Last weekend that Amazon Web Services Simple Storage Service (S3) system was disabled for a number of hours. The outage affected a huge number of startups and established companies that rely upon S3 for the operation of their systems.

Systems fail… they always do. This isn’t a screed declaring that cloud services are unreliable or that Amazon ought to be excoriated for being fallible. Failures happen to all systems. Those who strive for faultless operation are often the most disappointed when the inevitable occurs. When customers are on their back they can often lean quite heavily on their partners to help them find answers.

The public results of the root cause analysis have been posted. The short of it:

…we found that there were a handful of [system status] messages on Sunday morning that had a single bit corrupted such that the message was still intelligible, but the system state information was incorrect.

The details are a bit terse but one can surmise that the monitoring and failure analysis mechanisms used to manage the system were the mechanism that caused the failure. As they put it “…when the corruption occurred, we didn’t detect it and it spread throughout the system causing the [failure]…”

This reminded me of the 1990 AT&T Network “crash”. The entry of one system into a failure mode in fact propagated the failure across the entire network.

It’s one thing to know the routine operational state of a software system. There are litterally thousands (more likely millions) of people across the globe that understand the routine operation of the Linux kernel and the associated protocols, userspace daemons, and applications to a high degree of competency. I count myself as one of them for a number of subsystems.

I have a high respect for the engineers I’ve worked with at MontaVista and elsewhere who understand their codebases so well that they can litteraly imagine the consequences of unexpected situations that real code finds itself in and the correlate symptoms into root causes.

When the unexpected inevitably occurs you’ll want one of these people available to you. I’ve seen them make a difference. What’s your plan?

Brad

ShareThis

Something that I didn’t expect when I joined MontaVista was how much I would depend on our customer support portal (MontaVista Zone, or MVZ) for my day to day work. I’m a cynic, I guess. I just figured it wouldn’t be that useful. Years later I’m still a fan.

If you’ve not heard of MontaVista Zone it is a our customer-only support website. You can see a screenshot of the home page here: MVZ Home. MVZ is how we deliver products to our customers, distribute updates, notify everyone of new product content, and help to educate our customers. We’ve had MVZ since the early days of MontaVista and it is about a terabyte of product downloads and customer accessible information now.

Here’s a quick tour of my favorite features.

• RSS Feeds: You can subscribe to RSS feeds of MVZ content. Pop that into Google Reader and you’ve got a great way to stay abreast of new MVZ content. [Note: If you had previously subscribed log back into MVZ and get the new feed URL. It was updated with the latest refresh of the MVZ.] You can also use Google Reader to search the feed and history very easily.
• Searchable FAQs: This is a new feature released last month. All of the MVZ FAQs are now easily searchable by edition, version, and subject area.
• Email Notices: This isn’t a new feature but I’ll mention it anyways. You can sign up for email notification of all MontaVista Linux related security features and what we term “special notices”.
• Tutorials: You also may not know that there are tutorials available on MVZ for command line and DevRocket powered debugging, memory leak detection, kernel build and debug, and other topics.

So that’s my quick list of favorites both old and new on MontaVista Zone. We are open to new suggestions and would love to hear about it in the comments.

ShareThis

The signature attribute of the open source development practices employed by most projects is that change is fast, furious, and non-stop. The codebases for most community backed projects is almost always in a rapid state of change. Many blessings flow from this process:

• Rapid innovation and quick embrace of new concepts and capabilities
• A spotlight is cast upon defects and bugs that inflicted past versions of the software
• The rapid pace of development is a foundry for up and coming developers. If they have the chops they can make a name for themselves by pioneering new work. New members are the lifeblood of a vital and living community.

The disadvantage of these rapid code communities is also obvious: there is so much change that the codebase can become hard to grok.

Vendors can help by putting the brainpower (or sometimes just the boring brute force) required to grok the changes and backporting these changes to the version of the software employed by your product. This adds value by both through the creation of intellectual property (in the purest form… the understanding) and artifacts (the modified and tested patch).

There can literally be hundreds of thousands of lines changed between the product the customer is using and the version where the fix is expressed. Pulling out the few dozen lines that manifest the fix and backporting it reduces risk to the customer that has a fielded product.

Now I’m no pollyana who sees the world as only being helped by the behavior of vendors and the wide world of the open source communities as being some great unwashed morass of decrepit code. That’s not true. Thomas Gleixner at last year’s CELF 2007 opening keynote made some profound observations about the difficulty balancing community originated and vendor managed codebases. As both a recognized Linux kernel community participant and a vendor he has a great perspective on these matters.

The real conundrum is that change has real and concrete risk. Moving fast can be at odds with moving safe in the eyes of developers who are responsible for building commercial products rather than community projects. I’ve seen customers of ours be suspect of even taking periodic well understood kernel patches that solved identified problems in their products. Some commercial products are even prohibited from being updated.

No one person is right. No one group is wrong. There is just a real impedance mismatch between community speed and commercial speed in almost all cases that I’ve observed. Every company’s open source strategy (you do have one, right?) has to recognize and deal with this impedance mismatch.

ShareThis

Pogue’s latest post in the NY Times Blogs about technical support is sure to strike a chord in anyone who has ever bothered to call a technical support line. Every once in a great while I’ve bothered to do so it I have to admit a certain degree of loathing to the whole process. What I hate about it is always getting asked the “dumb” questions. Read the rest of this entry »

ShareThis

Last post started with a link to a humorous blog about some of the, ahem, “interesting” questions that some support teams get asked by their customers.

I can say, luckily, that our customers are likely far above the average. I respect what they do, the projects they pull off, and the questions they ask. I’ve even learned some great lessons from them and that’s the subject for this post.

First of all I think of a television show I saw a few years ago. The show was detailing the inner operations of a US nuclear submarine. My brother in law served as a nuclear engineer on one of these boats for several years. Both the show and my conversations with my brother in law convinced me that what makes a nuclear submarine “safe” to operate isn’t technology as much as it is process. There is no master computer that can make all of the risk of operating a combination nuclear reactor, weapons platform, dormitory, and submersible vehicle disappear. It is inherently risky. What makes the risk palatable, manageable, and acceptable to otherwise sane sailors is the process and the training and discipline of those who execute the process.

The boat is safer because the sailors and what they do make it safer.

Could technical support be better if we, both users and providers of technical support, made it better by the processes we follow? I’ve seen cases where I think that to be true.

At MontaVista we offer technical support to our customers for the embedded Linux distribution and a wide array of Open Source software that constitutes our product. I’m not on the support team but I’ve been on the sidelines enough to have made some observations. Most of these are guided by a key customer I’ve worked with for almost 4 years. They are top notch engineers and seem to excel at getting high quality support out of all of their vendors. They’ve shown my how their processes help get the best out of vendor support:

1. Help teach your vendor support team about your expectations, the kind of projects that you are engaged on, and the needs that your team will have in the upcoming months. Have these details ready before they are needed.
2. Be personal, friendly, and professional. Real people are on both ends of the email chain.
3. Both sides should be vigilant about follow-ups and meeting expectations.
4. Introduce technology providers who are delivering to the same customer product. You never know when they will have to work together to tackle a technical issue.
5. The customer’s expectations matter the most but it is important that the vendor support team is being asked for something they can reasonably be expected to deliver on. Work in advance to make sure both sides understand what that is. Find better support channels to the vendor if the ones you have aren’t right for the customer’s expectations.

If markets truly are conversations than it is likely essential to make sure that the conversation of technical support transcends emails and trouble tickets. Find ways to go beyond just the support email.

So in the comments I’d love to know if you would ever consider participating in…

• private blogging between customer and tech support to keep up-to-date on what’s new in your projects?
• a shared private wiki so customers and tech support folks could cooperate closely?
• social worknets to glue partners together with their mutual customer?

ShareThis