I just flew into our Santa Clara, CA office today. I’m a band geek… I played trombone, marched in high school, and marched for two years with the Georgia Tech Yellow Jacket Marching Band. Lots of fun!

I learned a lot in band and was reminded of it when I arrived at our corporate offices in Santa Clara, CA. Since school is out now there are teams of devoted musicians who are participate in drum and bugle corp competitions all around the US. One of the best known and elite corps, the Santa Clara Vanguard, practices in a parking lot about a 3 minute walk from the office. If you’ve ever heard a corp play in competition you’ll always know the distinctive sound when you here it. Listening to a corp is an experience your entire body participates in… not just your ears. It is that powerful.

One of the most important lessons I learned from marching was that even the most simple of tasks is profoundly complicated if you are trying to get a lot of people to do it together. We’d line up 20 high school band students, count down from three, and expect each of them to take one measured step with their left foot. This simple exercise could take an hour to get right. Double the number of people and it would likely take 4 times as long to get it right.

Mark Shuttleworth (Ubuntu’s BDFL) recently called for cooperation and release synchronicity in a recent blog post. His main argument was that if various major distributions agreed on a major release schedule that their efforts within the upstream communities could be coordinated. The distributions could also agree on a kernel, gcc, glibc, and major library versions to unify around and thus make life easier for ISV’s. Interesting idea… very user focused as one would expect from Mark.

There are competitive angles on this, of course. Ubuntu is probably 3rd or 4th place in server adoption behind Red Hat and Suse. That’s a cold place to be. Large ISV’s don’t really care beyond #1 and #2 unless they’ve got a key customer that wants you. Red Hat, of course, doesn’t see this as a problem that needs solving.

Synchronicity is also a challenge not only between Linux distributions but within a given Linux distribution. It is incredibly challenging to align the hundreds of community backed components that constitute a typical Linux distribution and get them to all be ready to provide to customers at the same time. This truth has often been misinterpreted as a criticism that these community projects are not suitable for production usage. That is not true… what is true is that there aren’t all suitable for production usage at the same time. Getting all of these components to take one measured step, with the same foot, in the same direction, at the same time is a profoundly non-trivial challenge.

Getting this to work between companies might be too much to expect without a massive groundswell of user and customer demand that would be needed to focus everyone’s attention. It is not clear to me that this demand is there.

Brad

ShareThis

Where we last left off our intrepid support team was just getting the FedEx package with the customer’s hardware in house. At this point, as the customers FAE, I’m pretty sure that the customer is going to rip me a new one any moment.

Thankfully (I appreciate it Geoff) he didn’t do that.

Since the error had been in reboot we were using some scripting to automatically race through the cycles. The kernel has a sequence it uses in booting up and not all of it actually produces printf output. You can also have issues where printf has been sent but not displayed. Adding printf instrumentation can be helpful… but it give you certain information. Some of you out there are thinking JTAG… but this was x86. Sorry.

The approach you take on this kind of defect is that you start simplifying the system. The customer had done some of this but we wanted to quickly duplicate his progress to make sure it was accurately conveyed. Then you might also start disabling BIOS features. Our support engineer soon noticed a correlation between the hang and activity happening on the parallel port I/O ports.

With a solid lead in hand the engineer took to just abusing the parallel port directly using a userspace test application. This particular system, as it turns out, had a hardware fault. Reading repeatedly from port 3Bc would, within minutes, cause the entire hardware platform to lock up.

Next the engineer went and wrote an x86 assembly code program that booted from a floppy that abused the port as described. This independently proved that the fault was in the hardware, not Linux.

I’ll probably always remember this issue because it wasn’t until right at the end that we became convinced that it wasn’t a software issue. We always suspected… but could never confirm.

Next debug horror story is about IDE drives.

Thanks for reading,

Brad

ShareThis

I was an Atlanta based event (StartupRiot.com, other coverage) yesterday and I was asked a few times about what MontaVista did. I mentioned that we helped our customers to get the most out of open source as they build their commercial devices.

Invariably the first comment I got was: “Oh… embedded software. So that’s doing Linux in really small systems, right?”

Well… sometimes. We’ve got many customers who are trying to build very small products that utilize a minimal amount of flash. We’ve also got customers who build huge systems that use multicore chips with 16 processors and many gigabytes of RAM.

That’s the great thing about Linux… it is scalable such that it can be adapted to reach the sweet spot on most designs without undue effort. It may not be appropriate for the extremes but it covers the great swath of everything else.

ShareThis

[ This is first in a series of some of the most unusual, challenging, or just plain odd support engagements I’ve observed over they years. Names and companies have been changed to protect the innocent. ]

CASE #1-M6P7/1771

LOCATION: New Jersey

(dun-DUH)

Customer reported a periodic hang when booting their system. The only direct symptom was the that the last kernel message displayed was: “Starting kswapd v1.8″. The behavior seemed to be a heisenbug since the frequency of the hang was only once every 10 boots. Some hardware hung more, some hung less. At times hitting the physical reset button recovered the system. One particular hardware instance would hard-lock resisting all recovery methods unless the system was hard power cycled by removing the AC plug.

Uh oh… that’s not good. Was the software so mucking the hardware up so much that a cold power cycle was the only thing that would fix it?

The customer jumped into a range of experiments adding and removing various hardware components and altering the amount of “power-off rest time” between test cycles. The results were conclusive: There is no one thing that made the situation better. Tweak the GigE, SMP, RT scheduler, boot device, nothing mattered.

On the MontaVista side we had been discussing this issue in depth with the customer all along and suggesting various tests to try. We also confirmed that we couldn’t reproduce the issue on the hardware that we had in house. This was an issue confined to the customer’s specific brand of x86 motherboard that they purchased. We were concerned because we sometimes see odd hardware attributes on custom hardware but when a commercial product is used the incidents of hardware weirdness are far less frequent.

Most of our customers are using hardware that MontaVista doesn’t have in our lab. That’s the nature of the embedded software industry. This imposes some natural constraints… we can’t kill defects that we can’t reproduce or observe reliably. Sometimes the only thing that can help is to get the customer’s hardware in-house.

FedEx to the rescue. What happened next was quite a surprise to me.

(dun-DUH)

TO BE CONTINUED.

ShareThis

Recently exposed in the NY Times and picked up on by good folks such as Bruce Schneier (read his whole post… it is a great summary) is an alarming study conducted by the Medical Devices Security Center identifying a host of privacy and integrity attacks that can be implemented by exploiting the wireless interface between a pacemaker and the external control unit.

Wow… being pwned never seemed like such a direct threat to human life.

The above is a snippit from the paper describing the kinds of attacks they were successful in demonstrating in vitro.The paper is fascinating. Part of their attack tooling was the Free GNU Radio project. GNU Radio and some radio interface boards created to be compatible with GNU Radio were used to both analyze the over the air communications and generate the transmissions used for the active attack.

What gets me is that while this is clearly a complex attack I’ve seen the genesis of these assaults first hand. I recently viewed a multi-page requirements document for an innovative device. Thousands of man hours of labor were implied by the requirements. There was a line item that simply stated one requirement:

• Security

That’s it. A single bullet.

I’ve seen much better from other folks. I’ve actually been in security reviews with customers, discussed the risks and security objectives of their designs. I’ve met folks who had that security mindset that can at once be confounding yet stimulating. Most, however, just see security as something that is bundled in a package (like the great OpenSSH) or is the result of installing all the updates.

There is more to it. Read some Ross Anderson. Read some Bruce Schneier. I’m just a security engineering wannabe but I still learned a lot.

Use your vendors, too. No one wants to see their product implicated as part of the next great security failing. MontaVista has people who can help you to understand your risk profile. If technologies are the right answer we can help you get the most out of Open Source to address those security risks.

But we don’t, and never have, sold security by the kilogram. Sorry.

ShareThis

Something that I didn’t expect when I joined MontaVista was how much I would depend on our customer support portal (MontaVista Zone, or MVZ) for my day to day work. I’m a cynic, I guess. I just figured it wouldn’t be that useful. Years later I’m still a fan.

If you’ve not heard of MontaVista Zone it is a our customer-only support website. You can see a screenshot of the home page here: MVZ Home. MVZ is how we deliver products to our customers, distribute updates, notify everyone of new product content, and help to educate our customers. We’ve had MVZ since the early days of MontaVista and it is about a terabyte of product downloads and customer accessible information now.

Here’s a quick tour of my favorite features.

• RSS Feeds: You can subscribe to RSS feeds of MVZ content. Pop that into Google Reader and you’ve got a great way to stay abreast of new MVZ content. [Note: If you had previously subscribed log back into MVZ and get the new feed URL. It was updated with the latest refresh of the MVZ.] You can also use Google Reader to search the feed and history very easily.
• Searchable FAQs: This is a new feature released last month. All of the MVZ FAQs are now easily searchable by edition, version, and subject area.
• Email Notices: This isn’t a new feature but I’ll mention it anyways. You can sign up for email notification of all MontaVista Linux related security features and what we term “special notices”.
• Tutorials: You also may not know that there are tutorials available on MVZ for command line and DevRocket powered debugging, memory leak detection, kernel build and debug, and other topics.

So that’s my quick list of favorites both old and new on MontaVista Zone. We are open to new suggestions and would love to hear about it in the comments.

ShareThis

Last month David A. Wheeler hosted a can’t miss webinar for those of you in the defense industry that have questions regarding the GPL and how that impacts your plans to either use or create open source software. I attended the webinar and David’s analysis was quite enlightening.

• “DoD and OSS” - OpenDocument
• “DoD and OSS” - PowerPoint
• “DoD and OSS” - PDF
• A audiovisual presentation is available via Google
• The Q&A.

There were enough questions afterwards that David wrote up a slew of questions covering a dozen different categories. Read the Q&A… do not skip it.

If you would like to hear a complimentary webinar that is focused on commercial usage of open source within embedded design projects you should take some time to listen to noted legal expert Jason Wacha’s presentation:

• GPL and Open Source Licensing Review: Join MontaVista Software and distinguished open source licensing authority Jason Wacha for a review of issues surrounding open source licensing. Starting from base principles of differences between laws, licenses, and commercial agreements, Mr. Wacha moves on to issues surrounding the use of a General Public License (GPL) and how it affects Linux and other open source software.

You’ll need to register but I think it is worth the trouble. I’ve listened to Jason’s GPL talks at least a dozen times and I still come away learning something new each time.

ShareThis

A short thought about the nature of communities and freedom has been bouncing around my head. Back in 2003 my wife and I took a few days to visit Boston. I’d made many trips there for work but typically only saw Logan airport and the ‘burbs. Boston has a lot more to offer than office parks.

We did the typical history walk through the city. Outside the Massachusetts State House is a statue of Mary Dyer who is remembered a Puritan turned Quaker who was hung for violating colonial law banning Quakers.

The sad irony in all of this is that the Puritan colony of Massachusetts was ostensibly started to preserve religious freedom. This sanctuary, however, seemingly applied only to Puritans and not to those of other faiths.

This was just a lesson to me that freedom for one class does not at all imply a respect of freedom for all or for all purposes. Tag that under “valuable_life_lesson”.

Remember this when evaluating open source communities. There is often a shared ethic of Free Software but sometimes that is all that is in common from one community to the next. One community might be open to new developers or alternate ideas and implementations. Others are relatively closed. Unlike denial of religious freedom, however, it is not possible to simply declare that all suppression of a community member’s freedom is repressive.

First of all… the right to modify a community codebase is not an expected freedom under any Free or Open Source software license that I’m aware of. Membership and the right contribution is a privilege extended by the respective communities under bylaws that rarely are documented but are there and should be respected.

Some communities do document their membership practices. Gentoo has an explicit membership process. Debian has a social contract the sets shared community expectations. Other communities have social processes that are part of the ambient structure of their world. You are advised to learn the implied procedures and policies if you intend to make progress and achieve you goals.

Secondly some degree of community control is likely to be seen as essential to the proper functioning of a FLOSS community. This is not some sort of hippie concert. Folks are making real software here. Linus Torvalds is jokingly referred to as the benevolent dictator… but his authority is real. Decisions need to be made and stuck to.

Sometimes, however, the decisions of the community leaders can and do cause division and frustration. Recently I was speaking with a commercial developer who had been working with a prominent open source distribution community for embedded devices. He expressed a high degree of frustration with this project and stated that there were “zealots” (his words) and this was keeping everything in a state of flux. His commercial project was, to a degree, hampered by this constant change and breakage. As his software stack expands from simply kernel and uClibc to graphics and windowing systems this breakage becomes more challenging.

I highly doubt this developer will be hung for his heretical statements. Nor are the members of this community anything like the Puritanical prosecutors of yore. I am also explicitly avoiding digging into the whole commercial vs. community supported argument. That’s for another day.

What I’m really trying to say is that when you evaluate a software base look at more than just code and the license. Try to understand the community that backs the project so that you can understand the operational dynamics of the community. How does it really work? Is the reality of this community going to serve your needs?

ShareThis

The signature attribute of the open source development practices employed by most projects is that change is fast, furious, and non-stop. The codebases for most community backed projects is almost always in a rapid state of change. Many blessings flow from this process:

• Rapid innovation and quick embrace of new concepts and capabilities
• A spotlight is cast upon defects and bugs that inflicted past versions of the software
• The rapid pace of development is a foundry for up and coming developers. If they have the chops they can make a name for themselves by pioneering new work. New members are the lifeblood of a vital and living community.

The disadvantage of these rapid code communities is also obvious: there is so much change that the codebase can become hard to grok.

Vendors can help by putting the brainpower (or sometimes just the boring brute force) required to grok the changes and backporting these changes to the version of the software employed by your product. This adds value by both through the creation of intellectual property (in the purest form… the understanding) and artifacts (the modified and tested patch).

There can literally be hundreds of thousands of lines changed between the product the customer is using and the version where the fix is expressed. Pulling out the few dozen lines that manifest the fix and backporting it reduces risk to the customer that has a fielded product.

Now I’m no pollyana who sees the world as only being helped by the behavior of vendors and the wide world of the open source communities as being some great unwashed morass of decrepit code. That’s not true. Thomas Gleixner at last year’s CELF 2007 opening keynote made some profound observations about the difficulty balancing community originated and vendor managed codebases. As both a recognized Linux kernel community participant and a vendor he has a great perspective on these matters.

The real conundrum is that change has real and concrete risk. Moving fast can be at odds with moving safe in the eyes of developers who are responsible for building commercial products rather than community projects. I’ve seen customers of ours be suspect of even taking periodic well understood kernel patches that solved identified problems in their products. Some commercial products are even prohibited from being updated.

No one person is right. No one group is wrong. There is just a real impedance mismatch between community speed and commercial speed in almost all cases that I’ve observed. Every company’s open source strategy (you do have one, right?) has to recognize and deal with this impedance mismatch.

ShareThis

So I’m working with a group at our company that is fond of shared Microsoft Exchange Calendars. I’m a pragmatic fellow and while I have run Linux as my day-to-day platform for 10 years now I just realize that the world sometimes sees stuff differently from me. Whatever… I really don’t mind. It’s just a program. Run it.

Let’s just boot Windows in a VMWare virtual machine and run Outlook’s Calendar. Easy as pie. Wrong.

I use OpenSSH to tunnel into our work network. I map a slew of local ports to remote machines over the SSH connection. So what ports need to be mapped to the Exchange server. Answer: All of them.

Exchange is built upon MS-RPC which is a tweaked version of OSF DCE/RPC and all of these protocols seem to just love to dynamically allocate ports to serve individual clients. I saw it in Wireshark myself. Outlook connects to the exchange server over port 153 and the reply points Outlook to another port to continue the conversation.

At this point I am filled with colorful epithets that I’ve disciplined myself not to utter verbally.

I know very little about all of these protocols. Wikipedia this stuff:

• “MSRPC (Microsoft Remote Procedure Call) is a modified version of DCE/RPC.”
• “DCE/RPC stands for “Distributed Computing Environment / Remote Procedure Calls”.
• “Distributed Computing Environment (DCE) is a software system developed in the early 1990s by a consortium…”

Eh… hold it. A consortium? Oh, that explains some of this. Tell me more…

“As part of the formation of OSF, various members contributed many of their ongoing research projects. At the time, network computing was quite popular, and many of the companies involved were working on similar RPC-based systems. By re-building these various utilities on a single “official” RPC mechanism, OSF could offer a major advantage over SVR4, allowing any DCE-supporting system (namely OSF/1) to interoperate in a larger network.”

So MSRPC is based on DCE/RPC which is based on a bunch of corporate R&D cast-offs? (maybe cast-offs is a little harsh)

I’m just dubious about a closed door corporate consortium getting stuff right. There is a huge difference between a community centric development model with corporate participation (like the Eclipse community) and a corporate consortium populated with commercial actors even if the intended output of the consortium is to be licensed under a FLOSS license.

I have no clue if a real community would have reigned in the madness of dynamic ports. Maybe they were de rigueur at that time. I have a hunch it would have been different, though.

ShareThis