I recommend that you read Bruce’s entire post, but I wanted to highlight a couple of thing that stood out. Take this part of his memo, for instance:

The code base is very fragile. A lot of it is bad old code that BigFirm didn’t have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value ‘3′, a constant named ‘ninety_eight’). Builds take all night. App releases don’t run acceptably, if at all, in a production environment. Developers check in files that won’t even compile.

Granted, this is an IT project and not an embedded project, but the principles remain the same. This is a perfect example of how project development costs can sneak up on you if you’re not careful. (Remember what I said about micro-leakage?) Sure your programmers are busy at their terminals, but what exactly are they working on? If you don’t know, you might have a problem.

Here’s another gem:

FUBAR was never properly architected and designed for the performance required. There is a current effort to increase performance after the fact, but the implementation makes that impossible. To make things worse, developers are having to scale the performance of and debug a seriously flawed application at the same time, making it very hard to stabilize the application.

Have you run into this before? Talk about trying to hit a moving target.

This paragraph, I think, arrives at the heart of the problem:

There isn’t enough intellectual honesty within the FUBAR project. Managers reject or explain away bad news and real problems, looking instead for people who will tell them what they want to hear.

A lot of design headaches can be avoided if the project leadership remain honest with themselves during the project. Ask yourself questions before the design process:

• Are my goals clear?
• Have I selected the right tools for the job?
• Is my development team up to the task?
• Are my deadlines feasible?

You need to be able to honestly answer those questions in every stage of the design process. If your project managers are more concerned with CYA politics than being honest with themselves and working to solve the problem, then the project is doomed to failure.

Oh, and if your current Linux project looks like the one that Bruce described, you might want to give me a call.

ShareThis

If you walk around any company that has a lot of software engineers, what do you see? You see a lot of people staring at terminals. What’s that guy doing? Well, he looks busy to me, but what’s he doing? He could be doing something that’s stellar or he could be doing something that’s a waste of time. How does management know that they’re doing either?

Let’s say that Joe Engineer downloads a profiling tool, and gets it working with Linux in a day. Let’s say he uses the tool the next day and it generates a 50 MB data file. The problem now is that he’s got all of the data, but can’t find the two data points that he needs. It becomes like finding a needle in a hay stack.

So Joe writes a PERL script to find the data points, which takes him three days to write. A week later, he writes another PERL script, and by the way, the guy down the hall just wrote the same script for a different project that he’s working on.

There’s three-plus days that are burned. If you look at Joe’s engineering hourly rate, you realize that he’s spent more in getting things to work with Linux than if they had just ponied up to buy some “ready-made” (no pun intended) tools.

The wastage (or micro-leakage as I like to call it) occurs in distributed small amounts that can add up quickly to a significant amount of money. Was Joe busy? Sure, he was really busy writing the PERL script. It was arguably a waste of time, but he was certainly working hard. If you look across a large company, you can see that there’s a real potential for an embarrassing amount of this insidious leakage.

In the hardware world, they have this wonderful, “gets your attention real quick” phenomenon that it costs you a million bucks to get a chip made, and if it doesn’t work the first time, it’s a colossal disaster, because you just burned a million bucks and you’ll get to spend another million fixing the problem. Because of that, nobody wants to be responsible for the first million dollar blunder, so they take great care in making sure that never happens.

You can burn that same million dollars in software, but it’s distributed, or leaked over time and over a group of developers. So it’s hard for management to understand how they’re wasting money, because it’s done an hour at a time, a developer at a time, in lots of small amounts. It’s sort of similar to how raindrops turn into the Mississippi River, or something like that.

So there’s the problem. How does a company know that it’s wasting time in software? Even in well managed companies, development costs can get away from them. We’ll explore some solutions in a future blog.

ShareThis

You can read my official response to that here. Kenton Williston also makes some interesting points here. But the debate brings up another issue I want to discuss.

For years, I’ve been saying an old phrase that goes, “You can’t hide from Linux.” If you look at the RTOS guys, ten years ago they were saying things like:

“Oh, you can’t use Linux at all.”

“It’s too slow.”

“It’s too big.”

“It’s not real time.”

“It’s written by 16 year olds.”

“It’s written by communists.”

Over the years, we’ve heard them all before, and we’ve blown all of those arguments away.

Now the RTOS guys are in a steady retreat from the onslaught of Linux, and from the looks of things, they’re starting to get desperate. We’re getting into the last bastions. They’ll say things like, “Well ok Linux can be used in set top boxes, but it will never be used in battery operated devices.”

Well, we knocked that one out the next year with millions of (battery operated) cell phones.

“Well, it can never be used in medical instruments.”

Again, that statement is untrue. We never made a lot of noise about it, but MontaVista Linux is in certified medical devices.

“Well, it will never be used in avionics.”

Wrong again. We are in avionics that also require certification.

My point is that there’s no place to hide. It’s just a matter of time. We’ve proven it. Just go back and look at the statements where the doubters have argued that Linux couldn’t be used, and we’ve just knocked them all down.

The RTOS players can fear monger all they want, but the truth remains the same. You can’t hide from Linux.

ShareThis

Although both consortiums are fundamentally different in their approaches, they both have the same end goal in taking down Microsoft in the mobile space. There is a real conundrum for the “anyone but Microsoft” crowd because if there isn’t going to be one company that counters Microsoft, then how do an aggregate of companies counter them in a way that’s coordinated? In fairness to Google and LiMo, that’s exactly what both consortiums are trying to do.

If you think about it, it makes complete sense. Where does Microsoft’s power on the desktop come from? It comes from the tyranny of all of the applications that run on Microsoft operating systems. That’s why it’s been impossible for Linux to displace Windows on the desktop.

With Android, Google has transferred that line of thinking from the desktop to the mobile device space. As it is with the desktop, the mobile computing space is once again all about the application. Google’s positioning on Android has made it all about having an environment where you can have lively, unpredictable application development because you never know where the next killer app is going to come from. So they’re trying to foster that in mobile computing, and everything else that they’re doing is a means to that end.

LiMo’s focus is arguably not as succinct, although in the end LiMo is still trying to do the same thing. However, I don’t think LiMo is as clearly focused as Google on the sole benefit being the application. Whereas LiMo has a broader focus, they are using different paths to the same end.

How much of an effect will LiMo or Google have on the mobile industry in 2008? That’s hard to say, especially since both appear to be a long way from releasing an actual product to the market. I will say, however, that MontaVista will come to market with real product in that space before either alliance does.

Even though MontaVista happily cooperates with LiMo and Google, I’m willing to bet that moving fast and getting a lot of units out there will be more defining than anything else. Who will be right? The market will dictate the answer to that, but it will be interesting to watch things as they unfold.

ShareThis

There are no set rules when it comes to open source, and each company has to decide how they relate to the community. From the beginning, we at MontaVista made the decision that we’d basically pour gas on the fire and be very active good citizens in the community, with the notion that the faster we make Linux embeddable, the more of a divot we take out of proprietary systems. Even though being very visible and active contributors might arguably empower our competitors, that’s the way it is. The bigger picture of eliminating the proprietary systems has always been more important.

However, in the embedded space, not all commercial Linux companies are created equal. Not all commercial embedded Linux companies value giving back to the open source community, or are even capable of making significant contributions for that matter.

If you’re trying to make a decision on who to trust when it comes to embedded Linux, follow the copyrights. The companies that are contributing back to the kernel in significant ways are the ones that have a true understanding of how to make Linux work for your device.

If you never touch the code, it’s obvious that you don’t have a lot of expertise in it. That doesn’t even factor in the fact that the software that you download from kernel.org is unfinished software, and if you don’t touch it and if your copyrights aren’t all over it, it’s full of bugs, it’s unstable, and it’s lacking in some technical areas that you have to add.

Commercial companies may leech off of the system and claim they are giving you “pristine” code. All that means is that they know how to download software, but what value are they adding? Anyone can download the stuff. So where is their expertise? What’s visible?

Just a little something to think about as you look at your options.

ShareThis

Well it’s pretty obvious that Linux world domination on the desktop is never going to happen. However, that doesn’t mean that Linux still can’t achieve the domination that the open source community has pined for all these years. The skirmish for the desktop may be lost, but the battle for computing is only beginning, and the new front is in the mobile computing space.

Although Linux ultimately failed to gain a strong foothold on the desktop, its use is exploding in mobile devices. What is happening is that a bazillion phones and little PCs and ultra mobile devices are all being built to run on Linux, and that is the path to dethroning Microsoft. Today there are as many of those devices on the net as there are Windows machines, and Microsoft knows it. That’s why they are pushing so hard to get into the mobile space.

As it stands, there are only two operating systems that look like they will survive: Microsoft, because they have infinite money and Linux because the rest of the world has infinite money. It’s simple for me to say, but that’s the way it breaks out. Basically the battle for computing is too big and costly for any one company other than Microsoft to win. So if someone is going to displace Microsoft, it’s going to be something that’s Switzerland-like, like Linux.

If the alternative to the monolith and monopoly of Microsoft is the rest of the world – as opposed to just one other company that can operate at the same level of uniformity – then the challenge and the weakness of the rest of the world is fragmentation and lack of interoperability, which is clearly what Google is trying to solve with Android and what Motorola is trying to solve with LiMo.

Google’s insight, which is true and certainly not unique to them, is the anchor that Microsoft has on the desktop that is impossible to displace is applications. It was a quirk of history that computers evolved the way they did, and that this sort of monolithic Wintel thing developed along the way. It’s arguably impossible to break that on the desktop.

However, the Internet relaxes people’s personal attachment to Wintel immeasurably, because they’re not dominated by applications that run on that device except for the browser, which is very neutral. As Google sees it, the browser is what will break up the Microsoft stranglehold. With the proliferation of Linux, mobile devices are now neutral and you can have a very capable device that runs no Microsoft software whatsoever. In fact, we at MontaVista are helping our customers build those things today.

We made a quote for IBM once, “It’s just about the future of computing.” That was true back then, and it’s just as true now.

ShareThis

Keeping that in mind, how the consortium of Google and everyone else handles the fact that they have no control over their foundation piece will be interesting. This is a problem that MontaVista has specialized in, of course, but that doesn’t change the fact that since the underlying platform that Android sits on is going to be old, Android either has to move forward (which presents its own set of problems) or it’s going to be out of date because Linux revs about every six months.

This is not an impossible problem, but it’s work and it’s an element that Google can’t control, which will be interesting to see how that aspect of it all plays out.

ShareThis

When you really understand Linux itself and the open source process, it is a clearly stated goal of the open source team (and the Linux team in particular) that the goal of the process is not to produce production quality, ready to go software at all. In reality, Linux is an R&D project. It’s a very fast moving R&D project that cycles every five to six months, and as part of the process the previous versions are essentially abandoned.

In that sense, Linux is a classic software engineer’s dream because it’s always about 90 percent done, and it’s been sort of roughly tested at best. It’s a wonderful process, but the process does not result in a commercial quality operating system. So, if you take a hard look at building your own variation of Linux, not only is it in general not worth your time to do this, but specifically, if you look at what you’ll actually have to do to the software (given that it was not designed to be used off the shelf) then it’s even more obvious that you don’t want to be doing this.

It’s an even more unstable starting point than you might have realized.

In that context, most business analysis would say that Linux is something that you’ll want to find a supplier for. There should be a software supply chain and that’s MontaVista’s role. We take all of that chaos with Linux and make it into something that you can inject into your product cycles and use. It’s something that you can really leverage, and in turn leverage your engineers to give you a different chain in values.

It’s pretty simple, and it’s why we’ve succeeded in the sense that we do have a lot of customers with all sorts of products that have gone to market and succeeded with our brand of Linux. At the highest level, it’s working, but I’ve seen surveys that say, yes Linux is going to be successful, and yes, people are going to use commercial Linux, but more people are going to roll their own. It doesn’t add up. I don’t understand why a company that makes instrumentation has engineers on their staff that are experts on the internals of a very complex operating system that their customers could care less if they’re using.

Even in our industry folks don’t understand software, and particularly don’t understand the software process and the costs. That’s why software projects are late, are buggy and run over cost. It’s a mysterious substance and you’d think that any manager, having an engineer with a CD with Linux on it with 30 million lines of source code, would say, “get that out of the building. We’re not doing well with the software we have already. Why do we want another 30 million lines of unstable code?”

It makes no sense to me.

ShareThis

Today, the temptation for device manufacturers to roll their own OS is still strong. The only difference between now and the old days is that the amount of software that we’re talking about now is huge. It’s probably a factor of 100 to 1000 more lines of software than it was back in the old days.

Presumably, when you download Linux, that source code is yours now, and as you start to modify it a bit and adapt it to your hardware, it’s sort of related to open source, but it’s your own copy now. But even though the open source software is “free” in a sense, there are still costs associated with it. So, if you’re considering rolling your own Linux, the question becomes is downloading Linux and adapting it yourself really cost effective? Should you be spending your engineering time doing that or should you buy a commercial version of Linux and have your engineers work on things that make your product better?

My answer is that in most cases, rolling your own Linux is a waste of time and engineering resources, especially if you think about differentiating your product. Every engineer should be oriented around whatever differentiation works in your market for your product, and you should have a supplier for the foundational pieces (such as Linux) that you need to work.

What’s remarkable is that many device manufacturers don’t even consider the commercial Linux options available to them. The common practice is for companies, under the notion that they think they’re saving money, to try the do-it-yourself approach. Unfortunately for them, the do-it-yourself approach generally costs more in development time than any temporary cost benefits they receive from sticking with open source in the first place.

That’s the fundamental value proposition from MontaVista is that even at the highest level, if you are capable of monkeying with Linux yourself, why bother?

ShareThis