Bruce Webster has an interesting blog post about “Project FUBAR,” an IT project gone bad, really bad. Although Bruce describes what appears to be the IT software project from hell, it is a real world example of some of the things that I described in my last blog post.

I recommend that you read Bruce’s entire post, but I wanted to highlight a couple of thing that stood out. Take this part of his memo, for instance:

The code base is very fragile. A lot of it is bad old code that BigFirm didn’t have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value ‘3′, a constant named ‘ninety_eight’). Builds take all night. App releases don’t run acceptably, if at all, in a production environment. Developers check in files that won’t even compile.

Granted, this is an IT project and not an embedded project, but the principles remain the same. This is a perfect example of how project development costs can sneak up on you if you’re not careful. (Remember what I said about micro-leakage?) Sure your programmers are busy at their terminals, but what exactly are they working on? If you don’t know, you might have a problem.

Here’s another gem:

FUBAR was never properly architected and designed for the performance required. There is a current effort to increase performance after the fact, but the implementation makes that impossible. To make things worse, developers are having to scale the performance of and debug a seriously flawed application at the same time, making it very hard to stabilize the application.

Have you run into this before? Talk about trying to hit a moving target.

This paragraph, I think, arrives at the heart of the problem:

There isn’t enough intellectual honesty within the FUBAR project. Managers reject or explain away bad news and real problems, looking instead for people who will tell them what they want to hear.

A lot of design headaches can be avoided if the project leadership remain honest with themselves during the project. Ask yourself questions before the design process:

• Are my goals clear?
• Have I selected the right tools for the job?
• Is my development team up to the task?
• Are my deadlines feasible?

You need to be able to honestly answer those questions in every stage of the design process. If your project managers are more concerned with CYA politics than being honest with themselves and working to solve the problem, then the project is doomed to failure.

Oh, and if your current Linux project looks like the one that Bruce described, you might want to give me a call.

ShareThis

When I take a close look at how device makers make software decisions during the design process, I get the sense that some companies do a less than stellar job of managing their software activity. In fact, general state of affairs as far as what it costs to develop software and management’s take on how they’re spending their money sure looks like a chronic headache for management. So that begs the question, are you contributing to that headache by wasting time on things that your customers could care less about?

If you walk around any company that has a lot of software engineers, what do you see? You see a lot of people staring at terminals. What’s that guy doing? Well, he looks busy to me, but what’s he doing? He could be doing something that’s stellar or he could be doing something that’s a waste of time. How does management know that they’re doing either?

Let’s say that Joe Engineer downloads a profiling tool, and gets it working with Linux in a day. Let’s say he uses the tool the next day and it generates a 50 MB data file. The problem now is that he’s got all of the data, but can’t find the two data points that he needs. It becomes like finding a needle in a hay stack.

So Joe writes a PERL script to find the data points, which takes him three days to write. A week later, he writes another PERL script, and by the way, the guy down the hall just wrote the same script for a different project that he’s working on.

There’s three-plus days that are burned. If you look at Joe’s engineering hourly rate, you realize that he’s spent more in getting things to work with Linux than if they had just ponied up to buy some “ready-made” (no pun intended) tools.

The wastage (or micro-leakage as I like to call it) occurs in distributed small amounts that can add up quickly to a significant amount of money. Was Joe busy? Sure, he was really busy writing the PERL script. It was arguably a waste of time, but he was certainly working hard. If you look across a large company, you can see that there’s a real potential for an embarrassing amount of this insidious leakage.

In the hardware world, they have this wonderful, “gets your attention real quick” phenomenon that it costs you a million bucks to get a chip made, and if it doesn’t work the first time, it’s a colossal disaster, because you just burned a million bucks and you’ll get to spend another million fixing the problem. Because of that, nobody wants to be responsible for the first million dollar blunder, so they take great care in making sure that never happens.

You can burn that same million dollars in software, but it’s distributed, or leaked over time and over a group of developers. So it’s hard for management to understand how they’re wasting money, because it’s done an hour at a time, a developer at a time, in lots of small amounts. It’s sort of similar to how raindrops turn into the Mississippi River, or something like that.

So there’s the problem. How does a company know that it’s wasting time in software? Even in well managed companies, development costs can get away from them. We’ll explore some solutions in a future blog.

ShareThis