I live in Atlanta and took some time last Tuesday night to attend CloudCamp Atlanta. I’ve worked with what we now call “cloud” technologies for quite a while. One of the highlights of my visit was a chance to meet Luke Kaines of Reductive Labs. As a few people around the office know I’m a big fan of Puppet. Luke was in town speaking at CCA from his native Nashville and I sat in on his session.

So if you don’t know about Puppet I’ll explain briefly. System administrators who run a _lot_ of systems can take at least two different approaches:

• Imperative: Write mountains of scripting code to diagnose what system administration actions need to be conducted.
• Declarative: Define a model which states how the system should look and let the tool bang the system into compliance.

Puppet is a declarative system. One uses a special domain specific language to describe a multi-system configuration. Puppet compiles this manifest and ships it out to nodes. Each locally installed version of puppet then tweaks the system until it matches the manifest.

I used Puppet on an internal project here at work… so that was why I was interested in meeting Luke. He had some interesting statements on the nature of open source community that I found intriguing and applicable to any realm of computing… even embedded. These aren’t quotes… just recollections.

A community will likely get you a lot of marketers and maybe some developers: There is a #puppet IRC channel on freenode.org that according to Luke routinely has 200+ developers sitting on it. There is also a
very active email list. These “marketers” are profoundly helpful and it was my observation that Luke didn’t mean it as a slight. He did go one to elaborate that there was really a very small core of developers who did core Puppet work.

Those marketers might be quite powerful: The room at 15-20 people in it and many of them displayed not just awareness of Puppet but actual expertise. That’s something for a tool which has minimal commercial backing and is rather young relative to its class (4-5 years). There was even, at the same time, a presentation on going at Linux.conf.au which was reportedly very popular.

In my mind I start to wonder what is the right balance between skilled developers and passionate marketers for a commercially backed open source project? Luke has taken a very open approach. Reductive Labs is a small shop right now that pays their bills and has a very passionate user and marketing base. Do they have the right balance for the future? I don’t think I’d be listening to Luke at all if it weren’t for his community of marketers.

What about our realm of embedded software? I often think about a few different mobile and open source projects/standards of recent origin:

1. Google’s Android
2. the LiMO Foundation
3. the LiPS Forum (now merged with LiMO)

or in the closer to earth real time Linux realm:

1. the Ingo/Gleixner RT work
2. ex-FSM Labs’ RT-Core
3. RTAI

who out of these two groups of projects/standards has the most effective balance of passionate marketers and skilled developers? I think I’d have to pick #1 on both of them… but that’s just my opinion. What’s yours?

Brad

I’ve worked with many customers on their open source and Linux strategies over the years. A reoccurring organizational pattern I’ve see is the “platform team”.

The platform team is usually only found in large corporations and it typically only created after a few embedded Linux based products have come and gone. Someday an executive in engineering or the CTO office declares that a central group should manage everything Linux related for product design and the team is created. Usually the first few projects go off well but after a while some cracks begin to occur.

Here are a few reasons I’ve seen platform teams fail to scale across the engineering community of these large companies:

Failure to consolidate: The symptom of this failure is that you’ll end up with several platform teams. It often gets so bad that you’ll see one or more per major division in the company. Executive fiats are often rendered to try and “fix” this problem but that is a seemingly difficult organizational edict to enforce due to technical limitations.

Failure to solve real project design problems: The symptom of this failure is design teams who start to quickly explore “alternatives” to the official platform due to technical requirements that just aren’t supported by the platform team. I’ve spoken with project leaders who said it would cost their platform team, and I kid you not on this, TWO MILLION DOLLARS to migrate their platform team created Linux to the new hardware design created by the project.

I spent time with a design team who’s central Linux platform team had allied with a competitor. This was over a year into the partnership and the project team had been given the thumbs down from the central Linux platform team. It seems that neither the central Linux platform team nor the vendor that they had allied with wanted to support the hardware platform they needed to use to meet their project’s performance requirements.

If you are a platform team and you aren’t serving your internal customers and your Linux commercialization partner can’t help you in this essential mission exactly what value are you adding to this game?

Failure to agree on how to split the tab: This is the one that seems to bite them all in the butt. The platform team does good work but the various project teams that want to use the platform can’t agree on how to fund the platform team and the maintenance of the platform. Each is willing to pay for the essentials to enable their project but none of them want to fund common technology investment.

Failure to remain connected to open source: Open source is the fount of all embedded Linux goodness. Platform teams who are more concerned about building the widget (their particular kernel and Linux release) than they are about building the process fall in this trap. This is why at MontaVista we emphasize that we help you to get the most out of open source. It isn’t about kernel 2.6.28… it is about the best way to get essential technology into your engineering team efficiently. Stay connected, build process, don’t become an island.

No ability to say “That’s a bad idea.”: Sometimes a project team will just have a bad idea. They’ll need something that isn’t right for the platform constituency at large. If you’ve built your platform team around consolidating on a set of particular widgets rather than a process to get the most out of open source you’ll have trouble. What if this project is your biggest funding source? Can you really say “no” to their bad idea?

If you are working at a company on a Linux platform team or are using the output of one of these platform teams I’d love to hear from you. I’m kicking around some interesting ideas. Comment lines are open.

Next week, November 18th, we’ll be running a great webinar entitled “Tips and techniques for improving embedded Linux startup time”. I encourage you to attend because one of our greatest assets here, Christopher Hallinan, will be speaking with Sridharan Subramanian of Freescale, and providing an overview of the boot sequence and kernel optimizations to consider for improving boot up. Chris, as you may know is has authored the number one embedded Linux book out there: Embedded Linux Primer.

Register and learn something new!

Amanda McPherson, Brian Proffitt and Ron Hale-Evans over at the Linux Foundation just released a new report on estimating the total value of the GNU and Linux software stack. If you’ve been around for a few years you’ve seen similar studies by David A. Wheeler and Bill Weinberg. All three of the studies use the COCOMO software costing model. COCOMO is an algorithm that uses some some algorithms to estimate the cost of a software codebase from the number of lines of code and the cost of the engineering team to write the code.

COCOMO spit out that Fedora 9 was a $10 billion dollar nugget of code.

Those out there with a more cynical mind will likely start complaining about the model, the cost of the engineer, the biases of the study authors, etc.

Let’s ignore all of that and just be brutally honest: There is zero financial incentive to create new competitive proprietary operating systems. The costs are too high because the barrier to compete is set very high by Linux. You won’t get rewarded by the market for writing a new block disk layer or scheduling algorithm under a proprietary license.

What does this mean for the embedded systems market? Lack of new competitive threats against the surviving proprietary operating systems. With few new competitive options the investment that might go into product improvement that could benefit the common user is withering. You can see it already… the real competitive front for many of these companies has shifted to milaero. They duke it out adding new features that 99% of embedded developers will never use.

So what to do? You’ve got to make your move and find a comfortable orbit around the world of GNU and Linux. That’s the innovation center. $10B is part of the proof of that. Next you need to find a way to get the most benefit from this new world of open source.

I returned home to Atlanta on Saturday still pumped about the great learning and fun that we had that week at the Vision 2008 Embedded Linux Conference. I wanted to share with you a few of them.

I’m sure you saw the splash on the Dell Lattitude ON announcement. Many of my MV colleagues have been nose to the grindstone for months on this project. They were asked to be the systems software provider (using Mobilinux 5.0) and to do the software integration for the application stack. I’ve not seen much of it myself personally but I hear good reports. Hopefully this will be a great success for Dell.

Since I was working and presenting at the event I didn’t get to attend many of the sessions. I did attend one of the last sessions of the conference by Matt Porter of Embedded Alley. His topic was the new Unsorted Block Image (UBI) and the UBI Filesystem (UBIFS) that can be layered upon it. What was superb about Matt’s presentation was that it wasn’t just a summary of information you could have learned off of Google. Matt went beyond the basics and delved into how to model your application’s usage profile to predict how quickly your product will consume the flash lifecycle of modern multilayer NAND flash memories.

I’ll repost if we can get a video of Matt’s presentation up on the web.

The other big highlight of the event was the Thursday night “Streets of San Francisco” party. This was the brainchild of our events manager Pamela Sherwood and it turned out great. The hotel ballroom was converted into street scenes from San Francisco. Tucked in the nooks and crannies was the dinner for the evening and ample adult beverages. All of this led up to what I think was the highlight of the evening… the Rock Band 2 show. I’ve played Rock Band quite a lot but it is a different experience with a stage, lights, and a real sound system.

We’ve got all the videos up on YouTube.

That’s our European FAE team giving a spirited rendition of Girls on Film.

Bootcamp!

The last few weeks at MontaVista have been quite hectic. Our Vision 2008 Embedded Linux Developer’s conference is finally here and getting everything prepared has been a huge effort for many of the MontaVista family. Now that the conference is in progress it is fun to see colleagues from around the world and meet embedded Linux enthusiasts from around the world. The photo above is from the “Bootcamp” training session being held right now. Most everyone has brought their laptops, loaded Linux in a virtualized environment, and are up and learning.

We’ve got two more days of the Vision conference and you haven’t missed out yet. If you are in the San Francisco area the get yourself to the registration page and use the promo code “BRAD”. You’ll get a discount that is steep enough you won’t mind having to upend your week to learn something very useful to your day job.

I’m having a great time at Vision… hope you can join us!

Brad

I just got off the plane from a great week visiting HQ in Santa Clara, CA. It is always fun to spend some time with my colleagues out there. Read to the bottom if you’d like to save some money and get some geek training.

I can’t say enough how thankful I am for everyone attending last week’s multicore webinar. The turnout was great. I enjoyed doing it and I hope you found it valuable. We’ve got some great upcoming sessions that we’ll mention closer to the broadcast date.

When preparing the session I tried very hard to focus on technology and techniques rather than products. That might seem a little odd coming from someone in the vendor community but I’m enough of a geek that I get a little bored listening to someone drone on about their datasheet. Let me know if that was the right balance for you in the comments, please.

If you didn’t watch the session you can catch the recording.

On the flight home I was reviewing presentations for the Vision 2008 Embedded Linux Conference. There are some really great sessions planned. If you’ve not heard Paul McKenney, Jonathan Corbet, or Chris Hallinan speak I think you’ll be in for a treat. Paul’s presentation wins kudos for best cartooning. I’m also very interested in catching up with Deepak Saxena on the work he is doing at One Laptop Per Child. There are also some hands on training events that I’m hoping I can shoehorn myself into.

I took some time in the office to strong-arm our event planner, Pamela, into giving me a discount code for my loyal blog readers. I could have asked for some of her chocolate chip cookies instead… but I was thinking of you. If you would like to attend Vision 2008 you can get a serious discount. Just go to the registration page and enter the discount code BRAD on the payment page. The discount isn’t as good as the student rate of $130 but it is darn close.

I’ll buy a beer for any blog reader (who I don’t already know from work) who says hello and joins me onstage for a song when we play Rock Band 2 on Thursday night.

Brad

Thank you to all of you who attended “Multicore 101: Migrating Embedded Applications to a Multicore Environment with Linux“. I am grateful to Ian Forsyth of Freescale as my co-presenter for the fruitful collaboration on this topic. Some quick reminders:

• Vision 2008: October 1-3, 2008 in San Francisco, California. This year’s top conference for embedded Linux developers will include a number of valuable technical presentations. Here are a few of them:

• Hands-on multicore Linux lab
• Linux synchronization mechanisms in driver development
• The Kernel Report by Jonathan Corbet of LWN.net
• Fast boot: Tips and techniques for improving Linux startup time
• Real time vs. real fast: How to choose? by Paul McKenney

Vision 2008 is sponsored in part by Freescale.

• Test Drive: If you’d like to take MontaVista Linux for a spin register for a free Test Drive session. You’ll be able to access a live system over the web including access to a Virtutech Simics powered hardware simulator.
• If you have more questions please enter your questions as a comment below. Ian and I will post all the answers we can get to.
• Subscribe to my blog for other posts from me concerning multicore, embedded Linux, security, the GPL and open source licensing and a little about MontaVista. Just click here for to subscribe via RSS.

Thank you again for your attendance and questions!

Brad Dixon

It seems a little odd to say thank you in a public setting to a customer you can’t name but that’s the odd spot I find myself in.

Back in 2003 I was drafted as an emergency fill in for a colleague at an important meeting. I thought I was just doing a quick favor. In actuality the potential customer had about a hundred fairly complicated questions to discuss about Linux in the telecom environment. Answering their questions turned into an all day meeting. I was exhausted at the end of it.

After a brief pause I’ve been associated with this customer almost continually. I’ve worked closely with them on technical, business, and of course around requirements for new products. I can say confidently that this engineering team is one of the most professional that I’ve ever worked with. Even when they are under fire they work with each other and their partners with respect.

They are also incredibly competent. The nature of their applications and their customers demands the utmost accountability for failure and expertise at every level of abstraction. One technical issue literally had to be tracked down to a defect in an upgraded firmware load delivered in a new batch of hard drives. The systematic analysis skills and discipline required to kill that problem are not demonstrated widely.

I’m taking on a new set of responsibilities at MontaVista and as such I’m having to, after 5 years, back away a bit from my favorite customer. As I do so they should know that they have my respect and that I’m always a phone call or email away.

Thank you!

Brad

It’s been a really busy week for me. Just put the wraps on a webinar that is premiering next week discussing multicore computing with Freescale. Ian Forsythe of Freescale will discuss the multicore challenge and their announced 8-core processor the P4080. I’ll follow up with a discussion of a strategy to migrate your application from single core to multicore.

Just a little taste of what will be in the session:

Linux’s inherent readiness for multicore

I hope you’ll attend. You can register for one of two timeslots:

• Thursday, September 11th 7 am ET /12 pm Western European Daylight

• Thursday, September 11th 10 am Pacific / 1 pm Eastern

Here’s the more official description:

When considering a move to a multicore environment, the number one
question embedded developers ask is “how do I leverage the work I’ve
already done in my applications?” Though multicore hardware has
provided improved performance for the embedded market, software
developers are challenged with taking advantage of the millions of
lines of single-threaded legacy code they have already written. This
creates a paradigm shift in how developers must think about and
implement future programs. This webinar will discuss the benefits of
embedded Linux as a solution with multicore and help systems engineers
consider what’s important as they make the transition. We will
conclude with an overview of MontaVista TestDrive, a virtual
evaluation of MontaVista Linux on a Virtutech Simics model of the
Freescale QorIQ P4080 and MPC8641D Power Architecture processors. We
invite attendees to sign-up for an evaluation of MontaVista software
on the latest Freescale multicore environments.