Open Device, Insert Code is no longer an active blog, but, we felt there was a lot of great information here and didn’t want to take it down. We’ve left the archives for you to read through and even comment on if you like. Please continue to enjoy this and all of our other blogs!

So what do you do when you get a new toy? You race around trying out all of the fun stuff you can do with it, of course. It is no different with me now that MVL6 is in beta and being used by developers.

As previously mentioned the integration platform capability of MVL6 is a big new feature. My last post mentioned many of the benefits including more easily reproducible builds and clear traceability of all build inputs to the outputs. A simplified diagram of the system looks like this:

The entire process is controlled by recipes and there is full transparency and traceability between source and metadata inputs and the build products that are produced. What kind of build products are there? Quite a few, in fact.

• Prebuilt filesystems that can be deployed to targets
• Packages in a variety of formats that can be used for deployed device upgrades
• Various manifest files describing what was placed in the built images
• Original source archives and patches suitable for distribution to satisfy various licensing obligations

As an example, after doing a quick test build of the busybox and less software packages you can see the source code ready for distribution:

$ ls tmp/deploy/sources/*/tmp/deploy/sources/BSD/:

less-418.tar.gz  less-418.tar.gz.md5tmp/deploy/sources/GPL/:

busybox-1.13.2-depmod.patch      busybox-1.13.2-modprobe.patch

busybox-1.13.2-depmod.patch.md5  busybox-1.13.2-modprobe.patch.md5

busybox-1.13.2-init.patch        busybox-1.13.2.tar.gz

busybox-1.13.2-init.patch.md5    busybox-1.13.2.tar.gz.md5

busybox-1.13.2-mdev.patch        busybox-1.13.2-tar.patch

busybox-1.13.2-mdev.patch.md5    busybox-1.13.2-tar.patch.md5

Is that source tarball actually the real original unmodified source from the upstream project? Let’s check and see by verifying the cryptographic signatures:

$ cp tmp/deploy/sources/BSD/less-418.tar.gz ./
$ wget ftp://ftp.gnu.org/gnu/less/less-418.tar.gz.sig
$ gpg -v less-418.tar.gz.sig
gpg: assuming signed data in `less-418.tar.gz'
gpg: Signature made Tue 08 Jan 2008 05:18:56 PM EST using DSA key ID 33235259
gpg: requesting key 33235259 from hkp server subkeys.pgp.net
gpg: armor header: Version: SKS 1.0.9
gpg: pub  1024D/33235259 2004-12-04  Mark Nudelman <markn@greenwoodsoftware.com>
gpg: using classic trust model
gpg: key 33235259: public key "Mark Nudelman <markn@greenwoodsoftware.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Good signature from "Mark Nudelman <markn@greenwoodsoftware.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: AE27 252B D684 6E7D 6EAE  1DD6 F153 A7C8 3323 5259
gpg: binary signature, digest algorithm SHA1

That looks good. How about busybox?

$ cp tmp/deploy/sources/GPL/busybox-1.13.2.tar.bz2 ./
$ wget http://www.busybox.net/downloads/busybox-1.13.2.tar.bz2.sign
$ gpg -v busybox-1.13.2.tar.bz2.sign
gpg: armor header: Hash: SHA1
gpg: armor header: Version: GnuPG v1.4.6 (GNU/Linux)
gpg: original file name=''
gpg: Signature made Tue 30 Dec 2008 10:37:37 PM EST using DSA key ID ACC9965B
gpg: requesting key ACC9965B from hkp server subkeys.pgp.net
gpg: armor header: Version: SKS 1.0.9
gpg: pub  1024D/ACC9965B 2006-12-12  Denis Vlasenko <vda.linux@googlemail.com>
gpg: using classic trust model
gpg: key ACC9965B: public key "Denis Vlasenko <vda.linux@googlemail.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Good signature from "Denis Vlasenko <vda.linux@googlemail.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C9E9 416F 76E6 10DB D09D  040F 47B7 0C55 ACC9 965B
gpg: textmode signature, digest algorithm SHA1

That checks out, too.

Now your developers and management can be assured that when it comes time satisfy any license obligations you have an easy and repeatable process for ensuring that the sources corresponding to distributed binaries is at hand.

If you’ve read up on MVL6 you’ve seen that we describe the BitBake tool as being used as the core of our new MontaVista Integration Platform. I’d like to explain what that means, from a technical standpoint, and how it rocks.

OpenEmbedded (OE) is a widely used, and I would consider a defacto standard, for community engineered embedded Linux distributions. OE is has been around for 7+ years and has an active contributor base of both commercial, independent contractor, and enthusiast developers. OE is what I call a family of related Linux distributions that share some common infrastructure. Each development team then customizes their distribution to meet their requirements. Some examples.

BitBake is one of these common infrastructure tools. BitBake is analogous to “make”. BitBake analyzes a set of directives and then builds a task dependency tree to satisfy a user command. BitBake then executes the defined tasks to completion. When paired with the OE metadata BitBake can compile from source to create the host development tools, cross-development tools, target binaries, and system flash/disk images needed.

On a technical basis MVL6 is, because of our use of BitBake and compatibility with the BitBake recipe syntax and the OE metadata, a peer to other OpenEmbedded-style distributions as listed above. Just like these peer distributions MVL6 has unique requirements driven by our customers.

For the MVL6 development tools we’ve invested to create several compelling benefits for developers over every option on the market both commercial and non-commercial:

• Dirt simple and fast installation. MVL6 is the fastest and easiest embedded Linux distribution to install and get productive in. I’ve used a bunch of them… commercial and non-commercial. Trust me on this. We’ve got a small required core to install and everything past that can be done incrementally on demand.
• It just works. When I interviewed developers about their experiences using several prominent community embedded Linux distributions there was a clear and consistent message: They were hard to use and took a lot of help and tweaking to get going. It is infuriating to kick off a build only to see it fail 7 hours into an 8 hour process because you were missing some important host tool or a website with the source code went down. MVL6 gets you started fast with sensible defaults, prebuilt binaries, and a quick path to get your hardware booted.
• One file defines your design. Using the extensions we’ve added you can use a single configuration file to define your entire design. We’ve got helpful docs to explain how to customize your project while not making an un-maintainable mess.
• MVL6 can start small and build up incrementally. Sometimes developers have to squeeze their designs to fit into flash. MVL6 starts with a small image and lets developers add incrementally. They can even break the bounds of restrictive binary-only distributions and downsize within software packages by removing features or files.
• MVL6 doesn’t cut you off from the world. While we provide a complete Market Specific Distribution (MSD) for your design you can, if you wish, supplement our product with components from OpenEmbedded. We’ve retained compatibility to give you great options.
• MVL6 doesn’t lock you in. Most commercial embedded Linux build systems are either under an ambiguous proprietary license or are so esoteric to be classified as vendorscript. MVL6 won’t lock you in like that due to its open core and usage of a defacto standard recipe syntax.
• A logical update system. MontaVista periodically releases updated software components to fix bugs and offer enhancements. Getting these updates into your MVL6 project is simple and risk free. Lock your project down so version updates don’t happen without permission. Pull down the updates using a simple automated tool. Try them out by commenting the version lockdown line. If you don’t like it just uncomment the version lockdown and no harm done.
• Works great behind a network proxy or even offline. Often developers have to work on restricted lab networks. Rather than depending on a slew of public HTTP, CVS, git, and Subversion servers across the Internet there is a single source for every original source archive and patch that goes into your MVL6 powered product. You can access the MontaVista Zone (MVZ) Content Server from behind a proxy or quickly mirror it for your own offline operations.
• We don’t even lock you into MVZ. Let’s say its 2018 and after the Great Quake of California that flooded the Bay Area you want to update your MVL6 powered design with some additional software. You forgot to download the sources for the libfoo package and now MontaVista is at the bottom of Lake Santa Clara. Luckily the MVL6 Integration Platform is smart enough to go to the original upstream repository. You go on as a happy developer.
• Better build portability. Often our customers need to be able to reproduce their builds for 10+ years into the future. The problem is that as their development PC’s get replaced they have to upgrade to new host Linux distributions. We’ve made sure MVL6 depends only on the well defined Linux Standard Base (LSB) components of the host operating system. This assures developers that, to the extent that is feasible without a crystal ball, their builds will be reproducible on future Linux distributions that have not yet been created.
• Better build reproducibility. What builds today has to build in 10 years. We’ve got MVL6 dialed-in so you can easily configuration manage the system without having to check 100,000 files into your revision control system. That’s goodness.

So that’s a long little love letter for the product. There are lots more cool tricks to show. Mark you calendars for an upcoming webinar by my esteemed colleague Nick Pollitt on all manners of MVL6-foo.

It is hard to imagine that it has been about 24 days since Meld launched. If you don’t know what Meld is it is a community for developers building products using embedded Linux. One of the most common questions I’ve had about Meld has been if it is truly for all embedded Linux developers or if it is just some sort of trick “we all talk about MontaVista and nothing but MontaVista” community.

Meld is really about embedded Linux. Take a look at the Wordle below:

That was built off of the “recent discussions” page. Does that put it all into perspective?

It has been a few weeks since Meld launched and I wanted to mention a few updates on it. We’ve had terrific uptake on Meld both in terms of sign-ups and in participation. The conversations have been what I would characterize as appropriately geeky. We’ve also had some great feedback from our Meld community on how we are doing:

Just join today to meld. Thanks for Montavista for hosting this forum. — from izjenie

Thanks for all the great responses guys. There was a lot of information gleamed in this thread. — from ngogineni

Meld is also covering a broad array of topics which I think is critical for those of us who are working with embedded systems. We can’t just specialize in one code base. Meld has hosted discussions on:

• drive to userspace communication
• Beagle board accessories
• booting linux in a multi-processor environment

You’ve also got another chance to learn about how community can help make your embedded Linux project easier at an upcoming webinar on Thursday March 26th. The two hosts for this webinar are Jonathan Corbet (founder of LWN.net… go subscribe now) and Troy Kitch of MV. Should be interesting to see how they discuss “Addressing the top 3 challenges of embedded Linux development with community“. Meld isn’t, of course, the only community out there and there is likely no better guide as to how to work with open source communities than Jonathan. You should read his other publications on this subject such as his “Guide to the Kernel Development Process“.

Yesterday MontaVista unveiled a new community for developers building commercial products using embedded Linux. We call it Meld and it is open to all. If you’ve got an interest in embedded Linux then please come and join in. MontaVista sponsors the community but it is not intended specific to MontaVista products or just for MontaVista customers.

So, why do this?

I’ve been involved with embedded Linux for almost 10 years now. Everything I’ve ever learned I learned from the generous assistance of others. Some knew they were helping me… but most didn’t. The generosity that powers open source and Free software is apparent.  Despite all of this, however, the particular branch of developers building commercial products using embedded Linux is in many ways under-connected.

I’m not an open source project developer… but I can help others who, like me, need an assist here and there to get a job done. Meld is one way to do that. Would you consider joining in to help, too?

Just a few quick FAQish comments:

• Meld isn’t intended to be a replacement or a rival to open source project communities like LKML, kernel.org, or the hundreds of other related projects out there. Our engineers send their patches to the various upstream repos and mailing lists. You should continue to do so as well. If you need help knowing where to go or how to engage we’d love to help you find your way to the right repo… just ask on Meld.
• While MontaVista may be mentioned from time to time on Meld it shouldn’t leave you with the impression Meld is just for MontaVista Linux. That is not our intent. Because neither embedded Linux in general or MontaVista Linux in particular is a mutant fork of the broader world of Linux and open source what we discuss should be generally applicable to many types of Linux which you use to get your jobs done.

If you’ve got questions the comment lines are open.

Brad

InternetNews published an article “Cisco Goes Deep for Linux and Open Source” by Sean Michael Kerner today. It highlights Cisco’s adoption of Linux and key contributions to open source software. The article, in a surprising bit of openness, shares some statements from Cisco executives about their uses of Linux and… well, just read:

“The initial condition is that we don’t want to burn cycles on engineering and development to build from a stock kernel up,” (Michael) Enescu (CTO of Open Source Initiatives at Cisco) said. “We start with a distribution and we have a very good relationship with Red Hat and MontaVista.”

What a nice thing to read on a Friday afternoon. Another quote from Michael inspired some commentary:

“We have both Red Hat and MontaVista as supplier to us. Occasionally there are pieces of a distro that we need to treat differently.”

What I find compelling about this statement is that it shows how what we call “roll-your-own” Linux is really a spectrum of behaviors. Most companies who embrace Linux and open source don’t do everything themselves or buy everything off the shelf. They are hybrids. Some custom, some stock.

It is how a vendor helps you to walk that gray area between stock and custom that can be the real differentiator. More on that later.

Last week MontaVista demo’ed the first version of our MID (Mobile Internet Device) platform. Our man-on-the-scene Dan Cauchy filmed a video showing it off. Excuse the sasquatch sighting style video… I guess he was hand holding it.

Maybe we’ll spring for a monopod for the flip camera

So just clearing out some stuff that I had meant to mention earlier but got too wrapped up in some irritating hardware issues.

Addressing the Top 5 Pains of Linux System Build and Design: Our friend and colleague Klaas van Gend is running this webinar tomorrow. Run like a Forrest Gump and go register right now. Why? Because Klaas knows his stuff cold. He is one of those people who I learn from each and every time he speaks. What makes listening to Klaas fun is that he’s opinionated and has a lot of knowledge all up and down the software stack. He’ll be rambling on real-time one moment and the next he’s discussing cryptographic trust networks.

Got a Twitter account? If you do and you are in to it you are welcome to follow MontaVista’s tweets. We tweet on the obviously named mvista Twitter account.

Today’s hardware irritations: not being able to find my SATA power cable that I ordered and lost in this slovenly office of mine. Two hard drive crashes within 3 months of each other. HDMI CEC commands that cause my A/V receiver to flip to the wrong input.

Today’s software loves: Puppet! When my laptop drive went kaput I was in the middle of upgrading to Ubuntu 8.10. I decided that I would take no system administration actions without recording what was being done as part of my Puppet manifest. When my laptop hard drive died I reinstalled Ubuntu, installed puppet and git, cloned my git repo that had my Puppet manifest in it, and then ran Puppet. After about 15 minutes all of the system tweaks I think are essential were up and running. Very cool!

Mike Hall (Windows Embedded Blog, of Microsoft) posted that Linux use in embedded systems is declining based on a whitepaper from an RTOS vendor and a study from Jerry Krasner of Embedded Market Forecasters. Mike writes a great blog… you should subscribe to it and read it regularly. Microsoft makes a compelling set of products that service true market needs. Nothing but respect for what the do… but I can’t let this slide.

Claim: 48% developers not considering Linux for their next embedded project

This is based on an unreferenced CMP survey cited in a whitepaper by RTOS vendor Express Logic. I sure wish they had a URL to the study because the chart they cite gives me pause. The little footnote that states that they question that supports this claim was changed in 2007 to be more specific about time frame was intriguing. Perhaps they asked a more specific question and got more specific answers? One might feel more freedom to answer liberally about what you hope to do in the future if there is no time frame to it.

Claim: New or modified lines of code on Windows CE 47,000 (ave) compared with 193,000 industry average

The Krasner report which supports this claim is far more interesting. Lets dig in!

A bit about EMF and Jerry Krasner: Jerry’s had years of experience watching this embedded systems market in particular. He’s said both positive and negative statements about embedded Linux. Back in 2003 he was unfairly vilified by many for his report “Total Cost of Development: A comprehensive cost estimation framework for evaluating embedded development platforms” that dinged embedded Linux for a high TCO and propped up Windows CE .NET and Windows XP Embedded. Linux Devices has a good historical summary.

In a December 2007 study “Embedded Linux Total Cost of Development Analyzed” he revisited this topic… this time with his own data collection and apparently free of any vendor funding or direction.

Some quotes from the December 2007 study that I find compelling:

In summary, Linux not only continues to gain market share across the embedded spectrum, but the definitive EMF data set forth herein demonstrates that developers using Linux have the same design outcomes compared with traditional RTOSes.

Uh… sorry Mike. Looks like you’ve reversed the polarity on what the LOC metric means. You’ve got to look at both the lines of code written and the design outcome data in the Krasner study. Just because a project is big in terms of lines of code doesn’t mean the underlying platform is a failure. Krasner mentions that “out analysis showed that the in-house developers wrote fewer lines of code and fared poorly when it came to final design outcomes as compared with their pre-design expectations.” He is using lines of code as a metric of developer contribution to product design requirements, not as a measure of project burden.

Consider what we get when we mash-up both the RTOS and Linux Krasner studies that were published in 2007:

So it looks like if you want to get quicker time to market you’ve got a fairly obvious choice. I also like how this data supports something I’ve felt empirically for 10 years now: Working with a commercial Linux vendor will result in a better project outcome:

Final design results within 30% of pre-design expectation for features and schedule:
Commercial Linux = 70.2%
Other Linux = 59.0%

I’m not bashing Windows CE/XP Embedded, Microsoft, or Mike. I just think that if you are going to go cite a market research study that you’ve got to take into account the overall dataset. The data indicates, to me, that:

1. in-house RTOS or Linux projects are painful
2. not all RTOS or Linux commercial vendors will support the same project outcome