It has been a very interesting year looking out the window at the embedded Linux scene. I see total mainstream adoption as a clear and present trend. I base this off of “man-on-the-street” conversations with acquaintances. I was at a training conference on a non-technical topic. Asked what MontaVista did I gave my non-technical explanation. Quick as anything the person replies… “oh, embedded Linux. Yes, our engineers use that.”

Linux and the importance it has to technology product companies is now common awareness. That means something significant.

I’ve also seen a strong trend in recognition that the compelling problems of today are well outside the realm of just kernels, toolchains, and booting. The profound increase in potential software capability that open source brings to bear on new products has its one set of challenges. You can do media, office applications, advanced networking, wireless networking, power management, etc… but making choices and integration are far more challenging.

We’ve been able to help many of our clients to fabricate compelling products using a whole range of technologies north of the kernel. We’ll have some fun time discussing these in the coming months, I hope.

So since the action just starts at the kernel let’s take a moment to plug some upcoming conferences where you can explore these issues:

• Linux Plumbers Conference, September 17th-19th in Portland, OR: If you’ve ever been to the Ottawa Linux Symposium then I imagine this will be like that… just with a focus on all of the interfaces that bridge the kernel/userspace/application divide.
• MontaVista’s own Vision 2008 conference, October 1-3rd in San Francisco, CA: We had 400 attendees from 13 countries attend last year on this inaugural event. If you’re focus is on commercial product creation using Linux then this is a great even to network, learn, and brainstorm at.
• DESIGN & ELEKTRONIK Entwicklerforum, Oct 15, 2008 in Stuttgart: I had not heard of this conference before but there are a number of interesting presentations scheduled including two of from my fine colleague, Klaas van Gend and Ned Miljevic.
• Embedded Linux Conference Europe, November 6-7th in Ede, The Netherlands: One of the big dogs of the annual embedded Linux conferences. The Klaas & Ned show is making an appearance here as well.

So there you go… blow your annual travel budget out and make them all!

If you can only pick one then pick Vision 2008.

Brad

ShareThis

In this line of work there are often things you can’t say. You just can’t go around talking about what our customers do with MontaVista Linux. Like I couldn’t possibly talk about the smile I get on my face when the chock-full-of-MontaVista-goodness high speed 3G cellular data network delivers packets to my <insert the name of the absurdly popular phone>. Can’t talk about that.

A few LinuxDevices.com articles will show you what other folks have said, though. Over on linux.com read up about how RipCode is helping MySpace to deliver video using MontaVista Linux.

Or you can read about how seven new MontaVista powered LiMo based phones were announced.

[Note… I don’t have a problem directly linking to LinuxDevices.com… maybe they’ll return the favor the next time they mention my blog!]

I stared with MontaVista back in 2000. Long time ago. I spent all my time answering questions about what Linux was, why open source wasn’t communism, and how someday our erstwhile competitors would come to regret the FUD they spread about Linux. Now a days I get to go to Best Buy and point out MontaVista powered products all over the place.

It makes me smile not because of some corporate dominance game. It makes me smile bcause I was part of this fight in the war for open source ascendancy and we won it. Reading Cuong Nguyen of RipCode say “We only considered Linux” makes me smile and would make me smile even if they weren’t our customers.

ShareThis

Last week’s webinar left off with several questions from the participants. I answered mine in an earlier post. My collaborator Kaitlin Murphy has provided answers to the hardware questions. Courtesy of Intel and Kaitlin I was permitted to post her answers here:

Q: What kind of power benefit we can gain by entering C6 of Atom CPU?

In the C6 state, the Intel(r) Atom processor will consume approximately 6% of the power it would consume in the C0 state. This roughly correlates to 100 mW, although the exact value may vary depending on which SKU of the processor is used and other factors. Obviously, the more time the Intel(r) Atom processor spends in the C6 state (or any other sleep state), the more power savings will be realized.

Q: What is the typical power consumption on a MID device using Atom on C0 and C6?

The power consumption of the Intel(r) Atom processor will vary significantly by usage conditions. For the MID type application, the average power is 160 - 220 mW, with ~100 mW idle power. (Idle and Average Power power quoted is using a mean leakage CPU which means that 50% of the CPUs will have leakage values below the median value and 50% will have leakage values above the median.) These numbers will differ in an embedded use case.

Q: Is it possible to disable part of the chipset (in our case, no video needed).

On the Intel(r) System Controller Hub US15W, it is possible to effectively disable the graphics core via clock gating. Intel does offer other “headless” products - such as the Intel(r) EP80579 which may work for your application, depending on your needs.

Q: …can [you] give me a [watt per MHZ] value?

This will be SKU dependant. For the Intel(r) Atom Z530 SKU at 1.6 GHz, the TDP is 2.2 W. Therefore, your theoretical W/MHz value is ~727 MHz/W. A similar calculation applied to the Z510 SKU yields ~550 MHz/W.

Q: How much as a percentage does the SpeedStep technology help if I have a real time application and cannot go to the lowest C state?

Intel(r) Enhanced Speed Step Technology can still be a big asset in power savings and power management - even if you can’t go into the lowest C state(s). The amount of time a processor can spend in a lower sleep state is primarily dependant on what it has been tasked to do. There is no reason a processor can’t handle real time requests when operating at a frequency lower than HFM. In fact, I’ve seen customers de-rate a processor (essentially forcing it to run at a lower frequency, simulating SpeedStep) if lower power consumption is a requirement.

Again: A big thank you to Kaitlin, Intel, and all of our participants.

ShareThis

If you missed the webinar this week you can view the slides here or the recorded version here.

These are the questions provided by the audience that I thought I could handle. Kaitlin Murphy from Intel is working on hers and we should get them together here shortly.

Q: Hi, you discussed the cpu-freq and dynamic frequency and power scaling? What about the control of idle sleep modes?

The questioner went on to correctly identified the cpu-idle subsystem as being responsible for putting the processor into various sleep states (C-states).

Q: How do make the application itself “power aware”?

As discussed in the call there are a few techniques that can be considered:

1. The application could itself serve as a userspace governor for voltage and frequency scaling.
2. The application can use the provided /proc and /sys interfaces to monitor the power status of the system including the AC & battery states, the frequency and voltage scaling behavior of the system. Very intelligent applications might use this data to defer power hungry tasks until later or even limit certain usage modes.
3. One should take care with all applications to ensure that they are not unnecessarily keeping power hungry devices open (which would prevent them from being powered down) or causing wakeups that are not needed. Dave Jones (RH) made a presentation at the Ottawa Linux Symposium 2006 about unneeded wakeups and other topics titled “Why Userspace Sucks” that is worth reading.

Q: Has this power management work been accepted by the open source community and the community will maintain it with kernel evolving?

All of the features I discussed are evolving in a public multi-party fashion in the mainline kernel tree and other contributing subsystem trees. Many companies play a role as do unaffiliated individuals. Notable contributors include Len Brown (Intel, ACPI), Ingo Molnar (RH, RT and everything) and Thomas Gleixner (Linutronix, dynticks, clockevents, etc.), Venki Pallipadi (Intel, deferrable timers). I know there are others but I’m getting tired of Googling. No offense intended at all.

What we’ve discovered over the years is that having the supporting technologies is important but then actually tuning them to work on a specific hardware, kernel, userspace, and application workload is essential.

You can’t download a kernel.org tree and expect power nirvana… but power management in the community has evolved and advanced significantly thanks to contributions from many.

Q: I guess CPUFreq driver should interact with clock framework. When and how they interact with each other?

Yes… you certainly want your timers to go off at the right time and your clocks to advance at the proper rate.

Quick grep study shows that the cpufreq_register_notifier() function is used in most architecture specific timekeeping code in order to register a notifier that updates the loops_per_jiffy. This is reinforced by the file Documentation/cpufreq/core.txt from the kernel source:

The CPUFreq core code is located in drivers/cpufreq/cpufreq.c. This
cpufreq code offers a standardized interface for the CPUFreq
architecture drivers (those pieces of code that do actual
frequency transitions), as well as to “notifiers”. These are device
drivers or other part of the kernel that need to be informed of
policy changes (ex. thermal modules like ACPI) or of all
frequency changes (ex. timing code) or even need to force certain
speed limits (like LCD drivers on ARM architecture). Additionally, the
kernel “constant” loops_per_jiffy is updated on frequency changes
here.

Q: on this slide you have APCI..is that a typo or a new Acronym?

Typo.

Q: What is the maximum amount of time a deferred timer may be deferred?

A: There is no maximal value defined or allowed to be defined. The timer will defer until a non-deferrable event such as another timer awakes the system. For more information this LWN.net article may be useful.

Q: Would the Atom processor be useful to run a real-time OS like MontaVista, and could sleeping between tasks be implemented in the real-time OS?

The question is a wonderful reminder that not all real time tasks are “fast”. There are a great number of RT applications which don’t require non-stop and rapid response to events. Some of these “slow” RT applications could benefit from utilizing power management capabilities in-between events.

Yes, the operating system could put the system to sleep in between events, even RT events. The responsiveness of the platform to the next RT event will be influenced by the speed that the platform can emerge from the sleep state. For example: If the hardware platform can emerge from sleep within 50ms (a totally made up number) and your event only has to be responded to within 100ms you might be fine. If your even requires a 1ms response time then you may have a problem.

Very deep sleep states (like the C6 state mention by Kaitlin on the Intel Atom processor) that flush caches may introduce secondary performance impacts. The caches will have to be refilled from main memory and this would introduce additional overhead as the system emerges from sleep.

If the performance penalty of emerging from sleep is unacceptable you could utilize the other power saving features (frequency and voltage scaling, I/O device shut-down, etc.) and avoid using sleep states.

Q: Could you recommend a development system to play with MontaVista and the Intel Atom processor?

As mentioned earlier we support the Intel® Atom™ Z500 Processor Board with our Professional Edition 5.0 product. You can find our full list of supported hardware here.

---

I forgot to include some image credits in my presentation. Here they are:Public Domain
http://en.wikipedia.org/wiki/Image:9700_HMS.PNG

CC Attribution 3.0 Unported
http://staticfree.info/projects/24h_clock/

CC Attribution-Share Alike 2.0 Generic
http://flickr.com/photos/jojakeman/2433418929/
http://flickr.com/photos/nirmalthacker/2187149611/
http://flickr.com/photos/yahtzeen/37102029/

ShareThis

A sincere thank you to Kaitlin Murphy of Intel who was my collaborator on the “Embedded Linux Power Management on the Intel Atom Processor” presentation. We just finished on time and had great attendance. Special thanks to Ken Molay (blog) of Webinar Success who was our host and coordinator.

If you missed the webinar you can register to download the presentation.

I didn’t get a chance to answer every question but I’ll do my homework and post them to this blog. Please post a comment if you had a question that you didn’t get to in the session. Once I get the full question list from Ken I’ll post them here.

Q: What Atom based reference boards does MontaVista support?

We support the Intel® Atom™ Z500 Processor Board with our Professional Edition 5.0 product. You can find our full list of supported hardware here.

ShareThis

On Wednesday, July 30 - 11:30am Pacific / 2:30pm Eastern I’m presenting a webinar with Intel’s Kaitlin Murphy on Linux power management and the Intel Atom processor. I hope you’ll consider attending. You can register at Linux Webinars.

As always I’m still hacking away on the agenda with Kaitlin but here is what should be in the final presentation:

1. Why care about power management… perspective from the retail POS industry
2. Power management as a system level design objective
3. ACPI and cpufreq in the Linux kernel
4. The Intel Atom processor
5. Enhanced Intel Speed Step technology
6. Deep Power Down technology
7. Dynamic cache sizing
8. A five step homework assignment for power management
9. Recommended reading

The webinar will help establish the fundamentals of power management and the techniques that can be used to exploit power management capabilities in the underlying Intel Atom hardware.

ShareThis

[back from a fun vacation and a quick trip to Ottawa… hello to our new readers from LinuxDevices]

Jonathan Corbet at LWN.net called my attention to a recent article in  the Patent Law blog which discusses a recent patent appeal outcome. The takeaway is that the appeal provided a rubric for evaluating the validity of patenting a software application that may mean that software algorithms that run only on general purpose computers are not patentable.

Whoa. That should cause some heartburn. If the analysis from the Patent Law blog is reliable then there could be a number of software domains which just aren’t candidates for protection using patents. Since IANAL I’ll refrain from adding my own analysis.

Let’s see if I can tempt Matt “I blog 98 times a day” Asay into saying something about this.

Brad

ShareThis

Next week I’ll be attending the Freescale Technology Forum in Orlando, FL. I’ll be presenting on power management for Linux based products on Wednesday at 5pm in the Tuscany A room.
For more info on the other presentations being conducted by my colleagues please see the list here.

ShareThis

Where we last left off our intrepid support team was just getting the FedEx package with the customer’s hardware in house. At this point, as the customers FAE, I’m pretty sure that the customer is going to rip me a new one any moment.

Thankfully (I appreciate it Geoff) he didn’t do that.

Since the error had been in reboot we were using some scripting to automatically race through the cycles. The kernel has a sequence it uses in booting up and not all of it actually produces printf output. You can also have issues where printf has been sent but not displayed. Adding printf instrumentation can be helpful… but it give you certain information. Some of you out there are thinking JTAG… but this was x86. Sorry.

The approach you take on this kind of defect is that you start simplifying the system. The customer had done some of this but we wanted to quickly duplicate his progress to make sure it was accurately conveyed. Then you might also start disabling BIOS features. Our support engineer soon noticed a correlation between the hang and activity happening on the parallel port I/O ports.

With a solid lead in hand the engineer took to just abusing the parallel port directly using a userspace test application. This particular system, as it turns out, had a hardware fault. Reading repeatedly from port 3Bc would, within minutes, cause the entire hardware platform to lock up.

Next the engineer went and wrote an x86 assembly code program that booted from a floppy that abused the port as described. This independently proved that the fault was in the hardware, not Linux.

I’ll probably always remember this issue because it wasn’t until right at the end that we became convinced that it wasn’t a software issue. We always suspected… but could never confirm.

Next debug horror story is about IDE drives.

Thanks for reading,

Brad

ShareThis

I was an Atlanta based event (StartupRiot.com, other coverage) yesterday and I was asked a few times about what MontaVista did. I mentioned that we helped our customers to get the most out of open source as they build their commercial devices.

Invariably the first comment I got was: “Oh… embedded software. So that’s doing Linux in really small systems, right?”

Well… sometimes. We’ve got many customers who are trying to build very small products that utilize a minimal amount of flash. We’ve also got customers who build huge systems that use multicore chips with 16 processors and many gigabytes of RAM.

That’s the great thing about Linux… it is scalable such that it can be adapted to reach the sweet spot on most designs without undue effort. It may not be appropriate for the extremes but it covers the great swath of everything else.

ShareThis