Open-Source Software and Its Role in Space Exploration

What did the rocket scientist say to the Free/Open Source Software developer? Let’s do launch! It’s only natural that they’d want to work together. Both communities are focused on the cutting edge: creating tools and capabilities that did not previously exist. Both dedicate their work to expanding humanity’s pool of information and want that information to float freely through society.

I am a software developer currently working on the NASA/JPL MSL (Mars Science Laboratory) rover, which launches in 2009. These are personal observations of how I encounter Free/Open Source Software (FOSS), and what I think about it.

Common Goals

Free floating information feeds a cycle of knowledge. Where the FOSS community donates code, algorithms and products, NASA and other organizations reciprocate with knowledge about weather systems, climate and basic science. Everyone contributes what they’re best at, and tightly chartered organizations can stay focused on deeper penetration of hard problems, confident that others are doing the same. Space exploration is necessarily a cooperative venture; it’s much too hard for anything less than all of humanity.

Look at these statements side by side, and you’ll see the philosophical similarities:

NASA codifies its dedication in Congress’ Space Act Charter:

[NASA shall] … provide for the widest practicable and appropriate dissemination of information concerning its activities and the results thereof…

The Open Source Initiative criteria for “Open Source” includes:

• Allow free redistribution
• Provide access to source code
• Allow modifications and the creation of “derived works”

FOSS developers codify that dedication in copyrights, copy-lefts, and license agreements like the GPL (GNU Public License), which says in part:

When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things.

Open Source in Space

Need a few examples?

FOSS explores our Solar System. We send robots to the moon, Mars and beyond to flyby, orbit or land. FOSS goes with them, pervasive in the real-time operating systems, math libraries and file systems. Consider the robotic decisions of where to rove, and realize the power given the human race by the Free Software Foundation’s (FSF) compilers, libraries, build scripts and so on.

“Electra” is NASA/JPL’s Software Defined Radio (SDR) product created to support the Mars Network, and the InterPlanetary Internet. Electra provides UHF radio links in compliance with Consultative Committee for Space Data Systems (CCSDS) protocols Proximity-1 (data link) and CFDP (file delivery).

The neat thing about SDRs is that we can still reconfigure the protocol and signal processing functions after launch. For example, on MRO some other hardware started “leaking” electro-magnetically, which interfered with the Electra radio. We sent up a software fix to Electra to reduce the impact. We already know MRO will act as a radio relay for future Mars probes not yet built, such as the Mars Science Laboratory (MSL), which will arrive in 2010. MSL and others will use protocols not yet invented, and MRO will have to be updated to learn them.

Please bear with me a moment; I’d like to make a point about how much FOSS enters into this. Each of the following are FOSS project technologies used in the program. The flight software is mostly C, with some assembly for trap handling. It lives on file servers running kerberized OpenAFS, in a CVS repository, and is cross-compiled on Linux for RTEMS on SPARCv7 target chip. The code is built with gcc, make and libtools, and linked with newlib. (There, that wasn’t so bad. I’ll mention it again later.)

Hubble “Pillars of Creation”

FOSS observes our Universe. All the striking Mars rover navigation images, all the roiling clouds of Jupiter, each new view of Saturn’s spectacular rings, every azure picture of deep-blue Neptune; the distant stars, far-away galaxies and stupendous galaxy clusters; all of these come to us touched in some way by FOSS. Think about the JPEG image formats themselves, the X11 workstation software to process them, and the MySQL databases to hold them.

Deep Space Network antenna

FOSS moves and analyzes ground data. When we prepare, check and double-check command sequences for uplink, the inputs and outputs travel variously across multiple sendmail e-mail systems, Linux platforms, and of course Internet Protocol stacks. The downlinked data, after its tiring journey across the solar system, bounces those last miles in mostly the same way the rest of our Web-connected world works. In the 1960s, NASA engineers had to invent lots of ways to move data around, and it became expensive to maintain.

FOSS methodologies develop space software. Some FOSS project efforts are very widely distributed, driving development methodologies to leverage all those eyes while integrating all that expertise and smoothing together all those styles. That problem is much tougher than, but similar to, our allocation and integration of functions across organizations, teams and contractors. Those methodologies are very attractive to us. There are several NASA examples of using an “agile software lifecycle,” and we look to open communities to show us that it can be done, and how to do it best.

The only way for the public to participate in seeing fresh images in near-real time is through an open architectures for public outreach.

FOSS develops the next-generation cutting-edge technology. Why can we put a man on the moon, but we still don’t have robot cars? Some key challenges are:

• Robots have different physical characteristics.
• Robots have different hardware architectures.
• Contributions made by multiple institutions.
• Advanced research requires a flexible framework.
• Software must support various platforms.
• Lack of common low-cost robotic platforms.
• Software must be unrestricted and accessible (ITAR and IP).
• Software must integrate legacy code bases.

The Coupled Layer Architecture for Robotic Autonomy (CLARAty) project brings together folks from many institutions. They develop unified and reusable software that provides robotic functionality and simplifies the integration of new technologies on robotic platforms. (They also have some funky movies of robots doing funny things.)

Ubiquitous FOSS

Our dirty little secret is that space agencies are companies just like everybody else. We too (am I shocking you?) use e-mail, Web servers, and all the usual non-space-qualified suspects. Here are some examples:

• Operating Systems, Systems Management: Rocks (cluster Linux), Ganglia, amanda
• Software Management: Depot, Subversion, Trac, Bugzilla
• Communications: OpenSSH, Apache, Jabber, Firefox/Mozilla, Sendmail, Mailman, Procmail, CUPS, OpenOffice, wikis (various)
• Data Visualization: ImageMagick, GMT, MatPlotLib
• Compilers, languages, code checkers: SunStudio, splint, Doxygen, valgrind, Java, Perl (some JPL history there), Python, Ruby
• Databases: MySQL

The Open Advantage

OK, so given that our cultures are similar, how does that translate into our bottom line? Why does FOSS have such a large role in space exploration? Here’s the top-10 list of what I see.

1. Schedule Margin

Planets move; launch windows don’t. The Spirit and Opportunity Mars Rovers had to go in the summer of 2003 or never. They are simply too massive to throw that far, for that budget, unless the planets aligned just so. (Mars and Earth line up every 26 months or so, but in 2003 they were unusually close together.) Procurement cycles for spending lots of government money can be months long, and they can dominate critical paths.

Quickly obtainable FOSS relieves that pressure and gives us some elbow room. Bug fix turnaround times can be critical. If we can fix the source code ourselves, we can keep a whole team moving forward. If the fix is accepted by the open-source community, we avoid long-term maintenance costs and have it for the next project. Feature additions (“Gee, if it only did this, too…”) have the same advantage but take longer to give back. Oddly, we can contract for new features but cannot easily give them away. The FOSS spirit hasn’t yet pervaded government contracting rules.

2) Risk Mitigation

Full system visibility is key to risk identification, characterization and resolution. The Mars robots are sent to encounter unfamiliar situations. Think how much information system engineers need to mitigate those risks. This is no place for a closed system.

All flight software goes through rigorous review, including the software (compilers) that builds the software (command sequence generators) that builds the software (commands). We do code walk-throughs which perforce means having the source code. We design white-box test plans by analysis of software decision paths, which is easier to do with the source code in hand. Our review process requires “outside experts, not working on the project” to review the code; well, that’s exactly what a FOSS community is all about, isn’t it? In essence, the open-source community is the world’s largest Review Board, only we don’t have to buy the doughnuts.

3) Reliability

When you leave Earth’s orbit, you also leave “push the reset button” and “reload from CD” far behind. We tend to find bugs that don’t bother other customers. We live at or beyond the border cases, and we push frontiers in all senses of the word. So all the critical bugs have to be found and squashed before we go.

The best way to shake out software bugs is to have lots of testers independent of development try it out in unfamiliar environments and in ways unforeseen—which pretty much describes the FOSS user community. By the time something’s on its 2.1 release, it’s usually been beaten up pretty thoroughly. And the beauty is, you have full disclosure about what broke in the 1.0 release, under what conditions, how it was fixed and what tests prove it’s gone.

4) Interoperability

Space exploration takes a lot of brain power—more brains than any one company or nation commands. Our industry, academic and international partners each have specialized expertise vital to the effort. And each partner, it seems, uses a different platform, language or protocol from the rest that’s optimized for that particular piece. Each builds a sub-assembly, and the thing has to work when you bolt it all together.

This is interoperability by definition. Software must be designed from the start to “play well with others” beyond your organizational control. Attempts to dictate uniform development platforms are not infrequent, and always fail. At worst, they represent a willful desire to ignore strict interface control, and interfaces are precisely what call for the greatest care.

As we build the space shuttle replacement and new spacecraft for the moon, interoperability is a top-level requirement. Under the current “Vision for Space Exploration,” the “Constellation Program’s Communications, Command, Control and Information (C3I) Interoperability Specification” was drafted early. Not surprisingly, the specification calls on open standards.

5) Portability

The Pioneer and Voyager spacecraft are older than disco, though younger than the Beatles. They are further from Silicon Valley than anything made by human hands, and getting further. Data from them continues to puzzle us. So, software to analyze that data has been ported to myriad computers. There’s never enough money to upgrade routinely, so we stick with a platform until it dies and/or its manufacturer goes out of business. This is only barely tenable through strict portability conventions.

Spacecraft parts are hideously expensive, what with radiation toleration, quality screening and so on. Software usually has to be developed on simulators and ported to a number of similar but not identical units. For every flight article, there may be a “qualification unit” (for testing to failure), two or three “form/fit” units for functional testing, and some “engineering units” for development. A simple radar algorithm has seen development on Mac OS X, Microsoft Windows and Linux (that I know of)—none of which is the final environment. It’s been coded in python, perl and C. The work would take far longer had we been locked into one vendor.

The Electra platform and code described earlier have been ported/inherited/reused for:

• a landing radar on MSL
• a spectrometer interface on ISRO’s (Indian Space Research Organisation) Chandrayaan-1 Moon Mineralogy Mapper
• a lunar radio architecture prototype C3I (Command, Control, Communications, and Information) Communications Adaptor (CCA)
• Radio Atmospheric Sounding and Scattering Instrument (RASSI)

—all in the space of a few years, each time by a different team.

6) Openness

I’ve seen technical information retrieved from hard copies of presentations, because people unable to open files that were only a few years old. The (closed) format had changed when a desktop computer was upgraded. That just won’t fly.

We’re hit hard by International Traffic in Arms Regulations (ITAR), Export Administration Regulations (EAR) and intellectual property concerns. Even innocuous software has to be screened for release, and is slow to share with an industrial or foreign partner who needs the same tool (possibly ported to their platform).

“Free as in free speech” can help; here’s an example. The thing about a Deep Space Network is, the Earth rotates. The big antennas are spaced out (if you’ll forgive the pun) around the globe in California, Spain and Australia. Once, it was time to upgrade some encryption software (ssh), and we discovered that it couldn’t be exported as binaries from the United States—which is funny, because the source code came from Finland. That gave us the solution: While sitting in California, we logged in to Australia and Spain, downloaded the Finnish code and built it there. “Finnishing” in a few minutes what would have taken weeks of paperwork.

Recently, we’ve experimented with auto-generation of Electra source code from UML state diagrams. It may or may not work out well enough to fly the code; we’ll see. It’s only because of the openness of the architecture and tools that we could even contemplate this radical modification to development.

7) Collaborative Software

With specialized instruments and parts coming from all over the globe, teleconferences and videoconferences are essential. There are a lot of really bad services out there, and the consistent failing theme is when the server expects your desktop to have some single-vendor thing that you’ve never heard of and shouldn’t need to know about.

Why is it college kids can talk to their friends using every platform made, and we can’t? Because they require interoperability from the beginning.

8) Confidence in the Future

At the risk of harping on the portability and openness points above, software projects have very long lives, and the selection process weighs in our confidence that it has a future. Developers like me are comforted by having the source code in hand as a risk mitigator.

Managers, however, prefer support contracts for the same reason: to get a commitment that someone is maintaining expertise in the product. We’ve had good success with companies whose core business is supporting FOSS projects. Large companies are seeing the same advantage, and have opened their formerly proprietary source code and funded organizations and universities that maintain them. This relatively new phenomenon is working well.

9) Security

It’s not like any launch codes are on my laptop, but I still don’t want some cracker breaking into my machine. I want to know the software I’m running is solid, to be told at once an issue was discovered and to have a patch immediately. I do not want a smiling happy representative telling me everything is fine, ignore that funny sound, and I can upgrade to the new version when it comes out next year. ‘Nuff said?

I read the weekly security announcements from the SANS Institute (SysAdmin, Audit, Network, Security) and pay attention to whether announced vulnerabilities have a fix available or whether it’s “being investigated by the vendor.”

10) The Cost Question

“Free as in free beer” can matter for small tasks or where labor is cheap (graduate students work for ramen noodles), but mostly the price isn’t a big factor for using FOSS. As CIOs know well, Total Cost of Ownership is dominated by learning curves, testing cycles, reviews, writing procedures, etc. So the cost savings is less in procurement than in labor, and in the way that openness saves time performing other activities.

Freedom Isn’t Free

Free speech doesn’t mean anyone has to listen. No one is entitled to a forum for their opinions, and freedom of the press belongs to them as owns the presses. Or in this case, the website. If you want to be heard, you have to convince your audience to invest their attention.

Perhaps you’re a member of an open-source community that hopes we’ll adopt your software. I can get my hands on all kinds of FOSS; what I download, build and fly is another matter. I’m always drawn to the products that include automated regression test suites. If the documentation includes a reference guide, user guide and maybe a QuickStart, it reflects an empathy for solving my problems.

We must meet very strict software quality-assurance and dissemination requirements. Here are some of the guidelines in NASA’s Software Policy (NPD 2820.1C):

• Require software providers (includes internal NASA providers) to have proven organizational capabilities and experience to deliver quality software on time, within budget and within technical acceptability.
• Require software providers to develop a plan to manage software throughout the program/project lifecycle. This plan shall include the collection and reporting of actual software related expenditures at the project level by lifecycle phases.
• Release software in accordance with NPR 2210.1, External Release of NASA Software, consistent with law and applicable agreements, for commercial, industrial, educational and governmental purposes.

You can find lots of other good stuff in that document by searching for “software.”

The point is that “assurance” means “do more than write solid code.” I need a story to go with it. Working demos go far, but not the last mile. These are the infamous paper trails that slow commercial projects. At one extreme is to incorporate your “development organization” and pay for an external audit of your processes. My personal jury is still out on this; I see lots of projects where the extra effort doesn’t actually raise quality (like most FISMA reporting) and it’s a waste. On the other hand, my current project is being appraised for Capability Maturity Model Integration (CMMI) level 3, and we’re actually changing how we do things, in a good way. The difference is made, naturally, by the quality of management attention.

Challenges Enough for Everybody

I hope this conveyed the impression that FOSS plays a significant role in space exploration. The more FOSS code we send to Mars, the more time I can spend working on different hard things.

FOSS’s strength is in the people, the community. Events like the International Conference on Open Source Systems build that strength. I’m honored to work with these folks, and I greatly appreciate what they do to help us all.

This article was adapted from an invited talk at the USENIX LISA 2006 conference in Washington, D.C.

This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.

Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer or otherwise, does not constitute or imply its endorsement by the United States Government or the Jet Propulsion Laboratory, California Institute of Technology.