Dynamic Languages: Not Just For Scripting Any More

Scripting languages have been around pretty much since operating systems were built to stay resident—back to DOS JCL, and no I don’t mean that DOS. I mean the Disk Operating System for the IBM 360. OS/360 JCL, Digital’s various command line interpreters, UNIX shell scripts and awk all allowed more-or-less complicated programming to perform repetitive tasks. Later, scripting languages like Rexx, Tcl and Perl gave script programmers more flexibility. Still, the general assumption was that even a program that could be prototyped using a scripting language would need to be rewritten into a compiled language.

This assumption started breaking down in Web-based applications. Many important, high-capacity Web systems were built with Perl or Tcl. At the same time, scripting languages languages like Python and Ruby began to violate long-held prejudices. Python and Ruby were clear, powerful, and new implementation techniques were making them increasingly efficient.

The result is that scripting languages occupy a greater and greater part of the programming landscape; the potential for high productivity overcame many objections. Still, scripting languages weren’t used in two areas of programming: systems programming and commercial, installable, “shrink-wrapped” products. Now those barriers have been broken as well.

Recently, I needed to build some new semantics around the Sun ZFS file system in order to prototype some new ideas. Traditionally, I might have done this with C. The existence of FUSE file systems in user space would simplify that. With a FUSE file system, I could write those new components in C in a user application, at least.

At about the same time, though, I ran into a lovely Mac product called ExpanDrive. ExpanDrive allows any remote system that is accessible through FTP or SFTP to be mounted on a Mac as if it were a native file system. It’s a nice step up from most FTP tools. With ExpanDrive, I simply remotely mount one of my websites as a file system; I can copy, paste, edit and rename exactly as if it were my own local disk. This led to reading Jeff Mancuso’s blog, he being one of the primary developers of ExpanDrive.

It turns out that ExpanDrive is very largely written in Python. As Mancuso describes it:

We rewrote SftpDrive [their windows product] from top to bottom in Python, with a GUI in Objective-C. It’s called ExpanDrive, and it took one third the time that SftpDrive took to develop. We greatly leverage Python and and many open source projects—just like a web developer. To minimize conflicts and to have the necessary control over the runtime environment, our build process extracts only the necessary bits from the full Python distribution and packages it into the .app. … We’re pushing out weekly updates which include more than just bug fixes. ExpanDrive has been a breeze to maintain and extend and the core remains perfectly cross platform.

With this as inspiration, I tried the same thing myself. By preference, I work on a Mac, and conveniently ZFS for Mac is a reasonably stable product. MacFUSE, from some folks at Google, is easily available, and there exists a good Python binding to FUSE. It took about a day’s effort to get them all installed and working together.

It then took only about another eight hours to have a full new file system, built on top of FUSE and ZFS, working on my desktop. Two days later, I was demoing the system for CTOs and vice presidents.

This wasn’t a finished product; in fact, I’m still adding new features and exploring the possibilities. This is made tremendously much easier because the important code is in Python: I edit the source, save the Python file, restart, and it’s running. No compile, no complicated build; as with writing a normal script, it’s nearly instant gratification.

What’s best is that the result is perfectly usable. It’s not a “prototype” in any limiting sense; it has all the functions I build and appears to the rest of the system as a normal component.

As with ExpanDrive, there is nothing about it to reveal that it’s really a “Python script.” It’s just another way of building a system component—just one that happens to be dramatically more productive, more flexible and (as a nice side effect) most of it is platform independent.

This won’t be the death of compiled languages. Just as we still need to write assembly language programs sometimes, concerns about complexity, performance and security will still lead to writing programs in compiled languages for years to come. But, with the ability to use Python to build custom programs, performing as operating system components, with all the advantages Python offers, it’s clear that Python—and by extension, other scripting languages—are not just for scripting any longer.