Docker has experienced exponential growth in the last two years. DockerHub downloads have gone from 100 million to 6 billion. To keep up with this growth, Docker broke its monolithic model into smaller open source components, including containerd, libnetwork, swarmkit and LinuxKit.
Open sourcing these components allowed Docker and its ecosystem to collaborate on each component and make them more efficient. It also helped Docker make its own platform modular and scalable, though it also meant that Docker itself would have to pull all these components together and integrate them to create the Docker platform.
That created a unique challenge for Docker.
Since everything is getting containerized, it’s not just Docker that’s experiencing the pressure of using common, standard components to support and build an increasing number of specialized platforms. There are many new players who want specialized Docker editions while using the same open source components.
That should not be a problem. It’s all standard based, open source technologies. All you need to do is put them all together and they will work magically, right?
As Docker set out to build specialized editions using components like LinuxKit, their own production model started to break down. They started to see redundancies and wasted efforts.
The gift became the curse!
They found that the modular design also meant that you have all these different teams pulling different open source components of your own projects to build different specialized systems. There was a lot of assembling, a lot of work that every team had to do themselves. It resulted in duplication and wasted effort.
Despite Docker's popularity, the company is relatively small. They have around 150 engineers doing all of that work. When Docker works on a new version of the platform, it doesn’t have hundreds of engineers on those projects. It couldn’t afford duplication of work within its own small team (which also defies the whole purpose of open source, where you reuse work instead of reinventing the wheel).
This problem was less about components and more about the process and assembly of the platform.
In describing how Docker fixed its engineering problem, Docker founder and CTO Solomon Hykes pointed to the car industry as an example. Different models of cars share the same components, like engines, wheels and chassis, but more importantly they share or collaborate on the assembly process so that they are not duplicating efforts.
Using the car industry model, Docker created a place within Docker where all of its teams collaborate on common assembly, in addition to collaborating with the ecosystem on the publicly available open source components. This reduced wasted effort, duplication and redundancy.
Hykes said that on average they have three or four engineers working on each edition of Docker and it typically takes them between three and six engineering months, from the first line of code to the first beta.
This internal assembly line project solved Docker’s own problem. But in the container ecosystem there was still an explosion of open source components and explosion of participants. So Docker decided to open up the assembly model that it developed. “Docker is a citizen of this container ecosystem and the only way for Docker to succeed is if the ecosystem succeeds,” said Hykes.
Docker took everything that it learned — the process, tools, components, framework, etc. — and contributed it to a new project that is very close to Hykes’ heart.
It’s called the Moby Project.
At its core, the Moby Project is a framework that allows anyone to assemble his or her own container system in a very efficient way so that there is no duplication or wasted effort.
The Moby Project has three core components:
- A library of containerized backend components (e.g., a low-level builder, logging facility, volume management, networking, image management, containerd, SwarmKit, …)
- A framework for assembling the components into a standalone container platform, and tooling to build, test and deploy artifacts for these assemblies.
- A reference assembly, called Moby Origin, which is the open base for the Docker container platform, as well as examples of container systems using various components from the Moby library or from other projects.
In addition to the 80+ core libraries and framework, the Moby Project offers ready-to-go reference assembly that anyone can use to spin their own system. Users can also modify it as much as they want to fit their own needs, or use it as inspiration to build their own systems.
The Moby Project will allow participants to use the framework to bring in their own components, assemble their own projects and build their own communities within Moby. “You don’t have to donate your code; you don’t have to change your code. All you need to do is show up with your code, build, assemble, collaborate, exchange ideas … Moby is an open place to collaborate,” said Hykes, “If you are a system builder, it means you can collaborate and participate in defining systems more efficiently without being tied to Docker.”
The Moby Project will become the heart of Docker. Hykes said that all of the open source contributions to Docker components, all of the pull requests that are sent to any of Docker's projects will go to Moby. Moby will be the place where Docker will test, integrate and discuss the design of the components. It will be the place to discuss how all pieces of Docker fit together.
“Essentially, Docker is building a LEGO club for the ecosystem with Docker at the center of it,” said Docker's VP of marketing David Messina.
The Moby Project will become a place where experimental, bleeding edge features of Docker will take shape. It will be the place where you will see the next release of Docker.
The Moby Project is to Docker what Fedora is to Red Hat Enterprise Linux
Patrick Chanezon, chief developer advocate at Docker, told me that back in 2003, Red Hat assembled the Linux kernel with lots of different components into the Fedora collaborative project. Fast forward to 2017, Docker is doing the same thing for containers with the Moby Project.
“We are going to make sure from the very start that it’s a community run project. We are designing open governance that’s inspired by successful projects that have done things like that in the past,” said Hykes, giving credit to Fedora and Red Hat. “We are not reinventing wheels here, we are standing on the shoulders of giants.”
Does that mean that the Moby Project will also become an independent project managed by an independent organization? Ben Golub, CEO of Docker, told me that the Moby Project will not be donated to an external organization. In the same way that Fedora is a Red Hat sponsored project, Moby is a Docker sponsored project.
This article is published as part of the IDG Contributor Network. Want to Join?