How the US Army is using OpenStack to train cyber soldiers

The United States Army Cyber School (USACYS) is using open source technologies like OpenStack to prepare soldiers for cyber warfare. 

The college was set-up roughly two years ago and while the organization is part of the government and the U.S. military, it’s essentially an educational organization.

Major Julianna Rodriguez (Director) and Chris Apsey (Deputy Director) of the Cyber Technical College at the U.S. Army Cyber School delivered a keynote speech at OpenStack Summit detailing how the school is leveraging open source technologies to keep up with the growth of technology.

The school trains more than 500 soldiers annually, equipping them with problem solving capabilities in the cyber domain. While the school doesn’t get to choose the courseware or what to teach, they do get to choose ‘how’ to teach.

One of the biggest challenges that the school was facing with the legacy model of updating and delivering courseware was not different from the challenges businesses face. Even a small change in the course material could take months and weeks to find its way into the material that’s delivered to the soldiers.

If the instructor came across some new or great idea to expose students something new, for example asking for a virtual machine to install a certain kind of malware on it with a certain kind of infrastructural set-up, the instructor would have to literally write down the requirements and send it to some contractor or organization who could provide that infrastructure. That could take up to 12-18 months to get that delivered.

That’s a typical problem that we see in the legacy model. USACYS doesn’t have the luxury of that kind of time, new threats keep coming up and soldiers need to be as agile and nimble as DevOps are in the IT world. 

That’s when they decided to move to modern technologies and adopted a DevOps culture at the school. They needed a model where everything was coded, where an instructor could deliver an idea quickly.

Apsey gave a demo at OpenStack Summit where he said that now everything is done in Git and they have managed to bring the delivery time from 12-18 months, to minutes and hours.

There is a master branch of the courseware that can be ‘forked’ or branched by an instructor to make changes or add new news. These changes, after review and discussion, can be merged back into the mainline. There you have new material immediately.

“This really rapid pace of being able to speed up an idea to fruition in a problem solving perspective is because we’re using OpenStack and other open source software to be able to put together infrastructure as a service,” said Maj. Rodriguez.

They are not using any vendor specific OpenStack solution; they built it from upstream and named it Broadband Handrail. “The idea is the staircase of learning, we want to make sure that you don’t fall off the staircase of learning so we’re providing a Broadband Handrail so students can, in a semi controlled environment, be able to spin up solutions or answer the solutions for their classes,” said Maj. Rodriguez.

But software is only half of the equation. Apsey said that it’s very hard to buy hardware in the federal government so they ‘found’ some unused servers and stacked them in the corner of a room, like a LAN party, cables hanging from the ceiling. They started off with a 40-core, 512GB of memory, 10TB of storage and LTE hotspot as an access point. They had a squid server to cache Debian packages to cache what was needed.

They shopped this infrastructure for around six months as a proof of concept for the leadership. Now they have a system with 2,000-core, 36TB of RAM and 4PB of storage.

But in the process, they learned some valuable lessons. One of the lessons they learned was that ‘silos’ of excellence don’t work and Maj. Rodriguez said that the right way to approach it is the way the open source community approaches it by identifying where it is that you are going and get a lot of stakeholders to share that vision so that you can go forward together.

The second lesson that they learned was version control so that you can see who is contributing, encourage their efforts and see if they need additional support. At the same time it also allows you to roll back changes if they didn’t work correctly.

Now the Broadband Handrail system is in place it’s helping with things they didn’t expect. Cyber warriors can now go back to use-cases that they worked on when they were students to keep their skills sharp. It also allows them to spin their own systems instead of having to buy their own servers, which saves costs.

Apsey concluded the keynote speech by saying, “I know the government isn’t thought a lot of as an innovator, but we like to think that we’re kind of moving the ball forward just a little bit.”