If you are a KDE neon user, reinstall your distro now!

The lead KDE neon developer Jonathan Riddell released a security advisory in which he urged KDE neon users to upgrade their systems or reinstall it.

TLDR;

The package archive of KDE neon was misconfigured, which allowed anyone to upload a package to the archive. It’s not known whether anyone uploaded any packages, but as a security measure they purged the archive, removed the ISOs built from it and they are now rebuilding it. The KDE neon team is urging users to upgrade to the latest packages once they are rebuilt. If you want peace of mind, reinstall KDE neon using the latest built ISO from the site.

The backstory: what actually happened?

I spoke with Riddell and asked him if there were some best practices or security protocols to be followed. He explained that since KDE neon is growing (in size) and now has almost all of KDE software which is maintained, the neon team had to move the archive that was on the same server as their build server to a server with more disk space.  

“Best practice when setting that up is to use the trusted SSH protocol to open a tunnel and upload through that,” said Riddell, “Unfortunately in setting up that new archive I had it listening for uploads on the whole network not just the local SSH tunnel.”

That’s it. That’s all that went wrong.

I recall my discussion with Richard Brown, the openSUSE Chairman, who stressed on building systems in a way that experienced engineers and developers are involved with a project so that not all the burden and responsibility is placed on a single person.

That was how things were with with Kubuntu, a project that Riddell started and then quit. Riddell told me, “At Ubuntu the archives are managed by a team of dedicated sysadmins and developed by the Launchpad team.”

Unfortunately, KDE Neon is a much newer and smaller project with a smaller team consisting of less than half a dozen contributors using off the shelf technologies to set it up. “We’re really proud of being able to recreate setups that companies spend millions developing ourselves,” said Riddell.

It’s commendable that Riddell’s smaller team did manage to fix things. But in the long run I think the KDE project should invest in smaller projects like neon, because in the end you are a KDE user.

Open source communities slow to adopt DevOps practices!

The irony is that open source kind of triggered the DevOps movement that stresses on agile development and agile teams, where as KDE and many open source communities are still living in the legacy world and are slow to adopt DevOps practices.

“This is strange because as teams who work solely on the internet we’re ideally situated to use systems which automatically build, test and deploy without human interference,” said Riddell.

While the KDE neon team did manage to rebuild their archive it was a painful process that took over three days. Not something that you would expect in the age of OpenStack, Docker containers and public cloud where the same goal can be achieved in matter of minutes and not days.

“We have a Jenkins instance listening to changes in the Git repositories or on the Tar download sites and rebuilding when it spots any,” said Riddell. ‘When a new release comes from KDE such as Plasma it can be built within a few hours with a high certainty that it’ll all compile and run because we fix issues in the developer editions each day.  We use cloud servers to build on and Docker to provide the build environment, the servers are fairly slow and cheap ones.  Rebuilding the User archive for this problem took about three days and was a useful exercise in discovering a few blockages in the process which we’ve noted for future work.”

Lesson to be learned

Ridder is not walking out of this accident without a lesson. When I asked what  he will do to ensure it won’t happen again, he said, “The tooling we have written for the CI system is called Pangea Tooling and it’s almost fully test-case covered.  I’ll write a test case to make sure this doesn’t happen again.”

He also stressed that other projects can learn from it and should write test cases before starting implementation. “While it’s a commonly said part of modern software development but it’s not always practical or adhered to alas,” said Riddell.

I also hope that the KDE project will dedicate more developers and adopt modern DevOps practices to ensure that the responsibility and onus is not on a single person or a small team. At the same time I also expect that this case will be treated as a learning lesson for the rest of the open source communities. Honestly speaking it was not a big deal or really bad case, but it happened.

If you are a KDE neon user, do what Riddell wrote in the advisory, reinstall the distro from the freshly built ISOs.