The lead KDE neon developer Jonathan Riddell released a security advisory in which he urged KDE neon users to upgrade their systems or reinstall it.\n\nTLDR;\n\nThe package archive of KDE neon was misconfigured, which allowed anyone\u00a0to upload a package to the archive. It\u2019s not known whether anyone uploaded any packages, but as a security measure they purged the archive, removed the ISOs built from it and they are now rebuilding it. The KDE neon team is urging users to upgrade to the latest packages once they are rebuilt. If you want peace of mind, reinstall KDE neon using the latest built ISO from the site.\n\nThe backstory: what actually happened?\n\nI spoke with Riddell and asked him if there were some best practices or security protocols to be followed. He explained that since KDE neon is growing (in size) and now has almost all of KDE software which is maintained, the neon team had to move the archive that was on the same server as their build server to a server with more disk space. \u00a0\n\n\n\u201cBest practice when setting that up is to use the trusted SSH protocol to open a tunnel and upload through that,\u201d said Riddell, \u201cUnfortunately in setting up that new archive I had it listening for uploads on the whole network not just the local SSH tunnel.\u201d\n\n\nThat\u2019s it. That\u2019s all that went wrong. \n\n\nI recall my discussion with Richard Brown, the openSUSE Chairman, who stressed on building systems in a way that experienced engineers and developers are involved with a project so that not all the burden and responsibility is placed on a single person.\n\n\nThat was how things were with with Kubuntu, a project that Riddell started and then quit. Riddell told me, \u201cAt Ubuntu the archives are managed by a team of dedicated sysadmins and developed by the Launchpad team.\u201d\n\n\nUnfortunately, KDE Neon is a much newer and smaller project with a smaller team consisting of less than half a dozen contributors using off the shelf technologies to set it up. \u201cWe're really proud of being able to recreate setups that companies spend millions developing ourselves,\u201d said Riddell.\n\n\nIt\u2019s commendable that Riddell\u2019s smaller team did manage to fix things. But in the long run I think the KDE project should invest in smaller projects like neon, because in the end you are a KDE user.\n\nOpen source communities slow to adopt DevOps practices!\n\nThe irony is that open source kind of triggered the DevOps movement that stresses on agile development and agile teams, where as KDE and many open source communities are still living in the legacy world and are slow to adopt DevOps practices.\n\n\n\u201cThis is strange because as teams who work solely on the internet we're ideally situated to use systems which automatically build, test and deploy without human interference,\u201d said Riddell.\n\n\nWhile the KDE neon team did manage to rebuild their archive it was a painful process that took over three days. Not something that you would expect in the age of OpenStack, Docker containers and public cloud where the same goal can be achieved in matter of minutes and not days.\n\n\n\u201cWe have a Jenkins instance listening to changes in the Git repositories or on the Tar download sites and rebuilding when it spots any,\u201d said Riddell. \u2018When a new release comes from KDE such as Plasma it can be built within a few hours with a high certainty that it'll all compile and run because we fix issues in the developer editions each day. \u00a0We use cloud servers to build on and Docker to provide the build environment, the servers are fairly slow and cheap ones. \u00a0Rebuilding the User archive for this problem took about three days and was a useful exercise in discovering a few blockages in the process which we've noted for future work.\u201d\n\n\nLesson to be learned\n\n\nRidder is not walking out of this accident without a lesson. When I asked what \u00a0he will do to ensure it won\u2019t happen again, he said, \u201cThe tooling we have written for the CI system is called Pangea Tooling and it's almost fully test-case covered. \u00a0I'll write a test case to make sure this doesn't happen again.\u201d\n\n\nHe also stressed that other projects can learn from it and should write test cases before starting implementation. \u201cWhile it\u2019s a commonly said part of modern software development but it's\u00a0not always practical or adhered to alas,\u201d said Riddell.\n\n\nI also hope that the KDE project will dedicate more developers and adopt modern DevOps practices to ensure that the responsibility and onus is not on a single person or a small team. At the same time I also expect that this case will be treated as a learning lesson for the rest of the open source communities. Honestly speaking it was not a big deal or really bad case, but it happened.\n\n\nIf you are a KDE neon user, do what Riddell wrote in the advisory, reinstall the distro from the freshly built ISOs.