Best practices for a secure and trustworthy container platform strategy

Containers have and will continue to disrupt software development and distribution. Containers enable software to be easily deployed across environments by forcing a consistency in the software’s development and operating environment. This, in turn, makes it easier for developers to build and deploy their code, and also restricts the number of production issues arising from differences in the operating and development environments. Container technology has a direct impact on the agility of a software development team and consequently have seen a huge increase in interest, adoption and usage.

Differences in development and production environments

Software production deployment typically faces hiccups due to the following types of differences between the production and development environment:

Tools, libraries and OS and infrastructure

Often, variations in the infrastructure, operating system, libraries, and tooling are responsible for the unexpected and unforeseen behavior of software outside the development environment. Applications that have been thoroughly tested and deployed in non-production environments can still exhibit patterns that do not match the intended design of the development team. This is because slight variations arising from different versions of libraries or from slightly different implementations of the tooling or differences in the state of the operating system can impact and change the behavior and reaction of a piece of software has to its inputs.

Network topologies

Production systems typically have a lot more fine-grained and robust network topologies to enable high levels of connectivity and availability. Developers typically will not have access to nor use topologies similar to production networks while developing the application. Given the differences in the software, hardware and networks associated with production and development environments, software that works reliably in a development environment network topology can face issues when running in a production environment network.

Security policies

Development environments almost always have vastly different security policies compared to production environments. Most commonly, developers typically have ‘root’ level privileges in their development machines whereas, in production, most applications do not operate with root level access. This means that software which might be operating with the highest levels of permissions and access to the local development environment may not be able to do so in production. In addition, security policies, access control etc. are often implemented with very different libraries, protocols, and techniques making it likely that the software will stumble in production.

Storage

Development environments typically let developers use local disk and file storage for application development purposes. Developers can easily access the local file system and use it to read, write and store data. However, in a production environment, storage mechanisms can vary from central, network storage or cloud storage. Such variations in storage can make the application behave differently and assumptions made during development might not hold in production.

Best practices to manage development to production transitions with containers

While containers offer a solution to the problem of incongruent development and production environments, they also open up new challenges if used incorrectly or inappropriately.

Using malicious images

Base container images sought by developers can be infected with malicious or low-quality code and configurations that infect every application built on top of the image. Such vulnerabilities can also make it possible for the application to be taken over by a remote attack once deployed to production. Developers should only download and use images from trusted sources and use one of several available image vulnerability scanning solutions available to ensure that the images are trustworthy. Central IT teams should ensure that enterprise developers cannot download container images from outside the enterprise unless and ensure that all images that are deployed to production have been certified by IT, Security and QA.

Creating unsecure images

Developers that are creating and contributing images to the central registry for other developers to find and reuse should be very careful about the data that they include in the images. Information such as secrets, passwords, application data, user data etc. can often be, inadvertently, left inside the image and the image with the data shared with other developers. Developers should double check that any such data is not left inside the image. In addition, tooling should be built and leveraged that can list and identify such data leakage issues.

Signing images

It is a good idea to have various development teams to sign the container images before the image is deployed in production. This ensures that the deployment team can verify that the images are known, trusted and verified for quality. This also mitigates the introduction of malicious or corrupted images from entering the production environment.

Time drift

For best application performance in an ecosystem of applications and services that communicate with each other, it is critical that all containers and the applications running in the containers are synced to the same clock to avoid time drift and the resulting unpredictable application behavior.

Monitoring container behavior

Another very useful technique to ensure high performing container based production deployments is that of profiling, baselining and monitoring. Automatically monitoring container behavior across the data it accesses, the services it accesses and the operations it performs is extremely important. Any deviations from the expected or usual behavior should be quickly detected and addressed.

Data backup and storage

Because containers offer fast boot up and teardown and are dependent on the continued availability of the underlying infrastructure, it is very important that container level backup is implemented. Procedures that backup container data and information continuously to protect against data loss from any infrastructure or production environment level outages and issues are well established and should be aggressively adopted.