Exploration of best practices for and ownership of the build and deployment of Containers
An exploration of the methods available to build container images.
An attempt to answer the following questions:
- who should own the container images?
- should the deployable artifact of a successful CI / CD build pipeline be:
- The app binary
- a ready to deploy container image containing the app binary + all necessary runtime config and binaries?
- If devs were solely responsible for building container images:
- How would operators deal with the operational burden involved in managing a large number of snowflakes?
- How would security sign off on each container?
- How would patching work?
- If operators were solely responsible for building container images:
- How would developers deal with handoff of their app binaries?
- How would developers know what was in each container?
- How would application code and configuration get added into a container?
Methods for building Container Images
- Use Buildpacks, a piece of software used to provision container images in Heroku and Cloud Foundry
- Begin with a base image (possible to roll your own)
- Either Launch the container and perform manual steps, or use a provisioning tool
- Manual Provisioning a Docker image
- Launch Docker container
- ssh into the container
- execute arbitrary commands
- commit image to repository
- ADD declarations can be used to bring files from an external location into the container image
- RUN declarations can be used to run arbitrary commands (e.g. apt get, run a shell script copied into the container)
- build image
- commit image definition to repository
- Docker API
- same options as above, but all via the Docker API
- Docker API was still a little buggy last time I checked; not all features work as expected
- ? find other possible ways to provision Docker images
Containers built & owned by Developers
- Developers select the base image type
- Whichever method is used to build images, developers are solely responsible for building the container
- For any prod app outages, devs will need to be brought in to help resolve issues since operators won’t necessarily know what’s running in these containers
- This approach leads to an increased operational burden
- Security teams will recognize the inherent risks associated with the potentially large number of image types and arbitrary tools that might be installed on a per-app basis
- Best practices will be difficult to manage across many teams in large enterprises
- Operators will not be comfortable taking a questionable container image from developers knowing that their job is to ensure stability, etc in production environments
- Aside from app binaries such as stateless microservices, would developers also be able to deploy containers containing persistent backends like MySQL, Redis, or Cassandra?
- CI / CD build pipelines could help with consistency, and provide guardrails, but to what extent?
- How would we prevent arbitrary binaries/code/scripts from being installed or executed during image creation?
- How would we limit the base image types in order to achieve a higher degree of consistency?
Containers built & owned by Operators
- Strong DevOps culture would need to exist in order to effectively handoff app binaries to operators
- operators would be responsible for creating container images, including base images with runtimes for apps built in languages such as Java, Ruby, Python, etc
- operators would be responsible for applying patches
- operators would be responsible for working with security for everything from the app down to the OS, which better resembles how most enterprises work today
- operators would need to build in hooks for developers to add in their app binaries and config as part of a build process
Containers co-owned between Developers & Operators
- Is it possible to allow operators to build base images which can be extended by devs in a safe way?
- How can operators maintain control over what gets added into a container image while still offering some degree of flexibility to developers?
- 12-factor approach to building applications applies very well here since environment variables are defined in the contract of the container image, and can be used to provide runtime config for apps
- Buildpacks provide a consistent method to produce container images while allowing for narrow means for developers to introduce app-specific config as necessary