This will serve as a generalized guide to developing reproducible virtual machines, and an overview of the steps using Packer and Ansible is given as an example. This approach can be used to vend the same virtual machine image version in many formats e.g. AMI, VHD, OVA. The reader has a background in systems administration or software development or both, is security conscious, and seeks to implement a trustworthy and maintainable systems development life cycle for the infrastructure layers of the application stack.
- Choose a source control manager (SCM) that supports version tagging e.g. Git to house your code and perform the following actions inside the code repository.
- Write a playbook declaring the desired OS configuration. To start, a single Ansible playbook file with several tasks may be sufficient. As your requirements become more complex you’ll employ Ansible roles which are reusable, parameterized configuration.
- Write a Packer template.
- Commit the configuration playbook and Packer template to SCM.
- Use a Git tag to “stamp” an immutable version string on the current SCM revision e.g. release-0.1.0.
- Run Packer to “build” the new OS image in the specified format.
Cloning vs Code
Reproducibility of known, desired configuration is key. Cloning a disk image reproduces the good, the bad, and the unknown. The starting point for your VM image production pipeline is an upstream, vanilla, verifiable OS image; and the goal is to transform it with code into something useful.
Choosing the Upstream Image
The obvious choice is the checksum-verified installation media from the OS maintainers, but it might make more sense to verify and trust a hypervisor-specific, vended OS image e.g. Rogue Wave Software on Azure, because they provide drivers and tunings for that platform.
If you require greater assurances or simply want more help with your image you may have more than one stage of upstream image transform that ultimately produces your functional, deployed image. One example of a multi-stage approach is to source the upstream image from a trusted vendor, and then to apply an OS hardening Ansible playbook. Alternatively, you could source a hardened OS image from the Center for Internet Security.
Either way, this is a time for diligence of risk management. You’ll want to be certain (enough) that you can verify the chain of custody of the upstream(s) to whatever level of rigor your application demands.
Configuring the Image
Now that you have a trusted upstream image you’ll set about customizing it for your application. This too could be more than one stage. You might bootstrap trust for a configuration management system certificate authority in a base image, and subsequently produce multiple application-specific functional images.
You’ll be glad you used a declarative configuration management system instead of shell scripts, and the fortunate soul that inherits your code will too. This means that the code that describes the configuration of your OS declares the desired end state e.g. “bind-utils is installed”, and the configuration system takes whatever actions are necessary to effect that state, and reports back whether this was a success with or without any change. In general, this approach offers a more principled approach which brings improved maintainability, strict error handling, debugging tools, and the like. To boot, you’ll probably learn some Python or Ruby along the way. It’s worth it.
Sealing the Image
It’s crucial to scrub and seal the OS before it becomes a reusable image. High-priority items include
- host SSH identities,
- hardware addresses in network configuration,
- deprovisioning any hypervisor agent e.g. waagent, and
- removing any artifacts of the Packer provisioner(s) e.g. ansible-local requires the playbooks to be uploaded to the prototype VM.
Versioning the Image
Before you run Packer to produce a new OS image, you’ll want to stamp the SCM repo that’s housing all this configuration with an immutable release version string e.g. git tag release-0.1.0. Ideally, this same string is stamped somewhere in the OS of the resultant image e.g. /etc/release and is also in the metadata e.g. entity tags of public cloud images like an AMI. Tagging is a feature of Packer’s “amazon-ebs” builder.
Releasing the Image
Most likely, this OS image you’re releasing will eventually become part of a release kit of many versioned components that together compose the full application stack. Importantly, releases are always immutable, and it is at this point that the OS image is “released” to and consumed by the stack. Any further features and fixes will be new release versions. Likewise, you’ll always be able to build an OS image with the current version by checking out the SCM tag on which it was originally built e.g. git checkout release-0.1.0.
Launching the Image
The new OS image should be referenced by its version string when launched by the application or a infrastructure-as-code provisioning tool like Terraform. This probably means some kind of lookup table that resolves the version string to the image ID. For example, if you used Packer’s “amazon-ebs” provisioner to assign an AMI tag “ImageVersion=release-0.1.0”, then you could find the AMI with that tag in a particular region with aws CLI.
aws ec2 describe-images \
–filters Name=owner-id,Values=$AWS_ACCOUNT_ID \
Upgrading Virtual Machines
You’ll have to judge whether your application is best served by an upgrade migration path or an ephemeral node replacement approach. The latter is more flexible and future forward and requires that the stack architecture abstracts away persistence from the purview of the VM. If you can see a path forward to that end, then “Go West”.
If not, and you opt for the upgrade path, and you’re using a replete configuration management system like Ansible, then you can write your configurations in such as way that they can always be safely re-declared without making unnecessary changes to effect the desired state i.e. idempotent. This opens the door to running a newer version of your Ansible playbook on a VM that was launched from an image built with an older version of the playbook.
For example, one of the tasks in the playbook could install a pinned version of some application. By using Ansible’s modules for package management you would be able to seamlessly replace the installed version when the newer version of the playbook is applied.
*This week’s blog was contributed by our very own, Kenneth Bingham – DevOps Engineer.*