Infrastructure as Code: When Automation isn’t Enough
Some of the best ideas in the tech industry, many of which have revolutionized the way organizations build software, have come from the cross-pollination of strategies and best practices that occurs when all teams from the organization leave their silos and work together on their most difficult problems. It’s widely understood that progress often comes from the sharing of ideas, so it’s no surprise that the fundamental principles behind DevOps are widely accepted as beneficial practices.Agile workflow methodologies were embraced early on by the development community, and today they are widely shared with business and operations teams. Continuous Delivery is an expansion of the concepts behind continuous integration. Continuous integration started as a development strategy and expanded to encompass production deployments, which meant bringing the operations team into the mix. When the principles of DevOps came into play, the concept of Continuous Delivery emerged.
One of the most powerful innovations spawned by the DevOps movement was the idea that operations departments could manage the configurations of servers and virtual machines the same way that developers manage the configurations in their software’s source code. This doesn’t simply mean command line scripts, which sysadmins were already familiar with. It means using a description language or full-fledged programming language to manipulate infrastructure in an automated, repeatable way.
Infrastructure as Code
With the arrival of tools like CFengine, Puppet, and Chef, the concept of Infrastructure as Code was born. These tools enable developers and sysadmins to abstract their problems around maintaining modular, automatable infrastructure, and being able to define them using a high-level language. Once teams started using these tools and techniques, building and maintaining their server environments began to closely resemble the way that software developers build and maintain application source code.
All of the testability, repeatability, and transparency of a modern software development process suddenly became available for people who were building infrastructure. The lofty goal for many of these cutting edge organizations was to be able to completely rebuild a business’ software systems with nothing more than physical server resources, a complete backup of their databases, and source code. Today, that’s how many modern successful tech businesses operate. They have the ability to do exactly that.
Infrastructure Automation at Scale
Infrastructure automation is the process of automatically running a set of custom actions to install an operating system on a virtual or physical server and then configure it according to specifications. When virtualization and cloud infrastructure became widely available, the new IT strategies that emerged were only achievable through some form of infrastructure automation. The ability to spin up new virtual machines in minutes meant that a manual installation with step-by-step instructions for the sysadmin wasn’t going to cut it anymore. Those previously manual steps would now need to be handled by software in conjunction with the creation of the VMs. Otherwise, the full value of cloud and virtualization strategies wouldn’t be realized.
However, infrastructure automation brings its own set of challenges. Any form of automation can cause a host of new issues if it is not carefully managed. Server image templates and copies (clones) are the first step in handling the scale of cloud and virtualization management, but if those templates or copies have any instructions that will cause problems, bad configurations get copied again and again, propagating the issue to a large number of servers. Automating correct infrastructure settings on a large scale can be a powerful ability, but it can be equally devastating if automation instructions are not composed carefully.
There are also dangers that come with the ease and speed of growing an organization’s infrastructure portfolio when they have automation in place. It’s very difficult with the basic automation that many organizations use to keep up with an infrastructure that is constantly growing and changing. The biggest problem that infrastructure automation can cause is Configuration Drift.From the Continuous Delivery Report glossary: [link to glossary page]
Many of today’s deployment problems and IT outages are caused by Configuration Drift. Configuration Drift isn’t limited to infrastructure configuration, either. Middleware settings also experience a great deal of untracked changes in many organizations. This causes numerous Configuration Drift-related errors, especially in testing and acceptance environments. Disaster recovery and high availability failures are also common results of Configuration Drift. Having a feature in your deployment automation solution, whether it’s custom or provided by a product vendor, is highly recommended. Detecting configuration drift is the first step to overcoming it.
Configuration Drift - A term for the general tendency of software and hardware configurations to drift, or become inconsistent, with the baseline or template version of the system due to manual ad hoc changes (like hotfixes) that are not introduced back into the template.
It’s important to note that infrastructure automation is not the same as Infrastructure as Code. Infrastructure automation can take repeatable actions and enact them across any number of servers, but Infrastructure as Code includes tools and capabilities from the software development world to build, maintain, and test those actions in a more manageable way.
Developer Lessons for Infrastructure as Code
The biggest danger when trying to integrate an Infrastructure as Code-enabling configuration management tool like Chef, SaltStack, or Ansible in your software production process is not applying the same lessons learned by the development world in the last ten years. Infrastructure code needs to be version controlled, tested, maintained, quickly deployed, and crafted for easy usability.
Development principles that date all the way back to the beginning of Extreme Programming will provide the best guidance for making Infrastructure as Code into a blessing rather than a curse. These are the principles that organizations who are new to Infrastructure as Code should focus on:
Include infrastructure code in the software development lifecycle by putting it through build, testing/QA, and production along with application code.
Start with and maintain a simple design for your infrastructure code and any new features. Use design patterns.
Have the infrastructure team agree on the design and direction of the system.
Institute code reviews with the help of team notifications for each commit. Pair programming is also helpful.
Maintain a shared set of coding standards for the infrastructure code.
Continuously test with as many of the same styles of development testing that could be applicable to Infrastructure as Code. This ensures that the code definitions produce the correct environments and don’t introduce problems.
Employ infrastructure monitoring for testing and pre-production environments as well as the production environment.
Refactor the infrastructure code to reduce the amount of maintenance needed.
Organizations don’t want to waste their time or money using tools that don’t provide the features needed to avoid the biggest pitfalls in Continuous Delivery. Beware of tools that let you easily create endless server templates but don’t allow proper traceability for template changes like the inevitable quick fixes that are prevalent in software production.
The software should be able to externalize infrastructure code definitions that can be stored in common version control systems. Organizations should be wary of tools that don’t allow the usage of community tools that benefit from large community knowledgebases. It’s better to use tools that are flexible and allow you to produce future-proof systems that can take advantage of the many innovations in the wider development tooling community.
Infrastructure as Code emerged from the development community through open source configuration management tools that were shared because so many organizations were having the same problems when using basic, non-programmatic infrastructure automation. The best piece of advice to internalize about using Infrastructure as Code is to stay connected to the vast community of innovative developers. If your systems remain open to the rapid changes in that community, you’ll be able to share and benefit from the cutting-edge ideas that will make your organization successful.