FP Complete builds on cutting-edge open-source devops technologies, providing devops solutions and consulting to a number of companies in life sciences & health, financial services, and secure Internet services. This exposure has given us a chance to work with some of the best engineering practices in devops.
Contact us if we can help you too
As we bring more companies forward into the world of devops, we will continue to share lessons learned and best practices with the IT community. Today’s best practice: immutability.
What is immutable?
As a software engineering concept, immutability means that once you assign a value or configuration to some entity, you never modify it in place -- if you want it to change, you make a new one and (optionally) tear down the old one.
As we know from functional programming (FP) and our work with Haskell, immutability boosts the reliability and predictability of system behavior -- preventing bugs and downtime, and increasing the speed and reproducibility of software development and deployment. A variable, program, or server is immutable if we guarantee, once it’s set up, that we won’t modify it in place. We can do this if replacements and deployments are cheap.
Devops has made it so much cheaper to build and deploy new software services and even whole servers and clusters, some companies are now taking advantage of immutability -- leading to much more reliable online services with less downtime and more frequent, reliable, repeatable updates.
Old-fashioned servers are mutable
Mutability describes how most traditional software, and most traditional server operations, are done. It means that once you create something, to avoid the terrible cost of creating another one, you just keep modifying the one you already put in place. Software patches, configuration file changes, even changing the value of a variable that’s already in use -- all these are examples of mutability. It’s always been a bit risky, but can be economical when (1) the cost of creating an entity is very high, and (2) the cost of bugs and mistakes is very low.
Unfortunately, mutability is a key cause of bugs and mistakes.
Consider this with a person instead of a computer. If I break my arm (an accidental mutation of my state), we could try to fix the problem using an immutable method: make a new Aaron, identical to the old one, but with a non-broken arm -- then tear down the old Aaron who is no longer needed. Obviously we just don’t have the technology to do this -- it’s beyond unaffordable -- so we are forced to use mutability. We patch up my arm, and wait for it to heal.
That’s way better than giving up, but now our managed service (Aaron) is in an unprecedented, irreproducible state. For weeks my arm is offline, in repair mode, while the rest of me runs. And for the rest of my life I may have to keep track of the fact that this arm is a little weaker, and there are some things I cannot do. My boss now has to remember: Aaron has this special flag that says he can’t lift some kinds of heavy boxes. What a pain. At least humans are flexible, so my colleagues won't just break with an "Error 504" when they try to shake my hand.
If only I could reconstruct the arm in its original state -- or even a whole new Aaron -- life would be so much easier. We may never have that for humans, certainly not in my lifetime. But thanks to modern devops technologies, we do have that option for servers. They don’t need to be modified in place, and they don’t need to run in unprecedented, irreproducible configurations that lead to many of today’s sysadmin emergencies, security breaches, and downtime.
How do we make servers immutable?
Our FP Deploy approach to devops is based on heavy use of containers (notably Docker), virtual machines, and where feasible, virtual networking, cloud computing (AWS, Azure, etc.), and virtual private clouds (VPCs). Every one of these technologies has something in common: it allows us to abstract away the work of creating a running online service. Configurations can be written declaratively, put under source control (say, in a Git repository), and run at any time (using, say, Kubernetes).
You want another server? Just run the deploy command again. You want another whole cluster (a “device” made of multiple servers and associated networking and data connections)? Just run that deploy command again.
By slashing the cost of deployment, we make it possible to create a whole new server painlessly, cheaply, and reproducibly. Developers just delivered a bug fix? Don’t patch the application server! Bring up a new instance based on the new software build. Once you’re happy that it’s running properly, bring down the old, less-good instance.
(In a future post we’ll talk about blue-green deployments and canary deployments -- cost-effective, easy techniques for making this transition cautiously and gracefully.)
An immutable server has a known, well-understood, source-controlled, reproducible configuration. No footnotes. We can be confident that the new production servers are the same as the engineering test servers, because they were created by running exactly the same deployment files -- not by a series of manual admin tweaks that could be incorrect or have latent side effects.
This also makes it easy to recover from disaster, or scale up for increased load, by redeploying a new server from the same deployment files.
We can afford to do this only because modern devops makes it so cheap to create new servers. When doing it right is automated, predictable, repeatable, and inexpensive, there’s no longer any reason to do it wrong.
Can whole clusters, or distributed devices, be immutable?
It’s easy to assert that an application server can be made immutable, because well-architected web app servers are fairly self-contained and fairly stateless. But what if you are making larger changes to a whole distributed device, consisting of many servers and network connections? What if, for example, you have made matching changes in both your front-end server and your back-end server? Or in any group of services in a service-oriented architecture (SOA)?
Then you step up to immutable devices, immutable clusters or VPCs or distributed systems, using exactly the same methodology. At FP Complete we routinely create whole 10+ virtual machine devices on command, even just for testing purposes, because we’ve automated it with FP Deploy. Again, why not do it right? Why not be confident that the whole distributed system is in a known, reproducible state? We can be much more assured that what worked in test is going to work in production.
However, sometimes we don’t have that luxury, for example if we are retrofitting modern devops onto parts of an older system that has not been uniformly upgraded -- or in any case where the service we are upgrading is far cheaper to redeploy than another, perhaps more stateful, service in the same distributed system. That’s ok: we can have immutable servers as parts of a mutable device. The administrator of the distributed system now has to track which services have been replaced with newer versions, but at least for any given service the servers can be immutable.
What about the database server?
Once we have made it cheap to rebuild and replace servers at any time, immutability can be a reality, and reliability and reproducibility go way up. However, not all servers are so easily replaced. In fact, servers providing key input and output channels may be outside the scope of our control -- so all we can do is treat them as external to the immutable device, and understand that hooking up the inputs and outputs needs to be part of the declarative script used to bring up any new version of our device.
The strongest example of this may be an enterprise database server. We typically have no intention of building a whole new database as part of our application build-and-deploy process. Databases are typically long-running and, fundamental to their purpose, they are extremely stateful, extremely mutable. Cloud services such as RDS make it easy to spin up new database instances, but often the contents of an enterprise database are too large or fast-changing to want to rehost -- or we just don’t have permission to do so. Instead, we leave it in place and accept that its contents are very mutable.
So even when we use immutability to make our application server clusters easier to upgrade and less prone to errors, we need to understand that they will almost always be connected up to other servers that lack this golden property. Automated deployment with modern devops, ideally with truly immutable servers, can ensure that your system looks just like the system that worked in test -- eliminating a lot of surprise downtime and deployment failures. But even at companies with modern devops, failures still happen -- and when they do, it’s because the test system was not exposed to the same kinds of inputs and outputs, and the same database state, as the production system. In a future blog post, we’ll look at some best practices for testing and quality control.
Have a look at your own software deployment practices. Are there too many (more than zero) manual steps involved in bringing up a new or updated server? Do you allow, or even require, sysadmins to make changes to servers that are already up and running? Maybe it's time to use automated, reproducible deployment, and make the move to immutable servers.
For more assistance
In addition to our customizable FP Deploy devops solution, FP Complete offers consulting services, from advice to hands-on engineering, to help companies migrate to modern devops.