From the Ashes Part 3: The Ugly Side of Infrastructure as Code

Last year, we built up the Spine platform’s infrastructure from scratch using an Infrastructure as Code (IaC) template inherited from an earlier company.

I’ve spoken at length about the ways that IaC was instrumental to this rebirth, but reflecting on our journey I’d also like to explore the messy cases where the technology itself became a hindrance.

As a counterpoint to both “The Good” (where IaC worked well to support our journey), and “The Bad” (where it couldn’t solve all our problems), this is the “Ugly” side of IaC – the messes, the pitfalls, and the downright unhelpful aspects of the technology.

This article is intended as a sharing of war stories and a highlighting of some particularly easy mistakes to make with IaC – terraform, in particular. I’m not aiming to discourage you from adopting these technologies, but I am trying to pass on a little healthy caution, earned through this somewhat extreme-case experience.

The Ugly

As before, everything I’m going to talk about came up while rebuilding Spine from IaC, mostly reflecting bad practice or curious corner cases rather than fundamental flaws in the IaC pattern. Unlike the “Bad” scenarios, however, these examples resulted in significantly more head-scratching, confusion, and frustration.

Let’s start by expanding on something I mentioned briefly in part 2:

• IaC security is usually left as an exercise to the user

  
  

A core aspect of IaC’s value is that users can define their infrastructure declaratively, and the framework takes on the responsibility of figuring out how to get to that declared state.

This means you don’t write the imperative steps for setting up your infrastructure:

e.g.

  ProvisionDatabase(…)
  ConfigureDatabase(…)

You simply declare the resource you want:

e.g.

  Database(…)

It’s the IaC framework’s responsibility to figure out what this means:

• Don’t have a Database instance? It will create one and then configure it.

• Already have a Database, but not matching your declared config? It will reconfigure the instance for you.

•  Already have a Database matching the declared config? It will leave things as-is.

To do this, the framework creates its own data record (referred to as the ‘statefile’ or simply ‘state’) to keep track of the cloud resource(s) it manages. This state is then compared with the IaC definition to figure out what the framework needs to do (more on this comparison later).

In short, the statefile is IaC’s memory and is essential for the long-term maintenance of an infrastructure stack.

It’s also its biggest security vulnerability.

That’s because the statefile doesn’t just track the identity or definition the associated IaC resources, it also stores any outputs generated by those resources. Database credentials, for example, or private certificates, or the intimate details of your secure networks. All stored in plaintext.

Doesn’t matter if you’re storing your statefile in source control (but you really shouldn’t) or as remote state in S3: if someone gets access to it, you’ve just given them the keys to your infrastructure kingdom.

What about sensitive variables though?

Unfortunately, “sensitive” only masks the values in logging output – it’s still stored in plaintext in state.

Ultimately, this is why terraform is quick to encourage you to encrypt your statefile at rest. However, this just shifts the security problem to being one of access management, which is a fairly big “exercise left to the reader”.

In recent years, terraform has introduced ephemeral resources which help by not storing the most sensitive values in state. However, these aren’t widely used for many resource types yet – and certainly won’t help any historical IaC you have kicking around.

Don’t worry though, security is probably not the biggest headache you’ll get looking back over old code. After all:

•  IaC often isn’t good code

HashiCorp Configuration Language (HCL) is the declarative language that (HashiCorp) Terraform uses to define IaC resources. The nominal “C” in IaC.

But, if I’m honest: it’s not a great code.

I’m not talking about its applicability outside of an IaC context, extensibility, or even performance. I mean that to someone familiar with coding in the application space, its features and syntax can be downright alien.

Part of this comes down to the declarative nature of IaC: HCL is not attempting to be a language that describes the process of deploying infrastructure, it’s a language that describes infrastructure. Specifically, infrastructure broken into independent “resource” units.

This means that HCL is optimised towards self-contained resource definitions. As such, any and all logic relating to a resource must be contained with that definition.

This is particularly counterintuitive for application developers when it comes to flow controls like conditionals (“if X then Y else Z”) or loops (“for X do Y”).

You cannot, for example do this in HCL:

if var.order == “sandwich” {

    resource “sandwich” “blt” {}
    
}

Instead you have to:

resource “sandwich” “blt” {
    # Do this once if my condition is true, otherwise do it 0 times
    count = var.order == “sandwich”? 1 : 0 
}

Which, ew.

And again, resources are independent, so you definitely can’t do this:

if var.order == “sandwich” {
    
    resource “sandwich” “blt” {}
    resource “sandwich” “cheese_and_ham” {}

}

Instead, you have to add the conditional to every affected resource, like so:

locals{
  sandwich_count = var.order == “sandwich”? 1 : 0
}

resource “sandwich” “blt” {

   count = local.sandwich_count

}

resource “sandwich” “cheese_and_ham” {

   count = local.sandwich_count

}

(Extracting the conditional into a local because I like my BLT DRY 😉)

Which is already becoming unreadable to the point of inscrutability.

You can, of course, introduce modules to group common resources together, but that’s like trying to crack a (syntactical) nut with a delivery truck. I shouldn’t have to start to delve into package management (with all the complexity and readability issues that introduces) just to get sane flow control.

Now sure, with enough exposure these patterns may become familiar or understandable to someone who lives and breathes infrastructure. But, the point of DevOps is to break down the barriers between application development and infrastructure operations. These kind of peculiar language patterns only serves to add a significant mental overhead when switching between application and infrastructure contexts.

To work around this limitation, Spine makes use of CDK for Terraform (CDKTF) for our platform infrastructure. This allows us to use a more powerful programming language (in our case, typescript) to wrap terraform, giving us all the power of a full programming language (proper flow control, type safety, testing frameworks, and object inheritance, etc, etc) to manage our more complex infrastructure projects.

However, the additional abstraction layer of another entire language introduces a second problem: how do you translate between HCL and an in-code representation of a terraform resource?

Depending on the resource, the answer varies wildly. One resource may be an almost-one-to-one transliteration of HCL structure, another might be fundamentally different, while a third might throw up its hands and force the user to render pseudo-HCL as a string (removing all the type safety and validation benefits which CDKTF brings).

As an aside: Tragically, Hashicorp sunset CDKTF at the end of 2025, although there is still hope that the community steps in to fill the void. This adds another major wrinkle to the transliteration problem, since the documentation for typescript (et al) resources is starting to fade away.

Anyway, why is there so much variation of resource behaviour?

You see, the terraform ecosystem is made up of a series of plugin packages called “providers”. Providers allow the IaC framework to be extensible to an endless list of possible cloud environments and resource types.

However,

•  IaC providers are not very consistent

Provider code is generally maintained by the (cloud) company responsible for the platform and/or resources in question, since having official IaC patterns makes your cloud offering more accessible.

In a lot of ways, this is a huge strength of the IaC ecosystem: the coming together of huge competing cloud companies behind an accessible standard that empowers end-consumers. On the flip side, it means that the IaC kitchen is filled with an awful lot of cooks, and most of them are bringing their own menu, cookware, and dietary requirements (to torture the metaphor).

The broth is looking awfully murky, is what I’m getting at. Who knows what that next spoonful will bring?

Sometimes, the provider developers will switch semantics (or syntax) between provider release versions – forcing a lot of painful re-writing of code and/or surprising changes then next time you run your IaC.

Of course, this is largely unavoidable: provider functionality is going to track the unique APIs, platform features, and stylistic choices of the development teams at each major vendor. It’s no surprise that we’ll get some amount of inconsistency.

But don’t forget – the overarching syntax is standardised (and largely outside of provider developer control). This means that you can have two resources side by side which have substantial differences in scope, lifecycle semantics, or idempotence (to name just a few) but which look almost identical.

This turns sufficiently complex IaC into a minefield of subtle gotchas and future bugs. At best, this adds yet more mental overhead to the developer who has to maintain it.

At worst, you can get production outages that you can’t easily revert, because:

•  Some IaC resources plan lazily

As a reminder, IaC. Is usually executed via a two phase “plan-then-apply” approach. The purpose of the Plan is to detect the changes necessary to update the running condition of the resources to match the IaC definition. The Apply stage then enacts those changes by talking to the underlying APIs of the cloud platform(s) involved.

The Plan gives the IaC user a chance to review the changes to be applied before actually applying them and ideally validate that the update could succeed. It can do that by performing a three-way comparison of the IaC definitions, the remote IaC state, and the running state of the resources (by querying the cloud APIs).

But I said “ideally” there, didn’t I?

In many cases, the final part of that comparison puzzle – the callout to the actual cloud APIs – is skipped. This is in part a performance consideration: you don’t want to make remote cloud calls unless you really need to.

This is fine, provided that you have complete confidence in:

1. The accuracy of your IaC state store

2. The provider’s ability to validate the IaC definition locally

3. That the cloud service expectations line up with what the provider is validating

If there are any weak points in this iron triangle, then your Plan stage outputs are going to give you false confidence for your Apply.

We’ll talk about some situations that case 1) doesn’t apply later in this post, but 2) and 3) are the main issue I’m talking about here.

Simply put, cloud infrastructure is now a very complex beast. It is very likely that an IaC provider won’t perfectly capture every possible edge case or requirement of every possible cloud resource. And even if it did, that complex set of conditions and requirements changes rapidly as the cloud tech evolves (hello again, incompatible version updates).

Let’s explore a representative scenario:

1. On Plan, the provider code does some local validation of the resource syntax, but misses some meaningful error in the semantics.

2. It checks against the remote state and figures out what needs to happen, and presents these changes to the user.

3. The user, brimming with happiness that their Plan contains the changes they expect runs the Apply.

4. On Apply, the provider code runs the same analysis of the IaC and state, and then calls the cloud APIs to enact the change.

5. The Cloud API responds with an error – the change either can’t happen, or it can’t happen that way, or can’t happen to that resource.

6. The Apply process fails.

7. The user puts down their cup of tea and begins to sweat.

This is obviously bad from a “fail fast” perspective, but it can be devastating depending on where it occurs in the Apply process.

Consider the following dependency chain:

1. Network resource 2. Security group 3. Custom Resource Definition 4. IAM role 5. Database 6. Managed service 7. Secrets 8. Message queue 9. Database 10. Application A deployment 11. Application B deployment 12. Monitoring rules

If your Apply error occurs on resource 1, then good news: you’ve still failed fast.

However, consider a scenario in which the error occurs on resource 5, having already updated resources 1 – 4. If you’re particularly unlucky, that renders resource 6 – 12  inoperable, since they’re set up for the previous versions of resource 1 – 4. And there’s no easy fix forwards, because your IaC for resource 5 is blocking the necessary updates. Maybe you’re also unlucky enough that you also can’t roll back some (or all) of resources 1 – 4 because of version compatibility.

Oh, and I hope you got your dependency ordering correct, or this is suddenly a mutually-blocking circular dependency.

That sensation you just felt going down your spine is operational dread (DreadOps), and it’s the reason your development team is weirdly tribal about pushing changes on a Friday.

Now, if the IaC provider simply talked to the cloud APIs in the Plan stage rather than just doing some local validation, then this failure could be caught in the Plan stage where it belongs.

Now I don’t want to seem to point the finger of blame solely at provider developers  here. The reason some resources don’t do this kind of remote validation is that they can’t: the cloud API required to validate your change may not exist.

But considering again that (most) IaC providers are developed by the same cloud company that develop the APIs (albeit likely very different teams), I definitely can be annoyed at the cloud vendor as a whole.

In fairness, this isn’t a universal problem: a lot of providers do implement a thorough 3-way change detection for the Plan phase. But again, this inconsistency of approach is just another landmine hiding in your IaC.

Worse, since details of the Plan methodology are generally not part of the standard provider documentation, this is a landmine you just have to step on before you can discover it.

And honestly, inconsistencies in resources validation are at least not a conceptual problem for IaC, unlike:

•  Some IaC resources don’t automate well

IaC – or any declarative framework, really – depends on three key principals:

• Determinism – that the desired end state is achieved regardless of the starting point

• Independence – that things happen in separate, complete units of work

• Idempotence – that nothing changes if nothing needs to change

Or put another way: if you define some IaC containing three resources, then regardless of the starting condition of any of those three resources, applying that IaC any number of times should result in your desired state every time.

But some cloud resources are either not independent (you need multiple of them to do some singular thing) or not properly deterministic* (can’t reach their end state without outside intervention).

* - This is partly why you can get into the kind of Apply-time failure I talked about in the last section: If all resources were truly idempotent you should always be able to fix forward (barring some entirely invalid input).

This anti-pattern manifests itself in a number of different ways: Some resources need to be walked through a particular lifecycle of changes (e.g. you can’t jump straight to the final desired state), some logical resources are split across multiple IaC resources, reflecting some two- (or more) phase process, such as a request and acceptance.

That last case is particularly problematic for IaC, as it may necessitate both halves of the equation to be run in different accounts. Trying to do both in a single IaC stack is a security problem at best, and a really fiddly exercise in juggling providers at worst.

If all of this is starting to sound very imperative, at odds with the declarative intent behind IaC, then you’ve spotted the reason this bothers me so much.

That all said, there may be very good reasons for the cloud resources to be designed this way:

Take twinned “request/accept” resources, for example – this kind of design is usually security-focused: you don’t want strangers to be able to unilaterally peer their Virtual Private Cloud (VPC) to your private network, after all. So instead, your sharing account needs to accept the peering request separate from the account requesting it.

Here the IaC resources are mimicking the human (well, admin) processes that would be carried out in some cloud console, but this results in a disjointed automation experience.  So the problem isn’t the cloud resource design per-se, but rather that the IaC pattern just isn’t suited for this kind of process.

But to repeat myself from part 1: not every part of your infrastructure needs to be IaC.

Ok, that’s enough body blows directed at the provider developers. Now it’s your turn.  

Yeah, you.

•  Manual changes will ruin your day

Be honest: you knew this was coming, right?

I’m talking about manually-applied changes to resources which are otherwise managed by IaC. Let me break it down for those of you who aren’t currently shuffling awkwardly and looking at your shoes. It goes like this:

1. Something caught fire™

2. You leap on the incident, determine the root cause, and identify an infrastructure-level change/fix/bodge that will put out that fire.

3. You could then:

   1. Figure out how to represent that change through the IaC runes

   2. Apply that change to your IaC codebase

   3. Run that IaC change through its pipeline – probably promoting through multiple increasingly production-esque environments (and potentially get distracted by unrelated test failures or other issues along the way)

   4. Get chewed out for taking so long to resolve a P1 incident

4. Or you could:

   1.  Make a few clicks in the Cloud dashboard

   2. Seem like some infrastructure wizard

An obvious choice.

And so 4) happens.

Is that kind of manual change good practice? Absolutely not.

Can it cause unintended consequences? For sure.

Does it make sense in the pressures of a customer-impacting outage? Unfortunately, yes.

This is the secret shame of many a platform engineer (myself included) but let those without IaSin cast the first stone.

But what’s the problem here exactly? Earlier, I discussed how IaC stores information to recording the running state of the infrastructure it manages: this is crucial for the IaC to figure out how to act because IaC is written declaratively.

What infrastructure manual changes do is to falsify the state record – making reality meaningfully different to what the IaC thinks should be there. This results in some non-deterministic failure modes for your IaC depending on the resource(s) affected, and the exact nature of the divergence of the state from reality: in some cases your Apply stage  will fail when some do-once operation is run for a second time (e.g. a database with the desired ID already exists), in others the earlier Plan stage will be filled with unexpected, possibly confusing, changes.

The solution is simple: just don’t change things outside of IaC. However, this effectively locks you out of using Cloud dashboards altogether – which is a shame, because these can be well-designed and effective products in their own right.

More importantly, these forbidden tools are often the more expedient option. Consider a typical urgent fix: a version bump, a redeploy, a new certificate – whatever. To do things properly™ you would need to:

1. Figure out which bit of IaC controls the affected resource

2. Figure out how to enact the change you want in IaC syntax

3. Run through an IaC plan to see what your change will actually do (and check there are no undesirable side-effects)

4. Run the IaC apply process

5. Hope you didn’t make a mistake anywhere in steps 1-4

Some of these steps might be buried inside a CI/CD pipeline and so also introduce time waiting for runners, tests, or other stages.

By contrast, if you tried to do this through the cloud tooling directly you would:

1. Figure out which dashboard (or subsection of a dashboard) controls the affected resource

2. Locate the button which does thing

3. Push that button

4. Feel vaguely anxious that it can’t be that easy

So, it is no surprise which option is selected by stressed engineers under the cosh of a high-stakes outage.

And, to absolve some of you a little: I think that there are a lot of justifiable reasons to take that expedient option, and that it is ok to do so if you close the loop after the fact and update your IaC to match what you did manually.

That is a very, very load-bearing “if”.

Still, suppose you remain well-behaved and only do things properly through IaC itself. Nothing can go wrong, right? After all, the tools themselves are built to avoid accidental mistakes, right? They protect against that sort of things, right?

Right???

• Delete protection doesn’t protect against deletion

Ok, this might be a terraform-specific bugbear, but I suspect this is going to shock a lot of you.

For context, in terraform IaC all resources have a special metadata block to describe certain lifecycle behaviours of that resource – what to do if the resource is destroyed, for instance. One widely-used option is delete protection: a failsafe to ensure that if you accidentally delete a resource, the IaC process will error out, rather than wreck your day.

Except that isn’t what it does at all.

Ok, ok, I’m being a little hyperbolic. There are some situations where it does do what you might expect.

Suppose your resource looks something like this:

resource “myResource” {
  certificate: data.certificate.private
}

i.e. I’m passing a private TLS certificate to my resource, named myResource, based on the parameter I’ve configured.

In most cases, if I updated one of those parameters in future, my cloud resource would be mutated in-place to reflect the new inputs. However, some parameters are so fundamental that changing them in-place does not make sense. Instead, changes to these parameters “force destruction”, which is a dramatic way of saying that the IaC apply process will destroy and then re-create the resource based on the new parameters.

In these sort of “force destruction” cases, delete protection will do exactly what you expect, and complain at you rather than allowing you to destroy your beloved resources.

Fantastic!

But what if you accidentally deleted the resource definition from your IaC altogether? Well then, err – delete protection won’t do anything to help, and your resource will be deleted.

That’s bad, but maybe understandable, I guess? But at least I just need to not delete resources from my code. But what if I were to just accidentally re-name the resource (“myResource” -> “ourResource”)?

Sorry, delete protection says this is fine, and your resource is gone. But, good news: you’ve got a fresh new resource instead. Hope you weren’t storing any data in that old resource!

What the hell is going on? Why isn’t delete protection protecting against these deletes???

Well, it all comes down to how the actual IaC code interacts with the statefile:

As I described earlier, if a resource is in-state then the IaC apply process tries to reconcile that state with the current code. But if the current code doesn’t have a resource which exists in-state, then the reconciliation action is to delete the resource.

Accidentally delete the resource definition? That’s a removal.

Rename a resource? Well, that new resource doesn’t match up to anything in-state, so that’s a removal (and a recreate).

So why doesn’t delete protection protect against those deletes? Because it is not stored in the remote state, only in the code.

So, if you delete the code definition of a resource – delete protection and all – then the IaC apply has no way to know that it should be delete protecting that resource.

If this sounds like a fairly fundamental issue with the technology, I agree. If you think this sounds like a really easy fix – just store the delete protection in the remote state – I also agree.

Hashicorp (the maintainers of terraform), don’t agree though. But you can certainly let them know on the long-standing github issue covering this behaviour.

In the meantime, (some) cloud services are having to take up the slack: exposing resource-level delete protection which behaves the way you’d actually expect (i.e. blocking changes which would remove a marked resource). So given the option, always prefer to use delete protection baked into resource definitions, not the terraform lifecycle version.

This is one final bit of provider inconsistency, of course. Hope you can keep that all in your head!

Conclusion

Hopefully this article has highlighted some useful (and entertaining) gremlins within the IaC ecosystem, maybe even illuminating the reason behind some issues you might have encountered personally in the past as I exercise my own personal bugbears.

However, these are still not really flaws with the conceptual IaC model, nor even issues with terraform as a particular technology choice. But do represent some of the nastiest, most counter-intuitive, or most annoying edge cases you may encounter when applying IaC at scale.

I do not want you to walk away from this article thinking you should reconsider adopting IaC in your project, but you should think twice about how broadly you intend on applying it – especially if you mean to do so as one infrastructure megaproject. And if this inspires you to go and review and refactor your IaC to exorcise some latent bugs, even better.

Next time we’ll wrap this whole journey up with some further gotchas to watch out for, and recommendations of best practices when using IaC as the backbone of your infrastructure.

Credits:

Phoenix image designed by Wannapik