Terraform Circular State

I am wanting to tackle an issue with best practice in AWS - IAM policies and resource policies.

The immediate reaction would be to say:

• All IAM policies should be scoped to the actions and resources that they interact with
• All resource policies should be scoped to the principals that interact with them

In a distributibuted micro-service architecture, where each application is deployed indepdently, this can become a bottleneck.

Generally, AWS resources are tied to micro-services - a service exists, has a datastore and all requests from that datastore are made to the micro-service. However, services such as lambda functions are a micro-service and are called “externally” (to the system) and lambda functions are govourned by resource policies.

So what might this look like:

# caller:
resource aws_iam_role caller {
  name = "caller-role"
}

data aws_iam_policy_document call_lambda {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["lambda:InvokeFunction"]

    resources = [
      "arn:aws:lambda:us-east-1:123456789012:function:lambda"
    ]
  }
}

## Attach policy to caller role
...


# lambda:
resource aws_lambda_function example {
  function_name = "example"
}

resource aws_lambda_permission cross_account {
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.example.function_name
  principal     = "arn:aws:iam::444455556666:role/caller-role"

  source_account         = "444455556666"
  function_url_auth_type = "AWS_IAM"
}

Of course, this isn’t nice - we have a hard-coded ARN. So, naturally, we’d:

...
    resources = [
      aws_lambda_function.example.arn
    ]
...

# and

...
  principal     = aws_iam_role.caller.arn
...

We’re somewhat now fortunate to be able to do this since, historically, the AWS provider had policies as attributes of resources, meaning the statements must have been fully-known at plan time in order to avoid circular dependencies. However, with these (not so) new resources for policies, such as aws_s3_bucket_policy etc. we can create the resources up-front and then apply policy resources.

HOWEVER

This works fine in a single repository, but let’s consider a large-scale micro-service architecture - Terraform is pushed to the applications and is distributed. And what does this mean?… it means our ability to reference resources defined in other projects requires another mechanism.

Terraform provides this out-of-the-box with remote state. However, here lies the problem, if I have two repoisities referencing each other this has two major issues:

1. Circular dependencies

The application depend on each other at deploy-time. Even though, architecturally there is a one-way dependency - the principal using the role invokes the lambda (and not the other way around), there is a hard dependency at deploy time. If we take any DR scenario, or even just the first release of these applications, both must have been deployed for the other to function.

This comes in two fold - the remote state must exist, otherwise Terraform will fail. The other is that the resources (lambda and IAM role) must exist for each to deploy - AWS may not always care if you reference an ARN that doesn’t exist in a policy, but Terraform isn’t going to make one up for you!

2. Fragile Interface

I’ve not once seen a publish spec where someone has outlined and versioned the outputs from their Terraform. Yet Terraform remote state is used. This means if the owner of either of these decides to change the name or format of the outputs, it will break the deployment of other. Of course, there is a possibility that since we are breaking deploy-time dependencies, that if someone messes up, it can’t even be reverted! We inherintly become trapped in the current deployed state in order to deploy, which is a nightmare situation to accidently get trapped in (and most likely even if it was intentional).

Defining the problem

So, if we want to solve this, what issues do we need to solve:

1. Interface definitions - a project should be able to define it’s outputs, the name and format. This should also define the case in all senarios - what if it’s not yet deployed.
2. Removal of circular dependencies - applications should be deployable at any time. It should be possible to define what happens if a referenced resource does not exist.
3. Have a defined and safe DR-ready approach to initial rollout

Architectural fixes

Before going wild, let’s see if there are some architectural changes that could fix this:

1. Principals up-front

If we were to create all principals up-front (e.g. all IAM roles), then this could avoid the situation.

1. Principals created (centrally, some dynamic/automated to avoid a big fat repo that everyone needs to modify)
2. Applications deploy, freely adding policies to their pre-existing role and using it with their applications
3. Resource policies reference the pre-created roles in resource policies.

Some of the downsides here are:

1. This up-front definition doesn’t come free - either increasing friction when creating applications OR some development of a centralised authority.
2. This won’t work for everything - IAM roles are only used in some cases, we still don’t know the ARNs of resources that don’t use them, e.g. data sources and lambda functions etc.

2. Policies afterwards

What if everything remained the same, but policies and permissions were created aftwards?

1. This would support all resources types

However:

1. Again, another big central repository that is difficult to maintain
2. This repository, to avoid hard-coding, would need remote state definitions for all applications, this makes the “lack of spec” problem even worse and becomes incredibly brittle - if a single application changes it’s outputs OR isn’t yet deployed, the whole thing becomes unusable
3. Since we deploy policies after an application, introducing new application versions that require policy changes become problematic - the policy would likely be deployed a good amount of time after the application.

Of course, some of these issues could be fixed if the policies were defined in more granular “policy” repos - however, we’re then talking about a massive increase in number of repositories and pipelines (let’s say double the number of applications repos).

3. One-sided policy definition locality

Define a standard in which the policies are defined in a location in one “direction” of the request, e.g. the consumer or the producer.

Let’s say, all policies are stored in the role with the application that is interacting with the remote resource.

This could work - but only with resources that support per-permission resources, e.g. S3 buckets have a single policy, which is defined once. Of course, some awful data-source -> resource (i.e. use data-source to get current policy, inject the required statements and use a resource to set it again), but let’s not go there.

4. One-way policies

Of course, this could be solved by saying: Permissions only exist on the resource (all principals use wild-card resources) OR permisisons only exist on the principal (and resource policies are not used, or at least set to the :root of the required accounts). Whilst this could work, one failure could break the whole thing - one application deployed with a wildcard policy (where resource policies aren’t used) would be allowed access to everything, and visa-versa for the other. Whilst it would be possible to use SCPs and RCPs to perhaps solve this, it becomes a layered security (and not layered through multiple concrete security defintions, but one where it only works when everything is perfect).

5. Use Terraform data sources

Of course, we could use Terrform data sources to lookup resources. However, this has two issues:

• It compounds on the issue of needing to know inner details of different systems for them to be able to interact
• To work cross account, the deployment pipeline would need access to both accounts

Let’s get creative

Instead, let’s think outside the box - let’s throw away everything we know about referencing reosurce attributes in Terraform and certainly it’s remote state.

Let’s imagine this:

resource aws_lambda_function this {
  function_name = "test"
}

resource arnregistry_resource lambda {
  type = "lambda:function"
  name = aws_lambda_function.this.name
  arn  = aws_lambda_function.this.arn
}

and:

resource aws_iam_role caller {
  name = "caller-role"
}

resource arnregistry_resource lambda {
  type = "iam:role"  # Yes, yes, we get this from the ARN, I know!
  name = aws_iam_role.caller.name
  arn  = aws_iam_role.caller.arn
}

The what could the permissions look like:

data arnregistry_resource caller {
  name = "caller-role"
}

resource aws_lambda_permission cross_account {
  count = data.arnregistry_resource.caller.exists ? 1 : 0

  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.example.function_name
  principal     = data.arnregistry_resource.caller.arn

  source_account         = data.arnregistry_resource.caller.aws_account
  function_url_auth_type = "AWS_IAM"
}


data arnregistry_resource lambda {
  name = "test"
}

data aws_iam_policy_document call_lambda {
  count = data.arnregistry_resource.lambda.exists ? 1 : 0

  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["lambda:InvokeFunction"]

    resources = [
      # Of course, if we had more, we could create a local list,
      # "compact" it and then perform a count based on the length of the list
      data.arnregistry_resource.lambda.arn
    ]
  }
}

Let’s consider how this changes things:

1. We have a fail-safe, the data call will never fail if the remote state doesn’t exist, but provides null values and tells you whether it exists.
2. We have a clean interface, we provide the name of the resource (and/or the type) and it provides the ARN

We can then expand this - we can incorporate additional details, such as environment (depending on the deployment pattern of the registry), determine the account and all sorts of other things.

On top, we can add some “nice to haves”, we’re now silently dropping policies another resource doesn’t exist, but we can show warnings during Terraform, so if something doesn’t exist, we can still let the user know.

Of course, some of this requires some slightly more complex handling, such as ensuring policies don’t end up with an empty list of resources, as this will fail to apply in some cases - so ensuring that in failure-cases, in which situation to ensure a blocking policy is in place and which other resources may need to be toggled. A lot of this becomes quite easy and Terraform modules become your friend.

Other solutions

Of course, there are other solutions, such as using Hashicorp Vault or even AWS parameter store etc, but…

1. AWS functionality won’t work cross-account out-of-the-box,
2. These introduce additional authentication complexity (though the above solution doesn’t suggest a way to solve this); and
3. neither enforce an interface, though this could be achieved with a simple Terraform module.

The slight nuance is ensuring that the Terraform provider support listing - data source have a tendency to fail-fast, but “list” data source can be used to toggle a count of a normal data source to ensure this doesn’t happen - it just depends if the provider has one!