Ruan Bekker's Blog

From a Curious mind to Posts on Github

Creating a Python Lambda Function With Terraform on AWS

In this tutorial I will explain how to deploy a AWS Lambda Function with Terraform using the Python runtime. It will include the permissions it needs to write its logs to AWS CloudWatch as well as to get information from the AWS API’s as a boilerplate for you to expand on it.

We will also use CloudWatch Events to trigger this lambda function every two hours.

Pre-Requisites

First you will need to have Terraform installed as well as authentication for Terraform to interact with your AWS account, I have written a post about it and you can follow that on “How to use the AWS Terraform Provider”.

Project Structure

The following code will be available on my github repository, but if you would like to follow along we will create everything step by step.

First create the project directory:

1

mkdir -p ~/workspace/aws-lambda-terraform

Then change into the directory:

1

cd ~/workspace/aws-lambda-terraform

First we want to create our modules directory:

1

mkdir -p modules/lambda-function

Then our environment directory:

1

mkdir -p environment/test

We will also create the directory for our function code:

1

mkdir -p modules/lambda-function/functions

And we can create the file for our python function:

1

touch modules/lambda-function/functions/demo.py

Now we will create our files inside our modules directory:

1

touch modules/lambda-function/{main,versions,outputs,variables}.tf

Then create the files inside our environments directory:

1

touch environment/test/{main,provider,output}.tf

Then in summary our project structure should look more or less like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

tree .
.
├── environment
│   └── test
│       ├── main.tf
│       ├── output.tf
│       └── provider.tf
└── modules
    └── lambda-function
        ├── functions
        │   └── demo.py
        ├── main.tf
        ├── outputs.tf
        ├── variables.tf
        └── versions.tf

5 directories, 8 files

Terraform Code

We will first start populating the modules bit, and start with modules/lambda-function/main.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

data "aws_iam_policy_document" "lambda" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

data "aws_iam_policy_document" "lambda_execution" {
  count = var.logs_enabled ? 1 : 0

  statement {
    sid     = "GetCallerIdentity"
    effect  = "Allow"

    actions = [
      "sts:GetCallerIdentity"
    ]

    resources = ["*"]

  }

  statement {
    sid     = "DescribeFunctionsInRegion"
    effect  = "Allow"

    actions = [
      "lambda:GetFunction"
    ]

    resources = ["*"]

    condition {
      test     = "StringEquals"
      variable = "aws:RequestedRegion"
      values = [var.aws_region]
    }
  }

}

resource "aws_iam_role_policy" "lambda_execution_policy" {
  count  = var.logs_enabled ? 1 : 0
  name   = "${var.project_name}-lambda-function-execution-policy"
  role   = aws_iam_role.lambda_role[count.index].id
  policy = data.aws_iam_policy_document.lambda_execution[count.index].json
}

data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/functions/demo.py"
  output_path = "${path.module}/lambda-archives/package.zip"
}

resource "aws_iam_role" "lambda_role" {
  count              = var.logs_enabled ? 1 : 0
  name               = "${var.project_name}-lambda-function-role"
  assume_role_policy = data.aws_iam_policy_document.lambda.json
}

resource "aws_lambda_function" "lambda" {
  count            = var.logs_enabled ? 1 : 0
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "${var.project_name}-lambda-function"
  role             = aws_iam_role.lambda_role[count.index].arn
  handler          = "demo.lambda_handler"
  source_code_hash = filebase64sha256(data.archive_file.lambda_zip.output_path)
  runtime          = "python3.8"
  timeout          = 30

  environment {
    variables = {
      PROJECT_NAME  = var.project_name
      FUNCTION_NAME = "${var.project_name}-lambda-function"
    }
  }

  depends_on = [
    data.archive_file.lambda_zip
  ]

}

resource "aws_cloudwatch_event_rule" "every_two_hours" {
  count               = var.logs_enabled ? 1 : 0
  name                = "${var.project_name}-every-two-hours"
  description         = "Fires every 2 hours"
  schedule_expression = "rate(2 hours)"
}

resource "aws_lambda_permission" "allow_cloudwatch" {
  count         = var.logs_enabled ? 1 : 0
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.lambda[count.index].function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.every_two_hours[count.index].arn
}

resource "aws_cloudwatch_event_target" "cloudwatch_event" {
  count     = var.logs_enabled ? 1 : 0
  rule      = aws_cloudwatch_event_rule.every_two_hours[count.index].name
  target_id = "${var.project_name}-snapshot-retention-target"
  arn       = aws_lambda_function.lambda[count.index].arn
}

// CloudWatch Logs
resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
  count     = var.logs_enabled ? 1 : 0
  name      = "/aws/lambda/${aws_lambda_function.lambda[count.index].function_name}"
  retention_in_days = 5
}

resource "aws_iam_role_policy_attachment" "lambda_exec_policy" {
  count      = var.logs_enabled ? 1 : 0
  role       = aws_iam_role.lambda_role[count.index].name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

The next one will be the modules/lambda-function/variables.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

variable "aws_region" {
  default = "eu-west-1"
  type    = string
}

variable "project_name" {
  default = "example"
  type    = string
}

variable "logs_enabled" {
  default = false
  type    = bool
}

Then define the modules output in modules/lambda-function/outputs.tf:

1
2
3

output "arn_string" {
  value = aws_lambda_function.lambda[*].arn
}

Then we define our python function code in modules/lambda-function/functions/demo.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

import os
import json
import logging
import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    client = boto3.client('lambda')
    logger.info(event)

    response = client.get_function(
        FunctionName=os.environ['FUNCTION_NAME']
    )

    logger.info(response)

    return {
        'statusCode' : 200,
        'body': response
    }

For our environment we want to specify the source as our module in environment/test/main.tf:

1
2
3
4
5

module "myfunction" {
  source       = "../../modules/lambda-function"
  project_name = "test"
  logs_enabled = true
}

Our outputs in environment/test/output.tf:

1
2
3

output "arn_string" {
  value = module.myfunction.arn_string
}

And since we are using AWS, we need to define our providers and the profile that we will use to authenticate against AWS, in my case, im using the default profile in environment/test/provider.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "4.23.0"
    }
  }
}

provider "aws" {
  region                   = "eu-west-1"
  profile                  = "default"
  shared_credentials_files = ["~/.aws/credentials"]
}

Terraform Plan

Now that we have defined our terraform code we can run:

1

terraform plan

And it should return something more or less like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.myfunction.aws_cloudwatch_event_rule.every_two_hours[0] will be created
  + resource "aws_cloudwatch_event_rule" "every_two_hours" {
      + arn                 = (known after apply)
      + description         = "Fires every 2 hours"
      + event_bus_name      = "default"
      + id                  = (known after apply)
      + is_enabled          = true
      + name                = "test-every-two-hours"
      + name_prefix         = (known after apply)
      + schedule_expression = "rate(2 hours)"
      + tags_all            = (known after apply)
    }

  # module.myfunction.aws_cloudwatch_event_target.cloudwatch_event[0] will be created
  + resource "aws_cloudwatch_event_target" "cloudwatch_event" {
      + arn            = (known after apply)
      + event_bus_name = "default"
      + id             = (known after apply)
      + rule           = "test-every-two-hours"
      + target_id      = "test-snapshot-retention-target"
    }

  # module.myfunction.aws_cloudwatch_log_group.cloudwatch_log_group[0] will be created
  + resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
      + arn               = (known after apply)
      + id                = (known after apply)
      + name              = "/aws/lambda/test-lambda-function"
      + retention_in_days = 5
      + tags_all          = (known after apply)
    }

  # module.myfunction.aws_iam_role.lambda_role[0] will be created
  + resource "aws_iam_role" "lambda_role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = "sts:GetCallerIdentity"
                      + Effect   = "Allow"
                      + Resource = "*"
                      + Sid      = "GetCallerIdentity"
                    },
                  + {
                      + Action    = "lambda:GetFunction"
                      + Condition = {
                          + StringEquals = {
                              + "aws:RequestedRegion" = "eu-west-1"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "DescribeFunctionsInRegion"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "test-lambda-function-role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)
    }

  # module.myfunction.aws_iam_role_policy.lambda_execution_policy[0] will be created
  + resource "aws_iam_role_policy" "lambda_execution_policy" {
      + id     = (known after apply)
      + name   = "test-lambda-function-execution-policy"
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = "sts:GetCallerIdentity"
                      + Effect   = "Allow"
                      + Resource = "*"
                      + Sid      = "GetCallerIdentity"
                    },
                  + {
                      + Action    = "lambda:GetFunction"
                      + Condition = {
                          + StringEquals = {
                              + "aws:RequestedRegion" = "eu-west-1"
                            }
                        }
                      + Effect    = "Allow"
                      + Resource  = "*"
                      + Sid       = "DescribeFunctionsInRegion"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + role   = (known after apply)
    }

  # module.myfunction.aws_iam_role_policy_attachment.lambda_exec_policy[0] will be created
  + resource "aws_iam_role_policy_attachment" "lambda_exec_policy" {
      + id         = (known after apply)
      + policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
      + role       = "test-lambda-function-role"
    }

  # module.myfunction.aws_lambda_function.lambda[0] will be created
  + resource "aws_lambda_function" "lambda" {
      + architectures                  = (known after apply)
      + arn                            = (known after apply)
      + filename                       = "../../modules/lambda-function/lambda-archives/package.zip"
      + function_name                  = "test-lambda-function"
      + handler                        = "demo.lambda_handler"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = false
      + qualified_arn                  = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "python3.8"
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = "MI7FD/KHgxRFh7cmPjzxg+w494pmyRGgQIr9Ls8Yups="
      + source_code_size               = (known after apply)
      + tags_all                       = (known after apply)
      + timeout                        = 30
      + version                        = (known after apply)

      + environment {
          + variables = {
              + "FUNCTION_NAME" = "test-lambda-function"
              + "PROJECT_NAME"  = "test"
            }
        }
    }

  # module.myfunction.aws_lambda_permission.allow_cloudwatch[0] will be created
  + resource "aws_lambda_permission" "allow_cloudwatch" {
      + action              = "lambda:InvokeFunction"
      + function_name       = "test-lambda-function"
      + id                  = (known after apply)
      + principal           = "events.amazonaws.com"
      + source_arn          = (known after apply)
      + statement_id        = "AllowExecutionFromCloudWatch"
      + statement_id_prefix = (known after apply)
    }

Plan: 8 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + arn_string = [
      + (known after apply),
    ]

Create Resources

If you are happy with the plan you can go ahead and run:

1

terraform apply

Which will create the resources in AWS. Upon creation we should see something like this:

1
2
3
4
5
6
7

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

arn_string = [
  "arn:aws:lambda:eu-west-1:000000000000:function:test-lambda-function",
]

Since we have our aws cli configured with a profile we can also test our lambda function:

1
2
3
4
5

$ aws --profile default lambda invoke --function-name test-lambda-function --cli-binary-format raw-in-base64-out --payload '{"name": "ruan"}' out.log
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

And the response from the invocation can be seen in the file we defined:

1
2

$ cat out.log
{"statusCode": 200, "body": {"ResponseMetadata": {"RequestId": "5171x", "HTTPStatusCode": 200, "HTTPHeaders": {"date": "Thu, 21 Dec 2023 06:34:13 GMT", "content-type": "application/json", "content-length": "3517", "connection": "keep-alive", "x-amzn-requestid": "5171x"}, "RetryAttempts": 0}, "Configuration": {"FunctionName": "test-lambda-function", "FunctionArn": "arn:aws:lambda:eu-west-1:000000000000:function:test-lambda-function", "Runtime": "python3.8", "Role": "arn:aws:iam::000000000000:role/test-lambda-function-role", "Handler": "demo.lambda_handler", "CodeSize": 401, "Description": "", "Timeout": 30, "MemorySize": 128, "LastModified": "2023-12-21T06:26:46.000+0000", "CodeSha256": "x", "Version": "$LATEST", "Environment": {"Variables": {"FUNCTION_NAME": "test-lambda-function", "PROJECT_NAME": "test"}}, "TracingConfig": {"Mode": "PassThrough"}, "RevisionId": "7faex", "State": "Active", "LastUpdateStatus": "Successful", "PackageType": "Zip", "Architectures": ["x86_64"], "EphemeralStorage": {"Size": 512}, "SnapStart": {"ApplyOn": "None", "OptimizationStatus": "Off"}, "RuntimeVersionConfig": {"RuntimeVersionArn": "arn:aws:lambda:eu-west-1::runtime:x"}}, "Code": {"RepositoryType": "S3", "Location": "https://awslambda-eu-west-1-tasks.s3.eu-west-1.amazonaws.com/snapshots/x/test-lambda-function-x?queryparameters"}}}

Updating Lambda Function Code

If we want to redeploy our function with updated code, we can change the content of functions/demo.py and then run:

1

terraform apply

Since our terraform code defined that if the source has of the function code changes, it will trigger a redeploy, and from the computed plan we can see that it will redeploy our function code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.myfunction.aws_lambda_function.lambda[0] will be updated in-place
  ~ resource "aws_lambda_function" "lambda" {
        id                             = "test-lambda-function"
      ~ last_modified                  = "2023-12-21T06:26:46.000+0000" -> (known after apply)
      ~ source_code_hash               = "8TLrm4GmTrfAxwfElmIjws1Vf9UDZ6k2w1+VEONJaCQ=" -> "RIQ62KCcjlcHh5lLCOlrkB7GioBpLY1Y5vN4UZGyN+c="
        tags                           = {}
        # (18 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

After entering “yes” we will update our function code

Discover AWS Console

If we logon to the AWS Console and head to Lambda we can inspect our function code:

image

If we manually want to trigger the function, select “Test”, then enter the “Event name” with something like “testing” then click “Test”:

image

If we follow the CloudWatch log link we can view the logs in CloudWatch:

image

Destroy Infrastructure

If you followed along and would like to destroy the created infrastructure:

1

terraform destroy

Resources

Terraform Examples

Thank You

Thanks for reading, feel free to check out my website, feel free to subscribe to my newsletter or follow me at @ruanbekker on Twitter.