Terraform:将Cloudwatch日志订阅交付配置为lambda?

问题描述

我需要将cloudwatch日志发送到日志分析服务.

I need to ship my cloudwatch logs to a log analysis service.

我已在此处阅读了这些文章和此处，并使其手动运行，不用担心.

I've followed along with these articles here and here and got it working by hand, no worries.

现在，我正在尝试使用Terraform(角色/策略，安全组，cloudwatch日志组，lambda以及从日志组触发lambda)来自动化所有这些操作.

Now I'm trying to automate all this with Terraform (roles/policies, security groups, cloudwatch log group, lambda, and triggering the lambda from the log group).

但是我不知道如何使用TF来配置AWS以从cloudwatch日志中触发lambda.

But I can't figure out how to use TF to configure AWS to trigger the lambda from the cloudwatch logs.

通过执行以下操作(在Lambda Web控制台UI中)，我可以手工将两个TF资源链接在一起:

I can link the two TF resources together by hand by doing the following (in the Lambda web console UI):

• 进入lambda函数的触发器"部分
• 点击添加触发器"
• 从触发器类型列表中选择"cloudwatch日志"
• 选择我要触发lambda的日志组
• 输入过滤器名称
• 将过滤器模式留空(这意味着在所有日志流上触发)
• 确保已选择启用触发器"
• 点击提交按钮

完成后，lambda将显示在cloudwatch日志控制台的订阅"列中-显示为"Lambda(cloudwatch-sumologic-lambda)".

Once that's done, the lambda shows up on the cloudwatch logs console in the subscriptions column - displays as "Lambda (cloudwatch-sumologic-lambda)".

我尝试使用以下TF资源创建订阅:

I tried to create the subscription with the following TF resource:

resource "aws_cloudwatch_log_subscription_filter" "cloudwatch-sumologic-lambda-subscription" {
  name            = "cloudwatch-sumologic-lambda-subscription"
  role_arn        = "${aws_iam_role.jordi-waf-cloudwatch-lambda-role.arn}"
  log_group_name  = "${aws_cloudwatch_log_group.jordi-waf-int-app-loggroup.name}"
  filter_pattern  = "logtype test"
  destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}

但是失败了:

我找到了有关为排定的事件设置类似操作的答案，但这似乎并不等同于上述控制台操作(控制台UI方法不会创建事件/规则)我可以看到).

I found this answer about setting up a similar thing for a scheduled event, but that doesn't seem to be equivalent to what the console actions I described above do (the console UI method doesn't create an event/rule that I can see).

请问有人可以告诉我我做错了什么吗?

Can someone give me a pointer on what I'm doing wrong please?

推荐答案

我没有正确定义aws_cloudwatch_log_subscription_filter资源-在这种情况下，您不应提供role_arn参数.

I had the aws_cloudwatch_log_subscription_filter resource defined incorrectly - you should not provide the role_arn argument in this situation.

您还需要添加aws_lambda_permission资源(在过滤器上定义了depends_on关系，否则TF可能会以错误的顺序执行此操作).

You also need to add an aws_lambda_permission resource (with a depends_on relationship defined on the filter or TF may do it in the wrong order).

请注意，AWS lambda控制台UI会为您隐式添加lambda权限，因此请注意，如果您之前在控制台UI中执行了相同的操作，则aws_cloudwatch_log_subscription_filter将在没有权限资源的情况下工作.

Note that the AWS lambda console UI adds the lambda permission for you invisibly, so beware that the aws_cloudwatch_log_subscription_filter will work without the permission resource if you happen to have done the same action before in the console UI.

必要的TF配置如下所示(最后两个资源是用于配置实际cloudwatch->lambda触发器的相关资源):

The necessary TF config looks like this (the last two resources are the relevant ones for configuring the actual cloudwatch->lambda trigger):

// intended for application logs (access logs, modsec, etc.)
resource "aws_cloudwatch_log_group" "test-app-loggroup" {
  name              = "test-app"
  retention_in_days = 90
}

resource "aws_security_group" "cloudwatch-sumologic-lambda-sg" {
  name = "cloudwatch-sumologic-lambda-sg"

  tags {
    Name = "cloudwatch-sumologic-lambda-sg"
  }

  description = "Security group for lambda to move logs from CWL to SumoLogic"
  vpc_id      = "${aws_vpc.dev-vpc.id}"
}

resource "aws_security_group_rule" "https-egress-cloudwatch-sumologic-to-internet" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  security_group_id = "${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"
  cidr_blocks       = ["0.0.0.0/0"]
}

resource "aws_iam_role" "test-cloudwatch-lambda-role" {
  name = "test-cloudwatch-lambda-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "test-cloudwatch-lambda-policy" {
  name = "test-cloudwatch-lambda-policy"
  role = "${aws_iam_role.test-cloudwatch-lambda-role.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole1",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface"
      ],
      "Resource": "*"
    },
    {
      "Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole2",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "arn:aws:ec2:ap-southeast-2:${var.dev_vpc_account_id}:network-interface/*"
    },

    {
      "Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole1",
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:*"
    },
    {
      "Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole2",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
    "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:log-group:/aws/lambda/*"
      ]
    },

    {
      "Sid": "CopiedFromTemplateAWSLambdaAMIExecutionRole",
      "Effect": "Allow",
      "Action": [
         "ec2:DescribeImages"
      ],
      "Resource": "*"
    }


  ]
}
EOF
}

resource "aws_lambda_function" "cloudwatch-sumologic-lambda" {
  function_name    = "cloudwatch-sumologic-lambda"
  filename         = "${var.lambda_dir}/cloudwatchSumologicLambda.zip"
  source_code_hash = "${base64sha256(file("${var.lambda_dir}/cloudwatchSumologicLambda.zip"))}"
  handler          = "cloudwatchSumologic.handler"

  role        = "${aws_iam_role.test-cloudwatch-lambda-role.arn}"
  memory_size = "128"
  runtime     = "nodejs4.3"

  // set low because I'm concerned about cost-blowout in the case of mis-configuration
  timeout = "15"

  vpc_config = {
    subnet_ids         = ["${aws_subnet.dev-private-subnet.id}"]
    security_group_ids = ["${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"]
  }
}

resource "aws_lambda_permission" "test-app-allow-cloudwatch" {
  statement_id  = "test-app-allow-cloudwatch"
  action        = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
  principal     = "logs.ap-southeast-2.amazonaws.com"
  source_arn    = "${aws_cloudwatch_log_group.test-app-loggroup.arn}"
}

resource "aws_cloudwatch_log_subscription_filter" "test-app-cloudwatch-sumologic-lambda-subscription" {
  depends_on      = ["aws_lambda_permission.test-app-allow-cloudwatch"]
  name            = "cloudwatch-sumologic-lambda-subscription"
  log_group_name  = "${aws_cloudwatch_log_group.test-app-loggroup.name}"
  filter_pattern  = ""
  destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}

编辑:请注意，上面的TF代码是几年前使用版本0.11.x编写的-仍然可以使用，但是可能有更好的处理方法.具体来说，除非需要，否则不要使用此类内联策略，请使用 aws_iam_policy_document 相反-随着时间的推移，它们只是更易于维护.

EDIT: Please note that the above TF code was written years ago, using version 0.11.x - it should still work but there may be better ways of doing things. Specifically, don't use an inline policy like this unless needed, use an aws_iam_policy_document instead - they're just way easier to maintain over time.

这篇关于Terraform:将Cloudwatch日志订阅交付配置为lambda?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！