在GKE集群上使用Terraform部署Helm工作负载

问题描述

我正在尝试使用Terraform Helm提供程序( https://www. terraform.io/docs/providers/helm/index.html )将工作负载部署到GKE集群.

I am trying to use Terraform Helm provider (https://www.terraform.io/docs/providers/helm/index.html) to deploy a workload to GKE cluster.

我或多或少遵循Google的示例- https://github.com/GoogleCloudPlatform/terraform-google-examples/blob/master/example-gke-k8s-helm/helm.tf ，但我确实想通过以下方式使用RBAC:手动创建服务帐户.

I am more or less following Google's example - https://github.com/GoogleCloudPlatform/terraform-google-examples/blob/master/example-gke-k8s-helm/helm.tf, but I do want to use RBAC by creating the service account manually.

我的helm.tf看起来像这样:

My helm.tf looks like this:

variable "helm_version" {
  default = "v2.13.1"
}

data "google_client_config" "current" {}

provider "helm" {
  tiller_image = "gcr.io/kubernetes-helm/tiller:${var.helm_version}"
  install_tiller = false # Temporary

  kubernetes {
    host                   = "${google_container_cluster.data-dome-cluster.endpoint}"
    token                  = "${data.google_client_config.current.access_token}"

    client_certificate     = "${base64decode(google_container_cluster.data-dome-cluster.master_auth.0.client_certificate)}"
    client_key             = "${base64decode(google_container_cluster.data-dome-cluster.master_auth.0.client_key)}"
    cluster_ca_certificate = "${base64decode(google_container_cluster.data-dome-cluster.master_auth.0.cluster_ca_certificate)}"
  }
}


resource "helm_release" "nginx-ingress" {
  name  = "ingress"
  chart = "stable/nginx-ingress"

  values = [<<EOF
rbac:
  create: false
controller:
  stats:
    enabled: true
  metrics:
    enabled: true
  service:
    annotations:
      cloud.google.com/load-balancer-type: "Internal"
    externalTrafficPolicy: "Local"
EOF
  ]

  depends_on = [
    "google_container_cluster.data-dome-cluster",
  ]
}

我遇到以下错误:

Error: Error applying plan:

1 error(s) occurred:

* module.data-dome-cluster.helm_release.nginx-ingress: 1 error(s) occurred:

* helm_release.nginx-ingress: error creating tunnel: "pods is forbidden: User \"client\" cannot list pods in the namespace \"kube-system\""

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

这是在我手动创建Helm RBAC并安装了Tiller之后发生的.

This happens after I manually created Helm RBAC and installed Tiller.

在安装Tiller时，我还尝试将"install_tiller = true"设置为完全相同的错误

I also tried to set "install_tiller=true" before with exactly the same error when Tiller was installed

"kubectl得到豆荚"的工作没有任何问题.

"kubectl get pods" works without any problems.

此用户客户端"是什么?为什么禁止其访问群集?

What is this user "client" and why it is forbidden from accessing the cluster?

谢谢

推荐答案

为服务帐户和群集角色绑定创建资源对我明确地起作用:

Creating resources for the service account and cluster role binding explicitly works for me:

resource "kubernetes_service_account" "helm_account" {
  depends_on = [
    "google_container_cluster.data-dome-cluster",
  ]
  metadata {
    name      = "${var.helm_account_name}"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "helm_role_binding" {
  metadata {
    name = "${kubernetes_service_account.helm_account.metadata.0.name}"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }
  subject {
    api_group = ""
    kind      = "ServiceAccount"
    name      = "${kubernetes_service_account.helm_account.metadata.0.name}"
    namespace = "kube-system"
  }
  provisioner "local-exec" {
    command = "sleep 15"
  }
}

provider "helm" {
  service_account = "${kubernetes_service_account.helm_account.metadata.0.name}"
  tiller_image = "gcr.io/kubernetes-helm/tiller:${var.helm_version}"
  #install_tiller = false # Temporary

  kubernetes {
    host                   = "${google_container_cluster.data-dome-cluster.endpoint}"
    token                  = "${data.google_client_config.current.access_token}"

    client_certificate     = "${base64decode(google_container_cluster.data-dome-cluster.master_auth.0.client_certificate)}"
    client_key             = "${base64decode(google_container_cluster.data-dome-cluster.master_auth.0.client_key)}"
    cluster_ca_certificate = "${base64decode(google_container_cluster.data-dome-cluster.master_auth.0.cluster_ca_certificate)}"
  }
}

这篇关于在GKE集群上使用Terraform部署Helm工作负载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！