Instantiating a Pulumi Kubernetes provider inside a Pod

Usually, we call the Kubernetes API from within a Pod using kubectl or by exposing the API via kubectl proxy and making HTTP requests to it.

The Pulumi Kubernetes provider requires a kubeconfig file, which is either loaded from disk (details) or passed as a keyword argument. Thus, we need to generate a kubeconfig that is valid from within a Pod.

Note that the kubeconfig becomes part of the stack state. Since the service account token is tied to a particular service account and rotated regularly, we cannot simply embed the token into the kubeconfig as in this gist by Rich Adams. Instead, we have to load the token from the Pod environment when the Kubernetes provider (which, just like kubectl, uses the client-go library) requests credentials.

Here's a utility function that generates a kubeconfig using the information available in the Pod:

import os
import pathlib


def get_kubeconfig():
    service_host = os.getenv("KUBERNETES_SERVICE HOST")
    service_port = os.getenv("KUBERNETES_SERVICE_PORT")
    service_scheme = (
        "http" if service_port in ("80", "8080", "8081") else "https"
    )
    server_url = f"{service_scheme}://{service_host}:{service_port}"

    service_account_dir = pathlib.Path(
        "/var/run/secrets/kubernetes.io/serviceaccount"
    )
    ca_crt_path = service_account_dir / "ca.crt"
    token_path = service_account_dir / "token"

    with open(service_account_dir / "namespace") as f:
        namespace = f.read()

    context = "my-context"

    return f"""apiVersion: v1
kind: Config
preferences: {{}}
current-context: {context}

clusters:
- name: myCluster
  cluster:
    server: {server_url}
    certificate-authority: {ca_crt_path}

users:
- name: podServiceAccount
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: bash
      args:
      - -c
      - |
        cat <<EOF
        {{
          "kind": "ExecCredential",
          "apiVersion": "client.authentication.k8s.io/v1beta1",
          "spec": {{}},
          "status": {{
            "token": "$(cat {token_path})"
          }}
        }}
        EOF    

contexts:
- name: {context}
  context:
    cluster: myCluster
    namespace: {namespace}
    user: podServiceAccount
"""

Note that the generated kubeconfig requires bash to be available in the Pod the Pulumi program is executed in.

Using this get_kubeconfig() utility function, we can generate a kubeconfig on-the-fly when instantiating the Pulumi Kubernetes provider and/or pass it on as a stack output:

import pulumi
import pulumi_kubernetes as k8s

k8s_provider = k8s.Provider("k8s-provider", 
                            kubeconfig=get_kubeconfig())

pulumi.export("kubeconfig", get_kubeconfig())