Some experiments with vcluster on APPUiO Cloud
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Tobias Brunner 339439529f add some control api pocs 2 years ago
control-api add some control api pocs 2 years ago
host-cluster update ca 2 years ago
in-cluster rename file to not apply 2 years ago
.gitignore add kubeconfig to git to have a good example 2 years ago
README.adoc its working 2 years ago
kubeconfig.yaml add kubeconfig to git to have a good example 2 years ago
values.yaml add username prefix and use sub as claim 2 years ago

README.adoc

vcluster on OpenShift

See also vcluster docs.

Notes

  • Issues with vcluster on OpenShift

    • Images do not run properly as non-root - workaround with multiple emptyDir mounts.

    • Default K3s CoreDNS deployment doesnt work on OpenShift Host Cluster → Custom deployment in in-cluster/coredns.yaml.

    • See vcluster on OpenShift 4 #171

  • Re-encrypt route only works with OIDC auth, not with certificate auth

Installation

  1. Create OpenShift Project

  2. Create and configure vcluster

    vcluster create vcluster-1 -n tobru-vcluster-poc -f values.yaml
    vcluster connect vcluster-1 --namespace tobru-vcluster-poc
    kubectl --kubeconfig=$(pwd)/kubeconfig.yaml apply -f in-cluster/
  3. Install Route

    CA=$(kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config view -o jsonpath='{.clusters[].cluster.certificate-authority-data}' --flatten | base64 -d)
    yq -i e ".spec.tls.destinationCACertificate = \"$CA\"" host-cluster/route.yaml
    oc apply -f host-cluster/route.yaml
  4. Configure kubeconfig for vcluster access

    kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config unset clusters.local.certificate-authority-data
    kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config set clusters.local.server https://vcluster-poc.apps.cloudscale-lpg-1.appuio.cloud
    kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config set-credentials oidc \
      --exec-api-version=client.authentication.k8s.io/v1beta1 \
      --exec-command=kubectl \
      --exec-arg=oidc-login \
      --exec-arg=get-token \
      --exec-arg=--oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev \
      --exec-arg=--oidc-client-id=tobru-vcluster-test \
      --exec-arg=--oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0
    kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config set-context Default --user=oidc

🚀

Background information: OIDC Authentication

vcluster config

vcluster:
  baseArgs:
[...]
    - --kube-controller-manager-arg=controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle
    - --kube-apiserver-arg=oidc-client-id=tobru-vcluster-test
    - --kube-apiserver-arg=oidc-groups-claim=groups
    - --kube-apiserver-arg=oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev
    - --kube-apiserver-arg=oidc-username-claim=email

kubectl plugin

# kubectl oidc-login setup --oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev --oidc-client-id=tobru-vcluster-test --oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0
authentication in progress...

## 2. Verify authentication

You got a token with the following claims:

{
  "exp": 000,
  "iat": 000,
  "auth_time": 000,
  "jti": "XXX",
  "iss": "https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev",
  "aud": "tobru-vcluster-test",
  "sub": "UUID",
  "typ": "ID",
  "azp": "tobru-vcluster-test",
  "nonce": "XXX",
  "session_state": "XXX",
  "at_hash": "XXX",
  "acr": "1",
  "sid": "XXX",
  "email_verified": true,
  "name": "Tobias Brunner",
  "groups": [
    "admin"
  ],
  "preferred_username": "tobias.brunner",
  "given_name": "Tobias",
  "family_name": "Brunner",
  "email": "tobias.brunner@vshn.net"
}

## 3. Bind a cluster role

Run the following command:

        kubectl create clusterrolebinding oidc-cluster-admin --clusterrole=cluster-admin --user='XXX'

## 4. Set up the Kubernetes API server

Add the following options to the kube-apiserver:

        --oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev
        --oidc-client-id=tobru-vcluster-test

## 5. Set up the kubeconfig

Run the following command:

        kubectl config set-credentials oidc \
          --exec-api-version=client.authentication.k8s.io/v1beta1 \
          --exec-command=kubectl \
          --exec-arg=oidc-login \
          --exec-arg=get-token \
          --exec-arg=--oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev \
          --exec-arg=--oidc-client-id=tobru-vcluster-test \
          --exec-arg=--oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0

## 6. Verify cluster access

Make sure you can access the Kubernetes cluster.

        kubectl --user=oidc get nodes

You can switch the default context to oidc.

        kubectl config set-context --current --user=oidc

You can share the kubeconfig to your team members for on-boarding.

# kubectl --kubeconfig ./kubeconfig.yaml config set-credentials oidc \
  --exec-api-version=client.authentication.k8s.io/v1beta1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg=--oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev \
  --exec-arg=--oidc-client-id=tobru-vcluster-test \
  --exec-arg=--oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0
# kubectl --kubeconfig ./kubeconfig.yaml --user=oidc get po -A

Keycloak client