162 lines
5.2 KiB
Plaintext
162 lines
5.2 KiB
Plaintext
= vcluster on OpenShift
|
|
|
|
See also https://www.vcluster.com/docs/[vcluster docs].
|
|
|
|
== Notes
|
|
|
|
* Issues with vcluster on OpenShift
|
|
** Images do not run properly as non-root - workaround with multiple `emptyDir` mounts.
|
|
** Default K3s CoreDNS deployment doesn't work on OpenShift Host Cluster -> Custom deployment in `in-cluster/coredns.yaml`.
|
|
** See https://github.com/loft-sh/vcluster/issues/171[vcluster on OpenShift 4 #171]
|
|
* Re-encrypt route only works with OIDC auth, not with certificate auth
|
|
|
|
== Installation
|
|
|
|
. Create OpenShift Project
|
|
|
|
. Create and configure vcluster
|
|
+
|
|
----
|
|
vcluster create vcluster-1 -n tobru-vcluster-poc -f values.yaml
|
|
vcluster connect vcluster-1 --namespace tobru-vcluster-poc
|
|
kubectl --kubeconfig=$(pwd)/kubeconfig.yaml apply -f in-cluster/
|
|
----
|
|
|
|
. Install Route
|
|
+
|
|
----
|
|
CA=$(kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config view -o jsonpath='{.clusters[].cluster.certificate-authority-data}' --flatten | base64 -d)
|
|
yq -i e ".spec.tls.destinationCACertificate = \"$CA\"" host-cluster/route.yaml
|
|
oc apply -f host-cluster/route.yaml
|
|
----
|
|
|
|
. Configure kubeconfig for vcluster access
|
|
+
|
|
----
|
|
kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config unset clusters.local.certificate-authority-data
|
|
kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config set clusters.local.server https://vcluster-poc.apps.cloudscale-lpg-1.appuio.cloud
|
|
kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config set-credentials oidc \
|
|
--exec-api-version=client.authentication.k8s.io/v1beta1 \
|
|
--exec-command=kubectl \
|
|
--exec-arg=oidc-login \
|
|
--exec-arg=get-token \
|
|
--exec-arg=--oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev \
|
|
--exec-arg=--oidc-client-id=tobru-vcluster-test \
|
|
--exec-arg=--oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0
|
|
kubectl --kubeconfig=$(pwd)/kubeconfig.yaml config set-context Default --user=oidc
|
|
----
|
|
|
|
🚀
|
|
|
|
== Background information: OIDC Authentication
|
|
|
|
* Blog: https://aaron-pejakovic.medium.com/kubernetes-authenticating-to-your-cluster-using-keycloak-eba81710f49b
|
|
* K8s Docs: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
|
|
* kubectl plugin: https://github.com/int128/kubelogin
|
|
|
|
=== vcluster config
|
|
|
|
```
|
|
vcluster:
|
|
baseArgs:
|
|
[...]
|
|
- --kube-controller-manager-arg=controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle
|
|
- --kube-apiserver-arg=oidc-client-id=tobru-vcluster-test
|
|
- --kube-apiserver-arg=oidc-groups-claim=groups
|
|
- --kube-apiserver-arg=oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev
|
|
- --kube-apiserver-arg=oidc-username-claim=email
|
|
```
|
|
|
|
=== kubectl plugin
|
|
|
|
```
|
|
# kubectl oidc-login setup --oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev --oidc-client-id=tobru-vcluster-test --oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0
|
|
authentication in progress...
|
|
|
|
## 2. Verify authentication
|
|
|
|
You got a token with the following claims:
|
|
|
|
{
|
|
"exp": 000,
|
|
"iat": 000,
|
|
"auth_time": 000,
|
|
"jti": "XXX",
|
|
"iss": "https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev",
|
|
"aud": "tobru-vcluster-test",
|
|
"sub": "UUID",
|
|
"typ": "ID",
|
|
"azp": "tobru-vcluster-test",
|
|
"nonce": "XXX",
|
|
"session_state": "XXX",
|
|
"at_hash": "XXX",
|
|
"acr": "1",
|
|
"sid": "XXX",
|
|
"email_verified": true,
|
|
"name": "Tobias Brunner",
|
|
"groups": [
|
|
"admin"
|
|
],
|
|
"preferred_username": "tobias.brunner",
|
|
"given_name": "Tobias",
|
|
"family_name": "Brunner",
|
|
"email": "tobias.brunner@vshn.net"
|
|
}
|
|
|
|
## 3. Bind a cluster role
|
|
|
|
Run the following command:
|
|
|
|
kubectl create clusterrolebinding oidc-cluster-admin --clusterrole=cluster-admin --user='XXX'
|
|
|
|
## 4. Set up the Kubernetes API server
|
|
|
|
Add the following options to the kube-apiserver:
|
|
|
|
--oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev
|
|
--oidc-client-id=tobru-vcluster-test
|
|
|
|
## 5. Set up the kubeconfig
|
|
|
|
Run the following command:
|
|
|
|
kubectl config set-credentials oidc \
|
|
--exec-api-version=client.authentication.k8s.io/v1beta1 \
|
|
--exec-command=kubectl \
|
|
--exec-arg=oidc-login \
|
|
--exec-arg=get-token \
|
|
--exec-arg=--oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev \
|
|
--exec-arg=--oidc-client-id=tobru-vcluster-test \
|
|
--exec-arg=--oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0
|
|
|
|
## 6. Verify cluster access
|
|
|
|
Make sure you can access the Kubernetes cluster.
|
|
|
|
kubectl --user=oidc get nodes
|
|
|
|
You can switch the default context to oidc.
|
|
|
|
kubectl config set-context --current --user=oidc
|
|
|
|
You can share the kubeconfig to your team members for on-boarding.
|
|
|
|
# kubectl --kubeconfig ./kubeconfig.yaml config set-credentials oidc \
|
|
--exec-api-version=client.authentication.k8s.io/v1beta1 \
|
|
--exec-command=kubectl \
|
|
--exec-arg=oidc-login \
|
|
--exec-arg=get-token \
|
|
--exec-arg=--oidc-issuer-url=https://id.dev.appuio.cloud/auth/realms/appuio-cloud-dev \
|
|
--exec-arg=--oidc-client-id=tobru-vcluster-test \
|
|
--exec-arg=--oidc-client-secret=63410f24-0721-447b-a290-4b0169c414e0
|
|
# kubectl --kubeconfig ./kubeconfig.yaml --user=oidc get po -A
|
|
```
|
|
|
|
== Keycloak client
|
|
|
|
* "Access Type" confidential
|
|
* Valid Redirect URIs:
|
|
http://localhost:18000
|
|
http://localhost:8000
|
|
(they are for the kubelogin plugin)
|