A Getting Started Guide
Cluster API is beautiful Kubernetes project which allows provisioning, upgrading, and operating multiple Kubernetes clusters using declarative APIs.
Kind is another beautiful project to build Kubernetes cluster on docker.
Both of the above projects are blessings for someone like me who wants to build cluster dynamically for development purpose. But the journey was not smooth when I stared using using using cluster-api version 3.8. Lot of hacks need to be made to make it working.
Although cluster api new version (at the time of writing it was v0.3.14) improves it a lot, the quick-start guide provided does not provide clarity on the dependency and thus I thought to write an article to build it on Docker without spending hours to find the issue.
Let’s understand the concept before getting stated. We could also just use kind to create multiple Kubernetes clusters, but it will not provide the functionality that cluster-api provides as well as real world scenario where it is not easy to build a Kubernetes Cluster.
Kind allows to create a Kubernetes cluster, which we provision as Management Cluster using the Cluster API by using clusterctl.
We also create multiple Kubernetes clusters configuration using cluserctl to be built by same Kubernetes cluster management tool — kubectl.
This is very important, since utilizing different version of above tool, may not provide a working cluster-api management cluster. I have used the following to build my environment.
- Ubuntu VM (20.04) with docker installed.
- Kind version 0.10.0
- Clusterctl version v.0.3.14
- Kubectl version v1.20.0
Install and Configure
1.Install kind and verify
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.10.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
2. Install kubectl and verify
curl -LO https://dl.k8s.io/release/v1.20.0/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
3. Install clusterctl and verify
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.3.14/clusterctl-linux-amd64 -o clusterctl
chmod +x ./clusterctl
sudo mv ./clusterctl /usr/local/bin/clusterctl
3. Install and configure Management Cluster
Use kind to create a Kubernetes cluster . Dot not forget to set the following environment variable, otherwise the cluster will be created in docker kind network which later will fail to communicate with cluster-api managed cluster which will be created in bridge network.
cat > kind-cluster-api.yaml <<EOF
- role: control-plane
- hostPath: /var/run/docker.sock
Here I have pinned the Kubernetes version 1.18.15 with hash tag to make sure it works every time. Please note here that although getting started guide mentions about v.1.18.16 for building workload cluster it fails, since it is not available.
Now create the cluster by using the config above -
kind create cluster --config ./kind-cluster-api.yaml --name clusterapi
Verify the above cluster is ready using — kubectl get nodes. The above installation modifies kubeconfig to use the cluster as default context.
To initialize the above cluster to be used as Management Cluster use the command below. It will install cluster-api, bootstrap, control plane and infrastructure provider (docker) CRDs.
clusterctl init --infrastructure docker
If the above is successful, you will find the following pods running without any issue on the management cluster — kubectl get po -A
4. Create a Workload Cluster
Now management cluster is ready to build workload clusters. Use the command like below to build Kubernetes v1.18.15 cluster.
clusterctl config cluster capi-k18 --flavor development \
--kubernetes-version v1.18.15 \
> capi-k18.yamlkubectl apply -f capi-k18.yaml
It will start building the workload cluster, it will be ready when all of the components of the cluster will have True in READY column when using command — clusterctl describe cluster capi-k18
At this stage control plane will be initialized but not ready since CNI plugin is not installed.
Let’s get kubernetes cluster config file using the command below -
clusterctl get kubeconfig capi-k18 > capi-k18.kubeconfig
Deploy the Calico CNI in the above workload cluster using the above kubeconfig file. Any other CNI can also be used.
kubectl --kubeconfig=./capi-k18.kubeconfig \
apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml
Now verify that both the Control Plane, Worker Nodes and Pods are ready.
kubectl get kubeadmcontrolplane --all-namespaces
kubectl --kubeconfig=./capi-k18.kubeconfig get nodes -A
kubectl --insecure-skip-tls-verify --kubeconfig=./capi-k18.kubeconfig get pods -A
I used insecure-skip-tls-verify option above since without that I get the following error -
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kubernetes”)
I put a alias in my .bashrc to avoid typing it every time,
alias kubectl='kubectl --insecure-skip-tls-verify'
It is also possible to just export the kubeconfig as environment variable, instead of running with every kubectl invokation.
During the installation and configuration process, I have encountered issues which did troubleshooting by looking at management cluster pod like the following -
a) kubectl -n capd-system logs capd-controller-manager-d459f8876-xg9zm manager
b) kubectl -n capd-system describe pod capd-controller-manager-d459f8876-l6rh2
If you get an issue like unknow flag: — metrics-addr it could be due to use of an old version.
The issue, MountVolume.SetUp failed for volume “cert” : secret “capi-webhook-service-cert” not found is due to use of old clusterctl version.
kubeadm init waiting could be due to workload and management cluster is different network.
You can also get error message intermittently while running kubectl — error: You must be logged in to the server (Unauthorized), just run the command again in that case.
Some other useful commands.
kubectl get cluster --all-namespaces
kubectl get machines
To delete workload cluster run below command. Let it complete. do not terminate the process. Otherwise it will be tedious manual process to cleanup.
kubectl delete cluster capi-k18
To delete management cluster use
kind delete cluster