Creating Highly Available clusters with kubeadm with external ETCD nodes

Overview

This article explains how to set up a highly available Kubernetes 1.16.2 cluster using kubeadm with external etcd nodes. This approach requires more infrastructure. The control plane nodes and etcd members are separated. See attached diagram below.

Highly Available Kubernetes cluster using kubeadm and external etcd nodes
Source: External etcd nodes

Prerequisites

I will cover steps for operating systems as mentioned below. Minimum server requirements:

  • Debian 9+, Ubuntu 18.04+ tested
  • 2 GB or more of RAM per node (Any less will leave little room for your Apps)
  • 2 CPUs or more on the Master nodes
  • Full network connectivity between all nodes in the cluster (public or private network is fine)
  • Root access
  • SSH access to copy Certificates access from ETCD nodes
  • Certain ports should be open on your nodes. See here for more details.
  • Swap disabled. You MUST disable swap in order for the kubelet to work properly.

Setup Environment

I have completed and tested this setup on Google Cloud but you can also work with bare metal or virtual machines. I tested with Debian 9+ and Ubuntu 18.04 Operating Systems.

NOTE: The below Hostnames and Ip addresses are just examples from my configuration

HostnameRoleIP Address
k8s-haproxyload balancer10.240.0.11
k8s-master-01master10.240.0.2
k8s-master-02master10.240.0.3
k8s-master-03master10.240.0.4
k8s-etcd-01etcd10.240.0.5
k8s-etcd-02etcd10.240.0.6
k8s-etcd-03etcd10.240.0.7
k8s-worker-01worker10.240.0.8
k8s-worker-02worker10.240.0.9
k8s-worker-03worker10.240.0.10

Install Docker CE

NOTE: Please perform these steps on all Master, Worker, Etcd nodes

Install packages to allow apt to use a repository over HTTPS

sudo su -
apt-get update && apt-get install -y apt-transport-https ca-certificates curl software-properties-common

Add Docker’s official GPG key, Add Docker apt repository

For Ubuntu 18.04+

# Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

# Add Docker apt repository
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

# Install Docker CE
apt-get update && apt-get install -y docker-ce=18.06.2~ce~3-0~ubuntu

For Debian 9+

# Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -

# Add Docker apt repository
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/debian \
  $(lsb_release -cs) \
  stable"

# Install Docker CE
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~debian

Setup the docker daemon & restart docker.

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker
systemctl daemon-reload && systemctl restart docker

Install Kubeadm, Kubectl, Kubelet

apt-get update -y
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

apt-get update -y && apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

Setup ETCD external nodes

NOTE: This step is to be performed on the 3 external Etcd nodes

Configure the kubelet to be a service manager for etcd

Since etcd was created first, we must override the service priority by creating a new unit file that has higher precedence than the kubeadm-provided kubelet unit file.

cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
#  Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
Restart=always
EOF

# Reload and restart the kubelet
systemctl daemon-reload && systemctl restart kubelet

Create configuration files for kubeadm

NOTE: This step is to be performed on One or first Etcd node

Generate one kubeadm configuration file for each host that will have an etcd member running on it using the following script. We will copy the configs across once complete.

# Update HOST0, HOST1, and HOST2 with the IP's or resolvable names of your ETCD hosts
export HOST0=10.240.0.5
export HOST1=10.240.0.6
export HOST2=10.240.0.7

Create temp directories to store config files that will end up on the other hosts

mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

Create the config files

ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("k8s-etcd-01" "k8s-etcd-02" "k8s-etcd-03")

for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta2"
kind: ClusterConfiguration
etcd:
    local:
        serverCertSANs:
        - "${HOST}"
        peerCertSANs:
        - "${HOST}"
        extraArgs:
            initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380,${NAMES[2]}=https://${ETCDHOSTS[2]}:2380
            initial-cluster-state: new
            name: ${NAME}
            listen-peer-urls: https://${HOST}:2380
            listen-client-urls: https://${HOST}:2379
            advertise-client-urls: https://${HOST}:2379
            initial-advertise-peer-urls: https://${HOST}:2380
EOF
done

Generate the Certificate Authority

NOTE: Run these commands on $HOST0 (where you generated the configuration files for kubeadm)

kubeadm init phase certs etcd-ca

This command creates these two files:

  • /etc/kubernetes/pki/etcd/ca.crt
  • /etc/kubernetes/pki/etcd/ca.key

Create certificates for each member

kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST2}/

# Cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/

# Cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml

# No need to move the certs because they are for HOST0

# Clean up certs that should not be copied off this host
find /tmp/${HOST2} -name ca.key -type f -delete
find /tmp/${HOST1} -name ca.key -type f -delete

Copy certificates and kubeadm configs

The certificates have been generated and now they must be moved to their respective hosts on k8s-etcd-02 and k8s-etcd-03

# k8s-etcd-02
USER=larry
HOST=${HOST1}
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
ssh ${USER}@${HOST}
USER@HOST $ sudo -Es
root@HOST $ chown -R root:root pki
root@HOST $ mv pki /etc/kubernetes/

# k8s-etcd-03
USER=larry
HOST=${HOST2}
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
ssh ${USER}@${HOST}
USER@HOST $ sudo -Es
root@HOST $ chown -R root:root pki
root@HOST $ mv pki /etc/kubernetes/

Ensure all expected files exist

The complete list of required files on $HOST0 is:

/tmp/${HOST0}
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
    ├── ca.crt
    ├── ca.key
    ├── healthcheck-client.crt
    ├── healthcheck-client.key
    ├── peer.crt
    ├── peer.key
    ├── server.crt
    └── server.key

On $HOST1

$HOME
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
    ├── ca.crt
    ├── healthcheck-client.crt
    ├── healthcheck-client.key
    ├── peer.crt
    ├── peer.key
    ├── server.crt
    └── server.key

On $HOST2

$HOME
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
    ├── ca.crt
    ├── healthcheck-client.crt
    ├── healthcheck-client.key
    ├── peer.crt
    ├── peer.key
    ├── server.crt
    └── server.key

Create the static pod manifests

Now that the certificates and configs are in place it’s time to create the manifests. On each host run the kubeadm command to generate a static manifest for etcd

NOTE: Run commands on $HOST0, $HOST1 and $HOST2

root@k8s-etcd-01 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
root@k8s-etcd-02 $ kubeadm init phase etcd local --config=/home/larry/kubeadmcfg.yaml
root@k8s-etcd-03 $ kubeadm init phase etcd local --config=/home/larry/kubeadmcfg.yaml

Optional: Check the cluster health

  • Set ${ETCD_TAG} to the version tag of your etcd image. For example v3.3.15
  • Set ${HOST1}to the IP address of the ETCD host you are testing.
# Example host IP's with ETCD to test
export HOST0=10.240.0.5
export HOST1=10.240.0.6
export HOST2=10.240.0.7

ETCD_TAG=v3.3.15

docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:${ETCD_TAG} etcdctl \
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
--key-file /etc/kubernetes/pki/etcd/peer.key \
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://${HOST1}:2379 cluster-health

# Result
cluster is healthy

Configure Load Balancer – HA proxy

NOTE: This step is to be performed on the load balancer node – k8s-haproxy

Since we are deploying three Kubernetes master nodes, we need to deploy a HA-Proxy load balancer in front of them to distribute the Api traffic.

Update the k8s-haproxy machine and install haproxy

apt-get update -y

# Install HA-Proxy
apt-get install haproxy -y

Configure the HA-Proxy load balancer

$ vim /etc/haproxy/haproxy.cfg

global
        ...

default
        ...

frontend kubernetes
        bind 10.240.0.11:6443
        option tcplog
        mode tcp
        default_backend kubernetes-master-nodes

backend kubernetes-master-nodes
        mode tcp
        balance roundrobin
        option tcp-check
        server k8s-master-01 10.240.0.2:6443 check fall 3 rise 2
        server k8s-master-02 10.240.0.3:6443 check fall 3 rise 2
        server k8s-master-03 10.240.0.4:6443 check fall 3 rise 2

Optional: Test HA-Proxy ports from 1st master

apt-get install netcat
nc -vn 10.240.0.11 6443

# Result should be
(UNKNOWN) [10.240.0.11] 6443 (?) open

Set up Master nodes

Copy the following files from any etcd node to the first control plane node

You can copy the following files from any etcd node in the cluster to the first control plane node. Replace the value of CONTROL_PLANE with the user@host of the first control plane plane node.

# I'm on ETCD $HOST0
export CONTROL_PLANE="[email protected]"
ssh ${CONTROL_PLANE}
mkdir -p /etc/kubernetes/pki/etcd/

# Exit ssh and copy files from $HOST0
scp /etc/kubernetes/pki/etcd/ca.crt "${CONTROL_PLANE}":/etc/kubernetes/pki/etcd/ca.crt
scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${CONTROL_PLANE}":/etc/kubernetes/pki/apiserver-etcd-client.crt
scp /etc/kubernetes/pki/apiserver-etcd-client.key "${CONTROL_PLANE}":/etc/kubernetes/pki/apiserver-etcd-client.crt

Set up the first control plane node – k8s-master-01

Create a file called kubeadm-config.yaml with the following contents. Replace the following variables in the config template with the appropriate values for your cluster

  • LOAD_BALANCER_DNS
  • LOAD_BALANCER_PORT
  • ETCD_1_IP
  • ETCD_2_IP
  • ETCD_3_IP
vim kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "k8s-haproxy:6443"
networking:
        podSubnet: "192.168.0.0/16"
etcd:
    external:
        endpoints:
        - https://10.240.0.5:2379
        - https://10.240.0.6:2379
        - https://10.240.0.7:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

Initialize the cluster

  • You can use the --kubernetes-version flag to set the Kubernetes version to use. It is recommended that the versions of kubeadm, kubelet, kubectl and Kubernetes match
  • The --control-plane-endpoint flag should be set to the address or DNS and port of the load balancer. In this instance the k8s-haproxy machine.

NOTE: Calico require a CIDR such as 192.168.0.0/16.To add a pod CIDR pass the flag --pod-network-cidr. We are using a kubeadm configuration file so we set the podSubnet field under the networking object of ClusterConfiguration in the kubeadm-config.yaml.

# Initialize cluster on 1st Master node
kubeadm init --config kubeadm-config.yaml --upload-certs

Results from kubeadm init, Follow steps from your output to complete joining cluster

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
    --discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253 \
    --control-plane --certificate-key 7caf342d8084c9cb310e38139d06e7c22adc1dc8fa8e1e5856105ec79fb0e39f

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
    --discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253
root@k8s-master-01:~#

Export your KUBECONFIG as normal user

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Join additional Master nodes

You can now join any number of the control-plane node running the following command on each as root:

kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
    --discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253 \
    --control-plane --certificate-key 7caf342d8084c9cb310e38139d06e7c22adc1dc8fa8e1e5856105ec79fb0e39f

# Output

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.


To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

Join the worker nodes

Use command from your kubeadm init output as stated above

kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
    --discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253

# Output

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Check the status of the nodes, not yet ready because there’s no network plugin

root@k8s-master-01:~# kubectl get nodes
NAME            STATUS     ROLES    AGE     VERSION
k8s-master-01   NotReady   master   7m19s   v1.16.3
k8s-master-02   NotReady   master   78s     v1.16.3
k8s-master-03   NotReady   master   38s     v1.16.3

Install Calico CNI

For Calico to work correctly, you need to pass --pod-network-cidr=192.168.0.0/16 to kubeadm init or update the calico.yml file to match your Pod network. Note that Calico works on amd64arm64, and ppc64le only

larry@k8s-master-01:~# kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml

# Output

larry@k8s-master-01:~# kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

Your cluster should be ready in a few minutes after calico CNI plugin installation. Please type the following and watch the pods of the control plane components get started:

kubectl get pod -n kube-system -w

You can run kubectl get nodes again. We can see all master nodes and worker nodes joined the cluster

root@k8s-master-01:~# kubectl get nodes
NAME            STATUS   ROLES    AGE     VERSION
k8s-master-01   Ready    master   15m     v1.16.3
k8s-master-02   Ready    master   9m55s   v1.16.3
k8s-master-03   Ready    master   9m15s   v1.16.3
k8s-worker-01   Ready    <none>   2m15s   v1.16.3
k8s-worker-02   Ready    <none>   86s     v1.16.3
k8s-worker-03   Ready    <none>   78s     v1.16.3

Test the cluster

Lets return all pods in all namespaces

root@k8s-master-01:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-55754f75c-7dv7v   1/1     Running   0          13m
kube-system   calico-node-2hqgf                         1/1     Running   0          8m11s
kube-system   calico-node-2r8hm                         1/1     Running   0          13m
kube-system   calico-node-htc9v                         1/1     Running   0          13m
kube-system   calico-node-l5kcc                         1/1     Running   0          7m14s
kube-system   calico-node-s5ndr                         1/1     Running   0          7m22s
kube-system   calico-node-svlsr                         1/1     Running   0          13m
kube-system   coredns-5644d7b6d9-nflz8                  1/1     Running   0          21m
kube-system   coredns-5644d7b6d9-zwlbb                  1/1     Running   0          21m
kube-system   kube-apiserver-k8s-master-01              1/1     Running   0          20m
kube-system   kube-apiserver-k8s-master-02              1/1     Running   0          14m
kube-system   kube-apiserver-k8s-master-03              1/1     Running   0          14m
kube-system   kube-controller-manager-k8s-master-01     1/1     Running   0          20m
kube-system   kube-controller-manager-k8s-master-02     1/1     Running   0          14m
kube-system   kube-controller-manager-k8s-master-03     1/1     Running   0          14m
kube-system   kube-proxy-4c5l6                          1/1     Running   0          15m
kube-system   kube-proxy-6bnq8                          1/1     Running   0          8m11s
kube-system   kube-proxy-9hwcm                          1/1     Running   0          15m
kube-system   kube-proxy-bs4bn                          1/1     Running   0          7m14s
kube-system   kube-proxy-pdhfj                          1/1     Running   0          7m22s
kube-system   kube-proxy-pmklj                          1/1     Running   0          21m
kube-system   kube-scheduler-k8s-master-01              1/1     Running   0          20m
kube-system   kube-scheduler-k8s-master-02              1/1     Running   0          14m
kube-system   kube-scheduler-k8s-master-03              1/1     Running   0          14m

Check ETCD pods on one ETCD node

larry@k8s-etcd-01:~# docker container ls
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
c59d9d2c57f4        b2756210eeab           "etcd --advertise-cl…"   4 hours ago         Up 4 hours                              k8s_etcd_etcd-k8s-etcd-01_kube-system_e5b787535d1d4c9aac5c9734aa76dcb4_2
4f945c09f90d        k8s.gcr.io/pause:3.1   "/pause"                 4 hours ago         Up 4 hours                              k8s_POD_etcd-k8s-etcd-01_kube-system_e5b787535d1d4c9aac5c9734aa76dcb4_0
larry@k8s-etcd-01:~#

SSH into container and check ETCD members

docker exec -it k8s_etcd_etcd-k8s-etcd-01_kube-system_e5b787535d1d4c9aac5c9734aa76dcb4_2 /bin/sh
# 
# ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://10.240.0.5:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key> > > >
2c60f2ee3316523f, started, k8s-etcd-01, https://10.240.0.5:2380, https://10.240.0.5:2379
770c999e97b442de, started, k8s-etcd-03, https://10.240.0.7:2380, https://10.240.0.7:2379
f5ee6099b36640dc, started, k8s-etcd-02, https://10.240.0.6:2380, https://10.240.0.6:2379
#

Test Cluster-Health

root@k8s-etcd-01:~# docker run --rm -it \
> --net host \
> -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.3.15 etcdctl \
> --cert-file /etc/kubernetes/pki/etcd/peer.crt \
> --key-file /etc/kubernetes/pki/etcd/peer.key \
> --ca-file /etc/kubernetes/pki/etcd/ca.crt \
> --endpoints https://10.240.0.5:2379 cluster-health
Unable to find image 'quay.io/coreos/etcd:v3.3.15' locally
v3.3.15: Pulling from coreos/etcd
050382585609: Already exists
849de450fe1b: Pull complete
509b1d7645fb: Pull complete
f3b2d4d7b140: Pull complete
8162e3515505: Pull complete
a19b09b60c11: Pull complete
Digest: sha256:3d544d83e97da5f20ae6c63c8beda516c9c8c478447ae6778ffdaf202946cf53
Status: Downloaded newer image for quay.io/coreos/etcd:v3.3.15
member 2c60f2ee3316523f is healthy: got healthy result from https://10.240.0.5:2379
member 770c999e97b442de is healthy: got healthy result from https://10.240.0.7:2379
member f5ee6099b36640dc is healthy: got healthy result from https://10.240.0.6:2379
cluster is healthy
root@k8s-etcd-01:~#

Create a test deployment nginx

larry@k8s-master-01:~# kubectl create deploy nginx --image=nginx:1.10
deployment.apps/nginx created

larry@k8s-master-01:~# kubectl scale deploy nginx --replicas=10
deployment.apps/nginx scaled

larry@k8s-master-01:~# kubectl expose deploy nginx --type=NodePort --port=80
service/nginx exposed

Check the Deployment, Service and Pods

root@k8s-master-01:~# kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-857df58577-4d4z8   1/1     Running   0          7m
pod/nginx-857df58577-4jvmk   1/1     Running   0          7m23s
pod/nginx-857df58577-c6kj8   1/1     Running   0          7m
pod/nginx-857df58577-dbp6h   1/1     Running   0          7m
pod/nginx-857df58577-jjbjb   1/1     Running   0          7m
pod/nginx-857df58577-phkct   1/1     Running   0          7m
pod/nginx-857df58577-pjzfd   1/1     Running   0          7m
pod/nginx-857df58577-qkj7d   1/1     Running   0          6m59s
pod/nginx-857df58577-ts7sc   1/1     Running   0          7m
pod/nginx-857df58577-wz57d   1/1     Running   0          6m59s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        40m
service/nginx        NodePort    10.103.136.216   <none>        80:30522/TCP   6m14s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   10/10   10           10          7m23s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-857df58577   10        10        10      7m23s

Describe the service, view endpoints and more

larry@k8s-master-01:~# kubectl describe svc nginx
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              <none>
Selector:                 app=nginx
Type:                     NodePort
IP:                       10.103.136.216
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30522/TCP
Endpoints:                192.168.118.65:80,192.168.118.66:80,192.168.118.67:80 + 7 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Your cluster should be up and working correctly. Install dashboards, monitoring to get insights from your cluster. Thank you.


Comments

4 responses to “Creating Highly Available clusters with kubeadm with external ETCD nodes”

  1. What are the requirements for the VMs? What does the firewall security of the VM’s look like? I am having trouble to to spin up VM’s. Can you help me with this.
    Thanks

  2. Jiwoong Song Avatar
    Jiwoong Song

    Amazing!
    It was very helpful for me. 🙂
    This post is very well organized for easy understanding.
    Thanks.

    I have couple of questions.

    1. How can I add a new etcd nodes to etcd cluster? or remove etcd nodes?

    2. How can i change a k8s cluster from local etcd to external etcd without reinstalling(or reset) the k8s cluster?

  3. hi.. the instructions are clear.. however, when i tried to test the cluster health (Optional: Check the cluster health)

    i encounter this issue.
    docker run –rm -it –net host -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:${ETCD_TAG} etcdctl –cert-file /etc/kubernetes/pki/etcd/peer.crt –key-file /etc/kubernetes/pki/etcd/peer.key –ca-file /etc/kubernetes/pki/etcd/ca.crt –endpoints https://10.148.0.5:2379 cluster-health
    cluster may be unhealthy: failed to list members
    Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 10.148.0.5:2379: connect: connection refused

    error #0: dial tcp 10.148.0.5:2379: connect: connection refused

    is there any additional configuration required?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.