Overview
This article explains how to set up a highly available Kubernetes 1.16.2 cluster using kubeadm with external etcd nodes. This approach requires more infrastructure. The control plane nodes and etcd members are separated. See attached diagram below.

Prerequisites
I will cover steps for operating systems as mentioned below. Minimum server requirements:
- Debian 9+, Ubuntu 18.04+ tested
- 2 GB or more of RAM per node (Any less will leave little room for your Apps)
- 2 CPUs or more on the Master nodes
- Full network connectivity between all nodes in the cluster (public or private network is fine)
- Root access
- SSH access to copy Certificates access from ETCD nodes
- Certain ports should be open on your nodes. See here for more details.
- Swap disabled. You MUST disable swap in order for the kubelet to work properly.
Setup Environment
I have completed and tested this setup on Google Cloud but you can also work with bare metal or virtual machines. I tested with Debian 9+ and Ubuntu 18.04 Operating Systems.
NOTE: The below Hostnames and Ip addresses are just examples from my configuration
Hostname | Role | IP Address |
k8s-haproxy | load balancer | 10.240.0.11 |
k8s-master-01 | master | 10.240.0.2 |
k8s-master-02 | master | 10.240.0.3 |
k8s-master-03 | master | 10.240.0.4 |
k8s-etcd-01 | etcd | 10.240.0.5 |
k8s-etcd-02 | etcd | 10.240.0.6 |
k8s-etcd-03 | etcd | 10.240.0.7 |
k8s-worker-01 | worker | 10.240.0.8 |
k8s-worker-02 | worker | 10.240.0.9 |
k8s-worker-03 | worker | 10.240.0.10 |
Install Docker CE
NOTE: Please perform these steps on all Master, Worker, Etcd nodes
Install packages to allow apt to use a repository over HTTPS
sudo su -
apt-get update && apt-get install -y apt-transport-https ca-certificates curl software-properties-common
Add Docker’s official GPG key, Add Docker apt repository
For Ubuntu 18.04+
# Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# Add Docker apt repository
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
# Install Docker CE
apt-get update && apt-get install -y docker-ce=18.06.2~ce~3-0~ubuntu
For Debian 9+
# Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
# Add Docker apt repository
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
# Install Docker CE
apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~debian
Setup the docker daemon & restart docker.
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart docker
systemctl daemon-reload && systemctl restart docker
Install Kubeadm, Kubectl, Kubelet
apt-get update -y
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update -y && apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
Setup ETCD external nodes
NOTE: This step is to be performed on the 3 external Etcd nodes
Configure the kubelet to be a service manager for etcd
Since etcd was created first, we must override the service priority by creating a new unit file that has higher precedence than the kubeadm-provided kubelet unit file.
cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
# Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
Restart=always
EOF
# Reload and restart the kubelet
systemctl daemon-reload && systemctl restart kubelet
Create configuration files for kubeadm
NOTE: This step is to be performed on One or first Etcd node
Generate one kubeadm configuration file for each host that will have an etcd member running on it using the following script. We will copy the configs across once complete.
# Update HOST0, HOST1, and HOST2 with the IP's or resolvable names of your ETCD hosts
export HOST0=10.240.0.5
export HOST1=10.240.0.6
export HOST2=10.240.0.7
Create temp directories to store config files that will end up on the other hosts
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
Create the config files
ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("k8s-etcd-01" "k8s-etcd-02" "k8s-etcd-03")
for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta2"
kind: ClusterConfiguration
etcd:
local:
serverCertSANs:
- "${HOST}"
peerCertSANs:
- "${HOST}"
extraArgs:
initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380,${NAMES[2]}=https://${ETCDHOSTS[2]}:2380
initial-cluster-state: new
name: ${NAME}
listen-peer-urls: https://${HOST}:2380
listen-client-urls: https://${HOST}:2379
advertise-client-urls: https://${HOST}:2379
initial-advertise-peer-urls: https://${HOST}:2380
EOF
done
Generate the Certificate Authority
NOTE: Run these commands on $HOST0
(where you generated the configuration files for kubeadm)
kubeadm init phase certs etcd-ca
This command creates these two files:
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key
Create certificates for each member
kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
# Cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
# Cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0
# Clean up certs that should not be copied off this host
find /tmp/${HOST2} -name ca.key -type f -delete
find /tmp/${HOST1} -name ca.key -type f -delete
Copy certificates and kubeadm configs
The certificates have been generated and now they must be moved to their respective hosts on k8s-etcd-02
and k8s-etcd-03
# k8s-etcd-02
USER=larry
HOST=${HOST1}
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
ssh ${USER}@${HOST}
USER@HOST $ sudo -Es
root@HOST $ chown -R root:root pki
root@HOST $ mv pki /etc/kubernetes/
# k8s-etcd-03
USER=larry
HOST=${HOST2}
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
ssh ${USER}@${HOST}
USER@HOST $ sudo -Es
root@HOST $ chown -R root:root pki
root@HOST $ mv pki /etc/kubernetes/
Ensure all expected files exist
The complete list of required files on $HOST0
is:
/tmp/${HOST0}
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
├── ca.crt
├── ca.key
├── healthcheck-client.crt
├── healthcheck-client.key
├── peer.crt
├── peer.key
├── server.crt
└── server.key
On $HOST1
$HOME
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
├── ca.crt
├── healthcheck-client.crt
├── healthcheck-client.key
├── peer.crt
├── peer.key
├── server.crt
└── server.key
On $HOST2
$HOME
└── kubeadmcfg.yaml
---
/etc/kubernetes/pki
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
├── ca.crt
├── healthcheck-client.crt
├── healthcheck-client.key
├── peer.crt
├── peer.key
├── server.crt
└── server.key
Create the static pod manifests
Now that the certificates and configs are in place it’s time to create the manifests. On each host run the kubeadm
command to generate a static manifest for etcd
NOTE: Run commands on $HOST0
, $HOST1
and $HOST2
root@k8s-etcd-01 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
root@k8s-etcd-02 $ kubeadm init phase etcd local --config=/home/larry/kubeadmcfg.yaml
root@k8s-etcd-03 $ kubeadm init phase etcd local --config=/home/larry/kubeadmcfg.yaml
Optional: Check the cluster health
- Set
${ETCD_TAG}
to the version tag of your etcd image. For examplev3.3.15
- Set
${HOST1}
to the IP address of the ETCD host you are testing.
# Example host IP's with ETCD to test
export HOST0=10.240.0.5
export HOST1=10.240.0.6
export HOST2=10.240.0.7
ETCD_TAG=v3.3.15
docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:${ETCD_TAG} etcdctl \
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
--key-file /etc/kubernetes/pki/etcd/peer.key \
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://${HOST1}:2379 cluster-health
# Result
cluster is healthy
Configure Load Balancer – HA proxy
NOTE: This step is to be performed on the load balancer node – k8s-haproxy
Since we are deploying three Kubernetes master nodes, we need to deploy a HA-Proxy load balancer in front of them to distribute the Api traffic.
Update the k8s-haproxy machine and install haproxy
apt-get update -y
# Install HA-Proxy
apt-get install haproxy -y
Configure the HA-Proxy load balancer
$ vim /etc/haproxy/haproxy.cfg
global
...
default
...
frontend kubernetes
bind 10.240.0.11:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master-01 10.240.0.2:6443 check fall 3 rise 2
server k8s-master-02 10.240.0.3:6443 check fall 3 rise 2
server k8s-master-03 10.240.0.4:6443 check fall 3 rise 2
Optional: Test HA-Proxy ports from 1st master
apt-get install netcat
nc -vn 10.240.0.11 6443
# Result should be
(UNKNOWN) [10.240.0.11] 6443 (?) open
Set up Master nodes
Copy the following files from any etcd node to the first control plane node
You can copy the following files from any etcd node in the cluster to the first control plane node. Replace the value of CONTROL_PLANE
with the user@host
of the first control plane plane node.
# I'm on ETCD $HOST0
export CONTROL_PLANE="[email protected]"
ssh ${CONTROL_PLANE}
mkdir -p /etc/kubernetes/pki/etcd/
# Exit ssh and copy files from $HOST0
scp /etc/kubernetes/pki/etcd/ca.crt "${CONTROL_PLANE}":/etc/kubernetes/pki/etcd/ca.crt
scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${CONTROL_PLANE}":/etc/kubernetes/pki/apiserver-etcd-client.crt
scp /etc/kubernetes/pki/apiserver-etcd-client.key "${CONTROL_PLANE}":/etc/kubernetes/pki/apiserver-etcd-client.crt
Set up the first control plane node – k8s-master-01
Create a file called kubeadm-config.yaml
with the following contents. Replace the following variables in the config template with the appropriate values for your cluster
LOAD_BALANCER_DNS
LOAD_BALANCER_PORT
ETCD_1_IP
ETCD_2_IP
ETCD_3_IP
vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "k8s-haproxy:6443"
networking:
podSubnet: "192.168.0.0/16"
etcd:
external:
endpoints:
- https://10.240.0.5:2379
- https://10.240.0.6:2379
- https://10.240.0.7:2379
caFile: /etc/kubernetes/pki/etcd/ca.crt
certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
Initialize the cluster
- You can use the
--kubernetes-version
flag to set the Kubernetes version to use. It is recommended that the versions of kubeadm, kubelet, kubectl and Kubernetes match - The
--control-plane-endpoint
flag should be set to the address or DNS and port of the load balancer. In this instance thek8s-haproxy
machine.
NOTE: Calico require a CIDR such as 192.168.0.0/16
.To add a pod CIDR pass the flag --pod-network-cidr
. We are using a kubeadm configuration file so we set the podSubnet
field under the networking
object of ClusterConfiguration
in the kubeadm-config.yaml
.
# Initialize cluster on 1st Master node
kubeadm init --config kubeadm-config.yaml --upload-certs
Results from kubeadm init
, Follow steps from your output to complete joining cluster
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
--discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253 \
--control-plane --certificate-key 7caf342d8084c9cb310e38139d06e7c22adc1dc8fa8e1e5856105ec79fb0e39f
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
--discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253
root@k8s-master-01:~#
Export your KUBECONFIG
as normal user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Join additional Master nodes
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
--discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253 \
--control-plane --certificate-key 7caf342d8084c9cb310e38139d06e7c22adc1dc8fa8e1e5856105ec79fb0e39f
# Output
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
Join the worker nodes
Use command from your kubeadm init
output as stated above
kubeadm join k8s-haproxy:6443 --token 92vwpe.fncgisukodl8w6vh \
--discovery-token-ca-cert-hash sha256:d133a360d51d23ebc6f94e81494332b99b49bf3ac568639770c830684ee6c253
# Output
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Check the status of the nodes, not yet ready because there’s no network plugin
root@k8s-master-01:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-01 NotReady master 7m19s v1.16.3
k8s-master-02 NotReady master 78s v1.16.3
k8s-master-03 NotReady master 38s v1.16.3
Install Calico CNI
For Calico to work correctly, you need to pass --pod-network-cidr=192.168.0.0/16
to kubeadm init
or update the calico.yml
file to match your Pod network. Note that Calico works on amd64
, arm64
, and ppc64le
only
larry@k8s-master-01:~# kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
# Output
larry@k8s-master-01:~# kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Your cluster should be ready in a few minutes after calico CNI plugin installation. Please type the following and watch the pods of the control plane components get started:
kubectl get pod -n kube-system -w
You can run kubectl get nodes
again. We can see all master nodes and worker nodes joined the cluster
root@k8s-master-01:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-01 Ready master 15m v1.16.3
k8s-master-02 Ready master 9m55s v1.16.3
k8s-master-03 Ready master 9m15s v1.16.3
k8s-worker-01 Ready <none> 2m15s v1.16.3
k8s-worker-02 Ready <none> 86s v1.16.3
k8s-worker-03 Ready <none> 78s v1.16.3
Test the cluster
Lets return all pods in all namespaces
root@k8s-master-01:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-55754f75c-7dv7v 1/1 Running 0 13m
kube-system calico-node-2hqgf 1/1 Running 0 8m11s
kube-system calico-node-2r8hm 1/1 Running 0 13m
kube-system calico-node-htc9v 1/1 Running 0 13m
kube-system calico-node-l5kcc 1/1 Running 0 7m14s
kube-system calico-node-s5ndr 1/1 Running 0 7m22s
kube-system calico-node-svlsr 1/1 Running 0 13m
kube-system coredns-5644d7b6d9-nflz8 1/1 Running 0 21m
kube-system coredns-5644d7b6d9-zwlbb 1/1 Running 0 21m
kube-system kube-apiserver-k8s-master-01 1/1 Running 0 20m
kube-system kube-apiserver-k8s-master-02 1/1 Running 0 14m
kube-system kube-apiserver-k8s-master-03 1/1 Running 0 14m
kube-system kube-controller-manager-k8s-master-01 1/1 Running 0 20m
kube-system kube-controller-manager-k8s-master-02 1/1 Running 0 14m
kube-system kube-controller-manager-k8s-master-03 1/1 Running 0 14m
kube-system kube-proxy-4c5l6 1/1 Running 0 15m
kube-system kube-proxy-6bnq8 1/1 Running 0 8m11s
kube-system kube-proxy-9hwcm 1/1 Running 0 15m
kube-system kube-proxy-bs4bn 1/1 Running 0 7m14s
kube-system kube-proxy-pdhfj 1/1 Running 0 7m22s
kube-system kube-proxy-pmklj 1/1 Running 0 21m
kube-system kube-scheduler-k8s-master-01 1/1 Running 0 20m
kube-system kube-scheduler-k8s-master-02 1/1 Running 0 14m
kube-system kube-scheduler-k8s-master-03 1/1 Running 0 14m
Check ETCD pods on one ETCD node
larry@k8s-etcd-01:~# docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c59d9d2c57f4 b2756210eeab "etcd --advertise-cl…" 4 hours ago Up 4 hours k8s_etcd_etcd-k8s-etcd-01_kube-system_e5b787535d1d4c9aac5c9734aa76dcb4_2
4f945c09f90d k8s.gcr.io/pause:3.1 "/pause" 4 hours ago Up 4 hours k8s_POD_etcd-k8s-etcd-01_kube-system_e5b787535d1d4c9aac5c9734aa76dcb4_0
larry@k8s-etcd-01:~#
SSH into container and check ETCD members
docker exec -it k8s_etcd_etcd-k8s-etcd-01_kube-system_e5b787535d1d4c9aac5c9734aa76dcb4_2 /bin/sh
#
# ETCDCTL_API=3 etcdctl member list \
--endpoints=https://10.240.0.5:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key> > > >
2c60f2ee3316523f, started, k8s-etcd-01, https://10.240.0.5:2380, https://10.240.0.5:2379
770c999e97b442de, started, k8s-etcd-03, https://10.240.0.7:2380, https://10.240.0.7:2379
f5ee6099b36640dc, started, k8s-etcd-02, https://10.240.0.6:2380, https://10.240.0.6:2379
#
Test Cluster-Health
root@k8s-etcd-01:~# docker run --rm -it \
> --net host \
> -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.3.15 etcdctl \
> --cert-file /etc/kubernetes/pki/etcd/peer.crt \
> --key-file /etc/kubernetes/pki/etcd/peer.key \
> --ca-file /etc/kubernetes/pki/etcd/ca.crt \
> --endpoints https://10.240.0.5:2379 cluster-health
Unable to find image 'quay.io/coreos/etcd:v3.3.15' locally
v3.3.15: Pulling from coreos/etcd
050382585609: Already exists
849de450fe1b: Pull complete
509b1d7645fb: Pull complete
f3b2d4d7b140: Pull complete
8162e3515505: Pull complete
a19b09b60c11: Pull complete
Digest: sha256:3d544d83e97da5f20ae6c63c8beda516c9c8c478447ae6778ffdaf202946cf53
Status: Downloaded newer image for quay.io/coreos/etcd:v3.3.15
member 2c60f2ee3316523f is healthy: got healthy result from https://10.240.0.5:2379
member 770c999e97b442de is healthy: got healthy result from https://10.240.0.7:2379
member f5ee6099b36640dc is healthy: got healthy result from https://10.240.0.6:2379
cluster is healthy
root@k8s-etcd-01:~#
Create a test deployment nginx
larry@k8s-master-01:~# kubectl create deploy nginx --image=nginx:1.10
deployment.apps/nginx created
larry@k8s-master-01:~# kubectl scale deploy nginx --replicas=10
deployment.apps/nginx scaled
larry@k8s-master-01:~# kubectl expose deploy nginx --type=NodePort --port=80
service/nginx exposed
Check the Deployment, Service and Pods
root@k8s-master-01:~# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-857df58577-4d4z8 1/1 Running 0 7m
pod/nginx-857df58577-4jvmk 1/1 Running 0 7m23s
pod/nginx-857df58577-c6kj8 1/1 Running 0 7m
pod/nginx-857df58577-dbp6h 1/1 Running 0 7m
pod/nginx-857df58577-jjbjb 1/1 Running 0 7m
pod/nginx-857df58577-phkct 1/1 Running 0 7m
pod/nginx-857df58577-pjzfd 1/1 Running 0 7m
pod/nginx-857df58577-qkj7d 1/1 Running 0 6m59s
pod/nginx-857df58577-ts7sc 1/1 Running 0 7m
pod/nginx-857df58577-wz57d 1/1 Running 0 6m59s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 40m
service/nginx NodePort 10.103.136.216 <none> 80:30522/TCP 6m14s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 10/10 10 10 7m23s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-857df58577 10 10 10 7m23s
Describe the service, view endpoints and more
larry@k8s-master-01:~# kubectl describe svc nginx
Name: nginx
Namespace: default
Labels: app=nginx
Annotations: <none>
Selector: app=nginx
Type: NodePort
IP: 10.103.136.216
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30522/TCP
Endpoints: 192.168.118.65:80,192.168.118.66:80,192.168.118.67:80 + 7 more...
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Your cluster should be up and working correctly. Install dashboards, monitoring to get insights from your cluster. Thank you.
Leave a Reply
You must be logged in to post a comment.