Guides

Install High-availability Kubernetes Cluster-1.18-CentOS

All the packages introduced below can be download from the link:

https://pan.baidu.com/s/1lWLyyT1jAbADGuxEG3m4gA Password: ge4d

System Preparation

Use the IOS installation system from the link above(Please use the CentOS-8.3.2011-x86_64-Minimal.iso file provided by SuperMap to install a clean system).
- Path: iManager 部署包/软件工具包/OS/OS-ISO/CentOS-8.3.2011-x86_64-Minimal.iso
When installing the system, all the partitioned file systems select ext4, except swap partition. The root(/) partition must occupies 90% or more resources of the total disk.
System Requirements:
- Number of processors: 4 cores+
- Memory: 8 GB+(The larger the memory, the better the performance)

Installation Package Preparation

Path: iManager部署包/软件工具包/Kubernetes离线安装包/kubernetes-1.18.20-centos-8.3-deploy.zip

Environment Preparation

For a high-availability Kubernetes cluster, to ensure the high availability of the cluster, prepare at least three physical machines as master nodes, and install each physical machine according to the System Preparation guide. A Kubernetes cluster composed of multiple master nodes has stronger fault tolerance. If a cluster consists of N master nodes, it can tolerate (N-1)/2 master nodes failing simultaneously (due to disconnection, hardware errors, disk damage, etc.). When the number of failed master nodes exceeds (N-1)/2, the entire Kubernetes cluster will fail. Refer to the table below for details.

Number of Master Node Fault Tolerance

1 0

2 0

3 1

4 1

5 2

6 2

7 3

8 3

9 4

As shown in the table above, it is preferable for a high-availability Kubernetes cluster to have an odd number of master nodes. Having an even number of master nodes does not increase the fault tolerance of the cluster.
Prepare an available IP address as the entry IP for the Kubernetes cluster. The IP address is preferably in the same network segment as the physical machines and is not in use.
Deployment of this environment is currently not supported on Alibaba Cloud and Huawei Cloud.

Number of Master Node	Fault Tolerance
1	0
2	0
3	1
4	1
5	2
6	2
7	3
8	3
9	4

Install Kubernetes Master Node

Unzip kubernetes-1.18.20-centos-8.3-deploy.zip, copy the offline-kubernetes-1.18.20-centos-8.3.tar.gz package to any directory of the machine.

Unzip:

tar ‐zxvf offline-kubernetes-1.18.20-centos-8.3.tar.gz

Enter the directory that you unzipped:

cd offline-kubernetes-1.18.20-centos-8.3

Edit the .env file to configure the Kubernetes cluster:

 # Required: the hostname of the current node, which must be unique among Kubernetes nodes. It is recommended not to include the string "master" and can be optionally replaced with "main" or "control". It is not recommended to include special characters "." too.
 HOSTNAME=
 # The IP address used to connect to the Kubernetes cluster. For a high availability cluster, CLUSTER_IP is a virtual IP, which is the previously prepared available IP address. For a cluster consisting of a single master node, CLUSTER_IP is the IP address of that master node.
 CLUSTER_IP=
 # The port used to connect to the Kubernetes cluster.
 # For a high availability cluster, CLUSTER_PORT is the port for connecting to the Kubernetes cluster, but it cannot be 6443. You can use ports such as 9443 that are not in use.
 # For a cluster consisting of a single master node, CLUSTER_PORT is the exposed 6443 port of that master node.
 CLUSTER_PORT=
 # Initialize/join the Kubernetes cluster in high availability mode.
 HA_ENABLED=true
 # Required when EANBLED_HA=true: the master node addresses for the high availability Kubernetes cluster load balancing, separated by commas. e.g. 172.16.120.144,172.16.120.145
 HA_APISERVER_BACKEND=
 # Required when EANBLED_HA=true: the network card name to bind the virtual IP on the current physical machine. The network card needs to be a functional one for communication.
 HA_NETWORK_CARD=
 # Specify the role of the current installed node in the Kubernetes cluster when joining. Options are MASTER, WORKER. If not specified, WORKER is used by default.
 JOIN_ROLE=
 # Required when joining nodes (worker nodes and control nodes): the token used to join the node, which can be created by kubeadm token create on the control node and expires in 24 hours.
 JOIN_TOKEN=
 # Required when JOIN_ROLE=MASTER: the key used for joining. It can be created by kubeadm init phase upload-certs --upload-certs and expires in 2 hours.
 JOIN_MASTER_CERT_KEY=
 # Whether the master node (including the initialized master node and the master node joined to the cluster) participates in service scheduling by default. The default is false, which means it does not participate in scheduling; true means it participates in scheduling.
 # Service scheduling: whether services on the Kubernetes cluster are allowed to be deployed to master nodes.
 #    For large clusters where master nodes manage node nodes, it is not recommended to participate in scheduling to reduce pressure.
 #    For small clusters where all physical machine resources need to be utilized as much as possible, master nodes can participate in scheduling. 
 MASTER_DISPATCH=false

The specific parameters that need to be configured can be referred to in the table below.

Name of the Parameters
HOSTNAME
CLUSTER_IP
CLUSTER_PORT
HA_ENABLED
HA_ APISERVER_BACKEND
HA_ NETWORK_CARD
MASTER_DISPATCH

Execute the following command with root permission:

chmod +x install.sh && ./install.sh master

Notes:

If you have multiple network cards on the machine, please assign one network card for Kubernetes, execute the command --apiserver-advertise-address IP(replace the IP in the command to the actual IP of network card). For instance, if the IP of network card is 172.16.16.9, the completely command should be chmod +x install.sh && ./install.sh --apiserver-advertise-address 172.16.16.9.

When you copy the completely command to use, if the pasted command wraps in a new line, it must be adjusted on the same line before executing the command.

After installing, you can visit Kubernetes UI by the address http://IP:31234, please replace the IP by the actual IP.

A single master node can also be used directly. By the increase of demands, add Kubernetes master/worker nodes anytime.

Join Kubernetes Master Node

If you need to add Kubernetes master nodes, please prepare another machine, the new master node can not be installed on the same machine of other master node. Time synchronization should be consistent across all nodes in the cluster. Please refer to FAQ Question 7 for instructions on how to synchronize time.

Unzip kubernetes-1.18.20-centos-8.3-deploy.zip, copy the offline-kubernetes-1.18.20-centos-8.3.tar.gz package to any directory of the machine.

Unzip:

tar ‐zxvf offline-kubernetes-1.18.20-centos-8.3.tar.gz

Enter the directory that you unzipped:

cd offline-kubernetes-1.18.20-centos-8.3

Refer to Install Kubernetes Master Node and edit the .env file according to the table below to configure the Kubernetes cluster.

Name of the Parameters
HOSTNAME
CLUSTER_IP
CLUSTER_PORT
HA_ENABLED
HA_ APISERVER_BACKEND
HA_ NETWORK_CARD
JOIN_ROLE
JOIN_TOKEN
JOIN_ MASTER_CERT _KEY
MASTER_DISPATCH

Notes:

If the current node is not planned in .env HAAPISERVERBACKEND and is added later, then the following operations need to be performed on each existing master node in the cluster:

(1) Edit the conf/haproxy.cfg file in the installation directory, and add the following line at the end.

server master-<ip> <ip>:6443 check

is the IP address of the current node.

For example: server master-172.16.120.146 172.16.120.146:6443 check

(2) Restart the container. docker restart supermap-haproxy

Execute the following command with root permission:

chmod +x install.sh && ./install.sh join

Notes:

If you have multiple network cards on the machine, please assign one network card for Kubernetes, execute the command --apiserver-advertise-address IP(replace the IP in the command to the actual IP of network card). For instance, if the IP of network card is 172.16.16.10, the completely command should be chmod +x install.sh && ./install.sh --apiserver-advertise-address 172.16.16.10.
chmod +x install.sh && ./install.sh join --apiserver-advertise-address 172.16.16.10

After successfully joining the cluster, execute the following command on the current node to view:

kubectl get nodes

Join Kubernetes Worker Node(Optional)

If you need to install the Kubernetes worker node, please prepare another machine, the worker node can not be installed on the same machine of master nodes. Time synchronization should be consistent across all nodes in the cluster. Please refer to FAQ Question 7 for instructions on how to synchronize time.

Copy the offline-kubernetes-1.18.20-centos-8.3.tar.gz package to any directory of the machine.

Unzip:

tar ‐zxvf offline-kubernetes-1.18.20-centos-8.3.tar.gz

Enter the directory that you unzipped:

cd offline-kubernetes-1.18.20-centos-8.3

Refer to Install Kubernetes Master Node and edit the .env file according to the table below to configure the Kubernetes cluster.

Name of the Parameters
HOSTNAME
CLUSTER_IP
CLUSTER_PORT
HA_ENABLED
HA_ APISERVER_BACKEND
HA_ NETWORK_CARD
JOIN_TOKEN
JOIN_ MASTER_CERT _KEY

Execute the following command with root permission:

chmod +x install.sh && ./install.sh join

Notes:

If you have multiple network cards on the machine, please assign one network card for Kubernetes, execute the command --apiserver-advertise-address IP(replace the IP in the command to the actual IP of network card). For instance, if the IP of network card is 172.16.16.10, the completely command should be chmod +x install.sh && ./install.sh --apiserver-advertise-address 172.16.16.10.
chmod +x install.sh && ./install.sh join --apiserver-advertise-address 172.16.16.10

After successfully joining the cluster, execute the following command on kubernetes master node to view:

kubectl get nodes

FAQ

If the error in below occurs when installing the worker node, it means the token was expired, you need to create a new token in the master node by the command ‘Kubeadm token create’.

Error:
```
couldn't validate the identify of the API server: abort connection to API server after timeout of 5m0s
```
If you want to remove the worker node from Kubernetes cluster, execute the command on the master node:
```
kubectl delete node <nodeName>
```
Notes:

<nodeName> is the name of the worker node you would like to remove.
If you want to remove a master node from Kubernetes cluster, execute the command on any other master node:
```
kubectl delete node <nodeName>
```
Notes:

<nodeName> is the name of the master node you would like to remove.

Enter the etcd container and delete the etcd service on the removed master node.

kubectl -n kube-system get pod | grep etcd
# Please use the actual etcd service name as reference, which is 'etcd-master01' here.
kubectl -n kube-system exec -it etcd-master01 sh

Enter the following command inside the container to view the existing etcd services.

 etcdctl --endpoints https://127.0.0.1:2379 --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/server.crt --key-file /etc/kubernetes/pki/etcd/server.key member list

The output is similar to the following:

4710f7f5592b46ed: name=master03 peerURLs=https://10.10.130.211:2380 clientURLs=https://10.10.130.211:2379 isLeader=false
b0634635e7aaf095: name=master02 peerURLs=https://10.10.185.16:2380 clientURLs=https://10.10.185.16:2379 isLeader=true
e3eff812e9c03ad9: name=master01 peerURLs=https://10.10.90.97:2380 clientURLs=https://10.10.90.97:2379 isLeader=false

Delete the etcd service on the removed node using the ID of the etcd service.

etcdctl --endpoints https://127.0.0.1:2379 --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/server.crt --key-file /etc/kubernetes/pki/etcd/server.key member remove <etcd-id>

<etcd-id> is the unique ID of the etcd service.

After the removal is complete, exit the etcd container, and execute the following command on the other Kubernetes master nodes.

In the installation directory, edit the conf/haproxy.cfg file, and delete the load balancing configuration information for the current node at the end of the file. The configuration information is similar to the following:

server master-<ip> <ip>:6443 check

<ip> is the IP address of the current node.

Restart the container:

docker restart supermap-haproxy

Go to the removed master node and clean up the virtual IP and load balancing.

docker stop supermap-haproxy supermap-keepalived
docker rm supermap-haproxy supermap-keepalived

Please execute the following command to reset the removed node.

kubeadm reset

If you want to rejoin the removed worker node in the Kubernetes cluster, Please perform the operation in 4. on the node to be removed first, and then follow the instructions for Join Kubernetes Master Node or Join Kubernetes Node(Optional) on the nodes joining the cluster.
If you want to uninstall Kubernetes, Please execute the following command in the root directories of each node, and then restart the physical machine to clean up the configuration.

chmod +x install.sh && ./install.sh down

CentOS 8 comes with chronyc as the time synchronization tool. To synchronize the time, you first need an accessible time server. Then follow these steps:

(1) Start the chronyd service
```
systemctl enable chronyd
```
(2) Edit the /etc/chrony.conf file and enter the time server address
```
server 172.16.120.139 iburst
```
Replace 172.16.120.139 with the actual time server address

(3) Execute the command to synchronize the time
```
systemctl start chronyd
chronyc -a makestep
```
If there is no time server, you can also configure the current machine as a time server using chronyd, following these steps:

(1) Edit the /etc/chrony.conf file and enter the following content
```
allow 172.16.0.0/16 # Set the subnet or network of machines allowed to connect to the time server
local stratum 10 # When the public NTP servers provided in the server section are unavailable, use the local time as the synchronization standard
```
(2) Start chronyd
```
systemctl restart chronyd
systemctl enable chronyd
```