Tidb三节点混合部署

1. 环境准备

分类 属性
虚拟机 virtualbox
操作系统 kylin v10:4c-6g-60g
数据库 tidb v6.5.6

1.1 虚拟机准备

制作好标准镜像后,克隆三台服务器,IP地址分别为:

IP地址 主机名 用途
192.168.56.111 tidb1 tiup部署节点,混合tidb,tikv,pd
192.168.56.112 tidb2 混合tidb,tikv,pd
192.168.56.113 tidb3 混合tidb,tikv,pd

环境配置修改

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# 1. IP修改
vi /etc/sysconfig/network-scripts/ifcfg-enp0s8
IPADDR=192.168.56.111

systemctl restart network


# 2. 主机名修改
hostnamectl set-hostname tidb1


# 3. hosts修改
echo "192.168.56.111 tidb1" >> /etc/hosts


# 4. 互信认证(采用私钥)
ssh-keygen -t  rsa
-- 两次回车
cat id_rsa.pub >> ~/.ssh/authorized_keys


# 5. 关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config 
setenforce 0  

# 6. 密码修改(修改为简单密码,kylin v10 密码复杂度修改比较麻烦,可以通过一下方法强制)

## 交互式
$ openssl passwd -6 -salt $(< /dev/urandom tr -dc '[:alnum:]' | head -c 32)
Password: 
$6$LOsCVmwRuh96ukYE$zRMlncIfTrDNo6aJsfhEa8fVMieCIMVGvo9pEKMH21L.D1dQn.1mV/0SCDsNy11RRLCHARm77VAYoCeGy89jk0
-- 上面出现的一串数字即为加密密码,写入到/etc/shadow中。
## 非交互式
echo password | openssl passwd -6 -salt $(< /dev/urandom tr -dc '[:alnum:]' | head -c 32) -stdin

kylin v10密码复杂度取消非常麻烦,可通过强制

1.2 集群yaml文件准备

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
global:
  user: "tidb"
  ssh_port: 22
  deploy_dir: "/tidb-deploy"
  data_dir: "/tidb-data"
server_configs: {}
pd_servers:
  - host: 192.168.56.111
  - host: 192.168.56.112
  - host: 192.168.56.113
tidb_servers:
  - host: 192.168.56.111
  - host: 192.168.56.112
  - host: 192.168.56.113
tikv_servers:
  - host: 192.168.56.111
  - host: 192.168.56.112
  - host: 192.168.56.113
monitoring_servers:
  - host: 192.168.56.111
grafana_servers:
  - host: 192.168.56.111
alertmanager_servers:
  - host: 192.168.56.111

这是一个减配版本,未涉及到具体的调优参数。

2. 离线安装部署

2.1 离线安装包合并

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# 配置tiup环境
tar xzvf tidb-community-server-v6.5.6-linux-amd64.tar.gz && \ 
sh tidb-community-server-v6.5.6-linux-amd64/local_install.sh && \ 
source /home/tidb/.bash_profile
# 合并介质
tar xf tidb-community-toolkit-v6.5.6-linux-amd64.tar.gz
ls -ld tidb-community-server-v6.5.6-linux-amd64 tidb-community-toolkit-v6.5.6-linux-amd64
cd tidb-community-server-v6.5.6-linux-amd64/
cp -rp keys ~/.tiup/
tiup mirror merge ../tidb-community-toolkit-v6.5.6-linux-amd64

2.2 部署

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
[root@tidb1 ~]# tiup cluster deploy tidb-test v6.5.6 ./topology.yaml --user root  -p
tiup is checking updates for component cluster ...
Starting component `cluster`: /root/.tiup/components/cluster/v1.14.0/tiup-cluster deploy tidb-test v6.5.6 ./topology.yaml --user root -p
Input SSH password: 



+ Detect CPU Arch Name
  - Detecting node 192.168.56.111 Arch info ... Done
  - Detecting node 192.168.56.112 Arch info ... Done
  - Detecting node 192.168.56.113 Arch info ... Done



+ Detect CPU OS Name
  - Detecting node 192.168.56.111 OS info ... Done
  - Detecting node 192.168.56.112 OS info ... Done
  - Detecting node 192.168.56.113 OS info ... Done
Please confirm your topology:
Cluster type:    tidb
Cluster name:    tidb-test
Cluster version: v6.5.6
Role          Host            Ports        OS/Arch       Directories
----          ----            -----        -------       -----------
pd            192.168.56.111  2379/2380    linux/x86_64  /tidb-deploy/pd-2379,/tidb-data/pd-2379
pd            192.168.56.112  2379/2380    linux/x86_64  /tidb-deploy/pd-2379,/tidb-data/pd-2379
pd            192.168.56.113  2379/2380    linux/x86_64  /tidb-deploy/pd-2379,/tidb-data/pd-2379
tikv          192.168.56.111  20160/20180  linux/x86_64  /tidb-deploy/tikv-20160,/tidb-data/tikv-20160
tikv          192.168.56.112  20160/20180  linux/x86_64  /tidb-deploy/tikv-20160,/tidb-data/tikv-20160
tikv          192.168.56.113  20160/20180  linux/x86_64  /tidb-deploy/tikv-20160,/tidb-data/tikv-20160
tidb          192.168.56.111  4000/10080   linux/x86_64  /tidb-deploy/tidb-4000
tidb          192.168.56.112  4000/10080   linux/x86_64  /tidb-deploy/tidb-4000
tidb          192.168.56.113  4000/10080   linux/x86_64  /tidb-deploy/tidb-4000
prometheus    192.168.56.111  9090/12020   linux/x86_64  /tidb-deploy/prometheus-9090,/tidb-data/prometheus-9090
grafana       192.168.56.111  3000         linux/x86_64  /tidb-deploy/grafana-3000
alertmanager  192.168.56.111  9093/9094    linux/x86_64  /tidb-deploy/alertmanager-9093,/tidb-data/alertmanager-9093
Attention:
    1. If the topology is not what you expected, check your yaml file.
    2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y
+ Generate SSH keys ... Done
+ Download TiDB components
  - Download pd:v6.5.6 (linux/amd64) ... Done
  - Download tikv:v6.5.6 (linux/amd64) ... Done
  - Download tidb:v6.5.6 (linux/amd64) ... Done
  - Download prometheus:v6.5.6 (linux/amd64) ... Done
  - Download grafana:v6.5.6 (linux/amd64) ... Done
  - Download alertmanager: (linux/amd64) ... Done
  - Download node_exporter: (linux/amd64) ... Done
  - Download blackbox_exporter: (linux/amd64) ... Done
+ Initialize target host environments
  - Prepare 192.168.56.113:22 ... Done
  - Prepare 192.168.56.111:22 ... Done
  - Prepare 192.168.56.112:22 ... Done
+ Deploy TiDB instance
  - Copy pd -> 192.168.56.111 ... Done
  - Copy pd -> 192.168.56.112 ... Done
  - Copy pd -> 192.168.56.113 ... Done
  - Copy tikv -> 192.168.56.111 ... Done
  - Copy tikv -> 192.168.56.112 ... Done
  - Copy tikv -> 192.168.56.113 ... Done
  - Copy tidb -> 192.168.56.111 ... Done
  - Copy tidb -> 192.168.56.112 ... Done
  - Copy tidb -> 192.168.56.113 ... Done
  - Copy prometheus -> 192.168.56.111 ... Done
  - Copy grafana -> 192.168.56.111 ... Done
  - Copy alertmanager -> 192.168.56.111 ... Done
  - Deploy node_exporter -> 192.168.56.111 ... Done
  - Deploy node_exporter -> 192.168.56.112 ... Done
  - Deploy node_exporter -> 192.168.56.113 ... Done
  - Deploy blackbox_exporter -> 192.168.56.111 ... Done
  - Deploy blackbox_exporter -> 192.168.56.112 ... Done
  - Deploy blackbox_exporter -> 192.168.56.113 ... Done
+ Copy certificate to remote host
+ Init instance configs
  - Generate config pd -> 192.168.56.111:2379 ... Done
  - Generate config pd -> 192.168.56.112:2379 ... Done
  - Generate config pd -> 192.168.56.113:2379 ... Done
  - Generate config tikv -> 192.168.56.111:20160 ... Done
  - Generate config tikv -> 192.168.56.112:20160 ... Done
  - Generate config tikv -> 192.168.56.113:20160 ... Done
  - Generate config tidb -> 192.168.56.111:4000 ... Done
  - Generate config tidb -> 192.168.56.112:4000 ... Done
  - Generate config tidb -> 192.168.56.113:4000 ... Done
  - Generate config prometheus -> 192.168.56.111:9090 ... Done
  - Generate config grafana -> 192.168.56.111:3000 ... Done
  - Generate config alertmanager -> 192.168.56.111:9093 ... Done
+ Init monitor configs
  - Generate config node_exporter -> 192.168.56.112 ... Done
  - Generate config node_exporter -> 192.168.56.113 ... Done
  - Generate config node_exporter -> 192.168.56.111 ... Done
  - Generate config blackbox_exporter -> 192.168.56.111 ... Done
  - Generate config blackbox_exporter -> 192.168.56.112 ... Done
  - Generate config blackbox_exporter -> 192.168.56.113 ... Done
Enabling component pd
        Enabling instance 192.168.56.111:2379
        Enabling instance 192.168.56.112:2379
        Enabling instance 192.168.56.113:2379
        Enable instance 192.168.56.113:2379 success
        Enable instance 192.168.56.112:2379 success
        Enable instance 192.168.56.111:2379 success
Enabling component tikv
        Enabling instance 192.168.56.113:20160
        Enabling instance 192.168.56.111:20160
        Enabling instance 192.168.56.112:20160
        Enable instance 192.168.56.111:20160 success
        Enable instance 192.168.56.112:20160 success
        Enable instance 192.168.56.113:20160 success
Enabling component tidb
        Enabling instance 192.168.56.113:4000
        Enabling instance 192.168.56.111:4000
        Enabling instance 192.168.56.112:4000
        Enable instance 192.168.56.111:4000 success
        Enable instance 192.168.56.112:4000 success
        Enable instance 192.168.56.113:4000 success
Enabling component prometheus
        Enabling instance 192.168.56.111:9090
        Enable instance 192.168.56.111:9090 success
Enabling component grafana
        Enabling instance 192.168.56.111:3000
        Enable instance 192.168.56.111:3000 success
Enabling component alertmanager
        Enabling instance 192.168.56.111:9093
        Enable instance 192.168.56.111:9093 success
Enabling component node_exporter
        Enabling instance 192.168.56.113
        Enabling instance 192.168.56.111
        Enabling instance 192.168.56.112
        Enable 192.168.56.111 success
        Enable 192.168.56.112 success
        Enable 192.168.56.113 success
Enabling component blackbox_exporter
        Enabling instance 192.168.56.113
        Enabling instance 192.168.56.111
        Enabling instance 192.168.56.112
        Enable 192.168.56.113 success
        Enable 192.168.56.112 success
        Enable 192.168.56.111 success
Cluster `tidb-test` deployed successfully, you can start it with command: `tiup cluster start tidb-test --init`

2.3 初始化启动

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
[root@tidb1 ~]# tiup cluster start tidb-test --init
tiup is checking updates for component cluster ...
Starting component `cluster`: /root/.tiup/components/cluster/v1.14.0/tiup-cluster start tidb-test --init
Starting cluster tidb-test...
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.112
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.113
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.112
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.112
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.113
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.113
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [ Serial ] - StartCluster
Starting component pd
        Starting instance 192.168.56.113:2379
        Starting instance 192.168.56.111:2379
        Starting instance 192.168.56.112:2379
        Start instance 192.168.56.111:2379 success
        Start instance 192.168.56.113:2379 success
        Start instance 192.168.56.112:2379 success
Starting component tikv
        Starting instance 192.168.56.113:20160
        Starting instance 192.168.56.111:20160
        Starting instance 192.168.56.112:20160
        Start instance 192.168.56.113:20160 success
        Start instance 192.168.56.112:20160 success
        Start instance 192.168.56.111:20160 success
Starting component tidb
        Starting instance 192.168.56.113:4000
        Starting instance 192.168.56.112:4000
        Starting instance 192.168.56.111:4000
        Start instance 192.168.56.113:4000 success
        Start instance 192.168.56.112:4000 success
        Start instance 192.168.56.111:4000 success
Starting component prometheus
        Starting instance 192.168.56.111:9090
        Start instance 192.168.56.111:9090 success
Starting component grafana
        Starting instance 192.168.56.111:3000
        Start instance 192.168.56.111:3000 success
Starting component alertmanager
        Starting instance 192.168.56.111:9093
        Start instance 192.168.56.111:9093 success
Starting component node_exporter
        Starting instance 192.168.56.113
        Starting instance 192.168.56.111
        Starting instance 192.168.56.112
        Start 192.168.56.112 success
        Start 192.168.56.113 success
        Start 192.168.56.111 success
Starting component blackbox_exporter
        Starting instance 192.168.56.113
        Starting instance 192.168.56.111
        Starting instance 192.168.56.112
        Start 192.168.56.113 success
        Start 192.168.56.112 success
        Start 192.168.56.111 success
+ [ Serial ] - UpdateTopology: cluster=tidb-test
Started cluster `tidb-test` successfully
The root password of TiDB database has been changed.
The new password is: 'A127D6bHz+-@*Mmg50'.
Copy and record it to somewhere safe, it is only displayed once, and will not be stored.
The generated password can NOT be get and shown again.

可通过tiup cluster status tidb-test 查看集群组件的状态。

3. 遇到的问题

3.1 初始化启动tidb组件失败

如下,实际上可能出现问题有:1.配置过低,执行慢,导致超过了上限2分钟。2.集群配置yaml文件的config配置部分参数不正确。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[root@tidb1 ~]# tiup cluster start tidb-test --init
tiup is checking updates for component cluster ...
Starting component `cluster`: /root/.tiup/components/cluster/v1.14.0/tiup-cluster start tidb-test --init
Starting cluster tidb-test...
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.112
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.113
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.113
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.113
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.112
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.111
+ [Parallel] - UserSSH: user=tidb, host=192.168.56.112
+ [ Serial ] - StartCluster
Starting component pd
        Starting instance 192.168.56.113:2379
        Starting instance 192.168.56.111:2379
        Starting instance 192.168.56.112:2379
        Start instance 192.168.56.112:2379 success
        Start instance 192.168.56.113:2379 success
        Start instance 192.168.56.111:2379 success
Starting component tikv
        Starting instance 192.168.56.113:20160
        Starting instance 192.168.56.112:20160
        Starting instance 192.168.56.111:20160
        Start instance 192.168.56.113:20160 success
        Start instance 192.168.56.112:20160 success
        Start instance 192.168.56.111:20160 success
Starting component tidb
        Starting instance 192.168.56.113:4000
        Starting instance 192.168.56.112:4000
        Starting instance 192.168.56.111:4000

Error: failed to start tidb: failed to start: 192.168.56.112 tidb-4000.service, please check the instance's log(/tidb-deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s

3.2 tidb-4000/log报disk full

实际上空间是够用的,可日志中还是有disk full的WARNINGERROR但不影响安装使用。

3.3 tiup cluster check中检查出FAULT?

如果是在低配置电脑上做实验,只要是互信通了,其他的告警都可以忽略。

updatedupdated2024-01-062024-01-06