部署高可用负载均衡集群
在部署生产可用的 kubernetes 集群之前，需要先部署 LoadBalancer 环境，这里使用 keepalived + haproxy 的方式实现负载均衡和高可用。
环境说明
单独拿两个节点部署 keepalived 和 haproxy 作为后端 kubernetes 控制平面的负载均衡器，拓扑结构如下:
两个节点上面分别部署 keepalived 和 haproxy 组成负载均衡集群，haproxy 的 backend 为后端的 kubernetes control plane node，vip(虚ip) 在这两个节点之间漂移形成高可用。
部署
keepalived 的主要作用是为 haproxy 提供 vip，在2个 haproxy 实例之间提供主备，降低当其中一个haproxy失效的时对服务的影响。
部署配置 keepalived
设置相关的环境变量，根据不同的环境自行配置。
# keepalived vip 地址
export K8SHA_VIP=10.168.222.18
# keepalived auth toke
export K8SHA_KA_AUTH=412f7dc3bfed32194d1600c483e10ad1d
# keepalived network interface
export K8SHA_NETIF=eth0
设置 sysctl 选项
$ cat <<EOF >>/etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
EOF
$ sysctl -p
安装 keepalived
$ yum install -y keepalived
添加配置
$ cat <<EOF >/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check_haproxy {
script "pidof haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface $K8SHA_NETIF
virtual_router_id 51
priority 250
advert_int 1
authentication {
auth_type PASS
auth_pass $K8SHA_KA_AUTH
}
virtual_ipaddress {
$K8SHA_VIP
}
track_script {
check_haproxy
}
}
EOF
启动 keepalived
$ systemctl enable --now keepalived
$ ip addr show $K8SHA_NETIF
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:22:ff:95:87:f7 brd ff:ff:ff:ff:ff:ff
inet 10.168.222.189/24 brd 10.168.222.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.168.222.18/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::222:ffff:fe95:87f7/64 scope link
valid_lft forever preferred_lft forever
部署配置 haproxy
此处的 haproxy 为 apiserver 提供反向代理，haproxy 将所有请求轮询转发到每个master节点上。
系统配置
export K8S_MASTER0=10.168.222.218
export K8S_MASTER1=10.168.222.197
export K8S_MASTER2=10.168.222.207
安装 haproxy
$ yum install -y haproxy
配置 haproxy
$ cat <<EOF >/etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
mode tcp
bind *:6443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server master-0 $K8S_MASTER0:6443 check
server master-1 $K8S_MASTER1:6443 check
server master-2 $K8S_MASTER2:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
bind *:1080
stats auth admin:awesomePassword
stats refresh 5s
stats realm HAProxy\ Statistics
stats uri /admin?stats
EOF
启动并检测服务
$ systemctl enable haproxy.service --now
$ systemctl status haproxy.service
$ netstat -tulnp | egrep '6443|1080'
tcp 0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 10033/haproxy
tcp 0 0 0.0.0.0:1080 0.0.0.0:* LISTEN 10033/haproxy