企业级反向代理 Haproxy

← 上一章：企业级容器技术 Docker ｜ 返回总目录 ｜ 下一章：高可用集群 KEEPALIVED →

本章内容

• HAProxy 介绍和架构

• HAProxy 安装

• HAProxy 基础配置

HAProxy 调度算法

HAProxy 高级功能

1 Web 架构介绍

2 HAProxy 简介

负载均衡：Load Balance，简称LB，是一种服务或基于硬件设备等实现的高可用反向代理技术，负载均衡将特定的业务(web服务、网络流量等)分担给指定的一个或多个后端特定的服务器或设备，从而提高了公司业务的并发处理能力、保证了业务的高可用性、方便了业务后期的水平动态扩展

阿里云SLB介绍 ：https://yq.aliyun.com/articles/1803

2.1 为什么使用负载均衡

增加业务并发访问及处理能力-->解决单服务器瓶颈问题

• 节约公网IP地址-->降低IT支出成本

• 隐藏内部服务器IP-->提高内部服务器安全性

• Web服务器的动态水平扩展-->对用户无感知

负载均衡配置简单-->固定格式的配置文件

负载均衡功能丰富-->支持四层和七层，支持动态下线主机

• 负载均衡性能较强-->并发数万甚至百万

2.2 负载均衡类型

四层：

LVS：Linux Virtual Server
Nginx：1.9版之后
HAProxy：High Availability Proxy

七层：

HAProxy
Nginx

硬件：

F5          https://f5.com/zh
Netscaler   https://www.citrix.com.cn/products/citrix-adc/
Array       https://www.arraynetworks.com.cn/
深信服       http://www.sangfor.com.cn/
北京灵州     http://www.lingzhou.com.cn/cpzx/llfzjh/

2.3 应用场景

四层：Redis、Mysql、RabbitMQ、Memcached等
七层：Nginx、Tomcat、Apache、PHP、图片、动静分离、API等

随着公司业务的发展，公司负载均衡服务既有四层的，又有七层的，通过LVS实现四层和Nginx实现七层的负载均衡对机器资源消耗比较大，并且管理复杂度提升，运维总监要求，目前需要对前端负载均衡服务进行一定的优化和复用，能否用一种服务同既能实现七层负载均衡，又能实现四层负载均衡，并且性能高效，配置管理容易，而且还是开源。

在企业生产环境中，每天会有很多的需求变更，比如增加服务器、新业务上线、url路由修改、域名配置等等，对于前端负载均衡设备来说，容易维护，复杂度低，是首选指标。在企业中，稳定压倒一切，与其搞得很复杂，经常出问题，不如做的简单和稳定。在企业中，90%以上的故障，来源于需求变更。可能是程序bug，也可能是人为故障，也可能是架构设计问题等。前端负载均衡设备为重中之重，在软件选型上一定充分考虑，能满足业务的前提下，尽可能降低复杂度，提高易维护性

2.4 HAProxy介绍

HAProxy is a free, very fast and reliable solution offering high availability,
load balancing, and proxying for TCP and HTTP-based applications.

HAProxy是法国开发者 威利塔罗 (Willy Tarreau) 在2000年使用C语言开发的一个开源软件，是一款具备高并发(一万以上)、高性能的TCP和HTTP负载均衡器，支持基于cookie的持久性，自动故障切换，支持正则表达式及web状态统计

历史版本：

历史版本更新功能：1.4  1.5  1.6  1.7  1.8 1.9  2.0 2.1 2.2 2.3 2.4 2.5-dev
1.8：多线程，HTTP/2缓存……

1.7：服务器动态配置，多类型证书……

1.6：DNS解析支持，HTTP连接多路复用……

1.5：开始支持SSL，IPV6，会话保持……

注意: 生产建议下载LTS版本

从2013年HAProxy 分为 社区版和企业版 ，企业版将提供更多的特性和功能以及全天24小时的技术支持等服务。

2.4.1 企业版

企业版网站：https://www.haproxy.com/

2.4.2 社区版

社区版网站：http://www.haproxy.org/

github：https://github.com/haproxy

2.4.3 版本对比

2.4.4 HAProxy功能

支持功能：

TCP和HTTP反向代理
SSL/TSL服务器
可以针对HTTP请求添加cookie，进行路由后端服务器
可平衡负载至后端服务器，并支持持久连接
支持所有主服务器故障切换至备用服务器
支持专用端口实现监控服务
支持停止接受新连接请求，而不影响现有连接
可以在双向添加，修改或删除HTTP报文首部
响应报文压缩
支持基于pattern实现连接请求的访问控制
通过特定的URI为授权用户提供详细的状态信息

支持http反向代理
支持动态程序的反向代理
支持基于数据库的反向代理

不具备的功能：

正向代理--squid，nginx
缓存代理--varnish
web服务--nginx、tengine、apache、php、tomcat
UDP--目前不支持UDP协议
单机性能--相比LVS性能较差

3 HAProxy 安装

介绍HAProxy的基础安装及基础配置

内网IP地址划分：

#外部网段：
192.168.10.0.0/24
#内部网段：
10.0.0.0/24

3.1 Ubuntu 安装

打开链接： https://haproxy.debian.net/ ，选择合适的版本，会自动出现下面安装提示

[root@ubuntu1804 ~]#apt-get install software-properties-common
[root@ubuntu1804 ~]#add-apt-repository ppa:vbernat/haproxy-2.0
[root@ubuntu1804 ~]#apt  update
[root@ubuntu1804 ~]#apt-cache madison haproxy

#安装指定版本
[root@ubuntu1804 ~]#apt install haproxy=2.0.4-1ppa1~bionic
#或安装最新版
[root@ubuntu1804 ~]#apt-get install haproxy=2.0.\*
#验证haproxy版本
[root@ubuntu1804 ~]#haproxy  -v

范例：

[root@ubuntu1804 ~]#apt-get install software-properties-common
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  python3-software-properties
The following packages will be upgraded:
  python3-software-properties software-properties-common
2 upgraded, 0 newly installed, 0 to remove and 252 not upgraded.
Need to get33.6 kB of archives.
After this operation, 13.3 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 software-
properties-common all 0.96.24.32.12 [10.0 kB]
Get:2 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 python3-
software-properties all 0.96.24.32.12 [23.6 kB]
Fetched 33.6 kB in 5s (6,528 B/s)
(Reading database ... 71030 files and directories currently installed.)
Preparing to unpack .../software-properties-common_0.96.24.32.12_all.deb ...
Unpacking software-properties-common (0.96.24.32.12) over (0.96.24.32.4) ...
Preparing to unpack .../python3-software-properties_0.96.24.32.12_all.deb ...
Unpacking python3-software-properties (0.96.24.32.12) over (0.96.24.32.4) ...
Processing triggers for man-db (2.8.3-2) ...

Setting up python3-software-properties (0.96.24.32.12) ...
Processing triggers for dbus (1.12.2-1ubuntu1) ...
Setting up software-properties-common (0.96.24.32.12) ...

[root@ubuntu1804 ~]#add-apt-repository ppa:vbernat/haproxy-2.0
 HAProxy is a free, very fast and reliable solution offering high availability,
load balancing, and proxying for TCP and HTTP-based applications. It is
particularly suited for web sites crawling under very high loads while needing
persistence or Layer7 processing. Supporting tens of thousands of connections is
clearly realistic with todays hardware. Its mode of operation makes its
integration into existing architectures very easy and riskless, while still
offering the possibility not to expose fragile web servers to the Net.
This PPA contains packages for HAProxy 2.0.
 More info: https://launchpad.net/~vbernat/+archive/ubuntu/haproxy-2.0
Press [ENTER] to continue or Ctrl-c to cancel adding it.
Hit:1 http://mirrors.aliyun.com/ubuntu bionic InRelease
Hit:2 http://mirrors.aliyun.com/ubuntu bionic-security InRelease
Hit:3 http://mirrors.aliyun.com/ubuntu bionic-updates InRelease
Hit:4 http://mirrors.aliyun.com/ubuntu bionic-proposed InRelease
Hit:5 http://mirrors.aliyun.com/ubuntu bionic-backports InRelease
Get:6 http://ppa.launchpad.net/vbernat/haproxy-2.0/ubuntu bionic InRelease [20.7
kB]
Get:7 http://ppa.launchpad.net/vbernat/haproxy-2.0/ubuntu bionic/main i386
Packages [984 B]
Get:8 http://ppa.launchpad.net/vbernat/haproxy-2.0/ubuntu bionic/main amd64
Packages [988 B]
Get:9 http://ppa.launchpad.net/vbernat/haproxy-2.0/ubuntu bionic/main
Translation-en [704 B]
Fetched 23.4 kB in 6s (3,971 B/s)
Reading package lists... Done

[root@ubuntu1804 ~]# apt-get update
Hit:1 http://mirrors.aliyun.com/ubuntu bionic InRelease
Hit:2 http://mirrors.aliyun.com/ubuntu bionic-security InRelease
Hit:3 http://mirrors.aliyun.com/ubuntu bionic-updates InRelease
Hit:4 http://mirrors.aliyun.com/ubuntu bionic-proposed InRelease
Hit:5 http://mirrors.aliyun.com/ubuntu bionic-backports InRelease
Hit:6 http://ppa.launchpad.net/vbernat/haproxy-2.0/ubuntu bionic InRelease
Reading package lists... Done

[root@ubuntu1804 ~]#apt-get install haproxy=2.0.\*
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '2.0.13-1ppa1~bionic' (HAProxy 2.0:18.04/bionic [amd64]) for
'haproxy'
The following additional packages will be installed:
  liblua5.3-0 libpcre2-8-0
Suggested packages:
  vim-haproxy haproxy-doc

The following NEW packages will be installed:
  haproxy liblua5.3-0 libpcre2-8-0
0 upgraded, 3 newly installed, 0 to remove and 252 not upgraded.
Need to get1,526 kB/1,819 kB of archives.
After this operation, 4,246 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ppa.launchpad.net/vbernat/haproxy-2.0/ubuntu bionic/main amd64
haproxy amd64 2.0.13-1ppa1~bionic [1,526 kB]
Fetched 572 kB in 26s (22.1 kB/s)
Selecting previously unselected package liblua5.3-0:amd64.
(Reading database ... 71032 files and directories currently installed.)
Preparing to unpack .../liblua5.3-0_5.3.3-1ubuntu0.18.04.1_amd64.deb ...
Unpacking liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ...
Selecting previously unselected package libpcre2-8-0:amd64.
Preparing to unpack .../libpcre2-8-0_10.31-2_amd64.deb ...
Unpacking libpcre2-8-0:amd64 (10.31-2) ...
Selecting previously unselected package haproxy.
Preparing to unpack .../haproxy_2.0.13-1ppa1~bionic_amd64.deb ...
Unpacking haproxy (2.0.13-1ppa1~bionic) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.3) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up libpcre2-8-0:amd64 (10.31-2) ...
Setting up liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ...
Processing triggers for rsyslog (8.32.0-1ubuntu4) ...
Setting up haproxy (2.0.13-1ppa1~bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service →
/lib/systemd/system/haproxy.service.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for systemd (237-3ubuntu10.3) ...
Processing triggers for rsyslog (8.32.0-1ubuntu4) ...
[root@ubuntu1804 ~]#apt-cache madison haproxy
   haproxy | 2.0.13-1ppa1~bionic | http://ppa.launchpad.net/vbernat/haproxy-
2.0/ubuntu bionic/main amd64 Packages
   haproxy | 1.8.8-1ubuntu0.9 | http://mirrors.aliyun.com/ubuntu bionic-
security/main amd64 Packages
   haproxy | 1.8.8-1ubuntu0.9 | http://mirrors.aliyun.com/ubuntu bionic-
updates/main amd64 Packages
   haproxy |    1.8.8-1 | http://mirrors.aliyun.com/ubuntu bionic/main amd64
Packages
   haproxy |    1.8.8-1 | http://mirrors.aliyun.com/ubuntu bionic/main Sources
   haproxy | 1.8.8-1ubuntu0.9 | http://mirrors.aliyun.com/ubuntu bionic-
security/main Sources
   haproxy | 1.8.8-1ubuntu0.9 | http://mirrors.aliyun.com/ubuntu bionic-
updates/main Sources
[root@ubuntu1804 ~]#haproxy  -v
HA-Proxy version 2.0.13-1ppa1~bionic 2020/02/15 - https://haproxy.org/

3.2 RHEL系统安装

在 RHEL/centos 系统上通过 yum、编译等多种安装方式

3.2.1 系统默认yum源

CentOS 7 的默认的base仓库中包含haproxy的安装包文件，但是版本比较旧，是1.5.18的版本，距离当前版本已经有较长时间没有更新，由于版本比较旧所以有很多功能不支持，如果对功能和性能没有要求可以使用此版本，否则推荐使用新版本。

范例：Rocky8

范例：CentOS 7 安装haproxy

[root@centos7 ~]# yum install haproxy -y

#验证haproxy版本
[root@centos7 ~]# haproxy -v
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

范例：CentOS 8 安装haproxy

[root@centos8 ~]#dnf -y install haproxy
[root@centos8 ~]#haproxy -v
HA-Proxy version 1.8.15 2018/12/13
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>
[root@centos8 ~]#haproxy -vv
HA-Proxy version 1.8.15 2018/12/13
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      =gcc
  CFLAGS  =-O2-g-fno-strict-aliasing-Wdeclaration-after-statement-fwrapv-
Wno-format-truncation-Wno-null-dereference-Wno-unused-label
  OPTIONS =USE_LINUX_TPROXY=1USE_CRYPT_H=1USE_GETADDRINFO=1USE_ZLIB=1
USE_REGPARM=1USE_OPENSSL=1USE_LUA=1USE_SYSTEMD=1USE_PCRE=1

Default settings :
  maxconn =2000, bufsize =16384, maxrewrite =1024, maxpollevents =200
Built with OpenSSL version : OpenSSL 1.1.1 FIPS  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1c FIPS  28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
    [SPOE] spoe
    [COMP] compression
    [TRACE] trace

3.2.2 第三方安装包

官方没有提供rpm相关的包,可以通过第三方仓库的rpm包

从第三方网站下载rpm包：https://pkgs.org/download/haproxy

范例：基于互联网第三方仓库在线安装

#wget http://www.nosuchhost.net/~cheese/fedora/packages/epel-7/x86_64/cheese-
release-7-1.noarch.rpm
#rpm -ivh cheese-release-7-1.noarch.rpm
#yum install haproxy
#验证haproxy版本
# haproxy  -v
HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

范例：利用第三方 yum 仓库安装

[root@centos7 ~]#wget https://centos7.iuscommunity.org/ius-release.rpm
[root@centos7 ~]#rpm -Uvh ius-release*rpm
[root@centos7 ~]#yum -y install epel-release
[root@centos7 ~]#rpm -Uvh ius-release*rpm
[root@centos7 ~]#yum install haproxy18u
[root@centos7 ~]#haproxy  -v
HA-Proxy version 1.8.23 2019/11/25
Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>

范例：下载rpm包离线安装

#下载安装lua库对应的版本
[root@centos7 ~]#wget
https://dl.iuscommunity.org/pub/ius/stable/CentOS/7/x86_64/lua53u-libs-5.3.4-
1.ius.centos7.x86_64.rpm

#安装lua库
[root@centos7 ~]#yum -y install lua53u-libs-5.3.4-1.ius.centos7.x86_64.rpm
#下载haproxy
[root@centos7 ~]#wget
https://dl.iuscommunity.org/pub/ius/stable/CentOS/7/x86_64/haproxy18u-1.8.20-
1.el7.ius.x86_64.rpm

#安装haproxy
[root@centos7 ~]#yum -y install haproxy18u-1.8.20-1.el7.ius.x86_64.rpm
[root@centos7 ~]#haproxy  -v
HA-Proxy version 1.8.20 2019/04/25
Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>
[root@centos7 ~]#systemctl start haproxy
[root@centos7 ~]#systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor
preset: disabled)

   Active: active (running) since Mon 2020-03-3021:17:23 CST; 56s ago

  Process: 1257ExecStartPre=/usr/sbin/haproxy -f$CONFIG-c-q(code=exited,
status=0/SUCCESS)
 Main PID: 1258 (haproxy)
   CGroup: /system.slice/haproxy.service
           ├─1258 /usr/sbin/haproxy -Ws-f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid
           └─1260 /usr/sbin/haproxy -Ws-f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid
Mar 3021:17:23 centos7.wangxiaochun.com systemd[1]: Starting HAProxy Load
Balancer...
Mar 3021:17:23 centos7.wangxiaochun.com systemd[1]: Started HAProxy Load
Balancer.

3.3 编译安装 HAProxy

编译安装HAProxy 2.0 LTS版本，更多源码包下载地址：http://www.haproxy.org/download/

3.3.1 解决 lua 环境

HAProxy 支持基于lua实现功能扩展，lua是一种小巧的脚本语言，于1993年由巴西里约热内卢天主教大学（Pontifical Catholic University of Rio de Janeiro）里的一个研究小组开发，其设计目的是为了嵌入应用程序中，从而为应用程序提供灵活的扩展和定制功能。

Lua 官网：www.lua.org

Lua 应用场景

游戏开发

• 独立应用脚本 Web 应用脚本

扩展和数据库插件，如MySQL Proxy

安全系统，如入侵检测系统

3.3.1.1 CentOS 基础环境

参考链接：http://www.lua.org/start.html

由于CentOS7 之前版本自带的lua版本比较低并不符合HAProxy要求的lua最低版本(5.3)的要求，因此需要编译安装较新版本的lua环境，然后才能编译安装HAProxy，过程如下：

#当前系统版本
[root@centos7 ~]#lua -v
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
#安装基础命令及编译依赖环境
[root@centos7 ~]# yum install gcc readline-devel
[root@centos7 ~]# wget http://www.lua.org/ftp/lua-5.3.5.tar.gz
[root@centos7 ~]# tar xvf  lua-5.3.5.tar.gz -C /usr/local/src
[root@centos7 ~]# cd /usr/local/src/lua-5.3.5
[root@centos7 lua-5.3.5]# make linux test

#查看编译安装的版本
[root@centos7 lua-5.3.5]#src/lua -v
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio

3.3.1.2 Ubuntu 基础环境

#安装基础命令及编译依赖环境
[root@ubuntu1804 ~]# apt -y install gcc make libssl-dev libpcre3 libpcre3-dev
zlib1g-dev libreadline-dev libsystemd-dev

[root@ubuntu1804 ~]# cd /usr/local/src
[root@ubuntu1804 ~]# wget http://www.lua.org/ftp/lua-5.3.5.tar.gz
[root@ubuntu1804 ~]# tar xvf  lua-5.3.5.tar.gz
[root@ubuntu1804 ~]# cd lua-5.3.5
[root@ubuntu1804 ~]# make linux test
[root@ubuntu1804 ~]#pwd
/usr/local/src/lua-5.3.5
[root@ubuntu1804 ~]# ./src/lua -v
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio

#或安装系统自带的lua
[root@ubuntu1804 ~]#apt install  lua5.3=5.3.3-1ubuntu0.18.04.1
[root@ubuntu1804 ~]#lua5.3  -v
Lua 5.3.3  Copyright (C) 1994-2016 Lua.org, PUC-Rio

3.3.2 编译安装HAProxy

#HAProxy  1.8及1.9版本编译参数：
makeARCH=x86_64 TARGET=linux2628 USE_PCRE=1USE_OPENSSL=1USE_ZLIB=1
USE_SYSTEMD=1USE_CPU_AFFINITY=1PREFIX=/usr/local/haproxy

#HAProxy 2.0以上版本编译参数：
[root@centos7 ~]#yum -y install gcc openssl-devel pcre-devel systemd-devel
[root@centos7 ~]#tar xvf haproxy-2.1.3.tar.gz  -C /usr/local/src
[root@centos7 ~]#cd /usr/local/src/haproxy-2.1.3/

#查看安装方法
[root@centos7 haproxy-2.1.3]#ll Makefile
-rw-rw-r--1 root root 40812 Feb 1223:18 Makefile

[root@centos7 haproxy-2.1.3]#cat README

[root@centos7 haproxy-2.1.3]#cat INSTALL

# 参考 INSTALL 文件进行编译安装

[root@centos7 haproxy-2.1.3]#make  ARCH=x86_64 TARGET=linux-glibc  USE_PCRE=1
USE_OPENSSL=1 USE_ZLIB=1  USE_SYSTEMD=1  USE_LUA=1 LUA_INC=/usr/local/src/lua-
5.3.5/src/  LUA_LIB=/usr/local/src/lua-5.3.5/src/
[root@centos7 haproxy-2.1.3]# make install PREFIX=/apps/haproxy
[root@centos7 haproxy-2.1.3]#ln -s /apps/haproxy/sbin/haproxy /usr/sbin/

# 查看生成的文件

[root@centos7 haproxy-2.1.3]#tree /apps/haproxy/
/apps/haproxy/
├── doc
│   └── haproxy
│       ├── 51Degrees-device-detection.txt
│       ├── architecture.txt
│       ├── close-options.txt
│       ├── configuration.txt
│       ├── cookie-options.txt
│       ├── DeviceAtlas-device-detection.txt
│       ├── intro.txt
│       ├── linux-syn-cookies.txt
│       ├── lua.txt
│       ├── management.txt
│       ├── netscaler-client-ip-insertion-protocol.txt
│       ├── network-namespaces.txt
│       ├── peers.txt
│       ├── peers-v2.0.txt
│       ├── proxy-protocol.txt
│       ├── regression-testing.txt
│       ├── seamless_reload.txt
│       ├── SOCKS4.protocol.txt
│       ├── SPOE.txt
│       └── WURFL-device-detection.txt
├── sbin
│   └── haproxy
└── share
    └── man
        └── man1
            └── haproxy.1
6 directories, 22 files

3.3.3 验证HAProxy版本

#验证HAProxy版本：
[root@centos7 ~]#which haproxy
/usr/sbin/haproxy
[root@centos7 ~]#haproxy  -v
HA-Proxy version 2.1.3 2020/02/12 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2021.
Known bugs: http://www.haproxy.org/bugs/bugs-2.1.3.html

[root@centos7 ~]#haproxy  -v
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/

#大写-V选项显示版本和帮助用法
[root@centos7 ~]#haproxy  -V
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/
Usage : haproxy [-f <cfgfile|cfgdir>]* [ -vdVD ] [ -n <maxconn> ] [ -N
<maxpconn> ]
        [ -p <pidfile> ] [ -m <max megs> ] [ -C <dir> ] [-- <cfgfile>*]
-v displays version ; -vv shows known build options.

• -d enters debug mode ; -db only disables background mode.

• -dM[<byte>] poisons memory with <byte> (defaults to 0x50)

• -V enters verbose mode (disables quiet mode)

• -D goes daemon ; -C changes to <dir> before loading files.

• -W master-worker mode.

• -Ws master-worker mode with systemd notify support.

• -q quiet mode : don't display messages

• -c check mode : only check config files and exit

-n sets the maximum total # of connections (uses ulimit -n)
-m limits the usable amount of memory (in MB)
-N sets the default, per-proxy maximum # of connections (0)
-Lset local peer name (default to hostname)

• -p writes pids of all children to this file

-de disables epoll() usage even when available

-dp disables poll() usage even when available

-dS disables splice usage (broken on old kernels)
-dG disables getaddrinfo() usage
-dR disables SO_REUSEPORT usage

-dr ignores server address resolution failures

-dV disables SSL verify on servers side

-sf/-st [pid ]* finishes/terminates old pids.

-x <unix_socket> get listening sockets from a unix socket

• -S <bind>[,<bind options>...] new master CLI

[root@centos7 ~]#haproxy  -vv
HA-Proxy version 2.1.3 2020/02/12 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2021.
Known bugs: http://www.haproxy.org/bugs/bugs-2.1.3.html
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      =gcc
  CFLAGS  =-m64-march=x86-64 -O2-g-fno-strict-aliasing-Wdeclaration-after-
statement-fwrapv-Wno-unused-label-Wno-sign-compare-Wno-unused-parameter-
Wno-old-style-declaration-Wno-ignored-qualifiers-Wno-clobbered-Wno-missing-
field-initializers-Wtype-limits
  OPTIONS =USE_PCRE=1USE_OPENSSL=1USE_LUA=1USE_ZLIB=1USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE-MY_EPOLL-MY_SPLICE+NETFILTER +PCRE -PCRE_JIT-
PCRE2-PCRE2_JIT+POLL -PRIVATE_CACHE+THREAD -PTHREAD_PSHARED-REGPARM-
STATIC_PCRE-STATIC_PCRE2+TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H
-VSYSCALL+GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4+ZLIB -SLZ
+CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS-51DEGREES-WURFL+SYSTEMD -
OBSOLETE_LINKER+PRCTL +THREAD_DUMP -EVPORTS

Default settings :

  bufsize =16384, maxrewrite =1024, maxpollevents =200

Built with multi-threading support (MAX_THREADS=64, default=4).

Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.5
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
       <default> : mode=TCP        side=FE|BE     mux=PASS
Available services : none
Available filters :
    [SPOE] spoe
    [CACHE] cache
    [FCGI] fcgi-app
    [TRACE] trace
    [COMP] compression

3.3.4 准备HAProxy启动文件

#默认缺少配置文件，无法启动
[root@centos7 ~]#systemctl daemon-reload
[root@centos7 ~]#systemctl start haproxy
Job for haproxy.service failed because the control process exited with error
code. See "systemctl status haproxy.service" and "journalctl -xe"for details.

[root@centos7 ~]#tail /var/log/messages
Mar 3022:28:50 centos7 systemd: haproxy.service: control process exited,
code=exited status=1
Mar 3022:28:50 centos7 systemd: Failed to start HAProxy Load Balancer.
Mar 3022:28:50 centos7 systemd: Unit haproxy.service entered failed state.
Mar 3022:28:50 centos7 systemd: haproxy.service failed.
Mar 3022:28:54 centos7 systemd: Starting HAProxy Load Balancer...

Mar 3022:28:54 centos7 haproxy: [ALERT] 089/222854 (28223) : Cannot open
configuration file/directory /etc/haproxy/haproxy.cfg : No such file or
directory
Mar 3022:28:54 centos7 systemd: haproxy.service: control process exited,
code=exited status=1
Mar 3022:28:54 centos7 systemd: Failed to start HAProxy Load Balancer.
Mar 3022:28:54 centos7 systemd: Unit haproxy.service entered failed state.
Mar 3022:28:54 centos7 systemd: haproxy.service failed.

#创建service文件
[root@centos7 ~]#vim  /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg  -c-q
ExecStart=/usr/sbin/haproxy -Ws-f /etc/haproxy/haproxy.cfg -p
/var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2$MAINPID
LimitNOFILE=100000

[Install]
WantedBy=multi-user.target

3.3.5 配置文件

#查看配置文件范例
[root@centos7 ~]#tree /usr/local/src/haproxy-2.1.3/examples/
/usr/local/src/haproxy-2.1.3/examples/
├── acl-content-sw.cfg
├── content-sw-sample.cfg
├── errorfiles
│   ├── 400.http
│   ├── 403.http
│   ├── 408.http
│   ├── 500.http
│   ├── 502.http
│   ├── 503.http
│   ├── 504.http
│   └── README
├── haproxy.init
├── option-http_proxy.cfg
├── socks4.cfg
├── transparent_proxy.cfg
└── wurfl-example.cfg
1 directory, 15 files

#创建自定义的配置文件
[root@centos7 ~]# mkdir  /etc/haproxy
[root@centos7 ~]# vim /etc/haproxy/haproxy.cfg
global
    maxconn 100000
chroot /apps/haproxy
    stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
#uid 99

#gid 99
    user  haproxy
    group haproxy
    daemon
#nbproc 4
#cpu-map 1 0
#cpu-map 2 1
#cpu-map 3 2
#cpu-map 4 3
    pidfile /var/lib/haproxy/haproxy.pid
    log 127.0.0.1 local2 info

defaults
    option http-keep-alive
    option  forwardfor
    maxconn 100000
    mode http
    timeout connect 300000ms
    timeout client  300000ms
    timeout server  300000ms
listen stats
    mode http
    bind 0.0.0.0:9999
    stats enable
    log global
    stats uri     /haproxy-status
    stats auth    haadmin:123456
listen  web_port
    bind 10.0.0.7:80
    mode http
    log global
    server web1  127.0.0.1:8080  check inter 3000 fall 2 rise 5

#检查配置文件语法
[root@centos7 ~]#haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid

3.3.6 启动 haproxy

#准备socket文件目录
[root@centos7 ~]# mkdir  /var/lib/haproxy

#设置用户和目录权限
[root@centos7 ~]# useradd -r -s /sbin/nologin -d /var/lib/haproxy haproxy
[root@centos7 ~]# systemctl  enable --now haproxy

3.3.7 验证 haproxy 状态

haproxy.cfg文件中定义了chroot、pidfile、user、group等参数，如果系统没有相应的资源会导致 haproxy无法启动，具体参考日志文件 /var/log/messages

[root@centos7 ~]#systemctl status haproxy
● haproxy.service - HAProxy Load Balancer

   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor
preset: disabled)
   Active: active (running) since Mon 2020-03-3020:45:20 CST; 4min 45s ago
  Process: 1362ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c-q
(code=exited, status=0/SUCCESS)
 Main PID: 1363 (haproxy)
   CGroup: /system.slice/haproxy.service
           ├─1363 /usr/sbin/haproxy -Ws-f /etc/haproxy/haproxy.cfg -p
/var/lib/haproxy/haproxy.pid
           └─1366 /usr/sbin/haproxy -Ws-f /etc/haproxy/haproxy.cfg -p
/var/lib/haproxy/haproxy.pid

Mar 3020:45:20 centos7.wangxiaochun.com systemd[1]: Starting HAProxy Load
Balancer...
Mar 3020:45:20 centos7.wangxiaochun.com systemd[1]: Started HAProxy Load
Balancer.
Mar 3020:45:20 centos7.wangxiaochun.com haproxy[1363]: [NOTICE] 089/204520
(1363) : New worker #1 (1366...ked
Mar 3020:45:20 centos7.wangxiaochun.com haproxy[1363]: [WARNING] 089/204520
(1366) : Server web_port/we...ue.
Mar 3020:45:20 centos7.wangxiaochun.com haproxy[1363]: [ALERT] 089/204520
(1366) : proxy 'web_port' has...le!
Hint: Some lines were ellipsized, use -l to show in full.
[root@centos7 ~]# pstree -p |grep haproxy
           |-haproxy(28101)---haproxy(28105)

3.3.8 查看haproxy的状态页面

浏览器访问： http://haproxy - server:9999/haproxy - status

3.4 基于 Docker 部署

Haproxy 官方镜像

https://hub.docker.com/_/haproxy/

范例: 基于 Docker 安装 Haproxy

mkdir /data/haproxy/ -p
cat  > /data/haproxy/haproxy.cfg
global
maxconn 100000
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
uid 99
gid 99
daemon
pidfile /var/lib/haproxy/haproxy.pid
log 127.0.0.1 local3 info
defaults
option http-keep-alive
option forwardfor
maxconn 100000
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000ms
listen stats
    mode http
    bind 0.0.0.0:9999
    stats enable
    log global
    stats uri /haproxy-status
    stats auth admin:123456

docker run -d--name my-running-haproxy -v
/data/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro -p9999:9999  --
sysctl net.ipv4.ip_unprivileged_port_start=0  haproxy:2.6.0-alpine3.16

docker run -d--name myhaproxy -v
/data/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro -p9999:9999  --
sysctl net.ipv4.ip_unprivileged_port_start=0  haproxy:2.6.6-alpine3.16

4 HAProxy 基础配置

配置文件官方帮助文档

官方文档：

http://cbonte.github.io/haproxy-dconv/
http://cbonte.github.io/haproxy-dconv/2.1/configuration.html

HAProxy 的配置文件haproxy.cfg由两大部分组成，分别是global和proxies部分

global：全局配置段

进程及安全配置相关的参数
性能调整相关参数
Debug参数

proxies：代理配置段

defaults：为frontend, backend, listen提供默认配置
frontend：前端，相当于nginx中的server {}
backend：后端，相当于nginx中的upstream {}
listen：同时拥有前端和后端配置,配置简单,生产推荐使用

4.1 Global配置

4.1.1 Global 配置参数说明

官方文档：http://cbonte.github.io/haproxy-dconv/2.4/configuration.html#3

chroot#锁定运行目录
deamon #以守护进程运行
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin process 1
#socket文件,并可以通过此文件管理
user, group, uid, gid  #运行haproxy的用户身份

#nbproc    n        #开启的haproxy worker 进程数，默认进程数是一个, nbproc从HAProxy
2.5开始不再支持

nbthread  1#和多进程 nbproc配置互斥（版本有关,CentOS8的haproxy1.8无此问题）,指定每个
haproxy进程开启的线程数，默认为每个进程一个线程
#如果同时启用nbproc和nbthread 会出现以下日志的错误，无法启动服务
Apr  714:46:23 haproxy haproxy: [ALERT] 097/144623 (1454) : config : cannot
enable multiple processes if multiple threads are configured. Please use either
nbproc or nbthread but not both.

#cpu-map 1 0    #绑定haproxy worker 进程至指定CPU，将第1个worker进程绑定至0号CPU
#cpu-map 2 1     #绑定haproxy worker 进程至指定CPU，将第2个worker进程绑定至1号CPU

cpu-map auto:1/1-8 0-7#haproxy2.4中启用nbthreads，在global配置中添加此选项，可以进行线
程和CPU的绑定，nbproc选项2.5版本中将会删除,每个进程中1-8个线程分别绑定0-7号CPU

maxconn  n      #每个haproxy进程的最大并发连接数
maxsslconn  n   #每个haproxy进程ssl最大连接数,用于haproxy配置了证书的场景下
maxconnrate n   #每个进程每秒创建的最大连接数量
spread-checks n #后端server状态check随机提前或延迟百分比时间，建议2-5(20%-50%)之间，默认
值0
pidfile         #指定pid文件路径
log 127.0.0.1  local2 info #定义全局的syslog服务器；日志服务器需要开启UDP协议，最多可以定
义两个

4.1.2 多进程和线程

从2.5以上新版中用多线程代替多worker进程

范例：多进程和socket文件

[root@haproxy ~]#vim /etc/haproxy/haproxy.cfg
global
maxconn 100000
chroot /apps/haproxy
stats socket /var/lib/haproxy/haproxy1.sock mode 600 level admin process 1
stats socket /var/lib/haproxy/haproxy2.sock mode 600 level admin process 2
stats socket /var/lib/haproxy/haproxy3.sock mode 600 level admin process 3
stats socket /var/lib/haproxy/haproxy4.sock mode 600 level admin process 4
uid 99
gid 99
daemon
nbproc 4
[root@haproxy ~]##systemctl restart haproxy
[root@haproxy ~]#pstree -p |grep haproxy
           |-haproxy(1933)-+-haproxy(1935)
           |               |-haproxy(1936)
           |               |-haproxy(1937)
           |               `-haproxy(1938)
[root@haproxy ~]#ll /var/lib/haproxy/
total 12
drwxr-xr-x  2 root root 4096 Sep  410:02 ./
drwxr-xr-x 38 root root 4096 Sep  222:37 ../
srw-------  1 root root    0 Sep  410:02 haproxy1.sock=
srw-------  1 root root    0 Sep  410:02 haproxy2.sock=
srw-------  1 root root    0 Sep  410:02 haproxy3.sock=
srw-------  1 root root    0 Sep  410:02 haproxy4.sock=

[root@haproxy ~]#ps axo pid,cmd,psr|grep haproxy
1933 /usr/sbin/haproxy -Ws-f /e   5
1935 /usr/sbin/haproxy -Ws-f /e   0
1936 /usr/sbin/haproxy -Ws-f /e   1
1937 /usr/sbin/haproxy -Ws-f /e   2
1938 /usr/sbin/haproxy -Ws-f /e   3
1965grep--color=auto haproxy     1

范例: 多线程和CPU绑定

#范例1
[root@haproxy ~]#vim /etc/haproxy/haproxy.cfg
global
    cpu-map auto:1/1-8 0-7
......
#语法检查
[root@haproxy ~]#haproxy  -c  -f  /etc/haproxy/haproxy.cfg
[root@haproxy ~]#systemctl restart haproxy.service

[root@haproxy ~]#pidstat -p 915  -t
Linux 4.15.0-112-generic (haproxy.wang.org)     Saturday, September 04, 2021
_x86_64_    (8 CPU)

07:46:56 CST   UID TGID   TID    %usr %system  %guest   %wait    %CPU   CPU
Command
07:46:56 CST   999915-0.00    0.00    0.00    0.00    0.00     0
haproxy
07:46:56 CST   999-9150.00    0.00    0.00    0.00    0.00     0
|__haproxy
07:46:56 CST   999-9160.00    0.00    0.00    0.00    0.00     1
|__haproxy
07:46:56 CST   999-9170.00    0.00    0.00    0.00    0.00     2
|__haproxy
07:46:56 CST   999-9180.00    0.00    0.00    0.00    0.00     3
|__haproxy
07:46:56 CST   999-9190.00    0.00    0.00    0.00    0.00     4
|__haproxy
07:46:56 CST   999-9200.00    0.00    0.00    0.00    0.00     5
|__haproxy
07:46:56 CST   999-9210.00    0.00    0.00    0.00    0.00     6
|__haproxy
07:46:56 CST   999-9220.00    0.00    0.00    0.00    0.00     7
|__haproxy

#范例2

[root@haproxy ~]#vim /etc/haproxy/haproxy.cfg
cpu-map auto:1/1-4 0-3

[root@haproxy ~]#ps  axo pid,cmd,psr -L  |grep haproxy
1437 /usr/sbin/haproxy -Ws-f /e   4

1439 /usr/sbin/haproxy -Ws-f /e   0
1439 /usr/sbin/haproxy -Ws-f /e   1
1439 /usr/sbin/haproxy -Ws-f /e   2
1439 /usr/sbin/haproxy -Ws-f /e   3

4.1.3 HAProxy日志配置

HAproxy本身不记录客户端的访问日志.此外为减少服务器负载,一般生产中HAProxy不记录日志. 也可以配置HAProxy利用rsyslog服务记录日志到指定日志文件中

4.1.3.1 HAProxy配置

#在global配置项定义：
log 127.0.0.1  local{1-7} info #基于syslog记录日志到指定设备，级别有(err、warning、
info、debug)

listen  web_port
  bind 127.0.0.1:80
  mode http
  log global        #开启当前web_port的日志功能，默认不记录日志
  server web1  127.0.0.1:8080  check inter 3000 fall 2 rise 5

# systemctl  restart haproxy

4.1.3.2 Rsyslog配置

vim /etc/rsyslog.conf
$ModLoad imudp
$UDPServerRun514
......
local3.*    /var/log/haproxy.log
......
# systemctl  restart rsyslog

4.1.3.3 验证HAProxy日志

重启syslog服务并访问app页面，然后验证是否生成日志

# tail -f /var/log/haproxy.log
Aug 1420:21:06 localhost haproxy[18253]: Connect from 192.168.0.1:3050 to
10.0.0.7:80 (web_host/HTTP)
Aug 1420:21:06 localhost haproxy[18253]: Connect from 192.168.0.1:3051 to
10.0.0.7:80 (web_host/HTTP)
Aug 1420:21:06 localhost haproxy[18253]: Connect from 192.168.0.1:3050 to
10.0.0.7:80 (web_host/HTTP)

4.1.3.4 实战案例：启动本地和远程日志

[root@haproxy ~]#vim /etc/haproxy/haproxy.cfg
log 127.0.0.1 local2 info
log 10.0.0.8 local2 info
[root@centos7 ~]#systemctl restart haproxy

#开启本地日志
[root@haproxy ~]#vim /etc/rsyslog.conf
$ModLoad imudp
$UDPServerRun514
......
local2.*                                        /var/log/haproxy.log
[root@haproxy ~]#systemctl restart rsyslog

#在syslog日志服务器开启远程主机日志
[root@centos8 ~]#vim /etc/rsyslog.conf
module(load="imudp") # needs to be done just once
input(type="imudp"port="514")
local2.*                   /var/log/haproxy.log
[root@centos8 ~]#systemctl restart rsyslog
#浏览器访问：http://haproxy-server:9999/haproxy-status,观察本机和远程主机生成的日志
[root@haproxy ~]#tail /var/log/haproxy.log
[root@haproxy ~]#cat /var/log/haproxy.log
Mar 3023:42:52 localhost haproxy[28643]: Connect from 10.0.0.1:7820 to
10.0.0.7:9999 (stats/HTTP)
Mar 3023:42:52 localhost haproxy[28643]: Connect from 10.0.0.1:7821 to
10.0.0.7:9999 (stats/HTTP)
Mar 3023:42:53 localhost haproxy[28643]: Connect from 10.0.0.1:7822 to
10.0.0.7:9999 (stats/HTTP)

[root@centos8 ~]#tail /var/log/haproxy.log
Mar 3023:42:53 10.0.0.7 haproxy[28643]: Connect from 10.0.0.1:7824 to
10.0.0.7:9999 (stats/HTTP)
Mar 3023:42:53 10.0.0.7 haproxy[28643]: Connect from 10.0.0.1:7825 to
10.0.0.7:9999 (stats/HTTP)
Mar 3023:42:53 10.0.0.7 haproxy[28643]: Connect from 10.0.0.1:7826 to
10.0.0.7:9999 (stats/HTTP)
Mar 3023:42:53 10.0.0.7 haproxy[28643]: Connect from 10.0.0.1:7827 to
10.0.0.7:9999 (stats/HTTP)