FastDFS集群搭建+keepalive+nginx高可用

这里以我搭建好的FastDFS集群高可用+负载均衡的虚拟机为例，下载地址：集群节点下载

一、环境准备

centOS-6.6，VMware

8个节点，node22，node23，node25~node30；其中node22，node23作为keepalived+nginx高可用负载均衡，具体思路看框架原理图。node25，node26作为tracker，node27~node30作为storage【node27，node28作为一个组，node29，node30作为一个组，文件上传到组中一个节点后，会根据内部算法同步到同组中的其他节点上】

框架结构图

原理：用keeplived模拟一个VIP，通过这个VIP请求负载均衡过的nginx的其中一个节点，再利用nginx做后端两个tracker的负载均衡上的nginx，最后通过tracker上的nginx访问其中的一组storage

fastDFS说明：

同一个组（Volume）中的节点是相互备份的，同一个组中的存储容量是以最小的那个节点的存储容量的大小来衡量的，同一个组中各个节点的存储数据是相同的，所以若是一个组中的节点数量多，那么该组的读操作能力就越强，类似该组内部实现了一个负载均衡，可以提高访问性能，一个集群中，不同的组起到的作用就是横向扩容的作用，每个组中的存储内容不一样，所以扩容增加组或者增加组中的节点的容量即可

资料：

相关安装包下载

二、安装

第一步：所有节点安装libfastcommon

将安装包分别发送到这6个节点

编译和安装所需的依赖包、环境，利用xshell的to all sessions操作，每个节点都安装环境

yum install make cmake gcc gcc-c++

暂时将所有的资料都放在root目录下，接下来所有节点安装libfastcommon

tar -zxvf libfastcommon-1.0.7.tar.gz

cd libfastcommon-1.0.7

./make.sh

./make.sh install

libfastcommon 默认安装到了 /usr/lib64/libfastcommon.so

/usr/lib64/libfdfsclient.so

（如果是32位系统,则在当前目录下将这个文件拷贝一份，64位则不用，cp /usr/lib64/libfastcommon.so /usr/lib/）

第二步：所有节点安装FastDFS

tar -zxvf FastDFS_v5.05.tar.gz

cd FastDFS

编译前要确保已经成功安装了 libfastcommon

./make.sh

./make.sh install

说明：默认服务脚本在/etc/init.d/目录下

/etc/init.d/fdfs_storaged

/etc/init.d/fdfs_tracker

说明：命令工具在/usr/bin/目录下的

fdfs_appender_test fdfs_appender_test1 fdfs_append_file fdfs_crc32 fdfs_delete_file fdfs_download_file fdfs_file_info fdfs_monitor fdfs_storaged fdfs_test fdfs_test1 fdfs_trackerd fdfs_upload_appender fdfs_upload_file stop.sh

restart.sh

服务脚本需要连接到/usr/bin/目录下的命令工具，默认服务脚本中连接到的却是/usr/local/bin

所以先将两个tracker节点node25，node26改成对应目录，

vi /etc/init.d/fdfs_trackerd

%s+/usr/local/bin+/usr/bin

保存退出

将剩下的四个storage节点进行如下更改

vi /etc/init.d/fdfs_storaged

%s+/usr/local/bin+/usr/bin

以上操作无论是配置 tracker 还是配置 storage 都是必须的，而 tracker 和storage 的区别主要是在安装完fastdfs 之后的配置过程中，自此各个节点的安装已经结束，下面是配置

三、配置

第一步：所有节点拷贝配置文件

样例配置文件在:

/etc/fdfs/client.conf.sample /etc/fdfs/storage.conf.sample /etc/fdfs/tracker.conf.sample

进入root目录

cd 回到root目录

cd FastDFS/conf/

to all sessions 所有节点执行下面操作

[root@node30 conf]# cp * /etc/fdfs/

cd /etc/fdfs

cd /home

mkdir fastdfs

cd fastdfs

新建目录，所有节点一次性将所有需要用到的目录都建立，to all sessions操作实现，这里的所有操作都是如此

mkdir client

mkdir storage

mkdir tracker

第二步：配置跟踪器：25，26两个tracker节点（暂时只需要tracker.conf），这里就不需要to all sessions

vi /etc/fdfs/tracker.conf

只要更改base_path

base_path=/home/fastdfs/tracker

防火墙中打开跟踪器端口（默认为 22122）：25，26两个tracker节点

vi /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT

重启25,26两个节点的防火墙

service iptables restart

启动tracker之前/home/fastdfs/traker目录下是没有文件的

25，26节点执行启动命令

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf

停止

/usr/bin/stop.sh /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf

初次启动会在/home/fastdfs/tracker目录下创建data，logs两个目录，可以通过两个方法查看traker是否启动成功

(1)查看 22122 端口监听情况:netstat -unltp|grep fdfs

(2)通过以下命令查看 tracker 的启动日志，看是否有错误

tail -100f /home/fastdfs/tracker/logs/trackerd.log

设置 FastDFS 跟踪器开机启动

vi /etc/rc.d/rc.local

## FastDFS Tracker /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf

至此，tracker配置成功

第三步：配置storage（27，28，29，30四个节点）

cd /etc/fdfs/

vi /etc/fdfs/storage.conf

修改的内容如下:（绿色不用改，红色要改）27,28和29,30在以下的配置只有group_name不同，其他配置相同disabled=false #启用配置文件group_name=group1 #组名（第一组（27,28）为 group1，第二组（29,30）为 group2）port=23000 #storage 的端口号,同一个组的 storage 端口号必须相同base_path=/home/fastdfs/storage #设置 storage 的日志目录

store_path0=/home/fastdfs/storage #存储路径

store_path_count=1 #存储路径个数，需要和 store_path 个数匹配tracker_server=192.168.25.125:22122 #tracker 服务器的 IP 地址和端口 tracker_server=192.168.25.126:22122 #多个 tracker 直接添加多条配置

http.server_port=8888 #设置 http 端口号

（其它参数保留默认配置，具体配置解释请参考官方文档说明：http://bbs.chinaunix.net/thread-1941456-1-1.html ）

防火墙开放端口，四个storage节点

防火墙中打开存储器端口（默认为 23000）: vi /etc/sysconfig/iptables 添加如下端口行：

-A INPUT -m state --state NEW -m tcp -p tcp --dport 23000 -j ACCEPT

重启防火墙：

service iptables restart

启动四个storage节点，启动storage之前/home/fastdfs/storage目录下是没有文件的

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf

启动之后，storage目录下生成data，logs目录，data目录下有很多的存储目录

检查是否成功

tail -f /home/fastdfs/storage/logs/storaged.log

查看 23000 端口监听情况：

netstat -unltp|grep fdfs

tracker节点26挂机测试，查看storage节点状态

/usr/bin/stop.sh /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf

storage节点日志查看，停止前leader是126，停止后leader是125节点，

26节点的tracker重新启动后，storage都会监听得到

所有 Storage 节点都启动之后，可以在任一 Storage 节点上使用如下命令查看集群信息：

/usr/bin/fdfs_monitor /etc/fdfs/storage.conf

[root@node27 fdfs]# /usr/bin/fdfs_monitor /etc/fdfs/storage.conf [2018-03-08 01:36:51] DEBUG - base_path=/home/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=2, server_index=1 tracker server is 192.168.25.126:22122 group count: 2 Group 1: group name = group1 disk total space = 12669 MB disk free space = 10158 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 192.168.25.127 ip_addr = 192.168.25.127 ACTIVE http domain = version = 5.05 join time = 2018-03-08 01:05:20 up time = 2018-03-08 01:05:20 total storage = 12669 MB free storage = 10158 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = 192.168.25.128 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2018-03-08 01:36:31 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Storage 2: id = 192.168.25.128 ip_addr = 192.168.25.128 ACTIVE http domain = version = 5.05 join time = 2018-03-08 01:05:16 up time = 2018-03-08 01:05:16 total storage = 12669 MB free storage = 10158 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2018-03-08 01:36:31 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Group 2: group name = group2 disk total space = 12669 MB disk free space = 10158 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 192.168.25.129 ip_addr = 192.168.25.129 ACTIVE http domain = version = 5.05 join time = 2018-03-08 01:05:13 up time = 2018-03-08 01:05:13 total storage = 12669 MB free storage = 10158 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = 192.168.25.130 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2018-03-08 01:36:34 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Storage 2: id = 192.168.25.130 ip_addr = 192.168.25.130 ACTIVE http domain = version = 5.05 join time = 2018-03-08 01:05:02 up time = 2018-03-08 01:05:02 total storage = 12669 MB free storage = 10158 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_open_count = 0 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 0 success_file_write_count = 0 last_heart_beat_time = 2018-03-08 01:36:42 last_source_update = 1970-01-01 08:00:00 last_sync_update = 1970-01-01 08:00:00

last_synced_timestamp = 1970-01-01 08:00:00

可以看到存储节点状态为 ACTIVE 则可

停止storage

/usr/bin/stop.sh /usr/bin/fdfs_storaged /etc/fdfs/storage.conf

给storage配置开机启动

vi /etc/rc.d/rc.local

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf

四、测试

第一步：修改 Tracker 服务器中的客户端配置文件client.conf（以25节点为例作为测试即可，26节点也可以做同样的配置）

vi /etc/fdfs/client.conf

base_path=/home/fastdfs/tracker tracker_server=192.168.25.125:22122 tracker_server=192.168.25.126:22122

文件上传测试

/usr/bin/fdfs_test /etc/fdfs/client.conf upload anti-steal.jpg

再测试一个，节点的同步时间还是很快的

/usr/bin/fdfs_upload_file /etc/fdfs/client.conf /root/FastDFS_v5.05.tar.gz

将上面两个测试再执行一次，文件则上传到了group2中

五、四个的storage节点nginx安装fastdfs模块

第一步：nginx安装fastdfs模块并且配置

cd 回到root目录

tar -zxvf fastdfs-nginx-module_v1.16.tar.gz

cd fastdfs-nginx-module

cd src/

pwd 复制

/root/fastdfs-nginx-module/src

vi config

将里面所有的local去掉保存这个路径修改是很重要，不然在 nginx 编译的时候会报错

本例子中所有的节点都预先安装了nginx，接下来只要重新编译安装nginx即可，

（如果没有安装则需要安装yum install gcc gcc-c++ make automake autoconf libtool pcre pcre-devel zlib zlib-devel

openssl openssl-devel）

所有storage节点都重新安装配置nginx

cd /root/nginx-1.8.0 （或者./configure --prefix=/usr/local/nginx --add-module=/usr/local/src/fastdfs-nginx-module/src）

./configure --prefix=/usr/nginx-1.8 \

--add-module=/root/fastdfs-nginx-module/src

make

make install

cp /root/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs

vi /etc/fdfs/mod_fastdfs.conf

第一组 Storage 的 mod_fastdfs.conf 配置与第二组配置只有 group_name 不同： group_name=group2

connect_timeout=10 base_path=/tmp tracker_server=192.168.25.125:22122 tracker_server=192.168.25.126:22122 storage_server_port=23000group_name=group1 #27,28节点group1 29,30节点group2 url_have_group_name = true store_path0=/home/fastdfs/storage group_count = 2 [group1] group_name=group1 storage_server_port=23000 store_path_count=1 store_path0=/home/fastdfs/storage [group2] group_name=group2 storage_server_port=23000 store_path_count=1 store_path0=/home/fastdfs/storage

其中一个组中配置文件

# connect timeout in seconds # default value is 30s connect_timeout=10 # network recv and send timeout in seconds # default value is 30s network_timeout=30 # the base path to store log files base_path=/tmp # if load FastDFS parameters from tracker server # since V1.12 # default value is false load_fdfs_parameters_from_tracker=true # storage sync file max delay seconds # same as tracker.conf # valid only when load_fdfs_parameters_from_tracker is false # since V1.12 # default value is 86400 seconds (one day) storage_sync_file_max_delay = 86400 # if use storage ID instead of IP address # same as tracker.conf # valid only when load_fdfs_parameters_from_tracker is false # default value is false # since V1.13 use_storage_id = false # specify storage ids filename, can use relative or absolute path # same as tracker.conf # valid only when load_fdfs_parameters_from_tracker is false # since V1.13 storage_ids_filename = storage_ids.conf # FastDFS tracker_server can ocur more than once, and tracker_server format is # "host:port", host can be hostname or ip address # valid only when load_fdfs_parameters_from_tracker is truetracker_server=192.168.25.125:22122 tracker_server=192.168.25.126:22122 # the port of the local storage server # the default value is 23000 storage_server_port=23000 # the group name of the local storage servergroup_name=group1 # if the url / uri including the group name # set to false when uri like /M00/00/00/xxx # set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx # default value is false url_have_group_name = true # path(disk or mount point) count, default value is 1 # must same as storage.conf store_path_count=1 # store_path#, based 0, if store_path0 not exists, it's value is base_path # the paths must be exist # must same as storage.confstore_path0=/home/fastdfs/storage #store_path1=/home/yuqing/fastdfs1 # standard log level as syslog, case insensitive, value list: ### emerg for emergency ### alert ### crit for critical ### error ### warn for warning ### notice ### info ### debug log_level=info # set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log # empty for output to stderr (apache and nginx error_log file) log_filename= # response mode when the file not exist in the local file system ## proxy: get the content from other storage server, then send to client ## redirect: redirect to the original storage server (HTTP Header is Location) response_mode=proxy # the NIC alias prefix, such as eth in Linux, you can see it by ifconfig -a # multi aliases split by comma. empty value means auto set by OS type # this paramter used to get all ip address of the local host # default values is empty if_alias_prefix= # use "#include" directive to include HTTP config file # NOTE: #include is an include directive, do NOT remove the # before include #include http.conf # if support flv # default value is false # since v1.15 flv_support = true # flv file extension name # default value is flv # since v1.15 flv_extension = flv # set the group count # set to none zero to support multi-group # set to 0 for single group only # groups settings section as [group1], [group2], ..., [groupN] # default value is 0 # since v1.14 group_count = 2 # group settings for group #1 # since v1.14 # when support multi-group, uncomment following section[group1] group_name=group1 storage_server_port=23000 store_path_count=1 store_path0=/home/fastdfs/storage #store_path1=/home/yuqing/fastdfs1 # group settings for group #2 # since v1.14 # when support multi-group, uncomment following section as neccessary[group2] group_name=group2 storage_server_port=23000 store_path_count=1 store_path0=/home/fastdfs/storage

第二步：四个storage节点配置nginx

cd /usr/nginx-1.8/conf

vi nginx.conf

server { listen 8888; server_name localhost; location ~/group([0-9])/M00 { ngx_fastdfs_module; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } }

说明： A、8888 端口值是要与/etc/fdfs/storage.conf 中的 http.server_port=8888 相对应，因为 http.server_port 默认为 8888,如果想改成 80，则要对应修改过来。 B、Storage 对应有多个 group 的情况下，访问路径带 group 名，如/group1/M00/00/00/xxx，对应的 Nginx 配置为： location ~/group([0-9])/M00 { ngx_fastdfs_module; }

C、如查下载时如发现老报 404，将 nginx.conf 第一行 user nobody 修改为 user root 后重新启动。

四个storage节点，在防火墙中打开 Nginx 的 8888 端口

vi /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8888 -j ACCEPT

service iptables restart

启动 Nginx

cd /usr/nginx-1.8

./sbin/nginx

设置 Nginx 开机启动

vi /etc/rc.local

/usr/nginx-1.8/sbin/nginx

浏览器访问测试时的上传文件

http://192.168.25.129:8888/group2/M00/00/00/wKgZgVqgKLKAVhNNAABdrZgsqUU428_big.jpg

http://192.168.25.129:8888/group2/M00/00/00/wKgZglqgKLWACQnpAAVFOL7FJU4.tar.gz

http://192.168.25.128:8888/group1/M00/00/00/wKgZgFqgJ3CACfMrAAVFOL7FJU4.tar.gz

第三步：在跟踪器tracker节点（ 192 .168.25.125 、 192 .168.25.126 ）上安装 Nginx清除缓存并且配置

ngx_cache_purge模块的作用：用于清除指定url的缓存

tar -zxvf ngx_cache_purge-2.3.tar.gz

cd /root/nginx-1.8.0

./configure --prefix=/usr/nginx-1.8 \

--add-module=/root/ngx_cache_purge-2.3

make && make install

配置 Nginx，设置负载均衡以及缓存

vi /usr/nginx-1.8/conf/nginx.conf

配置内容如下

user root; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; use epoll; } http { include mime.types; default_type application/octet-stream; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; #设置缓存 server_names_hash_bucket_size 128; client_header_buffer_size 32k; large_client_header_buffers 4 32k; client_max_body_size 300m; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 16k; proxy_buffers 4 64k; proxy_busy_buffers_size 128k; proxy_temp_file_write_size 128k; #设置缓存存储路径、存储方式、分配内存大小、磁盘最大空间、缓存期限 proxy_cache_path /home/fastdfs/cache/nginx/proxy_cache levels=1:2 keys_zone=http-cache:200m max_size=1g inactive=30d; proxy_temp_path /home/fastdfs/cache/nginx/proxy_cache/tmp; #设置 group1 的服务器 upstream fdfs_group1 { server 192.168.25.127:8888 weight=1 max_fails=2 fail_timeout=30s; server 192.168.25.128:8888 weight=1 max_fails=2 fail_timeout=30s; } #设置 group2 的服务器 upstream fdfs_group2 { server 192.168.25.129:8888 weight=1 max_fails=2 fail_timeout=30s; server 192.168.25.130:8888 weight=1 max_fails=2 fail_timeout=30s; } server { listen 8000; server_name localhost; #charset koi8-r; #access_log logs/host.access.log main; #设置 group 的负载均衡参数 location /group1/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache; proxy_cache_valid 200 304 12h; proxy_cache_key $uri$is_args$args; proxy_pass http://fdfs_group1; expires 30d; } location /group2/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache; proxy_cache_valid 200 304 12h; proxy_cache_key $uri$is_args$args; proxy_pass http://fdfs_group2; expires 30d; } #设置清除缓存的访问权限 location ~/purge(/.*) { allow 127.0.0.1; allow 192.168.1.0/24; deny all; proxy_cache_purge http-cache $1$is_args$args; } #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } } 按以上 nginx 配置文件的要求，创建对应的缓存目录： mkdir -p /home/fastdfs/cache/nginx/proxy_cache

mkdir -p /home/fastdfs/cache/nginx/proxy_cache/tmp

tracker节点中，系统防火墙打开对应的端口

vi /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8000 -j ACCEPT

service iptables restart

启动 Nginx

/usr/nginx-1.8/sbin/nginx

设置 Nginx 开机启动

vi /etc/rc.local

/usr/nginx-1.8/sbin/nginx

文件访问测试

前面直接通过访问 Storage 节点中的 Nginx 的文件

现在可以通过 Tracker 中的 Nginx 来进行访问

http://192.168.25.125:8000/group2/M00/00/00/wKgZgVqgKLKAVhNNAABdrZgsqUU428_big.jpg

http://192.168.25.125:8000/group1/M00/00/00/wKgZgFqgJ3CACfMrAAVFOL7FJU4.tar.gz

http://192.168.25.125:8000/group2/M00/00/00/wKgZglqgKLWACQnpAAVFOL7FJU4.tar.gz

六、配置keepalived+nginx高可用负载均衡（22节点和23节点）

说名：每一个 Tracker 中的 Nginx 都单独对后端的 Storage 组做了负载均衡，但整套 FastDFS 集群如果想对外提供统一的文件访问地址，还需要对两个 Tracker 中的 Nginx 进行 HA 集群

yum install gcc gcc-c++ make automake autoconf libtool pcre pcre-devel zlib-devel openssl openssl-devel

使用Keepalived + Nginx组成的高可用负载均衡集群做两个Tracker节点中Nginx的负载均衡

新增了两个节点（22-master，23-backup节点作为负载均衡所用），这两个节点已经安装了nginx和jdk，克隆所得

第一步：配置nginx，只更改红色部分

} }

vi /usr/nginx-1.8/html/index.html

192.168.25.122 中的标题加 2 Welcome to nginx! 2 192.168.25.123 中的标题加 3 Welcome to nginx! 3

系统防火墙打开对应的端口 88 vi /etc/sysconfig/iptables ## Nginx -A INPUT -m state --state NEW -m tcp -p tcp --dport 88 -j ACCEPT

service iptables restar

测试 Nginx 是否安装成功

/usr/nginx-1.8/sbin/nginx -t

出现下面两行说名配置成功

nginx: the configuration file /usr/nginx-1.8/conf/nginx.conf syntax is ok nginx: configuration file /usr/nginx-1.8/conf/nginx.conf test is successful

启动 Nginx /usr/nginx-1.8/sbin/nginx 重启 Nginx

/usr/nginx-1.8/sbin/nginx -s reload

设置 Nginx 开机启动

vi /etc/rc.local

/usr/nginx-1.8/sbin/nginx

分别访问两个 Nginx

192.168.25.122:88

192.168.25.123:88

第二步：安装 Keepalived

上传keepalived-1.4.1.tar.gz文件或者去官网下载 http://www.keepalived.org/download.html

解压安装

tar -zxvf keepalived-1.4.1.tar.gz

cd keepalived-1.4.1

./configure --prefix=/usr/keepalived

make && make install

将 keepalived 安装成 Linux 系统服务：

因为没有使用 keepalived 的默认路径安装（默认是/usr/local）,安装完成之后，需要做一些工作复制默认配置文件到默认路径

keepalived默认会读取/etc/keepalived/keepalived.conf配置文件，所以

mkdir /etc/keepalived

cp /usr/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

复制启动脚本到/etc/init.d下

cd /root/keepalived-1.4.1

cp ./keepalived/etc/init.d/keepalived /etc/init.d/

chmod 755 /etc/init.d/keepalived

复制sysconfig文件到/etc/sysconfig下

cp /usr/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

复制/sbin/keepalived到/usr/sbin下

cp /usr/keepalived/sbin/keepalived /usr/sbin/

设置 keepalived 服务开机启动

chkconfig keepalived on

修改 Keepalived 配置文件

(1) MASTER 节点配置文件（192.168.25.122）

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived global_defs { ## keepalived 自带的邮件提醒需要开启 sendmail 服务。建议用独立的监控或第三方 SMTP router_id edu-proxy-01 ## 标识本节点的字条串，通常为 hostname } ## keepalived 会定时执行脚本并对脚本执行的结果进行分析，动态调整 vrrp_instance 的优先级。如果脚本执行结果为 0，并且 weight 配置的值大于 0，则优先级相应的增加。如果脚本执行结果非 0，并且 weight 配置的值小于 0，则优先级相应的减少。其他情况，维持原本配置的优先级，即配置文件中 priority 对应的值。 vrrp_script chk_nginx { script "/etc/keepalived/nginx_check.sh" ## 检测 nginx 状态的脚本路径 interval 2 ## 检测时间间隔 weight -20 ## 如果条件成立，权重-20 } ## 定义虚拟路由，VI_1 为虚拟路由的标示符，自己定义名称 vrrp_instance VI_1 { state MASTER ## 主节点为 MASTER，对应的备份节点为 BACKUP interface eth0 ## 绑定虚拟 IP 的网络接口，与本机 IP 地址所在的网络接口相同，我的是 eth1 virtual_router_id 51 ## 虚拟路由的 ID 号，两个节点设置必须一样，可选 IP 最后一段使用, 相同的 VRID 为一个组，他将决定多播的 MAC 地址 mcast_src_ip 192.168.25.122 ## 本机 IP 地址 priority 100 ## 节点优先级，值范围 0-254，MASTER 要比 BACKUP 高nopreempt ## 优先级高的设置 nopreempt 解决异常恢复后再次抢占的问题 advert_int 1 ## 组播信息发送间隔，两个节点设置必须一样，默认 1s ## 设置验证信息，两个节点必须一致 authentication { auth_type PASS auth_pass 1111 ## 真实生产，按需求对应该过来 } ## 将 track_script 块加入 instance 配置块 track_script { chk_nginx ## 执行 Nginx 监控的服务 } ## 虚拟 IP 池, 两个节点设置必须一样 virtual_ipaddress { 192.168.25.50 ## 虚拟 ip，可以定义多个 }

}

(2)BACKUP 节点配置文件（192.168.25.123）: vi /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id edu-proxy-02 } vrrp_script chk_nginx { script "/etc/keepalived/nginx_check.sh" interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 mcast_src_ip 192.168.25.123 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { 192.168.25.50 }

}

（3）编写 Nginx 状态检测脚本 /etc/keepalived/nginx_check.sh (已在 keepalived.conf 中配置) 脚本要求：如果 nginx 停止运行，尝试启动，如果无法启动则杀死本机的 keepalived 进程，keepalived，将虚拟 ip 绑定到 BACKUP 机器上。内容如下：

vi /etc/keepalived/nginx_check.sh

#!/bin/bash A=`ps -C nginx –no-header |wc -l` if [ $A -eq 0 ];then /usr/local/nginx/sbin/nginx sleep 2 if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then killall keepalived fi fi

用意：请求进来的时候，nginx master宕机了，keepalived还没有将协议漂移过去，nginx的backup还没有上位，这时候请求进来还是在master上，请求无法处理，所以这时候要加一个脚本判断一下这个nginx是否正常，如果nginx不正常或者挂机了，进程中没有nginx则将本机的keepalived进程kill掉，让master切换到backup上来，因为真正处理用户请求的是nginx

保存后，给脚本赋执行权限

chmod +x /etc/keepalived/nginx_check.sh

启动 Keepalived service keepalived start

Starting keepalived: [ OK ]

Keepalived+Nginx 的高可用测试 (1)关闭 192.168.25.122 中的 Nginx，Keepalived 会将它重新启动 /usr/nginx-1.8/sbin/nginx -s stop (2)关闭 192.168.25.122 中的 Keepalived，VIP 会切换到 192.168.25.122 中

service keepalived stop

ip a 可以查看网络接口中的VIP

Keepalived 停止后，该节点的网络接口中的 VIP 将消失

(3)重新启动 192.168.1.51 中的 Keepalived，VIP 又会切回到 192.168.1.51 中来

service keepalived start

这时候就可以通过统一网址来访问集群了

Keepalived 服务管理命令：停止：service keepalived stop 启动：service keepalived start 重启：service keepalived restart 查看状态：service keepalived status 其他参考资料： keepalived 之 vrrp_script 总结：http://my.oschina.net/hncscwc/blog/158746 keepalived 双机热备实现故障时发送邮件通知：http://www.2cto.com/os/201407/317795.html

基于 keepalived 实现 VIP 转移，lvs，nginx 的高可用：http://www.tuicool.com/articles/eu26Vz

至此keepalived+nginx高可用已经配置结束

七、在 Keepalived+Nginx （22、23节点）实现高可用负载均衡集群中配置 Tracker 节点中 Nginx 的负载均衡反向代理

第一步：(192.168.25.122 和 192.168.25.123 中的 Nginx 执行相同的配置)

只需要在配置文件中添加以下配置即可

vi /usr/nginx-1.8/conf/nginx.conf

## FastDFS Tracker Proxy upstream fastdfs_tracker { server 192.168.25.125:8000 weight=1 max_fails=2 fail_timeout=30s; server 192.168.25.126:8000 weight=1 max_fails=2 fail_timeout=30s;

}

## FastDFS Tracker Proxy

location /dfs { root html; index index.html index.htm; proxy_pass http://fastdfs_tracker/; proxy_set_header Host $http_host; proxy_set_header Cookie $http_cookie; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; client_max_body_size 300m;

}

重启192.168.25.122 和 192.168.25.123中的nginx

通过 Keepalived+Nginx 组成的高可用负载集群的 VIP(192.168.25.50)来访问 FastDFS 集群中的文件

http://192.168.25.50:88/dfs/group2/M00/00/00/wKgZgVqgKLKAVhNNAABdrZgsqUU428_big.jpg

第二步：如何使用：

FILE_SYS_URL = http://192.168.25.50:88/dfs/

fdfs_client.conf中增加配置，tracker几个添加几行

tracker_server = 192.168.25.125:22122

tracker_server = 192.168.25.126:22122

http://192.168.25.50:88/dfs/group1/M00/00/00/wKgZf1qiVgWAKAofAAF9Se5IIHk477.jpg

FastDFS集群搭建+keepalive+nginx高可用

最近更新

热门博客

[ 申请 ]友情链接：