5.方案适用场景:
适用于数据库访问量不太大,短期内访问量增长不会太快,对数据库可用性要求非常高的场景。
6.测试环境介绍(如下所示,均已关闭防火墙及selinux,生产环境自行开放端口)
主机名 ip 系统 DRBD磁盘 heartbeat版本db-server-01 192.168.0.10 centos6.2 64bit /dev/sda5 3.0.4db-server-02 192.168.0.20 centos6.2 64bit /dev/sda5 3.0.4
7.软件安装以及环境配置
(1)安装drbd依赖组件(两台机器,安装以后重启系统,因为会升级内核版本,不重启会对不上内核版本,有知道不用重启的童鞋请给我留言^_^):
yum install -y kernel kernel-devel kernel-headers flex
(2)下载软件安装(两台机器操作一样)
wget http://oss.linbit.com/drbd/8.4/drbd-8.4.2.tar.gz
[AppleScript] 纯文本查看 复制代码
tar xf drbd-8.4.2.tar.gz
cd drbd-8.4.2
./configure --prefix=/usr/local/drbd --with-km
make KDIR=/usr/src/kernels/2.6.32-431.11.2.el6.x86_64/ #很多童鞋无法加载drbd模块,多半是正在运行的内核版本和新安装的不相符
make install
mkdir -p /usr/local/drbd/var/run/drbd
cp /usr/local/drbd/etc/rc.d/init.d/drbd /etc/rc.d/init.d
chmod 755 /etc/init.d/drbd
cd drbd
make clean
make KDIR=/usr/src/kernels/2.6.32-431.11.2.el6.x86_64/
cp drbd.ko /lib/modules/`uname -r`/kernel/lib/
modprobe drbd
检查是否加载了drbd模块
[root@192.168.0.10 ~]# lsmod | grep drbddrbd 314246 0 libcrc32c 1246 1 drbd[root@192.168.0.10 ~]#
(3)DRBD配置(配置之前需要先使用fdisk对 /dev/sda进行分区)
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# df -HT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext4 19G 2.6G 16G 15% /
tmpfs tmpfs 121M 0 121M 0% /dev/shm
/dev/sda1 ext4 204M 52M 141M 27% /boot
/dev/sda5 ext4 34G 185M 32G 1% /data
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
我这里两台机器之前都已经分区了,由于是自己笔记本上的虚拟机,所以懒得加磁盘了,我直接把 /data/卸载,然后格式化/dev/sda5,我两台机器都这样操作,如果你有空的磁盘,照样需要进行分区,比如可以将一个1T的盘分一个区就行了。
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# umount /data/
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# mkfs.ext4 /dev/sda5
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
2048000 inodes, 8185344 blocks
409267 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
250 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# fdisk -l
Disk /dev/sda: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000eb0ff
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 26 2321 18432000 83 Linux
/dev/sda3 2321 2451 1048576 82 Linux swap / Solaris
/dev/sda4 2451 6528 32742400 5 Extended
/dev/sda5 2451 6528 32741376 83 Linux
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
我这里还要在/etc/fstab里面注释一项:
#UUID=33958004-e8a7-4135-844f-707a5537e86a /data ext4 defaults 1 2
否则重启机器的时候提示无法挂载,会无法启动的。
修改/etc/hosts文件,两台服务器操作一样。
192.168.0.10 db-server-01192.168.0.20 db-server-02
drbd配置只需要修改/usr/local/drbd/etc/drbd.d/global_common.conf配置文件即可,修改后如下(两台服务器配置一样):
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# cat /usr/local/drbd/etc/drbd.d/global_common.conf
global { usage-count yes; }
common { syncer { rate 30M; } } #同步速率,视带宽而定
resource r0 { #创建一个资源,名字叫"r0"
protocol C; #选择的是drbd的C 协议(数据同步协议,C为收到数据并写入后返回,确认成功)
startup {
}
disk {
on-io-error detach;
}
net {
}
on db-server-01 { #设定一个节点,分别以各自的主机名命名
device /dev/drbd0; #设定资源设备/dev/drbd0 指向实际的物理分区 /dev/sda5
disk /dev/sda5;
address 192.168.0.10:7888; #设定监听地址以及端口
meta-disk internal;
}
on db-server-02 {
device /dev/drbd0;
disk /dev/sda5;
address 192.168.0.20:7888;
meta-disk internal; #internal表示是在同一个局域网内
}
}
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
(4)DRBD的管理与维护:
创建DRBD资源
配置好drbd以后,就需要使用命令创建配置的drbd资源,使用如下命令(两台服务器操作一样):
[root@192.168.0.10 ~]# dd if=/dev/zero of=/dev/sda5 bs=1M count=100 #不这样做的话,在创建资源的时候报错100+0 records in100+0 records out104857600 bytes (105 MB) copied, 3.34339 s, 31.4 MB/s[root@192.168.0.10 ~]#
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# drbdadm create-md r0
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
success
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
(5)DRBD的启动与状态查看(分别在两台服务器启动)
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# /etc/init.d/drbd start
Starting DRBD resources: [
create res: r0
prepare disk: r0
adjust disk: r0
adjust net: r0
]
.....
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# /etc/init.d/drbd start
Starting DRBD resources: [
create res: r0
prepare disk: r0
adjust disk: r0
adjust net: r0
]
.
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]#
查看drbd的状态:
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57
m:res cs ro ds p mounted fstype
0:r0 Connected Secondary/Secondary Inconsistent/Inconsistent C
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
可以看见都还没有主节点。设置当前节点(192.168.0.10)为主节点,并进行格式化和挂载 。
drbdadm -- --overwrite-data-of-peer primary allmkfs.ext4 /dev/drbd0mkdir /datamount /dev/drbd0 /data/
在另外一台服务器创建挂载目录,也创建/data
[root@192.168.0.20 ~]# mkdir /data
查看一下drbd的状态(可以看见还在同步):
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57
m:res cs ro ds p mounted fstype
... sync'ed: 13.7% (27596/31972)M
0:r0 SyncSource Primary/Secondary UpToDate/Inconsistent C /data ext4
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
(6)mysql安装,我这里为了简单直接安装编译好的二进制软件包(两台服务器都需要安装,操作一样,只是第二台mysql不需要初始化数据)
注意:两台服务器上的mysql用户的uid和gid要一样。不然切换后会导致mysql数据目录的属主不正确而启动失败。
[root@192.168.0.10 ~]# wget http://cdn.mysql.com/Downloads/M ... ux2.6-x86_64.tar.gz
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# tar xf mysql-5.5.37-linux2.6-x86_64.tar.gz -C /usr/local/
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# cd /usr/local/
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] local]# ln -s mysql-5.5.37-linux2.6-x86_64/ mysql
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] local]# groupadd mysql
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] local]# useradd -r -g mysql mysql
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] local]# cd mysql
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] mysql]# chown -R mysql .
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] mysql]# chgrp -R mysql .
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] mysql]# mkdir /data/mysql
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] mysql]# chown -R mysql.mysql /data/mysql/
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] mysql]# /usr/local/mysql/scripts/mysql_install_db --user=mysql --datadir=/data/mysql/ --basedir=/usr/local/mysql
[root@192.168.0.10 mysql]# chown -R root .[root@192.168.0.10 mysql]# cp support-files/my-medium.cnf /etc/my.cnf[root@192.168.0.10 mysql]# cp support-files/mysql.server /etc/init.d/mysqld[root@192.168.0.10 mysql]# chmod 755 /etc/init.d/mysqld
[root@192.168.0.10 mysql]# egrep 'datadir|basedir' /etc/my.cnf #两台服务器上的mysql配置文件都加入这里的配置 datadir=/data/mysqlbasedir=/usr/local/mysql [root@192.168.0.10 mysql]#
(7)手动切换drbd的主从。看另外一台服务器是否有数据(自动切换需要使用heartbeat,后面介绍):
[root@192.168.0.10 ~]# ll /data/total 20drwx------ 2 root root 16384 Apr 18 22:16 lost+founddrwxr-xr-x 5 mysql mysql 4096 Apr 18 23:01 mysql[root@192.168.0.10 ~]#
[root@192.168.0.20 ~]# ll /data/total 0[root@192.168.0.20 ~]#
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-01, 2014-04-18 21:15:57
m:res cs ro ds p mounted fstype
0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext4
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
可以看见当前服务器是主,也就是数据在这台服务器上,另外一台服务器是没有数据的。下面进行手动切换
主切换成从,需要先卸载文件系统,再执行降级为从的命令:
[root@192.168.0.10 ~]# umount /data/[root@192.168.0.10 ~]# drbdadm secondary all
从切换成主,要先执行升级成主的命令然后挂在文件系统:
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# drbdadm primary all
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# mount /dev/drbd0 /data/
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# ll /data/
total 20
drwx------ 2 root root 16384 Apr 18 22:16 lost+found
drwxr-xr-x 5 mysql mysql 4096 Apr 18 23:01 mysql
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@db-server-02, 2014-04-18 21:22:55
m:res cs ro ds p mounted fstype
0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext4
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]#
可以看见已经成功切换成主,并且mysql初始化数据也存在了。
DRBD脑裂后的处理
当DRBD出现脑裂后,会导致drbd两边的磁盘数据不一致,在确定要作为从的节点上切换成secondary,并放弃该资源的数据:
drbdadm secondary r0drbdadm -- --discard-my-data connect r0
在要作为primary的节点重新连接secondary(如果这个节点当前的连接状态为WFConnection的话,可以省略),使用如下命令连接:
drbdadm connect r0
(8)Heartbeat安装(两台服务器)
需要添加epel源,centos默认自己没有该软件包,当然你可以自己源码编译。
rpm -ivh http://dl.fedoraproject.org/pub/ ... ease-6-8.noarch.rpm
yum install heartbeat -y
创建DRBD脚本文件drbddisk两台服务器) 注意:
此处是一个大坑,因为默认yum安装Heartbeat,不会在/etc/ha.d/resource.d/创建drbddisk脚本,估计是版本太新了吧。记得前两年都不会这样的。囧。而且也无法在安装后从本地其他路径找到该文件。此处也是因为启动Heartbeat后无法PING通虚IP,最后通过查看/var/log/ha-log日志,找到一行ERROR: Cannot locate resource script drbddisk,然后进而到/etc/ha.d/resource.d/路径下发现竟然没有drbddisk脚本,最后在google上找到该代码,创建该脚本,终于测试通过:
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# chmod 755 /etc/ha.d/resource.d/drbddisk
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# cat /etc/ha.d/resource.d/drbddisk
#!/bin/bash
#
# This script is inteded to be used as resource script by heartbeat
#
# Copright 2003-2008 LINBIT Information Technologies
# Philipp Reisner, Lars Ellenberg
#
###
DEFAULTFILE="/etc/default/drbd"
DRBDADM="/sbin/drbdadm"
if [ -f $DEFAULTFILE ]; then
. $DEFAULTFILE
fi
if [ "$#" -eq 2 ]; then
RES="$1"
CMD="$2"
else
RES="all"
CMD="$1"
fi
## EXIT CODES
# since this is a "legacy heartbeat R1 resource agent" script,
# exit codes actually do not matter that much as long as we conform to
# [url=http://wiki.linux-ha.org/HeartbeatResourceAgent]http://wiki.linux-ha.org/HeartbeatResourceAgent[/url]
# but it does not hurt to conform to lsb init-script exit codes,
# where we can.
# [url=http://refspecs.linux-foundation.org/LSB_3.1.0/]http://refspecs.linux-foundation.org/LSB_3.1.0/[/url]
#LSB-Core-generic/LSB-Core-generic/iniscrptact.html
####
drbd_set_role_from_proc_drbd()
{
local out
if ! test -e /proc/drbd; then
ROLE="Unconfigured"
return
fi
dev=$( $DRBDADM sh-dev $RES )
minor=${dev#/dev/drbd}
if [[ $minor = *[!0-9]* ]] ; then
# sh-minor is only supported since drbd 8.3.1
minor=$( $DRBDADM sh-minor $RES )
fi
if [[ -z $minor ]] || [[ $minor = *[!0-9]* ]] ; then
ROLE=Unknown
return
fi
if out=$(sed -ne "/^ *$minor: cs:/ { s/:/ /g; p; q; }" /proc/drbd); then
set -- $out
ROLE=${5%/**}
: ${ROLE:=Unconfigured} # if it does not show up
else
ROLE=Unknown
fi
}
case "$CMD" in
start)
# try several times, in case heartbeat deadtime
# was smaller than drbd ping time
try=6
while true; do
$DRBDADM primary $RES && break
let "--try" || exit 1 # LSB generic error
sleep 1
done
;;
stop)
# heartbeat (haresources mode) will retry failed stop
# for a number of times in addition to this internal retry.
try=3
while true; do
$DRBDADM secondary $RES && break
# We used to lie here, and pretend success for anything != 11,
# to avoid the reboot on failed stop recovery for "simple
# config errors" and such. But that is incorrect.
# Don't lie to your cluster manager.
# And don't do config errors...
let --try || exit 1 # LSB generic error
sleep 1
done
;;
status)
if [ "$RES" = "all" ]; then
echo "A resource name is required for status inquiries."
exit 10
fi
ST=$( $DRBDADM role $RES )
ROLE=${ST%/**}
case $ROLE in
Primary|Secondary|Unconfigured)
# expected
;;
*)
# unexpected. whatever...
# If we are unsure about the state of a resource, we need to
# report it as possibly running, so heartbeat can, after failed
# stop, do a recovery by reboot.
# drbdsetup may fail for obscure reasons, e.g. if /var/lock/ is
# suddenly readonly. So we retry by parsing /proc/drbd.
drbd_set_role_from_proc_drbd
esac
case $ROLE in
Primary)
echo "running (Primary)"
exit 0 # LSB status "service is OK"
;;
Secondary|Unconfigured)
echo "stopped ($ROLE)"
exit 3 # LSB status "service is not running"
;;
*)
# NOTE the "running" in below message.
# this is a "heartbeat" resource script,
# the exit code is _ignored_.
echo "cannot determine status, may be running ($ROLE)"
exit 4 # LSB status "service status is unknown"
;;
esac
;;
*)
echo "Usage: drbddisk [resource] {start|stop|status}"
exit 1
;;
esac
exit 0
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]#
(9)heartbeat配置
Hearbeat的配置主要包括三个配置文件,authkeys,ha.cf和haresources的配置,下面就分别来看看:
Authkerys的配置(两台服务器配置一样)
这个文件用来配置密码认证方式,支持3种认证方式,crc,md5和sha1,从左到右安全性越来越高,消耗的资源也越多。因此如果heartbeat运行在安全的网路之上,比如私网,那么可以将验证方式设置成crc,master和backup的authkeys配置一样。我的authkeys文件配置如下:
[root@192.168.0.10 ~]# cat /etc/ha.d/authkeys auth 11 crc[root@192.168.0.10 ~]# chmod 600 /etc/ha.d/authkeys
注意:该文件权限必须是600
ha.cf的配置(两台机器稍微有点区别),Primary(192.168.0.10)如下:
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]# cat /etc/ha.d/ha.cf
logfile /var/log/ha-log
#定义Heartbeat的日志名字及位置
logfacility local0
keepalive 2
#设定心跳(监测)时间为2秒
deadtime 15
#设定死亡时间为15秒
ucast eth1 192.168.0.20
#采用单播的方式,IP地址指定为对方IP
auto_failback off
#当Primary机器发生故障切换到Secondary机器后Primary恢复后是否进行切回操作 (最好是我们有需求手动进行切换)
node db-server-01
node db-server-02
[[url=mailto:root@192.168.0.10]root@192.168.0.10[/url] ~]#
Secondary(192.168.0.20)如下:
[AppleScript] 纯文本查看 复制代码
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]# cat /etc/ha.d/ha.cf
logfile /var/log/ha-log
#定义Heartbeat的日志名字及位置
logfacility local0
keepalive 2
#设定心跳(监测)时间为2秒
deadtime 15
#设定死亡时间为15秒
ucast eth1 192.168.0.10
#采用单播的方式,IP地址指定为对方IP
auto_failback off
#当Primary机器发生故障切换到Secondary机器后Primary恢复后是否进行切回操作(一般我们可以看需求,否则不用自动切换)
node db-server-01
node db-server-02
[[url=mailto:root@192.168.0.20]root@192.168.0.20[/url] ~]#
haresources的配置(两台机器配置一样):
[root@192.168.0.10 ~]# cat /etc/ha.d/haresources db-server-01 IPaddr::192.168.0.88/24/eth1 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext4 mysqld [root@192.168.0.10 ~]#
注:该文件内IPaddr,Filesystem等脚本存放路径在/etc/ha.d/resource.d/下,也可在该目录下存放服务启动脚本(例如:mysqld),将相同脚本名称添到/etc/ha.d/haresources内容中,从而跟随heartbeat启动而启动该脚本。
IPaddr::192.168.0.88/24/eth1:用IPaddr脚本配置浮动VIP
drbddisk::r0:用drbddisk脚本实现DRBD主从节点资源组的挂载和卸载
Filesystem::/dev/drbd0::/data::ext4:用Filesystem脚本实现磁盘挂载和卸载
(10)heartbeat的管理
配置好heartbeat之后,需要将mysql从自启动服务器中去掉,因为主heartbeat启动的时候会挂载drdb文件系统以及启动mysql,切换的时候会将主上的mysql停止并卸载文件系统,从上会挂载文件系统,并启动mysql。因此需要做如下操作(两台服务器):
[root@192.168.0.10 ~]# chkconfig mysqld off[root@192.168.0.10 ~]# chkconfig heartbeat off[root@192.168.0.10 ~]# chkconfig drbd off