SLES11 Xen +HA CEO miyahara@virtualtech.jp VirtualTech Japan Inc. VTJ 2006 12 14,250,000 1-1-10 CEO CTO 8 5.5 URL http://virtualtech.jp/ 2 1
P2V Xen 3 High Availability Extension SUSE Linux Enterprise 11HA High Availability OpenAIS DRBD OCFS2 GUI 2
Xen+DRBD HA OS OS DRBD OpenAIS Xen DRBD OpenAIS Xen OS:SUSE Linux Enerprise 11 SLES 11 Xen SUSE Linux Enterprise 11 High Availability Extension SLES 11 HAE HA/ / 3
HA 1. SLES 11 HAE 2. /etc/hosts 3. NTP 4. OpenAIS 5. DRBD 6. DRBD 7. vm 8. vm SLES 11 HAE 1. SLES 11 ISO 2. [YaST] [] [] 3. 4. [] 5. SLES 11 ISO 4
xen1 xen2 1. [YaST] [] [ ] 2. [] 3. IP 192.168.0.1 xen1.example.com xen1 192.168.0.2 xen2.example.com xen2 NTP xen1 1. [YaST] [] [NTP ] 2. NTP 3. [Undisciplined Local Clock(LOCAL)] 4. [] 5. [ ] [Stratum] 5 5
NTP xen2 1. [YaST] [] [NTP ] 2. NTP 3. [] [ ] xen1 OpenAIS xen1 1. [YaST] [ ] [] 2. [Communication Channel] Bind Network Address 192.168.0.0 Multicast Address 224.0.0.1 Multicast Port 5405 3. ID 1 4. rrp mode active 6
OpenAIS OpenAIS xen2 1. [YaST] [ ] [] 2. [Communication Channel] Bind Network Address 192.168.0.0 Multicast Address 224.0.0.1 Multicast Port 5405 3. ID 2 4. rrp mode passive 7
/ 1. 2. hacluster xen1# passwd hacluster xen2# passwd hacluster 3. xen1# service openais start xen2# service openais start 4. GUI xen1# crm_gui & 5. Connection Server:127.0.0.1 User Name hacluster Password GUI 8
1. [CRM Config] 2. [No Quorum Policy] [ignore] 3. [Stonith Enabled] STONITH(Shoot The Other Node In The Head) 9
DRBD [YaST] ]] [ ] LVM lv4drbd vmconfig vmimages /dev/lv4drbd/vmconfig 3GB /dev/lv4drbd/vmimages 40GB LVM 10
DRBD (vmconfig) [YaST] [ ] [Drbd] Add DRBDvmconfig Node 1 vmconfig Node 2 Name xen1 xen2 Address:Port 192.168.0.1:7789 192.168.0.2:7789 Device /dev/drbd0 /dev/drbd0 Disk /dev/lv4drbd/vmconfig /dev/lv4drbd/vmconfig Meta-disk internal internal Syncer Rate 500M(Advanced Config 250KB) DRBD (vmimages) DRBDvmimages Node 1 Node 2 vmimages Name xen1 xen2 Address:Port 192.168.0.1:7790 192.168.0.2:7790 Device /dev/drbd1 /dev/drbd1 Disk /dev/lv4drbd/vmimages /dev/lv4drbd/vmimages Meta-disk internal internal Syncer Rate 500M(Advanced Config 250KB) 11
DRBD (1) DRBD (2) 12
DRBD /DRBD 1. /etc/drbd.conf 2. 3. DRBD xen1# service drbd start xen2# service drbd start 4. xen1# drbdadm create-md vmconfig xen1# drbdadm create-md vmimages xen2# drbdadm create-md vmconfig xen2# drbdadm create-md vmimages DRBD 1. DRBD xen1# drbdadm -- --overwrite-data-of-peer primary all 2. DRBDUpToDate xen1# service drbd status xen2# service drbd status 3. /dev/drbd0 /dev/drbd1 xen1# mkfs t xfs /dev/drbd0 xen1# mkfs t xfs /dev/drbd1 4. /DRBD # service drbd stop 13
DRBD 1. CUI # crm config 2. DRBD 1 vmconfig primitive vmconfig ocf:heartbeat:drbd params drbd_resource=vmconfig op monitor role=master interval=60s timeout=30s op monitor role=slave interval=60s timeout=30s ms ms-vmconfig vmconfig meta clone-max=2 notify=true globally-unique=false target-role=stopped primitive fs-vmconfig ocf:heartbeat:filesystem params fstype=xfs directory=/etc/xen/vm device=/dev/drbd0 meta target-role=stopped order ms-vmconfig-before-fs-vmconfig mandatory: msvmconfig:promote fs-vmconfig:start colocation fs-vmconfig-on-ms-vmconfig inf: fs-vmconfig msvmconfig:master commit quit 14
vmimages primitive vmimages ocf:heartbeat:drbd params drbd_resource=vmimages op monitor role=master interval=60s timeout=30s op monitor role=slave interval=60s timeout=30s ms ms-vmimages vmimages meta clone-max=2 notify=true globally-unique=false target-role=stopped primitive fs-vmimages ocf:heartbeat:filesystem params fstype=xfs directory=/var/lib/xen/images device=/dev/drbd1 meta target-role=stopped order ms-vmimages-before-fs-vmimages mandatory: msvmimages:promote fs-vmimages:start colocation fs-vmimages-on-ms-vmimages inf: fs-vmimages msvmimages:master commit quit DRBD 1. GUI # crm_gui 2. Connection 3. DRBD 4. master 5. df /dev/drbd0 /etc/xen/vm /dev/drbd1 /var/lib/xen/images 15
1. [YaST] [ ] [] 2. sles11-ha OS:SLES11 Windows CUI # crm config sles11-ha 1 16
primitive sles11_ha ocf:heartbeat:xen params name="sles11- ha" xmfile="/etc/xen/vm/sles11-ha op monitor interval="10s op start interval="0s" timeout="30s op stop interval="0s" timeout="300s" colocation sles11_ha-with-fs-vmconfig inf: sles11_ha fsvmconfig order sles11_ha-after-fs-vmconfig inf: fs-vmconfig:start sles11_ha:start colocation sles11_ha-with-fs-vmimages inf: sles11_ha fsvmimages order sles11_ha-after-fs-vmimages inf: fs-vmimages:start sles11_ha:start commit quit HA 1. 2. /GUI 3. OFF 4. 17
75% GUI 100% GUI CUI http://virtualtech.jp/mailmagazine/ 18
37 38 19
SSD A. SSD B. SSD 40 20
TPC-B 41 TPC-B 42 21
SAS 2.5 36.4GB 15krpm 2 SSD Intel X25-E(SLC) 32GB 2 RAIDOff On/Off FC SAN HP MSA1000 SCSI 146GB 10krpm 14RAID 5 512MB Cache Off R50%/W50% Cache On R0%/W100% PostgreSQL 8.3.7 pgbench -c 20 -t 3000 20 1120 43 CPU 4% CPUCPU Opteron 8VM CPU68%4 Opteron 2435 (2.6GHz) Xeon X5570 (2.93GHz HT On) Xeon X5570 (2.93GHz HT Off) 8VM 1553.1 68.6% 2175.3 96.0% 2180.3* 96.2% 12VM 2265.4* 100% 2207.5 97.4% 2171.8 95.9% 16VM 2249.1 99.3% 2248.7* 99.3% 2166.8 95.6% 44 22
Opteron Xeon16%18% Hyper Threading (3.8%:9.4%) Opteron 2435 (2.6GHz) Xeon X5570 (2.93GHz HT On) Xeon X5570 (2.93GHz HT Off) 8VM 200W 87.3% 259W 113.1% 253W* 110.5% 12VM 229W* 100% 272W 118.8% 258W 112.7% 16VM 229W 100% 279W* 121.8% 255W 111.4% 45 23