1 proxy running trash 2 jaula running keep/up jaula.dachary.org jaula.dachary.vm.gnt + http 5 earthquake running keep/up earthquake.pokersource.info earthquake.pokersource.vm.gnt + http 6 jspoker running keep/up jspoker.pokersource.info jspoker.pokersource.vm.gnt + http 7 packaging-farm running keep/up packaging-farm.dachary.org packaging-farm.dachary.vm.gnt + http/rsync 8 pokerconf2009 running keep/up pokerconf2009.pokersource.info pokerconf2009.pokersource.vm.gnt + http 9 pokersource running keep/up pokersource.pokersource.info pokersource.pokersource.vm.gnt + http 10 trau running keep/up trau.pokersource.info trau.ma trau.pokersource.vm.gnt + http 11 dnsslave running trash 14 ns10 running keep/prod ns10.fsffrance.org ns10.fsffrance.vm.gnt + http,dns 15 dedipoker0002 running keep/up pokersource.eu www.pokersource.eu eu.pokersource.vm.gnt + http,1938* (check firewall) 16 drupal-dev running keep/prod drupal-dev.pokersource.info drupal-dev.pokersource.vm.gnt + http - cspoker-bot shut off keep/down cspoker-bot.pokersource.info cspoker-bot.pokersource.vm.gnt + http - elgg shut off keep/up elgg.dachary.org elgg.dachary.vm.gnt + http + http://nagios.fsffrance.org/cgi-bin/nagios2/extinfo.cgi?type=1&host=elgg.rentre.tld - opensocial shut off trash - pokme shut off trash - scratch-virgin shut off trash - webkit-server shut off keep/down webkit-server.dachary.org webkit-server.dachary.vm.gnt + http + http://nagios.fsffrance.org/cgi-bin/nagios2/extinfo.cgi?type=1&host=webkit-server.dachary.rentre.tld
all the vm off can be archived.
memory used :
- proxy : 128m
- jaula : 128m
- earthquake: 128m
- jspoker : 1g
- packaging-farm : 128m
- pokerconf2009 : 128m
- pokersource : 512m
- trau : 256m
- dnsslave : 256m
- ns10 : 1g => change to 256m
- dedipoker0002 : 256m
- drupal-dev : 2g
disk :
bayou all -wi-a- 150.00G binbang all -wi-ao 100.00G cspoker-bot all -wi-ao 10.00G cspoker-drbd all -wi-ao 128.00M dedipoker all -wi-ao 10.00G dedipoker-drbd all -wi-ao 128.00M dedipoker0002 all -wi-ao 3.05G dnsslave all -wi-ao 1.00G drupal-dev all owi-ao 10.00G drupal-dev-drbd all -wi-ao 128.00M drupal-dev-remove-me all -wi-a- 10.00G drupal-dev-virgin all -wi-a- 10.00G drupal-dev-virgin-remove-me all -wi-a- 10.00G earthquake all -wi-ao 3.91G elgg all -wi-a- 10.00G jaula all -wi-ao 2.00G jaula-drbd all -wi-a- 128.00M jspoker all -wi-ao 18.55G jspoker-backup all -wi-a- 18.55G mediagateusa all -wi-ao 10.00G mediagateusa-drbd all -wi-ao 128.00M neufbox all -wi-a- 10.00G ns10 all -wi-ao 1.95G opensocial all -wi-a- 10.00G packaging-farm all -wi-ao 18.55G pokerconf2009 all -wi-ao 2.44G pokersource all -wi-ao 11.72G pokme all -wi-ao 5.00G pokme-drbd all -wi-ao 128.00M proxy all -wi-ao 1.00G scratch all -wi-a- 10.00G scratch-virgin all -wi-a- 10.00G shindig-1.0 all -wi-a- 10.00G trau all -wi-ao 10.00G webkit-server all -wi-a- 10.00G
drbd devices :
rentre:~# cat /proc/drbd version: 8.0.14 (api:86/proto:86) GIT-hash: bb447522fc9a87d0069b7e14f0234911ebdab0f7 build by phil@fat-tyre, 2008-11-12 16:40:33 #pokme, slave missing (was call) 0: cs:WFConnection st:Primary/Unknown ds:UpToDate/DUnknown A r--- ns:0 nr:0 dw:0 dr:0 al:0 bm:99 lo:0 pe:0 ua:0 ap:0 resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0 #/dev/all/dedipoker, primary is unique, but the vm is migrated. can be stopped 1: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate A r--- ns:0 nr:336 dw:336 dr:0 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0 #mediagateusa, primary on madia, but vm migrated, can be stopped. 2: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate A r--- ns:0 nr:2393600 dw:2393600 dr:0 al:0 bm:190 lo:0 pe:0 ua:0 ap:0 resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 #cspoker-bot, secondary was kutsu 3: cs:WFConnection st:Primary/Unknown ds:UpToDate/DUnknown A r--- ns:0 nr:0 dw:0 dr:0 al:0 bm:78 lo:0 pe:0 ua:0 ap:0 resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 #drupal-dev secondary was kutsu 4: cs:WFConnection st:Primary/Unknown ds:UpToDate/DUnknown A r--- ns:0 nr:0 dw:73517204 dr:9468284 al:23881 bm:23829 lo:0 pe:0 ua:0 ap:0 resync: used:0/61 hits:0 misses:0 starving:0 dirty:0 changed:0 act_log: used:0/127 hits:18355420 misses:25251 starving:0 dirty:1370 changed:23881
IPs :
- 87.98.253.6 for ns10.fsffrance.org. This IP will be *kept* and migrated to z2.
- 91.121.23.117 main rentre ip, will be lost
- 87.98.128.80 rentre1.pokersource.info , http(s) dnat to proxy, rsync/ssh to packaging-farm, 22019 on drupal-dev:ssh, used by both dachary's vms + drupal-dev.
- 87.98.128.95 rentre2.pokersource.info, not used
- 87.98.249.47 bayou2.pokersource.info, http(s)/icmp to proxy, used by all pokersource vms except drupal-dev
Downtimes :
proxy - not moved, only the config
jaula - can be down up to 2 days (dns propagation)
earthquake - can be down up to 2 days (dns propagation)
jspoker - can be down up to 2 days (dns propagation)
packaging-farm - can be down up to 2 days (dns propagation)
pokerconf2009 - can be down up to 2 days (dns propagation)
pokersource - can be down up to 2 days (dns propagation)
trau - can be down up to 2 days (dns propagation)
dnsslave - it is not working atm ('initramfs' message in console), do we migrate it ?
ns10 - not more than 30 minutes of downtime, the ip will be migrated to guarantee this
dedipoker0002 - can be down up to 2 days (dns propagation)
drupal-dev - can be down up to 2 days (dns propagation), if developers need it, they could add an entry in their /etc/hosts.
virtualisation type :
jaula : full virt earthquake : paravirt jspoker : paravirt packaging-farm : paravirt pokerconf2009 : paravirt pokersource : paravirt trau : full virt dnsslave : full-virt (but not working atm) ns10 : paravirt dedipoker0002 : paravirt drupal-dev : full-virt
New vm names :
jaula -> jaula.dachary.vm.gnt earthquake -> earthquake.pokersource.vm.gnt jspoker -> jspoker.pokersource.vm.gnt packaging-farm -> packaging-farm.dachary.vm.gnt pokerconf2009 -> pokerconf2009.pokersource.vm.gnt pokersource -> pokersource.pokersource.vm.gnt trau -> trau.pokersource.vm.gnt dnsslave ? ns10 -> ns10.fsffrance.vm.gnt dedipoker0002 -> dedipoker0002.pokersource.vm.gnt drupal-dev -> drupal-dev.pokersource.vm.gnt
archives
on dirtychip, make a backup of all lvs. list the dirtychip LV that already have backups and for which current LV
- drupal-dev : last backup 2010-01-17-drupal-dev
- jspoker : last backup 2009-07-19-jspoker
- opensocial : last backup 2009-12-22-opensocial
- pokme : a backup was done, but not in any subdir ...
migration
- define all dns on vpn-dns
- define all dhcp configs
- reload dns + dhcp on z2 cluster :
for i in 1 2 3 4 5 6 7 8;do ssh z2-$i "cd /etc/dhcp3 && hg pull -u && /etc/init.d/dhcp3-server restart && rm /var/cache/bind/* && /etc/init.d/bind9 restart";done
- pokersource vms will go on z2-3 (1 failover slot free to migrate bayou2)
- ns10 vm will go on z2-8 (1 failover slot free to migrate ns10.fsffrance)
The full-virt vms should be the easiest ones to migrate.
- on all the vms, check udev is installed, and if it was :
rm -rf /etc/udev/rules.d/*persistent-net.rules
full-virt vm migrations
jaula
- create the vm on z2, with a 2Go disk
- stop jaula vm on rentre :
virsh shutdown jaula
- sync with dd :
z2host:~# ssh rentre.tld "dd if=/dev/all/jaula" | dd of=/dev/drbdXX
- modify instance params :
gnt-instance modify -H kernel_path=,initrd_path= jaula.dachary.vm.gnt
- start the vm :
gnt-instance start jaula.dachary.vm.gnt
- update public dns jaula.dachary.org
- configure proxy to handle jaula.dachary.org, check it works by putting a line in /Etc/hosts
drupal-dev
- create the vm on z2, with a 10Go disk
- stop drupal-dev vm on rentre :
virsh shutdown drupal-dev
- sync with dd *from the drbd*:
z2host:~# ssh rentre.tld "dd if=/dev/drbd4" | dd of=/dev/drbdXX
- modify instance params :
gnt-instance modify -H kernel_path=,initrd_path= drupal-dev.pokersource.vm.gnt
- start the vm :
gnt-instance start drupal-dev.pokersource.vm.gnt
- update public dns drupal-dev.pokersource.info, configure proxy to handle it
- bind a port (22XXX, with XXX last byte of vm's ip) on failover ip
- check ssh and proxy works
- give informations on #pokersource@… to use the new vm without waiting for dns propagation
trau
- create the vm on z2, with a 10Go disk
- stop trau vm on rentre :
virsh shutdown trau
- sync with dd :
z2host:~# ssh rentre.tld "dd if=/dev/all/trau" | dd of=/dev/drbdXX
- modify instance params :
gnt-instance modify -H kernel_path=,initrd_path= trau.pokersource.vm.gnt
- start the vm :
gnt-instance start trau.pokersource.vm.gnt
- update public dns trau.pokersource.info *and* trau.ma, configure proxy to handle it
- check proxy works
dnsslave
Do I migrate it ?
paravirt vm
The goal is to migrate then and convert them into a full virt vm. Same process for :
- earthquake
- jspoker
- packaging-farm
- pokerconf2009
- pokersource
- dedipoker0002
(all but ns10).
- create the vm on z2, check lvs for disk size
- stop vm on rentre :
virsh shutdown <vm>
- mount it on rentre :
mkdir /mnt/vm mount /dev/all/vm /mnt/vm
- mount new vm on z2 :
z2-2:~# gnt-instance shutdown vm z2-2:~# gnt-instance activate-disks vm z2-Y.host.gnt:disk/0:/dev/drbdXX z2-2:~# ssh z2-Y z2-Y:~# kpartx -av /dev/drbdXX z2-Y:~# mkdir /mnt/vm z2-Y:~# mount /dev/mapper/drbdXXp1 /mnt/vm
- rsync from rentre :
z2-Y:~# rsync -avzH --numeric-ids --exclude=/dev --exclude=/proc --exclude=/sys root@rentre.tld:/mnt/vm/ /mnt/vm/
- chroot :
z2-Y:~# chroot /mnt/vm
- into the chroot :
z2-Y:/# cat /etc/network/interfaces ## check it contains : # The loopback network interface auto lo iface lo inet loopback # The primary network interface allow-hotplug eth0 iface eth0 inet dhcp z2-Y:/# vi /etc/fstab # add if not already : proc /proc proc defaults 0 0
- exit chroot
- umount vm :
z2-Y:~# umount /mnt/vm z2-Y:~# kpartx -dv /dev/drbdXX
- start vm :
gnt-instance start vm
- log into it, and install grub / kernel :
apt-get install grub linux-image-2.6.26-2-amd64 mkdir /boot/grub grub-mkdevicemap update-grub grub-install "(hd0)" #To be able reboot from inside the vm etc : apt-get install acpi-support
- modify instance params :
gnt-instance modify -H kernel_path=,initrd_path= trau.pokersource.vm.gnt
- restart the vm :
gnt-instance reboot vm
- it should boot.
- update public dns, configure proxy to handle it
- check proxy works
ns10
The process will be the same as previously, but to keep downtime as short as possible, we will : - snap the lv - make the sync from the lv - configure the new vm - migrate the ip - down the old vm
It's important than nobody add dns records during migration.
- create the vm on z2, check lvs for disk size
- create a snapshot of the lv on rentre :
lvcreate -L1G -s ns10-snapshot /dev/all/ns10
- mount it on rentre :
mkdir /mnt/ns10 mount /dev/all/ns10-snapshot /mnt/ns10
- mount new vm on z2 :
z2-2:~# gnt-instance shutdown ns10.fsffrance.vm.gnt z2-2:~# gnt-instance activate-disks ns10.fsffrance.vm.gnt z2-Y.host.gnt:disk/0:/dev/drbdXX z2-2:~# ssh z2-Y z2-Y:~# kpartx -av /dev/drbdXX z2-Y:~# mkdir /mnt/ns10 z2-Y:~# mount /dev/mapper/drbdXXp1 /mnt/ns10
- rsync from rentre :
z2-Y:~# rsync -avzH --numeric-ids --exclude=/dev --exclude=/proc --exclude=/sys root@rentre.tld:/mnt/ns10 /mnt/ns10/
- chroot :
z2-Y:~# chroot /mnt/ns10
- into the chroot :
z2-Y:/# cat /etc/network/interfaces ## check it contains : # The loopback network interface auto lo iface lo inet loopback # The primary network interface allow-hotplug eth0 iface eth0 inet dhcp z2-Y:/# vi /etc/fstab # add if not already : proc /proc proc defaults 0 0
- exit chroot
- umount vm :
z2-Y:~# umount /mnt/ns10 z2-Y:~# kpartx -dv /dev/drbdXX
- start vm :
gnt-instance start ns10
- log into it, and install grub / kernel :
apt-get install grub linux-image-2.6.26-2-amd64 mkdir /boot/grub grub-mkdevicemap update-grub grub-install "(hd0)" #To be able reboot from inside the vm etc : apt-get install acpi-support
- modify instance params :
gnt-instance modify -H kernel_path=,initrd_path= ns10.fsffrance.vm.gnt
- restart the vm :
gnt-instance reboot ns10
- it should boot.
- When it boot and bind is correctly started :
- configure shorewall to handle 87.98.253.6 and dnat domain,ssh,http to ns10 vm's ip.
- migrate the ip on ovh interface
- Check "telnet 87.98.253.6 53" works :
Trying 87.98.253.6... Connected to 87.98.253.6. Escape character is '^]'.
- check with dig :
dig @87.98.253.6 fsffrance.org
must display a result.
- shutdown vm on rentre :
virsh shutdown ns10
Upgrade /etc/dhcp3/.hg/hgrc
To elect another master by default
z2-2:~# cat /etc/dhcp3/.hg/hgrc [paths] default = /mnt/rentre.tld/etc/dhcp3
install rentre
- Reinstall rentre and add it to z2 using http://trac.dunnewind.net/dunnewind/wiki/GanetiOspfHowto .
At least for all pokersource vms previously migrated :
- move secondary to rentre (gnt-cluster replace-disks -n z2-9.host.gnt <vmname>
- be sure z2-9 proxy is synced with proxy.vm.gnt
- be sure z2-9 shorewall config is sync
- move ip from z2-8 to z2-9
- check everything still works
- failover the vms to z2-9 (gnt-instance failover <vmname>
