With dd
If the new disks have the same size (or a bigger one but you don't care if you lose extra space), you can copy disks using dd.
1) backup node
2) failover all vms to their secondary node. move failover IPs if needed. The route will be updated by ospf.
3) down the node
4) boot current node and new one on rescue system
5) on current node, start disc sync using (in a screen or with nohup) :
dd if=/dev/sda | ssh new.node "dd of=/dev/sda" dd if=/dev/sdb | ssh new.node "dd of=/dev/sdb"
6) When sync are finished, fsck /dev/sda1 on new node and any extra partitions
fsck -f /dev/sda1
7) mount /dev/sda1 somewhere, and update /etc/network/interfaces with new server IP (check ovh interface) 7bis) update DNS (z2-X.pokersource.info to use new IP)
8) start new node, check it works
9) migrate failover IP (z2-X1.pokersource.info) from old node to new one.
10) check node is back into the cluster
11) redist ganeti config :
gnt-cluster redist-conf
12) make a "gnt-cluster verify", it will display lot of :
Thu Dec 3 09:34:05 2009 * Verifying node z2-1.host.gnt (master candidate) Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 0 of instance home.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 1 of instance rheincode.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 2 of instance nagios.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 3 of instance trac.dachary.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 4 of instance trac.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 5 of instance trac.fsffrance.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 6 of instance z2work.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 7 of instance pm-monitor.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 9 of instance booken.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 10 of instance tsunami.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 11 of instance p4m.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 12 of instance proxy3.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 13 of instance cspoker-bot.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 14 of instance pioneer.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 15 of instance drupal-z2.pokersource.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 16 of instance proxy2.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 18 of instance x34xn.binbang.vm.gnt is not active Thu Dec 3 09:34:05 2009 - ERROR: drbd minor 22 of instance trac.binbang.vm.gnt is not active
all these instances, if started during node swap, will need a full resync of the drbd device. To do this without shutting down the vm :
#To remount the drbd device on slave
gnt-instance activate-disk <yourinstance>
Drbd should automatically sync the slave with the changes, to check this, you can use, for example :
watch -n 1 cat/proc/drbd
on new node, you should see a short sync each time you activate a disk.
Retry a "gnt-cluster verify" after that, you shouldn't have any error in "verifying z2-X.host.gnt" section.
13) switch back the vm on the node.
