HowTo create KVM Virtual Machines on the cluster
- Details Hits: 12282
This is a howto for creating KVM virtual machines on the NCIT-Cluster. You can run any virtual machine that has a qcow2 disk and you can use how many processors and memory you wish without disturbing the other users.Because we're using KVM you will have VNC access to the virtual machine and an IP in 10.42.X.X/16.
Objectives
You have a lab virtual machine and you want to run several instances of that virtual machine, and after your you're done, you want to revert to the main virtual machine. You can achieve this by using a master vm instance. You setup this lab into this master instance, shutdown the virtual machine and run as many copies you need by using KVMs copy-on-write options. KVM will use your harddisc as a readonly file, and write all diffs to another file. If you need to update the lab machine, shutdown all instances, delete the instance files, and just update the master machine and you're done.
Virtual Infrastructure Architecture
Basically you use the SunGridEngine batch system to reserve resources (cpu and memory). All virtual machines are hosted on LustreFS, so you have an infiniband connection on the opteron nodes and 4x1Gb maximal total throughput if running on the other nodes. Total cores available: 380 (for now).
[alexandru.herisanu@fep-53-1 ~]$ qstat -g c
all.q 0.52 0 0 180 380 200 0
ibm-nehalem.q -NA- 0 0 0 0 0 0
ibm-opteron.q 0.77 120 0 36 168 0 12
ibm-quad.q 0.34 80 0 144 224 0 0
Setup
First of all you need to be in the kvm-user group. There are some templates available, but you can use any KVM machine. The templates are located into /d02/home/ncit-cluster/kvm-templates.
There are two available templates:
- Oracle RHEL 5.2 64 bit
- Debian 5 with two OpenVZ containers
Copy one of those to your home folder ~/LustreFS. This wil copy it into your Lustre share (email us if you can not acces it).
It should look like this:
[alexandru.herisanu@fep-53-1 ~]$ id
uid=9552(alexandru.herisanu) gid=9000(profi) groups=514(app-gaussian03),515(vtune),518(kvm-user),9000(profi)
[alexandru.herisanu@fep-53-1 ~]$ cd LustreFS/virtual_machines/db2-image/
[alexandru.herisanu@fep-53-1 db2-image]$ ls
db2-hda.qcow2 hda-inst_ABDLab.qcow2
db2-hdb.qcow2 hdb-inst_ABDLab01.qcow2
db2-image.sh hdb-inst_ABDLab02.qcow2
Enterprise-R5-U5-Server-x86_64-dvd.iso hdb-inst_ABDLab.qcow2
hda-inst_ABDLab01.qcow2 out
hda-inst_ABDLab02.qcow2 start_machines.sh
There are several important files:
- db2-hda.qcow2 - your hdd
- hda-inst_ABDLab01.qcow2 - this is a copy on write instance of your hard disk. db2-hda will be a read-only copy, this file will contain only the diffs. Delete it and you revert you machine to the original one.
- start_machines.sh - your start script
- out - stdout directory for the batch system.
Setup your start_machines.sh startup script.It looks like this:
#!/bin/bash
##
## Master Copy
#
#apprun.sh kvm --queue
This email address is being protected from spambots. You need JavaScript enabled to view it.
--vmname ABDLab --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 0 --master
##
## Slave Copy
#
apprun.sh kvm --queue
This email address is being protected from spambots. You need JavaScript enabled to view it.
--vmname ABDLab01 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 1
apprun.sh kvm --queue
This email address is being protected from spambots. You need JavaScript enabled to view it.
--vmname ABDLab02 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:02 --vncport 1
Let me explain (you only have to write this once):
apprun.sh kvm --queue This email address is being protected from spambots. You need JavaScript enabled to view it. --vmname ABDLab --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 0 --master
- apprun.sh kvm - this is our wrapper script over sge
- --queue: where do you want to submit this machine to. You can use any queue, or you can also specify the host. ibm-quad.q, selects any host from this queue. This email address is being protected from spambots. You need JavaScript enabled to view it. - you want exactly that host.
- --vmname: must have a name
- --cpu: number of cpu's. quads have 2*4 cores, opterons 2*6, nehalem 2*8 cores available
- --memory: quads and opterons have 16Gb of memory, nehalems have 32Gb
- --hda: first harddisk in qcow2 format
- --hdb: [optional] secondary harddisk
- --cdrom: [optional] cdrom iso
- --status: will be uses (maybe)
- --mac: you MUST specify the full mac address. choose it so you don't overlap with others. It's best if you submit the mac range you use, so you get "static" ip addresses through DHCP
- --vncport: you MUST specify this so you do not overlap with anybody. It's from 0-20.
- --master: specifies you want to run the main instance. You will modify the --hda file. If you don't specify this, then a new harddisk is created that only logs the diffs.It's named hda-VMNAME_instance.
How do you connect?
You can connect to fep using SSH, VNC, FreeNX or SSH + X11 forwarding. From then on, if you know the ip of the machine, you can access it from fep, if not, you can connect to the machine using kvm like this:
[alexandru.herisanu@fep-53-1 db2-image]$ cat start_machines.sh | grep -v ^#
apprun.sh kvm --queue
This email address is being protected from spambots. You need JavaScript enabled to view it.
--vmname ABDLab01 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 1
apprun.sh kvm --queue
This email address is being protected from spambots. You need JavaScript enabled to view it.
--vmname ABDLab02 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:02 --vncport 1
This is how my machine startup looks like. I will run two instances on two separate hosts.
[alexandru.herisanu@fep-53-1 db2-image]$ ./start_machines.sh
Checking /export/home/ncit-cluster/prof/alexandru.herisanu/.ncit-cluster/ ...
Using /srv/ncit-cluster/apprun/kvm as environment ...
Running custom run script ...
JobId: 6732
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
6732 0.00000 ABDLab01-K alexandru.he qw 02/24/2011 15:01:26 2
Qsub command:
qsub -q
This email address is being protected from spambots. You need JavaScript enabled to view it.
-pe openmpi 2 -N ABDLab01-KVM -o out -e out -cwd /tmp/tmp.DPDOqGl533
Running script:
module load compilers/gcc-4.1.2
module load mpi/openmpi-1.5.1_gcc-4.1.2
sudo /srv/ncit-cluster/scripts/run-vm.sh --vmname ABDLab01 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 1 --user alexandru.herisanu
Checking /export/home/ncit-cluster/prof/alexandru.herisanu/.ncit-cluster/ ...
Using /srv/ncit-cluster/apprun/kvm as environment ...
Running custom run script ...
JobId: 6733
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
6732 0.00000 ABDLab01-K alexandru.he qw 02/24/2011 15:01:26 2
6733 0.00000 ABDLab02-K alexandru.he qw 02/24/2011 15:01:26 2
Qsub command:
qsub -q
This email address is being protected from spambots. You need JavaScript enabled to view it.
-pe openmpi 2 -N ABDLab02-KVM -o out -e out -cwd /tmp/tmp.rhCEktl561
Running script:
module load compilers/gcc-4.1.2
module load mpi/openmpi-1.5.1_gcc-4.1.2
sudo /srv/ncit-cluster/scripts/run-vm.sh --vmname ABDLab02 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:02 --vncport 1 --user alexandru.herisanu
[alexandru.herisanu@fep-53-1 db2-image]$
Great! They are up and running, and we can see we have those slots reserved. No batch jobs will harm us.
[alexandru.herisanu@fep-53-1 db2-image]$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
6732 0.51409 ABDLab01-K alexandru.he r 02/24/2011 15:01:34
This email address is being protected from spambots. You need JavaScript enabled to view it.
. 2
6733 0.51409 ABDLab02-K alexandru.he r 02/24/2011 15:01:36
This email address is being protected from spambots. You need JavaScript enabled to view it.
. 2
You know where you put the vm, so connect using vncviewer and get your ip. Or connect using your ip directly if you've set it up beforehand into dhcp.
How do I shut down the VM?
You can shutdown the vm from the virtual machine. shutdown - h now
Or you can use qdel
[alexandru.herisanu@fep-53-1 db2-image]$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
6732 0.51409 ABDLab01-K alexandru.he r 02/24/2011 15:01:34
This email address is being protected from spambots. You need JavaScript enabled to view it.
. 2
6733 0.51409 ABDLab02-K alexandru.he r 02/24/2011 15:01:36
This email address is being protected from spambots. You need JavaScript enabled to view it.
. 2
[alexandru.herisanu@fep-53-1 db2-image]$ qdel 6732
alexandru.herisanu has registered the job 6732 for deletion
[alexandru.herisanu@fep-53-1 db2-image]$
[alexandru.herisanu@fep-53-1 db2-image]$
[alexandru.herisanu@fep-53-1 db2-image]$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
6733 0.51409 ABDLab02-K alexandru.he r 02/24/2011 15:01:36
This email address is being protected from spambots. You need JavaScript enabled to view it.
. 2
[alexandru.herisanu@fep-53-1 db2-image]$
Voilla!
Heri