Category Archives: KVM

Ubuntu 14.04 QEMU/KVM Gaming Virtual Machine

Hello everyone. Today I want to show you a little project I have been working on for quite a while now. I found this great post a while ago and was pretty amazed by the fact that it is possible to play Current Gen 3D games inside of virtual machines! So I decided to try it on my own to get this working as well with my own setup. I gathered some hardware from ebay and Amazon and started building my test environment. Lets go:

Here is my setup:

Hardware:

Software:

  • Operating System: Ubuntu 14.0.4.1 LTS
  • Kernel: 3.13.0 – generic
  • Qemu: 2.0.0

Virtual Machine:

  • Operating System: Windows 8.1 Enterprise 64bit
  • Driver: AMD Catalyst Packages Omega (14.12) – (Display Driver ver. 14.501.1003)
  • Game: Left 4 Dead 2 (I admit that it is not really Current Gen but you know what I mean)

Prerequisites:

Before you start with the below instructions you need to enable IOMMU in your BIOS. Please make sure that this setting is working. I read that there are motherboards out there with faulty BIOS versions which will not allow you to enable this setting. (Maybe you will need to update your BIOS version so you can use this setting.)

Here is what I did to enable IOMMU on my Asrock 970 Extreme3 2.0 motherboard:

I entered the BIOS/UEFI by hitting DEL and found the setting in Advanced -> North Bridge Configuration -> IOMMU where I enabled it. (This setting will be disabled by default on the most motherboards.) Then I clicked on Exit -> Save Changes and Exit to complete this step.

After you have successfully enabled IOMMU in your BIOS you can proceed by installing Ubuntu.

You can check your Ubuntu and kernel version with the following command:

uname -a

Here are my Ubuntu and kernel version:

Linux test-desktop 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

As you can see I am currently using Ubuntu 14.04.1 LTS and my kernel version is 3.13.0 generic. (I know that 14.04.2 LTS is already out but I haven´t updated my system so far.)

I also installed an OpenSSH Server on my system so I can access my system remotely if I need to. (This step is completely optional.)

sudo apt-get install openssh-server

Installation QEMU/KVM

Now for the real deal. The first thing we need to do is to install QEMU and KVM. Simply run the below command to do so:

sudo apt-get install qemu-kvm

To check which version of QEMU got installed you can run this command:

kvm --version

The QEMU version I have installed at the moment is version 2.0.0 .

Edit modules and bootloader

The next thing you need to do is to edit the file “/etc/modules“. Simply add the modules listed below:


# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

lp
rtc
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_amd

Important: If you are using an Intel CPU you need to replace kvm_amd with kvm_intel.

Ok. Now you need to edit the file “/etc/default/grub“. There you need to find the line below and edit it to look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"

Important: If you are using an Intel CPU you need to replace amd_iommu with intel_iommu.

After you have edited the file it should look like this:


# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Now update Grub by running:

sudo update-grub

After the above command has finished reboot your computer to apply the changes:

sudo reboot

Blacklist Graphics Card

The next thing we need to do is to blacklist the graphics card we want to pass through to our virtual machine so that the host system will not use it for itself.

To get a list of PCI devices run the following command:

lspci -nn

Here is a list of my PCI devices:

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) [1002:5a14] (rev 02)
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) [1002:5a16]
00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) [1002:5a18]
00:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H) [1002:5a1c]
00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40)
00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42)
00:14.1 IDE interface [0101]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller [1002:439c] (rev 40)
00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383] (rev 40)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40)
00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40)
00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
00:15.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]
00:15.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 3) [1002:43a3]
00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 [1022:1600]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 [1022:1601]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 [1022:1602]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 [1022:1603]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 [1022:1604]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 [1022:1605]
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] [1002:6779]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] [1002:aa98]
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] [1002:6779]
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] [1002:aa98]
03:00.0 USB controller [0c03]: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller [1b6f:7052]
06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)

As you can see I have 2 identical graphics cards in my system which have the same identifier (This is normal). The strange thing is even after I had blacklisted them the GUI of my Ubuntu system would still continue working. The resolution dropped and windows started to open slower than before but nothing else. I´m not sure if this will cause issues in the future or not. (If you can use 2 different graphics cards. This will avoid possible issues.)

Ok. All you need to do here is to write down the addresses (02:00.0 and 02:00.1) and the identifiers (1002:6779 and 1002:aa98) of the graphics card you want to blacklist for your host and to pass through to your virtual machine.

Now you need to edit the file “/etc/initramfs-tools/modules” and add the identifiers of your graphics card. It should look like this:

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
pci_stub ids=1002:6779,1002:aa98

After you have done this you need to run the below command to rebuild initramfs:

sudo update-initramfs -u

And of course you need to reboot your computer after the above command has finished:

sudo reboot

After your computer is up and running again run the following command to make sure that your graphics card was successfully blacklisted:

dmesg | grep pci-stub

The command should return something like that:

[ 1.473148] pci-stub: add 1002:6779 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 1.473174] pci-stub 0000:01:00.0: claimed by stub
[ 1.473193] pci-stub 0000:02:00.0: claimed by stub
[ 1.473199] pci-stub: add 1002:AA98 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 1.473206] pci-stub 0000:01:00.1: claimed by stub
[ 1.473215] pci-stub 0000:02:00.1: claimed by stub

As I have mentioned before I use 2 graphics cards with the same identifiers which is the reason why there are 4 addresses listed in the above example. If you use 2 different graphics cards with different identifiers there will only be 2 addresses listed if you blacklist one of them.

If the command does not deliver an output like the above example or nothing at all it means that your attempt at blacklisting your graphics card failed and you need to find out why. Unfortunately I can not give you advice how to troubleshoot this issue since I could not reproduce an issue like this.

Create VFIO config file

Now we will create a file containing the addresses of our graphics card we want to pass through to our virtual machine. To do so we create a new file called “/etc/vfio-pci1.cfg” and enter our addresses.

Important: You need to add “0000:” at the beginning of your addresses to make it work. Unfortunately I do not know why.

Here is an example how this file should look:

0000:02:00.0
0000:02:00.1

Ok. That´s it. Our graphics card is now ready to use.

Create disk for the virtual machine

This step is really simple. We need a virtual disk where we can install our virtual machine. Simply run the below command to create one:

dd if=/dev/zero of=windows1.img bs=1M seek=120000 count=0

This will create a disk with 120GB storage. More than enough for our tests I think.

Create a script for the virtual machine

The next step is to create a script which will allow us to run our virtual machine. I simply used the script I found here and altered it a bit to match my system. You will need to do the same. You will need to replace the addresses and identifiers in the script to make it work. And you will need to alter the paths for the virtual disk we created in the step before and the path for your image file.

Here is my version of the script:


#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
dev="$1"
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
echo $line | grep ^# >/dev/null 2>&1 && continue
vfiobind $line
done

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \
-drive file=/home/test/windows1.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=/home/test/Downloads/Windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-usb -usbdevice host:093a:2510 -usbdevice host:046d:c316 \
-boot menu=on

exit 0

After you have altered the script to match your system you need to run the below command to make the script executable:

sudo chmod 755 /usr/vm1

As you may have noticed I added some USB devices to my virtual machine. A mouse (093a:2510) and a keyboard (046d:c316). So that I will be able to work with the machine properly.

I would advise you to do the same since I tought that this is the easiest way to add USB devices. To find out the identifiers of your USB devices simply run:

lsusb

Here is an example of how the output will look like:

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 004: ID 093a:2510 Pixart Imaging, Inc. Optical Mouse
Bus 001 Device 003: ID 046d:c316 Logitech, Inc. HID-Compliant Keyboard
Bus 001 Device 002: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 002: ID 1058:1021 Western Digital Technologies, Inc. Elements 2TB
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

You can use the mouse and keyboard of your host system like I did. It will not cause errors to pop up but it will make the devices unavailable for the host system. (That´s also one of the reasons why I installed an OpenSSH Server)

After I completed these steps I decided to reboot my system one last time just to be sure that everything works:

sudo reboot

After my computer was up and running again I logged on to my GUI and run the following command in a terminal window on the GUI:

sudo /usr/vm1

Important: I pointed out that I ran the above command on the GUI because I ran into problems when trying to run the command from an SSH session. I got the following error message “Could not initialize SDL (No available video device) – exiting“. I was not able to solve this so far so I decided to start the script on the GUI.

Right after I ran the script a black QEMU window popped up and my mouse and keyboard were passed through to the virtual machine.

The next thing I did was to connect a VGA cable to the VGA port of the graphics card I passed through to the virtual machine. And I was greeted by the Windows 8 Logo right after plugging in the cable.

Virtual Machine Installation

The installation of Windows 8.1 on my virtual machine was exactly the same as installing it on a normal physical computer. I just went through the steps and created a Local Account for my tests.

Driver Installation

Ok. After the virtual machine was up and running I decided to install my graphics card drivers so that I can play games. But first I verified that my graphics card got passed through correctly. So I installed GPU-Z to make sure that the model of my graphics card was recognized correctly. It was. The next step for me was to download the drivers from here. Next I needed some extra steps to get the drivers running.

Here are the steps how I got my drivers running:

  • First I opened Settings -> Control Panel -> System and Security -> Administrative Tools -> Computer Management -> Device Manager -> Display Adapters and right clicked on the Microsoft Basic Display Adapter and clicked on Update Driver Software… -> Search automatically for updated driver software -> This resulted in Windows downloading some drivers and my graphics card was recognized as Radeon HD6450 after the download and installation completed. After that I rebooted the virtual machine.
  • Next I started the exe I downloaded and canceled it after it finished extracting files. Then I returned to Display Adapters in the Device Manager and right clicked on Radeon HD6450 and clicked on Update Driver Software… -> Browse my computer for driver software and I chose the Path where the Driver files were extracted. After the installation my graphics card was recognized as Radeon HD7450. That was not so strange since the description of the graphics card also got messed when I used it on a normal physical computer. Then I rebooted the virtual machine.
  • And finally I started the exe one last time and went through the whole installation like I would do it on a normal physical computer and rebooted one last time.

After all the above steps I had AMD Catalyst installed correctly and I was able to unplug the VGA cable and replace it with an HDMI cable. Now everything worked as I wanted. Now it was testing time.

Important: The installation process of your drivers may differ from mine so don´t be frustrated if the above steps don´t work for you.

Game Installation

Now I needed a game to test this whole setup. So I installed Steam and started to download Left 4 Dead 2 (I thought this game should do a good job for testing). After the download and installation both finished I started the game right away. There was no trouble at all with the game. It started and behaved normally. Sound and Video both worked as expected and as far as I can judge there was no noticeable delay. The only tiny delay I felt was from my mouse and keyboard. The reacted a split second slower than normally I would say but maybe this is because I used the hosts devices or maybe because I did not pass through an USB Controller for the virtual machine. I don´t know for sure why it behaved like this. All in all I was surprised that it all worked quite well.

Summary

Well, this was indeed an interesting project which consumed quite a lot of my time and was sometimes a bit troublesome but I enjoyed working on it and I´m quite happy that I got it running. But I will not simply drop this project as it is now. No. I want to work on it a bit more and I will try to make screenshots for each step as well as a video of the gaming experience (If possible I want to try different graphic cards too). And I would be happy if some of you could post some of their experiences with KVM and QEMU in the comment section if you have tried this too. And I would be happy if you have a look at this awesome post and give these guys some credit for their awesome work.

Well, that´s it for this post. Sorry that this post got quite long this time but I hope it was interesting for you and I hope that I see you again next time 🙂

Sources:

http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

http://www.howtogeek.com/117635/how-to-install-kvm-and-create-virtual-machines-on-ubuntu/

http://ubuntuforums.org/showthread.php?t=2262280

http://apurv.me/vga-passthrough-using-kvm-in-ubuntu-14-04/

http://www.gitztalk.com/posts/2014/06/30/gaming-in-a-vm/

Tagged , , , ,