How to run Linux SMP in QEMU on a MIPS64 Release 6 CPU

After publishing my initial article about QEMU, many of you have sent queries about running software written for MIPS64 Release 6 CPUs (MIPS64r6) in QEMU and also about the current state of the MIPSr6 multithreading module implemented in QEMU.

I decided to answer these questions in this post and turn it into a tutorial on how to configure and build 64-bit Linux from scratch and run it on the QEMU simulator. In addition, I will be using a MIPS64r6 processor supporting VPs (Virtual Processors) which are part of MIPSr6 multithreading.

The MIPS Coherent Processing System (CPS) is one of key features needed to be emulated by QEMU to allow Linux to bring up VPs. Although implementing this feature is still a work in progress, we can have an early trial of what is available today.

The configuration and build process described in this short tutorial are simple and it should take less than half an hour on modern desktops. We will use the 4.7-rc1 version of Linux kernel (it is advisable to use final v4.7 once it becomes available) and the stable 2016.02 version of Buildroot to generate the root filesystem and the early available QEMU from the Codescape MIPS SDK.

Setting up QEMU

First of all, we need the platform on top of which we will run our operating system. We will use QEMU to build a virtual hardware platform that includes a MIPS CPU:

$ wget https://github.com/prplfoundation/qemu/archive/rel/2.5.0.2.0.tar.gz
$ tar -zxf 2.5.0.2.0.tar.gz
$ mkdir build && cd build
$ ../qemu-rel-2.5.0.2.0/configure --target-list=mips64el-softmmu
$ make -j8

If everything went fine, then the ./mips64el-softmmu/qemu-system-mips64el binary should appear. Our platform is now ready.

Root filesystem by Buildroot

Buildroot makes the process of building Linux for embedded systems very simple. It’s highly configurable but for our purpose we will use it only for generating a minimal root filesystem.

$ wget https://buildroot.org/downloads/buildroot-2016.02.tar.gz
$ tar -zxf buildroot-2016.02.tar.gz && cd buildroot-2016.02
$ make qemu_mips64el_malta_defconfig
$ make menuconfig

We have just applied the default configuration for Malta board, setting a little endian MIPS64 target. However, some further customization is needed; this is what you see below:

Target options  ---> Target Architecture Variant (mips 64r6) [ ] Use soft-float
Toolchain  ---> Toolchain type (External toolchain) Toolchain (Codescape IMG GNU Linux Toolchain 2015.10)
System configuration  ---> [*] Run a getty (login prompt) after boot  ---> (tty0) TTY port
Kernel  ---> [ ] Linux Kernel
Target packages  ---> Debugging, profiling and benchmark  ---> [*] stress-ng
Shell and utilities  ---> [*] screen
System tools  ---> [*] htop

I selected a target, an external toolchain and changed the TTY port. Also, I disabled building the kernel (as this will be done separately) and added just three target packages for the final test where we will stress the system and watch its current load. Since the configuration is done, we can start building:

$ make

Buildroot will download and build everything for you. After a few minutes the rootfs image should be available at ./output/images/rootfs.ext2.

The Linux kernel

Download the kernel and set the needed environment variables (we will use the toolchain which has already been downloaded by Buildroot):

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux && git checkout v4.7-rc1
$ export ARCH=mips
$ export CROSS_COMPILE=<buildroot-dir>/output/host/opt/ext-toolchain/bin/mips-img-linux-gnu-
$ make malta_defconfig
$ make menuconfig

Here are some additional changes I made:

CPU selection  ---> CPU type (MIPS64 Release 6)
Kernel type  ---> Kernel code model (64-bit kernel) [*] MIPS Coherent Processing System support [*] Support for the MIPS SIMD Architecture
Device Drivers  ---> Generic Driver Options  ---> [*]   Automount devtmpfs at /dev, after the kernel mounted the rootfs Graphics support  ---> <*> Direct Rendering Manager (XFree86 4.1.0 and higher DRI support)  ---> <*> DRM Support for bochs dispi vga interface (qemu stdvga) Console display driver support  ---> [*] VGA text console [*] Bootup logo  ---> Virtio drivers  ---> <*> PCI driver for virtio devices <*> Virtio input driver

I selected CPU features and enabled VGA text console as well as Virtio for the keyboard. And finally:

$ make –j8

Now we just need to wait a few minutes until vmlinux image is built.

Does it work?

At this point we have everything we need to run our operating system. We can skip building the bootloader; if it is not provided, then QEMU will determine the kernel’s entry point address and it will fill Malta’s flash memory with a pseudo-bootloader before the start of the guest execution This essentially contains a jump to the kernel.

Here is how we start our system:

$ qemu-system-mips64el -cpu I6400 -M malta -smp 8 -vga std –serial stdio -device virtio-keyboard-pci -kernel vmlinux -drive file=rootfs.ext2,format=raw -append "root=/dev/sda clocksource=GIC console=tty0"

After a few seconds, the command prompt is displayed. Now it is time for the final test which is stressing our system and watching the load on the VPs. For that, I used the stress-ng and htop apps which we included into the rootfs:

$ stress-ng --cpu 2

As we can see, the two VPs are fully utilised. And now the last test where we will try to hog all VPs:

$ stress-ng --cpu 8

And there it is! I hope you’ve enjoyed my tutorial. Let me know if you have any questions or suggestions in the comment box below.

If you are interested in the latest news and updates from Imagination, make sure to follow us on Twitter (@ImaginationTech, @MIPSguru, @MIPSdev), LinkedIn, Facebook and Google+.