So, in amongst my pile of crusty old hardware is the old netbook I used to use in the latter part of my univerity days. It is a Lemote Yeeloong, and sports a ~700MHz Loongson 2F CPU (MIPS III little endian ISA) and 1GB RAM.
Back in the day it was a brilliant little machine. It came out of the box running a localised (for China) version of Debian, and had pretty much everything you’d need. I natually repartitioned the machine, setting up Gentoo and I had a separate partition for Debian, so I could actually dual-boot between them.
Fast forward 10 years, the machine runs, but the battery is dead, and Debian no longer supports MIPS-III machines. Debian Jessie does, but Stretch, likely due for release some time this year, will not, if you haven’t got a CPU that supports mips32r2 or mips64r2, you’re stuffed.
I don’t want to throw this machine away. Being as esoteric as it is, it is an unlikely target for theft, as to the casual observer, it’ll just be “some crappy netbook”. If someone were to try and steal it, there’s a very high probability I’ll recover it with my data because the day its PMON2000 boot firmware successfully boots a x86-64 OS like Ubuntu or Windows without the assistance of a VM of some kind would be the day Satan puts a requisition order in for anti-freeze and winter mittens.
My use case is for a machine I can take with me on the bicycle. My needs aren’t huge: I won’t be playing video on this thing, it’ll be largely for web browsing and email. The web browser needs to support JavaScript, so that rules out options like ELinks or Dillo, my preferred browser is Firefox but I’ll settle for something Webkit-based if that’s all that’s out there.
So what operating systems do I have for a machine that sports a MIPS-III CPU and 1GB RAM? Fedora has a MIPS port, but that, like Debian, is for the newer MIPS systems. Arch Linux too is for newer architectures.
I could bootstrap Alpine Linux… and maybe that’s worth looking into, they seem to be doing some nice work in producing a small and capable Linux distribution. They don’t yet support MIPS though.
Linux From Scratch is an option, if a little labour intensive. (Been there, done that.)
OpenBSD directly supports this machine, and so I gave OpenBSD 6.0 a try. It’s a very capable OS, and while it isn’t Linux, there isn’t much that an experienced Linux user like myself needs to adapt to in order to effectively use the OS. pkgsrc is a great asset to OpenBSD, with a large selection of pre-built packages already available. Using that, it is possible to get a workable environment up and running very quickly. OpenBSD/loongson uses the n64 ABI.
Due to licensing worries, they use a particularly old version of binutils as their linker and assembler. The plan seems to be they wish to wean themselves off the GNU toolchain in favour of LLVM. At this time though, much of the system is built using the GNU toolchain with some custom patches. I found that, on the Yeeloong, 1GB RAM was not sufficient for compiling LLVM, even after adding additional swap files, and some packages I needed weren’t available in pkgsrc, nor would they build with the version of GNU tools available.
Maybe as they iron out the kinks in their build environment with LLVM, this will be worth re-visiting. They’ve done a nice job so far, but it’s not quite up to where I need it to be.
Gentoo actually gives me the choice of two possible ABIs: o32 and n32. o32 is the old 32-bit ABI, and suffers a number of performance problems, but generally works. It’s what Debian Jessie and earlier supplies, and what their mips32 port will produce from Stretch onwards.
n32 is the MIPS equivalent of what some of you may know as x32 on AMD64 platforms, it is a 32-bit environment with 64-bit long pointers… the idea being that very few applications actually benefit from the use of 64-bit data types, and so the usual quantities like int and long remain the same as what they’d be on o32, saving memory. The long long data type gets a boost because, although “32-bit”, the 64-bit operations are still available for use.
The trouble is, some applications have problems with this mode. Either the code sees “mips64” in the CHOST and assumes a full 64-bit system (aka n64), or it assumes the pointers are the same width as a long, or the build system makes silly assumptions as to where things get put. (virtualenv comes to mind, which is what started me on this journey. The same problem affects x32 on AMD64.)
So I thought, I’d give n64 a try. I’d see if I can build a cross-compiler on my AMD64 host, and bootstrap Gentoo from that.
Step 1: Cross-compiler
For the cross-compiler, Gentoo has a killer feature that I have not seen in too many other distributions: crossdev. This is a toolchain build tool that can generate cross-compiler toolchains for most processor architectures and environments.
This is installed by running emerge sys-devel/crossdev.
A gotcha with hardened
I run “hardened” AMD64 stages on my machines, and there’s a little gotcha to be aware of: the hardened USE flag gets set by crossdev, and that can cause fun and games if, like on MIPS, the hardening features haven’t been ported. My first attempt at this produced a n64 userland where pretty much everything generated a segmentation fault, the one exception being Python 2.7. If I booted with init=/bin/bash (or init=/bin/bb), my virtual environment died, if I booted with init=/usr/bin/python2.7, I’d be dropped straight into a Python shell, where I could import the subprocess module and try to run things.
Cleaning up, and forcing crossdev to leave off hardened support, got things working.
Building the toolchain
With the above gotcha in mind:
# crossdev --abis n64 \
--env 'USE="-hardened"' \
-s4 -t mips64el-unknown-linux-gnu
The --abis n64 tells crossdev you want a n64 ABI toolchain, and the --env will hopefully keep the hardened flag unset. Failing that, try this:
# cat > /etc/portage/package.use/mips64 <<EOF
cross-mips64el-unknown-linux-gnu/binutils -hardened
cross-mips64el-unknown-linux-gnu/gcc -hardened
cross-mips64el-unknown-linux-gnu/glibc -hardened
EOF
If you want a combination of specific toolchain components to try, I’m using:
- Binutils: 2.28
- GCC: 5.4.0-r3
- glibc: 2.25
- headers: 4.10
Step 2: Checking our toolchain
This is where I went wrong the first time, I tried building the entire OS, only to discover I had wasted hours of CPU time building non-functional binaries. Save yourself some frustration. Start with a small binary to test.
A good target for this is busybox. Run mips64el-unknown-linux-gnu-emerge busybox, and wait for a bit.
When it completes, you should hopefully have a busybox binary:
RC=0 stuartl@beast ~ $ file /usr/mips64el-unknown-linux-gnu/bin/busybox
/usr/mips64el-unknown-linux-gnu/bin/busybox: ELF 64-bit LSB executable, MIPS, MIPS-III version 1 (SYSV), statically linked, for GNU/Linux 3.2.0, stripped
Testing busybox
There is qemu-user-mips64el, but last time I tried it, I found it broken. So an easier option is to use real hardware or QEMU emulating a full system. In either case, you’ll want to ensure you have your system-of-choice running with a working 64-bit kernel already, if your real hardware isn’t already running a 64-bit Linux kernel, use QEMU.
For QEMU, the path-of-least-resistance I found was to use Debian. Aurélien Jarno has graciously provided QEMU images and corresponding kernels for a good number of ports, including little-endian MIPS.
Grab the Wheezy disk image and the corresponding kernel, then run the following command:
# qemu-system-mips64el -M malta \
-kernel vmlinux-3.2.0-4-5kc-malta \
-hda debian_wheezy_mipsel_standard.qcow2 \
-append "root=/dev/sda1 console=ttyS0,115200" \
-serial stdio -nographic -net nic -net user
Let it boot up, then log in with username root, password root.
Install openssh-client and rsync (this does not ship with the image):
# apt-get update
# apt-get install openssh-client rsync
Now, you can create a directory, and pull the relevant files from your host, then try the binary out:
# mkdir gentoo
# rsync -aP 10.0.2.2:/usr/mips64el-unknown-linux-gnu/ gentoo/
# chroot gentoo bin/busybox ash
With luck, you should be in the chroot now, using Busybox.
Step 3: Building the system
Having done a “hello world” test, we’re now ready to build everything else. Start by tweaking your /usr/mips64el-unknown-linux-gnu/etc/portage/make.conf to your liking then adjust /usr/mips64el-unknown-linux-gnu/etc/portage/make.profile to point to one of the MIPS profiles. For reference, on my system:
RC=0 stuartl@beast ~ $ ls -l /usr/mips64el-unknown-linux-gnu/etc/portage/make.profile
lrwxrwxrwx 1 root root 49 May 1 09:26 /usr/mips64el-unknown-linux-gnu/etc/portage/make.profile -> /usr/portage/profiles/default/linux/mips/13.0/n64
RC=0 stuartl@beast ~ $ cat /usr/mips64el-unknown-linux-gnu/etc/portage/make.conf
CHOST=mips64el-unknown-linux-gnu
CBUILD=x86_64-pc-linux-gnu
ARCH=mips
HOSTCC=x86_64-pc-linux-gnu-gcc
ROOT=/usr/${CHOST}/
ACCEPT_KEYWORDS="mips ~mips"
USE="${ARCH} -pam"
CFLAGS="-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
FEATURES="-collision-protect sandbox buildpkg noman noinfo nodoc"
# Be sure we dont overwrite pkgs from another repo..
PKGDIR=${ROOT}packages/
PORTAGE_TMPDIR=${ROOT}tmp/
ELIBC="glibc"
PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"
#PORTDIR_OVERLAY="/usr/portage/local/"
Now, you should be ready to start building:
# mips64el-unknown-linux-gnu-emerge -e \
--keep-going -j6 --load-average 12.0 @system
Now, go away, and do something else for several hours. It’ll take that long, depending on the speed of your machine. In my case, the machine is an AMD Phenom II x6 with 8GB RAM, which was brand new in 2010. It took a good day or so.
Step 4: Testing our system
We should have enough that we can boot our QEMU VM with this image instead. One way of trying it would be to copy across the userland tree the same way we did for pulling in busybox and chrooting back in again.
In my case, I took the opportunity to build a kernel specifically for the VM that I’m using, and made up a disk image using the new files.
Building a kernel
Your toolchain should be able to cross-build a kernel for the virtual machine. To get you started, here’s a kernel config file. Download it, decompress it, then drop it into your kernel source tree as .config.
Having done that, run make olddefconfig ARCH=mips to set the defaults, then make menuconfig ARCH=mips and customise to your hearts content. When finished, run make -j6 vmlinux modules CROSS_COMPILE=mips64el-unknown-linux-gnu- to build the kernel and modules.
Finally, run make modules_install firmware_install INSTALL_MOD_PATH=$PWD/modules CROSS_COMPILE=mips64el-unknown-linux-gnu- to install the kernel modules and firmware into a convenient place.
Making a root disk
Create a blank, raw disk image using qemu-img, then partition it as you like and mount it as a loopback device:
# qemu-img create -f raw gentoo.raw 8G
# fdisk gentoo.raw
(do your partitioning here)
# losetup -P /dev/loop0 $PWD/gentoo.raw
Now you can format the partitions /dev/loop0pX as you see fit, then mount them in some convenient place. I’ll assume that’s /mnt/vm for now. You’re ready to start copying everything in:
# rsync -aP /usr/mips64el-unknown-linux-gnu/ /mnt/vm/
# rsync -aP /path/to/kernel/tree/modules/ /mnt/vm/
You can use this opportunity to make some tweaks to configuration files, like updating etc/fstab, tweaking etc/portage/make.conf (changing ROOT, removing CBUILD), and setting up a getty on ttyS0. I also like to symlink lib to lib64 in non-multilib environments such as this: Don’t symlink lib and lib64! See below.
# cd /mnt/vm
# mv lib/* lib64
# rmdir lib
# ln -s lib64 lib
# cd usr
# mv lib/* lib64
# rmdir lib
# ln -s lib64 lib
When you’re done, unmount.
First boot
Run QEMU with the following arguments:
# qemu-system-mips64el -M malta \
-kernel /path/to/your/kernel/vmlinux \
-hda /path/to/your/gentoo.raw \
-append "root=/dev/sda1 console=ttyS0,115200 init=/bin/bash" \
-serial stdio -nographic -net nic -net user
It should boot straight to a bash prompt. Mount the root read/write, and then you can make any edits you need to do before boot, such as changing the root password. When done, re-mount the root as read-only, then exec /sbin/init.
# mount / -o rw,remount
# passwd
… etc
# mount / -o ro,remount
# exec /sbin/init
With luck, it should boot to completion.
Step 5: Making the VM a system service
Now, it’d be real nice if libvirt actually supported MIPS VMs, but it doesn’t appear that it does, or at least I couldn’t get it to work. virt-manager certainly doesn’t support it.
No matter, we can make do with a telnet console (on loopback), and supervisord to daemonise QEMU. I use the following supervisord configuration file to start my VMs:
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[program:qemu-mips64el]
command=/usr/bin/qemu-system-mips64el -cpu MIPS64R2-generic -m 2G -spice disable-ticketing,port=5900 -M malta -kernel /home/stuartl/kernels/qemu-mips/vmlinux -hda /var/lib/libvirt/images/gentoo-mips64el.raw -append "mem=256m@0x0 mem=1792m@0x90000000 root=/dev/sda1 console=ttyS0,115200" -chardev socket,id=char0,port=65223,host=::1,server,telnet,nowait -chardev socket,id=char1,port=65224,host=::1,server,telnet,nowait -serial chardev:char0 -mon chardev=char1,mode=readline -net nic -net bridge,helper=/usr/libexec/qemu-bridge-helper,br=br0
The following creates two telnet sockets, port 65223 is the VM’s console, 65224 is the QEMU control console. The VM has the maximum 2GB RAM possible and uses bridged networking to the network bridge br0. There is a graphical console available via SPICE.
All telnet and SPICE interfaces are bound to loopback, so one must use SSH tunnelling to reach those ports from another host. You can change the above command line to use VNC if that’s what you prefer.
At this point, the VM should be able to boot on its own. I’d start with installing some basic packages, and move on from there. You’ll find the environment is very sparse (my build had no Perl binary for example) but the basics for building everything should be there.
You may also find that what is there, isn’t quite installed right… I found that sshd wasn’t functional due to missing users… a problem soon fixed by doing an emerge -K openssh (the earlier step will have produced binary packages).
In my case, that’s installing a decent text editor (vim) and GNU screen so I can start a build, then detach. Lastly, I’ll need catalyst, which is Gentoo’s release engineering tool.
At the moment, this is where I’m at. GNU screen has indirectly pulled in Perl as a dependency, and that is building as I type this. It is building faster than the little netbook does, and I have the bonus that I can throw more RAM at the problem than I can on the real hardware. The plan from here:
- emerge -ek @system, to build everything that got missed before.
- ROOT=/tmp/seed emerge -eK @system, to bundle everything up into a staging area
- populating /tmp/seed/dev with device files
- tar-ing up /tmp/seed to make my initial “seed” stage for
catalyst.
- building the first n64 stages for Gentoo using
catalyst
- building the packages I want for the netbook in a
chroot
- transferring the
chroot to the netbook
Symlinking lib and lib64… don’t do it!
So, I was doing this years ago when n32 was experimental. I recall it being necessary then as this was before Portage having proper multilib support. The earlier mipsel n32 stages I built, which started out from kanaka‘s even more experimental multilib stages, required this kludge to work-around the lack of support in Portage.
Portage has changed, it now properly handles multilib, and so the symlink kludge is not only not necessary, it breaks things rather badly, as I discovered. When packages merge files to /lib, rather than following the symlink, they’ll replace it with a directory. At that point, all hell breaks loose, because stuff that “appeared” in /lib before is no longer there.
I was able to recover by rsync-ing /lib64 to /lib, which isn’t a pretty solution, but it’ll be enough to get an initial “seed” stage. Running that seed stage through Catalyst will clean up the remnants of that bungle.
Recent Comments