linux.codehelp.co.uk

Deprecating dpkg-cross

Neil Williams — Sun, 17 Jul 2016 11:06:31 -0100

Deprecating the dpkg-cross binary

After a discussion in the cross-toolchain BoF at DebConf16, the gross hack which is packaged as the dpkg-cross binary package and supporting perl module have finally been deprecated, long after multiarch was actually delivered. Various reasons have complicated the final steps for dpkg-cross and there remains one use for some of the files within the package although not the dpkg-cross binary itself.

2.6.14 has now been uploaded to unstable and introduces a new binary package cross-config, so will spend a time in NEW. The changes are summarised in the NEWS entry for 2.6.14.

The cross architecture configuration files have moved to the new cross-config package and the older dpkg-cross binary with supporting perl module are now deprecated. Future uploads will only include the cross-config package.

Use cross-config to retain support for autotools and CMake cross-building configuration.

If you use the deprecated dpkg-cross binary, now is the time to migrate away from these path changes. The dpkg-cross binary and the supporting perl module should NOT be expected to be part of Debian by the time of the Stretch release.

2.6.14 also marks the end of my involvement with dpkg-cross. The Uploaders list has been shortened but I'm still listed to be able to get 2.6.14 into NEW. A future release will drop the perl module and the dpkg-cross binary, retaining just the new cross-config package.

Moving to Pelican

Neil Williams — Tue, 03 May 2016 23:01:38 -0100

Prompted by Tollef, moving to Hugo, I investigated a replacement blog engine. The former site used Wordpress which is just overhead - my blog doesn't need to be generated on every view, it doesn't need the security implications of yet another website login and admin interface either.

The blog is static, so I've been looking at static generators. I didn't like the look of Hugo and wanted something where the syntax was familiar - so either Jinja2 or ReST.

So, I've chosen Pelican with the code living in a private git repo, naturally. I wanted a generator that was supported in Jessie. I first tried nikola but it turns out that nikola in jessie has syntax changes. I looked at creating backports but then there is a new upstream release which adds a python module not yet in Debian, so that would be an extra amount of work.

Hopefully, this won't flood planet - I've gone through the RSS content to update timestamps but the URLs have changed.

lava.debian.net

Neil Williams — Sat, 06 Feb 2016 14:33:38 +0000

With thanks to Iain Learmonth for the hardware, there is now a Debian instance of LAVA available for use and the Debian wiki page has been updated.

LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. Tests can be simple boot testing, bootloader testing and system level testing. Extra hardware may be required for some system tests. Results are tracked over time and data can be exported for further analysis.

LAVA has a long history of supporting continuous integration of the Linux kernel on ARM devices (ARMv7 & ARMv8). So if you are testing a Linux kernel image on armhf or arm64 devices, you will find a lot of similar tests already running on the other LAVA instances. The Debian LAVA instance seeks to widen that testing in a couple of ways:

A wider range of tests including use of Debian artifacts as well as mainline Linux builds
A wider range of devices by allowing developers to offer devices for testing from their own desks.
Letting developers share local test results easily with the community without losing the benefits of having the board on your desk.

This instance relies on the latest changes in lava-server and lava-dispatcher. The 2016.2 release has now deprecated the old, complex dispatcher and a whole new pipeline design is available. The Debian LAVA instance is running 2015.12 at the moment, I’ll upgrade to 2016.2 once the packages migrate into testing in a few days and I can do a backport to jessie.

What can LAVA do for Debian?

ARMMP kernel testing

Unreleased builds, experimental initramfs testing – this is the core of what LAVA is already doing behind the scenes of sites like http://kernelci.org/.

U-Boot ARM testing

This is what fully automated LAVA labs have not been able to deliver in the past, at least without a usable SD Mux

What’s next

LOTS. This post actually got published early (distracted by the rugby) – I’ll update things more in a later post. Contact me if you want to get involved, I’ll provide more information on how to use the instance and how to contribute to the testing in due course.

Experimenting with LXQt in Debian

Neil Williams — Tue, 29 Dec 2015 16:29:21 +0000

LXQt is a Qt lightweight desktop - the Qt port of LXDE. Packages exist in Debian - albeit without a top level metapackage or task package to make installing it easier. So I wrote up a simple-ish vmdebootstrap call:

$ sudo vmdebootstrap --image lxqt.img --size=5G --package=lxqt-panel \
--package=libqt5xcbqpa5 --package=qterminal --package=openbox \
--package=xdm --package=lxqt-session --package=lxqt-about \
--package=lxqt-policykit --package=lxqt-globalkeys \
--package=lxqt-notificationd --package=lxqt-sudo --package=dbus-x11 \
--package=lxqt-admin --package=lxqt-runner --package=lxqt-config \
--package=task-desktop --package=locales --package=xserver-xorg-core \
--package=oxygen-icon-theme --grub --distribution=unstable \
--mirror=http://mirror.bytemark.co.uk/debian --configure-apt -\
-enable-dhcp --serial-console --sudo --verbose --owner=neil \
--user='neil/neil'

(You'll need to adapt the last two commands to be a real user.)

This uses xdm instead of lxdm as this tests LXQt without having any GTK+ dependencies installed. lxdm does give a nicer experience at the cost of needing GTK+. YMMV.

Note

Note the explicit additions: --package=libqt5xcbqpa5 --package=dbus-x11 - as debootstrap does not follow Recommends, libqt5xcbqpa5 needs to be specified explicitly or the desktop will fail to start. dbus-x11 is also needed to get things working. task-desktop adds the Debian artwork and needs to be in the list of packages passed to debootstrap so that the Recommends of the task packages are not selected. (Note that I have so far failed to get LXQt to use the Debian artwork as a desktop background.)

So, what is it like? Well - alpha is how I might describe it. Not in terms of stability, more in terms of functionality. I do have a second install using lxdm which has been tweaked but it depends on your objective. If your aim is to not have GTK+ but not have KDE, then LXQt is a beginning only. In particular, if you really are intent on not having GTK+ at all, your choice of web browser is somewhat limited, to lynx. (There's no bare Qt file manager in Debian - pcmanfm-qt depends on libfm-modules which uses GTK+ - nor a bare text editor despite this being one of the simplest examples of a QApplication). There is a large gap in the software availability which is Qt but not KDE, despite the power and flexibility of Qt itself. (I've written applications using Qt directly before, it is much more flexible and configurable than GTK+). So there would seem to be a reason why a metapackage and a task package do not yet exist, there is a lot more to do. I'm happy to mix GTK+ applications, so my test environment can use iceweasel, chromium, leafpad and thunar.

Overall, this was an interesting diversion prompted by a separate discussion about the merits and controversies of GTK+, GNOME etc.

I failed to work out why the icon theme works if lxdm was installed but not with xdm (so there's a missing package but I'm not yet sure exactly which), so the screenshot is more bare than I expected.

With iceweasel installed and various other tweaks:

Finally, note #809339 - I have local changes which are being tested to use systemd-networkd but currently the masking of PredictableInterfaceNames as documented does not work, so some editing of /etc/network/interfaces.d/setup (or enable systemd-networkd yourself and add a suitable file to /etc/systemd/network/) will be needed to get a working network connection in the VM.

bashrc-git snippets

Neil Williams — Mon, 30 Nov 2015 22:49:44 +0000

Just in case someone else finds these useful, some bash functions I've got into the habit of having in ~/.bashrc:

mcd(){ mkdir "$1"; cd "$1"; }

gum(){ git checkout "$1" &#038;&#038; git rebase master &#038;&#038; git checkout master; }

gsb() { LIST=`git branch|egrep -v '(release|staging|trusty|playground|stale)'|tr '\n' ' '|tr -d '*'`; git show-branch $LIST; }

gleaf(){ git branch --merged master | egrep -v '(release|staging|trusty|playground|pipeline|review|stale)'; }

mcd is the oldest one and the simplest. The others are just useful git management shortcuts. I can use gum to bring a feature branch back to master and gsb to show me which branches need to be rebased on master, typically after a pull. The list of excluded branches includes branches which should not be rebased against master (I could do some processing of git branch -r to not have those hardcoded) but the odd one is stale. Sometimes, I get an idea for a feature which is too intrusive, too messy or just too incomplete to be rebased against master. Rather than losing the idea or wasting time rebasing, I'm getting into the habit of renaming the branch foo as stale-foo and gsb then leaves it alone. Equally, there are frequently times when I need to have a feature branch based on another feature branch, sometimes several feature branches deep. Identifying these branches and avoiding rebasing on the wrong branch is important to not waste time.

gsb takes a bit of getting used to, but basically the shorter and cleaner the output, the less work needs to be done. As shown, gsb is git show-branch under the hood. What I'm looking for is multiple commits listed between a branch and master. Then I know which branches to use with gum.

Finally, gleaf shows which feature branches can be dropped with git branch -d.

vmdebootstrap sprint

Neil Williams — Sat, 07 Nov 2015 12:01:03 +0000

At the miniDebConfUK in Cambridge, November 2015, there was a vmdebootstrap sprint vmdebootstrap is written in python and the sprint built on the changes I made during DebConf15. The primary aim was to split out the code from a single python script and create modules which would be useful to other tools and make the source code itself easier to follow. Whilst doing this, I worked with Steve McIntyre to implement UEFI support into vmdebootstrap. This version reached experimental shortly after DebConf15.

At the sprint, the squashfs support in vmdebootstrap was improved to be more useful in the preparation of live images rather than simply using mksquashfs as a compression algorithm for the entire image. I also improved and extended the documentation for vmdebootstrap. vmdebootstrap is now extensible and modular.

Iain Learmonth used this new modular support in the development of the live-build-ng wrapper which uses vmdebootstrap, live-boot and live-config to replace the role of live-build in the generation of official Debian live images by the debian-cd team. Iain Learmonth demonstrated that the new support in vmdebootstrap can be used to create working live images, including adding support for live images on any architecture supporting the Linux kernel in Debian and adding support for not only Grub but UEFI as well. A UEFI grub live image built by live-build-ng was demonstrated after only two days work on the vmdebootstrap base, wrapping support from live-boot and live-config with customisation for UEFI and grub config.

vmdebootstrap and live-build-ng have been explicitly designed within the debian-cd team to remove the need to run live-build to create official Debian live images, replacing live-build with live-build-ng and the vmdebootstrap support. This brings working support for multiple architectures and UEFI to live images.

To support the new functionality, the vmdebootstrap and debian-cd team have long sought a test suite for vmdebootstrap and Lars Wirzenius implemented one during the vmdebootstrap sprint. We now have fast tests with a pre-commit hook and build tests which are best with a local mirror. The test suite uses cmdtest & yarn. This test suite will be used for all future changes - patches will not be accepted if the test suite fails and any substantially new code <b>must</b> provide working test cases. The fast tests can run without sudo, I expect to be able to sort out ci.debian.net testing for the fast tests too.

There is also an outline for testing vmdebootstrap builds in (localhost) LAVA using a lava-submit.py example.

All this work will arrive in unstable as vmdebootstrap 1.2 soon and is now in the master branch. The old codehelp/modules branch has been merged and removed.

The problem of SD Mux

Neil Williams — Fri, 14 Aug 2015 07:19:24 +0000

I keep being asked about automated bootloader testing and the phrase which crops up is “SD mux” – hardware to multiplex SD card access (typically microSD). Each time, the questioner comes up with a simple solution which can be built over a weekend, so I’ve decided to write out the actual objective, requirements and constraints to hopefully illustrate that this is not a simple problem and the solution needs to be designed to a fully scalable, reliable and maintainable standard.

The objective

Support bootloader testing by allowing fully automated tests to write a
custom, patched, bootloader to the principal boot media of a test
device, hard reset the board and automatically recover if the bootloader
fails to boot the device by switching the media from the test device to
a known working support device with full write access to overwrite
everything on the card and write a known working bootloader.

The environment

100 test devices, one SD mux each (potentially), in a single lab with support for all or any to be switched simultaneously and repeatedly (maybe a hundred times a day to and fro) with 99.99% reliability.

The history

First attempt was a simplistic solution which failed to operate reliably. Next attempt was a complex solution (LMP) which failed to operate as designed in a production environment (partially due to a reliance on USB) and has since suffered from a lack of maintenance. The most recent attempt was another simplistic solution which delivered three devices for test with only one usable and even that became unreliable in testing.

The requirements

(None of these are negotiable and all are born from real bugs or real failures of previous solutions in the above environment.)

Ethernet – yes, really. Physical, cat5/6 RJ45 big, bulky, ugly gigabit ethernet port. No wifi. This is not about design elegance, this is about scalability, maintenance and reliability. Must have a fully working TCP/IP stack with stable and reliable DHCP client. Stable, predictable, unique MAC addresses for every single board - guaranteed. No dynamic MAC addresses, no hard coded MAC addresses which cannot be modified. Once modified, retain permanence of the required MAC address across reboots.
No USB involement – yes, really. The server writing to the media to recover a bricked device usually has only 2 USB ports but supports 20 devices. Powered hubs are not sufficiently reliable.
Removable media only – eMMC sounds useful but these are prototype development boards and some are already known to intermittently fry SD card controller chips causing permanent and irreversible damage to the SD card. If that happened to eMMC, the entire device would have to be discarded.
Cable connections to the test device. This is a solved problem, the cables already exist due to the second attempt at a fix for this problem which resulted in a serviceable design for just the required cables. Do not consider any fixed connection, the height of the connector will never match all test device requirements and will be a constant source of errors when devices are moved around within the rack.
Guaranteed unique, permanent and stable serial numbers for every device. With 100 devices in a lab, it is absolutely necessary that every single one is uniquely addressable.
Interrogation – there must be an interface for the control device to query the status of the SD mux and be assured that the results reflect reality at all times. The device must allow the control device to read and write to the media without requiring the test device to acknowledge the switch or even be powered on.
No feature creep. There is no need to make this be able to switch ethernet or HDMI or GPIO as well as SD. Follow the software principle of pick one job and do it properly.
Design for scalability – this is not a hobbyist project, this is a serious task requiring genuine design. The problem is not simple, it is not acceptable to make a simple solution.
Power – the device must boot directly from power-on without requiring manual intervention of any kind and boot into a default safe mode where the media is only accessible to the control device. 5V power with a barrel connector is preferable – definitely not power over USB. Device must raise the TCP/IP control interface automatically and be prepared to react to commands immediately that the interface is available.
Software: some logic to prevent queued requests from causing repeated switching without any interval in between, e.g. if the device had to be power cycled.
Ongoing support and maintenance of hardware, firmware and software. Test devices continue to develop and will require further changes or fixes as time goes on.
Mounting holes – sounds obvious but the board needs to be mounted in a sensible manner. Dangling off the end of a cat5 cable is not acceptable.

If any of those seem insurmountable or awkward or unappealing, please go back to the drawing board or leave well alone

Beyond the absolutes, there are other elements. The device is likely to need some kind of CPU and something ARM would be preferable, Cortex-M or Cortex-A if relevant, but creating a cape for a beaglebone-black is likely to be overkill. The available cables are short and the device will need to sit quite close to the test device. Test devices never put the SD card slot in the same place twice or in any location which is particularly accessible. Wherever possible, the components on the device should be commodity parts, replaceable and serviceable. The device would be best not densely populated – there is no need for the device to be any particular size and overly small boards tend to be awkward to position correctly once cables are connected. There are limits, of course, so boards which end up bigger than typical test devices would seem excessive.

So these are the reasons why we don’t have automated bootloader testing and won’t have it any time soon. If you’ve got this far, maybe there is a design which meets all the criteria so contact me and let’s see if this is a fixable problem after all.

vmextract – userspace VM helper

Neil Williams — Sat, 02 May 2015 18:31:26 +0000

In my previous post, I covered how to extend an initramfs to serve as a basis for tests and other purposes. In that example, the initramfs was simply copied off a device after installation. It occurred to me that python-guestfs would be a useful tool to make this step easier. So I've written the vmextract helper which is currently in the vmdebootstrap upstream git and will make it into the next release of vmdebootstrap for Debian (0.7).

Once vmdebootstrap has built an image, various files can be generated or modified during the install operations and some of these files can be useful when testing the image or packages which would be included in an updated image, before release. One example is the initrd built by the process of installing a Debian kernel. Rather than having to mount the image and copy the files manually, the <code>vmextract</code> helper can do it for you, without needing root privileges or any special setup.

$ /usr/share/vmdebootstrap/vmextract.py --verbose --boot \
  --image bbb/bbb-debian-armmp.img \
  --path /boot/initrd.img-3.14-2-armmp \
  --path /lib/arm-linux-gnueabihf/libresolv.so.2

This uses python-guestfs (which will become a Recommended package for vmdebootstrap) to prepare a read-only version of the image - in this case with the /boot partition also mounted - and copies files out into the current working directory.

Note the repeating use of --path to build a list of files to be copied out. To copy out an entire directory (and all sub-directories) as a single tarball, use:

$ /usr/share/vmdebootstrap/vmextract.py --verbose --boot \
  --image bbb/bbb-debian-armmp.img \
  --directory /boot/ \
  --filename bbb-armmp-boot.tgz

If --filename is not specified, the default filename is vmextract.tgz.

(vmextract uses gzip compression, just because.)

The other point to note is the use of the --boot option to mount the /boot partition as well as the root partition of the image as this example uses the beaglebone-black support which has a boot partition.

It's just a little helper which I hope will prove useful - if only to avoid both the need for sudo and the need for loopback mount operations with the inherent confusion of calculating offsets. Thanks to the developers of python-guestfs for making this workable in barely 100 lines of python. It uses the same cliapp support as vmdebootstrap, so can be used silently in scripts by omitting the --verbose option.

I haven't taken this on to the next step of unpacking the initrd and extending it, but that would just a bit of shell scripting using the files extracted by vmextract.py.

Next!

Next on the list will be extensions to vmdebootstrap to build live images of Debian, essentially adding Debian Installer to a vmdebootstrap image, which could actually be another python-guestfs helper (mounting read-write this time) to avoid adding lots more code to vmdebootstrap (which has grown to nearly 600 lines of python). That way, we can publish the bare VM images as well as a live conversion and reducing the number of times debootstrap needs to be called.

ARMMP Jessie

Neil Williams — Sat, 25 Apr 2015 22:31:59 +0000

Whilst the rest of Debian was very busy actually making the Jessie release, I was trying to prepare the ground to test the armhf installer images.

Cubietruck is fine, it's a nice install and the documentation on the wiki works nicely. I tried to automate it (as it would be something I'd like to test in LAVA & which is new in Jessie) with preseeding but although I have got a jessie-x86 preseed file which works in libvirt, the cubietruck installer doesn't seem to want to listen to all of the preseeding. Some of the options (notably the wifi firmware screen, hostname and user passwords) don't go away. x86 wasn't completely as expected either though & it works nicely in libvirt and Virtual Machine Manager but trying to (automate it again) do it manually on the command line with qemu raised an awkward grub2 bug where it couldn't work out where to install and failed to then reboot if manually installed.

iMX.53 was the next target & got a few steps along the road with that one, except that the kernel in the installer isn't able to see the SATA even though the bootloader can, nor can it see the USB stick. This means that the only media the installer can see is the SD card from which the installer was booted & trying to use that ends badly.

Beaglebone-black required a little bit of tweaking. Don't forget that the firmware & partition.img together don't give a lot of space, so creating a second partition on the SD card upon which to put the CD image itself is useful. The Debian ARMMP kernel in Jessie doesn't see USB, so the option here is actually what I wanted. Install rom SD card onto eMMC. It's a 2Gb space, but it's ideal for a default location, leaving the SD card for other testing. I ran out of time to see about preseeding BBB installs to eMMC but one word of warning, installing to the eMMC doesn't mean that you have a bootloader on the eMMC & the SD card is still needed to reboot, unless that is fixed manually post-install. (Haven't got that working yet either.)

Speaking of time, this post is more brief on the detail than I would like for a couple of reasons:

I haven't actually got things working well enough (clearly from some of the end states above)
the new code to test these in LAVA isn't ready yet either

I did mean to test on an iMX6.quad Wandboard & will see about that, hopefully, later.

In general, the armhf SD card support is the best start for installing Jessie on a range of ARMv7 hardware. (I haven't got as far as ARMv8 yet either.) However, when I describe that as "the best start", it is lacking in many areas & typically due to restrictions in the u-boot support, mainline kernel support and general board variability. It's a lot easier than it used to be but ARMv7 boards are some distance from a "single installer setup" or even a single set of installer documentation, for any distribution.

Extending an existing ARMMP initramfs

Neil Williams — Tue, 14 Apr 2015 12:27:06 +0000

Note

The actual use of this extension is still in development and the log files are not currently publicly visible, but it may still be useful for people to know the what and why.

The Debian ARMMP kernel can be used for multiple devices, just changing the DTB. I've already done tests with this for Cubietruck and Beaglebone-Black, iMX.53 was one of the original test devices too. Whilst these tests can deploy a full image (there are examples of building such images in the vmdebootstrap), it is a lot quicker to do simple tests of a kernel using a ramdisk. The default Debian initramfs has a focused objective but still forms a useful base for extension. In particular, I want to be able to test one initramfs on multiple boards (so multiple dtbs) with the same kernel image. I then want to be able, on selected boards, to mount a SATA drive or write an image to eMMC or a USB stick or whatever. LAVA (via the ongoing refactoring, not necessarily in the current dispatcher code) can automate such tests, e.g. to allow me to boot a Cubietruck into a standard Debian ARMMP armhf install on the SATA drive but using a modified (or updated) ARMMP kernel over TFTP without needing to install it on the device itself. That same kernel image can then be tested on multiple boards to see if the changes have benefitted one board at the expense of another. Automating all of that could be of a lot of benefit to the ARM kernel developers in Debian and outside Debian.

So, the start point. Install Debian onto a Cubietruck & in my case, with a SATA drive attached. All well and good so far, standard Debian Jessie ARMMP. (Cubietruck uses the LPAE kernel flavour but that won't matter for the initramfs.)

Rather than building the initramfs manually, this provides a shortcut & at some point I may investigate how to do this in QEMU but for now, it's just as quick to SSH onto the Cubietruck and update.

I've already written a little script to download the relevant linux-image package for ARMMP, unpack it and pull out the vmlinuz, the dtbs and a selected list of modules. The list is selective because TFTP has a 32Mb download limit and there are more modules than that. So I borrowed a snippet from the Xen folks (already shown previously here).

The script is in a support repository for LAVA but can be used anywhere. (You'll need to edit the package name in the script to choose between ARMMP and ARMMP LPAE.

Steps

Get a working initramfs from an installed device running Debian ARMMP and copy some files for use later.

Note

use the name of the symlink in the copy so that the file in /tmp/ is the actual file, using the name of the symlink as the filename. This is important later as it saves a step of having to make the (unnecessary) symlink inside the initramfs. Also, mkinitramfs, which built this initrd.img file in the first place, uses the same shared libraries as the main system, so copying these into the initramfs still works. (This is really useful when you get your ramdisk to support the attached secondary storage, allowing you to simply mount the original Debian install and fixup the initramfs by copying files off the main Debian install.) The relevant files are to support DNS lookup inside the initramfs which then allows a test to download a new image to put onto the attached media before rebooting into it.
```
cp /boot/initrd.img-3.16.0-4-armmp-lpae /tmp/
cp /lib/arm-linux-gnueabihf/libresolv.so.2 /tmp/
cp /mnt/lib/arm-linux-gnueabihf/libnss_dns.so.2 /tmp/
```
Copy these off the device for local adjustment:
```
scp IP_ADDR:/tmp/FILE .
```

Decompress the initrd.img:

cp initrd.img-3.16.0-4-armmp-lpae initrd.img-3.16.0-4-armmp-lpae.gzip
gunzip initrd.img-3.16.0-4-armmp-lpae.gzip

Make a new empty directory:
```
mkdir initramfs
cd initramfs
```

Unpack:

sudo cpio -id < initrd.img-3.16.0-4-armmp-lpae

Remove the old modules (LAVA can add these later, allowing tests to use an updated build with updated modules):
```
sudo rm -rf ./lib/modules/*
```
Start to customise - need a script for udhcpc and two of the libraries from the installed system to allow the initramfs to do DNS lookups successfully:
```
cp ../libresolv.so.2 ./lib/arm-linux-gnueabihf/
cp ../libnss_dns.so.2 ./lib/arm-linux-gnueabihf/
```

Copy the udhcpc default script into place:

mkdir ./etc/udhcpc/
sudo cp ../udhcpc.d ./etc/udhcpc/default.script
sudo chmod 0755 ./etc/udhcpc/default.script

Rebuild the cpio archive:

find . | cpio -H newc -o > ../initrd.img-armmp.cpio

Recompress:
```
cd ..
gzip initrd.img-armmp.cpio
```

If using u-boot, add the UBoot header:

mkimage -A arm -T ramdisk -C none -d initrd.img-armmp.cpio.gz initrd.img-armmp.cpio.gz.u-boot

Checksum the final file so that you can check that against the LAVA logs:
```
md5sum initrd.img-armmp.cpio.gz.u-boot
```

Each type of device will need a set of modules modprobed before tests can start. With the refactoring code, I can use an inline YAML and use dmesg -n 5 to reduce the kernel message noise. The actual module names here are just those for the Cubietruck but by having these only in the job submission, it makes it easier to test particular combinations and requirements:

- dmesg -n 5
- lava-test-case udevadm --shell udevadm hwdb --update
- lava-test-case depmod --shell depmod -a
- lava-test-case sata-mod --shell modprobe -a stmmac ahci_sunxi sd_mod sg ext4
- lava-test-case ifconfig --shell ifconfig eth0 up
- lava-test-case udhcpc --shell udhcpc
- dmesg -n 7

In due course, this will be added to the main LAVA documentation to allow others to keep the initramfs up to date and to support further test development.

OpenTAC mailing list

Neil Williams — Fri, 13 Feb 2015 02:10:01 +0000

After the OpenTAC session at Linaro Connect, we do now have a mailing list to support any and all discussions related to OpenTAC. Thanks to Daniel Silverstone for the list.

List archive: http://listmaster.pepperfish.net/pipermail/opentac-vero-apparatus.com

More information on OpenTAC: http://wiki.vero-apparatus.com/OpenTAC

OpenTAC hardware in manufacture

Neil Williams — Mon, 29 Dec 2014 14:19:01 +0000

A bit of news on the development of OpenTAC - the Open Hardware Test Automation Controller. I've talked about this at the MiniDebConf 2014 in Cambridge. (Video available). The development is being tracked on the Vero-Apparatus wiki and as this is Open Hardware, the files are attached to the wiki (All files are CC BY-SA 4.0).

Andy completed the schematics at the start of November, allowing work to start on the routing. With routing completed, orders were placed for manufacture of the first PCBs on the 19th December 2014. Whilst waiting for the PCBs to arrive we're working on the device tree database (my first real experience with creating a device tree) which will underpin the PDU and serial console services available to the user as well as the admin interface for control of the USB subsystems, fan control, power control and thermal monitoring. The first thing we need to do is create a data dictionary - a table to correlate software identifiers with the real hardware pins. We'll then follow that through with a default device tree overlay that will leave all the associated I/O lines in a safe initial state.

Once we have some code, I'll be pushing to a branch on GitHub. We've also got an internal git repo on vero-apparatus for OpenTAC files which we will add to in due course.

Image of the board as rendered prior to prototype production:

More to come once we have the hardware on the desk.

On getting NEW packages into stable

Neil Williams — Mon, 10 Nov 2014 21:16:52 +0000

There's a lot of discussion / moaning /arguing at this time, so I thought I'd post something about how LAVA got into Debian Jessie, the work involved and the lessons I've learnt. Hopefully, it will help someone avoid the disappointment of having their package missing the migration into a future stable release. This was going to be a talk at the Minidebconf-uk in Cambridge but I decided to put this out as a permanent blog entry in the hope that it will be a useful reference for the future, not just Jessie.

Context

LAVA relies on a number of dependencies which were - at the time all this started - NEW to Debian as well as many others already in Debian. I'd been running LAVA using packages on my own system for a few months before the packages were ready for use on the main servers (I never actually installed LAVA using the old virtualenv method on my own systems, except in a VM). I did do quite a lot of this on my own but I also had a team supporting the effort and valuing the benefits of moving to a packaged system.

At the time, LAVA was based on Ubuntu (12.04 LTS Precise Pangolin) and a new Ubuntu LTS was close (Trusty Tahr 14.04) but I started work on this in 2013. By the time my packages were ready for general usage, it was winter 2013 and much too close to get anything into Ubuntu in time for Trusty. So I started a local repo using space provided by Linaro. At the same time, I started uploading the dependencies to Debian. json-schema-validator, django-testscenarios and others arrived in April and May 2014. (Trusty was released in April). LAVA arrived in NEW in May, being accepted into unstable at the end of June. LAVA arrived in testing for the first time in July 2014.

Upstream development continued apace and a regular monthly upload, with some hotfixes in between, continued until close to the freeze.

At this point, note that although upstream is a medium sized team, the Debian packaging also has a team but all the uploads were made by me. I planned ahead. I knew that I would be going to Macau for Linaro Connect in February - a critical stage in the finalisation of the packages and the migration of existing instances from the old methods. I knew that I would be on vacation from August through to the end of September 2014 - including at least two weeks with absolutely no connectivity of any kind.

Right at this time, Django1.7 arrived in experimental with the intent to go into unstable and hence into Jessie. This was a headache for me, I initially sought to delay the migration until after Jessie. However, we discussed it upstream, allocated time within the busy schedule and also sought help from within Debian with the RFH tag. Raphaël Hertzog contributed patches for django1.7 support and we worked on those patches upstream, once I was back from vacation. (The final week of my vacation was a work conference, so we had everyone together at one hacking table.)

Still there was more to do, the django1.7 patches allowed the unit tests to complete but broke other parts of the lava-server package and needed subsequent tweaks and fixes.

Even with all this, the auto-removal from testing for packages affected by RC bugs in their dependencies became very important to monitor (it still is). It would be useful if some packages had less complex dependency chains (I'm looking at you, uwsgi) as the auto-removal also covers build-depends. This led to some more headaches with libmatheval. I'm not good with functional programming languages, I did have some exposure to Scheme when working on Gnucash upstream but it wasn't pleasant. The thought of fixing a scheme problem in the test suite of libmatheval was daunting. Again though, asking for help, I found people in the upstream team who wanted to refresh their use of scheme and were able to help out. The fix migrated into testing in October.

Just for added complications, lava-server gained a few RC bugs of it's own during that time too - fixed upstream but awkward nonetheless.

Achievement unlocked

So that's how a complex package like lava-server gets into stable. With a lot of help. The main problem with top-level packages like this is the sheer weight of the dependency chain. Something seemingly unrelated (like libmatheval) can seriously derail the migrations. The package doesn't use the matheval support provided by uwsgi. The bug in matheval wasn't in the parts of matheval used by uwsgi. It wasn't in a language I am at all comfortable in fixing - but it's my name on the changelog of the NMU. That happened because I asked for help. OK, when django1.7 was scheduled to arrive in Debian unstable and I knew that lava was not ready, I reacted out of fear and anxiety. However, I sought help, help was provided and that help was enough to get upstream to a point where the side-effects of the required changes could be fixed.

Maintaining a top-level package in Debian is becoming more like maintaining a core package in Debian and that is a good thing. When your package has a lot of dependencies, those dependencies become part of the maintenance workload of your package. It doesn't matter if those are install time dependencies, build dependencies or reverse dependencies. It doesn't actually matter if the issues in those packages are in languages you would personally wish to be expunged from the archive. It becomes your problem but not yours alone.

Debian has a lot of flames right now and Enrico encouraged us to look at what else is actually happening in Debian besides those arguments. Well, on top of all this with lava, I also did what I could to help the arm64 port along and I'm very happy that this has been accepted into Jessie as an official release architecture. That's a much bigger story than LAVA - yet LAVA was and remains instrumental in how arm64 gained the support in the kernel and various upstreams which allowed patches to be accepted and fixes to be incorporated into Debian packages.

So a roll call of helpers who may otherwise not have been recognised via changelogs, in no particular order:

Steve McIntyre (Debian & Linaro)
Raphaël Hertzog (Debian)
Dave Pigott (Linaro)
Rémi Duraffort (Linaro)
Sjoerd Simons (Debian)
Antonio Terceiro (Debian and formerly Linaro)
Martin Pitt (Debian)
Jordi Mallach (Debian)
Hector Oron (Debian)
Colin Watson (Debian)

Also general thanks to the Debian FTP and Release teams.

Lessons learnt

Allow time! None of the deadlines or timings involved in this entire process were hidden or unexpected. NEW always takes a finite but fairly lengthy amount of time but that was the only timeframe with any amount of uncertainty. That is actually a benefit - it reminds you that this entire process is going to take a significant amount of time and the only loser if you try to rush it is going to be you and your package. Plan for the time - and be sceptical about how much time is actually required.
Ask for help! Everyone in Debian is a volunteer. Yes, the upstream for this project is a team of developers paid to work on this code (and largely only this code) but the upstream also has priorities, requirements, objectives and deadlines. It's no good expecting upstream to do everything. It's no good leaving upstream insufficient time to fit the required work into the existing upstream schedules. So ask for help within upstream and within Debian - ask for help wherever you can. You don't know who may be able to help you until you ask. Be clear when asking for help - how would someone test their proposed fix? Exactly what are you asking for help doing? (Hint: "everything" is not a good answer.)
Keep on top of announcements and changes. The release team in Debian have made the timetable strict and have published regular updates, guidelines and status notes. As maintainer, it is your responsibility to keep up with those changes and make others in the upstream team aware of the changes and the implications. Upstream will rely on you to provide accurate information about these requirements. This is almost more important than actually providing the uploads or fixes. Without keeping people informed, even asking for help can turn out to be counter-productive. Communicate within Debian too - talk to the teams, send status updates to bugs (even if the status is tag 123456 + help).
Be realistic! Life happens around us, things change, personal timetables get torn up. Time for voluntary activity can appear and disappear (it tends to disappear far more often than extends, so take that into account too).
Do not expect others to do the work for you - asking for help is one thing, leaving the work to others is quite another. No complaining to the release team that they are "blocking" your work and avoid pleading or arguing when a decision is made. The policies and procedures within Debian are generally clear and there are quite enough arguments without adding more. Read the policies, read the guidelines, watch how other packages and other maintainers are handled and avoid those mistakes. Make it easy for others to help deliver what you want.
Get to know your dependency chain - follow the links on the packages.debian.org pages and get a handle on which packages are relevant to your package. Subscribe to the bug pages for some of the more "high-risk" packages. There are tools to help. rc-alert can help you spot problems with runtime dependencies (you do have your own package installed on a system running unstable - if not, get that running NOW). Watching build-dependencies is more difficult, especially build-dependencies of a runtime dependency, so watch the RC bug lists for packages in your dependency chain.

Above all else, remember why you and upstream want the packages in Debian in the first place. Debian is a respected distribution and has an acknowledged reputation for stability and portability. The very qualities that you and your upstream desire from having your package in Debian have direct implications for the amount of work and the amount of time that will be required to get your packages into Debian and keep them there. Having your package in Debian will bring considerable benefits but you will be required to invest a considerable amount of time. It is this contribution which is valuable to Debian and it is this work which will deliver the benefits you seek.

Being an expert in the one package is wildly inadequate. Debian is about the system, the whole distribution and sooner or later, you - as the maintainer - will be absolutely required to handle something which is so far out of your comfort zone it's untrue. The reality is that you are not expected to fix that problem - you are expected to handle that problem and that includes seeking and acknowledging the help of others.

The story isn't over until release day. Having your package in testing the day before the freeze is one step. It may be a large step, but it is only one. The status of that package still needs monitoring. That long dependency chain can still come back and bite.

Don't wait for problems to surprise you.

Finally

One thing I do ask is that other upstream teams and maintainers think about the dependency chain they are creating. It may sound nice to have bindings for every interpreted language possible when releasing your compiled library but it does not help people using that library. Yes, it is more work releasing the bindings separately because a stable API is going to be needed to allow version 1.2.3 to work alongside 1.2.2 and 1.3.0 or the entire effort is pointless. Consider how your upstream package migrates. Consider how adding yet another build-dependency for an optional component makes things exponentially harder for those who need to rely on that upstream. If it is truly optional, release it separately and keep backwards compatibility on each side. It is more work but in reality, all that is happening is that the work is being transferred from the distribution (where it helps only that one distribution and causes duplication into other distributions) into the upstream (where it helps all distributions). Think carefully about what constitutes core functionality and release the rest separately.

Combining bindings for php, ruby, python, java, lua and xslt into a single upstream release tarball is a complete nonsense. It simply means that the package gets blocked from new uploads by the constant churn of being involved in every transition that occurs in the distribution. There is a very real risk that the package will miss a stable release simply by having fingers in too many pies. That hurts not only this upstream but every upstream trying to use any part of your code. Every developer likes to think that people are using and benefiting from their effort. It's not nice to directly harm the interests of other developers trying to use your code. It is not enough for the binary packages to be discrete - migrations happen by source package and the released tarball needs to not include the optional bindings. It must be this way because it is the source package which determines whether version 1.2.3 of plugin foo can work with version 1.2.0 of the library as well as with version 1.3.0.

Maintainers regularly deal with these issues - so talk to your upstream teams and explain why this is important to that particular team. Help other maintainers use your code and help make it easier to make a stable release of Debian. The quicker the freeze / release process becomes, the quicker new upstream versions can be uploaded and backported.

OpenTAC – an automation lab in a box

Neil Williams — Sun, 19 Oct 2014 22:34:16 +0000

I've previously covered running LAVA on ARM devices, now that the packages are in Debian. I've also covered setting up the home lab, including the difficulty in obtaining the PDU and relying on another machine to provide USB serial converters with inherent problems of needing power to keep the same devices assigned to the same ser2net ports.

There have been ideas about how to improve the situation. Conferences are a prime example - setting up a demo involving LAVA means bringing a range of equipment, separate power bricks, separate network switches (with power bricks), a device of some kind to connect up the USB serial converters (and power brick) and then the LAVA server (with SATA drive and power brick) - that is without the actual devices and their cables and power. Each of those power cables tend to be a metre long, with networking and serial, it quickly becomes a cable spaghetti.

Ideas around this also have application inside larger deployments, so the hardware would need to daisy-chain to provide services to a rack full of test devices.

The objective is a single case providing network, power and serial connectivity to a number of test devices over a single power input and network uplink. Naturally, with a strong free software and open development bias, the unit will be Open Hardware running Debian, albeit with a custom Beaglebone Linux kernel. It's a Test Automation Controller, so we're using the name OpenTAC (Open Hardware Test Automation Controller).

Progress

Open hardware ARM device running Debian to automate tests on 4 to 8 devices, initially aimed at LAVA support for Linaro engineers. Power distribution, serial console, network and optional GPIO extensions.

The design involves:

A Beaglebone Black (revC)
- USB hotplug support required, certainly during development.
Custom PCB connected as a Beaglebone Cape, designed by Andy Simpkins.
Base board provides 4 channels:
- 5V Power - delivered over USB
- Ethernet - standard Cat5, no LEDs
- Serial connectivity
  - RS232
  - UART
- GPIO
Internal gigabit network switch
Space for a board like a CubieTruck (with SATA drive) to act as LAVA server
Daughter board:
- Same basic design as the base board, providing another 4 channels, equivalent to the base channels. When the daughter board is fitted, a second network switch would be added instead of the CubieTruck.
Power consumption measurement per channel
queries made via the Beaglebone Black over arbitrary time periods, including during the test itself.
The GPIO lines can be used to work around issues with development boards under test, including closing connections which may be required to get a device to reboot automatically, without manual intervention.
Serial connections to test devices can be isolated during device power-cycles - this allows for devices which pull power over the serial connection. (These are typically hardware design issues but the devices still need to be tested until the boards can be modified or replaced.)
Thermal control, individual fan control via the Beaglebone Black.
1U case - rackable or used alone on the desk of developers.
Software design:
- lavapdu backend module for PDU control (opentac.py) & opentac daemon on the BBB
  - telnet opentac-01 3225
- ser2net for serial console control
  - telnet opentac-01 4000

The initial schematics are now complete and undergoing design review. A lot of work remains ...

vmdebootstrap images for ARMMP on BBB

Neil Williams — Tue, 26 Aug 2014 20:08:39 -0100

After patches from Petter, add foreign architecture support and picking up some scripting from freedombox, I've just built a Debian unstable image using the ARMMP kernel on Beaglebone-black.

A few changes to vmdebootstrap will need to go into the next version (0.3), including an example customise script to setup the u-boot support. With the changes, the command would be:

sudo ./vmdebootstrap --owner `whoami` --verbose --size 2G \
--mirror http://mirror.bytemark.co.uk/debian --log beaglebone-black.log \
--log-level debug --arch armhf --foreign /usr/bin/qemu-arm-static \
--no-extlinux --no-kernel --package u-boot --package linux-image-armmp \
--distribution sid --enable-dhcp --configure-apt \
--serial-console-command '/sbin/getty -L ttyO0 115200 vt100' \
--customize examples/beagleboneblack-customise.sh --bootsize 50m \
--boottype vfat --image bbb.img

Some of those commands are new but there are a few important elements:

use of --arch and --foreign to provide the emulation needed to run the debootstrap second stage.
drop extlinux and install u-boot as a package.
linux-image-armmp kernel
new command to configure an apt source
serial-console-command as the BBB doesn't use the default /dev/ttyS0
choice of sid to get the latest ARMMP and u-boot versions
customize command -- this is a script which does two things:
- copies the dtbs into the boot partition
- copies the u-boot files and creates a u-boot environment to use those files.
use of a boot partition -- note that it needs to be large enough to include the ARMMP kernel and a backup of the same files.

With this in place, a simple dd to an SD card and the BBB boots directly into Debian ARMMP.

The examples are now in my branch and include an initial cubieboard script which is unfinished.

The current image is available for download. (222Mb).

I hope to upload the new vmdebootstrap soon -- let me know if you do try the version in the branch.

Validating ARMMP device tree blobs

Neil Williams — Tue, 22 Jul 2014 20:07:15 -0100

I've done various bits with ARMMP and LAVA on this blog already, usually waiting until I've got all the issues ironed out before writing it up. However, this time I'm just going to do a dump of where it's at, how it works and what can be done.

I'm aware that LAVA can seem mysterious at first, the package description has improved enormously recently, thanks to exposure in Debian: LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. Tests can be simple boot testing, bootloader testing and system level testing, although extra hardware may be required for some system tests. Results are tracked over time and data can be exported for further analysis.

The LAVA documentation has a glossary of terms like result bundle and all the documentation is also available in the lava-server-doc package.

The goal is to validate the dtbs built for the Debian ARMMP kernel. One of the most accessible ways to get the ARMMP kernel onto a board for testing is tftp using the Debian daily DI builds. Actually using the DI initrd can come later, once I've got a complete preseed config so that the entire install can be automated. (There are some things to sort out in LAVA too before a full install can be deployed and booted but those are at an early stage.) It's enough at first to download the vmlinuz which is common to all ARMMP deployments, supply the relevant dtb, partner those with a minimal initrd and see if the board boots.

The first change comes when this process is compared to how boards are commonly tested in LAVA - with a zImage or uImage and all/most of the modules already built in. Packaged kernels won't necessarily raise a network interface or see the filesystem without modules, so the first step is to extend a minimal initramfs to include the armmp modules.

apt install pax u-boot-tools

The minimal initramfs I selected is one often used within LAVA:

wget http://images.armcloud.us/lava/common/linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot

It has a u-boot header added, as most devices using this would be using u-boot and this makes it easier to debug boot failures as the initramfs doesn't need to have the header added, it can simply be downloaded to a local directory and passed to the board as a tftp location. To modify it, the u-boot header needs to be removed. Rather than assuming the size, the u-boot tools can (indirectly) show the size:

$ ls -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
-rw-r--r-- 1 neil neil  5179571 Nov 26  2013 linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot

$ mkimage -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
Image Name:   linaro-image-minimal-initramfs-g
Created:      Tue Nov 26 22:30:49 2013
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    5179507 Bytes = 5058.11 kB = 4.94 MB
Load Address: 00000000
Entry Point:  00000000

Referencing http://www.omappedia.com/wiki/Development_With_Ubuntu, the header size is the file size minus the data size listed by mkimage.

5179571 - 5179507 == 64

So, create a second file without the header:

dd if=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot of=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz skip=64 bs=1

decompress it:

gunzip linaro-image-minimal-initramfs-genericarmv7a.cpio.gz

Now for the additions:

dget http://ftp.uk.debian.org/debian/pool/main/l/linux/linux-image-3.14-1-armmp_3.14.12-1_armhf.deb

(Yes, this process will need to be repeated when this package is rebuilt, so I'll want to script this at some point.):

dpkg -x linux-image-3.14-1-armmp_3.14.12-1_armhf.deb kernel-dir
cd kernel-dir

Pulling in the modules we need for most needs, comes thanks to a script written by the Xen folks. The set is basically disk, net, filesystems and LVM.

find lib -type d -o -type f -name modules.\*  -o -type f -name \*.ko \
 \( -path \*/kernel/lib/\* -o  -path \*/kernel/crypto/\* \
 -o  -path \*/kernel/fs/mbcache.ko -o  -path \*/kernel/fs/ext\* \
 -o  -path \*/kernel/fs/jbd\* -o  -path \*/kernel/drivers/net/\* \
 -o  -path \*/kernel/drivers/ata/\* -o  -path \*/kernel/drivers/scsi/\* \
 -o -path \*/kernel/drivers/md/\* \) | pax -x sv4cpio \
 -s '%lib%/lib%' -d -w >../cpio
gzip -9f cpio

original Xen script (GPL-3+)

I found it a bit confusing that i is used for extract by cpio, but that's how it is. Extract the minimal initramfs to a new directory:

sudo cpio -id < ../linaro-image-minimal-initramfs-genericarmv7a.cpio

Extract the new cpio into the same location. (Yes, I could do this the other way around and pipe the output of find into the already extracted location but that's for when I get a script to do this):

sudo cpio --no-absolute-filenames -id < ../ramfs/cpio

CPIO Manual

Use newc format, the new (SVR4) portable format, which supports file systems having more than 65536 i-nodes. (4294967295 bytes) (41M)

find . | cpio -H newc -o > ../armmp-image.cpio

... and add the u-boot header back:

mkimage -A arm -T ramdisk -C none -d armmp-image.cpio.gz debian-armmp-initrd.cpio.gz.u-boot

Now what?

Now send the combination to LAVA and test it.

Results bundle for a local LAVA test job using this technique. (18k)

submission JSON - uses file:// references, so would need modification before being submitted to LAVA elsewhere.

complete log of the test job (72k)

Those familiar with LAVA will spot that I haven't optimised this job, it boots the ARMMP kernel into a minimal initramfs and then expects to find apt and other tools. Actual tests providing useful results would use available tools, add more tools or specify a richer rootfs.

The tests themselves are very quick (the job described above took 3 minutes to run) and don't need to be run particularly often, just once per board type per upload of the ARMMP kernel. LAVA can easily run those jobs in parallel and submission can be automated using authentication tokens and the <code>lava-tool</code> CLI. ` lava-tool <https://packages.debian.org/jessie/lava-tool>`_ can be installed without <code>lava-server</code>, so can be used in hooks for automated submissions.

Extensions

That's just one DTB and one board. I have a range of boards available locally:

iMX6Q Wandboard (used for this test)
iMX.53 Quick Start Board (needs updated u-boot)
Beaglebone Black
Cubie2
CubieTruck
arndale (no dtb?)
pandaboard

Other devices available could involve ARMv7 devices hosted at www.armv7.com and validation.linaro.org - as part of a thank you to the Debian community for providing the OS which is (now) running all of the LAVA infrastructure.

That doesn't cover all of the current DTBs (and includes many devices which have no DTBs) so there is plenty of scope for others to get involved.

Hopefully, the above will help get people started with a suitable kernel+dtb+initrd and I'd encourage anyone interested to install lava-server and have a go at running test jobs based on those so that we start to build data about as many of the variants as possible.

(If anyone in DI fancies producing a suitable initrd with modules alongside the DI initrd for armhf builds, or if anyone comes up with a patch for DI to do that, it would help enormously.)

This will at least help Debian answer the question of ` what the Debian ARMMP package can actually support <https://lists.debian.org/debian-devel/2014/07/msg00833.html>`_

For help on LAVA, do read through the documentation and then come to us at #linaro-lava or the linaro-validation mailing list or file bugs in Debian: reportbug lava-server.

so you can ask me.

I'm giving one talk on the LAVA software and there will be a BoF on validation and CI in Debian.

LAVA in Debian unstable

Neil Williams — Tue, 01 Jul 2014 20:08:03 -0100

lava-server has arrived in Debian unstable. No need for third party repositories (unless you want to), a simple apt install lava-server. What happens from that point isn't so simple. I've made the single instance installation as straightforward as I could but there is a lot more to do to get LAVA working in a useful manner. First, a little history to explain some of the hysterical raisins which may appear later.

Validation

So, you've made a change to the code, good. It compiles, yay! Before you ship it, does it pass the unit tests? Right, now you have an improved patch which does what you want and keeps the unit tests running. Validation is all about asking that awkward question: does your change work? Does it introduce side effects on other systems / services in the same code? Does it break programs which use services exported by the code? LAVA is one step along the road to system testing, starting at the bottom.

Automation

Well you could do all that yourself. Write the image to the device yourself, apply power yourself, connect to serial, copy test scripts to the booted image and pull the results off, somehow. Much better if this is automated - maybe every commit or every push or as often as the tests can be run on the number of devices available.

LAVA Linaro Automated Validation Architecture. Linaro is a not-for-profit building the future of the Linux kernel on ARM devices. Lava was built for and by Linaro, so the initial focus is clearly on validating the Linux kernel on ARM devices. There are likely to be gotchas in the code for those wanting to use LAVA for other kernels - Linaro can't support lots of different kernels, but if there are changes which make it easier to use other kernels without impacting on validation of Linux, those would likely be accepted. The experience with using LAVA is all with ARM but if there is interest in extending LAVA to work with devices of other architectures, again, patches would be welcome.

The development of the packaging to make LAVA suitable for Debian has also meant that LAVA will run on hardware other than x86. e.g. armv7.com. I'm running a mini lab at home based around an arndale board, I've also got a mini lab at work based on a cubie2. Requirements for such setups are described in the documentation and in previous entries on this blog (principally you will need SATA, lots of RAM and as many CPU cores as you can find. If you want to run LAVA on other architectures, go ahead and let us know if there are issues.

Available versions

The versions uploaded to Debian will (from now on) be production releases. The same code as is running on http://validation.linaro.org/ - development builds and test builds are supported using helpers in the lava-dev package. Releases to unstable will automatically migrate into Ubuntu Unicorn. I'll continue building interim versions at the former locations on http://people.linaro.org/~neil.williams/, including builds for Ubuntu Trusty 14.04LTS as well as providing packages for Jessie until the uwsgi package can migrate. LAVA is looking to work with anyone who can prepare and maintain packages for other distributions like Fedora.

Bugs

LAVA is migrating from Launchpad bugs to http://bugs.linaro.org which is a bugzilla interface. Now that LAVA is also in Debian, anyone is welcome to use the Debian BTS which does not require any login or account setup. The maintainers (including me) will then forward those bugs as appropriate.

Documentation

The immediate task is to update the documentation in the lava-server-doc package (and the Debian wiki) to reflect the availability of LAVA from Debian unstable and how to choose which release of LAVA to use in your systems. However, there is a large amount of documentation already available - click the Help link in the menu bar of any current LAVA instance. As with many projects, the docs have been written by the development team. If there are things which are unclear or if sections need to be moved around to make it easier for people new to LAVA to pick it up, please file bugs.

Hardware

It isn't easy to run a LAVA lab, there is a lot more to it than simply installing lava-server. LAVA would not be possible without the lab team and any other LAVA instance is going to need the same level of excellence in system administration, device support and cooperation. I've covered a little bit of that in previous entries on this blog about my home lab but that is nothing compared to the work required to provide a working lab like the one in Cambridge. Access to that lab is restricted to people within Linaro but LAVA is also reaching out to the community and if there are tests or hardware you want to see within a LAVA instance, not necessarily in the main lab, then talk to us.

Contact

Bug reports are preferable but you can also find LAVA people on #linaro-lava on OFTC or contact us on the linaro-validation mailing list.

Future

There is a lot more work to do on LAVA yet. There are assumptions about partition layout within images (hysterical raisins) and issues with unused software being required for remote worker installations. Both are now part of the next major development work within LAVA.

DebConf14

There is lot more to talk about with LAVA - if you are attending DebConf14 then there will be talks on LAVA and plenty of time to answer questions.

LAVA packages for Debian

Neil Williams — Wed, 26 Mar 2014 19:35:39 -0100

Packaging LAVA for Debian unstable

with notes on other distributions

I've been building packages for LAVA on Debian unstable for several months now and I've been running LAVA jobs on the laptop and on devices in my home lab and on an ARMv7 arndale too.

Current LAVA installations use lava-deployment-tool which has only supported Ubuntu 12.04 LTS Precise Pangolin. There has been a desire in LAVA to move away from a virtual environment, to put configuration files in FHS compliant paths, to use standard distribution packages for dependencies and so to make LAVA available on more platforms than just precise. Packaging opens the door to installing LAVA on Debian, Ubuntu, Fedora and any other recent distribution. Despite LAVA currently being reliant on 12.04 Precise, some of the python dependencies of LAVA have been able to move forward using the virtual environment provided by builtout and pypi. This means that LAVA, as packaged, requires a newer base OS suite than precise - for Ubuntu, the minimal base is Saucy Salamander 13.10 and for Debian it would be Jessie (testing) although there is currently a transition ongoing in Debian which means that uwsgi is not in testing and Debian unstable would be needed instead.

The work to migrate configuration snippets out of deployment-tool and to ensure that the tarball built using setuptools contains all of the necessary files for the package has already been done. The packaging itself is clean and most of the work is done upstream. There is, as ever, more to do but the packages work smoothly for single install LAVA servers where the dispatcher is on the same machine as the django web frontend.

The packages have also migrated to Django1.6, something which is proving difficult with the deployment-tool as it has not kept pace with the changes outside the virtual environment, even if other parts of LAVA have.

LAVA will be switching to packages for installation instead of deployment-tool and this will mean changes to how LAVA works outside the Cambridge lab. When the time comes to swich to packaging, the plan is to update deployment-tool so that it no longer updates /srv/lava/ but instead migrates the instance to packages.

Main changes

Configuration files move into /etc/
- Device configuration files /etc/lava-dispatcher/devices/
- Instance configuration files /etc/lava-server/
Log files move into /var/log/
- Adding logrotate support - no more multi-Gb log files in /srv/lava/
Commitment to keeping the upstream code up to date with dependencies
Support for migrating existing instances, using South.
Packaging helpers
- add devices over SSH instead of via a combination of web frontend and SSH.
Developer builds with easily identifiable version strings, built as packages direct from your git tree.
New frontend
Although django1.6 does not change the design of the web frontend at all, LAVA will take the opportunity to apply a bootstrap frontend which has greater support for browsers on a variety of devices, including mobile. This also helps identify a packaged LAVA from a deployment LAVA.
Documentation and regular updates

The Plan

LAVA has made regular releases based on a monthly cycle and these will be provided as source tarballs at http://www.linaro.org/downloads/ for distributions to download. The official monthly release and any intervening updates will be made available for distributions to use for their own packaging. Additionally, Debian packages will be regularly built for use within LAVA and these will be available for those who choose to migrate from Ubuntu Precise to Debian Jessie. LAVA will assist maintainers who want to package LAVA for their distributions and we welcome patches from such maintainers. This can include changes to the developer build support script to automate the process of supporting development outside LAVA.

Initially, LAVA will migrate to packaging internally, to prove the process and to smooth out the migration. Other LAVA instances are welcome to follow this migration or wait until the problems have been ironed out.

The Issues

Remote workers - this is work to be completed during the

migration to packaging within LAVA as well as pending work upstream on the internals of the connection between a remote worker and the master scheduler. Expect some churn in this area whilst the code is being finalised. A lava-worker package is being prepared which borrows enough code from lava-server to run the lava-scheduler-daemon on the remote worker, until such time as the remote worker communications are refactored.
Integration - there are plans to integrate some of the modules which LAVA uses which are not commonly packaged: linaro-dashboard-bundle and linaro-django-xmlrpc
OpenID - the certificates which underpin OpenID use with Launchpad have been removed in Debian and LAVA is currently investigating alternatives.
json-schema-validator currently has a broken test suite, so this will need to be patched by LAVA to allow the package to build. The code may be replaced with a different validator if the issue persists.
namespace handling - the current package install is unnecessarily noisy with complaints about the lava namespace. This will need a fix but does not affect current operation.
Unit tests - LAVA is working hard to add a much larger coverag of internal unit tests. These use a temporary database which is not generally available during the build of a distribution package. LAVA is already running continuous integration tests to ensure that these tests continue to pass and the packages will gain documentation on how to run these tests after installation.

Build Dependencies

For Debian unstable, the list of packages which must be installed on your Debian system to be able to build packages from the lava-server and lava-dispatcher source code trees are:

debhelper (>= 8.0.0) python | python-all | python-dev | python-all-dev
python-sphinx (>= 1.0.7+dfsg) | python3-sphinx python-mocker
python-setuptools python-versiontools

(python-versiontools may disappear before the packages are finalised)

In addition, to be able to install lava-server, these packages need to be built from tarballs released by Linaro (the list may shorten as changes upstream are applied):

linaro-django-pagination,
python-django-restricted-resource (>= 0.2.7),
lava-tool (>= 0.2), lava-utils-interface (>= 1.0),
linaro-django-xmlrpc (>= 0.4),
python-versiontools (>= 1.8),
python-longerusername,
linaro-dashboard-bundle (>= 1.10.2),
lava-dispatcher (>= 0.33.3)
lava-coordinator, lava-server-doc

The list for lava-dispatcher is currently:

python-json-schema-validator, lava-tool (>= 0.4),
lava-utils-interface, linaro-dashboard-bundle (>= 1.10.2),

The packages available from my experimental repository are using a new packaging branch of lava-server and lava-dispatcher where we are also migrating the CSS to Bootstrap CSS.

Installing LAVA on Debian unstable

$ sudo apt-get install emdebian-archive-keyring<br />

Add the link to my experimental repository (amd64, i386, armhf & arm64) to your apt sources, e.g. by creating a file /etc/apt/sources.list.d/lava.list containing:

deb http://people.linaro.org/~neil.williams/lava sid main<br />

Update with the new key:

$ sudo apt-get update

It is always best to install postgresql first:

$ sudo apt-get install postgresql

There are then three options for the packages to install (Please be careful with remote worker setup, it is not suitable for important installations at this time.):

Single instance, server and dispatcher with recommended tools:
```
apt-get install lava
```

Installs linaro-image-tools and guestfs tools.

Single instance, server and dispatcher:
```
apt-get install lava-server
```

Installs enough to run LAVA on a single machine, running jobs on boards on the same LAN.

Experimental remote worker support:
```
apt-get install lava-worker
```

Needs a normal lava-server installation to act as the master scheduler but is aimed at supporting a dispatcher and boards which are remote from that master.

The packages do not assume that your apache2.4 setup is identical to that used in other LAVA installations, so the LAVA apache config is installed to /etc/apache2/sites-available/ but is not enabled by default. If you choose to use the packaged apache config, you can simply run:

$ sudo a2ensite lava-server
$ sudo apache2ctl restart

(If this is a fresh apache install, use a2dissite to disable to default configuration before restarting.)

Information on creating a superuser, adding devices and administering your LAVA install is provided in the README.Debian file in lava-server:

$ zless /usr/share/doc/lava-server/README.Debian.gz

Provisos and limitations

Please be aware that packaged LAVA is still a work-in-progress but do let us know if there are problems. Now is the time to iron out install bugs and other issues as well as to prepare LAVA packages for other distributions.

It will be a little while before the packages are ready for upload to Debian - I've got to arrange the download location and upload the dependencies first - and some of that work will wait until more work has gone in upstream to consolidate some of the current dependencies.

LAVA server on an arndale

Neil Williams — Fri, 14 Mar 2014 20:06:27 +0000

(ARMv7 A15 dual core)

I've covered how I set up my home LAVA lab and how I got two iMX.53 boards running, the next step was to sort out the LAVA server.

My initial setup relied on using my Thinkpad as the server. This was somewhat convenient as that's how I develop the code, but didn't help me run jobs like health checks because the laptop was usually somewhere other than on the home LAN. There had been a long-standing problem with using LAVA on ARM - the uwsgi support didn't compile - but the code being used was old. As the same code has been in Debian for a long time, I suspected that this was fixed with a newer version. So as part of migrating LAVA to django1.6, apache2.4, postgres9.3, django-tables2 0.13 and a few other updated dependencies, I have been using the version of uwsgi from Debian unstable for quite a while without problems on x86. A simple tweak to the LAVA setup scripts and a newer version of uwsgi became available and it builds just fine on armhf. (I haven't tried armel because Linaro is ARMv7 or newer and we don't test with ARMv5/ARMv6 boards, so there's no point in trying armel - it's not likely to be an issue for this fix though.) (Only the current lava-deployment-tool methods need to actually compile uwsgi directly from upstream without any patches - the packaging which will replace l-d-t uses precompiled binaries, just as any admin would expect.)

That initial install was on one of the iMX.53 boards, which ran as slowly as anyone who has used an iMX.53 would expect. I haven't run test jobs on that install, it had a hard enough time serving the frontend.

Clearly some more cores would be a good step and when an arndale became available for the home lab, the next step was to get a SATA drive onto the arndale, put Debian unstable on that and then deploy LAVA to the arndale using the Debian packages. As a bonus, these packages are built for django1.6 (using an unreleased branch).

Sounds simple enough. It works on iMX.53, it should work on an arndale. Well, it does - but only after some unexpected faff.

arndale, u-boot and SATA

I hate u-boot - every single time I ever touch it, the little I know about any other u-boot device is completely useless. Uniformity is a good thing - infinite permutations are just rope to hang ourselves with `cute embedded nonsense hacks <https://plus.google.com/+JonMasters/posts/j5Vdu1LKv3b>`_ and I'm bored of that whole scene.

So, boot the arndale, interrupt the bootloader to get to the prompt. What's the first thing most people would want? help. Denied - this particular u-boot has been compiled without the ability to display any help messages. u-boot has just sunk further into the depths of "please can we have something else which is uniform across devices".

OK, well maybe I can get by without help. sata init - unknown command 'sata'. (Go away, do something else for a bit.)

The arndale isn't one of the newest boards in the ARMv7 world, LAVA has been using them for a while, so someone has probably done some work to fix this …. nope. There's a workaround - a nice detailed guide by Phil Endecott which details that the kernel needs to continue to be loaded from SD and then the SATA can be used for just the rootfs.

Mildly annoyed (I was hoping to be able to update the arndale kernel without the hassle of mounting the SD card at all), the rest was straight-forward.

Preparing the SATA and installing Debian

Due to the complexity of some of the transitions currently ongoing in Debian ahead of the Jessie release freeze, uwsgi is not in Debian testing right now so this install has to be Debian unstable. That's OK, I've been running Debian unstable on all my machines for several years now - usually updating ~5 times a week.

# parted /dev/sda -s unit mb mktable msdos
# parted /dev/sda -s unit mb mkpart primary 1 -- "-0"
# mkfs.ext4 /dev/sda1
# mount /dev/sda1 /mnt/root/

(note the -- in the mkpart command, that catches me out most times I try this. Without it, the special -0 gets interpreted as an option to parted but parted doesn't then tell you how to fix it, it just complains about an unknown option.)

The SD image I am using is a LAVA master image, based on Ubuntu, so it was easy to put debootstrap onto it.

# apt-get install debootstrap
# debootstrap --arch=armhf sid /mnt/root/ http://ftp.uk.debian.org/debian
# chroot /mnt/root/

Inside the chroot, it's Debian as usual:

apt-get update
apt-get install emdebian-archive-keyring
passwd

emdebian-archive-keyring is a package I (ab)use to sign my LAVA packaging as it has the advantage of being in Debian and Ubuntu going back to the time of Lenny, so it is always available from a repository which has already been authenticated to apt and doesn't involve downloading some random key from a website.

I use passwd manually because this rootfs won't be using auto-login, it will have regular users. root passwords can be automated, if necessary.

Thinking about login, remember to give the rootfs a usable tty. (The LAVA master image has a nasty habit of mangling editors like nano and vi when used over serial, so I'm using echo.) For the arndale you need:

# echo T0:23:respawn:/sbin/getty -L ttySAC2 115200 vt102 >> ./etc/inittab<
# echo >> ./etc/securetty
# echo ttySAC2 >> ./etc/securetty

Then to make the hostname useful, adapt this to your needs:

echo 127.0.0.1       arndale.codehelp arndale >> ./etc/hosts

Finally, get the network running at boot and add the LAVA repository:

# echo auto lo eth0 > ./etc/network/interfaces.d/arndale
# echo iface lo inet loopback >> ./etc/network/interfaces.d/arndale
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/arndale
# echo deb http://people.linaro.org/~neil.williams/lava sid main >> ./etc/apt/sources.list.d/lava.list
# apt-get update
# exit

Once the LAVA packaging is finalised, the packages will be uploaded to a normal Linaro PPA, I'll update the packages to use an 'official' download location and we'll move to using sane version string handling. For now, the packages are still in development.

Code for the packaging is in Alioth but there are issues with anonymous access to the pkg-linaro-lava repositories currently, so I've also pushed to http://github.com/Linaro if you are interested in building the packages yourself. See also my packaging guide which is also linked from the Links page on my own site.

With that done, time to reboot, set the SATA drive as the rootfs using the horrid u-boot and install LAVA:

# reboot
> setenv bootargs console=tty0 console=ttySAC2,115200n8 drm_kms_helper.edid_firmware=edid-1920x1080.fw  root=/dev/sda1
> boot

'login':

# apt-get install postgresql

After this operation, 63.0 MB of additional disk space will be used:

# apt-get install lava-server

After this operation, 231 MB of additional disk space will be used:

# a2dissite 000-default
# a2ensite lava-server
# apache2ctl restart
# service lava-server restart

Take note of the IP address of your new ARM LAVA server and sign in with your regular details, e.g. using OpenID before creating a temporary superuser:

# lava-server manage createsuperuser --username default --email=user@example.com

Choose a temporary password, log out, log in as the temporary superuser, promote your regular user to superuser and default owner of unrestricted devices in the django admin interface. Finally, log out and login as regular user to delete the temporary user. (Yes, that probably could do with some improvement - remind me later and I'll see if a known user can be promoted from the command line.)

See also Senthil's recent post on LAVA unit tests. Getting those to run without so many steps is also on the TODO list.

Results

IP Address: 127.0.0.1
Is Master? : True
Uptime: 6:02:23.700000
Architecture: armv7l
Platform: Linux-3.10.1.0-1-linaro-arndale-armv7l-with-debian-jessie-sid
Insignal Arndale evaluation board based on EXYNOS5250
System memory: 1998MiB

Next!

I'm working on some code upstream to make it easier to add devices from the command line. The aim is that a set of simple commands will set up an initial device, run a job on that device and submit a result bundle to the LAVA server as job #1. This gives some assurance that everything has gone well.

Now I can finally add health checks for the iMX.53 boards and start regular tests on those boards. The cubie boards will wait for a later blog post, there are interesting problems with the bootloader and serial connections. That's a whole different rant.

Debian ARMMP in LAVA using iMX.53

Neil Williams — Sun, 09 Feb 2014 18:51:10 -0100

The home lab is my LAVA development environment and I've recently got two iMX53 Quick Start Boards working with a typical LAVA master image based on a Linaro build of Linux 3.1.0 for mx5 with a Ubuntu Oneiric Ocelot 11.10 rootfs:

3.1.0-1002-linaro-lt-mx5 (buildd@hubbard) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #13-Ubuntu PREEMPT Fri Dec 16 01:21:07 UTC 2011

As part of my Debian work, it was clearly time to look at a current, Debian, kernel and rootfs and as I'm developing and testing on Debian unstable, this would necessarily mean testing the Debian ARMMP (multi-platform) kernel which replaces the mx5 build used in Wheezy:

Linux version 3.12-1-armmp (debian-kernel@lists.debian.org) (gcc version 4.8.2 (Debian 4.8.2-10) ) #1 SMP Debian 3.12.9-1 (2014-02-01)

I will be scripting the creation of a suitable image for these tests and there are other changes planned in LAVA to make it easier to build suitable images, but it is useful to document just how I worked out how to build the first image.

Manual build steps

First, I've already discovered that the u-boot on the iMX53 doesn't like ext4, so the first step was to prepare an ext3 filesystem for the image. With a SATA drive attached, it was also much better to use that than the SD card, at least for generating the image. I'm also doing this natively, so I am working inside the booted master image. This is fine as the master is designed to manipulate test images, so the only package I needed to install on the LAVA master image was debootstrap I had an empty SATA drive to play with for these tests, so first prepare an ext3 filesystem:

# mkfs.ext3 /dev/sda1
# mkdir /mnt/sata
# mount /dev/sda1 /mnt/sata
# mkdir /mnt/sata/chroots/

Start the debootstrap:

# apt-get update
# apt-get install debootstrap
# debootstrap --arch=armhf --include=linux-image-armmp \
 --verbose unstable \
 /mnt/sata/chroots/unstable-armhf http://ftp.uk.debian.org/debian

Various actions in this chroot will need proc, so mount it here:

# chroot /mnt/sata/chroots/unstable-armhf
# mount proc -t proc /proc
# mount devpts -t devpts /dev/pts
# exit

You may also have to edit the apt sources - the LAVA master image doesn't have editors installed, so either use echo or download an edited file. I'm using:

deb http://ftp.uk.debian.org/debian sid main

flash-kernel needs changes

For the initial tests, I've got to get this image to boot directly from u-boot, so flash-kernel is going to be needed inside the chroot and to get the iMX53 to work with the ARMMP kernel and Device Tree, flash-kernel will need an update which will mean a patch:

# chroot /mnt/sata/chroots/unstable-armhf
# apt-get update
# apt-get install patch flash-kernel
# cp /usr/share/flash-kernel/db/all.db /home
# cd /home
# patch -p2

The patch itself goes through a couple of iterations. Initially, it is enough to use:

 Machine: Freescale MX53 LOCO Board
-Kernel-Flavors: mx5
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append: yes

Later, once the device has booted with the ARMMP kernel, the Machine Id can be updated to distinguish it from the mx5 flavour (from /proc/cpuinfo) and use the model name from the Device Tree (/proc/device-tree/model):

diff --git a/db/all.db b/db/all.db
index fab3407..41f6c78 100644
--- a/db/all.db
+++ b/db/all.db
@@ -62,6 +62,18 @@ U-Boot-Initrd-Address: 0x00800000
 Required-Packages: u-boot-tools
 Bootloader-Sets-Incorrect-Root: yes

+Machine: Freescale i.MX53 Quick Start Board
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append-From: 3.12
+Boot-DTB-Path: /boot/dtb
+U-Boot-Kernel-Address: 0x70008000
+U-Boot-Initrd-Address: 0x0
+Boot-Kernel-Path: /boot/uImage
+Boot-Initrd-Path: /boot/uInitrd
+Required-Packages: u-boot-tools
+Bootloader-Sets-Incorrect-Root: no
+
 Machine: Freescale MX53 LOCO Board
 Kernel-Flavors: mx5
 U-Boot-Kernel-Address: 0x70008000

(I will be filing this patch in a bug report against flash-kernel soon.) With that patched, update and run flash-kernel:

# mv all.db /usr/share/flash-kernel/db/all.db
# flash-kernel
flash-kernel: installing version 3.12-1-armmp
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
Installing new dtb.
# exit

LAVA overlays

This will be a LAVA test image and it needs an overlay - if you want a vanilla image, set up a passwd inside the chroot instead:

# cd /mnt/sata/chroots/unstable-armhf/
# wget --no-check-certificate \
  https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay-minimal_1112.2_all.deb
# wget --no-check-certificate \
  https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay_1112.2_all.deb
# chroot /mnt/sata/chroots/unstable-armhf/
# dpkg -i linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# rm linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# exit

Changes to allow the chroot to boot

Now the chroot needs setting up as a boot image:

# chroot /mnt/sata/chroots/unstable-armhf/
# echo T0:23:respawn:/sbin/getty -L ttymxc0 115200 vt102 >> ./etc/inittab
# echo auto lo eth0 > ./etc/network/interfaces.d/mx53loco
# echo iface lo inet loopback >> ./etc/network/interfaces.d/mx53loco
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/mx53loco
# apt-get clean
# umount /dev/pts
# umount /proc
# exit

Partitioning as a LAVA test image

This would be enough as a single partition test image but, currently, LAVA expects a much more hard-wired image. Depending on the history of the device and the need for LAVA to be able to put fresh kernel builds together with a known rootfs, LAVA has used a separate /boot and / partition in the test image for nearly all boards. Standard LAVA test images for many boards (like the iMX53) also have a small unallocated space at the start of the SD card, so until I can get LAVA upstream to handle test images of arbitrary design, I'm adapting the image to suit the expectations inside LAVA. Yes, I know but it's better to get something working before spending time fixing things to make it work better. It will be fixed, in time.

So I needed to separate out the /boot contents from the rest of the rootfs - whilst keeping the chroot itself in a state which I can easily update and use to create other images:

# cd /mnt/sata/chroots/unstable-armhf/boot/
# tar -czf ../../boot.tar.gz ./*
# cd ..
# tar -czf ../root.tar.gz ./*

Now the test image file and its partitions:

# dd if=/dev/zero of=/mnt/sata/images/empty/debian.img bs=1M count=1024
# cp /mnt/sata/images/empty/debian.img /mnt/sata/images/debian-unstable-armhf-armmp.img
# losetup /dev/loop0 /mnt/sata/images/debian-unstable-armhf-armmp.img
# parted /dev/loop0 -s unit mb mktable msdos
# parted /dev/loop0 -s unit mb mkpart primary 1 10
# parted /dev/loop0 -s unit mb mkpart primary 11 110
# parted /dev/loop0 -s unit mb mkpart primary 111 1024

I did make the mistake of using kpartx at this stage but there are many areas of confusion when translating the kpartx output to offsets for mount when parted is easier:

# parted /dev/loop0 unit B -s print
Model:  (file)
Disk /dev/loop0: 1073741824B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start       End          Size        Type     File system  Flags
 1      1048576B    10485759B    9437184B    primary
 2      10485760B   110100479B   99614720B   primary
 3      110100480B  1024458751B  914358272B  primary

Use the Start numbers and use losetup to create the loop devices for each partition:

# losetup -o 10485760 /dev/loop1 /dev/loop0
# losetup -o 110100480 /dev/loop2 /dev/loop0
# mkfs.vfat /dev/loop1
# mkfs.ext3 /dev/loop2

Now clean up the loop mountpoints:

# losetup -d /dev/loop2
# losetup -d /dev/loop1
# losetup -d /dev/loop0
# losetup -a

losetup -a should return nothing. If it doesn't, investigate the contents of /dev/mapper and use dmsetup remove until losetup -a does report empty. Otherwise the subsequent stages will fail. Now deploy the /boot contents into the empty image:

# mount -oloop,offset=10485760 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/boot/
# pushd /mnt/boot/
# tar -xzf /mnt/sata/chroots/boot.tar.gz
# popd
# sync
# umount /mnt/boot/

and the / contents (removing the duplicate ./boot contents):

# mount -oloop,offset=110100480 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/root/
# pushd /mnt/root
# tar -xzf /mnt/sata/chroots/root.tar.gz
# rm ./boot/*
# popd
# sync
# umount /mnt/root

Check the image

# mount -oloop,offset=10485760 debian-unstable-armhf-armmp.img /mnt/boot
# ls /mnt/boot
config-3.12-1-armmp  initrd.img-3.12-1-armmp  System.map-3.12-1-armmp  uImage  uInitrd  vmlinuz-3.12-1-armmp
# umount /mnt/boot
# mount -oloop,offset=110100480 debian-unstable-armhf-armmp.img /mnt/root
# ls /mnt/root
bin  boot  dev  etc  home  initrd.img  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vmlinuz
# ls /mnt/root/bin/auto-serial-console
/mnt/root/bin/auto-serial-console
# umount /mnt/root
# md5sum debian-unstable-armhf-armmp.img

Downloads

Yes, I have put that file online, if you are interested. Do read the readme first though.

Getting the image off the device

# ip a
# python -m SimpleHTTPServer 80 2>/dev/null

then wget (just http://, IP address / and the filename), md5sum - finish with Ctrl-C. Depending on setup, it may be quicker to transfer the uncompressed file over a LAN than to let the device compress it. Your main machine will do the compression much faster, even with a larger download. (It also helps to not compress the image on the device, you can test the mount offsets more easily - and do check that the image can be mounted with the offsets from parted.)

Results

complete log of the first successful LAVA test of a Debian ARMMP kernel on an iMX53 (153k)
LAVA submission JSON (only usable with a local URL of the image and a few tweaks to the standard LAVA mx53loco device configuration)
LAVA result bundle (JSON) (75k)

# cat /proc/cpuinfo
processor      : 0
model name     : ARMv7 Processor rev 5 (v7l)
Features       : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer        : 0x41
CPU architecture: 7
CPU variant    : 0x2
CPU part       : 0xc08
CPU revision   : 5

Hardware       : Freescale i.MX53 (Device Tree Support)
Revision       : 0000
Serial         : 0000000000000000

# uname -a
Linux imx53-02 3.12-1-armmp #1 SMP Debian 3.12.9-1 (2014-02-01) armv7l GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:        Debian
Description:   Debian GNU/Linux unstable (sid)
Release:       unstable
Codename:      sid

Next!

Yes, there is a lot to do from here on.

The creation of the test image needs to be smoothed out and scripted. I'm thinking that this would go into vmdebootstrap. Lars?
LAVA needs to be less presumptive about the contents and partitioning of the test image.
the Debian ARMMP kernel itself needs to regain SATA support (an update is already pending).
I've also got two Cubieboard2 devices and there is a sun7i-a20-cubieboard2.dtb file in the ARMMP kernel which deserves some investigation.

Then there is the whole issue of actually making this multi-platform. After all, it is far from ideal that a multi-platform kernel package has to have platform-specific operations using flash-kernel at each update. So Grub on ARM is going to be on the list of things to investigate.

iMX.53 in home lab

Neil Williams — Thu, 06 Feb 2014 20:21:30 -0100

The iMX53 Quick Start Board has been a mainstay of the Debian armhf port and has been part of LAVA until recently. It's a natural choice for seeing how LAVA and Debian can work together more closely. So, in the first steps to getting my home lab running, I set about restoring two of these boards to LAVA support.</p>

I will be looking into each of the possible boot methods with these boards, but the way to start is with a LAVA master image on an SD card. With the pause in iMX53 development within LAVA, this raises a couple of issues:

u-boot on the iMX53 is old and the Linaro image tools have advanced without making allowances for the unsupported mx53loco type.
- For LAVA, this means that u-boot on the iMX53 does not understand ext4 and although there are guides on updating it, I wanted to stay with the original, at least at the start.
- Using ext4 will lead to this error:
```
mmc0: error 110 whilst initialising SD card
```
iMX53 quick start boards come with a working Ubuntu installation on an SD card but the release is Lucid Lynx 10.04LTS. LAVA has been on Precise Pangolin 12.04LTS for some time.
- This causes issues with debootstrap which cannot unpack packages from Debian sid due to changes in the compression format inside the .deb files. It is possible to use wheezy and note that lucid, by default, will produce armel. Use `--arch=armhf, then apt-get upgrade and apt-get dist-upgrade to testing and then to sid (not in one single step). Remember to mount proc before starting the upgrade.

LAVA master image on iMX53

The LAVA master image is more recent and can fix these problems. See lava-image-creation - preparing a master image

$ git clone git://git.linaro.org/lava/lava-master-image-scripts.git
$ cd lava-master-image-scripts

Use patch -p1 to apply this patch (apologies for the line wrapping in the blog):

diff --git a/lava-create-master b/lava-create-master
index 2e1d0e8..a1e6131 100755
--- a/lava-create-master
+++ b/lava-create-master
@@ -366,7 +366,7 @@ make_master() {
         verbose " * Building vanilla image with linaro-media-create..."
         if ! linaro-media-create \
             --dev $LMC_DEV \
-            --rootfs ext4 \
+            --rootfs ext3 \
             --image-file $CACHE_DIR/pristine-images/$dev-vanilla.img \
             --image-size 1G \
             --align-boot-part \

$ sudo ./lava-create-master imx53

Without the patch applied, I simply removed the SD card, mounted the third partition on my main machine and created a tarball of the contents:

$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -czf /tmp/imx53-rootfs.tgz ./*
$ popd
$ sudo umount /mnt/

Reformat p3 to ext3 (or ext2 if you prefer)

$ sudo mkfs.ext3 /dev/mmcblk0p3

Put the rootfs back onto the replacement ext3 partition.

$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -xzvf /tmp/imx53-rootfs.tgz
$ popd
$ sudo sync
$ sudo umount /mnt/

The bootscript is buggy, so run:

::: > run loaduimage > run mmcboot

Then fix bootscript or you won't be able to reboot: Simplest way may be to set u-boot to ignore boot.scr

> setenv script
> saveenv
> reset

Device configuration in LAVA

I found screen to be a bit awkward as a serial connection, (use Ctrl-A : quit) to quit. I tried autologin for the lucid image but that was not needed for the master image which uses the linaro overlay. The PDU commands are for my own setup. The daemon (the machine running the pdu daemon scripts), hostname (the hostname or IP of the PDU itself) and the port numbers will change between labs. Again, the blog has changed the line endings.

device_type = mx53loco
hostname = imx53-01
connection_command = telnet hobbes 4002
#login_prompt = lucid-desktop login:
#password_prompt = Password:
#username = lucid
#password = lucid
hard_reset_command = /usr/bin/pduclient --daemon localhost --hostname pdu --command reboot --port 05
power_off_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command off --port 05
power_on_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command on --port 05
testboot_offset = 3

Critical testboot_offset is currently undocumented. Without it, you will see:

** Partition 4 not valid on device 0 **

You can trace this back in a LAVA test log to:

setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:4 0x72000000 uInitrd;
fatload mmc 0:4 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"

The correct line needs to use mmc 0:5 in each case.

This is what testboot_offset modifies - the default is 2:

setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:5 0x72000000 uInitrd;
fatload mmc 0:5 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"

Hacking session in LAVA

A hacking session in LAVA is a way of getting an SSH login directly into a deployed test image, as root. It's a test image, so look around, trash stuff, build stuff, reboot the board. When you are done, use stop-hacking and the test completes, leaving the master image unchanged, ready for the next rootfs to go onto the test image you were using.

This will be improved in later blog entries but as a starter and so that you can play with the SATA drive with a recent installation, this is the JSON for a LAVA job which gives you an SSH session inside the LAVA test image on an iMX53. The test image comes from Linaro releases 12.01. I haven't set up a proxy yet, so to save time, I downloaded it once and used a local file:// URL in the testjob.

Note

the rootfstype parameter - without this, LAVA will helpfully format the partition onto which the rootfs is installed to ext4, with predictable results. (This bug is currently being fixed in LAVA so that LAVA uses the same filesystem as the downloaded image. All part of making LAVA less presumptive.)

The smoke-tests-basic testdef is gratuitous and can be removed but this particular run won't be quick - it took over an hour to get into the hacking session, but future topics here will improve that - so the extra few minutes for the smoke-tests is not a big win.

Change the value for GATEWAY to something sane for your network, change the value for PUB_KEY to your SSH public key. Change the image location to something which works for your setup.

{
   "health_check": false,
   "job_name": "iMX53-hacking",
   "logging_level": "DEBUG",
   "device_type": "mx53loco",
   "timeout": 900,
   "actions": [
       {
           "command": "deploy_linaro_image",
           "parameters": {
               "image": "file:///home/linaro/lava/mx53loco-ubuntu-desktop.img.gz",
               "rootfstype": "ext3"
           }
       },
       {
           "command": "lava_test_shell",
           "parameters": {
               "testdef_repos": [
                   {
                       "git-repo": "git://git.linaro.org/qa/test-definitions.git",
                       "testdef": "ubuntu/smoke-tests-basic.yaml"
                   },
                   {
                       "git-repo": "http://git.linaro.org/git/lava-team/hacking-session.git",
                       "parameters": {
                           "GATEWAY": "10.15.0.1",
                           "PUB_KEY": "...."
                       },
                       "testdef": "hacking-session-debian.yaml"
                   }
               ],
               "timeout": 18000
           }
       },
       {
           "command": "submit_results_on_host",
           "parameters": {
               "server": "http://sylvester.codehelp/RPC2/",
               "stream": "/anonymous/codehelp/"
           }
       }
   ]
}

(Use http://jsonlint.com to reformat.)

I was part of the way through running debootstrap on the SATA drive when ser2net timed out because I hadn't changed the values at that time.

Configuring set2net for LAVA

ser2net can run on any machine capable of providing the serial connections, including a dedicated device.

Decide on a port range to use on the machine running ser2net.

If the device has more than one network interface, ensure that ser2net only offers connections on the relevant interface by prefixing the port with the IP address or hostname.

telnet is generally the easiest ser2net state to use with LAVA:

#  <TCP port>:<state>:<timeout>:<device>:<options>
#     TCP port
#            Name   or  number of the TCP/IP port to accept con-
#            nections from for this device.  A port number may
#            be of the form [host,]port, such as 127.0.0.1,2000
#            or localhost,2000.  If this is specified, it will
#            only bind to the IP address specified. Otherwise
#            it will bind to all the ports on the machine.
# 10.15.0.1,4000:telnet:600:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner

Clear the timeout or long running jobs will fail.

10.15.0.1,4000:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner
10.15.0.1,4001:telnet:0:/dev/ttyUSB1:115200 8DATABITS NONE 1STOPBIT banner

I've tried relatively carefully to preserve the instructions here from my notes. However, if you do try this yourself and there are problems, let me know and I'll update.

I'm also planning on putting all of the config files, patches, JSON examples and other code into a git repo fairly soon to make things easier.

Home server rack

Neil Williams — Sun, 26 Jan 2014 18:24:55 -0100

The uses of this particular rack I'll cover in future entries - this is about how I made the rack itself, with help from friends (Steve McIntyre & Andy Simpkins). A common theme is making allowances for using dev kit boards - ready-to-rack ARM servers are not here yet. My aim was to have 4, possibly 6 ARM dev kit boards running services from home, so there was no need for a standard 42U rack, a 9U rack is enough. Hence:

Wall Mounted 9U 450mm deep 19 inch rack enclosure & glass door

To run various other services around the house (firewall, file server etc.), a microserver was also necessary:

HP 704941-421 ProLiant Micro Server (AMD Turion II Neo N54L 2.2GHz, 2GB RAM, 250GB HDD, 2 Core, 7th Generation)

I chose to mount that on a bookcase beneath the wall mounted rack as it kept all the cables at the bottom of the rack itself. The microserver needed a second gigabit network card fitted to cope with being a firewall as well, if you do the same, ensure you find a suitable card with a low profile bracket. Some are described as being low profile but do not package the low profile bracket, only a low profile card and a full height bracket.

Intel EXPI9301CTBLK PRO1000 Network Card & note the low profile bracket in the pack.

The first of the dev kit requirements is the lack of boards which can be racked, so shelves are going to be needed, leading on to something to stop the boards wandering across the shelf when the cables are adjusted & velcro pads in my case.

Second requirement is that dev kit boards are notorious for not rebooting cleanly. Nothing to do with the image being run, the board just doesn't cut power or doesn't come back after cutting power. Sometimes this is down to early revisions of the board, sometimes the board pulls enough power through the USB serial converter to remain alive, whatever the cause, it won't reboot without user involvement. So a PDU becomes necessary - remotely controllable. New units tend to be expensive and/or only suitable for larger racks, I managed to find an older 8 port APC unit, something like:

(Don't be surprised if that becomes a dead link - search for APC Smart Slot Master Switch Power Distribution Unit).

Talking of power, I'm looking to use SATA drives with these boards and the boards themselves come with a variety of wall wart plugs or USB cables, so a number of IEC sockets are needed - not the usual plugs:

Power cable - IEC C14 plug - 13A socket - 25 cm

or, for devices which genuinely need 2A to boot (use the 1A for attached SATA or leave empty):

Black Universal 3.1A - 15W - 5V Dual USB Rapid Mains Charger Plug

Check the power output rating of the USB plugs used to connect to the mains as well - many are 1A or less. Keep the right plug for the right board.

Power is going to also be a problem if, like me, you want to run SATA drives off boards which support SATA. The lack of a standard case means that ATX power is awkward, so check out some cheap SATA enclosures to get a SATA connector with USB power.

I am using these enclosures (prices seem to have risen since I obtained them):

Startech 2.5 inch eSATA USB External Hard Drive Enclosure for SATA HDD

Along with these:

eSATA to SATA Serial External Shielded Cable 1m because the iMx53 boards have SATA connectors but the enclosure exports eSATA. Whilst this might seem awkward, the merit of having both eSATA and simple USB power on one enclosure is not to be under-estimated. (Avoid the stiffer black cables - space will get tight inside the rack.)

Naturally, a 2.5 inch SATA drive is going to be needed for each enclosure, I'm using HDD but SSD is also an option.

Also, consider simple 2 or 3 way fused adaptors so that the board and the SATA drive can be powered from a single PDU port, this makes rebooting much simpler if the board needs a power supply with integrated plug instead of over USB.

Now to the networking (2 x 8 port was cheaper than 1 x 16):

Netgear GS108 8-port Gigabit Ethernet Unmanaged Switch

Don't forget the cat5 cables too - you'll want lots of short cables, 1m or shorter inside the rack and a few longer ones going to the microserver and wall socket. I used 8x1m.

Naturally, on the floor below your rack you are going to put a UPS, so the PDU power then needs to be converted to draw from the UPS via IEC plugs instead of standard mains. I decided to use a 6 gang extension 1m cable with an IEC plug - it was the only bit of wiring I had to do and even those are available ready-made if you want to do it that way.

Depending on the board, you may need your own serial to USB converters, you'll certainly need some powered USB hubs.

I'm using a wall mounted 9U rack, so I also needed a masonry drill and 4 heavy duty masonry M8 bolts. The rack comes with a mounting plate which needs to be bolted to the wall but nothing else is included. This step is much easier with someone else to take the weight of the rack as it is guided into the brackets on the mounting plate - the bracket may need a little persuasion to allow for the bolt heads to not get in the way during mounting. Once mounted, the holes in the back of the rack allow for plenty of room, it's just getting to that point.

The rack has side panels which latch out easily, providing easy maintenance. The glass door can be easily reversed to open from the opposite side. However, the key in the glass door is largely useless. The expectation is that the units in the rack are attached at the front but the dev boards on shelves are not going to be 'protected' by a key in the front door. The key therefore ends up being little more than a handle for the glass door.

OK. If you've got this far, it's a case of putting things together:

Economy Cage Nut Tool 19 inch racking for cagenut extraction

Yes, you really do want one. Fine, do without the premium one, but the economy one will save you a lot of (physical) pain.

At this stage, it becomes clear that the normal 19 inch server rack shelves don't leave a lot of room at the back of the rack for the cables - and there are a lot of cables.

Each board has power, USB serial connection and network. The SATA has power too. The PDU has a power lead and you'll need network switches too. The network switches need power and an uplink cable.

I positioned the supports in the rack as far forward as possible and attached the shelves to give enough room for the PDU on the base of the rack, the network switches resting on top and the extension bar (with the heavier, stiffer cables) at the back. (I need to bring my shelves another 1 or 2 positions further forward as there is barely enough room for one cable between the shelf and the back of the rack and that makes moving boards around harder than it needs to be.)

The PDU defaults to enabling all ports at startup, so connect to it over telnet and turn off the ports before connecting things and sorting out the network interface to what the rest of the lab needs. (I'm using a 10. range and the PDU was originally set to use 192.168.1.)

That's about it as far as the hardware setup is concerned. Just time to label up each board, note down which PDU port does which device, which serial to USB converter is on which device on the microserver and check the power - my initial problem with one board was traced to the inability of the board to power SATA off the on-board USB port even when using the provided 2A power supply. That meant adding a standard mains adaptor to feed both the SATA power and the board power off the one PDU port - there is little point powering off the board but not the SATA, or vice versa.

I haven't totalled up the expenditure but the biggest expenses were the microserver and the wall mounted rack but don't underestimate how much it will cost to buy 6 IEC plugs, various USB serial converters and how much you may spend on peripheral items.

There is quite a lot of room on the 2 shelves for more boards, what will limit the deployment in this rack is space for the cables, especially power. The shorter the power cables, the easier it will be to maintain the devices in the rack. It might be worth looking at a 12U rack, if only to give plenty of space for cables.

Once I've got the software working, I'll describe what this rack will be doing ... it's got something to do with Debian, ARM, Linux and tests but you've probably already guessed that much ...