Planet Skolelinux

September 26, 2014

Petter Reinholdtsen

How to test Debian Edu Jessie despite some fatal problems with the installer

The Debian Edu / Skolelinux project provide a Linux solution for schools, including a powerful desktop with education software, a central server providing web pages, user database, user home directories, central login and PXE boot of both clients without disk and the installation to install Debian Edu on machines with disk (and a few other services perhaps to small to mention here). We in the Debian Edu team are currently working on the Jessie based version, trying to get everything in shape before the freeze, to avoid having to maintain our own package repository in the future. The current status can be seen on the Debian wiki, and there is still heaps of work left. Some fatal problems block testing, breaking the installer, but it is possible to work around these to get anyway. Here is a recipe on how to get the installation limping along.

First, download the test ISO via ftp, http or rsync (use ftp.skolelinux.org::cd-edu-testing-nolocal-netinst/debian-edu-amd64-i386-NETINST-1.iso). The ISO build was broken on Tuesday, so we do not get a new ISO every 12 hours or so, but thankfully the ISO we already got we are able to install with some tweaking.

When you get to the Debian Edu profile question, go to tty2 (use Alt-Ctrl-F2), run

nano /usr/bin/edu-eatmydata-install

and add 'exit 0' as the second line, disabling the eatmydata optimization. Return to the installation, select the profile you want and continue. Without this change, exim4-config will fail to install due to a known bug in eatmydata.

When you get the grub question at the end, answer /dev/sda (or if this do not work, figure out what your correct value would be. All my test machines need /dev/sda, so I have no advice if it do not fit your need.

If you installed a profile including a graphical desktop, log in as root after the initial boot from hard drive, and install the education-desktop-XXX metapackage. XXX can be kde, gnome, lxde, xfce or mate. If you want several desktop options, install more than one metapackage. Once this is done, reboot and you should have a working graphical login screen. This workaround should no longer be needed once the education-tasks package version 1.801 enter testing in two days.

I believe the ISO build will start working on two days when the new tasksel package enter testing and Steve McIntyre get a chance to update the debian-cd git repository. The eatmydata, grub and desktop issues are already fixed in unstable and testing, and should show up on the ISO as soon as the ISO build start working again. Well the eatmydata optimization is really just disabled. The proper fix require an upload by the eatmydata maintainer applying the patch provided in bug #702711. The rest have proper fixes in unstable.

I hope this get you going with the installation testing, as we are quickly running out of time trying to get our Jessie based installation ready before the distribution freeze in a month.

September 26, 2014 10:20 AM

September 25, 2014

Petter Reinholdtsen

Suddenly I am the new upstream of the lsdvd command line tool

I use the lsdvd tool to handle my fairly large DVD collection. It is a nice command line tool to get details about a DVD, like title, tracks, track length, etc, in XML, Perl or human readable format. But lsdvd have not seen any new development since 2006 and had a few irritating bugs affecting its use with some DVDs. Upstream seemed to be dead, and in January I sent a small probe asking for a version control repository for the project, without any reply. But I use it regularly and would like to get an updated version into Debian. So two weeks ago I tried harder to get in touch with the project admin, and after getting a reply from him explaining that he was no longer interested in the project, I asked if I could take over. And yesterday, I became project admin.

I've been in touch with a Gentoo developer and the Debian maintainer interested in joining forces to maintain the upstream project, and I hope we can get a new release out fairly quickly, collecting the patches spread around on the internet into on place. I've added the relevant Debian patches to the freshly created git repository, and expect the Gentoo patches to make it too. If you got a DVD collection and care about command line tools, check out the git source and join the project mailing list. :)

September 25, 2014 09:20 AM

September 21, 2014

Petter Reinholdtsen

Hva henger under skibrua over E16 på Sollihøgda?

Rundt omkring i Oslo og Østlandsområdet henger det bokser over veiene som jeg har lurt på hva gjør. De har ut fra plassering og vinkling sett ut som bokser som sniffer ut et eller annet fra forbipasserende trafikk, men det har vært uklart for meg hva det er de leser av. Her om dagen tok jeg bilde av en slik boks som henger under ei skibru på Sollihøgda:

Boksen er tydelig merket «Kapsch >>>», logoen til det sveitsiske selskapet Kapsch som blant annet lager sensorsystemer for veitrafikk. Men de lager mye forskjellig, og jeg kjente ikke igjen boksen på utseendet etter en kjapp titt på produktlista til selskapet.

I og med at boksen henger over veien E16, en riksvei vedlikeholdt av Statens Vegvesen, så antok jeg at det burde være mulig å bruke REST-API-et som gir tilgang til vegvesenets database over veier, skilter og annet veirelatert til å finne ut hva i alle dager dette kunne være. De har både en datakatalog og et søk, der en kan søke etter ulike typer oppføringer innen for et gitt geografisk område. Jeg laget et enkelt shell-script for å hente ut antall av en gitt type innenfor området skibrua dekker, og listet opp navnet på typene som ble funnet. Orket ikke slå opp hvordan URL-koding av aktuelle strenger kunne gjøres mer generisk, og brukte en stygg sed-linje i stedet.

#!/bin/sh
urlmap() {
    sed \
    -e 's/  / /g'   -e 's/{/%7B/g'  \
    -e 's/}/%7D/g'  -e 's/\[/%5B/g' \
    -e 's/\]/%5D/g' -e 's/ /%20/g'  \
    -e 's/,/%2C/g'  -e 's/\"/%22/g' \
    -e 's/:/%3A/g'
}

lookup() {
    url="$1"
    curl -s -H 'Accept: application/vnd.vegvesen.nvdb-v1+xml' \
       "https://www.vegvesen.no/nvdb/api$url" | xmllint --format -
}

for id in $(seq 1 874) ; do
    search="{
  lokasjon: {
    bbox: \"10.34425,59.96386,10.34458,59.96409\",
    srid: \"WGS84\"
  },
   objektTyper: [{
     id: $id, antall: 10
   }]
}"

    query=/sok?kriterie=$(echo $search | urlmap)
    if lookup "$query" |
    grep -q '<totaltAntallReturnert>0<'
    then
    :
    else
    echo $id
    lookup "/datakatalog/objekttyper/$id" |grep '^  <navn>'
    fi
done

exit 0
Aktuelt ID-område 1-874 var riktig i datakatalogen da jeg laget scriptet. Det vil endre seg over tid. Skriptet listet så opp aktuelle typer i og rundt skibrua:
5
  <navn>Rekkverk</navn>
14
  <navn>Rekkverksende</navn>
47
  <navn>Trafikklomme</navn>
49
  <navn>Trafikkøy</navn>
60
  <navn>Bru</navn>
79
  <navn>Stikkrenne/Kulvert</navn>
80
  <navn>Grøft, åpen</navn>
86
  <navn>Belysningsstrekning</navn>
95
  <navn>Skiltpunkt</navn>
96
  <navn>Skiltplate</navn>
98
  <navn>Referansestolpe</navn>
99
  <navn>Vegoppmerking, langsgående</navn>
105
  <navn>Fartsgrense</navn>
106
  <navn>Vinterdriftsstrategi</navn>
172
  <navn>Trafikkdeler</navn>
241
  <navn>Vegdekke</navn>
293
  <navn>Breddemåling</navn>
301
  <navn>Kantklippareal</navn>
318
  <navn>Snø-/isrydding</navn>
445
  <navn>Skred</navn>
446
  <navn>Dokumentasjon</navn>
452
  <navn>Undergang</navn>
528
  <navn>Tverrprofil</navn>
532
  <navn>Vegreferanse</navn>
534
  <navn>Region</navn>
535
  <navn>Fylke</navn>
536
  <navn>Kommune</navn>
538
  <navn>Gate</navn>
539
  <navn>Transportlenke</navn>
540
  <navn>Trafikkmengde</navn>
570
  <navn>Trafikkulykke</navn>
571
  <navn>Ulykkesinvolvert enhet</navn>
572
  <navn>Ulykkesinvolvert person</navn>
579
  <navn>Politidistrikt</navn>
583
  <navn>Vegbredde</navn>
591
  <navn>Høydebegrensning</navn>
592
  <navn>Nedbøyningsmåling</navn>
597
  <navn>Støy-luft, Strekningsdata</navn>
601
  <navn>Oppgravingsdata</navn>
602
  <navn>Oppgravingslag</navn>
603
  <navn>PMS-parsell</navn>
604
  <navn>Vegnormalstrekning</navn>
605
  <navn>Værrelatert strekning</navn>
616
  <navn>Feltstrekning</navn>
617
  <navn>Adressepunkt</navn>
626
  <navn>Friksjonsmåleserie</navn>
629
  <navn>Vegdekke, flatelapping</navn>
639
  <navn>Kurvatur, horisontalelement</navn>
640
  <navn>Kurvatur, vertikalelement</navn>
642
  <navn>Kurvatur, vertikalpunkt</navn>
643
  <navn>Statistikk, trafikkmengde</navn>
647
  <navn>Statistikk, vegbredde</navn>
774
  <navn>Nedbøyningsmåleserie</navn>
775
  <navn>ATK, influensstrekning</navn>
794
  <navn>Systemobjekt</navn>
810
  <navn>Vinterdriftsklasse</navn>
821
  <navn>Funksjonell vegklasse</navn>
825
  <navn>Kurvatur, stigning</navn>
838
  <navn>Vegbredde, beregnet</navn>
862
  <navn>Reisetidsregistreringspunkt</navn>
871
  <navn>Bruksklasse</navn>

Av disse ser ID 775 og 862 mest relevant ut. ID 775 antar jeg refererer til fotoboksen som står like ved brua, mens «Reisetidsregistreringspunkt» kanskje kan være boksen som henger der. Hvordan finner jeg så ut hva dette kan være for noe. En titt på datakatalogsiden for ID 862/Reisetidsregistreringspunkt viser at det er finnes 53 slike målere i Norge, og hvor de er plassert, men gir ellers få detaljer. Det er plassert 40 på østlandet og 13 i Trondheimsregionen. Men siden nevner «AutoPASS», og hvis en slår opp oppføringen på Sollihøgda nevner den «Ciber AS» som ID for eksternt system. (Kan det være snakk om Ciber Norge AS, et selskap eid av Ciber Europe Bv?) Et nettsøk på «Ciber AS autopass» fører meg til en artikkel fra NRK Trøndelag i 2013 med tittel «Sjekk dette hvis du vil unngå kø». Artikkelen henviser til vegvesenets nettside reisetider.no som har en kartside for Østlandet som viser at det måles mellom Sandvika og Sollihøgda. Det kan dermed se ut til at jeg har funnet ut hva boksene gjør.

Hvis det stemmer, så er dette bokser som leser av AutoPASS-ID-en til alle passerende biler med AutoPASS-brikke, og dermed gjør det mulig for de som kontrollerer boksene å holde rede på hvor en gitt bil er når den passerte et slikt målepunkt. NRK-artikkelen forteller at denne informasjonen i dag kun brukes til å koble to AutoPASS-brikkepasseringer passeringer sammen for å beregne reisetiden, og at bruken er godkjent av Datatilsynet. Det er desverre ikke mulig for en sjåfør som passerer under en slik boks å kontrollere at AutoPASS-ID-en kun brukes til dette i dag og i fremtiden.

I tillegg til denne type AutoPASS-sniffere vet jeg at det også finnes mange automatiske stasjoner som tar betalt pr. passering (aka bomstasjoner), og der lagres informasjon om tid, sted og bilnummer i 10 år. Finnes det andre slike sniffere plassert ut på veiene?

Personlig har jeg valgt å ikke bruke AutoPASS-brikke, for å gjøre det vanskeligere og mer kostbart for de som vil invadere privatsfæren og holde rede på hvor bilen min beveger seg til enhver tid. Jeg håper flere vil gjøre det samme, selv om det gir litt høyere private utgifter (dyrere bompassering). Vern om privatsfæren koster i disse dager.

Takk til Jan Kristian Jensen i Statens Vegvesen for tips om dokumentasjon på vegvesenets REST-API.

September 21, 2014 07:50 AM

September 16, 2014

Petter Reinholdtsen

Speeding up the Debian installer using eatmydata and dpkg-divert

The Debian installer could be a lot quicker. When we install more than 2000 packages in Skolelinux / Debian Edu using tasksel in the installer, unpacking the binary packages take forever. A part of the slow I/O issue was discussed in bug #613428 about too much file system sync-ing done by dpkg, which is the package responsible for unpacking the binary packages. Other parts (like code executed by postinst scripts) might also sync to disk during installation. All this sync-ing to disk do not really make sense to me. If the machine crash half-way through, I start over, I do not try to salvage the half installed system. So the failure sync-ing is supposed to protect against, hardware or system crash, is not really relevant while the installer is running.

A few days ago, I thought of a way to get rid of all the file system sync()-ing in a fairly non-intrusive way, without the need to change the code in several packages. The idea is not new, but I have not heard anyone propose the approach using dpkg-divert before. It depend on the small and clever package eatmydata, which uses LD_PRELOAD to replace the system functions for syncing data to disk with functions doing nothing, thus allowing programs to live dangerous while speeding up disk I/O significantly. Instead of modifying the implementation of dpkg, apt and tasksel (which are the packages responsible for selecting, fetching and installing packages), it occurred to me that we could just divert the programs away, replace them with a simple shell wrapper calling "eatmydata $program $@", to get the same effect. Two days ago I decided to test the idea, and wrapped up a simple implementation for the Debian Edu udeb.

The effect was stunning. In my first test it reduced the running time of the pkgsel step (installing tasks) from 64 to less than 44 minutes (20 minutes shaved off the installation) on an old Dell Latitude D505 machine. I am not quite sure what the optimised time would have been, as I messed up the testing a bit, causing the debconf priority to get low enough for two questions to pop up during installation. As soon as I saw the questions I moved the installation along, but do not know how long the question were holding up the installation. I did some more measurements using Debian Edu Jessie, and got these results. The time measured is the time stamp in /var/log/syslog between the "pkgsel: starting tasksel" and the "pkgsel: finishing up" lines, if you want to do the same measurement yourself. In Debian Edu, the tasksel dialog do not show up, and the timing thus do not depend on how quickly the user handle the tasksel dialog.

Machine/setup Original tasksel Optimised tasksel Reduction
Latitude D505 Main+LTSP LXDE 64 min (07:46-08:50) 44 min (11:27-12:11) >20 min 18%
Latitude D505 Roaming LXDE 57 min (08:48-09:45) 34 min (07:43-08:17) 23 min 40%
Latitude D505 Minimal 22 min (10:37-10:59) 11 min (11:16-11:27) 11 min 50%
Thinkpad X200 Minimal 6 min (08:19-08:25) 4 min (08:04-08:08) 2 min 33%
Thinkpad X200 Roaming KDE 19 min (09:21-09:40) 15 min (10:25-10:40) 4 min 21%

The test is done using a netinst ISO on a USB stick, so some of the time is spent downloading packages. The connection to the Internet was 100Mbit/s during testing, so downloading should not be a significant factor in the measurement. Download typically took a few seconds to a few minutes, depending on the amount of packages being installed.

The speedup is implemented by using two hooks in Debian Installer, the pre-pkgsel.d hook to set up the diverts, and the finish-install.d hook to remove the divert at the end of the installation. I picked the pre-pkgsel.d hook instead of the post-base-installer.d hook because I test using an ISO without the eatmydata package included, and the post-base-installer.d hook in Debian Edu can only operate on packages included in the ISO. The negative effect of this is that I am unable to activate this optimization for the kernel installation step in d-i. If the code is moved to the post-base-installer.d hook, the speedup would be larger for the entire installation.

I've implemented this in the debian-edu-install git repository, and plan to provide the optimization as part of the Debian Edu installation. If you want to test this yourself, you can create two files in the installer (or in an udeb). One shell script need do go into /usr/lib/pre-pkgsel.d/, with content like this:

#!/bin/sh
set -e
. /usr/share/debconf/confmodule
info() {
    logger -t my-pkgsel "info: $*"
}
error() {
    logger -t my-pkgsel "error: $*"
}
override_install() {
    apt-install eatmydata || true
    if [ -x /target/usr/bin/eatmydata ] ; then
        for bin in dpkg apt-get aptitude tasksel ; do
            file=/usr/bin/$bin
            # Test that the file exist and have not been diverted already.
            if [ -f /target$file ] ; then
                info "diverting $file using eatmydata"
                printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
                    > /target$file.edu
                chmod 755 /target$file.edu
                in-target dpkg-divert --package debian-edu-config \
                    --rename --quiet --add $file
                ln -sf ./$bin.edu /target$file
            else
                error "unable to divert $file, as it is missing."
            fi
        done
    else
        error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
    fi
}

override_install

To clean up, another shell script should go into /usr/lib/finish-install.d/ with code like this:

#! /bin/sh -e
. /usr/share/debconf/confmodule
error() {
    logger -t my-finish-install "error: $@"
}
remove_install_override() {
    for bin in dpkg apt-get aptitude tasksel ; do
        file=/usr/bin/$bin
        if [ -x /target$file.edu ] ; then
            rm /target$file
            in-target dpkg-divert --package debian-edu-config \
                --rename --quiet --remove $file
            rm /target$file.edu
        else
            error "Missing divert for $file."
        fi
    done
    sync # Flush file buffers before continuing
}

remove_install_override

In Debian Edu, I placed both code fragments in a separate script edu-eatmydata-install and call it from the pre-pkgsel.d and finish-install.d scripts.

By now you might ask if this change should get into the normal Debian installer too? I suspect it should, but am not sure the current debian-installer coordinators find it useful enough. It also depend on the side effects of the change. I'm not aware of any, but I guess we will see if the change is safe after some more testing. Perhaps there is some package in Debian depending on sync() and fsync() having effect? Perhaps it should go into its own udeb, to allow those of us wanting to enable it to do so without affecting everyone.

Update 2014-09-24: Since a few days ago, enabling this optimization will break installation of all programs using gnutls because of bug #702711. An updated eatmydata package in Debian will solve it.

September 16, 2014 12:00 PM

September 10, 2014

Petter Reinholdtsen

Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net

Yesterday, I had the pleasure of attending a talk with the Norwegian Unix User Group about the OpenPGP keyserver pool sks-keyservers.net, and was very happy to learn that there is a large set of publicly available key servers to use when looking for peoples public key. So far I have used subkeys.pgp.net, and some times wwwkeys.nl.pgp.net when the former were misbehaving, but those days are ended. The servers I have used up until yesterday have been slow and some times unavailable. I hope those problems are gone now.

Behind the round robin DNS entry of the sks-keyservers.net service there is a pool of more than 100 keyservers which are checked every day to ensure they are well connected and up to date. It must be better than what I have used so far. :)

Yesterdays speaker told me that the service is the default keyserver provided by the default configuration in GnuPG, but this do not seem to be used in Debian. Perhaps it should?

Anyway, I've updated my ~/.gnupg/options file to now include this line:

keyserver pool.sks-keyservers.net

With GnuPG version 2 one can also locate the keyserver using SRV entries in DNS. Just for fun, I did just that at work, so now every user of GnuPG at the University of Oslo should find a OpenGPG keyserver automatically should their need it:

% host -t srv _pgpkey-http._tcp.uio.no
_pgpkey-http._tcp.uio.no has SRV record 0 100 11371 pool.sks-keyservers.net.
%

Now if only the HKP lookup protocol supported finding signature paths, I would be very happy. It can look up a given key or search for a user ID, but I normally do not want that, but to find a trust path from my key to another key. Given a user ID or key ID, I would like to find (and download) the keys representing a signature path from my key to the key in question, to be able to get a trust path between the two keys. This is as far as I can tell not possible today. Perhaps something for a future version of the protocol?

September 10, 2014 11:10 AM