Simon St James discusses his new KDE 4 testing initiative,
KDE4Daily:
Hi all! My name is Simon and I am the creator of the KDE4Daily project, which is my first contribution to KDE! The idea originally came from hanging around in #oxygen and observing that, whenever Nuno Fernades Pinheiro wished to see some changes to the widget style, he would need Ruphy to SSH into his machine and update and compile it for him. I thought things could be a bit more streamlined if Riccardo Iaconelli could somehow just add the changed code to a personal repository and have Nuno update to it automatically, but there were probably binary compatibility problems if they used different distributions which would prove awkward. Since virtualisation offers a way to distribute self-contained installations and have them run on different platforms, I thought this might be a way forward.
A little later I realised that this might be a good way of lowering the barrier for other people who want to try out bleeding-edge KDE but who don't run a distribution that provides up-to-date packages, or don't want to deal with the hassle of compiling. So I ran it past the KDE Promotion team and several people expressed their support for the idea, and so I set to work straight away.
The KDE4-install updater itself is not that sophisticated, technically - each build has all files hashed, and when a file changes between builds, the new version is simply globbed into a big bzip'd file. When the user updates, the list of changes is simply replayed and the replacement files are extracted into the correct places and symlinks are re-arranged, as required. This replaying is designed to be robust, so that if it fails (for whatever reason), it can be restarted without ill-effect and hopefully conclude successfully. While this aspect of the project was reasonably simple, dealing with debugging information was annoyingly difficult.
One of the technical challenges was fixing bugs / implementing features "in the field", as it were, without having to re-upload a brand new torrent, so one of the first things I worked on was a way of deploying script updates to users. Several bug fixes have been transparently slipstreamed in since the torrent was first added.
The debugging info for the KDE
modules included in the KDE4Daily install is very big - approximately 512MB when bzip'd. Worse, even small changes in the source code can result in hundreds of megabytes of debug files that are not binary equals of the debug files from the previous build - generating an additional 300 MB per day of debugging info is not unusual. My upload bandwidth makes uploading this much per day unfeasible, but KDE4Daily's usefulness as a debugging aid is significantly reduced if it cannot be provided when an app crashes - without it, the backtraces given by DrKonqui are not really useful to developers.
Thankfully, the debugging info is fantastically susceptible to utilities such as
bsdiff and bspatch, which help bring this down to a much more manageable 10-20MB per day of patches. So, after compiling a build, the fat, debug-filled executables are stripped and separated from their debug info and tagged in a way that associates them with this extracted debug info. Ordinarily, the way the GNU tools work means that an otherwise unchanged executable would change whenever its debugging info did (a fairly common occurrence), but a bit of jiggery-pokery and a naughty hack to GDB eliminate this inefficiency. If (not when ;)) an app crashes in KDE4Daily, a FUSE daemon written in Ruby intercepts requests for debug information from GDB, downloads a mapping that associates the executable's tag with its debug info, downloads the "original" debug info for this executable and the small patch required to bring it up to date, combines the two and then delivers the full, up-to-date debugging info to GDB for backtrace generation.
Preparing an install is about 99% automated. I only deploy a build if all modules have compiled cleanly and while there are occasional patches where one or two modules might have problems, on the whole compilation is pretty reliable. Building takes about 2-5 hours, and stripping and diffing the debugging info takes 1-3 hours. Generally, I need to upload anywhere between 20 and 70 MB per day, all-inclusive.
The torrent was uploaded two weeks ago but not widely publicised. Eventually, Aaron Seigo, Jos Poortvliet, Sebastian Kügler and Anne-Marie Mahfouf goaded me into writing a Dot article (my first!) and Jonathan Riddell helped knock it into shape. Many more people downloaded it than I had expected - I was expecting maybe one hundred or so people over the lifetime of the project, but at its peak, more than 1100 people were on the torrent! Dirk Mueller very kindly provided hosting for the updates and debugging info, and Martin and Kiyoshi provided hosting for the image itself for people who have no access to BitTorrent. Many people approached me with support and extra information (Liquidat blogged on
how to convert the image so that VirtualBox users could use it) and left
encouraging comments on the Dot article, and a gratifying number of people in #kde4-krush seemed to be using it! :)
I always thought it was pretty cool that when I first upped the torrent, Aaron Seigo himself was the first to hop on and has continued to seed ever since - I think this nicely highlights the "community" aspect of KDE :)
On the whole, it's been a fun challenge and overall a very positive first KDE experience - post-4.0, I hope to continue to contribute to KDE with some libs work and some apps :)
