prev
Issue 116
22nd June 2008
by Danny Allen
next


This Week...
Work on a "Grid" containment for Plasmoids. A Plasma applet to monitor the WiFi signal strength (on Linux systems). Infrastructure in place for a network settings daemon in the NetworkManager Plasmoid. An Akonadi Plasma data engine, intended for initial use by a "Plasmobiff" applet. "Previewer", a new Plasmoid for previewing files using KParts technology. KDevPlatform (the basis of KDevelop4) gets a plugin for basic Git source versioning control. Start of resurrecting C# support in KDevelop. Key bindings added to display various debugging information in KStars. Progress in sound and scripting support in Parley. KTron gets a new maintainer, with work on porting to KDE 4 and moving to SVG graphics. "Find duplicates" tool in Digikam is now regarded as fully functional. Initial efforts at using PolicyKit for the management of system-wide fonts. Work on scrolling interaction in KOrganizer, and further development in the new "Message List View" of KMail. Copying music tracks to an MP3Tunes locker, and experimental auto-fetching of album artwork in Amarok 2.0. Support for authentication algorithm selection for WEP connections in KNetworkManager. Early beginnings of scripting plugin for KTorrent. Initial support for searching for placemarks, and undo/redo support in the "GeoShape" of KOffice. Beginnings of work on the "Presenter View" in KPresenter, with two synchronised canvasses. "Web forms" component of Kexi becomes based on the kde.org template, for consistency and accessibility reasons. Provisional .wav support in TagLib. Initial import of a Perldoc KIOSlave for KDE 4. Import of a KDE 4 port of sysinfo:/ (as seen in openSUSE 11). KDE 4.1 Beta 2 is tagged for release.

Szymon Stefanek introduces the concepts behind his Summer of Code for KMail, "Enhancements and porting to native Qt4":
Now that KDE 4.1 is about to be released and we should be doing bugfixes only, what are these large commits, file additions, deletions, moves, giant SVN merges that jump out in Commit-Digest once in a while?

...it's me playing in branches :)

Well... in fact it's me working on my Google Summer of Code project and specifically on KMail.

My Summer of Code project has the following goals:
  • Help, and possibly complete, the port of KMail to native Qt4
  • Replace the old Qt3-based KMHeaders with a super-cool "Message List View"
  • Implement a view for "whole threads" of messages
I'm actually in the middle of the project. Several widgets have already been converted to native Qt4 and i'm working on the super-cool "Message List View" which already displays the contents of folders.

Just to make you a bit curious about the sources i'll introduce an interesting problem encountered in the development: the message list view filling algorithm.

The Rich Message List View
When you click on a folder in any e-mail client you usually get its contents displayed in some window. The contents of a folder are obviously messages and you likely want the list to be displayed with some kind of "preferred arrangement". When you're searching for the latest messages received or sent you probably to want the messages to be sorted by date. When you're searching for a particular sender you might want the messages to be sorted by that field. When you're looking at a mailing list you usually want the messages to be arranged in trees which represent discussion threads. Many e-mail clients (soon including KMail) allow you to group the messages and threads by some kind of criteria. For example, you might want to group the messages by "smart" date ranges: "Messages received Today", "Messages received Yesterday", "Messages received Last Week", "Messages from Last Year"...

The mail folders can be very large. It's not unusual to find folders with more than 50000 messages inside (just stay subscribed to the linux-kernel mailing list - or kde-commits! - for some weeks and you'll end up having an example of such a folder). Scanning and organizing all the messages inside such a folder *will* be time consuming. The current KMail implementation is blocking in this sense and even if it uses clever optimisations it still tends to freeze for several seconds when opening a large folder. Freezing is a symptom we need to avoid.

So now you should be getting the big picture. Large folders but no freezing, grouping or no grouping, threading or no threading, sorting by that or by something else. How to approach such an algorithm?

Well.. the actual kmail-soc branch source code has an implementation for this :) I'll leave you the fun of finding the details by reading the comments in the source and i'll outline only the general idea.

First of all, we take care of the freezing problem. The idea is that the UI must be able to process events while the filling algorithm is running. KMail structure is not thread safe at all: multithreading is not an option. Calling "processEvents()" once in a while is also not very nice: it looks so "visual basic"-ish and may lead to a lot of unnecessary calls. So we approach the problem by idle time processing. This is a typical algorithm for UI-based programs.

We set up a zero-interval timer that the Qt core will trigger every time it has nothing else to do (that is, the event loop is "idle"). Once we're called we will be able to do a chunk of work. Idle time processing imposes an obvious restriction on our implementation: it *must* be interruptible. That is, it must be able to perform a part of the work, stop at a certain point and later resume exactly from that point.

The underlying folder is assumed to be a "flat" structure in that the messages can be retrieved by using a numeric index. The natural splitting approach is to cover a contiguous part of the message index space in each work chunk.

We start at the initial index and go upwards... well... the direction and order of the messages to access in the underlying storage is yet another part of the story but i'll omit it for now.

We actually define a "maximum chunk processing time" as a constant in the sources and while processing messages, once in a while, we check if we ran out of time. If we did, then we store our current index as initial index for the next job and return control to Qt. And this works nicely, even if the real-world implementation is really more complex than the one i've just described.

Next we take care of message threading. Have you ever wondered how your e-mail clients knows that a message is a reply to another message? The answer is, it either gets this information (and trusts) the sender's e-mail client or it attempts to guess. The basic idea is that each message should contain a "Message-Id" field with an unique ID string. Then every (direct) reply to that message should contain this unique ID in the "In-Reply-To" header field. This doesn't always work because of broken or featureless clients. There is also a problem when one of the thread messages is missing since it literally breaks the tree in two non-joinable pieces. So we have the "References-Id" field. Smart e-mail clients put there several unique identifiers of messages belonging to the same thread. There is a complex rationale for that we're using only one specific entry in that References-Id (see sources for more information). But anyway, we have two somewhat "direct" means for threading messages. Then there are hopeless e-mail clients which do not include these fields in replies. So we try to guess by looking at the subjects... but i'll leave the details here and go on with the algorithm.

Messages then "come in" from the folder in some order. The order does not necessarily need to be the "right one". That is, a reply to a message X may come in before the message X itself. This means that initially we'll miss the parent for the reply and *later* we have to find out that the original message is its (perfect?) parent. This complicates the things a bit since we can't just append the messages to the trees. We can append them when a perfect parent is found but when no parent is found we don't know if there will be one later. The current solution tries to keep the message without a parent unattacched as long as possible (during a single execution of the fill view algorithm) hoping that the parent will jump out. When we reach the end of the folder without finding the parent then the parentless child must be attacched to the view and shown to the user (even if an orphan, it's still a valid message).

But then the user presses the "check mail" button and new messages are downloaded from the server... and the missing parent suddently appears. Then we have an additional hazard: we must be able to change the hierarchy of the already displayed messages... and we must do it on-the-fly, possibly without bothering the user that might be actually reading *another* message in the same folder (and maybe, as per Murphy's Law, belonging to the very same thread).

Then we have message grouping. Determining the group that a *single* messsage belongs to is quite simple. A bigger problem comes out when we want to be able to group threads by something that is not the first e-mail in the thread. Maybe it's not totally obvious but think of a group of "all the discussions that had activity today". This is actually the group of threads for that the user has received at least one message today. To implement this, in certain configurations, groups must be held up until the end while in others they can be attached directly to the view.

Anyway, at the moment of writing, the sources in the kmail-soc branch have a nearly working 4 pass solution for the problem (which in fact has several additional hardcore details). It works but it's complex and hard to read. This might also mean that it will be hard to mantain in the future. Something must be done in this direction too.

Then there is the potential Akonadi port... but that's yet another part of the story. Ah, and skinning too! ...we have both eye-candy lovers and conservators that want the view "just like it was in KMail 1.0" :D


Well...I hope to have interested you at least a bit :)

I always try to make sure that the branch code compiles. You're encouraged to try it out and send feedback!

About Me
My name is Szymon Stefanek. I live in Italy and i'm studying Informatic Engineering at the University of Siena. Currently i've passed 29 of the 31 required exams and i'm planning to graduate in December 2008.

i'm an open source software enthusiast and I have written free code since 1998. i'm the leader of the KVIrc project that is actually in a mature state. Other developers are taking care of it and i'm trying to expand my horizons by joining another open source team.

I have used KDE since 1998. In the past, I have contributed some artwork and several pieces of my code have indirectly ended up in the KDE codebase.

Because we can't get enough of Parley, Daniel Laidig writes about how he is adding a welcome screen to Parley:
A few days ago I committed a rather small addition to Parley that changes the visual appearance a lot: a welcome screen. Until now, Parley appeared showing a table view which is not exactly the most beautiful appearance I can think of, and certainly doesn't give the impression that learning vocabulary is fun (which is true, but the aim of Parley should be to make it fun!).

One way to make Parley look more appealing and to make it easier to use at the same time was to introduce a welcome screen. This screen is displayed instead of the table when no document is loaded and provides easy access to the most important actions, like creating a new collection, opening existing collections, and downloading new files via Get Hot New Stuff. There is a list of recent files which coincidentally resembles the welcome page of Gwenview a lot and provides Goya-powered buttons for opening the file or directly start practicing. With this, in 99% of the cases you just need one click to get started, however users who prefer the old behavior can still choose to automatically open the last file in the settings dialog.

My hope is that this will not be the last change to create an even better user interface for Parley in KDE 4.2. Frederik and I were discussing to start using KParts for the different modules (welcome, editor and practice) to create a more consistent interface. I don't know yet how easy that will be, since KParts don't support dock widgets yet, but hopefully that will change in the next few months.



Statistics
Commits: 2292 by 247 developers, 7529 lines modified, 1455 new files.
Open Bugs: 16678
Open Wishes: 14277
Bugs Opened: 463 in the last 7 days.
Bugs Closed: 421 in the last 7 days.

Commit Summary
Module Commits
/trunk/KDE
693
/trunk/l10n-kde4
347
/trunk/extragear
240
/branches/work
216
/trunk/playground
215
/trunk/koffice
153
/branches/kdepim
97
/branches/stable
64
/branches/extragear
57
/trunk/www
42
Lines Developer Commits
1043
Marc Mutz
244
222
Hamish Rodda
81
139
Gilles Caulier
59
31
Frank Osterfeld
53
191
David Nolden
52
118
Lukas Appelhans
52
99
Albert Astals Cid
50
189
Allen Winter
50
158
Stefan Nikolaus
50
9
Ivan Čukić
45

Internationalisation (i18n) Status
Language Percentage Complete
Ukrainian
99%
Portuguese
99%
Swedish
99%
Greek
99%
Estonian
95%
Galician
93%
French
92%
Low Saxon
90%
Japanese
90%
Spanish
89%

Bug Killers and Buzz
Program Buzz
Amarok
  9815
K3B
  4875
KMail
  4840
Kopete
  3320
KDevelop
  2595
Plasma
  2489
Kaffeine
  2037
Kate
  2001
Solid
  1873
Kontact
  1790


Person Buzz
David Faure
  2110
Stephan Kulow
  1749
Aaron Seigo
  1390
Torsten Rahn
  1367
Jonathan Riddell
  1132
Laurent Montel
  1030
Stephan Binner
  782
Thiago Macieira
  668
Zack Rusin
  638
Adriaan de Groot
  631
Commit Countries

Commit Demographics
Sex
112 %       Male
7.29 %       (unknown)
0.561 %       Female
Motivation
55.6 %       Volunteer
53.7 %       (unknown)
11.4 %       Commercial
 
Ages
74.0 %       (unknown)
20.5 %       25 to 34
15.4 %       18 to 24
4.86 %       35 to 44
3.41 %       45 to 54
2.52 %       Under 18


Contents
  Bug Fixes Features Optimise Security Other
Accessibility
Development Tools [*] [*]
Educational [*] [*] [*]
Graphics [*] [*] [*] [*]
KDE-Base [*] [*] [*]
KDE-PIM [*] [*]
Office [*] [*] [*] [*]
Konqueror
Multimedia [*] [*] [*]
Networking Tools [*] [*]
User Interface [*]
Utilities [*] [*] [*]
Games [*] [*] [*]
Other [*]

There are 131 selections this week.

Bug Fixes
Educational
Frederik Gladhorn committed changes in /trunk/KDE/kdeedu:
Nice catch Avgoustinos!
And in time for 4.1.

Editing languages should work now.
- after adding and then immediately removing a language one of the existing languages would be blown away
- reload the languages after they have been changed
Diffs: 1, 2 Revision 821561

David Capel committed changes in /branches/work/soc-parley/parley/src:
Fix fixes, timers added (for now its only invisible ones -- visible ones will come later)
the theme was updated so a button could be shown (makes things more obvious for testing purposes)

Note to the wise: distro kde4 packages can cause trouble...
Diffs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 (+ 3 more) Revision 822109
View Visual Changes (to 1 file)

Jason Harris committed changes in /trunk/KDE/kdeedu/kstars/kstars:
Fix UI bugs in the Options window, Catalogs tab. Certain controls (the star magnitude limits and
deep-sky catalogs) did not enable the Apply button when modified. The deep-sky catalogs changed
the map immediately without needing Apply or Ok, which is inconsistent. Now these controls behave
like all of the other options: Apply, Ok, and Cancel should all work on them as expected.

Thanks for pointing this out, James.
Diffs: 1, 2 Revision 822426

Jason Harris committed changes in /trunk/KDE/kdeedu/kstars/kstars:
Fixing a usability issue in the options window.
Toggling the view toolbar buttons now toggles the corresponding control in the options dialog.
This is useful for those who keep the options window open while using the program.

TODO: there's no meta-option for deep-sky objects in the options window. I'll ask translators if it's ok to add one for 4.2.

Thanks for pointing this out, James.
Diffs: 1, 2 Revision 822804

Games
Ian Wadham committed changes in /trunk/KDE/kdegames/kubrick/src:
Fix the integration of Singmaster moves with all menu actions.
Change Demo shortcut to Ctrl+D and the Down move to D.
Clean up and remove debugging outputs.
Diffs: 1, 2, 3, 4, 5, 6, 7 Revision 820786

Graphics
Jesper Pedersen committed changes in /trunk/extragear/graphics/kphotoalbum:
Deleting an image that was no longer on disk resulted in an error dialog telling that it could not be deleted.

Now it will just act as if it did actually delete it, and take the image out of the database.
Diffs: 1, 2 Revision 820940