KDE

About a week ago I started the KDE Relicensing Effort.

Things are progressing slowly but steady. I think within the next few weeks we can relicense most of our code base if we manage to dig up the missing people and get their permission in time.

Tom Albers added a nice toy last weekend that combines the Krazy2 based licensing checker which Allen and I wrote and the relicense checking script into a simple but easy to use web application.

Feel invited to browse it and help getting the listed files relicensed!

Another long standing TODO item finally implemented: With the help of frerich, I was able to add an RSS output feed to the KDE dashboard in no time. There is no excuse anymore: If you break the build without noticing, you're not running KDE 4!

The feed url is

http://developer.kde.org/~dirk/dashboard/index.rss

.

I even added it to my cell phone so that I can continuously watch when the build breaks

Last Friday, the new server hardware for our main SVN server arrived. As you probably noticed, the old server was slowly exceeding its capacity, given the steady and healthy growth of our project.

I couldn't wait and was visiting my office during vacation to unwrap the machine and take a few photos, start to configure the RAID and start some memory and stress testing scripts to make sure that the machine is okay.

But first below some pretty shots of the machine, more will come later.. Continue reading "New KDE SVN server hardware arrived"

Today I've started experimenting a little with CRM114. My bias against statistical spam detectors is big: In the times of geek spam and rapidly changing spam corpus, I don't think it can work as effective as it did a few years ago. I believe that a carefully tuned heuristic scanner can work better, as it is much better at ignoring statistical poisoning. A heuristic scanner can also classify mail it has never seen before correctly, while a statistical tester can not work as good with it. And my best point against statistical analysis is: if it does something wrong, you don't know why. You can only train it and hope in good faith that it won't do that error again.

Anyway, I've read some papers about approximate regular expression matching and the algorithm used by CRM114 and I have to agree it sounds great. So I decided to try it and installed it. I've started with a collection of ham and spam mails and let crm114 sort them, and only trained mis-classified emails (as recommended by the manual). The first impression: It is fast, much faster than SpamAssassin. Second impression: Very disappointing. While it was perfect during a re-test (e.g. the same corpus was tested after the false-classification were trained), it lost accuracy after having learned from the other corpus. For example, after having achieved perfect precision on the spam corpus, I tried the ham corpus, and there a few errors (about 0.1% were incorrectly classified). Therefore, I've re-trained the mis-classified emails, and then decided to re-classify the spam corpus. Result: while the filter had 100% accuracy in the run before, it mis-classified almost 40% of the mails. whoa.

But well, we know that training is a painful process, so lets re-run and re-train against the two collections. And indeed after another round of classify-and-retrain the results were finally usable.

For now, I've only added crm114 to emails sent to me, but I hope to be able to extend it to other kde.org users, to see if it is feasible as a site-wide filtering setup. Especially if a lot of different native languages are involved, this is a difficult situation, and I don't have high hopes.

My main objective for this is to reduce the CPU overhead of the current spam filtering setup for kde.org, because our SPAM volume regularly increases, while the hardware stays the same.

Unfortunately, for the second time this year mail.kdemail.net has a longer downtime. kdemail.net was my personal pet-peeve which I started a few years ago and which was then later converted to be a full-fledged kolab installation hosted somewhere else. Also, I usually don't directly administrate the server anymore, instead David Solbach is helping out with that.

From what I've heard so far is that there was another power supply related hickup that blew away at least one of the harddisks in the RAID array that hosts the kolab mail spool. Currently the raid is being rebuild with spare disks and hopefully some or all data can be rescued. Stay tuned.

Thanks to the helping hands of David and Urs Wolfer, several KDE.org subsites have been converted to the Oxygen layout. This includes lxr.kde.org, download.kde.org and websvn.kde.org. Websvn even has been upgraded to a much newer software, incorporating a few new features and over 2 years worth of bugfixes!

From the other side of sysadmin things, there are lots of pending things to do, and I was only a few days on vacation. The TODO list includes fixing the donation page, upgrading Spamassassin to a security-fixed version and installing the Subversion 1.4 packages on svn.kde.org. Also I plan to tackle the long-standing todo of the new bugzilla installation in the next days.

From the non-sysadmin side of things I've just finished creating tarballs for 3.80.3, the third KDE 4 development snapshot. Since we've created KDE4 packages for openSUSE, we're finding lots of issues and there is a lot of stuff to hack on, and it does look like I'll have more time for hacking soon.

For a few months now I'm playing with an experimental build setup, that triggered by SVN commits, will rebuild KDE svn modules in a clean chroot'ed environment. The purpose is to find unexpected build breakages, because every once in a while somebody forgets to adds a new file to SVN, or to commit a rule to install some header file that is now needed.

Those who regularly join the kde4-devel channel will already know dashbot, an irc bot that is reporting build failures as they're found by the build bot.

Only few people however know that there is also a mailing list where build failures are mailed to. And even fewer seem to know about the status web site where you can get an overview about which modules are failing (or about compiler warnings if the module is not failing at all). You can even guess the week day from that website: If all is red, then it is usually monday. If all is green, it tends to be weekend

Right now, almost all modules are red due to the KListBox->KListWidget migration. Join and hack those errors!

Up to a few days ago we've had a global filter installed which discarded any message that was sent to a kde.org mailing list and was considered to be spam by our spam filtering setup. This was working fine for more than 4 years.

However, there were complains that mails were not arriving to certain mailing lists. I've tried to find out which mails were affected, but failed, because the affected person was unable to forward me the mail in question. Therefore, I've had to bite the bullet and disable the global spam filtering.

Since then, most mailing lists that do not filter by subscribed members have faced massive spamming. Therefore, I've decided to switch them to hold messages of unsubscribed members by default, and installed a script that will monitor these settings and switch it back whenever the list moderator decides to disable that configuration option again.

The number one complaint about this change is that bugzilla email is not automatically sent to the mailing list anymore. The easiest fix is to edit the list settings under Privacy/Spam filters and add a rule with the regexp:

^X-Bugzilla-URL.*http://bugs.kde.org/

and the Action "Accept".

Similarly, mailing lists that absolutely have to be open for all posters should at least add a rule that filters based ^X-Spam-Flag: YES and hold those posts for moderation.

In addition to that I've spent some more hours on tuning the filtering setup to adjust for the recent spam outbreaks, so that things should become normal again soon.

So, it seems, it just happened as I predicted already yesterday on IRC: Instead of talking to each other and to write an email, the involved people blog about it. Well, so be it. Let me just blog about it (because blogging is cool and we don't talk via irc or, by all means heaven forbid, e-mail anymore). Fact is: Misunderstandings happen. Bad intentions are not involved. If you want something to get done by KDE Sysadmins, then its a comparable bad idea to post things to a random IRC channel and expect somebody from the sysadmin team to pick it up. It is also a bad idea to blog about it. It might be a good idea and to start the good oldfashioned e-mail reader and drop an email to sysadmin@kde.org that contains a clear description of what you want to happen and why. For the rest of the ridiculous claims I'd just quote the irc log which should be available to any of the involved parties (but is supposed to be private, so I can't publish it in a hole. Although privateness doesn't seem to be respected anymore when it comes to blogging. Or so it seems). Anyway, the line that is important reads:

[Wed Jan 17 2007] [14:51:33] <danimo> dirk: then make it devzone

So for the very last time: I will not respond to random IRC postings anymore that involve sysadmin stuff from those people. Ever. If you want something, drop an email.

Yay, so vacation time is over. I've just waded through the various pending requests on sysadmin@ and moderated around 200 pending mailing list postings. It seems a lot of things broke or changed during the years, so I'll be busy fixing stuff over the weekend. And who wrote those 17134 emails that arrived in my inbox ? ;-(

Today the database of bugs.kde.org failed unexpectedly, as a lot of messages from users to webmaster@kde.org reported. This has been the first serious database corruption so far, and it's not MySQL alone to blame it seems. Looking into it, there were several SCSI bus resets this morning. However, the RAID selfcheck reports that all disks are still healthy. Lets just hope that it is correct and this was just a one-time event. I've verified that we still regularly backup everything to tape, so any loss shouldn't be too bad.

Luckily I was able to restore the database without loss. If you notice anything strange happening with bugs.kde.org, let me know.

Honestly, I just love hacking while traveling by train - no IRC, no ICQ, no email that steals attention. However, most of the time I don't sit near a power supply socket so my laptop runs from battery. How much do you need for running KDE? I don't mean CPU or RAM ressources, but battery power. I know that you can enhance the battery lifetime by installing more RAM into your laptop - so that the hard disk has less work to do. But how much could you save by not running KDE ?

And I had time to figure out. I've fixed busy-looping in KDE-applications already in the past, but it seems there are new offenders added to our tree on a daily basis. Even if those busy-looping applications (with short timeouts) never appear as significant in the top output - they eat processing time.. and power. For my laptop, running KDE (not doing any user interaction) requires about 1.2 W/h. Thats 10%. With other words, I could hack about 10% longer if those are fixed. Thats a lot, so I started stracing to find the worst offenders. And the hitlist is:

• artsd. No further comment necessary, its the first thing I kill (or don't even install) anyway.
• kded (with dnotify it seems to busy-loop due to debug-output<->directory changed trigger loop)
• klaptopdaemon. The code just does horrible stuff, including installing event listeners everywhere for no reason. Second thing to kill if you actually use a laptop.
• kicker. One reason is the clock, which busy-loops even if nothing is to be expected to be done. But its doing a lot more, I haven't quite figured out yet what it does, but its definitely unnecessary as long as there is no user interaction.

Over the next few weeks I might try fixing those worst offenders. Any help with that is highly appreciated.

Due to an accident I stumbled over Bug 103272, which requests a highscore about the quickest bugfixer. I immediately liked the idea so I sat down and started to build a SQL query for it. Look, its a real beauty:

SELECT
    profiles.login_name, bugs.bug_id, min(bugs_activity.bug_when-bugs.creation_ts) as open_time
FROM
    bugs, bugs_activity, profiles
WHERE
    bugs.resolution='FIXED'
AND
    bugs.bug_status='RESOLVED'
AND
    bugs_activity.bug_when  >= FROM_DAYS(TO_DAYS(NOW())-$days)
AND
    bugs.bug_id=bugs_activity.bug_id
AND
    bugs_activity.added='FIXED'
AND
    bugs_activity.who=profiles.userid
GROUP BY
    profiles.login_name, bugs.bug_id
ORDER BY
    open_time ASC
LIMIT
    $number

Have a look at the result.