KDE sysadmin

Akademy 2008 was a blast, I enjoyed my time there very much. It felt just perfectly organized, so big thank you to all of you who helped to make it happen. It was running very professional with very very few (if any at all) noticeable glitches. I think our biggest issue was the it's own success: all the events were fairly crowded

As I didn't find time to actually get something done during Akademy, I've spent yesterday on catching up with long overdue sysadmin tasks, which included to upgrade one of our core servers to a software state which isn't 4 years out of date. Unfortunately the new database didn't really like to be treated by bugzilla anymore the way it used to be, so after the successful upgrade I ended up with a nonworking bugs.kde.org installation.

Luckily, Matt Rogers and myself have already prepared a new bugzilla installation on a test system for the last year or so, so out of the two possible options of debugging and fixing our outdated buzilla installation to work again with the new database and the switch to the new bugzilla I chose the latter. It has the drawback of causing some regression pain in the near future, but we finally have a much easier to maintain setup that we can upgrade with newer features easier than the old version. The new version for example allows many of the custom features that we added before via patches to be configured out of the box. Configuration is always better than Customisation, so thats a really good thing that I'm very happy about.

Currently we have quite some quirks yet to go through and to sort out, but things should be working much better than before quite soon, and I'm confident that we get that done. I need some help with the web template to add some useful grouping, colors, icons and other basic layout stuff to make it at least somewhat useable again. Contact me if you're interested. Helping hands are more than welcome.

short stories:

• kdetalk.net back online, after Martin Konold was able to resurrect the hard drive
• KDE 4.1 snapshot 4.0.63 released, openSUSE 10.3 packages are available
• Fosdem 2008 is coming up, hope to meet you there!

I'm scared as well, but not because of the state of things - I think it is improving rapidly. And after all - it is named RC1, not final. I'm just scared because we`re overwhelmed by the download traffic (between 1-2 Gigabit per second) and we definitely need more bandwidth for the next release!

Finally! After long waiting and some fun with German custom duty, the new Hardware donation to KDE e.V. arrived a few days ago. Watch the dot for an exact announcement once the details are figured out

Continue reading "KDE e.V. received another server"

Last Friday, the new server hardware for our main SVN server arrived. As you probably noticed, the old server was slowly exceeding its capacity, given the steady and healthy growth of our project.

I couldn't wait and was visiting my office during vacation to unwrap the machine and take a few photos, start to configure the RAID and start some memory and stress testing scripts to make sure that the machine is okay.

But first below some pretty shots of the machine, more will come later.. Continue reading "New KDE SVN server hardware arrived"

Thanks to the helping hands of David and Urs Wolfer, several KDE.org subsites have been converted to the Oxygen layout. This includes lxr.kde.org, download.kde.org and websvn.kde.org. Websvn even has been upgraded to a much newer software, incorporating a few new features and over 2 years worth of bugfixes!

From the other side of sysadmin things, there are lots of pending things to do, and I was only a few days on vacation. The TODO list includes fixing the donation page, upgrading Spamassassin to a security-fixed version and installing the Subversion 1.4 packages on svn.kde.org. Also I plan to tackle the long-standing todo of the new bugzilla installation in the next days.

From the non-sysadmin side of things I've just finished creating tarballs for 3.80.3, the third KDE 4 development snapshot. Since we've created KDE4 packages for openSUSE, we're finding lots of issues and there is a lot of stuff to hack on, and it does look like I'll have more time for hacking soon.

Up to a few days ago we've had a global filter installed which discarded any message that was sent to a kde.org mailing list and was considered to be spam by our spam filtering setup. This was working fine for more than 4 years.

However, there were complains that mails were not arriving to certain mailing lists. I've tried to find out which mails were affected, but failed, because the affected person was unable to forward me the mail in question. Therefore, I've had to bite the bullet and disable the global spam filtering.

Since then, most mailing lists that do not filter by subscribed members have faced massive spamming. Therefore, I've decided to switch them to hold messages of unsubscribed members by default, and installed a script that will monitor these settings and switch it back whenever the list moderator decides to disable that configuration option again.

The number one complaint about this change is that bugzilla email is not automatically sent to the mailing list anymore. The easiest fix is to edit the list settings under Privacy/Spam filters and add a rule with the regexp:

^X-Bugzilla-URL.*http://bugs.kde.org/

and the Action "Accept".

Similarly, mailing lists that absolutely have to be open for all posters should at least add a rule that filters based ^X-Spam-Flag: YES and hold those posts for moderation.

In addition to that I've spent some more hours on tuning the filtering setup to adjust for the recent spam outbreaks, so that things should become normal again soon.

Yay, so vacation time is over. I've just waded through the various pending requests on sysadmin@ and moderated around 200 pending mailing list postings. It seems a lot of things broke or changed during the years, so I'll be busy fixing stuff over the weekend. And who wrote those 17134 emails that arrived in my inbox ? ;-(

Today the database of bugs.kde.org failed unexpectedly, as a lot of messages from users to webmaster@kde.org reported. This has been the first serious database corruption so far, and it's not MySQL alone to blame it seems. Looking into it, there were several SCSI bus resets this morning. However, the RAID selfcheck reports that all disks are still healthy. Lets just hope that it is correct and this was just a one-time event. I've verified that we still regularly backup everything to tape, so any loss shouldn't be too bad.

Luckily I was able to restore the database without loss. If you notice anything strange happening with bugs.kde.org, let me know.

Over the last few weeks we've been hit by web robots trying to traverse our bugs database, with different sophistication and query rate. I'm not sure what the point of that action is - since we're not serving any harvestable email address to anonymous visitors. Most likely this is a for the only reason of a pure DDoS attack - with increasing success. Therefore I've enabled more drastic counter-measures by filtering requests which match certain HTTP request header patterns.

Unfortunately this wasn't enough, so today I tried another (unintrusive) counter-measure: bugs.kde.org now requires SSL. I hope that the usual traversing bots do not support SSL. Thats just a guess of course, I might be wrong. We'll see. It's the last step (in my imagination) before requiring a login for viewing bug reports, which would be quite inconvenient to our users. If you have any better idea how to get rid of those robots, please post a comment.

This has been made possible by GoDaddy who kindly signed our SSL certificate for free. Finally no self-signed invalid certificates on KDE.org anymore!

Ugh. Lots and lots of German Nazi SPAM hitting our mail server right now. We're trying to filter them via the following rules:

header __SOBER_P_MSGID Message-ID =~ /<[0-9a-f\.]{15,22}\@/
header __SOBER_P_CTYPE Content-Type =~ /text\/plain.*charset=\"us-ascii\"/
header __SOBER_P_PRIO X-Priority =~ /^3 /
header __SOBER_P_IMP Importance =~ /^Normal/

meta SOBER_P_SPAM (__SOBER_P_MSGID && __SOBER_P_CTYPE && __SOBER_P_PRIO && __SOBER_P_IMP )
score SOBER_P_SPAM 18.0
describe SOBER_P_SPAM Rassistische Mail Sober-P

Let me know if you find any mail that still slips through.

Just a few words.. checkouts on our core KDE servers are crawling because the main svn.kde.org server is hammered by probably hundreds of developers checking out their sources. I've been waiting for 6 hours now for one single checkout to finish and its not even 30% finished yet. I'm trying to finish fixing the scripts as fast as possible, but there are so many open ends right now that we will probably take another day until the updating of web pages works fine again. Thanks for your patience.

Last night I continued to prepare the KDE CVS conversion to [http://subversion.tigris.org Subversion]. http://websvn.kde.org/ is now up and running and awaiting your testing!

David complained today about not being able to recieve mail from Laurent. I'm not sure why, but I bet one cent that the error is not on kde.org's mail setup. As it turned out after some debugging, I lost the bet. ClamAV decided that the attachment of the mail is encrypted, though it was not, and due to our configuration we block mails with encrypted zip archives as attachment. Investigation showed that this was again the very same bug in ClamAV I fixed over a year ago reintroduced in the CVS snapshot we use, which made it detect any zip archive containing a file with options to detect it as encrypted. How much use is a virus scanner that doesn't even have an automatic regression test for such lame things? So I went on and patched our installation and wrote a very angry mail with patch to the ClamAV developers. It seems the patch was included. Ok, at least something.