Google might know how to write a web browser, but writing an OS certainly isn't their forte.

I've written up a feature proposal on how we can use Btrfs snapshots to enable system rollbacks in Fedora 13, by gluing together the existing kernel code to do Btrfs snapshots, a UI for performing rollbacks, and a yum plugin to make snapshots automatically before each yum transaction. Lots of good comments so far, and LWN has written an article about it (subscribers only, for the moment).

Fedora 12 got released yesterday, with plenty of nice new features.

This is an article introducing a new email reading system called notmuch, written by Carl Worth with comments from me (and a few minor patches).

Abandon Fail Boat

Almost two months ago, when I updated my debian system to the latest and greatest bits, I happened to get a new version of evolution, 2.28. As has become the tradition with new versions of evolution, a few more things broke.

I’ve suffered through evolution ‘upgrades’ several times and had slowly reduced my usage of evolution features to try and keep it working. This time, I got stuck. The accumulated bugs in this mailer made it impossible for me to get my work done any more.

And, yes, it’s a sad commentary on the Linux desktop that the most important feature for many people using Linux has no credible GUI application (yes, I’ve tried a lot of email applications; I have too much mail for them to cope).

Exploring Sup

Carl had given up on Evolution a few weeks before and was using sup. From his description, and from a brief bit of experimentation, I decided to give it a try. Sup has four main features:

1. It is entirely search based. All messages are indexed by a ‘real’ indexing system, xapian which provides reasonable full text search for email.

2. You can mark (automatically, or manually) messages with labels; the ‘inbox’ view just shows the results of a search for messages with the ‘inbox’ label.

3. It never modifies the actual mail store. All state is stored inside the database in the form of labels.

4. Most operations act on threads, not messages. Viewing a thread shows you the unread messages in the whole thread in a single page, making following the conversation easy.

This feature set is exactly what I’ve been trying to get Evolution to use for several years; I used the virtual folders to automatically sort mail into several ‘catagories’. Unfortunately, the evolution vfolder support was terrible to start with (way too slow to be actually useful) and has gotten far worse over time (no more nested vfolders?).

Sup works quite well for a small amount of email. With my message store (dating back to 1984), it took “a while” to do the initial scan of to construct the database. After that, searches are zippy fast.

Sup has a couple of fairly serious mis-features though:

1. It’s written in ruby. Yet another language disaster in my book; syntax horror-show similar to perl, and a lack of static typechecking means that obvious bugs in the program wouldn’t be caught until you happened to execute that particular line of code. Ruby is also no speed demon—I spend a lot of my day reading email, waiting for ruby is not on my list of desired activities.

2. It has a magic curses UI. This is actually pretty good for reading email, but it’s not scriptable at all, which is useful for mass patch-application, and it completely fails when composing new mail as it forks off emacs and waits for it to complete, meaning that you cannot see any mail while composing a message.

3. It saves a bunch of label changes inside the application, and Xapian saves most of the database changes too. Having sup crash often means re-viewing a lot of mail.

Carl and I started fixing sup in various ways; making the mime-viewer run asynchronously (so you could see attachments while viewing the rest of the message), sorting the inbox oldest first and various other changes. Nothing serious, but it did show us how sup was built and just how simple it was inside.

It turns out that sup is just a bit of UI goo over a powerful full-text database; the complicated code is not the UI but the database. Of course, the sup UI is great for viewing mail, but that’s fortunately easy to clone.

A Minimal Mail Reader

Having seen just how easy it was to build a really nice mail reading system, Carl and I sat down and sketched out what the foundations of our ‘ideal’ system would look like:

1. Xapian based. I haven’t seen anything close to Xapian in terms of features or performance. It has only one serious bug—it’s written in C++. Fortunately, we can wrap the C++ mess with a simple C wrapper and ignore that aspect.

2. Command line driven. Any UI would be constructed on top of the command line interface. And, by UI, we mean emacs major mode. If someone wants to write a GUI, we won’t stop them though.

3. Otherwise, work a lot like Sup (thread-based, immutable mail store, user-defined tags).

Carl started by playing with Xapian, using the existing sup database; one possibility would have been to retain compatibility with the sup format and just provide a new interface. Unfortunately, there were a lot of ‘ruby-isms’ in the sup database, and reconstructing that would have been pretty difficult from a non-ruby application.

Introducing Notmuch: Not much of an email program

Notmuch really isn’t much of an email program; it doesn’t talk to mail servers to receive or send mail, it doesn’t even really know what Maildir should look like. All it does is construct a database for all of your mail messages and allow you to search and show email messages.

Notmuch has two pieces—a C program that uses Xapian to search and tag mail messages, and an emacs major mode which provides a fairly simple user interface. Like git, the notmuch C program places a bunch of commands within a single executable:

1. setup
   Interactively setup notmuch for first use.

2. new
   Find and import any new messages.

3. search search-term […]
   Search for threads matching the given search terms.

4. reply search-terms […]
   Formats a reply from a set of existing messages.

5. show search-terms […]
   Shows all messages matching the search terms.

6. tag +tag|-tag […] [—] search-term […]
   Add/remove tags for all messages matching the search terms.

7. dump [filename]
   Create a plain-text dump of the tags for each message.

8. restore filename
   Restore the tags from the given dump file (see ‘dump’).

9. help [command]
   This message, or more detailed help for the named command.

(The above text was taken directly from notmuch itself and was written by Carl).

As you can see, all of the commands which talk about messages take an arbitrary search pattern. The search command outputs thread identifiers in search-term form, so you can easily script things by pulling that out of the search output and passing it to additional notmuch commands. Learning how to do searching in notmuch is the key to using it successfully.

Xapian Search Terms

Matching words anyplace in the message is fairly simple; just list the set of words you want to match. Notmuch also adds some special syntax to direct the match at specific header fields:

• tag:tag
  match messages with the specified tag

• thread:thread-id
  match messages associated with the specified thread

• id:id
  match the message with the given id. Message ids are those set by the message sender in the Message-Id: header field.

• from:word
  match messages with word in the from address field.

• to:word
  match messages with word in either the To: or Cc: headers.

• attachment:word
  match messages with word in an attachment filename.

• subject:word
  match messages with word in the subject field.

Aside from these additions, notmuch uses standard Xapian search syntax, including support for AND, OR etc. Xapian’s query parser is not the most robust piece of code though, so sometimes you need to mess with the query to get it to do what you want.

Notmuch emacs mode

There are a lot of email clients available for emacs; notmuch adds only the email reading part and uses the existing ‘message’ module for composing and sending mail. Even still, notmuch.el is almost 1000 lines long. It offers two different modes — the search display, where a list of email threads are presented, and the thread display, where a single thread is displayed.

The search display presents the output of ‘notmuch search’ in a window, eliding the thread id. When a thread is selected, a thread display buffer is constructed with the thread contents as formatted by ‘notmuch show’.

‘notmuch show’ structures the thread to make the display more useful in emacs; it splits messages into headers and bodies and marks the thread depth of each message. The header of each message will be shrunk to a single line (in reverse video). Previously read portions of the thread will be hidden by default, along with signature lines, quotations and attachments. Each of these can be viewed by use of a suitable command. Carl stole much of this from Sup and adopted it for use inside emacs, along with some of the key bindings.

How well does it work right now?

Frankly, notmuch is pretty rough today; I’m using it to read email, but I’m finding lots of stuff to fix. Fortunately, most of the fixes are pretty simple at this point. The good news is that it’s plenty fast, fast enough that I can count how many threads I’ve exchanged with my good friend Bart in the past 25 years (2686) in only a few seconds.

The biggest performance issue is some lazy code within Xapian. When you want to change the set of tags related to a document in the database (a single mail message), Xapian replaces the entire document. Try removing the ‘inbox’ tag from half a million messages and Xapian will carefully rewrite 5GB of data. That takes a while. The Xapian developers have suggested that this shouldn’t be hard to fix though, at which point re-tagging messages should get a lot faster.

For those interested in playing along, the notmuch sources are available from the notmuch web site along with a pointer to the mailing list.

Someone pointed out to me that I'm featured in two recent Boycott Novell posts. In the first one, I learnt that I'm not a representative of Free/open source software, while the second post shows how Novell still controls GNOME’s direction through me (and this could be a way for Novell to influence GNOME into becoming more dependent on Mono).

Oh, I'm certainly not perfect, but I would think that having contributed to GNOME and free software in general for something like 6 years during my free time before I joined Novell could, you know, count a little bit. Had I stopped contributing in my free time, maybe it would make some sense? Actually, I should do that: I'd have much more time for sleep ;-)

Also, I must admit I didn't know Novell is telling me where GNOME should go; but now I wonder: could I be hypnotised? Or someone could be invading my dreams in some way to influence me! Or wait, I know: there has to be some subliminal messages sent on my screen! That's it, surely! More seriously, how to explain this... I think I'm way more a GNOME person in Novell trying to make Novell change, than a Novell person in GNOME trying to make GNOME change.

But thanks for the good laugh! The conclusion of all this, I guess, is that it means I'm now a pop star! Woohoo :-)

So I (and other radeon developers) debug a lot of radeon problems, both locally and with people over irc/bugzilla, and I often am quite slow to deal with bugs that I can reproduce locally, its usually a last resort to do remote debugging and its unfortunate for people who have hw bugs that we can't reproduce locally.

So what prompted this post?
https://bugzilla.redhat.com/show_bug.cgi?id=527874
KMS:RV515:X1400 Thinkpad T60 resume fails

So first up, my local Thinkpad T60p with an rv530 always resumed fine, my local T42 with 7500 mostly works okay as well, so
there goes my local reproduction.

So Peng works for Red Hat in Beijing and the week before kernel summit I sat down on irc for most of two days with him running various tests. We tracked it down in that 2 days to the fact that his video RAM wasn't getting setup properly on resume. The NMI on resume let me track down that when the gpu accessed the ring, it generated an error on the PCI bus, this led to checking the contents of the PCIE gart table (with a detour through kernel vmap page handling). The PCIE gart table is in VRAM, and upon checking it on resume noticed when we copied it back into VRAM it was getting mangled. So we could deduce VRAM was broken.

I handed off to Jerome and he got traces of the BIOS posting using VBE and using ATOM (which we use in the kernel), Jerome ruled out different parts of the engines and we got reports of it working for some people when powered down for a long time or other randomness, and we were going back and forth with ideas on what might be going wrong, and had started thinking we should power off various parts of the hw before suspend, and the problem was due to inconsistent hw state on resume.

Peng happened to visit Brisbane for a Red Hat meeting this week and brought the T60, and this morning I swapped my laptop for his. First of all I tried to play by plugging in a VGA monitor, this produced another bug where the LVDS would die when starting KMS, so I fixed that quickly first. So the VGA screen was also corrupted, and VRAM wasn't enabled at all. Next I tested vbetool posting worked, also suspend/resume to corrupt, unload radeon, vbetool post and load radeon worked. Then I started testing with Jerome's userspace atom init tool, doing a s/r, unload radeon, atom post, load radeon also worked fine. This is where it started to make no sense, since Jeromes tool was doing the exact same thing as the kernel parser. I started by blaming the atom delay code in the kernel but that proved a dead end after an hour or two. Next thing I enabled the kernel atom debugging and all of a sudden it resumed fine. So it was a timing issue somewhere in atom parser running the init code.

So enabling debugging put enough of a delay between operations that something that wasn't working before now succeeds. I started bisecting the debug messages, I removed half the debugging at a time until after about 3 hrs I got it down to one printk happening between two atombios commands. The surrounding code was reading and writing one of the memory controller setup registers on the GPU, so it pointed to some register write not getting fully into the hw before we read it back and write it again later. I changed the atom code to do a read back before writing regs for certain operations and viola all resumed fine.

So this took the best part of 8 hours, I reckon if I'd been doing the same over irc with Peng it would have taken at least a month of back and forth on irc to figure it out. Having the hardware locally even for a day made it possible to track it down and figure it out so much quicker and efficently. So the bad news for anyone with bugs we can't reproduce locally is that we generally will fix any bugs we can locally first just from a efficiency point of view, since we can fix them so much quicker and faster.

Building on what Havoc wrote two years ago about the fallacies of OOM safety (Out Of Memory) in user code I'd like to point you to this little mail I just posted to jack-devel which tries to give you the bigger picture. Should be interesting for non-audio folks, too.

Say NO to OOM safety!

Many of you are probably familiar with evtest, a small debugging utility written by Vojtek Pavlik to check what input device data is leaving the kernel. I use it a lot and one of the standard requests I make in bugreports is run evtest against the device file. The information it prints tells us the capabilities a device has and the events being sent when a certain action is performed.

I've used this information to add uinput device to my uinput test devices collection to debug various server and/or driver failures.
That worked for some cases but suffered from a major caveat: it was incredibly hard to reproduce issues that resulted from complex interactions. For months, I've been meaning to fix this and last weekend I finally had time to sit down and hack something up. That work is now in the evtest repository.

It works quite simple:
The user runs evtest-capture against the device and performs the action that reproduces the bug. evtest-capture saves the event stream into an xml file, this xml file can then be converted into a standalone uinput-based C program that resembles both the device and the interaction. I can re-create and run that program on my test box and reproduce and hopefully debug the issue.

The full set of steps is (as root where necessary):

$> evtest-capture /dev/input/event0 pointer-crashes.xml
$> xsltproc evtest-create-device.xsl pointer-crashes.xml > pointer-crash-device.c
$> gcc -o pointer-crash-device pointer-crash-device.c
$> ./pointer-crash-device
#verify the issue happens with this test device.

Like so many things, it's not perfect but it will hopefully help us reproduce a lot more bugs and thus make it simpler to find and address these bugs.

This afternoon Andre released telepathy-qt4 0.2.0, the first version that builds a shared library, and I've uploaded it to Debian (it'll get into unstable as soon as it clears the NEW queue).

This is a major milestone for telepathy-qt4: over the course of 16 months' development it's gone from a collection of auto-generated constants (Olli Salli's initial commit, back in July 2008) to a shared library with a frozen API and ABI. We encourage Qt/KDE application developers to treat telepathy-qt4 as the preferred Qt 4 binding for Telepathy.

Andre and the rest of the Telepathy project will continue to add high-level bindings for more Telepathy features over the course of the 0.3.x series; we plan to keep the API and ABI backwards-compatible until shortly before the next milestone, 0.4.0, at which point we'll consider breaking ABI to drop deprecated functionality.

The GNOME release team held a meeting 10 days ago. Yep, it's hardly believable for many people, I guess: you've always thought the release team members are lazy contributors who just pretend they're doing something useful. So, the truth is that this is also a valid statement ;-) But from time to time, we're having meetings (and doing other things that are not completely useless). Even though we have quite some meeting experience now, we're still optimistic about having enough time to discuss every topics. It turns out we managed to discuss everything in only three hours this time. Quite amazing, if you ask me!

One important topic was of course when to release GNOME 3.0. We've said since quite some time that we would decide in November whether GNOME 3.0 would be released in March 2010 or in September 2010. And in November we managed to decide: GNOME 3.0 is planned for September 2010. We started having a good vision of where we are standing thanks to informal feedback, but we've also gathered direct feedback. Interestingly, this request for feedback was interpreted in some news as a clue for a September release date (hrm, don't ask me if I can read the last link ;-)), while it really was just a request for feedback. There are already some articles about the release date announcement.

The other big topic of the release team meeting was of course the new modules that were proposed for inclusion. The decisions were also announced yesterday. Some details of the decisions are, I would think, quite instructive:

• Clutter is currently blocked because of the copyright assignment required by Intel, which is a situation we'd like to avoid. We've approached Intel about this, but this is not something that can be solved in only a few days. This will hopefully get fixed in the future. This lead us to wonder, though: copyright assignment to a company is something we prefer to avoid, but would copyright assignment to the GNOME Foundation or another non-profit organization be okay for our community?

• dconf isn't approved for 2.30, but is pre-approved for 3.0. This shows our commitment to this new technology, but this also illustrates that we can learn at least a bit from our experience :-) We've had some issues integrating technologies a bit too early in the past, and while doing the same error is sometimes required, this wasn't case here. Still, the community is invited to take an in-depth look at dconf and to start branches that will use it so we can make sure it enters GNOME in the best state possible!

• Tracker, which has been a topic for long threads on our mailing list is approved as an external dependency. We certainly hope this will help fix the chicken-and-egg problem where people don't see the benefit of tracker because nothing is using it. Also, it's worth mentioning that the release team really encourages tracker developers to clearly separate the indexer from the store to avoid confusion that has been plaguing the discussion.

Of course, work on 2.30 has already started and there's even a 2.29.1 release already out. The fact that the decision about new modules is out earlier starting with this cycle should help make sure the new modules and new external dependencies are well integrated through our whole desktop.

Also, since I haven't written anything about GNOME 2.28, I guess it's not too late to celebrate this release. I'll admit it's a bit late, but I'll stay with my "not too late". And I have a good excuse: I was lost in Germany with nearly no internet access at the time of the release. As usual, I'm a big fan of the new GNOME release! I'm just a bit sad I couldn't do more, since I was planning to fix a few things and I couldn't make it...

All the GNOME 2.28 goodness will of course be available in openSUSE 11.2 (as well as Fedora 12, Mandriva 2010.0, Ubuntu 9.10 if you want to try other distributions). We've been doing quite some work in the GNOME team for 11.2, and as usual, it's hard to remember everything we changed ;-) One thing I'm quite happy with, though, is that I believe we're better upstream citizen now — it's still not perfect, of course. Oh, and we have some really great theme, thanks to Jakub, with a nice little touch in the form of the font used for window frames! 11.2 will be out next Thursday.

I'll try to write a bit more about recent openSUSE news on Thursday :-) But work has started again in Factory for the next version of openSUSE. Woohoo!

I had to use GNOME today on a random computer. gnome-terminal was beeping all the time. I say no problem, I know how to disable this:

xset b off

It did not work... Hum. I go in the right menu and see a checkbox "Terminal bell". I click on it. All I got is all my terminal windows going away and:

gnome-terminal3557: segfault at 4 ip 0806f417 sp bf9fd000 error 4 in gnome-terminal8048000+3a000

No kidding. You don't want to use X standard ways and prefers to crash at my face. Fuck you.

So... I got a new addition for my graphics card museum a week or so ago. I found this on eBay when I was searching for something else entirely.

I haven't installed it in a system yet, so it may not work. Why not? Because I know it doesn't work at all with the r128 driver in Mesa. I might therefore be tempted to hack on that driver. That would be bad.

Folks! Since quite some time now the kernel exports the DMI machine information below /sys/class/dmi/id/. You may stop now parsing the output of dmidecode thus depending on external tools and privileged code.

For example, to read your BIOS vendor string all you need to do is this:

$ read bv < /sys/class/dmi/id/bios_vendor
$ echo $bv

Which is of course much simpler, and cleaner, and safer than anything involving dmidecode.

Thank you for your time!

Please welcome Gaetan Nadon to the Xorg development team, mentored initially by Peter Hutterer (whot)

If you were to install the new Moblin 2.1 somewhere, you'd be getting a gnome-bluetooth powered Bluetooth panel.

All the code lives upstream in the gnome-bluetooth module on master.

I just pushed out code which adds MPEG2 slice decoding to my graphics driver. It is based on XvMC, but unlike "standard" XvMC implementations, it sends MPEG slices over to the graphics driver over the X protocol.

The base idea is the following: The MPEG engine gets MPEG slices, and outputs to a buffer. This buffer needs to then be displayed by the overlay engine. So, we need to spend most of our time managing the communication and syncing of those engines. We already have the other video engines implemented nicely, so, why not stick the MPEG code next to that and have a nice and clean implementation?

The XvMC protocol, X-wise, is mostly about telling the driver that the MPEG hardware is in use, and subsequently claiming buffers in the framebuffer, managed by the X driver. Everything else is expected to happen in the client library. For what reason, i do not know, but part of it could be that, this way, X is not seen to eat any CPU cycles. In any case, this makes it a very weird protocol, with things spread all over the stack.

Things were made worse with the advent of the XvMC wrapper. Instead of expanding the XvMC protocol slightly to provide the name of the XvMC client library to be loaded, DRI is abused for this purpose. So... a pointless hard dependency on DRI is added, and now, no working DRI means no working XvMC... Curious. Makes the pointless dependency on Shm look harmless.

So what i did now is send all the data over the X protocol, over a tiny X extenstion, so that it could be fed into the hardware and synced inside the X driver. An XvPutImage with a longword buffer containing the mpeg buffer id then makes sure that everything gets displayed correctly. And while the overlay is being set up, the mpeg engine can finish its work, and at the very last minute, the overlay code waits for the mpeg engine to finish, and then the overlay gets told to display the new image.

Other XvMC implementations went and completely reimplemented the overlay in the client library, and even involved 2d acceleration to be able to send mpeg slices to the hw a bit faster. A syncing nightmare. Another advantage is that my implementation can implement the newer mpeg2 engines in just a few hundred highly hardware specific lines.

Of course, sending this data over the X protocol in tiny bits does incur some more cpu cycles, and i also am not feeding mpeg data into the hardware over the command buffers. Because of this, my code uses about 30-35% for a normal DVD (write a comment if you guessed which :)) on a VIA C3 Samuel2 (yes, half speed FPU, not quite PPro compatible) at 600Mhz, while openchrome uses 20-25%, roughly 2/3rds. But the performance of my code is still very good, good enough to not bother with speeding things up just yet.

As usual, it is easy to get this new code. It builds and runs against all Xorg versions that are common, and the debian package build system has also been updated for the xvmc code.

For xine there is one caveat, due to the horrible implementation of video_out_xxmc. We need to fool xine into thinking that we do support subpictures (we don't as it the xxmc way of implementing things didn't even get close to how the hardware implemented it). For that, the following option needs to be set:

Option "XvMCBrokenXine" "True"

in the device section of xorg.conf.

Enjoy!

Today, as some of you guessed from my teaser yesterday, I finished implementing on-disk buffering in Totem, using playbin2's new features.

I have to say, I really liked the idea of FatELF. I'm not terribly surprised at the... er... push-back. We've seen this a lot of times in the past. A number of years ago SGI proposed some patches to use a "spare" 32-bits in a per-thread data structure to hold the GL context pointer. The patches were rejected, and the submitters were flamed. Fast forward a few years (and by I a few I mean 5-ish) we have TLS to store this sort of data in a generic way.

What's the point?

Well, some times someone has the right idea to solve a particular problem. It might not be everyone's problem, and it might not be something we (the "royal" we, of course) want to maintain moving forward. However, it might be something useful and there might be a way to generalize it to make it really useful.

How so?

The first idea that idea had about FatELF was to put multiple versions of a code compiled with optimizations for different targets in the same binary. So, put the -march=i686 and -march=core2 code in the same binary. Yeah, GCC has some features that work sort of like this. Last I tried them, it was a pain in the ass and carried some annoying "caveats".

I would also love a solution to the mixed 32-bit / 64-bit distros. Anyone that thinks /lib and /lib64 (or /lib32 of you're Ubuntu) is a good solution needs a kick in the teeth. Seriously. FatELF isn't a solution "out of the box" for that problem (how do you install a 32-bit library into an existing 64-bit library install) yet, but it doesn't seem like an insurmountable problem.

In the past couple of weeks, we've been hard at work fixing bugs for the next Fedora release, Fedora 12.

One task I been trying quite a few times with Transmageddon is to port it from libglade to gtkbuilder. So far I have always failed for some reason or the other. A big part of it is that I have tons of examples out there for how things are done with libglade, but not so much for gtkbuilder yet.

That said I am also convinced that someone with the right skills could do the port in about 30 minutes or so. Which is the reason for this blog post. Is there anyone out there who would be willing to cook up a patch for me to port Transmageddon to gtkbuilder? (Its written in Python). If so please grab either the latest release or check out git master from GNOME git.

Any help with this would be much appreciated.

Update: Multiple patches received, much appreciated. I will use weekend to try to merge

Thanks to Alan Coopersmith's efforts, X11R7.5 was released a few days ago. Except - what does that mean?

This post is intends to shed some light onto the components of the X11R7.5 release and where the version number comes from.

X Window System

If you're running a desktop system other than Windows or OS X, you're most likely running some instance of the X Window System, also referred to as "X11" or simply "X". X consists of several components that all make up the "X Window System", yet some of them are more visible than others.

X Protocol

The core component of X is the X Protocol. This is what defines X, it is essentially the API.
The X Protocol consists of the core protocol, dating back to the 1980s and a number of protocol extensions, essentially additions to the core protocol. If you hear terms like X Input, XRandR, RENDER, etc., all of these are protocol extensions.

X Server and the drivers

The X Server is the process that talks to the hardware drivers and listens to requests from applications to draw things. It also handles input events and passes them on to the right application. Depending on your hardware, you have a number of drivers. These days many setups have evdev and synaptics for input, and intel, ATI or nvidia for graphics.

The X Server supports the core protocol and most protocol extensions, but different X servers may support different versions. Generally, the most recent X Server supports the latest version of the protocol.

Xlib and friends

Xlib (or libX11) is the library that allows applications to talk X Protocol to the server. It wraps the low-level protocol into a slightly higher-level API. These days, most applications that display a GUI use Xlib at some point - though Xlib is usually abstracted away by a saner toolkit such as GTK or Qt.

Xlib has been the single toolkit to talk X Protocol for ages, but in recent years XCB is gaining some traction (and in fact recent versions of Xlib use XCB at the lowest level).

X applications

A number of applications are traditionally part of the X Window System. One of the well known ones is xeyes, but other, crucial tools such as setxkbmap and xkbcomp are part of these applications as well.

Misc other stuff

There are a number of other packages that include fonts, misc utils, data packages etc. I'll skip the details, it's just important to know they're there.

X11R7.what?

Back a few years ago, all the above component were part of one repository. To build one of the components, you also had to build the others. To release one, you'd have to release the whole lot. Over time, the version numbers crept up to 6.9 for this so-called monolithic tree.

X11R6.9 (X11 Release 6.9) was the last monolithic release. Around 2005, the monolithic tree was split up into separate repositories for each component. This also reset the version numbers for most of the components - those that inherited the 6.9 version numbers (or even 7.0) were reset to 1.0.

Since then, the X11R7.x releases (referred to as "katamari") are quite like distributions. They cherry-pick a bunch of module versions known to work together and combine them into one set. The modules themselves move mostly independent of the katamaris and thus their version numbers may skip between katamaris. For example, X11R7.4 had the X Server 1.5, X11R7.5 has X Server 1.7.

This is where much confusion comes from. Many users don't know whether they're running 1.7, 7.5, 1.0 or 6.8. The intent of a katamari is simply to provide a set of modules that are sufficient to get a basic GUI running. That's why over time modules get added or removed from the katamari as well. A module that was part of X11R7.5 may not be part of X11R7.6 and of course the other way round (a full list of which versions are included is at the top of the X11R7.5 Changelog).

Which version actually matters?

Katamaris matter mostly for distributors. They represent a set of versions known working together and make for easy picking. A distribution is free to start out with a katamari and then update to newer modules as they are released. The katamari is merely a starting point, not more.

For this reason, it rarely matters to an individual user whether a module they're running is part of a katamari. For bug reporting, developers need to know the versions of the individual modules affected so they can narrow down which bug may be triggered.

To get the versions for the X Server and the drivers, look at the /var/log/Xorg.0.log. The first line states the version of the X server. Drivers are loaded dynamically, so you need to search for them in the log. For example, my log says:

X.Org X Server 1.7.0

[...]

(II) Module intel: vendor="X.Org Foundation"
       compiled for 1.6.99.903, module version = 2.9.0
       Module class: X.Org Video Driver
       ABI class: X.Org Video Driver, version 6.0

[...]

(II) Module evdev: vendor="X.Org Foundation"
       compiled for 1.7.0, module version = 2.3.0
       Module class: X.Org XInput Driver
       ABI class: X.Org XInput driver, version 7.0

[...]

(II) Loading /usr/lib/xorg/modules/input/synaptics_drv.so
(II) Module synaptics: vendor="X.Org Foundation"
       compiled for 1.6.99.900, module version = 1.1.99
       Module class: X.Org XInput Driver
       ABI class: X.Org XInput driver, version 7.0

So right now I'm running X Server 1.7.0, with evdev 2.3.0, intel 2.9.0 and synaptics 1.1.99. Whether these versions are part of a katamari doesn't matter.

X apps usually have some -version switch. For libraries, it's best to use the distribution's packaging system (e.g. rpm -q libX11) to get the version number.

Edward pointed my to this blog today which brought up a point I myself have been making in regards to Android. I spoke to several people at the CE Linux meeting a couple of weeks ago about this for one. To quote from the blog:

Android is an island of its own, and useful code sharing is largely limited to the kernel.

At Collabora Multimedia we are currently working with both Maemo and Android systems and while I can see the appeal of Android from a phone makers perspective I can’t help but be a little saddened by how worthless it is to the general linux eco-system. One of the things I always loved about Nokia’s Maemo effort is that since its using so many of the standard components that we use on the Linux Desktop, it means that when a feature is added or a bug is fixed in Maemo, it directly helps also the linux desktop. Nokia and Maemo has had a strong and direct impact on a lot of open source projects, ranging from GStreamer, D-bus, GTK+, Telepathy, Matchbox, X Window System and more. And Nokia’s work on Qt going forward will of course have a direct impact on the quality of KDE.

Android on the other side has a much more marginal impact. I know they have contributed some patches to Webkit, but apart from that they offer little value to the rest of the linux eco-system. Been even told by some kernel developers that an Android kernel driver is about as immediately useful for the mainstream kernel as a FreeBSD or OpenSolaris driver. Meaning that porting is needed.

So for me personally I can’t help but feel a lot more positive about Maemo (or Moblin for that matter as they too share the same kind of philosophy as Maemo) and getting a N900 is definitely on my TODO list. That said Android is a work in progress and hopefully we can get them to abandon their essentially proprietary stack going forward and instead incorporate more and more shared libraries with the server and desktop. Maemo has proved that for a smartphone these libraries works just as well as Googles homebrew. Some of the efforts we are involved with are pushing in that direction and hopefully Google will realize that the secret to the success of open source is synergy.

Boston was good to me last week — I got to see two talks from two particularly inspiring and heroic people, a day apart. (And a Rodrigo y Gabriela gig on Friday, which perhaps wasn't as heroic, but was also awesome: check out Tamacun, Orion and Captain Casanova on Youtube.)

Now, on to the talks:

William Kamkwamba

When he was 14, William Kamkwamba built a working windmill at his house in Malawi, despite having dropped out of high school a few years earlier because his parents weren't able to afford to send him anymore. He knew what to build by looking at pictures of a windmill in a science textbook in a library, using a dictionary to translate the words that referred to the pictures from English to his native language of Chichewa, and believing that the presence of the photo meant that someone must have built one before, therefore it must be possible for him to do it too. He also had some experience with repairing radios, taking them apart and working out what each component was doing by trial and error. His story is so inspiring because he lacked enough schooling in English and Science to be expected to gain the knowledge of electromagnetics he picked up, lacked any money to buy parts to work with, but somehow achieved his goal anyway. He gave a humorous and fascinating talk at MIT last week with Bryan Mealer, the co-author of the book that tells his story: The Boy Who Harnessed The Wind.

Reading the book after the talk totally changed my understanding of what he'd done and why, though; the background for his wanting to build a windmill is not always mentioned in articles and interviews with him. The season before he started, there had been a famine throughout Malawi — there was a drought, and corrupt government officials had sold off the country's strategic grain reserves and kept the money, meaning that the government did nothing to help feed millions of subsistence farmers (including William's family) who were left with a small fraction of the amount of food they needed for that season. William's family lost a lot of weight, sold their possessions and dropped out of school to pay for food, and watched many of their friends and other villagers waste away and die from hunger over a period of months. The book contains detailed descriptions of what it's like to live and go to bed hungry, after maybe a few mouthfuls of food all day, that make me deeply ashamed that we allow this to happen to anyone in the world.

Given all this context, it becomes totally obvious what William was doing the next year at the library: the textbook said that windmills could be used to power water pumps, which would mean freedom for his village from having to go through another drought and famine. The surprising conclusion you're left with is: "Of course he built a windmill, teaching himself a massive amount of electronics that was described in a language he barely understood in order to do so — what else was he supposed to do?".

Peter Singer

The second talk was from Peter Singer, who's an applied ethicist at Princeton, and writes about modern ethical questions from a utilitarian perspective. He wrote Animal Liberation thirty years ago, which is thought of as having founded the animal rights movement; his work persuaded me to start approaching vegetarianism, then become vegetarian nearly three years ago, and mostly-vegan earlier this year (vegan at home, vegetarian when eating out with friends or if it's difficult to find vegan food). Lately he's been writing about poverty and the nature of our responsibility to people suffering due to poverty in countries other than our own, and has a powerful argument that we aren't doing nearly enough. I first encountered his anti-poverty work with What Should a Billionaire Give — and What Should You? in the New York Times, and he's since written a book on the subject, The Life You Can Save. The book is excellent — as well as describing the moral basis for aid, he handles common objections to charitable giving, including what responsibility we have when others aren't accepting their share of it, where it's okay for us to stop and feel like we've done enough, why we shouldn't be giving money locally instead, and how we can find efficient and life-changing charities to donate to. Here's the book's opening and most provocative question, first proposed in his 1972 paper Famine, Affluence, and Morality:

"On your way to work, you pass a small pond. On hot days, children sometimes play in the pond, which is only about knee-deep. The weather's cool today, though, and the hour is early, so you are surprised to see a child splashing about in the pond. As you get closer, you see that it is a very young child, just a toddler, who is flailing about, unable to stay upright or walk out of the pond. You look for the parents or babysitter, but there is no one else around. The child is unable to keep his head above the water for more than a few seconds at a time. If you don't wade in and pull him out, he seems likely to drown. Wading in is easy and safe, but you will ruin the new shoes you bought only a few days ago, and get your suit wet and muddy. By the time you hand the child over to someone responsible for him, and change your clothes, you'll be late for work. What should you do?"

Of course, we all answer that the minor inconvenience of having to buy a new pair of shoes and change our clothes are not valid excuses for refusing to save a life, and the follow-up hits you like a ton of bricks: there actually are people dying from poverty every day, around 30,000 of them, and we really can save a life for not much more than the cost of a good pair of shoes. Furthermore, more than two and a half billion people live on the equivalent of less than USD $2 per day. How, then, can we say that we're different from the person who walks by the lake, sees the child drowning, realizes they could save them, and does nothing?

Singer's book explores the differences between the two situations — primarily that in one a child is next to you while in another they are far away — and concludes that this cannot be a sufficiently different situation to present a different moral answer, if we claim to hold ethical beliefs like "the value of a human life is the same no matter where in the world it is" and "a human life is worth more than a pair of shoes".

Now, I didn't mean for this post to make you feel guilty — in fact, I'm feeling very optimistic about this issue. Singer observes at the beginning of the book that the struggle to reduce suffering due to poverty has historically been a sort of climb towards an unreachable, unknowably distant mountain peak; but now we have cleared the clouds and can see the summit, our ability to do this is clearly within our means. I want other people to enjoy life the way I did at the gig on Friday night. While I can't give everyone tickets to go to concerts, I can certainly work towards them having enough food so that the William Kamkwambas of the world can, rather than trying to fall asleep in darkness and hunger, enjoy a full stomach and some good music with a radio they've managed to repair.

We're living interesting times, the web is gaining momentum, the explosion of the smartphones and the mobile market is changing the ways people think about computing, the model to deliver applications and services to the consumer and his role in the usage has radically changed.

The way people create, build and deploy applications has radically changed, 10 years ago you couldn't even think about deploying a large application without packaging it in a box with a manual and spend large amounts of money to deliver it to the shop shelves, you couldn't even think about people actually buying it without some sort of advertising in the hi tech magazines.

Today, all that it takes to check what's new and exciting is a browser, even if the application you want to try is locally installed. In this regard, the traditional closed desktop provides a pretty well understood deployment model for ISVs willing to create and deliver an application, a .msi installer on windows, a .dmg image for Mac OS X where you just drag and drop an icon to your app folder.

Platforms like the iPhone and Android makes application creation, packaging and distribution a well documented straightforward process, except for one thing, their app stores are censored so we have an advantage here that we are not exploiting.

The Unix world in general, and the Linux world in particular is a little special in this regard. By nature, we have a diversity of tools to manage and deploy applications and libraries in our system (yum, apt, ips, ports...), tools like PackageKit can overcome some of the problems of such diversity, although it is getting mainstream slower that we would like to. In general users should be alright with this as things stand today.

However, it is not the users the ones I'm worried about here, it is the CS students and enthusiast wanting to do small fancy apps, it is the small companies with no resources to employ a team of package maintainers to create and maintain a dozen versions of packaging scripts. The very people with the talent to create new exciting apps that can attract and engage users, the very people that can create an ecosystem where creating and distributing large apps for Linux is not a path full of pain. Applications like Photoshop, Autocad, SPSS are not going to get open sourced anytime soon, some of them may never be, the question is, can we attract them to make the free desktop a more appealing option for users? For some people and institutions those apps are the one and only reason to stick to Windows or Mac, so this problem is worth considering.

Yet, it doesn't look we are getting close to support application distribution models klik in the upstream desktop environments where you could download a file and run your app straight away. This is what the developers want. This is what the users want. We are just not listening them here.

Creating apps for our stack is a pain in the ass, the good practices are fairly undocumented, essential resources are fragmented within several web sites, with different APIs models, there's a lack of consistency and ease of use. Yes, I know, this is open source, this is the way it's been so far, you can't kill diversity.

But... I think we are in a point in time where it is critical to assure our success, or the spreading of freedom we envisioned might be threatened by our competition and we will lose large amounts of control over our technologies as a community once again. Most importantly, I do not only think these radical changes are necessary, I think that we can actually make them happen with no much effort, we just in need for some focus and learn from what others have done to make their platforms attractive. We can bring our stack back to the peak of innovation and leave behind our early 90's development and distribution style once and for all.

We already have a compelling desktop people want to use, it's been a hell of an effort to get where we are, things like the new Shell and Zeitgeist can give us some hype on OSNews and Slashdot, but at the end of the day, innovation happens elsewhere, so we better focus on empowering all that creative and passionate people out there who wants to create apps that their family and friends can use.

Back from Japan at last (I think United lost my sleep schedule on the way home though, trying to retrieve it this weekend has been a challenge).

Both KS and JLS went well I thought. It was really good to connect with some of the Japanese developers that until now I’ve only interacted with through email.

The summit went well this year I thought. We didn’t have a big set of controversial issues to discuss, but we did sort out some development process issues. The highlight for me was the two customer panels. On the first day we had some people from TV and other vendors talk about how they’re using the kernel and other open source software. It’s interesting that some of them are stuck way back on 2.4 and very early 2.6 kernels. Part of the reason is long product development cycles, but mostly it’s because the SoCs used in many products only have support in a limited set of kernels (usually custom patches for specific kernels provided by companies like Montavista). The “platformization” work done by tglx and the x86 team recently (partly motivated by Intel “Moorestown” support, but also in preparation for more x86 based SoCs in the future) should help with this for x86 stuff. We definitely want to avoid an ARM-like situation where each SoC requires a specific kernel with incompatible firmware and hardware support. I had some good discussions with Linus and Paul on that topic; the tricky part will be ensuring that vendors adhere to some level of standardization in their platform and firmware support. Doing so will have big benefits: upstream kernel support should be better and much more flexible (good for the SoC vendors and their customers), and the platform maintainers should have a much easier job integrating support for new platforms without a huge set of ifdefs and incompatible firmware interfaces. Managed to get a few bugs fixed at KS as well, Ted & Dirk didn’t have anywhere to run when I wanted them to test some patches for problems they’d reported!

The JLS conference was interesting too, with a few good talks on things like barcode delivery of oops info and btrfs

Tokyo is a pretty amazing city. This was my first trip to Japan and a few of us were fortunate enough to have Paul Mundt guide us for a couple of evenings to explore the city. The narrow alleyways and tiny bars in the Shinjuku (at least I think that’s where we ended up) were really fun. We even checked out a Mexican bar called Bonita; Mexican stuff outside the southwest US and Mexico is always interesting, but the Japanese mix made things even more so. Overall a fun night including Japanese Denny’s food, passed out salarymen, and an everything store with some bizarre costumes, including some furry outfits we were tempted to buy… A bit later in the week we had a contrasting experience by going to Seamon (one of the dozens of one star Michelin sushi restaurants in Tokyo) and a high end scotch and cigar bar afterwards.

Ok now back to catching up on the huge backlog of patches that have accrued due to travel neglect.

We have recently added 3 new members to our growing Multimedia team and GStreamer consulting business. The first one onboard was Thiago Sousa Santos who I think many of you probably already know as he has been a regular GStreamer contributor for the last few years. He also wrote some important plugins for GStreamer as part of the last two Google Summer of Code projects, namely the Quicktime/MP4/3GPP muxer for GStreamer and this year the ASF muxer and ASF RTP payloader. Having been so impressed with his work as part of the community over the last few years we made sure to snatch him up as soon as he graduated from University

The second person we added to our team was Robert Swain. He might not be familiar to people following GStreamer or GNOME, but he has been an active contributor to the ffmpeg project, working for instance on improving the AAC support in ffmpeg. A lot of the work we do at Collabora Multimedia is of course low level multimedia handling and optimisations and Robert will strengthen our capabilities in that field. Also with his experience with ffmpeg we can hopefully use his knowledge to improve the GStreamer ffmpeg plugin where possible.

And finally we have Arun Raghavan, who will be joining us next Month. Arun comes to us recommended by Pulse Audio maintainer Lennart Poettering and will be part of our effort to officially support the Pulse Audio sound server as part of our portfolio of open source projects we offer expertise and consulting services around. Wim Taymans have been moonlighting a bit as a pulse audio developer over the last year, but with Arun on the team we now have a person dedicated to Pulse Audio development, making sure Pulse Audio works great for our customers on their embedded systems. We also hope his efforts will pay dividends for Pulse Audio users on the desktop too in terms of more features and better stability. The synergy we are able to create between the embedded world and the desktop is part of our core mission here at Collabora and with Arun on the team we hope to continue and deepen the great working relationship we have established with Lennart. As a sidenote Arun comes to us from NVidia so maybe we can even have him help improve the GStreamer vdpau plugins

Speaking of synergies between embedded and desktop work, I hope everyone read Guillaume Desmottes blog post about Collabora’s increased effort behind the Empathy chat,VoIP and video conferencing client

Please help me !

How can i add more than four keyboard layouts on my Ubuntu 9.04 ? Is it fixable in this version of Ubuntu or i just should wait for next version (9.10 , where i hope this bug is fixed?)



Thanx !

One thing that came to my mind after the summit was that after John McCann's presentations on various GNOME Shell topics, people went from the so called 'gnomeshellskepticism' to actually start getting it, and by the end of the event most people were using it by default.

A lot is yet to be done to get it to a final state and they seem to have a real vision for it. But it was still bugging me the fact that an explanation of the philosophy and showing some of the features in action made such a difference.

After realizing this I somewhat started to considering that a first login introduction in form of an assistant or a video could make a huge different on terms of success of adoption of the new shell, this approach works pretty well for games, where usually the first time you play it there's some sort of introduction showing you the basic controls and the interface.

Just got a bug report (SIGPIPE when playing with nmap), so I released a 11th version of sysrqd.

<rant>

So in the past Ubuntu packaged PA in a way that, let's say, was not exactly optimal. I thought they'd gotten around fixing things since then. Turns out they didn't. Seems in their upcoming release they again did some genius thing to make PA on Ubuntu perform worse than it could. The Ubuntu kernel contains all kind of closed-source and other crap to no limits, but backporting a tiny patch that is blessed and merged upstream and in Fedora for ages, that they won't do. Gah.

And it doesn't stop there. This patch is an outright insult. This is disappointing.

Madness. Not good, Ubuntu, really not good! And I'll get all the complaints for this f**up again. Thanks!

/me is disappointed. Ubuntu, you really can do better than this.

</rant>

Kinda fun watching this video. As it seems the big new features of the Windows 7 audio stack are the ability to move streams while they are live, to do role-based policy routing, and to pause streams during phone calls. Hah! That's so yesterday! A certain sound server I happen to know very well has been supporting this for a longer time already, and you can even buy that logic in various consumer products.

Nice to know that in some areas of the audio stack it's not us who need to play catch-up with them, but they are the ones who need to play catch-up with us.

Its been an interesting week here Grenoble, been talking with a lot of people about linux on consumer electronics in general, but also of course about the GStreamer consulting we offer at Collabora Multimedia. It is also always encouraging to see the number of people at an event like this who already have heard about Collabora, be it in conjunction with GStreamer or Telepthay or Webkit or any of the other projects we either have the lead on or are contributing heavily or been told about us by an existing customer.

We ended up having a very nice conference dinner yesterday evening at one of the restaurants on top of the mountain travelling there by cable car.

Getting ready to start my journey back home now, and while I have to say Grenoble has made a very positive impression on me, I am looking forward to getting home to Cambridge.

I am currently in the town of Grenoble in France, attending the CE Linux conference. Or rather the official conference starts tomorrow, so today I am attending a workshop hosted by ST Ericsson talking about their open source effort around the Nomadik platform, more specfically the NHK-15 platform. Looks like a very interesting piece of kit and I also got a nice development board to take home. Met a few known faces already here, for instance Dave Neary is also attending the workshop today, but I am sure there will be more people when the official conference kicks of tomorrow.

Anyway, if anyone else are attending CE Linux and want to talk about Collabora, GStreamer, Telepathy, PulseAudio and so on, be sure to look me up.

I also noticed that I tend to try to speak Spanish to everyone here. Not sure why, but I guess my mind on some level assume that they might have a better chance to guess what I mean if I speak Spanish and they only speak French. Or maybe its because my new housemate, Abigail, is Spanish, so due to speaking with her my mind is now tuned to jump to trying to use Spanish words

The first Telepathy session session on Saturday evening at the Boston GNOME Summit was very much of a Q&A where myself and Will answered various technical and roadmap issues from a handful of developers and downstream distributors. It showed me that there’s a fair amount of roadmap information we should do better at communicating outside of the Telepathy project, so in the hope its useful to others, read on…

MC5

Ted Gould was wondering why Mission Control 4 had an API for getting/setting presence for all of your accounts whereas MC5 does not. MC5 has a per-account management of desired/current presence which is more powerful, but loses the convenience of setting presence in one place. Strictly speaking, doing things like deciding which presence to fall back on (a common example being if you have asked for invisible but the connection doesn’t support it) is a UI-level policy which MC should not take care of, but in practice there aren’t many different policies which make sense, and the key thing is that MC should tell the presence UI when the desired presence isn’t available so it could make a per-account choice if that was preferable. As a related side point, Telepathy should implement more of the invisibility mechanisms in XMPP so it’s more reliably available, and then we could more meaningfully tell users which presence values were available before connecting, allowing signing on as invisible.

Since MC5 gained support for gnome-keyring, its not possible to initialise Empathy’s account manager object without MC5 prompting the user for their keyring password un-necessarily (especially if the client is Ubuntu’s session presence applet and the user isn’t using Empathy in the current session but has some accounts configured). Currently the accounts D-Bus API requires that all properties including the password are presented to the client to edit. A short-term fix might be to tweak the spec so that accounts don’t have to provide their password property unless it’s explicitly queried, but this might break the ABI of tp-glib. Ultimately, passwords being stored and passed around in this way should go away when we write an authentication interface which will pass authentication challenges up to the Telepathy client to deal with, enabling a unified interface for OAuth/Google/etc web token, Kerberos or SIP intermediate proxy authentication, and answering password requests from the keyring lazily or user on demand.

Stability & Security

Jonathan Blandford was concerned about the churn level of Telepathy, from the perspective of distributions with long-term support commitments, and how well compatibility will be maintained. Generally the D-Bus API and the tp-glib library APIs are maintained to the GNOME standards of making only additive changes and leaving all existing methods/signals working even if they are deprecated and superseded by newer interfaces. A lot of new interfaces have been added over the past year or so, many of which replace existing functionality with a more flexible or more efficient interface. However, over the next 4-6 months we hope to finalise the new world interfaces (such as multi-person media calls, roster, authentication, certificate verification, more accurate offline protocol information, chat room property/role management), and make a D-Bus API break to remove the duplicated cruft. Telepathy-glib would undergo an ABI revision in this case to also remove those symbols, possibly synchronised with a move from dbus-glib to GVariant/etc, but in many cases clients which only use modern interfaces and client helper classes should not need much more than a rebuild.

Relatedly there was a query about the security of Telepathy, and how much it had been through the mill on security issues compared to Pidgin. In the case of closed IM protocols (except MSN where we have our own implementation) then we re-use the code from libpurple, so the same risks apply, although the architecture of Telepathy means its possible to subject the backend processes to more stringent lockdowns using SElinux or other security isolation such as UIDs, limiting the impact of compromises. Other network code in Telepathy is based on more widely-used libraries with a less chequered security history thus far.

OTR

The next topic was about support for OTR in Telepathy. Architecturally, it’s hard for us to support the same kind of message-mangling plugins as Pidgin allows because there is no one point in Telepathy that messages go through. There are multiple backends depending on the protocol, multiple UIs can be involved in saving (eg a headless logger) or displaying messages (consider GNOME Shell integration which previews messages before passing conversations on to Empathy), and the only other centralised component (Mission Control 5) does not act as an intermediary for messages. Historically, we’ve always claimed OTR to be less appealing than native protocol-level end-to-end encryption support, such as the proposals for Jingle + peer to peer XMPP + TLS which are favoured by the XMPP community, mostly because if people can switch to an unofficial 3rd party client to get encryption, they could switch to a decent protocol too, and because protocol-level support can encrypt other traffic like SRTP call set-up, presence, etc.

However, there is an existing deployed OTR user base, including the likes of Adium users on the Mac, who might often end up using end to end encryption without being aware of it, who we would be doing a disservice by Telepathy not supporting OTR conversations with these people. This is a compelling argument which was also made to me by representatives from the EFF, and the only one to date which actually held some merit with me compared to just implementing XMPP E2E encryption. Later in the summit we went on to discuss how we might achieve it in Telepathy, and how our planned work towards XMPP encryption could also help.

Tubes

We also had a bit of discussion about Tubes, such as how the handlers are invoked. Since the introduction of MC5, clients can register interest in certain channel types (tubes or any other) by implementing the client interface and making filters for the channels they are interested in. MC5 will first invoke all matching observers for all channels (incoming and outgoing) until all of them have responded or timed out (eg to let a logger daemon hook up signal callbacks before channel handling proceeds), all matching approvers for incoming channels until one of them replies (eg to notify the user of the new channel before launching the full UI), and then sending it to the handler with the most specific filter (eg Tomboy could register for file transfers with the right MIME type and receive those in favour to Empathy whose filter has no type on it). Tubes can be shared with chat rooms, either as a stream tube where one member shares a socket for others to connect to (allowing re-sharing an existing service implementation), or a D-Bus tube where every member’s application is one endpoint on a simulated D-Bus bus, and Telepathy provides a mapping between the D-Bus names and the members of the room.

In terms of Tube applications, now we’ve got working A/V calling in Empathy, as well as desktop sharing, and an R&D project on multi-user calls, our next priority is on performance and Tube-enabling some more apps such as collaborative editing (Gobby, AbiWord, GEdit, Tomboy…?). There was a question about whether Tube handlers can be installed on demand when one of your contacts initiates that application with you. It’d be possible to simulate this by finding out (eg from the archive) which handlers are available, and dynamically registering a handler for all of those channel types, so that MC5 will advertise those capabilities, but also register as an approver. When an incoming channel actually arrives at the approval stage, prompt the user to install the required application and then tell MC5 to invoke it as the handler.

Colin Walters asked about how Telepathy did NAT traversal. Currently, Telepathy makes use of libnice to do ICE (like STUN between every possible pair of addresses both parties have, works in over 90% of cases) for the UDP packets involved in calls signalled over XMPP, either the Google Talk variant which can benefit from Google’s relay servers if one or other party has a Google account, so is more reliable, or the latest IETF draft which can theoretically use TURN relays but its not really hooked up in Telepathy and few people have access to them. XMPP file transfers and one-to-one tube connections use TCP which is great if you have IPv6, but otherwise impossible to NAT traverse reliably, so often ends up using strictly rate-limited “SOCKS5″-ish XMPP proxies, or worse, in-band base64 in the XML stream. We hope to incorporate (and standardise in XMPP) a reliability layer which will allow us to use Jingle and ICE-UDP for file transfers and tubes too, allowing peer to peer connections and higher-bandwidth relays to enhance throughput significantly.

Future

Ted Gould had some good questions about the future of Telepathy…

Should Empathy just disappear on the desktop as things like presence applets or GNOME Shell take over parts of its function? Maybe, yes. In some ways its goal is just to bring Telepathy to end users and the desktop so that its worth other things integrating into Telepathy, but Telepathy allows us to do a lot better than a conventional IM client. Maemo and Sugar on the OLPC use Telepathy but totally integrates it into the device experience rather than having any single distinct IM client, and although Moblin uses Empathy currently it has its own presence chooser and people panel, and may go on to replace other parts of the user experience too. GNOME Shell looks set to move in this direction too and use Telepathy to integrate communications with the desktop workflow.

Should Telepathy take care of talking to social networking sites such as Facebook, Twitter, etc? There’s no hard and fast rule – Telepathy only makes sense for real-time communications, so it’s good for exposing and integrating the Facebook chat, but pretty lame for dealing with wall posts, event invitations and the like. Similarly on the N900, Telepathy is used for the parts of the cellular stack that overlap with real-time communications like calling and SMS, but there is no sense pushing unrelated stuff like configuration messages through it. For Twitter, the main question is whether you actually want tweets to appear in the same UI, logging and notification framework as other messages. Probably not anything but the 1-to-1 tweets, meaning something like Moblin’s Mojito probably makes more sense for that. Later in the summit I took a look at Google Latitude APIs, which seem like something which Telepathy can expose via its contact location interface, but probably not usefully until we have support for metacontacts in the desktop.

Can/will Telepathy support IAX2? It can, although we’d have to do a local demultiplexer for the RTP streams involved in separate calls. It’s not been a priority of ours so far, but we can help people get started (or Collabora can chat to people who have a commercial need for it). Similarly nobody has looked at implementing iChat-compatible calling because our primary interest lies with open protocols, but if people were interested we could give pointers – its probably just SIP and RTP after you dig through a layer of obfuscation or two.

If you want to know more about Telepathy feel free to comment with some follow-up questions, talk to us in #telepathy on Freenode, or post to the mailing list.

I spent this weekend in Boston for the annual GNOME summit. I really enjoyed it this year, although there were fewer attendees than previously it felt very focussed and productive. There’s some cool stuff going on, and it’s always great to catch up with all of the usual free software suspects in Boston. Some highlights from the weekend:

• Corridor session with David Zeuthen, Ryan Lortie, Matthias Clasen and later joined by me and Will Thompson from Collabora, discussed a lot of the issues of integrating D-Bus into Glib and I think achieved a pretty good consensus about how GVariant and GDBus should fit together and start getting the pieces merged. Really looking forwards to it, GVariant looks mad ninja.
• Some nice discussions about Gtk+ 3.0 roadmap, although I’m still worried that the sealing/accessor work will take so much developer time there won’t be that much time to make the improvements its supposed to enable. Theming was mentioned but what else should there be? Also spotted Kristian Høgsberg, Cody Russell and Matthias again talking about client-side decorations, presumably Wayland scheming… :)
• Nice session about messaging and other notifications in GNOME Shell. I really like the way the project is going, and hopefully we can join in and spend some time hooking Telepathy up here, although as a fallback to make the existing stuff (which doesn’t have specific code to hook in and make a UI), then it’d be nice if the shell could also be a frontend for the notify/indicate stuff which Ubuntu have been pushing in GNOME 2.x already.
• Had some good brainstorming about Telepathy integration in games and Tomboy sharing too, as well as some slightly less conclusive pondering about how to deliver metacontacts (ie, merging multiple sources such as IM, social networking and other address books) in GNOME. We need to sync up with what Moblin is doing here as it looks quite promising and should give us some components to re-use.
• Will, Sjoerd and I spent an hour or two around a blackboard working out how XMPP end-to-end TLS encryption might be exposed to clients in Telepathy, in order to work out how best we’d expose OTR too. It looks like we have a fairly workable proposal now which we’ll be explaining in due course, but it means at least we can give more useful advice to people who are interested in implementing it, or move forwards on implementing it ourselves.

I was really impressed by Jason Clinton and others’ summaries of the sessions, which I think are really valuable for the people who couldn’t make it to the summit. He asked me to take some notes about the first Telepathy session on Saturday evening while he was taking notes about the Outreach session. Rather than lumber him with my deranged scratchings from Tomboy, I’ll blog them separately.

Outsiders of the group of developers regularly complain that it's very hard to get an understanding of what happens in development of the Radeon driver(s). They have convinced me.

So I've spent some time to refactor the documentation on the DRI Wiki, creating a new, cleanup up and up-to-date portal page in the process. However, this refactoring is not complete, and the amount of information is not yet entirely satisfying. The real test now is whether this can be more than a one-shot effort by me, so this is a shout out to others - particularly to the users who have been asking for this - to help improve the documentation.

Partly, this can be done in very mundane ways by normal users. The application support matrix needs to be updated for Mesa 7.6 (there have been a lot of significant improvements), which mostly requires testing by a lot of people who own the necessary hardware/software combinations.

I have also created a stub page for 3D troubleshooting and known problems, and I want to rely mostly on users to help filling that with useful information. After all, the kind of trouble I run into usually has to do with me breaking the driver during development, which is on a rather different level from the kind of trouble users run into.

Now my main goal is actually to improve the way we present ourselves to potential new developers. This is also quite tricky because after having been immersed in this stuff for so long, it is hard for me to judge what kind of information and documentation would be most helpful to newcomers. If you have a concrete question, or set of questions, I'd be curious to know, and I may end up blogging on it, or documenting it on the Wiki, or documenting it in the source code. However, please give me more than a muddy "this stuff is complicated, can you hold my hand through it?", because yes, unfortunately, this stuff is complicated.

In my experience, the first step to understanding the 3D driver is to understand OpenGL. After all, the purpose of the 3D driver is ultimately to implement an API - and you just won't understand the driver if you don't understand that API. So if you're curious about 3D driver development, hack on little OpenGL samples first, or even at the same time! It doesn't have to be a large game engine; the samples that come with Mesa may be more than enough as a starting point. Then maybe you want to step through your program and what the driver does in a debugger, or experiment with other little ways that can help you understand the driver.

I hope that we will be able to welcome you into the fold one day.

I just posted two OpenGL tutorials. This is the start of what I hope will be a long series of tutorials. Right now there's just a "Hello, world!" tutorial for the post-OpenGL 3.1 world (i.e., no immediate mode, no fixed-function) and a more in-depth tutorial for vertex shader inputs.

Since I want to produce PDF and HTML output, I'm authoring them in DocBook (or is it SGML? WTF?). It's mostly working out, but there are some things that I just can't seem to get to work right. The most frustrating part is that documentation for all of the SGML toolchains that I can find are crap. It's kind of ironic that the documentation for a document creation tool is so lacking...

My two biggest peeves are the crap figure handling in the PDF output and the crap table formatting in the HTML output. I can fix the HTML problems with CSS, but how is the tool makeing the output so bad in the first place? For the PDFs, seriously, a page break in the middle of a figure? Whose genius plan is that?!?

The other problem that I initial had was the lack of XInclude support in Jade. I "fixed" that using xmllint, but this seems like a fundamental feature to be missing from the tool.

Before anyone comments, I did consider using LaTeX. The main problem I have with LaTeX is the HTML output that it generates make me want to stab somebody. Every single time I come across an academics webpage that has been generated from LaTeX, I shed a lonely tear...

As ajax quite elegantly summed up, due to a series of catastrophic power failures at PSU, where fd.o is hosted, we were down for a good chunk of yesterday. Despite the machines having redundant power supplies, being connected to separate power rails in the rack, which were hooked up to independent, UPS-backed, power supplies, we still (like a good chunk of Portland, and certainly everyone in the PSU machine room) lost our power.

As far as we can tell, when annarchy.fd.o (websites, people.fd.o, cgit, anongit, et al) came back up, power was again interrupted while the ext3 journal was being replayed. When it came up the n'th time, fsck dumped almost the entire filesystem in lost+found, then started saying increasingly unhappy things about the state of the filesystem on its second pass. In the end, we just went with mkfs, and now we have a brand new and shiny filesystem.

It's worth pointing out that even if this was another filesystem, such as /srv, which hosts all project data, we would've been fine, as they're all backed up. But, unfortunately for some, we made a decision a while ago to not back /home up, and didn't advertise that as widely as we should have. So, if you had stuff in annarchy:/home, it's now gone, and I hope you have backups.

Sorry about that. On the upside, I got to see PSU's new and really very nice machine room this morning, thanks to XDC being about 250m away from the PSU machine room, and fd.o is otherwise running fine. We've been talking this week about replacing our ageing hardware, which would also allow for more redundancy as well as better performance from those machines. But we still have no plans to back up /home, so if you put stuff there, please, please keep your own backups (or make sure the Wayback Machine knows about it).

X.Org 7.5 will be releasing fairly soon -- within the month. Check the updated schedule for more details.

Today's the Day:

If Collabora has seemed quiet recently, it's because we've been slaving away on various parts of a really awesome project, which we can finally start talking about. Maemo 5 is coming! \o\ \o/ /o/

In other news, xserver 1.7 status update next week.

The Call for Papers for LCA 2010 is closing this Friday, the 24th of July. I can't speak for the rest of the committee, of course, but there seems to be a bit of a trend towards more left-field talks this year, which is great. Remember to get your talk in by Friday, and see you in January!

Boycott Novell seem to be the open source (inasmuch as anyone who's never contributed anything beyond a blog can be called part of the open source community) equivalent of people on the street shouting at the sky about the government.

(Waiting for my inbox to fill up with accusations of being a Microsoft and Novell shill and/or Lefty sockpuppet due to using 'open source' instead of 'free software'. Whatever.)

If you're bored to tears, you might want to read some gutter journalism, to the extent a four-page bullying of a college student (plus some bonus guilt by association) can be called 'journalism'. Pretty sad to see that this is apparently the standard of debate these days.

Off to GCDS to join the rest of the happy Collabora massive, hurrah. :)

Received in an SMS from a friend: 'I just walked past a bloke with pasty white skin and a ponytail wearing a t-shirt that said 'I do it with Ubuntu'.'

I don't use Emacs myself, and I don't recall a single Emacs user complaining about accidentally triggering Ctrl-Alt-Backspace on their way to M-C-E-A-S paste-output-of-doctor-into-irc. Most of the grumbling came from actual users (i.e. people who don't know what an X server is, let alone how to configure it, let alone to email xorg-devel@ about it), rather than people who are perfectly capable of changing the default[0].

Regardless, Peter came up with a perfectly sane plan which makes it very easy indeed to optimise for clients with stuck grabs (being that termination requires you to be processing input events in the first place, in which case you're likely doing reasonably well anyway).

[0]: Yes, the text is woeful. Sorry.

Dom: Yeah, we used PayPal to accept payments for accommodation for the 2008 X Developers' Summit, but a combination of staggering US bank incompetence and PayPal being, well, PayPal, means that we lost about $US4500 we'll almost certainly never see again. The whole thing was a nightmare. After that, I switched to Google Checkout and didn't have a single problem, aside from it wanting to give me the whole interface in Finnish for a while and not offering a choice.

I'm at the Boston GNOME Summit at M.I.T. nice to visit the US a second time and get a grasp on another city. This is my first summit and so far it's been really nice to experience a different kind of event than GUADEC. I like the environment and the fact that is less crowded and more relaxed. It allows you to have the time to actually talk quietly with the people around.

Gtk+ Font Dialog Revamp

A while ago I tried to approach the Gtk+ standard font dialog usability issues and started some discussions in my blog.

After having some discussions that didn't really help me to get anywhere, I decided to find someone with a real usability background so I poked Máirín Duffy and she agreed to team up to solve this problem by doing some mockups for me to implement in pygtk and the run usability tests with the prototypes.

I'm glad I found someone to actually approach this problem from a proper UI design cycle rather than making things up based on gut feelings and random suggestions from other developers based on their personal preferences.

In about a week we should be able to come up with the first prototype.

GSettings/dconf

Ryan and myself have done some PR on dconf and GSetings and trying to push the proposals of inclusion of the basic infrastructure into GLib to move forward. It seems we have a way forward. This is good news for the de-Bonobo-ization goal for GNOME 3.0. and getting a more performant configuration system and a proper GObject friendly API for the platform.

GNOME Shell

Loads of sessions and audience around the GNOME Shell projects, I have to say that I've been exceptic about the direction of the Shell, but after the sessions and the overview of the RedHat guys and been running it from Karmic for a while I think it's coming along not that badly and that we could get.

Pushed out a new release of Transmageddon today. It is mostly
about fixing bugs and trying to make things more robust. But I also added the PSP and Google G1 profiles to this release.
Remuxing should be more robust now and if it lacks the plugins it needs it will let you know and let you choose something else instead.

My next step is going to be to combine the device profiles with remuxing, so that if the device you are targeting supports
for instance the audio and/or video format used in the incoming media Transmageddon will just remux it instead of decode and re-encode it. Should eventually in combination with a AC3 parser plugin enable you to just remux Matroska files with H264 and AC3 audio to MPEG TS when you choose the PS3 profile to get a playable file. Only problem there of course is the bitrate requirements of AC3 when used in MPEG TS on the PS3.

My Transmageddon hacking has slowed down a little over the last few Months. But I am still working on it, fixing bugs and adding little features. However a lot of the stuff I have been doing recently is adding code to work around or detect errors. Error handling is nice in the sense its code that help my application work on computers other than my own, but it is also something which I guess people find rather uninteresting. Its like you compare the last 3 versions and from a feature standpoint they are almost identical, even though I added quite a bit of code to handle all the kind of problems people reported to me.

For instance I spent quite a few hours yesterday adding code to make sure I could handle the situation of missing audio and video parsers. Currently if you choose passthrough mode and missed the needed parser plugin the application would just hand, with a lot of ugly spew on the command line. Well thanks to changing 70 lines of code and spending hours coming up with those lines the application now handles it gracefully. Of course for someone not running into this problem the application does nothing it didn’t before.

So part of my feel that these sort of fixes are quite boring and uninteresting, but on the other hand I guess they are exactly the stuff that is the difference between an application that obviously was never meant to work on any system apart from that of the application writer and an application that most people can actually use. And when people tell me they successfully used Transmageddon it do make me more happy than when I am told they tried it and it failed horribly. I mean the point of releasing Transmageddon to the public was not to make them familiar with python error messages

I have a small question to the more python savvy people out there though. I have been trying to set a environment variable for Transmageddon in python, but so far it doesn’t seem to work. If I in the shell do:
export GST_DEBUG_DUMP_DOT_DIR= "/tmp"
that works fine. But if I in my python code do
os.environ["GST_DEBUG_DUMP_DOT_DIR"] = "/tmp"
os.putenv('GST_DEBUG_DUMP_DIR_DIR', '/tmp')
neither of them seem to have any effect. Anyone got a clue to what I am doing wrong?
Edit: Turns out I was setting the environment variables to late in my file, I needed to do it before import gst was called Thanks Edward.

It has been about half a year since the last release, but finally, over a hundred git commits later, we have version 1.3.0 of the radeonhd driver.

You may think that a release "cycle" of 6 months is... not that much. However, as most open source projects radeonhd is pretty much understaffed. Together with lots of additional work on Novell's side (which of course reduces the amount of time Egbert and I can spend on radeonhd) it took us a while to finally find some time for polishing. Because 2D acceleration is active by default now on (almost) all chipsets, we were seeing more regressions than usual.

Never mind, you're probably more interested about the new release. These are the main changes:

• Added support for RV740, M92, M93, M97.


• Added more support for HDMI audio, XVideo color spaces, backlight control.


• Added support for power management.


• 2D acceleration (EXA) is enabled by default now, except on RV740.


• Tons of bug fixes (AtomBIOS, Cursor, DDC, EXA, LUT, MC, Quirks, RandR).

For more read the Phoronix article, the announcement mail or the README of the driver.

CIO has an interview with me.

Over a year ago openChrome 0.2.903 has been released. Since then development happened in svn only, without any releases. Up to today :)

So we have 0.2.904 with some bug fixes, many new chips support and hardware cursor and RandR improvements. It was nice to see new person - Bartosz Kosiorek - involved in this project. He tracked and killed many bugs recently.

ANNOUNCEMENT

Unfortunately no one seem to have enought time to accomplish new openchrome DRM and Mesa drivers. Porting patches that gone to openChrome's ttm branch is already hard job and there are still issues that have to be fixed.

[read more]

LWN covers Paul's and my talk at the Audio MC at LPC, Portland. (Subscribers only for now)

Update: Here's a free subscriber link.