So I've been slowly writing the hotplug support v3 in between all the real jobs I have to do.

[side note: When I started out on hotplug. one of my goals was to avoid changing the server API/ABI too much so I could continue side by side testing.]

how did I get to v3?
v0.1: was called dynerama it failed miserably and proved that using Xinerama as the plugging layer was a bad plan.
v1: was the first time I decided to use an impedance layer between some server objects and driver objects.
v2: was the a major rebase of v1.

v2 was trucking along nicely and I managed to get the design to the stage where PRIME offloading intel/nouveau worked, USB device hotplug with udl worked, and GPU switch worked between two drivers. However v2 duplicated a lot of code and invented a whole new set of API objects called DrvXRec, so DrvScreenRec, DrvPixmapRec, DrvGCRec etc, this lead me to looking at the pain of merging this into the drivers and the server, and my goals of avoiding changing the API/ABI was getting in my way.

So before starting v3 I decided to rework some of the server "APIs".

The X server has two main bodies of code, one called DIX, and one called DDX. The DIX (device independent X) code and the DDX (Device dependent X code). In the X.org tree the dix lives up in the top level dirs, and for X.org server the DDX lives in hw/xfree86. The main object with info about protocol screens and GPUs is called ScreenRec in the DDX and ScrnInfoRec in the DIX. These are stored in two arrays, screenInfo.screens in the DIX and xf86Screens in the DDX, when code wants to convert between these it can do one of a few things.

a) lookup by index, both structs have an index value, so to go from ScrnInfo to Screen you look at screenInfo.screens[scrninfo->scrnIndex] and other way is xf86Screens[screen->myNum]. This is like the I didn't try and make an API, I just exposed everything.

b) ScrnInfo has a ScreenPtr in it, so some code can do ScrnInfo->pScreen to get the pointer to the dix struct. But this pointer is initialised after a bunch of code is called, so you really can't guarantee this pointer is going to be useful for you.

c) XF86SCRNINFO uses the DIX private subsystem to lookup the Scrn in the Screen's privates. This is the least used and probably slowest method.

So also screenInfo.screens contains the protocol screens we exposed to the clients, so this array cannot really change or move around. So I'd like to add screeninfo.gpuscreens and xf86GPUScreens and not have drivers know which set of info they are working on, however (a) totally screws this idea, since the indices are always looked up directly in the global arrays.

Now lots of the Screen/ScrnInfo APIs exposed to the drivers pass an int index as the first parameter, the function in the driver then goes and looks up the global arrays.

So my first API changes introduce some standard conversion functions xf86ScreenToScrn and xf86ScrnToScreen, and converts a lot of the server to use those. Yay an API. The second set of changes then changes all of the index passing APIs to pass ScrnInfoPtr or ScreenPtr, so the drivers don't go poking into global arrays. Now this is a major API change, it will involve slightly messy code in drivers that want to work with both servers, but I can't see a nicer way to do it. I've done a compat header file that will hopefully allows to cover a lot of this stuff where we don't see it.

I've ono other API introduction on the list, Glyph Pictures are another global array indexed by screen index, I've gone and added an accessor function so that drivers don't use the index anymore to get at the array contents directly.

Once this stuff lands in the server, a team of people will go forward and port the drivers to the new APIs (who am I kidding).

Compute support with clover and r600g has been progressing very nicely over the last few months. This is due to some great work by EVOC student Francisco Jerez on the clover state tracker and gallium compute interface, and also Ádám Rák who wrote an r600g compute implementation. With these pieces in place it is possible to run simple OpenCL programs using clover and r600g! Here is what you can do to try it out:

** UPDATE: Installation instructions can now be found here.

After all that you should be ready to go. I have posted some simple OpenCL programs here. Most of these should work with clover and r600g.
There is still a lot of work left to do, so don't be surprised if your favorite OpenCL program doesn't work, but I am excited about the work that has been done so far, and I'm optimistic the open source compute stack will continue to improve.

This news is actually a bit old, but I thought I’d make use of my new presence on planet.freedesktop.org to say that the Media Player Remote Interfacing Specification is now using FreeDesktop.org infrastructure, instead of a hodge-podge of other services.

You can view the specifiation, file a bug, view the git repository and participate on the mailing list.

If you’re creating an MPRIS2-capable media player, you may want to make use of the MPRIS tester application (which is actually not hosted on fd.o, but on GitHub).

I was replying to an email, and got side-tracked into writing some Wayland anti-FUD. There are lots of myths about Wayland out there, so I thought to better make it into a blog post.

This post is about the very small overhead of a Wayland (system) compositor, and why Wayland over network will be much better than X-over-ssh.

I predict that on desktops and other systems that may have accounts for more than one person, there will actually be two Wayland compositors stacked. There is a system compositor at the bottom, handling fast user switching, replacing VT switching, etc., and then a session compositor that actually provides the desktop environment. This is not my idea, it has been written in the Wayland FAQ under "Is Wayland replacing the X server?" for a long time.

My point is: Wayland compositors will not make 3D games suck because of compositing. While explaining why, I also continue to explaining why network transparency will not suck either. Now, do not mix up these things, I am not claiming that remoting 3D games over network will magically become feasible.

The overhead of adding a system compositor in the Wayland stack will be very small. A system compositor normally does not do any real work for compositing, it only takes the buffer handle from a client compositor, and flips it onto the screen. No rendering and no image copying involved in the system compositor.

It is the same with a full-screen game vs. any Wayland compositor: the compositor will not do any real work. A game renders its image into a buffer, passes the buffer handle to the compositor, and the compositor tells the hardware to scan out the buffer. No extra copying, no extra rendering.

The overhead that will appear with adding a system compositor, is relaying input events and buffer flips. The amount of data is small, and at least buffer flips will happen at most once per vertical refresh per monitor. There is also the idea of relaying input events only once per frame. This means that CPU process context switches will increase only by few per frame, when adding another compositor in the stack. Ideally the increase is 2 per frame: a switch to system compositor, system compositor handles input and output, and a switch back.

The overhead can be this small, because the protocol has been designed to avoid round-trips. A round-trip means that one process is waiting for another to reply before it can continue. The protocol also favors batching: accumulate a bunch of data, and then send as a batch. Both of these principles minimize the number CPU process context switches.

Because of these design principles, no Wayland developer is worried about the performance of a possible network transparency layer. Minimizing CPU context switches translates directly to minimizing the effect of network latency. Some believe, that even a simple Wayland network transport which practically just relays the Wayland protocol messages as is, and adds transferring of buffer data, will clearly outperform the traditional X-over-ssh.

Now, if you still claim that X-over-ssh would be better, you a) underestimate the effect of latency, and b) forget that modern applications do not send small rendering commands through the X protocol like 10-20 years ago. Modern applications render their content client-side, and send images to the X server. Wayland simply makes images the only way to send content to the server, allowing to drop the whole rendering machinery from the server and avoiding a huge amount of protocol.

Every so-often I get asked the question of how to test the X server (or drivers) from git. The setup I have is quite simple: I have a full tree in /opt/xorg, next to the system-installed binaries in /usr. A symlink and some environment variables allow me to switch between git versions of the server and clients and the system-installed ones.

Installing the tree

Getting that setup is quite easy these days:

mkdir -p xorg/util
git clone git://anongit.freedesktop.org/git/xorg/util/modular xorg/util/modular
cd xorg
./util/modular/build.sh --clone --autoresume built.modules /opt/xorg

That takes a while but if any component fails to build (usually due to missing dependencies) just re-run the last command. The built.modules file contains all successfully built modules and the script will simply continue from the last component. Despite the name, build.sh will also install each component into the specified prefix.

You get everything here, including a shiny new copy of xeyes. Yes, what you always wanted, I know

Note that build.sh is just a shell script, so you can make changes to it. Disable the parts you don't want (fonts, for example) by commenting them out. Or alternatively, generate a list of all modules, remove the ones you don't want or need and build with that set only:

./util/modular/build.sh -L > module_list
vim module_list # you can skip fonts, apps (except xkbcomp) and exotic drivers
./util/modular/build.sh --clone --autoresume built.modules --modfile module_list /opt/xorg

Either way, you end up with /opt/xorg/bin/Xorg, the X server binary. I just move my system-installed and then symlink the new one.

sudo mv /usr/bin/Xorg /usr/bin/Xorg_old
sudo ln -s /opt/xorg/bin/Xorg /usr/bin/Xorg

Next time when gdm starts the server, it'll start the one from git. You can now update modules from git one-by-one as you need to and just run make install in all of them. Alternatively, running the build.sh script again without the --clone parameter will simply git pull in each module.

Setting up the environment

What I then define is a few environment variables. In my .zshrc I have

alias mpx=". $HOME/.exportrc.xorg"

and that file contains

export PKG_CONFIG_PATH=/opt/xorg/lib/pkgconfig:/opt/xorg/share/pkgconfig
export LD_LIBRARY_PATH=/opt/xorg/lib/
export PATH=/opt/xorg/bin:$PATH
export ACLOCAL="aclocal -I /opt/xorg/share/aclocal"
export MANPATH=/opt/xorg/share/man/

So running "mpx" will start git versions of the clients, link clients against git versions of the libraries, or build against git versions of the protocol.

Why this setup?

The biggest advantage of this setup is simple: the system install doesn't get touched at all and if the git X server breaks changing the symlink back to /usr/bin/Xorg_old gives me a working X again. And it's equally quick to test Fedora rpms, just flick the symlink back and restart the server. I have similar trees for gnome, wayland, and a few other large projects.

It also makes it simple to test if a specific bug is a distribution bug or an upstream bug. Install the matching X server branch instead of master and with a bit of symlink flicking you can check if the bug reproduces in both. For example, only a few weeks ago I noticed that xinput got BadAtom errors when run from /usr/bin but not when run from /opt/xorg/bin. Turns out it was a thing fixed in the upstream libXi but not backported to Fedora yet.

The drawback of this setup is that whenever the xorg-x11-server-Xorg module is updated, I need to move and symlink again. That could be automated with a script but so far I've just been too lazy to do it.

[Update 11.05.12: typo and minor fixes, explain build.sh -L]

I'm sick with a cold, but since I ran out of movies to watch I thought it best to try to distract myself by seeing whether the N900 could run my Doom 3 branch. It turns out the answer is "yes, but very slowly."

I suspect due to the completely non-optimized code, poor choice of formats (should be using RGB565), and avoid using highp everywhere in the shaders.


Nevertheless, it does run despite the currently poor performance and few rendering bugs...

Notification to speakers

The GUADEC 2012 programme committee took a bit more time than first anticipated to evaluate all talk submissions, but it's now all done: this morning, we finally sent the notification to speakers. Thanks to everyone who submitted a talk: it looks like we'll have a great GUADEC :-) Of course, we still need to create the schedule, but that should be trivial, right? (hmm...)

If you submitted a talk and didn't get a positive or negative answer by mail, please first check your spam folder: mail is from guadec-papers, and contains Your talk at GUADEC 2012 in the subject. If you don't find anything, feel free to ping me.

Help organize the lightning talks!

Next step is the call for lightning talks and for BoFs! I guess this will happen in the next few days. I don't think we have anyone in charge of this yet, so if that's something you'd like to help with, just drop us a quick mail on guadec-list and we'll happily give you a I'm fantastic: I'm helping organize GUADEC badge ;-)

Sometimes it feels that each project I work on has different indentation settings. Not quite true but still annoying. I don't know of a way to tell vim to auto-detect the indentation settings based on the current file (which, for X.Org projects wouldn't work anyway) but what has been incredibly useful is the vimdir script. It simply scans the directory tree upwards to find a .vimdir file and loads the settings from there. So I keep files like this around:

setlocal noexpandtab shiftwidth=8 tabstop=8

The alternative is to add a snippet to the file itself but not every maintainer is happy with that.

/* vim: set noexpandtab tabstop=8 shiftwidth=8: */

I spent some time hacking on Doom 3 adding support for EGL and OpenGL ES2.0; obviously there is still a significant amount of work to be completed before this looks even remotely like the game. Some of you might make out the text console in this image, rendered with the incorrect shader. :-) Phoronix, go nuts... :-)

(Updated again 2012-05-07: Doom 3 is nothing without stencil shadow volumes!)





(Updated 2012-05-07: I discovered and fixed a little bug in the modelViewProjection matrix...)

My attempt at Letting Go, by Andrew York:

Two weeks ago we built a slackline setup in our back yard. The issue we had was that we don't have any trees back there to tie up to. Common solutions in this case involve building an A frame and using whatever sort of anchor you can come up with, with plenty of options available.

We wanted better. The yard could only go to about 40 ft of line, and we didn't want to sacrifice precious length between our anchors and the A frame.

The first plan we were working with was to put a pipe in some cement, then slide a smaller pipe into it, and use that as our fake tree to anchor to: Now there's a solid anchor, but it's removable if I decide to sell the house or something some day. I found some numbers for guidelines for building railings, though, that indicated that you'd need massive steel pipe to support the loads we're talking about.

What we went with in the end was a wooden 4x4. We'd heard that slackliners were successfully using those in home setups. But we were a little wary of trusting a wood 4x4 more than a steel pipe. So what we buried in the cement was a post sleeve so that we could just slide our 4x4 into the cement hole after it was set. The cement was 3 feet deep and just over 1 foot across (if you decide to go this route: post hole diggers are *awesome*). This let us put an 8 foot 4x4 in each and be able to set a line at heights up to around 4 feet off the ground. But just in case, we also dropped some heavy chains into the cement as well in case we want anchors for A frames if this posts thing doesn't work out.

We first used the system last Sunday with great success. It's a typical 4-carabiner primitive system but we used a double pulley system behind that to get enough tension from a single person tightening that you'd stay off the ground in the middle. There was a disturbing amount of bending and some creaking in the 4x4s, but they held.

Today Scott was setting up the line again, and said "I got it nice and tight, look at that!", and I hopped on. I made it about 1/3 of the way, when there was a snapping sound and suddenly I was on the ground. Luckily failure wasn't as catastrophic as we feared. The post had just bent over, and not detached and gone flying.

Our next plan was to use steel I-beams: the backup plan that justified the 4x4 sleeves. I'm still concerned though -- a beam stress calculator program says that for what we're thinking is like up to 1600lbs of force at 4 feet from the support point, we end up with a maximum bending stress at the support point of 164 ksi on a S3x7.5 I-beam (the biggest that will fit in our sleeves as far as I can see). If I'm supposed to compare this number to the yield stress of the steel the beam would be made of, that number is only 22 ksi.

The plan for the moment is to throw together some A frames (actually, X frames -- Scott built and used some of those successfully this week, and it sounds easy enough) and use that unless we can figure out that I was wrong and steel will hold.

Much too often, I see patches that add code copied from other sections of the same repository. The usual excuse is that, well, we know that block of code works, it's easy to copy and we immediately get the result we need.

This is rather short-sighted. Whenever code is copied, the two instances will go and live separate lives. Code is never static, over time that copy becomes a partial reimplementation of the original.

There are a few conditions when copy-paste is acceptable:

• You can guarantee that the original code does not have any bugs and thus the copy does not have any bugs, now or in the future. Otehr


• You can guarantee that anyone making changes to this code in the future is aware of the copy and the original and their respective contexts.


• You can guarantee that the context of the original and the copy never changes in a different manner.


• You are happy to reimburse testers and developers for the time wasted tracking down bugs caused by ignoring any of the three above.

If the above are true, copying code is ok. And you probably get some price for having found an impossible piece of code. In all other cases, write a helper function and share the code. If the helper function is to unwieldy, maybe it's time to think about the design and refactor some things.

This post is part of a series on multi-touch support in the X.Org X server.

1. Multitouch in X - Getting events
2. Multitouch in X - Pointer emulation
3. Multitouch in X - Touch grab handling
4. Multitouch in X - Multitouch-touchpads

In this post, I'll outline the handling of touch events for multitouch-capable touchpads. Multi-touch touchpads that are supported are those that provide position information for more than one finger. The current version of the synaptics X driver does some tricks to pretend two-finger interaction on single-finger touchpads - such devices are not applicable here.

Touchpads are primarily pointer devices and any multi-touch interaction is usually a gesture. In the protocol, such devices are of the type XIDependentDevice and the server does adjust touch event delivery. I've already hinted at this here but this time I'll give a more detailed explanation.

Touch event delivery

Unlike for direct touch devices such as touchscreens, dependent devices have a different picking mechanism in the server. We assume that all gestures are semantically associated with the cursor position. For example, for scrolling, you would move the cursor on top of the window to be scrolled, then you would start scrolling. The server thus adjusts event delivery accordingly. Whereas for direct touch devices the touch events are delivered to whichever window is at the position of the touch, touch events from dependent devices are always delivered to the window underneath the pointer (grab semantics are adjusted to follow the same rules). So if you start a gesture in the top-left corner of the touchpad, the window underneath the cursor gets the events with the top-left coordinates. Note that the event and root coordinates always reflect the pointer position.

The average multi-touch touchpad has two modes of operation: single-finger operation where the purpose is to move the visible cursor and multi-finger operation which is usually interpreted into a gesture on the client. These two modes are important, as they too affect event delivery. The protocol specifies that any interaction with the device that serves to move the visible cursor only should not generate touch events, and that touch events will start once that interaction becomes a true multi-touch interaction. This leaves the drivers a little bit of leeway, but the current implementation in the synaptics driver is the following:

1. A user places one finger on the touchpad and moves. The client will receive regular pointer events.
2. The user places a second finger on the touchpad. The client will now receive a TouchBegin event for the first and the second touch, at their respective current positions in device coordinate range.
3. Movement of either finger now will generate touch events, but no pointer events.
4. Any other fingers will generate touch events only.
5. When one of two remaining fingers on the touchpoint ceases the touch, a TouchEnd is sent for both the terminating touch and the remaining touch. The remaining finger will revert to sending pointer events.

Legacy in-driver gestures

As you are likely aware, the synaptics driver currently supports several pseudo gestures such as tap-to-click or two-finger scrolling. These gestures are interpreted in the driver, thus the server and client never see the raw data for them.

With proper multi-touch support these gestures are now somewhat misplaced. On the one hand, we want the clients to interpret multitouch, on the other hand we want the gestures to be handled in the same manner in all applications. (Coincidentally, this is also a problem that we need to solve for Wayland).

We toyed with ideas of marking emulated events so clients can filter but since we do need to be compatible to the core and XI 1.x behaviours, we only found one solution: any in-driver gestures that alter event deliver must not generate touch events. Thus, if the driver is set to handle two-finger scrolling, the clients will only see the pointer events and scroll events, they will not see touch events from two-fingers. To get two-finger scrolling handled by the client, the in-driver gesture must be disabled. The obvious side-effect of that is that you then cannot scroll in applications that don't support the gestures. Oh well, it's the price we have to pay for having integrated gesture support in the wrong place.

xf86-input-synaptics 1.6.0 was released today and it packs a bunch of changes. This post outlines some of these changes and future directions for the drive.

Clickpad support

Clickpads are touchpads without physical buttons. Instead, the user has to press the whole pad down to trigger a click. These pads are becoming increasingly common in laptops but need a bunch of software workarounds to make them useful with traditional interaction methods. I've got a post dedicated to Clickpad support in the synaptics driver, so best to read that for more details.

The important bit about clickpad support is that it's only available on multi-touch capable X servers. If you're running an older server, you won't see clickpad-y goodness. Sorry.

Smooth scrolling

Smooth scrolling is now supported by the driver as well. On XI 2.1-compatible servers (read: server 1.12 [1]), the driver will submit scrolling valuators instead of scroll events. This makes for a smoother scrolling experience on clients that support it.

We're in a transition period where clients start to enable smooth scrolling but many other clients don't support it yet. For now, we left coasting enabled by default in the driver to provide a consistent experience. This will likely be disabled once smooth scrolling support and client-side coasting becomes more prevalent.

Meanwhile, clients can disable it by setting the "Synaptics Coasting Speed" property to 0, 0.

Bugs, bugs, bugs

Luckily I was able to spend quite some time on synaptics over the last weeks to fix various bugs and I think this is the most solid release of synaptics in quite a while. A few bugs that you may have seen that have now been fixed are:

• touchpad unresponsive after suspend  #49161
• weird cursor jumps while dragging #48777
• broken double-tap #31854
• slow or too fast scrolling on pre-1.12 servers #46617

and a few more.

Future of synaptics

What becomes more and more obvious is that the driver is on the edge of being unmaintainable. Pushing new features into the driver always always means that some other feature is broken and sometimes we don't notice for months. Having around 75 options in the driver doesn't help here, testing all combinations is impossible.

I can't claim that I have big plans because I tend to get pre-empted by misc things all the time but the rough plan is to keep a 1.6.x series while re-doing some of the internals of the driver for the 1.7 release. Expect the git master branch to see a few big changes though.


[1] Ubuntu 12.04 ships a 1.11 with the 1.12 input stack, so the above applies to that server too.

We are working on putting together a miniconf on the topic of Boot & Base OS for the Linux Plumbers Conference 2012 in San Diego (Aug 29-31). And we need your submission!

Are you working on some exciting project related to Boot and Base OS and would like to present your work? Then please submit something following these guidelines, but please CC Kay Sievers and Lennart Poettering.

I hope that at this point the Linux Plumbers Conference needs little introduction, so I will spare any further prose on how great and useful and the best conference ever it is for everybody who works on the plumbing layer of Linux. However, there's one conference that will be co-located with LPC that is still little known, because it happens for the first time: The C Conference, organized by Brandon Philips and friends. It covers all things C, and they are still looking for more topics, in a reverse CFP. Please consider submitting a proposal and registering to the conference!

There's one feature In the upcoming Fedora 17 release that is immensly useful but very little known, since its feature page 'ckremoval' does not explicitly refer to it in its name: true automatic multi-seat support for Linux.

A multi-seat computer is a system that offers not only one local seat for a user, but multiple, at the same time. A seat refers to a combination of a screen, a set of input devices (such as mice and keyboards), and maybe an audio card or webcam, as individual local workplace for a user. A multi-seat computer can drive an entire class room of seats with only a fraction of the cost in hardware, energy, administration and space: you only have one PC, which usually has way enough CPU power to drive 10 or more workplaces. (In fact, even a Netbook has fast enough to drive a couple of seats!) Automatic multi-seat refers to an entirely automatically managed seat setup: whenever a new seat is plugged in a new login screen immediately appears -- without any manual configuration --, and when the seat is unplugged all user sessions on it are removed without delay.

In Fedora 17 we added this functionality to the low-level user and device tracking of systemd, replacing the previous ConsoleKit logic that lacked support for automatic multi-seat. With all the ground work done in systemd, udev and the other components of our plumbing layer the last remaining bits were surprisingly easy to add.

Currently, the automatic multi-seat logic works best with the USB multi-seat hardware from Plugable you can buy cheaply on Amazon (US). These devices require exactly zero configuration with the new scheme implemented in Fedora 17: just plug them in at any time, login screens pop up on them, and you have your additional seats. Alternatively you can also assemble your seat manually with a few easy loginctl attach commands, from any kind of hardware you might have lying around. To get a full seat you need multiple graphics cards, keyboards and mice: one set for each seat. (Later on we'll probably have a graphical setup utility for additional seats, but that's not a pressing issue we believe, as the plug-n-play multi-seat support with the Plugable devices is so awesomely nice.)

Plugable provided us for free with hardware for testing multi-seat. They are also involved with the upstream development of the USB DisplayLink driver for Linux. Due to their positive involvement with Linux we can only recommend to buy their hardware. They are good guys, and support Free Software the way all hardware vendors should! (And besides that, their hardware is also nicely put together. For example, in contrast to most similar vendors they actually assign proper vendor/product IDs to their USB hardware so that we can easily recognize their hardware when plugged in to set up automatic seats.)

Currently, all this magic is only implemented in the GNOME stack with the biggest component getting updated being the GNOME Display Manager. On the Plugable USB hardware you get a full GNOME Shell session with all the usual graphical gimmicks, the same way as on any other hardware. (Yes, GNOME 3 works perfectly fine on simpler graphics cards such as these USB devices!) If you are hacking on a different desktop environment, or on a different display manager, please have a look at the multi-seat documentation we put together, and particularly at our short piece about writing display managers which are multi-seat capable.

If you work on a major desktop environment or display manager and would like to implement multi-seat support for it, but lack the aforementioned Plugable hardware, we might be able to provide you with the hardware for free. Please contact us directly, and we might be able to send you a device. Note that we don't have unlimited devices available, hence we'll probably not be able to pass hardware to everybody who asks, and we will pass the hardware preferably to people who work on well-known software or otherwise have contributed good code to the community already. Anyway, if in doubt, ping us, and explain to us why you should get the hardware, and we'll consider you! (Oh, and this not only applies to display managers, if you hack on some other software where multi-seat awareness would be truly useful, then don't hesitate and ping us!)

Phoronix has this story about this new multi-seat support which is quite interesting and full of pictures. Please have a look.

Plugable started a Pledge drive to lower the price of the Plugable USB multi-seat terminals further. It's full of pictures (and a video showing all this in action!), and uses the code we now make available in Fedora 17 as base. Please consider pledging a few bucks.

Recently David Zeuthen added multi-seat support to udisks as well. With this in place, a user logged in on a specific seat can only see the USB storage plugged into his individual seat, but does not see any USB storage plugged into any other local seat. With this in place we closed the last missing bit of multi-seat support in our desktop stack.

With this code in Fedora 17 we cover the big use cases of multi-seat already: internet cafes, class rooms and similar installations can provide PC workplaces cheaply and easily without any manual configuration. Later on we want to build on this and make this useful for different uses too: for example, the ability to get a login screen as easily as plugging in a USB connector makes this not useful only for saving money in setups for many people, but also in embedded environments (consider monitoring/debugging screens made available via this hotplug logic) or servers (get trivially quick local access to your otherwise head-less server). To be truly useful in these areas we need one more thing though: the ability to run a simply getty (i.e. text login) on the seat, without necessarily involving a graphical UI.

The well-known X successor Wayland already comes out of the box with multi-seat support based on this logic.

Oh, and BTW, as Ubuntu appears to be "focussing" on "clarity" in the "cloud" now ;-), and chose Upstart instead of systemd, this feature won't be available in Ubuntu any time soon. That's (one detail of) the price Ubuntu has to pay for choosing to maintain it's own (largely legacy, such as ConsoleKit) plumbing stack.

Multi-seat has a long history on Unix. Since the earliest days Unix systems could be accessed by multiple local terminals at the same time. Since then local terminal support (and hence multi-seat) gradually moved out of view in computing. The fewest machines these days have more than one seat, the concept of terminals survived almost exclusively in the context of PTYs (i.e. fully virtualized API objects, disconnected from any real hardware seat) and VCs (i.e. a single virtualized local seat), but almost not in any other way (well, server setups still use serial terminals for emergency remote access, but they almost never have more than one serial terminal). All what we do in systemd is based on the ideas originally brought forward in Unix; with systemd we now try to bring back a number of the good ideas of Unix that since the old times were lost on the roadside. For example, in true Unix style we already started to expose the concept of a service in the file system (in /sys/fs/cgroup/systemd/system/), something where on Linux the (often misunderstood) "everything is a file" mantra previously fell short. With automatic multi-seat support we bring back support for terminals, but updated with all the features of today's desktops: plug and play, zero configuration, full graphics, and not limited to input devices and screens, but extending to all kinds of devices, such as audio, webcams or USB memory sticks.

Anyway, this is all for now; I'd like to thank everybody who was involved with making multi-seat work so nicely and natively on the Linux platform. You know who you are! Thanks a ton!

This came up on the linuxwacom-devel list today and I think it warrants further spread through this post.

Different projects have different patch review requirements but the biggest difference is pre-review and post-review. That is, do patches get reviewed before or after they hit the repositories. Not too long ago, the X server employed a post-review process. Everyone with access pushed and bugs would get discovered by those reading commit logs. Patches that ended up on the list were mainly from those that didn't have commit access. Beginning with server 1.8 we introduced a hard review requirement and every patch to make it into the repos ended up on the list, so we switched from post-review to pre-review.

Aside from enforcing that someone gives the formal ACK for a patch, a side-effects is to allow for a passive "drive-by" learning of the code base. Rather than having to explicitly look up commit logs, patches are delivered into one's inbox, outlining where the development currently happens, what it is about and - perhaps most importantly - issues that may have been caused bugs in rejected patches. Ideally that is then archived, so you can link to that discussion later.

The example I linked to from above is automake's INCLUDES versus AM_CPPFLAGS. I wouldn't have know about them if it wasn't for skimming through Gaetan's patches to various X.Org-related projects. That again allowed me to contribute a (in this case admittedly minor) patch to another project. And since that patch ended up on another list, the knowledge can spread on.

Next time when you think of committing directly to a repo, consider sending the patches out. Not just for review, but also to make learning easier for others.

In my last post I talked about Clickpad support in the synaptics driver. Unfortunately, right around the same time a kernel bug surfaced that messed with the events from some clickpads (the Synaptics-branded ones, Apple and others were unaffected). Affected was the 3.3.x series and obviously 3.4. Thanks to +Benjamin Herrenschmidt, that bug was fixed and it is now available in a variety of kernels. The upstream commit is in Linus tree now, still making it into stable (not in 3.3.4 yet). Fedora users can grab Fedora 17 or Fedora 16 kernels that carry this patch.

Another month is at end, and it looks like a good timing for another update about the Intel Linux Graphics project.

As usual, starting with Kernel, I can say that this way a very, very busy month, and we had more kernel patches during this time than I can remember from the past. Several patchbombs came through, such as:

• Several rounds of Ben‘s patches for timed GPU execution, or BO waits. Those patches add the ability to specify the desired timeout parameter to the core GPU functions, and allow the userspace to specify how log to wait until they are done – instead of waiting for them indefinitely. It is a nice addition to the way userspace cooperates with the Kernel part for the GPU-related tasks, and a helpful addition to support the GL_ARB_sync OpenGL extension.
• Also from Ben, we had some patches that fixed a couple of sparse warnings. They should not change anything functionality or performance-wise, but they make the driver code more elegant .
• From Daniel Vetter, we had a 22-patch series for dragon slaughtering consolidation of the legacy DRI1 code paths. Those patches do a huge cleanup and properly split the DRI1 and DRI2 calls in a way that everything related to DRI1 should belong to the i915_dma.c module now.
• Chris Wilson and Jesse Barnes have been working on a better way to handle the PCH PLLs separately from the display pipes. This patch greatly simplifies the interaction with the components of the GPU located on the chipset, and allows to allocate or re-use different PLLs on-demand. This came particularly handy for my Haswell enablement efforts, where most of the things are being done by the CPU part of the graphics subsystem, and PCH is used for non-essential tasks.
• Also from Chris Wilson, we had a 28-patch series for reusing the stolen memory early in the GPU initialization. Stolen memory is being allocated by BIOS for its need, and previously, when Kernel GPU driver was loaded, most of the memory was re-initialized. With those new patches, it is possible to reuse BIOS-allocated objects and even allocate frame buffer from the stolen memory. Yet another step towards a faster and flicker-less boot!
• Ben Widawsky has sent his patches for L3 cache remapping. Those patches account for the row-level remapping of cache content when a parity error is detected, and this task is carried transparently to the software. On Ivy Bridge (which those patches are targeted), this situation is detected by means of an interrupt, and when this interrupt gets to the i915 kernel driver, it is able to cope with it and keep track of the failed bits. In other words, you may think on it as on a some sort of badblocks handling which is used for hard disks disks, but for L3 cache. Except that the L3 errors are much, much rare to occur.
• Some power subsystem refactoring came from me, where all the power-related components of the driver (rc6, watermarks, framebuffer compression, power monitoring and turbo-related bits) got moved into the intel_pm.c module; and Daniel Vetter has also complemented this new module with additional power-related items I missed in my original move.
• As another major patchbombing, I’ve send another round of Haswell GPU enablement patches, which add a new intel_ddi.c module, improve support for HDMI outputs, and address most of the comments I’ve received for the past months about those patches.
• And finally, still on Intel Linux Graphics page but on an entirely different hardware, Alan Cox has sent a round of updated kernel patches for GMA500 support to inclusion into the drm-next tree.

Besides those major patch series, we had several patches for i8xx interrupts handling and gen2 fixes (which also helps to answer the question whether the old i8xx graphics cards are still supported ), improvements in page flipping, turbo initialization, Sandy Bridge GPU hangs in Google Maps/Earth, proper IPS polling which should be limited to Gen5 architecture now, pineview clock gating and several backlight fixes, besides all the other smaller ones.

On 2D driver side, the major news of the past month was the release of Glamor 0.4 acceleration backend. The biggest changes are the support for DRI2 and texture from pixmap functionalities, co-existance with AIGLX, many performance optimizations for pixmap uploading/downloading, full GLES2 color formats support and a new texture cache pool mechanism to reduce texture/fbo creating and destruction overhead and considerable improve overall performance.

Still on the 2D side of things, a new Cairo 1.12.2 release went out. This is a largely bugfixing release, which fixed a large number of issues which were found out since the Cairo 1.12 got released. Chris Wilson has also carried out a performance evaluation of Cairo 1.12.2 with different backends in his blog.

On Mesa side, lots of bugs were fixed, as usual, and among the most interesting updates for the past months that I caught were:

• Support for RGBX_8888 format used in Android native buffers from Sean Kelley.
• Support for additional gbm interfaces from Ander, used to allow weston compositor to reuse KMS framebuffer objects instead of creating and destroying them for each rendered frame.
• Addition of the DRI image v4 extensions to MESA_drm_image, and support for additional DRI image formats from Gwenole.
• Also from Gwenole, 5-patch series for the DRI2 changes to add support for the new Wayland and VA/EGL proposals to handle YUV buffers, and differentiate progressive/interlaced contents.
• A couple of fixes for mipmap offsets when used with HiZ and separate stencil buffers, from Paul Berry.
• Lots of work on GLSL support, from Eric, Ken, Ian and Paul.
• And finally, still from Gwenole, came a patch series to allow caching data regions according to a given offset.

Moving to Wayland, the patches that caught my attention were:

• Collabora’s Pekka Paalanen‘s patches for adding Android support for Wayland (which were already covered by phoronix as well).
• Ander’s patches for reusing KMS framebuffer objects.
• Gwenole‘s patches that added support for querying for the list of supported surface formats, support for YUV buffers and generic buffer formats.
• And Tiago Vignatti‘s patches for xwayland improvements, customizable modifier key biddings.

And, last but not least, we’ve gone through the list of the open freedesktop.org and kernel.org bugzillas, and managed to scrub dozens of fixes in the past few weeks, solving a huge variety of issues – from i8xx-specific bugs up to recent GPU turbo issues after idle on newer machines. If you have a bug open with us, take a look if it is still active – chances are that many of the issues you could have were already gone . And if it still there, it would be a great time to check it again and verify if the drm-intel-next tree is still affected by it. As Kernel 3.5 merge window gets closer, it is a very good timing to try to address the remaining issues prior to its opening.

That’s all for now – stay tuned for future news from the Intel Linux Graphics land, and enjoy the International Workers’ Day meanwhile – if your country considers it as a holiday, of course !

The X.org window server is the foundation of the Linux desktop today. There are hopes that we will be moving to Wayland in the near future, but for now everything in Ubuntu and other desktop Linux distributions depend on X.org as the window server.

Unfortunately, X.org has very limited testing. There are some unit tests of internal functions, but much of the bugs in X are caused by specific server state that cannot be easily replicated in unit testing environments. There are some testing functionalities provided by XTest, but it does not provide a testing environment or framework. This poor test support has left us with very little testing. Practically no bug fixes are accompanied by regression tests, so bugs can reappear in later releases without anyone knowing.

I have been working on a project to bring better integration testing to X.org. Thomas Voss and I created xorg-gtest, which is a gtest wrapper to provide a real X.org environment for running tests and some useful utility functions and test fixtures. We have been using it for uTouch testing, which you can see at the Ubuntu Jenkins console, and last week I proposed the first regression test for the upstream X.org server.

There are a few key things that xorg-gtest provides:

• A test X.org server that is started before tests are run and stopped when tests finish
• A well abstracted testing framework (Google Test)
• Support for playing recorded input event sequences through the Linux kernel

Making testcase writing easy is a key goal of xorg-gtest. The testcase developer can subclass the xorg-gtest base test fixture, which sets up an X11 display connection, and then start writing the test case. Due to the features of gtest, the test case does not need to be explicitly registered; linking the test case into the test program is all that is needed. Further, the developer does not need to write a main() function for the test program. An xorg-gtest main() function is provided as a standalone library and should be sufficient for almost all testing requirements. The following is a really simple example testcase:

#include <xorg/gtest/test.h>

using namespace xorg::testing;

TEST_F(Test, MyExampleTest) {
  EXPECT_NE(0, DefaultRootWindow(Display()));
}

All that is needed is to compile this file and link it with the xorg-gtest and xorg-gtest_main libraries. The following will be output when the test runs:

$ ./xorg-gtest-example

[==========] Running 1 test from 1 test case.

[----------] Global test environment set-up.
[----------] 1 test from Test
[ RUN ] Test.MyExampleTest
[ OK ] Test.MyExampleTest (175 ms)
[----------] 1 test from Test (175 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (3181 ms total)
[ PASSED ] 1 test.

The test X.org server is started and stopped in the “Global test environment set-up” and “Global test environment tear-down” phases. In five lines of code we have created a simple test that runs against a headless X.org server. Simple!

There are limitations, of course. The default test server configuration provided by xorg-gtest uses the xf86-video-dummy driver. This allows for running the test server without interrupting your current desktop session. For input testing this works great, but it falls short of what would be needed for some graphical and graphics driver testing. For 2D graphic tests that are not driver specific it may be possible to query the dummy framebuffer contents. For 3D or driver-specific tests the dummy graphics driver won’t work. However, the xorg-gtest provided main() function has the –no-dummy-server option to force tests to be run in the existing X session. Thus, it may be possible to use xorg-gtest for xf86-video-intel testing when running in an existing X session on Intel hardware.

For more information on xorg-gtest, please see the documentation at http://people.freedesktop.org/~cndougla/xorg-gtest/. For a real world X.org testcase, see my proposed test for XIQueryPointer with touchscreens here. Note the parameterized test instantiation at the very bottom of the patch. It shows how to easily test different behavior for different XInput client versions.

A month in, and we’re starting to get some good feedback from people trying cairo-1.12. Unfortunately, it appears that we’ve exposed some bugs in a few drivers, though hopefully we will see driver updates to resolve those issues shortly. We also received a few bug reports against Cairo itself. The most bizarre perhaps was that LibreOffice failed to display the presentation slideshow. That turned out to be an inadvertent bug caught by better error detection – though since that affected the established API, we had to relax the checks somewhat. Along with a few other bug fixes, Cairo 1.12.2 was released.

In the last round of benchmarking I performed, some of you noticed that the glamor backend for the Intel driver was not included. It was left out for the simple reason that it was not able to complete the task. However with the first stable release of glamor-0.4, it is able to complete a benchmarking run. And so without further ado, let’s see how all the current drivers fare on a SandyBridge i5-2500 desktop with GT1 integrated graphics and a Radeon HD 5770 discrete GPU with cairo-1.12.2.

This time the results are normalized to the performance with Xvfb. Any driver that performs a test faster is above the centre, any that were slower below. Again the general state of the drivers leave much to be desired, and despite the bold claims for glamor, in my testing it fails to improve upon UXA. Early days you might say.

Sometime you feel like you need to cry out loud the painful process that GPU driver development is. Over the last few weeks i have been working on DP->VGA bridge (named nutmeg) for AMD A serie integrated GPU. Up until now the VGA output of anything using this nutmeg bridge never light up, of course all the documentation we had told us that we were doing things properly, but when the monitor desperetly stays black, you have to face the true, it doesn't work.

So i went on a journey, when working on GPU, journey always turns up in some kind of heroic quest. But first a bit of background on how you get a picture on the monitor :

framebuffer -> crtc -> encoder -> transmitter -> connector -> monitor

Of course things can go wrong in any or all or some of those stages (see http://www.botchco.com/agd5f/?p=51  if you want a longer description). Display port add more fun to the mix, the main idea behind display port is that you have train the link between the source, your GPU, and the sink, usualy your monitor but in this case the nutmeg bridge that convert the DP to VGA. Link training is one of those things that never go quite right. It's as if it was designed to be able to fail in an unlimited number of way, creativities in failure is a slogan that would fit display port. Don't get me wrong, i prefer display port of DVI or HDMI any time of the day. Display port is just almost perfect, but people designing it must had a bad day when they came up with link training.

So display port link training between the integrated GPU and the nutmeg VGA bridge was failing reliably for me. As usual i spend few days trying all sort of incentation to make it works, bending the display port specification in all possible way, interpreting it in the most non logical and backward way. I was brave and discarded no possibilities.

Because GPU would never accept to surrender to your will easily, all my attempt at fixing the link training were in vain. So it was time for me to go look at what the good old vesa bios was doing. Because the hard true is that vesa was successfull at bringing up VGA. Running the vesa bios in an emulator and catching register read/write was something we usualy did back in the day, so we already have a tool for that. But nowadays register read/write are too verbose, there is too many of them and it's way too tedious to figure out what and why such register have such value.

Before going further i must quickly describe atombios, the poor folks at AMD that work on video bios decided to come up with a simple langage (opcode is allmost limited to jump,nop,delay,mov,test,add,sub,shift,mask) that could be reuse by the driver to perform common task that OEM might need to tweak for their specific design. You see each OEM have some freedom around what kind of component it pair the GPU with. For instance different OEM can choose different DDR chips, each chips has its own timing and needs its own special initialization. For sake of simplicity anything that is specific to an OEM should be hidden behind atombios data or code. Why not using x86 directly ? Well nowadays running x86 real mode code can be tricky and x86 is not the only beast in town. Atombios offer a simple langage for which one can easily write an interpreter.

One of the thing that can be, and should be, share is the modesetting code, this code might need some special tweak by each OEM depending on voltage, frequency of the board, and possibly associated external bridge. So atombios is the perfect place for the modesetting code and it allows the video bios and the driver (wether closed or open source) to reuse the same code.

Here i am, i don't want to trace register read/write of the vesa bios but i would like to trace atombios execution of the vesa bios. You see the x86 vesa bios is allmost only just an atombios interpreter that call and execute various atombios function. What is interesting to know is the arguments the vesa bios is giving to those atombios function. Because if there is a difference between the open source driver and the vesa bios it must be in the argument to the atombios function, or an atombios function we don't call, or one we should not call.

So i needed to  trace the vesa atombios execution and catch the argument it gave to the atombios function. Thing about video bios is that there is no symbol, it's just pure x86 real mode assembler. Nevertheless there must be an entry point to the atombios interpreter in the vesa bios. The entry point offset will be in some of the call x86 opcode. So here i am editing the x86emu and tracing all the call opcode and printing register value and stack at each call. You would think that video bios is small and that there isn't much function in it, but of course it's not. I ended up with several thousand of different call, of all those call ~400 different function (offset). That's where you know you need to think a bit and find a way to beat the machine. 

I want over all the atombios function argument and looked for some i could easily guess the values the vesa bios would use, for instance the video mode size. One more thing about atombios function, they are all identified by an unique index. A given function has a fixed index and you look up its offset inside a table (using the index) in the video bios. The atombios interpreter entry point must take as argument the index of the atombios function and the arguments to the atombios function. Now i knew what kind of number i was looking for. After few (if you ever wanted to meet an euphemism you now did) false positive i finaly spoted this interpreter offset and identified on the stack the arguments to the atombios function.

It wasn't all downhill from that point, i now had a full trace of vesa setting a mode with atombios. I first replayed the trace to make sure that i did capture thing properly and that i got everything, and of course it worked. But this trace was way too big, it had too many call to too many atombios function. I had to trim it down, and so i did. This is the usal start from non working condition, remove some atombios function call, run the trace if it bring up the VGA output, it means this atombios call is not crucial so one can overlook it. Well you get it, refine and repeat over and over until you come up with a minimal trace.

Of course, GPU is a twisted thing, and nothing goes as plan with it. So i started comparing minimal trace with the open source driver atombios execution and parameter and found nothings, or so i thought. You see, most of the atombios function take 1 or 2 dword as arguments, everythings is tightly packed, each field take only the minimun number of bit needed for it. And there was my demise, i only paid attention to the first 2 dwords, but the tiny single bit difference between the open source driver and the vesa bios was in the third dword, hidden from my scrutiny.

Of course i can only blame my self for not being thorough enough, but when you spend hour looking at hexadecimal strings, you just can't help being lazy, at least i can't.

This was the tell of how to fix the VGA output of AMD A integrated GPU (also known as llano) in 3 weeks. You should not worry about my next journey, it will once again prove tedious and frustrating. I also hope that this little story shed some light on the difficulties of modesetting, many people believe that 3D engine is the most complicated piece of the GPU, well modesetting is the most unwilling and failure creative piece of the GPU (for all fairness sometime the monitor helps with broken EDID, narrow minded working frequency, limited tolerance, ...). One last thing about modesetting, without it nothing on the screen ... that could disappoint people that want to use their computer.

A couple of months ago, Collabora assigned me first to research and then make a proof of concept port of Wayland on Android. I had never even seen an Android before. Yesterday, Weston on Android achieved first light!

Galaxy Nexus running Weston and simple-shm.

That is a Samsung Galaxy Nexus smart phone, running a self-built image of Android 4.0.1. Weston is driving the screen, where you see the simple-shm Wayland client. There is no desktop nor wallpaper, because right now, simple-shm is the only ported client.

How is that possible? Android has no DRI, no DRM, no KMS (the DRM API), no GBM, no Mesa, and for this device the graphics drivers are proprietary and I do not have access to the closed driver source.

Fortunately, Android's self-invented graphics stack has pretty similar requirements to Weston. All it took was to write a new Android specific backend for Weston, that interfaces to the Android APIs. Writing it took roughly three days.

And the rest of the two months? I spent some time in studying Android's graphics stack, but the majority of the time sunk into porting the minimum required library dependencies, libwayland, Weston, and simple-shm to the Android platform and build environment. Simply getting the Android build system to build stuff properly took a huge effort, and then I got to write workarounds to features missing in Android's C library (Bionic). Features, that we have taken for granted on standard Linux operating systems for years. I also had to completely remove signal handling and timers from libwayland, because signalfd and timerfd interfaces do not exist in Bionic. Those need to be reinvented still.

Android has gralloc and fb hardware abstraction layer (HAL) APIs. Hardware vendors are required to implement these APIs, and provide EGL and GL support libraries. These implementations are usually closed and proprietary. On top of these is the Android wrapper-libEGL, written in C++, open source. My first thought was to use the gralloc and fb HAL APIs directly, but turned out that the wrapper-libEGL does not support using them in the application side. Instead, I was forced to use some Android C++ API (there is no C API for this, as far as I can tell) to get access to the framebuffer in an EGL-compatible way. In the end, I had to write a lot less code than using the HALs directly.

The Android backend for Weston so far only provides basic graphics output to the screen, and offers (presumably) accelerated GLES2 via EGL for the server. No input devices are hooked up yet, so you cannot interact with Weston. I do not know how to get pageflip completion events (if possible?), so that is hacked over.

Simple-shm is the only client that runs for now. There is no support for EGL/GL in Wayland clients. Toytoolkit clients are waiting for Cairo and dependencies to be ported.

The framebuffer can be used by one program at a time. Normally that program is SurfaceFlinger, the Android system compositor. To be able to run Weston, I have to kill SurfaceFlinger and make sure it stays down. Killing SurfaceFlinger also kills the whole Android UI infrastructure. You cannot even power off the phone by pressing or holding down the physical power button!

A video about simple-shm running on Galaxy Nexus:

The sources with Android build integration and other hacks can be found here:

• http://cgit.collabora.com/git/user/pq/wayland.git/log/?h=android
• http://cgit.collabora.com/git/user/pq/wayland-demos.git/log/?h=android
• http://cgit.collabora.com/git/user/pq/wayland_aggregate.git/

The wayland_aggregate is how I actually connect to the Android build system. Building is not trivial, and you cannot simply do a checkout and start compiling. You have to get the right Android tree for your device, add my local manifest (which still points to repositories on my hard drive, i.e., won't work for you), download and extract binary blobs, and whatnot. Be warned, I will rebase the above branches.

This is the beginning of pushing a Wayland stack into Android. Next I need to clean up, send stuff upstream, add input support, find out about that pageflip, reinvent signal handling and timerfd, and then move on to the second major task: supporting Wayland GL clients. I hope it is possible to implement the Wayland platform in the wrapper-libEGL.

This work is sponsored by Collabora, Ltd, and I also thank my fellow collaborans for guidance through the vast jungles of Android.

Of the Wayland demo clients in the Weston repository, simple-shm is the simplest. All the related code is in that one file, and it interfaces directly with libwayland. It does not use GL or EGL, so it can be ran on systems where the EGL stack does not support the Wayland platform nor extensions. However, what it renders, is surprising:

The original simple-shm client on a Weston desktop.

The square with apparently garbage texture is the original simple-shm. To any graphics developer, who does not know any better, that immediately looks like something is wrong with the image stride somewhere in the graphics stack. That really is what it was supposed to look like, not a bug.

I decided to propose a different rendering, that would not look so much like a bug, and had some real diagnostic value.

The proposed appearance of simple-shm, the way it is supposed to look like.

The new appearance has some vertical bars moving from left to right, some horizontal bars moving upwards, and some circles that shrink into the center. With these, you can actually see if there is a stride bug somewhere, or non-uniform scaling. There is one more diagnostic feature.

This is how the proposed simple-shm looks like when the X-channel is mistaken as alpha.

Simple-shm uses XRGB buffers. If the compositor does not properly ignore the X-channel, and uses it as alpha, you will see a cross over the image. Depending on whether the compositor repaints what is below simple-shm or not, the cross will either saturate to white or show the background through. It is best to have a bright background picture to clearly see it.

I do hope no-one gets hypnotized by the animation. ;-)

SUSE is hiring people for the Boosters team! This is the team I've been involved in in the last few years, so I thought I'd share with you a few words on this...

The Boosters are working on enabling openSUSE contributors to reach their goals. This can involve technical diving, an artistic vision (not required, obviously, or I woulnd't be in the team ;-)), marketing fun, talking at events, discussing issues, etc.: all skills are welcome in our team, as all skills are welcome and needed in the community! It's really an amazing job where you're simply part of the community and your goal is to help the community move in the right direction. On top of that, I have to mention that the Boosters team is full of great minds, and we're enjoying every day working on something we love!

Dream job, some might say :-)

Are you interested? Check out the details and apply! You can also check the other open positions at SUSE, there might be the one you're looking for... ­Oh, and as we keep hiring, remember to check out the careers page every now and then to see the latest openings!

Looking back in time

Jason just pointed out in this email that the fact that we had a window of opportunity during the netbook boom that was lost was a failure of GNOME as a project. I'd like to point out that first of all, I think that if an ISV and an OEM failed to use GNOME and engage with our community to affect change so that they could build their own product, it's their failure, not ours. Nonetheless he has a point.

It's worth noting that people buying netbooks just wanted more portable and cheaper version of the traditional Windows desktop, and that's why demand for netbooks shrunken after a while. It failed to match the expectation of the user, and no matter how good and perfect GNOME was back then, we would have failed to match that expectation.

However we can learn a few things about what's been happening since then.

I am not saying that we are not making steps towards this goal at all, a lot of work is being done, but if we could concentrate the same amount of energy that we did for the 3.0 release on this goal, the possibilities are pretty encouraging.

I recently implemented a GLSL renderer backend for Doom 3. Yes, there are already a couple of backends existing (e.g. raynorpat's) unfortunately these did not run successfully on my hardware and had serious rendering and pixel errors.

These images are from the first implementation of my backend, where I had accidentally called normalize() on a vector which was almost normalized. The result is pixel-imperfection when compared to the standard ARB2 backend, and the cost of pointless normalization in the fragment shader.

You can also see the importance of running a comparison or image-diff program when implementing a new backend. Can you see the differences between the first two images immediately, with the naked eye? I couldn't.



Finally, here is the backend running the hellhole level. The black regions are areas that would be rendered by the (currently unimplemented in GLSL) heatHaze shader. Not bad for an i965 GPU.



Just for the laughs, here is what happens when Doom 3 decides to try LSD; or fails to pass initialized texture coordinates from the vertex program to the fragment program in the ARB2 backend.

Having signed and sent of my contract I think the time has come to let the every one know that I am starting to work for what I think is a very cool company in just two Months, Red Hat. I have been a Red Hat and later Fedora user ever since I first tried Linux back in the late 90ties (by installing either 4.1 or 4.2 of Red Hat Linux), so it does feel like a bit of a homecoming for me. And in my opinion Red Hat is still what they where back then, the biggest driving force behind pushing linux and open source forward. And this is what I hope to be a part of by joining Red Hat, to play my part in continuing to push new innovations into the operating system and through that make it an even better choice for more and more users and organisations out there.

It is still with a bit of melancholy that I leave Collabora behind as I am very proud of the team we built there over the last five years. The amount of features and maturity we managed to put into GStreamer over those years and all the new usecases we managed to cover is quite amazing. But with Rob and Philippe at the helm I am sure the company will continue to prosper even without me.
I will of course continue to be a part of the GStreamer community and will be working with both Collabora and the GStreamer community for instance on organizing this years GStreamer Conference (hope to have the GStreamer Conference 2012 page up in a few days). So while I am now moving onto a new challenge I do not plan on leaving all the great people and friends I made behind, in fact since my new role at Red Hat will be as part of the Desktop team I will continue to haunt a lot of the same conferences and gatherings in the years to come .

Also as part of this new job I will be moving to Brno in the Czech Republic, joining the 400 strong Red Hat team there. For me, the best thing about living in Brno, in addition to being a very nice city, is that it will allow me to put even more pressure on Jimmac to create a new icon for Transmageddon, as he will be living only a few hours away by car

I have discovered some bugs in my previous entry regarding Google Calendar synchronization. Firstly there is a bug in the locking mechanism causing it to sometimes hang while running fetchmail. The following is the corrected .procmailrc entries:

#
# === Google Calendar (from Microsoft Exchange; iCalendar) ===
#
:0 c:
* H ?? ^Content-Type: multipart/(alternative|mixed)
* B ?? ^Content-Type: text/calendar
{
 :0 Wac: Mail/multipart.lock
 | munpack -q -t -C ~/Mail/.munpack 2> /dev/null
 :0 Wac: Mail/multipart.lock
 | /home/oliver/bin/calendar.sh
 :0 Wac: Mail/multipart.lock
 | rm -f ~/Mail/.munpack/*
}

I discovered the second bug when synchronizing a particular iCalender format file containing several BEGIN:VEVENT blocks. Events are expected to have a UID field. I do not know whether this should be unique within the scope of your entire calendar, but it must be unique within the scope BEGIN:VCALENDAR block (i.e. the entire iCalender file.)

Exchange, in all its infinite stupidity decides to assign the same UID to every VEVENT, and as soon as you attempt to upload it to Google Calender via WebDav you will receive a cryptic error message:
500 Server error code: ........ (Pseudo-random characters)

You'll receive an even less helpful error message when trying the "Import Calendar" feature of the web interface:
Failed to import events: Could not upload your events because you do not have sufficient access on the target calendar.

What is the solution? Well, funnily enough a Unique Identifier (UID) should actually be unique! Changing those UIDs to some unique values and uploading the file again works perfectly! (Anything works, I used echo $RANDOM | md5sum | cut -d ' ' -f 1 i.e. MD5 hash of a pseudo-random number.)

I am only aware of one such formatted calendar entry in my dataset, so I adjusted it manually. However, I'd feel safer using sed or awk to replace each "UID" with a real unique identifier. I'll post such a modified calendar.sh in the second part, whenever I feel like playing with awk again! You could always beat me to it... Happy hacking!

I spent many hours scouring Google before finally figuring out how to implement this easily. Calendar synchronization is very important to me; I want to be able to look at my phone, my tablet, or my browser and see my entire schedule (otherwise I'd never show up anywhere.)

.procmailrc:

:0 Wc: Mail/multipart.lock
* H ?? ^Content-Type: multipart/(alternative|mixed)
* B ?? ^Content-Type: text/calendar
{
        :0 Wac:
        | munpack -q -t -C ~/Mail/.munpack 2> /dev/null
        :0 Wac:
        | /home/oliver/bin/calendar.sh
        :0 Wc:
        | rm -f ~/Mail/.munpack/*
}

This beautiful undocumented recipe is actually quite simple, but perhaps needs some explanation. From man procmailrc:

A line starting with ':' marks the beginning of a recipe.
It has the following format:

       :0 [flags] [ : [locallockfile] ]

       zero or more conditions (one per line)
       exactly one action line

W means wait until the program finishes and ignore any failure messages. a means the preceding recipe must have successfully completed before procmail will execute this recipe. Finally, c means carbon-copy (this recipe does not handle delivery of the mail, so it must proceed further through the chain of recipes.)

It may be more appropriation to use f (this pipe is a filter) instead. I am new to procmail configuration; c works for me.

First match messages containing a Content-Type header of either multipart/alternative or multipart/mixed, second check the message body for a Content-Type of text/calendar (which is the data we want.) If these conditions are true unpack the MIME multipart message into separate files with munpack into a temporary directory. Proceed and run calendar.sh which does the magic, and finally clean up after ourselves.

calendar.sh:

#!/bin/bash

http_proxy=http://proxy.example.com:8080

CALENDAR_ID='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX@group.calendar.google.com'
for i in ~/Mail/.munpack/*; do
        if [ -n "$(grep 'BEGIN:VCALENDAR' ${i} 2> /dev/null)" ]; then
                expect -dc \
                "spawn /usr/bin/cadaver -p ${http_proxy/http:\/\/} \
                https://www.google.com/calendar/dav/${CALENDAR_ID}/events ; \
                expect \"dav:\" ; \
                send \"put ${i}\\r\" ; \
                expect \"dav:\" ;
                send \"bye\\r\""
        fi
done

Newlines have been inserted into the script so that it doesn't break layouts on other pages; you will need to fix those if copying this script.

I use expect and cadaver to upload the iCalender file extracted by munpack to Google Calendar via their WebDav interface. Finding the CALENDAR_ID is a little confusing. You can find it from the "Settings" option when you login to the Google Calender via a browser. You must use the public Calendar Address, not the Private Address, but you do not need to share the calender publicly; Google will request your login details via WebDav. This is easiest to configure in .netrc. chmod 600 the file for some additional safety.

machine www.google.com
login john.doe
password example123

The login should not include the @gmail.com otherwise it will fail. Happy hacking!

Swift is the software behind the OpenStack Object Storage service.

This service provides a simple storage service for applications using RESTful interfaces, providing maximum data availability and storage capacity.

I explain here how some parts of the storage and replication in Swift works, and show some of its current limitations.

If you don't know Swift and want to read a more "shallow" overview first, you can read John Dickinson's Swift Tech Overview.

How Swift storage works

If we refer to the CAP theorem, Swift chose availability and partition tolerance and dropped consistency. That means that you'll always get your data, they will be dispersed on many places, but you could get an old version of them (or no data at all) in some odd cases (like some server overload or failure). This compromise is made to allow maximum availability and scalability of the storage platform.

But there are mechanisms built into Swift to minimize the potential data inconsistency window: they are responsible for data replication and consistency.

The official Swift documentation explains the internal storage in a certain way, but I'm going to write my own explanation here about this.

Consistent hashing

Swift uses the principle of consistent hashing. It builds what it calls a ring. A ring represents the space of all possible computed hash values divided in equivalent parts. Each part of this space is called a partition.

The following schema (stolen from the Riak project) shows the principle nicely:

In a simple world, if you wanted to store some objects and distribute them on 4 nodes, you would split your hash space in 4. You would have 4 partitions, and computing hash(object) modulo 4 would tell you where to store your object: on node 0, 1, 2 or 3.

But since you want to be able to extend your storage cluster to more nodes without breaking the whole hash mapping and moving everything around, you need to build a lot more partitions. Let's say we're going to build 2¹⁰ partitions. Since we have 4 nodes, each node will have 210 ÷ 4 = 256 partitions. If we ever want to add a 5^th node, it's easy: we just have to re-balance the partitions and move 1⁄4 of the partitions from each node to this 5^th node. That means all our nodes will end up with 210 ÷ 5 ≈ 204 partitions. We can also define a weight for each node, in order for some nodes to get more partitions than others.

With 2¹⁰ partitions, we can have up to 2¹⁰ nodes in our cluster. Yeepee.

For reference, Gregory Holt, one of the Swift authors, also wrote an explanation post about the ring.

Concretely, when building one Swift ring, you'll have to say how much partitions you want, and this is what this value is really about.

Data duplication

Now, to assure availability and partitioning (as seen in the CAP theorem) we also want to store replicas of our objects. By default, Swift stores 3 copies of every objects, but that's configurable.

In that case, we need to store each partition defined above not only on 1 node, but on 2 others. So Swift adds another concept: zones. A zone is an isolated space that does not depends on other zone, so in case of an outage on a zone, the other zones are still available. Concretely, a zone is likely to be a disk, a server, or a whole cabinet, depending on the size of your cluster. It's up to you to chose anyway.

Consequently, each partitions has not to be mapped to 1 host only anymore, but to N hosts. Each node will therefore store this number of partitions:

number of partition stored on one node = number of replicas × total number of partitions ÷ number of node

Examples:

We split the ring in 2¹⁰ = 1024 partitions. We have 3 nodes. We want 3 replicas of data.
→ Each node will store a copy of the full partition space: 3 × 210 ÷ 3 = 210 = 1024 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 5 nodes. We want 3 replicas of data.
→ Each node will store 211 × 3 ÷ 5 ≈ 1129 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 6 nodes. We want 3 replicas of data.
→ Each node will store 211 × 3 ÷ 6 = 1024 partitions.

Three rings to rule them all

In Swift, there is 3 categories of thing to store: account, container and objects.

An account is what you'd expect it to be, a user account. An account contains containers (the equivalent of Amazon S3's buckets). Each container can contains user-defined key and values (just like a hash table or a dictionary): values are what Swift call objects.

Swift wants you to build 3 different and independent rings to store its 3 kind of things (accounts, containers and objects).

Internally, the two first categories are stored as SQLite databases, whereas the last one is stored using regular files.

Note that this 3 rings can be stored and managed on 3 completely different set of servers.

Data replication

Now that we have our storage theory in place (accounts, containers and objects distributed into partitions, themselves stored into multiple zones), let's go the replication practice.

When you put something in one of the 3 rings (being an account, a container or an object) it is uploaded into all the zones responsible for the ring partition the object belongs to. This upload into the different zones is the responsibility of the swift-proxy daemon.

But if one of the zone is failing, you can't upload all your copies in all zones at the upload time. So you need a mechanism to be sure the failing zone will catch up to a correct state at some point.

That's the role of the swift-{container,account,object}-replicator processes. This processes are running on each node part of a zone and replicates their contents to nodes of the other zones.

When they run, they walk through all the contents from all the partitions on the whole file system and for each partition, issue a special REPLICATE HTTP request to all the other zones responsible for that same partition. The other zone responds with information about the local state of the partition. That allows the replicator process to decide if the remote zone has an up-to-date version of the partition.

In case of account and containers, it doesn't check at the partition level, but check each account/container contained inside each partition.

If something is not up-to-date, it will be pushed using rsync by the replicator process. This is why you'll read that the replication updates are "push based" in Swift documentation.

# Pseudo code describing replication process for accounts
# The principle is exactly the same for containers
for account in accounts:
    # Determine the partition used to store this account
    partition = hash(account) % number_of_partitions
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-account-server process
        version_of_account = zone.send_HTTP_REPLICATE_for(account):
        if version_of_account < account.version()
            account.sync_to(zone)

This replication process is O(number of account × number of replicas). The more your number of account will increase and the more you will want replicas for your data, the more the replication time for your accounts will grow. The same rule applies for containers.

# Pseudo code describing replication process for objects
for partition in partitions_storing_objects:
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-object-server process
        verion_of_partition = zone.send_HTTP_REPLICATE_for(partition):
        if version_of_partition < partition.version()
            # Use rsync to synchronize the whole partition
            # and all its objects
            partition.rsync_to(zone)

This replication process is O(number of objects partitions × number of replicas). The more your number of objects partitions will increase, and the more you will want replicas for your data, the more the replication time for your objects will grow.

I think this is something important to know when deciding how to build your Swift architecture. Choose the right number the number of replicas, partitions and nodes.

Replication process bottlenecks

File accesses

The problem, as you might have guessed, is that to replicate, it walks through every damn things, things being accounts, containers, or object's partition hash files. This means it need to open and read (part of) a every file your node stores to check that data need or not to be replicated!

For accounts & containers replication, this is done every 30 seconds by default, but it will likely take more than 30 seconds as soon as you hit around 12 000 containers on a node (see measurements below). Therefore you'll end up checking consistency of accounts & containers on each all node all the time, using obviously a lot of CPU time.

For reference, Alex Yang also did an analysis of that same problem.

TCP connections

Worst, the HTTP connections used to send the REPLICATE commands are not pooled: a new TCP connection is established each time something has to be checked against the same thing stored on a remote zone.

This is why you'll see in the Swift's Deployment Guide this lines listed under "general system tuning":

# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1

# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144

In my humble opinion, this is more an ugly hack than a tuning. If you don't activate this and if you have a lot of containers on your node, you'll end up soon with thousands of connections in TIME_WAIT state, and you indeed risk to overload the IP conntrack module.

Container deletion

We also should talk about container deletion. When a user deletes a container from its account, the container is marked as deleted. And that's it. It's not deleted. Therefore the SQLite database file representing the container will continue to be checked for synchronization, over and over.

The only way to have a container permanently deleted is to mark an account as deleted. This way the swift-account-reaper will delete all its containers and, finally, the account.

Measurement

On a pretty big server, I measured the replications to be done at a speed of around 350 {account,container,object-partitions}/second, which can be a real problem if you chose to build a lots of partition and you have a low number_of_node ⁄ number_of_replicas ratio.

For example, the default parameters runs the container replication every 30 seconds. To check replication status of 12 000 containers stored on one node at the speed of 350 containers/seconds, you'll need around 34 seconds to do so. In the end, you'll never stop checking replication of your containers, and the more you'll have containers, the more your inconsistency window will increase.

Conclusion

Until some of the code is fixed (the HTTP connection pooling probably being the "easiest" one), I warmly recommend to chose correctly the different Swift parameters for your setup. The replication process optimization consists in having the minimum amount of partitions per node, which can be done by:

• decreasing the number of partitions
• decreasing the number of replicas
• increasing the number of node

For very large setups, some code to speed up accounts and containers synchronization, and remove deleted containers will be required, but this does not exist yet, as far as I know.

The opinions expressed here represent my own and not those of my employer.

It has been way too long since my last status update on systemd. Here's another short, incomprehensive status update on what we worked on for systemd since then.

We have been working hard to turn systemd into the most viable set of components to build operating systems, appliances and devices from, and make it the best choice for servers, for desktops and for embedded environments alike. I think we have a really convincing set of features now, but we are actively working on making it even better.

Here's a list of some more and some less interesting features, in no particular order:

1. We added an automatic pager to systemctl (and related tools), similar to how git has it.
2. systemctl learnt a new switch --failed, to show only failed services.
3. You may now start services immediately, overrding all dependency logic by passing --ignore-dependencies to systemctl. This is mostly a debugging tool and nothing people should use in real life.
4. Sending SIGKILL as final part of the implicit shutdown logic of services is now optional and may be configured with the SendSIGKILL= option individually for each service.
5. We split off the Vala/Gtk tools into its own project systemd-ui.
6. systemd-tmpfiles learnt file globbing and creating FIFO special files as well as character and block device nodes, and symlinks. It also is capable of relabelling certain directories at boot now (in the SELinux sense).
7. Immediately before shuttding dow we will now invoke all binaries found in /lib/systemd/system-shutdown/, which is useful for debugging late shutdown.
8. You may now globally control where STDOUT/STDERR of services goes (unless individual service configuration overrides it).
9. There's a new ConditionVirtualization= option, that makes systemd skip a specific service if a certain virtualization technology is found or not found. Similar, we now have a new option to detect whether a certain security technology (such as SELinux) is available, called ConditionSecurity=. There's also ConditionCapability= to check whether a certain process capability is in the capability bounding set of the system. There's also a new ConditionFileIsExecutable=, ConditionPathIsMountPoint=, ConditionPathIsReadWrite=, ConditionPathIsSymbolicLink=.
10. The file system condition directives now support globbing.
11. Service conditions may now be "triggering" and "mandatory", meaning that they can be a necessary requirement to hold for a service to start, or simply one trigger among many.
12. At boot time we now print warnings if: /usr is on a split-off partition but not already mounted by an initrd; if /etc/mtab is not a symlink to /proc/mounts; CONFIG_CGROUPS is not enabled in the kernel. We'll also expose this as tainted flag on the bus.
13. You may now boot the same OS image on a bare metal machine and in Linux namespace containers and will get a clean boot in both cases. This is more complicated than it sounds since device management with udev or write access to /sys, /proc/sys or things like /dev/kmsg is not available in a container. This makes systemd a first-class choice for managing thin container setups. This is all tested with systemd's own systemd-nspawn tool but should work fine in LXC setups, too. Basically this means that you do not have to adjust your OS manually to make it work in a container environment, but will just work out of the box. It also makes it easier to convert real systems into containers.
14. We now automatically spawn gettys on HVC ttys when booting in VMs.
15. We introduced /etc/machine-id as a generalization of D-Bus machine ID logic. See this blog story for more information. On stateless/read-only systems the machine ID is initialized randomly at boot. In virtualized environments it may be passed in from the machine manager (with qemu's -uuid switch, or via the container interface).
16. All of the systemd-specific /etc/fstab mount options are now in the x-systemd-xyz format.
17. To make it easy to find non-converted services we will now implicitly prefix all LSB and SysV init script descriptions with the strings "LSB:" resp. "SYSV:".
18. We introduced /run and made it a hard dependency of systemd. This directory is now widely accepted and implemented on all relevant Linux distributions.
19. systemctl can now execute all its operations remotely too (-H switch).
20. We now ship systemd-nspawn, a really powerful tool that can be used to start containers for debugging, building and testing, much like chroot(1). It is useful to just get a shell inside a build tree, but is good enough to boot up a full system in it, too.
21. If we query the user for a hard disk password at boot he may hit TAB to hide the asterisks we normally show for each key that is entered, for extra paranoia.
22. We don't enable udev-settle.service anymore, which is only required for certain legacy software that still hasn't been updated to follow devices coming and going cleanly.
23. We now include a tool that can plot boot speed graphs, similar to bootchartd, called systemd-analyze.
24. At boot, we now initialize the kernel's binfmt_misc logic with the data from /etc/binfmt.d.
25. systemctl now recognizes if it is run in a chroot() environment and will work accordingly (i.e. apply changes to the tree it is run in, instead of talking to the actual PID 1 for this). It also has a new --root= switch to work on an OS tree from outside of it.
26. There's a new unit dependency type OnFailureIsolate= that allows entering a different target whenever a certain unit fails. For example, this is interesting to enter emergency mode if file system checks of crucial file systems failed.
27. Socket units may now listen on Netlink sockets, special files from /proc and POSIX message queues, too.
28. There's a new IgnoreOnIsolate= flag which may be used to ensure certain units are left untouched by isolation requests. There's a new IgnoreOnSnapshot= flag which may be used to exclude certain units from snapshot units when they are created.
29. There's now small mechanism services for changing the local hostname and other host meta data, changing the system locale and console settings and the system clock.
30. We now limit the capability bounding set for a number of our internal services by default.
31. Plymouth may now be disabled globally with plymouth.enable=0 on the kernel command line.
32. We now disallocate VTs when a getty finished running (and optionally other tools run on VTs). This adds extra security since it clears up the scrollback buffer so that subsequent users cannot get access to a user's session output.
33. In socket units there are now options to control the IP_TRANSPARENT, SO_BROADCAST, SO_PASSCRED, SO_PASSSEC socket options.
34. The receive and send buffers of socket units may now be set larger than the default system settings if needed by using SO_{RCV,SND}BUFFORCE.
35. We now set the hardware timezone as one of the first things in PID 1, in order to avoid time jumps during normal userspace operation, and to guarantee sensible times on all generated logs. We also no longer save the system clock to the RTC on shutdown, assuming that this is done by the clock control tool when the user modifies the time, or automatically by the kernel if NTP is enabled.
36. The SELinux directory got moved from /selinux to /sys/fs/selinux.
37. We added a small service systemd-logind that keeps tracks of logged in users and their sessions. It creates control groups for them, implements the XDG_RUNTIME_DIR specification for them, maintains seats and device node ACLs and implements shutdown/idle inhibiting for clients. It auto-spawns gettys on all local VTs when the user switches to them (instead of starting six of them unconditionally), thus reducing the resource foot print by default. It has a D-Bus interface as well as a simple synchronous library interface. This mechanism obsoletes ConsoleKit which is now deprecated and should no longer be used.
38. There's now full, automatic multi-seat support, and this is enabled in GNOME 3.4. Just by pluging in new seat hardware you get a new login screen on your seat's screen.
39. There is now an option ControlGroupModify= to allow services to change the properties of their control groups dynamically, and one to make control groups persistent in the tree (ControlGroupPersistent=) so that they can be created and maintained by external tools.
40. We now jump back into the initrd in shutdown, so that it can detach the root file system and the storage devices backing it. This allows (for the first time!) to reliably undo complex storage setups on shutdown and leave them in a clean state.
41. systemctl now supports presets, a way for distributions and administrators to define their own policies on whether services should be enabled or disabled by default on package installation.
42. systemctl now has high-level verbs for masking/unmasking units. There's also a new command (systemctl list-unit-files) for determining the list of all installed unit file files and whether they are enabled or not.
43. We now apply sysctl variables to each new network device, as it appears. This makes /etc/sysctl.d compatible with hot-plug network devices.
44. There's limited profiling for SELinux start-up perfomance built into PID 1.
45. There's a new switch PrivateNetwork= to turn of any network access for a specific service.
46. Service units may now include configuration for control group parameters. A few (such as MemoryLimit=) are exposed with high-level options, and all others are available via the generic ControlGroupAttribute= setting.
47. There's now the option to mount certain cgroup controllers jointly at boot. We do this now for cpu and cpuacct by default.
48. We added the journal and turned it on by default.
49. All service output is now written to the Journal by default, regardless whether it is sent via syslog or simply written to stdout/stderr. Both message streams end up in the same location and are interleaved the way they should. All log messages even from the kernel and from early boot end up in the journal. Now, no service output gets unnoticed and is saved and indexed at the same location.
50. systemctl status will now show the last 10 log lines for each service, directly from the journal.
51. We now show the progress of fsck at boot on the console, again. We also show the much loved colorful [ OK ] status messages at boot again, as known from most SysV implementations.
52. We merged udev into systemd.
53. We implemented and documented interfaces to container managers and initrds for passing execution data to systemd. We also implemented and documented an interface for storage daemons that are required to back the root file system.
54. There are two new options in service files to propagate reload requests between several units.
55. systemd-cgls won't show kernel threads by default anymore, or show empty control groups.
56. We added a new tool systemd-cgtop that shows resource usage of whole services in a top(1) like fasion.
57. systemd may now supervise services in watchdog style. If enabled for a service the daemon daemon has to ping PID 1 in regular intervals or is otherwise considered failed (which might then result in restarting it, or even rebooting the machine, as configured). Also, PID 1 is capable of pinging a hardware watchdog. Putting this together, the hardware watchdogs PID 1 and PID 1 then watchdogs specific services. This is highly useful for high-availability servers as well as embedded machines. Since watchdog hardware is noawadays built into all modern chipsets (including desktop chipsets), this should hopefully help to make this a more widely used functionality.
58. We added support for a new kernel command line option systemd.setenv= to set an environment variable system-wide.
59. By default services which are started by systemd will have SIGPIPE set to ignored. The Unix SIGPIPE logic is used to reliably implement shell pipelines and when left enabled in services is usually just a source of bugs and problems.
60. You may now configure the rate limiting that is applied to restarts of specific services. Previously the rate limiting parameters were hard-coded (similar to SysV).
61. There's now support for loading the IMA integrity policy into the kernel early in PID 1, similar to how we already did it with the SELinux policy.
62. There's now an official API to schedule and query scheduled shutdowns.
63. We changed the license from GPL2+ to LGPL2.1+.
64. We made systemd-detect-virt an official tool in the tool set. Since we already had code to detect certain VM and container environments we now added an official tool for administrators to make use of in shell scripts and suchlike.
65. We documented numerous interfaces systemd introduced.

Much of the stuff above is already available in Fedora 15 and 16, or will be made available in the upcoming Fedora 17.

And that's it for now. There's a lot of other stuff in the git commits, but most of it is smaller and I will it thus spare you.

I'd like to thank everybody who contributed to systemd over the past years.

Thanks for your interest!

I finally decided to configure my computers for UTF-8, mostly due to the frustration of attempting to determine which Finnish character should have been displayed (and the probable annoyance of Finns having their names mangled by my mail client.)

• /etc/locale.gen is useful for generating only the locales you actually need. I selected English (US and UK), Finnish, and Swedish as UTF-8, ISO-8859-1, and ISO-8859-15 (where appropriate.)
• .bashrc: export LANG=en_US.UTF-8
• Setup .Xdefaults:

XTerm*Background: Black
XTerm*Foreground: White
XTerm*UTF8: 2
XTerm*utf8Fonts: 2

Actually the UTF-8 options are not required due to the way I am launching terminals from my window manager, but it does not cause any damage. Black terminal background is mandatory!

The fun part was figuring out why my xterm, specified in .ratpoisonrc as bind c exec xterm -u8 did not launch a UTF-8 terminal. I then realized that Ratpoison was launching xterm from a bare-metal shell and the LANG environment variable was never set; quickly changing the binding to bind c exec xterm +lc -u8 (switch off automatic selection of encoding and respect my -u8 option) resolved the final issue.

Now I can happily read the ä's, ö's, and Å's from a language I do not understand (excluding a few keywords and phrases.) Cool!

LED Fireworks. Or, parent helpers go overboard

My friend, Mike Ward, and I help out at our kids school by teaching 5th and 6th graders programming in conjunction with a Lego class. For several years, I was teaching Logo on ancient Apple II machines. The teacher had a great curriculum and the kids learned a lot, but this year we decided to switch things around and try them out on Arduinos.

This spring, the kids are building a Lego display centered around a birthday party theme for the star elephant at the Oregon Zoo (Packy), who turned 50 last weekend. In a moment of fun, I suggested that perhaps we should have some fireworks, along with the presents and birthday cake. How hard could it be to wire up a couple of LEDs to an Arduino to make something that looked like fireworks?

Selecting parts.

Mike and I sat down to order some parts from Digikey. We found these nice, cheap, LED drivers from Sharp. Then we found a selection of LEDs from Cree along with current-programming resistors (12.7kΩ) to drive the LEDs at 20mA, and some bypass caps (0.1μF) to keep the parts happy.

The LED drivers are simple shift registers, with separate clock and data. They have both input and output pins, so you can daisy chain them and drive as many as you like from a single pair of CPU pins. We figured 160 LEDs would be plenty, so we ordered 10 chips.

The LED drivers came in a through-hole, DIP package; it seemed like we’d be able to just stuff them into a breadboard and quickly wire things up.

Prototyping the design

The LED driver chips turn out to use a much finer pitch package than the usual 0.1” spacing. There’s no way we could stuff them into a breadboard and quickly wire things up. Obviously, a bit more care while ordering would have uncovered this, but my experience with through-hole parts is limited to parts from the mid 1970s…

I soldered extension wires onto one chip to test the circuit, and managed to get one chip talking to an Arduino. Here’s a short movie of that in action:

You can see the LED driver chip sitting on a set of extension leads plugged into the breadboard. It’s hooked to the SPI port on the Arduino board, with the clock set as fast as it will go. If you watch the video, you can see that the LEDs actually vary in brightness. They’re actually being pulse-width modulated. Clocked at 2MHz, there’s enough bandwidth in the SPI bus to drive 160 LEDs with 127 discrete pulse widths at nearly 100Hz (100 * 127 * 160 = 2032000).

Is bread boarding practical?

With the chip propped up on its extension wires, and the LEDs plugged in, the whole setup was not surprisingly fragile. Getting five of these chips wired up and running for the length of the Lego show (one day at school and two days at the zoo) seemed unlikely. So, we decided to build some custom circuit boards.

Using the GEDA tools to make circuit boards

My business partner, Bdale Garbee, and I build rocketry electronics through Altus Metrum. He’s the hardware guy and I spend my weekends writing firmware for various micro-controllers. Of course, I use free software tools for the firmware, and Bdale uses the free gEDA tools for the hardware. I’ve gotten vaguely familiar with those by helping Bdale review circuit diagrams and PC board layout over the years. So I figured I’d be able to learn a bit more about them and come up with some designs.

I decided to stick two LED drivers on each board. As the chips can drive each LED with up to 50mA of current, each board could potentially consume 1.6A at 5V. Sticking a linear regulator on each board capable of supplying that much current would have required a hefty heat sink. Instead, I decided to stick a mini USB connector on the board and use some cheap USB power supplies.

Of course, there wasn’t a pre-packaged schematic symbol or footprint for the LED driver, so I first had to construct those. But, with symbols for all of the parts I used available, I drew this circuit:

Wiring up 32 LEDs

It’s really easy to draw wires showing 32 LEDs hooked up to each board. Actually selecting connectors that can be easily hooked up took a bit more time. The idea of crimping individual two-pin connectors for 160 LEDs didn’t seem like fun, plus individual connectors were going to cost a small fortune (the best option I found would have cost about $0.40 per LED). Bdale suggested using ribbon cable with a crimp-on connector — crimp a connector across the whole cable and then split out two-wire sets for each LED. I’d still have to individually solder each LED to the wire, but on the board end, all I’d have to do is crimp the connector on to the cable.

Designing the circuit boards

The gEDA tools separate schematic design and PCB layout into two separate tools; the schematic program, gschem, exports a netlist and set of components which the schematic program, PCB, can import. This sounded a bit clunky to me, as you have to re-export and re-import the schematic data into the PCB each time you make changes, but it seems to work reasonably well in practice.

The first time the schematic data is imported to the PCB tool, all of the component footprints are simply stacked on top of one another. With those moved to sensible locations, the tool will show ‘rats’ for each netlist.

Having drawn the schematic with symbols that strongly resembled the actual components, it was pretty easy to work through the rats one at a time by painting copper on the board. I’d noticed that all of the signals could easily be routed on one side of the board, so I flooded the back side with a ground plane to keep things quiet.

The final circuit board design looks like this:

Circuit boards.

All Altus Metrum products are manufactured by Advanced Circuits, and we’ve been quite happy with them. They offer a cheap and fast prototyping service through their barebonespcb.com site. These boards are two layers with no silk screen or solder mask.

I uploaded my design and in a couple of days I got back some shiny circuit boards:

The footprint I designed for the LED drivers turned out to have a very small gap between the pads for each pin and the ground plane. This lead to a couple of shorted pins in the five boards, which were pretty easy to diagnose as the LED driven by that pin would be stuck on.

Here’s another picture with the ribbon cable connected and an LED lit up:

Mounting the LEDs

I started by writing a small nickle program to simulate the trajectory of a ballistic fireworks shell followed by an explosion at apogee and subsequent ballistic tracks for the resulting pieces. I made the initial conditions of the launch somewhat random, and Mike and I sat and watched that draw things until we saw a couple that looked ‘good’. The output from that was a series of X/Y locations spaced uniformly in time. Mike used those to drill holes in a piece of acrylic from TAP plastics. He glued the LEDs into that with some white glue (so that we have a chance of getting them back out):

A custom maple box?

I spent last weekend visiting my father for his birthday. Meanwhile, Mike was busy back home building a case for the project. I got home and found that he had fabricated a solid maple case. It’s beautiful, and holds all of the LEDs, boards, wires and a couple of 4-outlet USB power supplies.

Here’s a picture of the joint he made in the box corners:

The finished product

Here’s the inside of the box:

And here’s a movie of the whole thing in action. It’s missing 8 white LEDs; I hadn’t ordered enough from Digikey.

Downloading schematics and PCB artwork

The hardware design is licensed under the TAPR Open Hardware License. It’s all stored in git (of course):

git://keithp.com/git/hw/fireworks

All of the symbols I created are also available through git:

git://keithp.com/git/hw/keithp

Things I’d do differently

One of the frustrating things about moving atoms around instead of just bits is that once you’ve got a physical artifact, it’s really expensive to change it. There are a couple of minor things I would change if I were going to build more of these:

• Add mounting holes. Not having any way to mount the boards means that they’re flopping around inside the cabinet. We’ll probably find a way to tie them down somehow, but a couple of holes in the boards would have made that really easy.

• Increase the gap between the pin pads and the ground plane. The narrow space provided ample opportunity for solder bridges. I’ve increased that spacing in the git repository version of the boards.

• Used a variable resistor for LED current control. With a fixed value, it’s very difficult to adjust the relative brightness of the LEDs. Even though we used LEDs from the same series, the different colors have different apparent brightness.

• Used surface mount passives. We had bought through-hole parts when the plan was to build these on a breadboard, but soldering through-hole parts onto the board is more work than using surface mount parts. 0805 (or even 1206) parts are plenty large to solder by hand and are faster to deal with than having to fold leads, insert them into holes, flip the boards over, solder them down and trim the leads.

• Perhaps we went a bit overboard on this project…

Things that worked well

There were a couple of things that went better than I had expected:

• The gEDA tools aren’t that hard to use. Constructing the circuit board really was a simple matter of painting with copper. If you can drive inkscape, you can make circuit boards.

• Advanced Circuits bare-bones PCB service. It was really cool to upload the output files from gEDA and have actual boards arrive in the mail a couple of days later.

• Building the boards. I really didn’t expect all of the components to ‘just fit’ in the boards; I figured I’d mess up the footprints or drill sizes and end up kludging something onto the boards. Bdale did catch a small drill size on the LED driver pins, which I bumped up a bit, but other than that, all of the parts slipped in and soldered easily. Each board took about 10 minutes to put together.

[update 20 Apr: clarify middle finger emulation]
One of the features added to the upcoming xf86-input-synaptics version 1.6 is support for clickpad-style devices. This post outlines what clickpads are, how they are supported and how you go about enabling the new features.

What are clickpads

The name ClickPad comes from the Synaptics product of the same name. It describes a touchpad that does not have physical buttons. Instead, the whole touchpad works as a button. Devices like this have been around for a while, most notably the touchpads found in Apple laptops and the Dell Mini 10 series. The challenges for us were to handle the data streams correctly. Most of this work was done by Chase Douglas.

Clickpads give us the position for each finger, and a button 1 event when the pad is pressed. The design of the hardware however means that whenever you click the pad, you always add one more finger to the mix. Decoupling that finger from the ones that actually matter is the challenge. And integrating it with all the other whacky features that the synaptics driver currently has.

Clickpad Support

Clickpad support requires server 1.12 [0]. It heavily relies on multitouch support.
Central to the new feature is the "ClickPad" option and property. It is enabled if the kernel sets the INPUT_PROP_BUTTONPAD property on the device, otherwise you can enable it in an xorg.conf.d snippet or at runtime with either xinput or synclient:

xinput set-prop "device name" "Synaptics ClickPad" 1
synclient ClickPad=1

This toggles a few driver behaviours to make the clickpad much more usable. Most notably, you can use one finger to press the button and another finger to move the cursor around.

Word of warning here: if you enable clickpad support manually at runtime, you will also have to manually disable traditional middle mouse button emulation (synclient EmulateMidButtonTime=0). For autoconfigured devices or xorg.conf.d configured devices this is done automatically.

The second big feature for clickpads is support for software buttons. Since the device only gives us left button clicks, we expose an option to allow for right and middle button clicks. By default, we ship this xorg.conf.d snippet:

Section "InputClass"
        Identifier "Default clickpad buttons"
        MatchDriver "synaptics"
        Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
EndSection

The order of soft button edges is left, right, top, bottom for the right button, then left, right, top, bottom for the middle button. So the above snippet sets the right half of the bottom 18% to be a right button, with no middle button present. A coordinate of 0 means "stretch to edge of touchpad". [1]

Traditional middle mouse button emulation is disabled by default and we don't handle it for clickpads. Traditional middle mouse button emulation refers to the method of generating a button 2 press when both button 1 and button 3 are pressed simultaneously. Mostly because, well, I'm not sure how you would even trigger middle mouse button emulation for clickpads given that all buttons except the first are already emulated anyway. You can still emulate middle mouse button emulations through clickfingers (see below), tapping, or the soft button areas as described above.

Tapping works as on any other touchpad.

ClickFinger functionality works, albeit slightly different. In the default ClickFinger setting, if you have 2 fingers on the touchpad and you press the left button, you will see a ClickFinger2 action performed (right button by default). On clickpads we guess the number of fingers by proximity (since you need one finger to actually press the button). Fingers closer together than 30% of the touchpad's size [2] will count towards ClickFinger actions, others will be skipped. So in the default use-case, where you have two fingers down on the touchpad in the middle and you use a third to click the button, you will still get a ClickFinger2 action. Likewise, if you press with two fingers down, you will also get a ClickFinger2 action.

All other functions are not affected by clickpad support and behave as before.

• [0] Ubuntu 12.04 will ship a 1.11/1.12 mix, that will work as well
• [1] "edge of touchpads" should be 100% but it isn't due to a years-old mishandling of the coordinate ranges. So 0 is better to use here.
• [2] All touchpads lie about their size, so we just hardcode 30% since anything we infer is going to be wrong anyways

Off-and-on for a couple years (no kidding) I've been working on a branch that compiles the assembly shaders (e.g., GL_ARB_vertex_program) to GLSL IR. I even talked about it at XDC 2011. This project has turned into such an incredible rat's nest of irritation that I can't even believe it.

All of that asside, the project reached a significant milestone today. I can run retail Doom3 binaries. There are a couple caveats (incorrect rendering, have to disable HiZ on Sandybridge, etc.), but it's still significant progress. The images below are i965 on Mesa 8.0.2 (with HiZ disabled), i965 on my branch (with HiZ disabled), and classic swrast on my branch. At least at the time of this writing, there are some bugs related to separate stencil on Sandybridge that only occur when HiZ is enabled.

Today I release a Python client library to query Munin servers.

I wrote it as part of some experiments I did a few weeks ago. I discovered there was no client library to query a Munin server. There's PyMunin or python-munin which help developing Munin plugins, but nothing to access the munin-node and retrieve its data.

So I decided to write a quick and simple one, and it's released under the name of PyMuninCli, providing the munin.client Python module.

Fontconfig 2.9.0 uploaded to Debian Experimental

Thanks to Akira Tagoh for taking over maintenance of fontconfig. As a result of this switch, there’s a new release, 2.9.0.

Along with this new upstream release, I’ve responded to Debian bug 651493 and removed all defoma support from the fontconfig packages. Installing the new version should remove all traces of the old defoma support as well.

I’d love to hear from people with success (or not) in testing this version of the package; 2.9.0 should be uploaded to unstable in time for the next freeze.

[updated Apr 16 2012 for smooth scrolling]
The latest version of Mac OSX (Lion) introduced a feature referred to natural scrolling. Since I've been getting a few requests to add the same feature to the synaptics driver. At it's base natural scrolling is an inversion of the scroll direction*. We've had the ability to do this for years, though more by accident than intent.
The synaptics driver (until X server 1.12's smooth scrolling) uses buttons 4/5/6/7 for up/down/left/right scrolling. If you want up to scroll down, just swap them and you're good to go.

xinput set-button-map <device name> 1 2 3 5 4

Now down is up and up is down. The device name can be obtained as usual with xinput list. This will work for servers up to including 1.11.
As of synaptics 1.6RC3 (1.5.99.903), the options VertScrollDelta and HorizScrollDelta can now be set to a negative value. For servers supporting smooth scrolling (X server 1.12+), this effectively inverts the scroll direction.
* I realise that natural scrolling in OSX may have a few other details to make it do whatever it does. Synaptics supports scroll inversion and scrolling inertia.

As Starks used to say, “Winter is coming”. So in this sense, time has come for another round of updates from the Intel Linux Graphics project.

This time, it took me less time than on previous round of updates, so I think I’ll be able to give you some more details about some of the most interesting changes I’ve spotted for the past weeks.

Starting with the Kernel, many and many patches has landed into the intel-gfx mailing list. Among some of the most interesting were:

• Another round of Valley View enabling patches from Jesse Barnes. Some of both Valley View and Haswell architecture patches were already included into the drm-intel-next branch by Daniel Vetter, but obviously most of the things are still being worked on. Valley View patches are getting really close to their final form, but some of the changes come from the fact that its architecture handles most of the display-related operations in a very orthogonal way to the other GPUs we have. So we’ll still see more rounds of patchbombing on ValleyView front.
• Daniel Vetter sent another round of patches for PPGTT handling, most of which were already included in his drm-intel-next tree by now. Besides those, Daniel came with another round of Sandy Bridge hardware-related workarounds for the kernel as well. None of those were (apparently) hit by our usual code paths in Kernel, but in order to prevent possible future problems it is better to be safe than sorry on this matter.
• Jesse also came with a patch series which reworked and simplified some shared code paths in the driver, mostly related to the Display Port support and interaction with the PCH-based items (e.g., parts of the display that are based on the chipset and not on the CPU itself).
• Chris Wilson sent some patches for properly handle some Graphics interrupts corner cases, some additional debugging patches, additional LVDS-related fixes and a large number of patches that improve the way GPU rings interact between each other.
• Adam Jackson from Redhat, Rodrigo Vivi, Paulo Zanoni and Takashi Iwai from SuSE have been working on supporting new video modes that we weren’t handling properly (such as modes that are not properly described in EDID metadata that monitors report even when those modes are actually supported by them). Those patches resulted in a series of very interesting discussions about how to handle such cases as well, including all the scary terms like CVT, DMT, EDID, GTF2, GTF, and all the other WTF-like terms and all the related stuff.
• Ben Widawsky posted some patches for proper cache-level parity handling and better RC6 visibility from userspace, by using the sysfs facilities.
• Last but not least, I’ve sent out the 3rd round of Haswell enablement patches. Phoronix already did a great coverage of them in the past weeks, so as the major highlights of this new series is support of digital outputs (like HDMI and DVI), and some additional tweaks to make things work better in overall. This time, the patchbombing was of mere 29 patches thanks to Daniel Vetter who had already included around as many patches into his drm-intel-next-queued tree last week. It was an amazingly interesting experience for me to work on this hardware enablement, and it is great to see the results of this work out in the wild. And of course, I’d like to thank Jesse for helping me with most of the hardest issues I had, and Ben for his help when I got completely stuck on some of them .
• And finally, still related to the Intel Graphics but also to all the other drm-based Linux projects, Paulo Zanoni have been working on an infoframes programming tool, which was finally included into the intel-gpu-tools suite; underscan properties support and on generic CRTC properties development.

Moving to Mesa, lots of patches on all the areas as well. The most interesting ones related to the Intel GPUs that I managed to look at were:

• Eric Anholt’s work on GLSL 1.40 support and performance tuning in shaders for Intel GPUs. The GLSL 1.40 support is a big step for the open-source GL library, and it will be a very welcomed addition for the next GL versions support. And performance-related patches are always a nice thing to have .
• Ian Romanick sent the patches that rework the initial uniform handling in Mesa.
• And Neil Roberts came with some patches that improve wayland-specific Mesa code paths.

Besides those changes, we had hundreds of patches that I left out of the scope of this update. Lots of activities are going on on many Mesa-related development, and the progress of this open-source GL implementation is looking great!

On 2D driver side, lots of patches entered the xf86-video-intel project for the past weeks, mostly related to the SNA acceleration backend, but also some glyph-related fixes and other bugfixes for UXA. All of them came from Chris Wilson, who is doing an unbelievable job on the project for the past years!

On Wayland project, as usual, lots of activity was going on. Among the most interesting developments was the addition of piglit support to the project by Chad Versace, which allows to run the piglit tests under GLX, Wayland and X11/EGL while selecting the desired window system at runtime.

And finally, on a related project, Jeremy Huddleston has released the first maintenance release for xorg-server, which fixes many input-related issues together with some RandR fixes and small memory leaks.

I guess that’s it for today. Stay tuned for future updates, and enjoy the next season of The Game of Thrones meanwhile (which works amazingly well on my Intel-based card with vaapi acceleration by the way ).

If you want to give a talk at GUADEC 2012 in A Coruña (Spain), hurry up: the deadline for the GUADEC CfP is tomorrow. Don't think twice, just go ahead and submit your talk!

This afternoon I’m giving a brief talk on the Nuanti C++/CLI compiler built on Clang at the LLVM European Conference, London.

We started this project a couple of years ago to cover the language specification in ECMA-372, having found it a pain to integrate applications and SDK written in C and C++ with .NET. Since then, the compiler has developed into a self-hosting toolchain that’s been used to port entire applications to run on Mono, Silverlight, Mac OS X and Linux.

There are two codegen modes: (1) fully managed .NET bytecode, and (2) native optimized machine code with hooks into the Mono runtime for invocations. There’s some flexibility to mix and match the two. Sometimes it’s handy to call into an existing managed library from your C++ application and this use case is simple as adding an import header and calling the methods you need. No bindings or wrappers, and if using a runtime like Mono the stack is garbage collected behind the scenes.

The frontend has all the parser support you’d expect plus CLR metadata import. Meanwhile there’s a CIL backend using the LLVM codegen infrastructure. The two work together to generate native-looking SDK-quality assemblies out of all kinds of unmanaged code.

The implementation embraces and extends the ECMA spec with dynamic types from C# 4.0, LINQ for C++, expression trees and full-blown DLR syntax trees, going beyond the original specification in many ways. This is the real interesting part, since it’s what we use to bootstrap the compiler (so it can compile itself).

We were able to port User Mode Linux and WebKit to Silverlight using this with a certain amount of patching. There are other fascinating integration points like a pretty neat Objective-C++/CLI bridge.

Our main focus has been in-game ‘scripting’ and IOS applications. Going forward we’re concentrating more on open web standards and JavaScript for the platform but this continues to be a fun R&D project with often surprising uses, and it’ll be a fun story to tell.

See you there!

(Oh, and we’re looking for developers to join our compiler team in London.)

TL;DR: systemd does not require the performance-sensitive bits of Linux control groups enabled in the kernel. However, it does require some non-performance-sensitive bits of the control group logic.

In some areas of the community there's still some confusion about Linux control groups and their performance impact, and what precisely it is that systemd requires of them. In the hope to clear this up a bit, I'd like to point out a few things:

Control Groups are two things: (A) a way to hierarchally group and label processes, and (B) a way to then apply resource limits to these groups. systemd only requires the former (A), and not the latter (B). That means you can compile your kernel without any control group resource controllers (B) and systemd will work perfectly on it. However, if you in addition disable the grouping feature entirely (A) then systemd will loudly complain at boot and proceed only reluctantly with a big warning and in a limited functionality mode.

At compile time, the grouping/labelling feature in the kernel is enabled by CONFIG_CGROUPS=y, the individual controllers by CONFIG_CGROUP_FREEZER=y, CONFIG_CGROUP_DEVICE=y, CONFIG_CGROUP_CPUACCT=y, CONFIG_CGROUP_MEM_RES_CTLR=y, CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y, CONFIG_CGROUP_MEM_RES_CTLR_KMEM=y, CONFIG_CGROUP_PERF=y, CONFIG_CGROUP_SCHED=y, CONFIG_BLK_CGROUP=y, CONFIG_NET_CLS_CGROUP=y, CONFIG_NETPRIO_CGROUP=y. And since (as mentioned) we only need the former (A), not the latter (B) you may disable all of the latter options while enabling CONFIG_CGROUPS=y, if you want to run systemd on your system.

What about the performance impact of these options? Well, every bit of code comes at some price, so none of these options come entirely for free. However, the grouping feature (A) alters the general logic very little, it just sticks hierarchial labels on processes, and its impact is minimal since that is usually not in any hot path of the OS. This is different for the various controllers (B) which have a much bigger impact since they influence the resource management of the OS and are full of hot paths. This means that the kernel feature that systemd mandatorily requires (A) has a minimal effect on system performance, but the actually performance-sensitive features of control groups (B) are entirely optional.

On boot, systemd will mount all controller hierarchies it finds enabled in the kernel to individual directories below /sys/fs/cgroup/. This is the official place where kernel controllers are mounted to these days. The /sys/fs/cgroup/ mount point in the kernel was created precisely for this purpose. Since the control group controllers are a shared facility that might be used by a number of different subsystems a few projects have agreed on a set of rules in order to avoid that the various bits of code step on each other's toes when using these directories.

systemd will also maintain its own, private, controller-less, named control group hierarchy which is mounted to /sys/fs/cgroup/systemd/. This hierarchy is private property of systemd, and other software should not try to interfere with it. This hierarchy is how systemd makes use of the naming and grouping feature of control groups (A) without actually requiring any kernel controller enabled for that.

Now, you might notice that by default systemd does create per-service cgroups in the "cpu" controller if it finds it enabled in the kernel. This is entirely optional, however. We chose to make use of it by default to even out CPU usage between system services. Example: On a traditional web server machine Apache might end up having 100 CGI worker processes around, while MySQL only has 5 processes running. Without the use of the "cpu" controller this means that Apache all together ends up having 20x more CPU available than MySQL since the kernel tries to provide every process with the same amount of CPU time. On the other hand, if we add these two services to the "cpu" controller in individual groups by default, Apache and MySQL get the same amount of CPU, which we think is a good default.

Note that if the CPU controller is not enabled in the kernel systemd will not attempt to make use of the "cpu" hierarchy as described above. Also, even if it is enabled in the kernel it is trivial to tell systemd not to make use of it: Simply edit /etc/systemd/system.conf and set DefaultControllers= to the empty string.

Let's discuss a few frequently heard complaints regarding systemd's use of control groups:

• systemd mounts all controllers to /sys/fs/cgroup/ even though my software requires it at /dev/cgroup/ (or some other place)! The standardization of /sys/fs/cgroup/ as mount point of the hierarchies is a relatively recent change in the kernel. Some software has not been updated yet for it. If you cannot change the software in question you are welcome to unmount the hierarchies from /sys/fs/cgroup/ and mount them wherever you need them instead. However, make sure to leave /sys/fs/cgroup/systemd/ untouched.
• systemd makes use of the "cpu" hierarchy, but it should leave its dirty fingers from it! As mentioned above, just set the DefaultControllers= option of systemd to the empty string.
• I need my two controllers "foo" and "bar" mounted into one hierarchy, but systemd mounts them in two! Use the JoinControllers= setting in /etc/systemd/system.conf to mount several controllers into a single hierarchy.
• Control groups are evil and they make everything slower! Well, please read the text above and understand the difference between "control-groups-as-in-naming-and-grouping" (A) and "cgroups-as-in-controllers" (B). Then, please turn off all controllers in you kernel build (B) but leave CONFIG_CGROUPS=y (A) enabled.
• I have heard some kernel developers really hate control groups and think systemd is evil because it requires them! Well, there are a couple of things behind the dislike of control groups by some folks. Primarily, this is probably caused because the hackers in question do not distuingish the naming-and-grouping bits of the control group logic (A) and the controllers that are based on it (B). Mainly, their beef is with the latter (which systemd does not require, which is the key point I am trying to make in the text above), but there are other issues as well: for example, the code of the grouping logic is not the most beautiful bit of code ever written by man (which is thankfully likely to get better now, since the control groups subsystem now has an active maintainer again). And then for some developers it is important that they can compare the runtime behaviour of many historic kernel versions in order to find bugs (git bisect). Since systemd requires kernels with basic control group support enabled, and this is a relatively recent feature addition to the kernel, this makes it difficult for them to use a newer distribution with all these old kernels that predate cgroups. Anyway, the summary is probably that what matters to developers is different from what matters to users and administrators.

I hope this explanation was useful for a reader or two! Thank you for your time!

In case you haven't submitted your talk proposal for GUADEC 2012 in A Coruña, Spain yet, hurry: the deadline is on April 14th, i.e. this saturday! Read der Call for Participation! Submit a proposal!

Short summary: layout of the blog changed. Long summary: One of the issues I ran into repeatedly was that any code postings would overrun the rather narrow column width. Now I've done the lazy thing and applied a new standard template that appears to be wider. So in the future, code snippets are hopefully easier to read.

(Ohanami refers to the Japanese festival of cherry-blossom viewing, celebrated during the week or or two in which the trees bloom each year. We've been getting together with friends to sit under the trees here for the last few years.)

Ok, I was away on holidays for the last week so I missed everything. Thanks to Simon Thum, I found the Chrome team's April fools joke: Chrome's Multitask Mode. Allegedly a mode for Chrome that allows to use multiple pointers simultaneously.

Great, except that it's just an April fool's joke, they didn't actually implement it. Even though, well, it's been a standard feature in the X.Org X server since September 2009. Yep, right. Fedora 12 had it, for example, and every version since. It's ubiquitous on the GNU/Linux desktop. GTK3 has support for it.

Being able to use two hands is quite useful, research papers on the benefits go back well over 20 years. Use cases are generally as dominant hand/non-dominant hand methods (e.g. an artist holding the palette with one hand while painting with the other one) or as equal bimanual interaction (aligning a rectangular mask around an object). All this isn't new, and as I said above this is all a standard feature in X since 2009. You really just [1] need to add it in your application. So Chrome's April fool's joke is pretty much a joke about not implementing a feature. Which, well, uhm... ok. Haha.

The video is a bit outlandish, with a guy playing golf and a shooter simultaneously. However, listening to just the audio on the video largely makes sense (except the switching off your computer part). Using two mice became natural for me quite quickly. I even conducted a user-study about users using two browser windows simultaneously to research and gather information about a specific topic. Yep, they could do it, including typing up the results in a decidedly single-user Abiword without major conflicts.

Now I'm waiting for next year's joke, maybe it's about how we drive cars with, wait for it, steering wheels.

[1] I say "just", but implementing it is really hard. It opens a can of worms about assumptions in user interface paradigms that aren't quite that simple to deal with. From a technical point of view, it's a "just" though...

After a long period of very slow development of Transmageddon where most work has been in debugging issues with GStreamer 0.11 and porting existing features to GTK3 and GStreamer 0.11, I decided it was time to start thinking about where to go forward again. That said, there are still some issues with the new GTK3 and GStreamer 0.11 version, but I needed a break from just tedious porting/re-implementation work and instead start looking at some new ideas and features.

There are two major items I am thinking of. The first is what to do with the usecase of people just wanting to create a manual transcode. The goal of Transmageddon is to make transcoding simple, but the problem is that I am trying to simplify something that is inherently quite complex. The current user interface doesn’t really let you set a lot of things and I know that sometimes that will create files that doesn’t conform to the needs of the user, for instance a lot of settings are just kept from the original output file, like number of audio channels or bitrate used and so on. And many others just rely on the default values of the GStreamer elements used. I don’t want to try to support tweaking all of them through the user interface as there is no way of doing so without making the user interface either cluttered or filled with what will be for most people just gibberish. It is the problem I feel for instance Handbrake is suffering from, that yes it does let you set everything you could ever need, so it can create files that will be useable in some cases where the default user interface of Transmageddon falls short, but it also becomes a hard to navigate jungle for people. My answer to the need for that level of settings is and will continue to be the device profiles option, which I also have some plans for. That said there are a couple of features I do think would be useful to enable from the non-preset interface, like being able to resize the video, choose if you want deinterlacing or not and to allow you to choose stereo output for the audio and finally I want to allow the creation of transcoding queues, so you don’t have to wait until one transcode is finished before configuring another one.

The one special feature I already got in there, the ability to rotate the video, so that if your video for instance was shot with a camera held sideways, also felt rather arbitrary (it was put in there due to a very early bug request) to have exposed in the userinterface.

So what I have been experimenting with is as user interface which has an Advanced menu option in which you can set certain things and also enable certain extra options in the user interface.

So as you see in the screenshot above there are certain options which you just turn on/off and certain options which when enabled will add extra elements to the userinterface as shown below.
If I go down this route the question of course comes what options to put into this Advanced menu and also if I should make the settings persistent accross runs of Transmageddon or if I should let Transmageddon return to its default settings every time you start it.

The Transmageddon editor

Of course the Advanced menu will never become the solution to all transcoding challenges, the device profiles will still be that. But I realized that editing textfiles is not for everyone and it also makes creating advanced profiles a task for even fewer people than there needs to be. I don’t know how many people out there have made their own profiles, but the only one I ever got submitted was from Stefan Kost who made one for the N900. That is probably mostly due to lack of documentation of how to create profiles and where to put them, a problem I have been planning to remedy. That said I realized that maybe creating some kind of editor would be an even better solution, as it could provide a lot of helpful tools for profile creation and thus making it accessible to a lot more people. Which is why I been trying to prototype what such an user interface could look like.

The interface above is what I got so far, and it is just a glorified profile viewer atm, but my hope is to make it fully functional and hopefully useful to people. One feature I really want to do is to allow you to take an existing video, have the profile editor analyze it and create a new profile that will allow you to replicate the settings of that file. So when you get a new phone or device the manufacturer most likely put a sample file on it, you can then load that sample into the editor and the editor will create a profile that matches it. This will enable you to transcode other files to that profile and thus make them work on your device.

That said this will be a major task creating this editor, because I want to to contain a lot of clever logic, so that it doesn’t just end up being a glorified text editor, but I will need to test and experiement to figure out what that logic will be and how to expose it in the user interface.

Anyway, I am hoping to hear back from the community on these two new things and playing with to hear what you think, both from a usability standpoint and of course ideas for how the features could work or should not work, and what you would need to make Transmageddon suit your needs.

Smurf com•mit [smɚf kə'mĭt]
noun

1. Commit made, usually during a long series of commits, that makes incremental changes or tweaks while developing some larger feature or performing some larger refactor. Such commits often feature commite messages such as "almost there," "will be working soon," or "not much further now."

2. Commit made by a Smurf.

Students, this Friday at 1900 UTC is the deadline to apply for this year’s GSoC. It’s an awesome program that pays you to work on open-source projects for a summer (where you == a university/college student).

It’s by no means too late, but start your application today. You can find more information on Gentoo’s projects here (click on the Ideas page to get started; also see our application guidelines) and on the broader GSoC program here.

Good luck!

Tagged: community, development, gentoo, gsoc

Apache 2.4 being out, I noticed that my good old mod_defensible did not compile anymore.

The changes in the new Apache 2.4 API were small for its concern, so it was pretty easy to update this software to make it compile again.

Honestly, I'm not sure that this module is really used into the wild, but I still think that it can serve as a good prototype for doing other things so I like keeping it around. :-)

All this has been triggered by the Apache 2.4 arrival into Debian experimental. Therefore I've updated the mod_defensible package to use the new dh_apache2, and imported it into Git at the same time.

I love a good photo project. Here are some favourites -- feel free to suggest any that you've enjoyed in the comments.

Pierre Beteille's Books/Livres:

Irina Werning's Back To The Future and Back To The Future 2:

Natsumi Hayashi's Levitating Girl:

Adele Enersen's When My Baby Dreams:

Carli Davidson's Portraits of loving pets with disabilities:

Adde Adesokan’s Triptychs of Strangers:

I didn't pay much attention to sports when I lived in England. They're much harder to ignore now that I live in the US, especially in a competitive town like Boston where each of the national titles in basketball, American football, ice hockey and baseball have been won by a local team at least once since 2000.

Mad just won her lab's March Madness bracket (this involves making predictions on the winners and losers of 63 college basketball games, and getting more of them correct than the other people who are competing with you) so we've watched a lot of basketball games recently, and I like to watch my countryman¹ Andy Murray play tennis when I can.

I've also been trying to decide what my favorite sport is. Sometimes people ask you things like that here. I think they're mainly asking to see whether I say something "outlandish" about soccer being better than American Football, or cricket being the zenith of physical competition between teams. I find that I enjoy watching basically all of them, especially with friends. My actual answer is long and complicated, so I'll write it out here. I'm going to concentrate on popular sports in the US, since that's where I live now.

Basketball

As Mako pointed out to me, it's rarely worth tuning in to a basketball game before the last five to ten minutes. A 10-15 point deficit is by no means insurmountable with ten minutes left to play, and teams are rarely more than 20 points ahead by then. You generally won't have missed the decisive portion of the game if you wait until the last quarter to start watching, which makes watching the first three quarters feel unsatisfying somehow.

The NBA tries to make the league more competitive between teams by having a draft where the teams that didn't make the playoffs in the previous season get the first choice of upcoming college players in the new season; it also has a salary cap and luxury tax. I like these ideas in principle, but they don't seem to be working in practice yet: the NBA has teams that repeat championship wins more often than the NFL, showing that these rules are not working to break up domination by a few teams.

Morale seems to be a particularly visible aspect of basketball given the frequent scoring — one team going on a 10-0 run is common, often with the other team coming back with a reversed run later, and that's satisfying to watch.

Basketball seems especially prone to having games be decided by controversial referee decisions, most of which can not be appealed or corrected.

Baseball

Baseball has no salary cap, and has the far worse equivalent of a luxury tax which sends money to the MLB organization itself when a team pays over a salary limit. This does not bother teams like the New York Yankees or Boston Red Sox, which continue to spend ridiculous amounts on players undeterred. The last game of the 2011 season, between the Red Sox and the Tampa Bay Rays, was between a team with a $156M payroll and a team with a $37M payroll². Why is that supposed to be fun or fair to watch? I guess it could be fun if you always cheer for the team that is vastly outperforming its payroll (as Tampa Bay did against Boston that night!), but otherwise I don't see the point.

As for Mako's decisive-moment metric, baseball almost has the inverse problem to basketball — a bases-loaded homer in the 3rd inning (for example) can often decide the game right there, so you have to start watching at the beginning to be confident of not missing a decisive play. The games are pretty long too. I think there's a middle ground between early-decisiveness and last-minute-decisiveness that I prefer instead.

American (NFL) Football

I like that the NFL has a direct salary cap, so no team can spend many times more on players than another team as they do in baseball. I also like the amazing speed, agility and split-second instincts that NFL players exhibit. The decisiveness seems about right. The system for appealing rulings and referring to instant replays works well.

But I hate that the NFL has a problem with chronic traumatic encephalopathy that it is basically doing nothing about. Many veteran players are suffering from this form of early-onset dementia. This exposé by GQ in 2009 (GQ, really?!) is completely damning, and I think it should be required reading for anyone who watches NFL football. There's also Malcolm Gladwell's article in the New Yorker directly comparing American football to dogfighting, which seems sound to me.

But wait, you say! This season the NFL instituted new rules on what type of hits are allowed. These rules entirely miss the point — as you'll read in the GQ article, professional footballers take the equivalent of multiple car crashes of sub-concussion every game, and none of the plays that cause these sub-concussions were illegal before the new rules or are illegal after them. The new "devastating tackle" rule bans tackles that look gruesome, while leaving alone the hits that are known to cause permanent brain injury when repeated as they are on almost every play. It's a cover up, and the player-safety equivalent of security theater.

We also don't get to see NFL teams practice, yet the HITS data referred to by the GQ article suggests that practices contain many sub-concussions too, and can be as damaging as the games themselves. If the players spend 80% of their time practicing and 20% playing in season games, then practice could be responsible for 80% of the brain injuries due to concussion they get, and we don't even see it happening.

My preferred solution is to remove helmets altogether, as rugby does. Faced with the option of colliding heads at 100 gs of force but without a helmet, it seems likely that players would just decide not to do something so ridiculous with their heads.

Ice Hockey

I'm not at all expert in ice hockey, but it seems to me that it has the same safety problems as football, plus legalized and celebrated fighting between players. Gross.

Tennis

It's nice to have an individual sport as well as team ones. Psychology and willpower often seem to become the dominant factor in who wins a game, and that's exciting to watch. I guess it ties back into my experience playing Chess and Go, and individual tennis might be my favorite sport for that reason alone.

At first glance it seems like tennis has the same decisiveness problems as baseball, with it being hard to come back from an early deficit, but it doesn't really. Tennis doesn't keep an absolute cumulative score in the same way as baseball — you can lose the first set in a 1-6 blowout and win the second set 7-6, and then everything's all square again even though you only won 8 games and your opponent won 12. The baseball equivalent would be something like if you could make up for your opponent's third-inning bases-loaded homer with a well-timed single run afterwards, and that would certainly keep the games closer.

Tennis refereeing decisions can be appealed with a challenge system that works well.

Soccer

Has the "beginning of the game can decide the match" problem of baseball. Clubs refuse to institute a salary cap, leading to the same kind of disparity that baseball has between teams — in 2008, the median Premier League total team salary was $44M/year, yet Chelsea spent³ $172M/year!

Worse, continuing to refuse to use instant replay to correct poor referee decisions (in 2012!) is basically unforgivable, and caused significant international dispute during the last World Cup. Peter Singer wrote an article about the ethics of cheating in football that I agree with wholeheartedly; I don't think soccer players make good role models under the current rules (with the possible exception of Lionel Messi's refusal to flop).

Singer looks for an instance of overtly ethical play in soccer, but even the one example he manages to find is not a good one. He mentions Robbie Fowler in 1997 being awarded a penalty which he tries to decline, but the referee insists that he take it. Singer says that Fowler takes the penalty "in a manner that enables the goalkeeper to save it", but that's not what happens — the keeper does get a hand to the shot, but then it spills back out and one of Fowler's teammates puts in the rebound for a goal. If Fowler actually wanted to decline the penalty, he could have simply not aimed it at the inside of the goal. Fowler says that he was not intending to miss, and was playing with acceptance of the referee's decision even though he didn't agree with it.


Okay, so I have complaints and likes about every sport. How do I decide which one is my favorite? In the true spirit of utilitarianism, I have a nerdy idea about comparing sports against each other using the axes I care about directly: the satisfyingness of Mako's decisive moment's timing, the long term safety of the players, the impact of referee decisions on the outcome and the impact that money has on the strength of each team; but I'll break this post up here and wait until my next post to detail my calculation. Of course, these variables might be much more important to me than to you. What do you think? Are there more variables that you'd add for consideration? How does your favorite sport stack up in these metrics? Do you want to set me straight on ice hockey?

¹: No, I'm not trying to claim that Andy Murray is English like me; he is Scottish. But apparently the United Kingdom is a country that consists of countries, even though that makes no sense, so he's still my countryman.

²: Figures are from http://baseballplayersalaries.com/.

³: Stats from http://www.trophy4toon.co.uk/salaries.html. I can't find total salary data that's newer than 2008.

I do have a SandyBridge desktop. This system is a bit special as it is the one in which I can run a discrete card in direct comparison to the ingrated processor graphics, i.e. run both GPUs simultaneously (but not yet co-operatively). In this system I put a Radeon HD5770 which at the time was a mid-range GPU offering good value for performance. So how does Cairo fare on this box? In particular are the DDX drivers any better than pixman, the backend for Cairo’s software rasteriser?

Once again the white baseline in the centre represents the performance of the image backend, how fast we expect to be able to render the contents ourselves. Above that baseline, the driver is faster; below slower, laggy and power hungry. This is quite a busy graph as it compares the performance of the propietary fglrx driver and the open-source radeon driver along with the alternative acceleration methods for the integrated graphics. I’d recommend viewing at full size, but the gist of it is that in many cases the propietary driver lags behind the open-source effort. Neither are truly comparable to just using the CPU and only the open-source effort is ever faster. In contrast, we have the integrated processor graphics. Even this GT1 desktop system (only half the GPU capability of a GT2 system such as found on mobiles and on a few desktop chips) can outclass the CPU. When not limited by a poor driver, that is.