One of the platforms we’ve been working on for a while at Collabora is the Raspberry Pi. Obviously the $25 pricepoint makes it hugely appealing to a lot of people — including free software developers who up until now have managed to avoid the agonyjoy we experience on a daily basis working on embedded and mobile platforms — but there are a couple of aspects which speak specifically to us as a company.

Firstly, we did quite a bit of work on OLPC through the years, which had a similar, very laudable, educational mission encouraging not just deep computer literacy in children, but also open source involvement. The Raspberry Pi has broadly the same aims, a very education-friendly pricepoint, and has seen huge success.

Less loftily, it’s a great example of a number of architectures we’ve been quietly working on for quite some time, where hugely powerful special-purpose (i.e. not OpenGL ES) graphics hardware goes nearly unused, in favour of heavily loading the less powerful CPU, or pushing everything through GL.

background etc

The Raspberry Pi has a Broadcom BCM2385 SoC in it, containing an extremely beefy (roughly set-top-box-grade) video, media and graphics processor called the VideoCore (somewhat akin to a display controller, GPU and DSP hybrid), and a … somewhat less beefy general-purpose ARMv6¹ CPU. The ARM side does everything you’d expect, whereas the VideoCore is a multi-functional beast, acting as the GPU for OpenGL ES, the display engine for outputs/overlays/etc, and also any general-purpose processing (e.g. accelerated JPEG decode).

In terms of how this looks from the ARM, the VideoCore exposes its display functionality through DispManX, an API for display control similar, in capability at least, to KMS. The DispManX exposes a number of output displays, each of which can have a number of planes (sometimes called overlays or sprites) which can each be in different colourspaces (think: video), scaled, alpha-blended, or variously stacked. No surprise there, as this is how most GPUs and display controllers look everywhere: from your phone, to your desktop, to your set-top box.

A recurring theme for us is how to properly use and expose these overlays. There’s a huge benefit in doing so over using GL: not only are they a lot faster, but they also have hugely better quality when doing colourspace conversion and scaling, and extra filters for much better image quality². There’s also a pretty strong power argument to be made; at one stage, measuring on a phone, we found a 20% runtime difference — from 4 hours to a bit over 5 — when watching videos using the overlay, compared to pumping them through GL ES. And that was without the zerocopy support we enjoy nowadays!

The X Video extension (Xv) exposes overlays in a very limiting and frustrating way, meaning they’re only really suitable for video, and even then aren’t capable of zerocopy.  DRI2 video support was proposed to fix this, but so far TI’s OMAP is the only real deployment of this, and client support is very patchy.

Even then, this is only applicable to video. Under the X11 model, effectively the only way to do compositing is for an external process to render the entire screen as a single image, then pass that image to the X server to display. This is fine for GL, since that’s exactly what it’s built for, but it means you’re never going to get to use all your lovely 2D compositing hardware. Either that, or you do offload compositing inside the X server: something very difficult which almost no-one does, not only because it means no compositing. And no compositing means that your desktop is all back to 1995, with those attractive flickering backgrounds and jittery resizes. :(

i thought this was about rpi … ?

We’ve been working with the Raspberry Pi Foundation since last year, when we first brought up Weston on Raspberry Pi. At that stage, Weston was still very heavily GLES-based, but was able to opportunistically pull individual surfaces out into overlays if the conditions were right. This worked, and was all well and good, but was an awkward abuse of Weston’s internal model, and made RPi look rather unlike any other backend.

Fast forward to a couple of months ago, and Weston had come a long way along the road towards 1.1. A pretty vital piece for us was the renderer infrastructure: whereas Weston previously only had pluggable backends controlling the final display output backend (think: KMS, fbdev, RDP), it now gained a similar split for the renderers.

The original split was between the original GLES renderer and a purely-software Pixman renderer (which was still capable of everything the GLES renderer was!), but it was also a perfect fit for DispManX. So, the first thing Pekka did, and the base of all our work since, was to split the RPi backend into a backend and a renderer, allowing us to guarantee that we’d never fall back to GLES. This lets us feed the entire set of windows into the hardware, complete with stacking order, alpha blending, and 2D transforms, and have the VideoCore take care of everything, all fully synchronised and tear-free. Without using GL!

Pekka’s blog has (or will have, when he publishes it and I sneakily amend this article) more of the gory details on both the Weston internals, and how DispManX works.

shiny demos

Seemingly no-one can talk about open source graphics without getting all misty-eyed over wobbly windows; what good is straight tech with nothing to show for it? So, while Pekka worked on the renderer, Louis-Francis and myself knocked up a couple of demos to show off what we could do with 2D compositing alone.

The first, and most compelling, demo is in our case study,  in which Louis-Francis drags some windows around under both X11 and Weston for the camera. X11 struggles badly, whereas Weston with its VideoCore backend keeps up like a champ.

Louis-Francis then added an optional fade layer, where all the background windows were dimmed, complete with a nice fading animation between foreground and background. This is done internally with one or two solid-colour surfaces, with varying alpha.

Last but not least, I added a reimplementation of GNOME3′s window overview mode. It is a little on the rudimentary side, and does certainly suffer from not having an established layout/packing framework to use, but does work, and I think provides a pretty good demonstration of the kinds of smooth and fluid animations that are totally possible without ever touching GLES. And the best part is that windows zoom around, scaled and alpha-blended, at 60fps, whereas X11 struggles to get to 10fps just moving an unscaled, opaque, window. Yeesh.

And then some of the details were just that: details. Pekka add a new background scaling mode which preserves aspect ratio, Louis-Francis and Pekka worked to make sure the memory usage was always as low as humanly possible, including an optional mode where we trade strict visual correctness for performance and memory usage, we fixed some bugs in the XWayland emulation, and some bugfixes.

sounds great; how do i get it?

The source is already winging its way upstream; currently, the renderer has been merged, but some of the effects are still oustanding. In the meantime, you can clone Pekka’s repository:

$ git clone git://git.collabora.co.uk/git/user/pq/weston
$ cd weston
$ git checkout origin/raspberrypi-dispmanx-wip

Or we’ve also got Raspbian packages, based on the stable 1.0.x rather than unreleased git:

# echo deb http://raspberrypi.collabora.com wheezy rpi >> /etc/apt/sources.list
# apt-get update
# apt-get install weston

tl;dr

A lot of hardware — particularly in the media/embedded/mobile space — ships with hugely powerful special-purpose graphics hardware, aside from the usual OpenGL ES. Up until now, this special-purpose hardware has gone unused, but for equally special-purpose hacks. The work we’ve done with Weston and Raspberry Pi, using only its dedicated 2D compositing hardware, is a fantastic example of what we can do with Wayland and these kinds of platforms in general, without using GL ES.

now convince your boss

Would you love us to shut up and take your money, but just need that little bit extra to persuade those humourless lot up in purchasing? Fear not: we have an entire case study for the work we’ve done on the Raspberry Pi, as well as a general graphics page. Or maybe you’re more impressed (ha) by press releases? Either way, our trained operators are standing by to take your call.

1. ARMv6 is the sixth iteration of the ARM architecture; the chip family is ARM11, which was the generation immediately pre-Cortex. Various ARM11s were used in, e.g., the Nokia N810, the iPhone 3G, and the Nintendo 3DS. If you’re confused by the ARM nomenclature, Wikipedia has a really good table.
2. Fun side note: if you want zero-copy video through GL, thanks to the unbelievably harsh wording of GL_OES_EGL_image_external, then you’re only allowed to use linear or nearest filtering. Not even bilinear, and definitely nothing near what overlays can do.

Just tagged a 1.0.0 release for libmbim, a library which helps you talk to MBIM-capable modems. You can read more about the MBIM protocol in the libmbim introduction blogpost I wrote some months ago. The 1.0.0 tarball is ready for download from freedesktop.org:
 
http://www.freedesktop.org/software/libmbim/libmbim-1.0.0.tar.xz

If you want to easily talk to a MBIM device from a GLib-based application, you may want to check the libmbim API documentation.

libmbim is currently used by ModemManager (git master), but you can also now use it in standalone mode with either mbimcli (the command line utility) or mbim-network (a helper script to launch a connection):

# echo "APN=Internet" > /etc/mbim-network.conf
 
# mbim-network /dev/cdc-wdm0 start
Loading profile...
    APN: Internet
Starting network with 'mbimcli -d /dev/cdc-wdm0 --connect=Internet --no-close'...
Network started successfully
 
# mbim-network /dev/cdc-wdm0 status
Loading profile...
    APN: Internet
Getting status with 'mbimcli -d /dev/cdc-wdm0 --query-connection-state --no-close'...
Status: activated
 
# mbim-network /dev/cdc-wdm0 stop
Loading profile...
    APN: Internet
Stopping network with 'mbimcli -d /dev/cdc-wdm0 --disconnect'...
Network stopped successfully


As with libqmi’s qmi-network script, you’ll still need to run a DHCP client on the wwan interface after getting connected through MBIM. Note that your modem may not support DHCP… if that’s your case then patches are welcome to update the script to dump the IP configuration Or just use ModemManager, which works nicely with the static IP setup.

Enjoy!

AltOS 1.2.1 — TeleBT support, bug fixes and new AltosUI features

Bdale and I are pleased to announce the release of AltOS version 1.2.1.

AltOS is the core of the software for all of the Altus Metrum products. It consists of cc1111-based micro-controller firmware and Java-based ground station software.

The biggest new feature for AltOS is the addition of support for TeleBT, our ground station designed to operate with Android phones and tablets. In addition, there’s a change in the TeleDongle radio configuration that should improve range, some other minor bug fixes and new features in AltosUI

AltOS Firmware — Features and fixes

There are bug fixes in both ground station and flight software, so you should plan on re-flashing both units at some point. However, there aren’t any incompatible changes, so you don’t have to do it all at once.

New features:

• TeleBT support.

• Improved radio sensitivity. The TeleDongle receiver parameters have been tweaked to provide better reception.

• TeleMini now completely resets all radio parameters in recovery mode (with the two outer debug pins connected) — 434.550MHz, N0CALL, factory radio cal.

Bug fixes:

• USB device fixes. This improves operation with Windows, avoiding hangs and errors in many cases.

• Correct the Kalman filter error covariance matrix; the old parameters were built assuming continuous measurements.

AltosUI — Easier to use

AltosUI has also seen quite a bit of work for the 1.2.1 release. It’s got several fun new features and a few bug fixes.

New Graph UI features:

• Show tool-tips with the value near the cursor.

• Make the set of displayed values configurable. Add all of the available data values just in case you want to see them.

• Added a Map tab showing the ground track of the whole flight.

• The flight summary tab now includes the final GPS position. This lets you figure out where your rocket landed without replaying the whole flight.

Other new AltosUI features:

• TeleBT support, including Bluetooth connections (Linux-only, at present).

• Shows the callsign in the Monitor Idle and other command-mode windows so that you can tell what callsign is being used.

• Show the block number when downloading flight data. This lets you see something happen even for longer flights.

• Make the initial position of the AltosUI configurable so that you can position it out of the way of the rest of you desktop.

• Distribute Mac OS X in .dmg format (Mac OS Disk Image); this means you don’t need to explicitly unpack the bits.

Bug fixes:

• Deal with broken networking while downloading map tiles. Tiles are now always downloaded asynchronously so that the UI doesn’t freeze when the network is slow.

Because of some event I cannot remember, I was curious to see which developers had how many lines of code. It turned out to be more of a challenge than I thought (mostly because of Unicode). Here is the top 10 from the very latest drm-intel-nightly.

>./count_i915.py drivers/gpu/drm/i915/

total lines counted: 67417 (compare with git ls-files — ./drivers/gpu/drm/i915/ | xargs wc -l)
[('Jesse Barnes', 16255),
('Chris Wilson', 13174),
('Daniel Vetter', 11066),
('Eugeni Dodonov', 3636),
('Paulo Zanoni', 3434),
('Ben Widawsky', 2935),
('Eric Anholt', 2176),
('Keith Packard', 1773),
('Zhenyu Wang', 1703),
('Ville Syrjälä', 1130)]

And here is the slightly better than one-off script I used:

#!/usr/bin/env python3
import sys
import os
import subprocess
import re
import collections
import pprint

count = collections.Counter()
p = re.compile(r'.+ \(([\w\W]+) [0-9]{4}-[0-9]{2}-[0-9]{2}.*', re.UNICODE)

path=sys.argv[1]
git_path=os.path.dirname(sys.argv[0]) + "/" + path
os.chdir(git_path)
for file in os.listdir(os.getcwd()):
        blame = subprocess.check_output(["git", "blame", "-w", "--abbrev=10", file],
                        stderr=subprocess.STDOUT)
        for line in blame.splitlines():
                m = p.match(str(line, 'utf-8'))
                name = str(m.group(1))
                name = name.rstrip()
                count[name] += 1

print("total lines counted: " + str(sum(count.values())) + " (compare with git ls-files -- " + git_path + " | xargs wc -l)")
pprint.pprint(count.most_common(10))

EDIT: The code pretty printer doesn’t like ‘\’ and therefore 3 instances were missing in the first post. I’ve actually copy/pasted the version from the blog now and run it.

I was asked by some people to write a status report about the whole PK/AS/LI stuff – sorry guys that it took so much time to write it .

PackageKit

(PackageKit is an abstraction layer for package-management systems, allowing applications to access the package-manager using simple DBus calls)

PackageKit is an incredibly successful project. With the 0.8.x series, it received many performance improvements, and has now the same speed on my computer than the distribution’s native tools. PackageKit is used in almost all major Linux distributions, except for Ubuntu[1]. But even Ubuntu has written some compatibility layer, so most calls to PackageKit will work.

The only major distro where PackageKit is currently not available, seems to be Gentoo (and I am not sure about the shape of the Gentoo PackageKit backend too).

Debian Wheezy includes PackageKit by default, and in Jessy we are going to replace some distribution-specific tools with PackageKit frontends (mostly the old and unmaintained update-notifier and Software-Updater – no worries, we are not going for a Synaptic replacement (currently this won’t be possible with PK anyway))

Unfortunately, some PackageKit backends are still not adjusted for the 0.8.x API and are only running on 0.7.x. This is bad, since 0.8.x is a huge step forward for PackageKit. But the situation is slowly improving, with the latest OpenSUSE release, the Zypper backend is now available on 0.8.x too.

Being able to run a PackageKit from the 0.8.x series is a requirement for both AppStream and Listaller.

AppStream

(AppStream is a cross-distro effort for building Software-Center-like applications. It contains stuff like a screenshot-service, ratings&reviews etc. The most important component is a Xapian database, storing information about all available applications in the distribution’s repositories. The Xapian DB is distro-agnostik, but distributors need to provide data to fill it. AppStream offers an application-centric way to look at a package database)

The AppStream side doesn’t look incredibly great, but the situation is improving. As far as I know, OpenSUSE is shipping AppStream XML to generate the database. Ubuntu ships the desktop-files, and I am working on AppStream support in Debian’s Archive Kit. On the Fedora side, negotiations with the infrastructure-team are still going on. I haven’t heard anything from Mageia and other AppStream participants yet.

Unfortunately, at least for OpenSUSE, the AppStream efforts seem to be stalled, due to people having moved to different tasks. But efforts to add the remaining missing things exist.

On the software side, Apper (KDE PackageKit frontend) has full support for AppStream. Apper just needs to be compiled with some extra flags to make it use the AppStream database.

On the GNOME-side, GNOME-Software is being developed. The tool will make use of the AppStream database, on distributions where it is available.

Also, a Software-Center for Elementary and other GTK+-based desktops is being developed, which is based on AppStream (already quite usable!).

Using the Ubuntu Software Center on not-Ubuntu-based distributions ist still not much fun, but with the AppStream database available and a working PackageKit 0.8.x with a backend which supports parallel transactions, it is possible to use it.

On the infrastructure side: I recently landed some patches in AppStream-Core, which will improve the search function a lot. AppStream-Core contains all tools necessary to generate the AppStream database. It also contains libappstream, a GObject-based library which can be used to access the AppStream database.

Also, we discuss dropping PackageKit’s internal desktop-file-cache in favour of using the AppStream database. If we do that, we will also add software descriptions to the AppStream db, to improve search results and to speed up search for applications. Because we have to deprecate API for that, I expect this change to happen with PackageKit 0.9.x.

As soon as the Freedesktop-Wiki is alive again and my account is re-enabled, I will create compatibility-list, showing which distribution implements what of the PK/AS/LI stuff, especially focusing on components needed for AppStream.

Only a few distributions package AppStream-Core so far. Although it is beta-software, creating packages for it and shipping the required data to generate the AppStream database would be a very huge step forward.

Listaller

(Listaller is a cross-distro 3rd-party software installer, which integrates into PackageKit and AppStream. It allows installing 3rd-party applications, which are not part of the distributor’s repositories, using standard tools used also for native-package handling. Everything which uses PackageKit can make use of Listaller packages too. Listaller also allows sandboxing of new applications, and uses an AppDir-like approach for installing software.)

Listaller is currently undergoing it’s last transition before a release with stable API and specifications can be made. Dependency solving will be improved a lot during the current release-cycle, making it less powerful, but working on all distributions instead. (Fedora always had an advantage in dependency-solving, due to RPM providing more package metadata for Listaller to use) This change was delayed due to discussing a possible use of ZeroInstall-feeds to provide missing dependencies with the ZeroInstall team. We did not come to a conclusion about extending the XML, so Listaller will contain an own format to define a dependency, which can reference a ZeroInstall feed. That should be a good solution for everyone.

All these changes will result in IPK1.2, a new version of the IPK spec with small changes in the pkoptions file syntax and huge changes in dependency-handling. The new code is slowly stabilizing in a separate branch, and will soon be merged into master.

The next Listaller release will be the last one of the 0.5.x series, we will start 0.6.x then. KDE currently has support for Listaller through Apper, which is enabled on a few distributions. In GNOME, optional Listaller support is being developed and will be available in one of the upcoming releases.

Currently, to my knowledge, only a few distributions package Listaller. This should improve, so it is easier for application developers to deploy IPK packages.

The upcoming changes in KDE and GNOME to build stable developer platforms will help Listaller a lot in finding matching dependencies, and for stuff which only depends on one software frameworks, installations should be a matter of seconds.

As you can see, lots of things are happening, and there is improvement in all components related to installing and presenting software on Linux machines. However, all these projects have a severe lack of manpower, especially AppStream and Listaller have the lowest number of developers working on the tools (at time, only two active developers). This is the main reason for the slow development. But I am confident that we will have something shipped in the next distribution releases. At least AppStream should be ready then.

[1]: I don’t blame Ubuntu for that – during the time they wrote an own solution, PackageKit did not have all the required features. (This situation has changed now, fortunately.)

NOTE: I might extend this post with feedback from the different distributions, as soon as I get it.

One of my recent innocent tweet about Gerrit vs Github triggered much more reponses and debate that I expected it to. I realize that it might be worth explaining a bit what I meant, in a text longer than 140 characters.

I'm having a hard time now contributing to projects not using Gerrit. Github isn't that good.

— Julien Danjou (@juldanjou) May 8, 2013

The problems with Github pull-requests

I always looked at Github from a distant eye, mainly because I always disliked their pull-request handling, and saw no value in the social hype it brings. Why?

One click away isn't one click effort

The pull-request system looks like an incredible easy way to contribute to any project hosted on Github. You're a click away to send your contribution to any software. But the problem is that any worthy contribution isn't an effort of a single click.

Doing any proper and useful contribution to a software is never done right the first time. There's a dance you will have to play. A slowly rhythmed back and forth between you and the software maintainer or team. You'll have to dance it until your contribution is correct and can be merged.

But as a software maintainer, not everybody is going to follow you on this choregraphy, and you'll end up with pull-request you'll never get finished unless you wrap things up yourself. So the gain in pull-requests here, isn't really bigger than a good bug report in most cases.

This is where the social argument of Github isn't anymore. As soon as you're talking about projects bigger than a color theme for your favorite text editor, this feature is overrated.

Contribution rework

If you're lucky enough, your contributor will play along and follow you on this pull-request review process. You'll make suggestions, he will listen and will modify his pull-request to follow your advice.

At this point, there's two technics he can use to please you.

Technic #1: the Topping

Github's pull-requests invite you to send an entire branch, eclipsing the fact that it is composed of several commits. The problem is that a lot of one-click-away contributors do not masterize Git and/or do not make efforts to build a logical patchset, and nothing warns them that their branch history is wrong. So they tend to change stuff around, commit, make a mistake, commit, fix this mistake, commit, etc. This kind of branch is composed of the whole brain's construction process of your contributor, and is a real pain to review. To the point I quite often give up.

A typical case: 3 commits to build a 4 lines long file.

Without Github, the old method that all software used, and that many software still use (e.g. Linux), is to send a patch set over e-mail (or any other medium like Gerrit). This method has one positive effect, that it forces the contributor to acknowledge the list of commits he is going to publish. So, if the contributor he has fixup commits in his history, they are going to be seen as first class citizen. And nobody is going to want to see that, neither your contributor, nor the software maintainers. Therefore, such a system tend to push contributors to write atomic, logical and self-contained patchset that can be more easily reviewed.

Technic #2: the History Rewriter

This is actually the good way to build a working and logical patchset using Git. Rewriting history and amending problematic patches using the famous git rebase --interactive trick.

The problem is that if your contributor does this and then repush the branch composing your pull-request to Github, you will both lose the previous review done, each time. There's no history on the different versions of the branch that has been pushed.

In the old alternative system like e-mail, no information is lost when reworked patches are resent, obviously. This is far better because it eases the following of the iterative discussions that the patch triggered.

Of course, it would be possible for Github to enhance this and fix it, but currently it doesn't handle this use case correctly..

Exercise for the doubtful readers: good luck finding all revisions of my patch in the pull-request #157 of Hy.

A quick look at OpenStack workflow

It's not a secret for anyone that I've been contributing to OpenStack as a daily routine for the last 18 months. The more I contribute, the more I like the contribution workflow and process. It's already well and longly described on the wiki, so I'll summarize here my view and what I like about it.

Gerrit

To send a contribution to any OpenStack project, you need to pass via Gerrit. This is way simpler than doing a pull-request on Github actually, all you have to do is do your commit(s), and type git review. That's it. Your patch will be pushed to Gerrit and available for review.

Gerrit allows other developers to review your patch, add comments anywhere on it, and score your patch up or down. You can build any rule you want for the score needed for a patch to be merged; OpenStack requires one positive scoring from two core developers before the patch is merged.

Until a patch is validated, it can be reworked and amended locally using Git, and then resent using git review again. That simple. The historic and the different version of the patches are available, with the whole comments. Gerrit doesn't lose any historic information on your workflow.

Finally, you'll notice that this is actually the same kind of workflow projects use when they work by patch sent over e-mail. Gerrit just build a single place to regroup and keep track of patchsets, which is really handy. It's also much easier for people to actually send patch using a command line tool than their MUA or git send-email.

Gate testing

Testing is mandatory for any patch sent to OpenStack. Unit tests and functionnals test are run for each version of each patch of the patchset sent. And until your patch passes all tests, it will be impossible to merge it. Yes, this implies that all patches in a patchset must be working commits and can be merged on their own, without the entire patchset going in! With such a restricution, it's impossible to have "fixup commits" merged in your project and pollute the history and the testability of the project.

Once your patch is validated by core developers, the system checks that there is not any merge conflicts. If there's not, tests are re-run, since the branch you are pushing to might have changed, and if everything's fine, the patch is merged.

This is an uncredible force for the quality of the project. This implies that no broken patchset can ever sneak in, and that the project pass always all tests.

Conclusion: accessibility vs code review

In the end, I think that one of the key of any development process, which is code review, is not well covered by Github pull-request system. It is, along with history integrity, damaged by the goal of making contributions easier.

Choosing between these features is probably a trade-off that each project should do carefully, considering what are its core goals and the quality of code it want to reach.

I tend to find that OpenStack found one of the best trade-off available using Gerrit and plugging testing automation via Jenkins on it, and I would probably recommend it for any project taking seriously code reviews and testing.

If you’re a university student, time is running out! You could get paid to hack on Gentoo or other open-source software this summer, but you’ve gotta act now. The deadline to apply for the Google Summer of Code is this Friday.

If this sounds like your dream come true, you can find some Gentoo project ideas here and Gentoo’s GSoC homepage here. For non-Gentoo projects, you can scan through the GSoC website to find the details.

Shared Memory Fences

In our last adventure, dri3k first steps, one of the ‘future work’ items was to deal with synchronization between the direct rendering application and the X server. DRI2 “handles” this by performing a round trip each time the application starts using a buffer that was being used by the X server.

As DRI3 manages buffer allocation within the application, there’s really no reason to talk to the server, so this implicit serialization point just isn’t available to us. As I mentioned last time, James Jones and Aaron Plattner added an explicit GPU serialization system to the Sync extension. These SyncFences serializing rendering between two X clients, but within the server there are hooks provided for the driver to use hardware-specific serialization primitives.

The existing Linux DRM interfaces queue rendering to the GPU in the order requests are made to the kernel, so we don’t need the ability to serialize within the GPU, we just need to serialize requests to the kernel. Simple CPU-based serialization gating access to the GPU will suffice here, at least for the current set of drivers. GPU access which is not mediated by the kernel will presumably require serialization that involves the GPU itself. We’ll leave that for a future adventure though; the goal today is to build something that works with the current Linux DRM interfaces.

SyncFence Semantics

The semantics required by SyncFences is for multiple clients to block on a fence which a single client then triggers. All of the blocked clients start executing requests immediately after the trigger fires.

There are four basic operations on SyncFences:

• Trigger. Mark the fence as ready and wake up all waiting clients

• Await. Block until the fence is ready.

• Query. Retrieve the current state of the fence.

• Reset. Unset the fence; future Await requests will block.

SyncFences are the same as Events as provided by Python and other systems. Of course all of the names have been changed to keep things interesting. I’ll call them Fences here, to be consistent with the current X usage.

Using Pthread Primitives

One fact about pthreads that I recently learned is that the synchronization primitives (mutexes, barriers and semaphores) are actually supposed to work across process boundaries, if those objects are in shared memory mapped by each process. That seemed like a great simplification for this project; allocate a page of shared memory, map into the X server and direct rendering application and use the existing pthreads APIs.

Alas, the pthread objects are architecture specific. I’m pretty sure that when that spec was written, no-one ever thought of running multiple architectures within the same memory space. I went and looked at the code to check, and found that each of these objects has a different size and structure on x86 and x86_64 architectures. That makes it pretty hard to use this API within X as we often have both 32- and 64- bit applications talking to the same (presumably 64-bit) X server.

As a last resort, I read through a bunch of articles on using futexes directly within applications and decided that it was probably possible to implement what I needed in an architecture-independent fashion.

Futexes

Linux Futexes live in this strange limbo of being a not-quite-public kernel interface. Glibc uses them internally to implement locking primitives, but it doesn’t export any direct interface to the system call. Certainly they’re easy to use incorrectly, but it’s unusual in the Linux space to have our fundamental tools locked away ‘for our own safety’.

Fortunately, we can still get at futexes by creating our own syscall wrappers.

static inline long sys_futex(void *addr1, int op, int val1,
                 struct timespec *timeout, void *addr2, int val3)
{
    return syscall(SYS_futex, addr1, op, val1, timeout, addr2, val3);
}

For this little exercise, I created two simple wrappers, one to block on a futex:

static inline int futex_wait(int32_t *addr, int32_t value) {
    return sys_futex(addr, FUTEX_WAIT, value, NULL, NULL, 0);
}

and one to wake up all futex waiters:

static inline int futex_wake(int32_t *addr) {
    return sys_futex(addr, FUTEX_WAKE, MAXINT, NULL, NULL, 0);
}

Atomic Memory Operations

I need atomic memory operations to keep separate cores from seeing different values of the fence value, GCC defines a few such primitives and I picked _syncboolcompareandswap and _syncvalcompareandswap. I also need fetch and store operations that the compiler won’t shuffle around:

#define barrier() __asm__ __volatile__("": : :"memory")

static inline void atomic_store(int32_t *f, int32_t v)
{
    barrier();
    *f = v;
    barrier();
}

static inline int32_t atomic_fetch(int32_t *a)
{
    int32_t v;
    barrier();
    v = *a;
    barrier();
    return v;
}

If your machine doesn’t make these two operations atomic, then you would redefine these as needed.

Futex-based Fences

These wake-all semantics of Fences greatly simplify reasoning about the operation as there’s no need to ensure that only a single thread runs past Await, the only requirement is that no threads pass the Await operation until the fence is triggered.

A Fence is defined by a single 32-bit integer which can take one of three values:

• 0 - The fence is not triggered, and there are no waiters.
• 1 - The fence is triggered (there can be no waiters at this point).
• -1 - The fence is not triggered, and there are waiters (one or more).

With those, I built the fence operations as follows. Here’s Await:

int fence_await(int32_t *f)
{
    while (__sync_val_compare_and_swap(f, 0, -1) != 1) {
        if (futex_wait(f, -1)) {
            if (errno != EWOULDBLOCK)
                return -1;
        }
    }
    return 0;
}

The basic requirement that the thread not run until the fence is triggered is met by fetching the current value of the fence and comparing it with 1. Until it is signaled, that comparison will return false.

The compareandswap operation makes sure the fence is -1 before the thread calls futex_wait, either it was already -1 in the case where there were other waiters, or it was 0 before and is now -1 in the case where there were no waiters before. This needs to be an atomic operation so that the fence value will be seen as -1 by the trigger operation if there are any threads in the syscall.

The futex_wait call will return once the value is no longer -1, it also ensures that the thread won’t block if the trigger occurs between the swap and the syscall.

Here’s the Trigger function:

int fence_trigger(int32_t *f)
{
    if (__sync_val_compare_and_swap(f, 0, 1) == -1) {
        atomic_store(f, 1);
        if (futex_wake(f) < 0)
            return -1;
    }
    return 0;
}

The atomic compareandswap operation will make sure that no Await thread swaps the 0 for a -1 while the trigger is changing the value from 0 to 1; either the Await switches from 0 to -1 or the Trigger switches from 0 to 1.

If the value before the compareandswap was -1, then there may be threads waiting on the Fence. An atomic store, constructed with two memory barriers and a regular store operation, to mark the Fence triggered is followed by the futex_wake call to unblock all Awaiting threads.

The Query function is just an atomic fetch:

int fence_query(int32_t *f)
{
    return atomic_fetch(f) == 1;
}

Reset requires a compareandswap so that it doesn’t disturb things if the fence has already been reset and there are threads waiting on it:

void fence_reset(int32_t *f)
{
    __sync_bool_compare_and_swap(f, 1, 0);
}

A Request for Review

Ok, so we’ve all tried to create synchronization primitives only to find that our ‘obvious’ implementations were full of holes. I’d love to hear from you if you’ve identified any problems in the above code, or if you can figure out how to use the existing glibc primitives for this operation.

Last week was the OpenStack Design Summit in Portland, OR where we, developers, discussed and designed the new OpenStack release (Havana) coming up.

The summit has been wonderful. It was my first OpenStack design summit -- even more as a PTL -- and bumping into various people I've never met so far and worked with online only was a real pleasure!

Me and Nick ready to talk about Ceilometer new features.

Nick Barcet from eNovance, our dear previous Ceilometer PTL, and myself, talked about Ceilometer and presented the work that bas been done for Grizzly, with some previews of what we'll like to see done for its Havana release. You can take a look at the slides if you're curious.

Design sessions

Ceilometer had his design sessions during the last days of the summit. We noted a lot of things and commented during the sessions in our Etherpads instances.

The first session was a description of Ceilometer core architecture for interested people, and was a wonderful success considering that the room was packed. Our Doug Hellmann did a wonderful job introducing people to Ceilometer and answering question.

Doug explaining Ceilometer architecture.

The next session was about getting feedbacks from our users. We had a lot of surprise to discover wonderful real use-cases and deployments, like the CERN using Ceilometer and generating 2 GB of data per day!

The following sessions ran on Thursday and were much more about new features discussion. A lot ot already existing blueprints were discussed and quickly validated during the first morning session. Then, Sandy Walsh introduced the architecture they use inside StackTach, so we can start thinking about getting things from it into Ceilometer.

API improvements were discussed without surprises and with a good consensus on what needs to be done. The four following sessions that occupied a lot of the days were related to alarming. All were lead by Eoghan Glynn, from RedHat, who did an amazing job presenting the possible architectures with theirs pros and cons. Actually, all we had to do was to nod to his designs and acknowledge the plan on how to build this.

That last two sessions were about discussing advanced models for billing where we got some interesting feedback from Daniel Dyer from HP, and then were a quick follow-up of the StackTach presentation from the morning session.

Havana roadmap

The list of blueprints targetting Havana is available and should be finished by next week. If you want to propose blueprints, you're free to do so and inform us about it so we can validate it. The same applies if you wish to implement one of them!

API extension

I do think the API version 2 is going to be heavily extended during this release cycle. We need more feature, like the group-by functionnality.

Healthnmon

In parallel of the design sessions, discussions took place in the unconference room with the Healthnmon developers to figure out a plan in order to merge some of their efforts into Ceilometer. They should provide a component to help Ceilometer supports more hypervisors than it currently does.

Alarming

Alarming is definitely going to be the next big project for Ceilometer. Today, Eoghan and I started building blueprints on alarming, centralized in a general blueprint.

We know this is going to happen for real and very soon, thanks to the engagements of eNovance and RedHat who are commiting resources to this amazing project!

This is some news I have been wanting to share with my friends and the community for quite a while, starting on the 15th of June I will be working for Red Hat.

I will be filling Christian's position before he got promoted to lead the entire desktop group, that means I will be managing the engineers working on Evolution, Firefox and LibreOffice. This means that I will be closer to the tree upstreams I have cared the most for my entire career: GNOME, Mozilla and LibreOffice (and OO.o before them) and I get to do it in one of the greatest companies of the open source ecosystem.

I will be moving to Brno, Czech Republic, to work along side 600 redhatters in the European engineering HQs, also, this means I don't have to figure out flights and accommodation for GUADEC this year ;-)

I can't express how thrilled I am for being given this chance. This has been, however, one of the hardest decisions I had to make in my entire life.

Moving to Dublin when I was 22 to work for Sun was certainly easy, I was fed up with college at that point, I never left home before and I was young enough so that my decisions did not necessarily affect how the next 10 years would look like. It turned out quite well, I loved working for Sun and learned a lot, my English improved quite a lot and spent probably one of the best years of my life (2008).

Moving to Manchester from Dublin to work for Codethink in 2009 was also somewhat easy, most of my friends left or where about to leave Dublin and things at Sun were starting to get a bit stiff because of the acquisition process from Oracle. It was a good move after all and I fell in love with the UK and Manchester as a place to live. I also learned a great deal alongside Rob and the rest of the Codethink bunch.

However this time is different, I am 29 (yeah I know, not an old fart yet, but still...) I came back to Gran Canaria and started working for Canonical a year and a half ago after 5 years abroad, I rebuilt my social circles, I am closer to my family, I've just lost 15kg recently and I am going through a very happy stage of my life now, not to mention that moving to a country whose language I'm far from barely speaking is certainly frightening.

It's the first time this kind of move feels like a sacrifice to me, but my gut feeling keeps telling me that it will be worth it, and it has never failed me before for these kind of matters.

Exciting and challenging times ahead!

I would like to give everyone a friendly reminder that Saturday the 27th of April is the official deadline for the GUADEC 2013 Call for Papers. So make sure to get your proposal submitted.

We hope to have a wide range of talks this year, including talks on related subjects such as Wayland and Multimedia, so don’t automatically assume that you will not get a talk approved because its not ‘pure GNOME’.

GUADEC this year will be in Brno in the Czech Republic, so I hope to see as many of you as possible here.

(C) tasayu @ Flickr – CC by-nc-nd

MBIM!

Winter is over, Spring is here, and along with the heat, rains and insects, ModemManager arrives with support for modems using the new MBIM protocol, available through the new cdc-mbim kernel driver in Linux >= 3.8 and a preliminary version of libmbim.

The current implementation provides:

• PIN unlock/change/enable/disable capabilities
• 3GPP modem support (GSM, HSPA, LTE…)
• Basic connectivity support

It doesn’t provide yet most of the side-features available in other modems (e.g. SMS messaging, location…), but it is a good start. If you don’t want to use MBIM just yet, check Bjørn’s list of issues and alternatives.

Where do I get it?

The ‘libmbim’ library can be found in the following Lanedo-maintained gitorious repository:

• git: https://gitorious.org/lanedo/libmbim

 
We also released a tarball in libqmi’s release place:
http://www.freedesktop.org/software/libqmi/libmbim-0.0.1.tar.xz

If you’re part of the freedesktop.org sitewranglers group, please check the still open libmbim project request.

The support is already in ModemManager git master, and will be enabled if libmbim is found during configure. You can also explicitly request to use MBIM support with the new –with-mbim configure switch.

Thanks

Special thanks go to Lanedo for sponsoring the initial libmbim development, Bjørn Mork for his suggestions and tests with multiple MBIM-capable modems, and Huawei for sending me a E367 with MBIM firmware.

Rather by accident, Dave Airlie and I found a minor security issue in the X server last week (read the story here) This issue has been assigned CVE-2013-1940 and is now publicly available. The corresponding bug reports are here: https://bugs.freedesktop.org/63353 and https://bugzilla.redhat.com/950438.

X servers receive notifications from HAL/udev about new input devices, even when you vt-switched to the tty or another server. Input devices added while the server is not the owner of the vt will be added but not enabled, so events from such devices are ignored. On vt-switch back, the device is enabled and the fd is added to the select set used by the server. Future events will trigger a SIGIO and will be processed as expected

evdev holds the fd open between PreInit and enabling the device. If the device is hot-plugged while the server is vt-switched away events accumulate on the fd. evdev calls xf86FlushInput() to discard these events but a bug in that function made it essentially a noop for evdev devices. Thus, once the server is the VT owner again, events from that device are still on the fd and are processed whenever the next event comes along on that device.

Reproducer is fairly simple: open a text editor, vt-switch, hotplug a keyboard, type something on that keyboard, vt-switch back and the events will be replayed on the existing server.

This issue is now fixed upstream and I have released xserver 1.13.4 and xserver 1.14.1. It is of relatively low impact but the fix is easy so I recommend to patch your X servers.

Next Wednesday, April 24th, will be my last day at Canonical.

During these 18 months I've met a great bunch of people and I've been part of a worthwhile project, to bring free software to the consumer space, something I hope they achieve even if I am not part of the team anymore.

It has been an amazing ride and I think I've grown quite a lot as an engineer in the meantime, I am very greatful for their trust and support and I will miss the vibe during sprints, the colleagues and friends I've made during this time.

So what's next you ask? You'll know soon enough, all I can say for now is that after the 24th I will have some time for myself and I will sort things out before my next gig.

[read more]

DRI3K — First Steps

Here’s an update on DRI3000. I’ll start by describing what I’ve managed to get working and then summarize discussions that happened on the xorg-devel mailing list.

Private Back Buffers

One of the big goals for DRI3000 is to finish the job of moving buffer management out of the X server and into applications. The only thing still allocated by DRI2 in the X server are back buffers; everything else moved to the client side. Yes, I know, this breaks the GLX requirement for sharing buffers between applications, but we just don’t care anymore.

As a quick hack, I figured out how to do this with DRI2 today — allocate our back buffers separately by creating X pixmaps for them, and then using the existing DRI2GetBuffersWithFormat request to get a GEM handle for them.

Of course, now that all I’ve got is a pixmap, I can’t use the existing DRI2 swap buffer support, so for now I’m just using CopyArea to get stuff on the screen. But, that works fine, as long as you don’t care about synchronization.

Handling Window Resize

The biggest pain in DRI2 has been dealing with window resize. When the window resizes in the X server, a new back buffer is allocated and the old one discarded. An event is delivered to ‘invalidate’ the old back buffer, but anything done between the time the back buffer is discarded and when the application responds to the event is lost.

You can easily see this with any GL application today — resize the window and you’ll see occasional black frames.

By allocating the back buffer in the application, the application handles the resize within GL; at some point in the rendering process the resize is discovered, and GL creates a new buffer, copies the existing data over, and continues rendering. So, the rendered data are never lost, and every frame gets displayed on the screen (although, perhaps at the wrong size).

The puzzle here was how to tell that the window was resized. Ideally, we’d have the application tell us when it received the X configure notify event and was drawing the frame at the new size. We thought of a cute hack that might do this; track GL calls to change the viewport and make sure the back buffer could hold the viewport contents. In theory, the application would receive the X configure notify event, change the viewport and render at the new size.

Tracking the viewport settings for an entire frame and constructing their bounding box should describe the size of the window; at least it should describe the intended size of the window.

There’s at least one serious problem with this plan — applications may well call glClear before calling glViewport, and as glClear does not use the current viewport, instead clearing the “whole” window, we couldn’t use the viewport as an indication of the current window size.

However, what this exercise did lead us to realize was that we don’t care what size the window actually is, we only care what size the application thinks it is. More accurately, the GL library just needs to be aware of any window configuration changes before the application, so that it will construct a buffer that is not older than the application knowledge of the window size.

I came up with two possible mechanisms here; the first was to construct a shared memory block between application and X server where the X server would store window configuration changes and signal the application by incrementing a sequence number in the shared page; the GL library would simply look at the sequence number and reallocate buffers when it changed.

The problem with the shared memory plan was that it wouldn’t work across the network, and we have a future project in mind to replace GLX indirect rendering with local direct rendering and PutImage which still needs accurate window size tracking. More about that project in a future post though…

X Events to the Rescue

So, I decided to just have the X server send me events when the window size changed. I could simply use the existing X configure notify events, but that would require a huge infrastructure change in the application so that my GL library could get those events and have the application also see them. Not knowing what the application is up to, we’d have to track every ChangeWindowAttributes call and make sure the event_mask included the right bits. Ick.

Fortunately, there’s another reason to use a new event — we need more information than is provided in the ConfigureNotify event; as you know, the Swap extension wants to have applications draw their content within a larger buffer that can have the window decorations placed around it to avoid a copy from back buffer to window buffer. So, our new ConfigureNotify event would also contain that information.

Making sure that ConfigureNotify event is delivered before the core ConfigureNotify event ensures that the GL library should always be able to know about window size changes before the application.

Splitting the XCB Event Stream

Ok, so I’ve got these new events coming from the X server. I don’t want the application to have to receive them and hand them down to the GL library; that would mean changing every application on the planet, something which doesn’t seem very likely at all.

Xlib does this kind of thing by allowing applications to stick themselves into the middle of the event processing code with a callback to filter out the events they’re interested in before they hit the main event queue. That’s how DRI2 captures Invalidate events, and it “works”, but using callbacks from the middle of the X event processing code creates all kinds of locking nightmares.

As discussed above, I don’t care when GL sees the configure events, as long as it gets them before the application finds about about the window size change. So, we don’t need to synchronously handle these events, we just need to be able to know they’ve arrived and then handle them on the next call to a GL drawing function.

What I’ve created as a prototype is the ability to identify specific events and place them in a separate event queue, and when events are placed in that event queue, to bump a ‘sequence number’ so that the application can quickly identify that there’s something to process.

Making the Event Mask Per-API Instead of Per-Client

The problem described above about using the core ConfigureNotify events made me think about how to manage multiple APIs all wanting to track window configuration. For core events, the selection of which events to receive is all based on the client; each client has a single event mask, and each client receives one copy of each event.

Monolithic applications work fine with this model; there’s one place in the application selecting for events and one place processing them. However, modern applications end up using different APIs for 3D, 2D and media. Getting those libraries to cooperate and use a common API for event management seems pretty intractable. Making the X server treat each API as a separate entity seemed a whole lot easier; if two APIs want events, just have them register separately and deliver two events flagged for the separate APIs.

So, the new DRI3 configure notify events are created with their own XID to identify the client-side owner of the event. Within the X server, this required a tiny change; we already needed to allocate an XID for each event selection so that it could be automatically cleaned up when the client exited, so the only change was to use the one provided by the client instead of allocating one in the server.

On the wire, the event includes this new XID so that the library can use it to sort out which event queue to stick the event in using the new XCB event stream splitting code.

Current Status

The above section describes the work that I’ve got running; with it, I can run GL applications and have them correctly track window size changes without losing a frame. It’s all available on the ‘dri3’ branches of my various repositories for xcb proto, libxcb, dri3proto and the X server.

Future Directions

The first obvious change needed is to move the configuration events from the DRI3 extension to the as-yet-unspecified new ‘Swap’ extension (which I may rename as ‘Present’, as in ‘please present this pixmap in this window’). That’s because they aren’t related to direct rendering, but rather to tracking window sizes for off-screen rendering, either direct, indirect or even with the CPU to memory.

DRI3 and Fences

Right now, I’m not synchronizing the direct rendering with the CopyArea call; that means the X server will end up with essentially random contents as the application may be mid-way through the next frame before it processes the CopyArea. A simple XSync call would suffice to fix that, but I want a more efficient way of doing this.

With the current Linux DRI kernel APIs, it is sufficient to serialize calls that post rendering requests to the kernel to ensure that the rendering requests are themselves serialized. So, all I need to do is have the application wait until the X server has sent the CopyArea request down to the kernel.

I could do that by having the X server send me an X event, but I think there’s a better way that will extend to systems that don’t offer the kernel serialization guarantee. James Jones and Aaron Plattner put together a proposal to add Fences to the X Sync extension. In the X world, those offer a method to serialize rendering between two X applications, but of course the real goal is to expose those fences to GL applications through the various GL sync extensions (including GLARBsync and GLNVfence).

With the current Linux DRI implementation, I think it would be pretty easy to implement these fences using pthread semaphores in a block of memory shared between the server and application. That would be DRI-specific; other direct rendering interfaces would use alternate means to share the fences between X server and application.

Swap/Present — The Second Extension

By simply using CopyArea for my application presentation step, I think I’ve neatly split this problem into manageable pieces. Once I’ve got the DRI3 piece working, I’ll move on to fixing the presentation issue.

By making that depend solely on existing core Pixmap objects as the source of data to present, I can develop that without any reference to DRI. This will make the extension useful to existing X applications that currently have only CopyArea for this operation.

Presentation of application contents occurs in two phases; the first is to identify which objects are involved in the presentation. The second is to perform the presentation operation, either using CopyArea, or by swapping pages or the entire frame buffer. For offscreen objects, these can occur at the same time. For onscreen, the presentation will likely be synchronized with the scanout engine.

The second form will mean that the Fences that mark when the presentation has occurred will need to signaled only once the operation completes.

A CopyArea operation means that the source pixmap is “ready” immediately after the Copy has completed. Doing the presentation by using the source pixmap as the new front buffer means that the source pixmap doesn’t become “ready” until after the next swap completes.

What I don’t know now is whether we’ll need to report up-front whether the presentation will involve a copy or a swap. At this point, I don’t think so — the application will need two back buffers in all cases to avoid blocking between the presentation request and the presentation execution. Yes, it could use a fence for this, but that still sticks a bubble in the 3D hardware where it’s blocked waiting for vblank instead of starting on the next frame immediately.

Plan of Attack

Right now, I’m working on finishing up the DRI3 piece:

• Replace the DRI2 buffer allocation kludge with actual local buffer allocation, mapping them into pixmaps using FD passing.

• Replace the DRI2 authentication scheme with having the X server open the DRI object, preparing it for rendering and passing it back to the application.

• Working on the XCB pieces to get the split event-queue stuff landed upstream.

• Implementing the Fencing stuff to correctly serialize access to the pixmap.

The first three seem fairly straight forward. The fencing stuff will involve working with James and Aaron to integrate their XSync changes into the server.

After that, I’ll start working on the presentation piece. Foremost there is figuring out the right name for this new extension; I started with the name ‘Swap’ as that’s the GL call it implements. However, ‘Swap’ is quite misleading as to the actual functionality; a name more like ‘Present’ might provide a better indication of what it actually does. Of course, ‘Present’ is both a verb and a noun, with very different connotations. Suggestions on this most complicated part of the project are welcome!

So... I've been on an old games kick for some time now. As part of that, I recently purchased a Namco neGcon Playstation controller. I'm not going to dig out a copy of Wipeout... I want to support it in some of the demo programs I write for the graphics programming class I teach... because I can.

A tiny bit of background for people too lazy to click the Wikipedia link... This is an old Playstation controller. It came out long before even the Dual Analog (April 1997, according to the Wikipedia article. I couldn't find a firm release date, but I remember seeing adds for it around the launch time of the Playstation. I could find some rec.games.video.sony posts from March 1996 about importing it from Japan. What makes it special are the quirky analog inputs. The left trigger and the two red buttons are analog. The real kicker is twisting it in the middle is also an analog input.

I hooked it up to my laptop with a generic Playstation-to-USB converter, and hacked up a demo program in SDL to see how this thing reports itself. The first disappointing thing is the name. SDL just reports it as "USB Gamepad " (yes, with a space at the end). I'm sure that's a quirk of the adapter. Since I have several controllers that I use, I use the name to set default button mappings. My Logitech DualShock look-a-like reports as "Logitech Logitech Dual Action" (yes, Logitech twice), and my PS3 Sixaxis reports as "Sony PLAYSTATION(R)3 Controller".

It shows up as 5 axes, 12 buttons, and a hat. Let's look at the mapping to see it get crazy:

• Twisting: axis 0
• Buttons I and II (the red ones): axis 1. Yes, the two jolly, red, candy-like analog buttons show up, together, as axis 1. Button I gets the negative values, and button II gets the positive values.
• Button A: button 1
• Button B: button 0
• Left trigger: axis 3
• Right trigger: button 7
• Start: button 9
• D-pad: Hat 0. I hate controllers that advertise the d-pad as a hat. My Logitech controller does that, but the Sixaxis just shows them as buttons.

All of this begs the question, "WTF?" It also begs a couple follow-up questions. Is all of this madness caused by the adapter, or is it endemic to the controller itself? I suspect it shows up as 12 buttons because of the DualShock. The DualShock actually has 13 buttons (the "Analog" selector), but I don't think that gets sent over the protocol. I think that just changes the function of the controller itself. My Logitech has a similar "mode" button, and that doesn't go over the protocol.

Did anyone ever use a neGcon with a parallel port adpater? How did it show up? It looks like the Linux kernel has supported it for ages, so someone must have done it...

* Update all the code necessary to run bfgminer has been pushed upstream

On Monday, I started to work on getting one of the many bitcoin mining applications, bfgminer, to work with clover and r600g.  Since then, I have uncovered and fixed a number of bugs in the compute stack and now with some final bug fixes from R600 backend developer Vincent Lejeune, I have bfgminer working with the Open Source drivers! (Also, thanks to Aaron Watry for implementing the rotate builtin for libclc!)

If anyone is interested in testing, you can follow the installation instructions for gallium compute here, and instead of the standard branches listed in the instructions checkout r179204 from the clang tree, and use the bfgminer branch from the following repos:

git://people.freedesktop.org/~tstellar/mesa
git://people.freedesktop.org/~tstellar/llvm
git://people.freedesktop.org/~tstellar/libclc

Additionally, you will need to invoke bfgminer using the command line options in this script (you will need to modify the script to add your preferred bitcoin address). Currently, bfgminer will only work on Evergreen and non-Cayman Northern Islands GPUs.

There is still one bug that will occasionally cause a hash value to be miscalculated.  When this happens, bfgminer will report a hw error, but this is not serious and the program will continue to run.

The compiled kernel code for bfgminer is not very well optimized yet and there are still a lot of things that can be done to improve performance.  For now, I’m going to work on getting these patches upstream and writing some piglit tests for the bugs, but maybe in the future I’ll have some time take a closer look at the kernel performance.  I have a few ideas for things to do to improve performance, so if anyone is interested in playing around with the code, just ping me on irc.  My nick is tstellar, and I’m usually in #radeon and #dri-devel on irc.freenode.net.  There may even be a few things that would make good Google Summer of Code projects for any students out there interested in GPU performance and/or bitcoin.

* Update: bfgminer now autodetects the Mesa platform, so the bfgminer patch is no longer required.

• An Sandybridge/Ivybridge Intel GPU (Intel HD 2000/3000/2500/4000 in marketing speak).
• A "enthusiast" motherboard which allows you to set gpu overclocking settings (higher frequency and voltage).
• Lastet drm-intel-nightly kernel branch from drm-intel git or at least a kernel built with this patch from Ben Widawsky. The patch should apply to pretty much any recent stable kernel. Without that patch gpu turbo support is broken.
• Preferably a desktop gpu with a big cooling rig. Overclocking on modern intel chips essentially just increases the turbo headroom into the unvalidated range (with the potential for hangs and corruptions in rendering). And if your cooling system can't keep up with the increased heat output the on-chip controller will quickly clamp your clocks down to the non-turbo frequency.
• Your favorite gpu benchmark. The reason for that is the thermal throttling when running in the turbo range: It happens behind the driver's back and the only observable effect is a slower gpu - the current frequency value in sysfs is still the same!

So, in my earlier post, I mentioned that I’d been working on a library called xkbcommon. Since it’s got a massively misleading name (and renaming hasn’t been totally ruled out yet …), it’s probably worth an introductory post.

tl;dr: It loads XKB keymaps and can manage tricky things like modifier state for you.

tell me more!

At its core, xkbcommon is nothing to do, as you might think, with X11. It’s about two things: parsing and loading keymaps, and managing their ongoing state. State, in terms of keyboards, is the usual suspects — modifiers (e.g. Shift, Alt), multiple layouts (mostly for multiple languages), and LEDs. In general, you only want one person keeping a canonical copy of the state, and distributing it to its clients, as both the X server and Wayland do today. xkbcommon allows for this mode of operation, and is indeed how all current Wayland clients handle keyboard input.

One notable thing xkbcommon isn’t, is an input method. Anything more complex than ‘I press one key and get a symbol’, such as phonetic compound composition for CJK/Thai/etc (where you type out a word, which, when completed, becomes much fewer characters on screen) or menu-based selections as common in both Arabic and Mandarin (where you type out the beginnings of a word, and are then offered a selection of choices to complete), or even just the seemingly-straightforward compose key (e.g. right Alt + o + / → ø), are implemented internally to the toolkit, and don’t really use xkbcommon at all. Supporting the two (key-based vs. string-based) methods in the same protocol seems straightforward, but quickly becomes hugely unwieldy if you ever need to do anything weird like, say, process shortcuts. Anyway, if you want to find out more about input methods on Wayland, Michael Hasselmann has a very nice update about Maalit and ibus.

This makes life rather more simple for the keyboard-based protocols: all we have to do is tell xkbcommon every time someone presses (or releases) a key, and ask it which symbols resulted from that.

There are two methods of operation. The self-contained approach, suitable if you’re writing programs which back directly on to a terminal and have absolute full control over the entire keyboard, is used by most of our tests, such as interactive.c, which is a reasonably summary of how to implement one of these clients. Here, the same client manages the entire state, and never needs to deal with any external entities.

However, the most common method of operation is the one used by Wayland and X11, where a central server manages the state and informs the clients. Under this model, the clients never update the state implicitly through xkb_state_update_key(); they only ever use xkb_state_update_mask() in response to its master sending it a new and complete copy of the state.

ok — so how do I use it?

Firstly, create a context:

ctx = xkb_context_new(0);

which will be the base for all your xkbcommon operations. Secondly, we need to compile a keymap. If you have a named keymap you want to compile from (e.g. the user has specified ‘Icelandic Dvorak with Caps Lock mapped to Ctrl’), use:

keymap = xkb_keymap_new_from_names(ctx, rules, model, layout, variant, options);

On Linux, the rules and model are almost always ‘evdev’. Layout is usually the ISO two-character country (not language!) code, variant is specific to the layout, and options are global. In this case, we would use:

keymap = xkb_keymap_new_from_names(ctx, "evdev", "evdev", "is", "dvorak", "ctrl:nocaps");

A keymap is a static and immutable ruleset describing the translation between key events and the resulting output (e.g. if d is pressed while shift is held down, then output D). On top of the keymap, for every keyboard we use, we generate a new state object:

state = xkb_state_new(keymap);

which will track things like which keys are currently held down. Every time a key is pressed or released, we first get the symbols it produces:

sym = xkb_state_key_get_one_sym(state, keycode);

which returns a key symbol (listed in xkbcommon-keysyms.h, and identical to the X11 keysyms; xkb_keysym_to_utf8() may be used to translate this to a string)  generated by that keypress. For example, pressing the B key on Icelandic Dvorak with AltGr held down would return XKB_KEY_ssharp (i.e. ß). After we have got the symbol to be used, we then update the state object with the keypress. It is crucial the order of these two operations is not reversed! Anyhow, we update the state object like so for sole-control programs:

changed = xkb_state_update_key(state, keycode, XKB_KEY_DOWN);

or thusly for clients who get their master state from an external program, e.g. Wayland compositor:

changed = xkb_state_update_mask(state, mods_down, mods_latched, mods_locked, ...);

and we have now completed our key processing. Hurrah!

Note that you do not need any special cases anywhere: xkbcommon takes care of the mechanics of handling special keys under the hood for you. The one case where you need extended functionality is shortcut processing (e.g. capture Ctrl+P), but that’s a story for another day.

shouldn’t all this be in the documentation?

… yeah.

PS: Funny story: I was planning to explain in this footnote how I’d just picked Icelandic Dvorak for a laugh and no-one would make a layout that ridiculous. But no, it actually exists …

Some sad news, GStreamer did not get accepted to the 2013 Google Summer of Code. We don’t know the exact reasons, but all we can do at this point is redouble our efforts to get accepted again next year.

In the meantime not all hope is lost. You can still do a Google Summer of Code involving GStreamer with some of the other organisations that did get approved. For instance if you have an idea for a great multimedia desktop application using GStreamer you can try submitting that to for instance GNOME or KDE depending on your UI toolkit of choice as one example. In the past GNOME has also been open to hosting some pure GStreamer projects so you can always try submitting such too. In general I recommend people to take a look at the list of accepted organisations and see if there are projects which technologies would be relevant to the GSoC project you want to do and the apply with them. Also remember you can submit the same project to multiple organisations if it fits multiple projects.

Also the PiTiVi video editor did get approved so I strongly encourage you to take a look at their wonderful ideas page and submit a proposal to do a PiTiVi project.

My fellow Berliners! There's another Berlin Open Source Meetup scheduled for this Sunday! You are invited!

See you on Sunday!

Hello everyone! I am very excited to report about the awesome progress we made with Tanglu, the new Debian-based Linux-Distribution.

First of all, some off-topic info: I don’t feel comfortable with posting too much Tanglu stuff to Planet-KDE, as this is often not KDE-related. So, in future Planet-KDE won’t get Tanglu information, unless it is KDE-related You might want to take a look at Planet-Tanglu for (much) more information.

So, what happened during the last weeks? Because I haven’t had lectures, I nearly worked full-time on Tanglu, setting up most of the infrastructure we need. (this will change with the next week, where I have lectures again, and I also have work to do on other projects, not just Tanglu ^^) Also, we already have an awesome community of translators, designers and developers. Thanks to them, the Tanglu-Website is now translated to 6 languages, more are in the pipeline and will be merged later. Also, a new website based on the Django framework is in progress.

The Logo-Contest

We’ve run a logo-contest, to find a new and official Tanglu logo, as the previous logo draft was too close to the Debian logo (I asked the trademark people at Debian). More than 30 valid votes (you had to be subscribed to a Tanglu Mailinglist) were received for 7 logo proposals, and we now have a final logo:

I like it very much

Fun with dak

I decided to use dak, the Debian Archive Kit, to handle the Tanglu archive. Choosing dak over smaller and easy-to-use solutions had multiple reasons, the main reason is that dak is way more flexible than the smaller solution (like reprepro or min-dak) and able to handle the large archive of Tanglu. Also, dak is lightning fast. And I would have been involved with dak sooner or later anyway, because I will implement the DEP-11 extension to the Debian Archive later (making the archive application-friendly).

Working with dak is not exactly fun. The documentation is not that awesome, and dak contains many hardcoded stuff for Debian, e.g. it often expects the “unstable” suite to be present. Also, running dak on Debian Wheezy turned out to be a problem, as the Python module apt_pkg changed the API and dak naturally had problems with that. But with the great help of some Debian ftpmasters (many thanks to that!), dak is now working for Tanglu, managing the whole archive. There are still some quirks which need to be fixed, but the archive is in an usable state, accepting and integrating packages.

The work on dak is also great for Debian: I resolved many issues with non-Debian dak installations, and made many parts of dak Wheezy-proof. Also, I added a few functions which might also be useful for Debian itself. All patches have of course been submitted to upstream-dak.

Wanna-build and buildds

This is also nearly finished Wanna-build, the software which manages all buildds for an archive, is a bit complicated to use. I still have some issues with it, but it does it’s job so far. (need to talk to the Debian wanna-build admins for help, e.g. wanna-build seems to be unable to handle arch:all-only packages, also, build logs are only submitted in parts)

The status of Tanglu builds can be viewed at the provisoric Buildd-Status pages.

Setting up a working buildd is also a tricky thing, it involves patching sbuild to escape bug #696841 and applying various workarounds to make the buildd work and upload packages correctly. I will write instructions how to set-up and maintain a buildd soon. At time, we have only one i386 buildd up and running, but more servers (in numbers: 3) are prepared and need to be turned into buildds.

After working on Wanna-build and dak, I fully understand why Canonical developed Launchpad and it’s Soyuz module for Ubuntu. But I think we might be able to achieve something similar, using just the tools Debian already uses (maybe a little less confortable than LP, but setting up an own LP instance would have been much more trouble).

Debian archive import

The import of packages from the Debian archive has finished. Importing the archive resulted in many issues and some odd findings (I didn’t know that there are packages in the archive which didn’t receive an upload since 2004!), but it has finally finished, and the archive is in a consistent state at time. To have a continuous package import from Debian while a distribution is in development, we need some changes to wanna-build, which will hopefully be possible.

Online package search

The Online-Package-Search is (after resolving many issues, who expected that? ) up and running. You can search for any package there. Some issues are remaining, e.g. the file-contents-listing doesn’t work, and changelog support is broken, but the basic functionality is there.

Tanglu Bugtracker

We now also have a bugtracker which is based on the Trac software. The Tanglu-Bugtracker is automatically synced with the Tanglu archive, meaning that you find all packages in Trac to report bugs against them. The dak will automatically update new packages every day. Trac still needs a few confort-adjustments, e.g. submitting replies via email or tracking package versions.

Tanglu base system

The Tanglu metapackages have been published in a first alpha version. We will support GNOME-3 and KDE4, as long as this is possible (= enough people working on the packaging). The Tanglu packages will also depend on systemd, which we will need in GNOME anyway, and which also allows some great new features in KDE.

Side-effect of using systemd is – at least for the start – that Tanglu boots a bit slow, because we haven’t done any systemd adjustments, and because systemd is very old. We will have to wait for the systemd and udev maintainers to merge the packages and release a new version first, before this will improve. (I don’t want to do this downstream in Tanglu, because I don’t know the plans for that at Debian (I only know the information Tollef Fog Heen & Co. provided at FOSDEM))

The community

The community really surprised me! We got an incredible amount of great feedback on Tanglu, and most of the people liked the idea of Tanglu. I think we are one of the less-flamed new distributions ever started . Also, without the very active community, kickstarting Tanglu would not have been possible. My guess was that we might be able to have something running next year. Now, with the community help, I see a chance for a release in October

The only thing people complained about was the name of the distribution. And to be really honest, I am not too happy with the name. But finding a name was an incredibe difficult process (finding something all parties liked), and Tanglu was a good compromise. “Tanglu” has absolutely no meaning, it was taken because it sounded interesting. The name was created by combining the Brazilian “Tangerine” (Clementine) and the German “Iglu” (igloo). I also dont think the name matters that much, and I am more interested in the system itself than the name of it. Also, companies produce a lot of incredibly weird names, Tanglu is relatively harmless compared to that .

In general, thanks to everyone participating in Tanglu! You are driving the project forward!

The first (planned) release

I hereby announce the name of the first Tanglu release, 1.1 “Aequorea Victoria“. It is Daniel’s fault that Tanglu releases will be named after jellyfishes, you can ask him why if you want I picked Aequorea, because this kind of jellyfish was particularly important for research in molecular biology. The GFP protein, a green fluorescent protein (=GFP), caused a small revolution in science and resulted in a Nobel Prize in 2008 for the researchers involved in GFP research (for the interested people: You can tag proteins with GFP and determine their position using light microscopy. GFP also made many other fancy new lab methods possible).

Because Tanglu itself is more or less experimental at time, I found the connection to research just right for the very first release. We don’t have a time yet when this version will be officially released, but I expect it to be in October, if the development speed increases a little and more developers get interested and work on it.

Project Policy

We will need to formalize the Tanglu project policy soon, both the technical and the social policies. In general, regarding free software and technical aspects, we strictly adhere to the Dbian Free Software Guidelines, the Debian Social Contract and the Debian Policy. Some extra stuff will be written later, please be patient!

Tanglu OIN membership

I was approached by the Open Invention Network, to join it as member. In general, I don’t have objections to do that, because it will benefit Tanglu. However, the OIN has a very tolerant public stance on software patents, which I don’t like that much. Debian did not join the OIN for this reason. For Tanglu, I think we could still join the OIN without someone thinking that we support the stance on software patents. Joining would simply be pragmatic: We support the OIN as a way to protect the Linux ecosystem from software patents, even if we don’t like the stance on software patents and see it differently.

Because this affects the whole Tanglu project, I don’t want to decide this alone, but get some feedback from the Tanglu community before making a decision.

Can I install Tanglu now?

Yes and no. We don’t provide installation images yet, so trying Tanglu is a difficult task (you need to install Debian and then upgrade it to Tanglu) – if you want to experiment with it, I recomment trying Tanglu in a VM.

I want to help!

Great, then please catch one of us on IRC or subscribe to the mailinglists. The best thing is not to ask for work, but suggest something you want to do, others will then tell you if that is possible and maybe help with the task.

Packages can for now only be uploaded by Debian Developers, Ubuntu Developers or Debian Maintainers who contacted me directly and whose keys have been verified. This will be changed later, but at the current state of the Tanglu archive (= less safety checks for packages), I only want people to upload stuff who definitely have the knowledge to create sane packages (you can also proove that otherwise, of course). We will later establish a new-member process.

If you want to provide a Tanglu archive mirror, we would be very happy, so that the main server doesn’t have to carry all the load.

If you have experience in creating Linux Live-CDs or have worked with the Ubiquity installer, helping with these parts would be awesome!

Unfortunately, we cannot reuse parts of Linux Mint Debian, because many of their packages don’t build from source and are repackaged binaries, which is a no-go for the Tanglu main archive.

Sneak peek

And here is a screenshot of the very first Tanglu installation (currently more Debian than Tanglu):

Something else…

I am involved in Debian for a very long time now, first as Debian Maintainer and then as Debian Developer – and I never thought much about the work the Debian system administrators do. I didn’t know how dak worked or how Wanna-build handles the buildds and what exactly the ftpmasters have to do. By not knowing, I mean I knew the very basic theory and what these people do. But this is something different than experiencing how much work setting up and maintaining the infrastructure is and what an awesome job the people do for Debian, keeping it all up and running and secure! Kudos for that, to all people maintaining Debian infrastructure! You rock! (And I will never ever complain about slow buildds or packages which stay in NEW for too long )

hi

Yeesh, it really has been a while, hasn’t it.

Last time we spoke, I was working on X11, and my blog was being served off a FreeBSD 4.3 machine (obviously named ‘amnesiac’), from the datacentre of an ISP who had long since forgotten which ex-employee’s Pentium II was sitting in that rack. Unfortunately I didn’t back it up, and no longer even have a copy of the RSS feed to import. Looking over the Wayback Machine copy though, I don’t think we’re missing much by losing that one. So here we are with WordPress, having blatantly ripped off Lucas’s theme.

Shortly after that machine died — early in 2012 — I finished up my last proper X11 project for a customer, went back to Australia for linux.conf.au and a holiday, and came back to start working on Wayland’s input subsystem (including xkbcommon, on which more in a future post). Working on Wayland has been really refreshing; not only did it feel like time for a change after nine years of working on X11, but diminishing returns meant that we were increasingly spending enormous amounts of time on X11 to achieve very marginal results.

The last major thing I did there was work with Chase Douglas (ex-Canonical) and Peter Hutterer (Red Hat) — though sadly due to timing, we almost entirely worked in relay, rather than together — on the XI2.3 multitouch protocol and implementation. I was quite proud to see it eventually make its way upstream, but having subsequently both tried to explain to others how it all hangs together, and watched Peter continue to fix corner-case after corner-case in its interactions with other parts of X11 input, it really solidified my view that X11 is a dead end for any new development.

stop whining and get on with it

A couple of months ago, in Canberra for linux.conf.au 2013, I gave a presentation about that very realisation.  Originally I thought of giving a kind of dispassionate comparison and explanation of the two, but it ended up becoming a fairly linear narrative of where X began, where it is now, and how Wayland organically developed from this. It’s a fairly well-worn line that Wayland is in many ways the culmination of the last ten years of X.Org development, but it really is true.

Part of that has not just been shedding the X11 cruft and getting the opportunity to start over ­­— though, given the state of the X11 protocol and its 28 common extensions, plus its codebase, that’s really no bad thing — but some radically different design decisions, down to the real fundamentals of the interfaces. Giving compositors a lot more context opens up a huge amount of possibilities for us (beginning with actually being able to activate a screensaver while a pop-up menu’s active …), and I’m really excited about the kinds of things we can do.

It’s been really interesting to see Wayland grow and develop into a fully-fledged window system. At Collabora, we’ve had Pekka, Louis-Francis, Fred and others working on Wayland support across the board: compositors, client toolkits/apps, and hardware enablement too. After the groundwork we’ve been doing, a lot of it is really starting to come to fruition and hopefully we’ll have some more shiny demos soon. Along the way, I managed to fulfill one of my goals I set a while ago, which was to get out of the bubble of only ever working on the server (since X clients were so uniformly unpleasant), and work throughout the stack. Over the past few months, I’ve been able to work on GStreamer, WebKit, Clutter, COGL, Mesa, and other EGL/GLES stacks, which has been really interesting.

where next?

With GTK+, Qt and Clutter all shipping Wayland support in their stable releases, the next big thing for me personally is watching the GNOME port progress. We’ve got a stable and solid core protocol and library now, but there are definitely some gaps to be filled in by having a real desktop environment ported, and certainly around input we’ve got a lot of things to support (e.g. touchpad support on par with xf86-input-synaptics, better mouse acceleration, support for synchronising keyboard repeat rates between clients, tablets, etc). But it’s quite exciting to see what people will be able to make of our work so far.

So thanks to crowd sourcing we are getting a lot of linux games coming out, titles such as Double Fine Adventure, Wasteland 2, Spaceventure, Project Eternity, Hero-U: Rogue to Redemption, Godus, Torment: Tides of Numenera,Arcade Racer and a lot more are all coming to the Linux platform thanks to crowdsourcing.

The question that one would want to answer is if crowd sourcing can work for open source projects creating useful software or other kind of tools. One project I am really hoping gets funded currently is Geary, the open source email client from the great guys at Yorba. Not that I personally are desperate for a new email client, I am a happy user of the Evolution email client, but I also know that email is a very personal thing for many people and an area where having different client offering different experiences might make a difference. Also proving that a non-profit outfit like Yorba can fund themselves through crowdsourcing is an important thing to demonstrate. So I have already pledged to Geary and I hope you will too.

They are not the only such project out there of course. Another project I have pledged to is the Phantom Open Emoji project which wants to create a full set of liberally licensed emoji/smileys covering the Unicode 6.0 set of over 800 such things. They only need to reach 25,000 USD in funding to create the full set, so I hope they can make that goal and we can then include support for these in Empathy.

The final project I want to mention is the OpenShot kickstarter. While I do think the best solution would be a GStreamer based one like PiTiVi for various reasons I am still happy that this kickstarter has reached is minimum funding goal already as it is does show success at funding open source development projects.

That said what strikes me is that the 3 open source projects above are all actually quite cheap, in the sense that the funding amounts they ask for are not high at all. And when you consider that games such as Torment reach 4 million USD in crowd sourcing funding one could wish that people would be a bit more prepared to bankroll open source projects. That said I guess just like games had their breakthrough moment with the Double Fine Adventure kickstarter, maybe open source development needs its own shinning star to lead the way. So if you haven’t already please pledge to one or more of the open source efforts above.

There will be more attempts of exploring this space though I am sure, I am even planning to be involved in one such effort myself, but more details on that later.

• Trimmed down API, just for positioning
• Smaller, but more integrated, selection of ways to get the positioning (GPS from your integrated WWAN modem, Wi-Fi AP data, IP address, no discrete GPS or manual location)
• Power-saving, by aggregating requests from all the applications
• and finally, privacy, where only applications that you allow to request your location can get it, and only with an accuracy as needed for the application.

I've meant to look at Hy since Paul Tagliamonte started to talk to me about it, but never took a chance until now. Yesterday, Paul indicated it was a good time for me to start looking at it, so I spent a few hours playing.

But what's Hy?

Python is very nice: it has a great community and a wide range of useful libraries. But let's face it, it misses a great language.

Hy is an implementation of a Lisp on top of Python.

Technically, Hy is built directly with a custom made parser (for now) which then translates expressions using the Python AST module to generate code, which is then run by Python. Therefore, it shares the same properties as Python, and is a Lisp-1 (i.e. with a single namespace for symbols and functions).

If you're interested to listen Paul talking about Hy during last PyCon US, I recommend watching his lightning talk. As the name implies, it's only a few minutes long.

Does it work?

I've been cloning the code and played around a bit with Hy. And to my greatest surprise and pleasure, it works quite well. You can imagine writing Python from there easily. Part of the syntax smells like Clojure's, which looks like a good thing since they're playing in the same area.

You can try a Hy REPL in your Web browser if you want.

Here's what some code look like:

(import requests)
 
(setv req (requests.get "http://hy.pault.ag"))
(if (= req.status_code 200)
  (for (kv (.iteritems req.headers))
    (print kv))
  (throw (Exception "Wrong status code")))

This code would ouput:

('date', 'Wed, 03 Apr 2013 12:09:23 GMT')
('connection', 'keep-alive')
('content-encoding', 'gzip')
('transfer-encoding', 'chunked')
('content-type', 'text/html; charset=utf-8')
('server', 'nginx/1.2.6')

As you can see, it's really simple to write Lispy code that really uses Python idioms.

There's obviously still a lots of missing features in Hy. The language if far from complete and many parts are moving, but it's really promising, and Paul's doing a great job implementing every idea.

I actually started to hack a bit on Hy, and will try to continue to do so, since I'm really eager to learn a bit more about both Lisp and Python internals in the process. I've already send a few patches on small bugs I've encountered, and proposed a few ideas. It's really exciting to be able to influence early a language design that I'll love to use! Being a recent fan of Common Lisp, I tend to grab the good stuff from it to add them into Hy.

As those of you following the GStreamer development mailing list or the GStreamer Google Plus profile know, we have been having a GStreamer hackfest in Milan over the last few days. We have 17 people here, all hammering away at our laptops or discussing various technical challenges sitting at a nice place called the Milan Hub.

A lot of progress has been made during these days with some highlights including work on fixing the use of Gnonlin with GStreamer 1.0, which is a prerequisite for getting PiTiVi and Jokosher running with GStreamer 1.0. Jeff Fortin, Thibault Saunier, Nicolas Dufresne, Edward Hervey, Peteris Krishanis and Emanuele Aina has all been helping out with this in addition to fixing various other issues in PiTiVi and Jokosher.

Sebastian Dröge has put a lot of work during the hackfest into providing the basic building blocks for doing hardware codecs nicely in GStreamer, and Víctor Jáquez has been working on making VAAPI work well using these building blocks, with the plan among other things to make sure you have hardware accelerated decoding working with WebKit. In that regards Philippe Normand has spent the hackfest investigating and improving various bits of the GStreamer backend in Webkit, like improving the on-disk buffering method used. Also in terms of hardware codec support Edward Hervey also found a bit of time to work a little on the VDPAU plugins.

Speaking of web browsers Alessandro Decina has been working on porting Firefox to GStreamer 1.0, he has also been our local host making sure we have found places to eat lunch and dinner that where able to host our big group. So a big thank you to Alessandro for this.

Wim Taymans has been working on properly dealing with chroma keying in GStreamer, improving picture quality significantly in some cases, in addition to being constantly barraged with questions and discussions about various enhancements, bugs and other challenges.

Edward Hervey has in addition to help out with GNonlin also been working on improvements in our DVB support and improving encodebin so that you can now request a named profile when requesting pads, the last item being a crucial piece in terms of allowing me to proceed with Transmageddons multistream support.

Stefan Sauer spent time on fixing various bugs in the GStreamer 1.0 port of Buzztard and a first stab at designing a tracing framework for GStreamer.

Arun Raghavan was working on various bugs related to Pulse Audio and GStreamer and also implemented a SBC RTP depayloader element for GStreamer.

Tim-Philipp Müller has been working on implementing a stream selection flag in order for GStreamer player to be able to follow any in-file hints about which streams to default to or to not default to for that matter.

As for myself I been mostly working on Transmageddon trying to get the multistream and DVD support working. Thanks to some crucial bugfixes from Edward Hervey and Wim Taymans I was able to make good progress and I have ripped my first DVD with Transmageddon now. There is still a lot of work that needs doing, both in terms of presentation, features and general robustness, but I am very pleased by the progress made.

Title selection screen, needs a bit more polish, but getting there.

As you see above you can now choose to transcode to different codecs for each sound stream, or drop the streams you don’t care about. The main usecase for different codecs is to you a different codecs for surround sound as opposed to stereo or mono streams.

A big thank you to Collabora and Fluendo for sponsoring us with dinner during the hackfest.

Also a big thank you to Collabora, Fluendo, Google, Igalia, Red Hat and Spotify for letting their employees attend the hackfest.

Hi Everyone,

It’s been a while since my last update (over a month!) so it’s a good time to talk about what’s been going on.

Firstly, GVFS 1.16 is out – so that’s the first stable release with the MTP backend in it. w00t!

Before you wonder, it doesn’t include my work to support the Android Direct I/O extensions (that allow normal read/write access to files on the device). I’ve now got those to a point where I’m ready to get them in, but I’m waiting on a review in bugzilla. Since my last update, all the libmtp changes have been merged and released in version 1.1.16.

The second big thing I’ve done is completely change how mtp URIs work. In previous posts, I’ve talked about how I was putting entity IDs as path elements to save having to maintain an ID->filename mapping, and then relying on the gvfs display and copy name properties to make the files appear to have normal names when looked at. I ultimately decided to abandon this approach for a couple of reasons. The main one is that with Direct I/O support, every application that can operate on files can be used with an MTP device, and most of those apps don’t know anything about gvfs and can’t use the special properties. The second reason is that there are edge cases where it’s impossible to tell if you’re looking at a filename that’s all numbers or an entity ID. So, I’ve added a mapping system and URIs now use filenames.

Finally, I’ve fixed a bug in gvfs that only got triggered when unmounting an mtp device in Ubuntu 13.04 betas. The code in question hasn’t changed in gvfs for a long time, but the bug didn’t appear anywhere else. Still, there is a real code problem in there, so I’ve got a fix out for it.

I’ve updated my with builds that contain all these pending patches (although the raring gvfs got updated while mine was building so it’s now considered out-of-date) and the new libmtp, so please try the new stuff out.

For the curious, here are the GNOME bugzilla entries tracking these changes:

• Add support for Android direct I/O extensions
• Implement filename cache to enable name based URIs
• gvfsmonitor.c: backend_died needs to take a ref while unsubscribing subscribers

Enjoy!

While working on the XLcloud project (HPC on cloud) it appeared clear to us that OpenStack was missing a critical component towards resource reservations.

A capacity leasing service is something really needed by service providers, especially in the context of cloud platforms dedicated to HPC style workload. Instead of building something really specific, the decision has been made to build a new standalone OpenStack components aiming to provide this kind of functionnality to OpenStack. In the spirit of others OpenStack components, it will be extensible to fullfil a large panel of needs around this problematic.

The project is named Climate, and is hosted on StackForge. It will follow the standard OpenStack development modal. This service will be able to handle a calendar of reservations for various resources, based on various criteria.

The project is still at its early design stage, and we plan to have a unconference session during the next OpenStack summit in Portland to present our plans and ideas for the future!

Update: This isn't actually that much better than letting them access the private key, since nothing is stopping the user from running their own SSH agent, which can be run under strace. A better solution is in the works. Thanks Timo Juhani Lindfors and Bob Proulx for both pointing this out.

At work, we have a shared SSH key between the different people manning the support queue. So far, this has just been a file in a directory where everybody could read it and people would sudo to the support user and then run SSH.

This has bugged me a fair bit, since there was nothing stopping a person from making a copy of the key onto their laptop, except policy.

Thanks to a tip, I got around to implementing this and figured writing up how to do it would be useful.

First, you need a directory readable by root only, I use /var/local/support-ssh here. The other bits you need are a small sudo snippet and a profile.d script.

My sudo snippet looks like:

Defaults!/usr/bin/ssh-add env_keep += "SSH_AUTH_SOCK"
%support ALL=(root)  NOPASSWD: /usr/bin/ssh-add /var/local/support-ssh/id_rsa

Everybody in group support can run ssh-add as root.

The profile.d goes in /etc/profile.d/support.sh and looks like:

if [ -n "$(groups | grep -E "(^| )support( |$)")" ]; then
    export SSH_AUTH_ENV="$HOME/.ssh/agent-env"
    if [ -f "$SSH_AUTH_ENV" ]; then
        . "$SSH_AUTH_ENV"
    fi
    ssh-add -l >/dev/null 2>&1
    if [ $? = 2 ]; then
        mkdir -p "$HOME/.ssh"
        rm -f "$SSH_AUTH_ENV"
        ssh-agent > "$SSH_AUTH_ENV"
        . "$SSH_AUTH_ENV"
    fi
    sudo ssh-add /var/local/support-ssh/id_rsa
fi

The key is unavailable for the user in question because ssh-add is sgid and so runs with group ssh and the process is only debuggable for root. The only thing missing is there's no way to have the agent prompt to use a key and I would like it to die or at least unload keys when the last session for a user is closed, but that doesn't seem trivial to do.

When I’ve wanted to play in some new areas lately, it’s been a real frustration because Gentoo hasn’t had a complete set of packages ready in any of them. I feel like these are some opportunities for Gentoo to be awesome and gain access to new sets of users (or at least avoid chasing away existing users who want better tools):

• Data science. Package Hadoop. Package streaming options like Storm. How about related tools like Flume? RabbitMQ is in Gentoo, though. I’ve heard anecdotally that a well-optimized Hadoop-on-Gentoo installation showed double-digit performance increases over the usual Hadoop distributions (i.e., not Linux distributions, but companies specializing in providing Hadoop solutions). Just heard from Tim Harder (radhermit) than he’s got some packages in progress for a lot of this, which is great news.
• DevOps. This is an area where Gentoo historically did pretty well, in part because our own infrastructure team and the group at the Open Source Lab have run tools like CFEngine and Puppet. But we’re lagging behind the times. We don’t have Jenkins or Travis. Seriously? Although we’ve got Vagrant packaged, for example, we don’t have Veewee. We could be integrating the creation of Vagrant boxes into our release-engineering process.
• Relatedly: Monitoring. Look for some of the increasingly popular open-source tools today, things like Graphite, StatsD, Logstash, Lumberjack, ElasticSearch, Kibana, Sensu, Tasseo, Descartes, Riemann. None of those are there.
• Cloud. Public cloud and on-premise IaaS/PaaS. How about IaaS: OpenStack, CloudStack, Eucalyptus, or OpenNebula? Not there, although some work is happening for OpenStack according to Matthew Thode (prometheanfire). How about a PaaS like Cloud Foundry or OpenShift? Nope. None of the Netflix open-source tools are there. On the public side, things are a bit better — we’ve got lots of AWS tools packaged, even stretching to things like Boto. We could be integrating the creation of AWS images into our release engineering to ensure AWS users always have a recent, official Gentoo image.
• NoSQL. We’ve got a pretty decent set here with some holes. We’ve got Redis, Mongo, and CouchDB not to mention Memcached, but how about graph databases like Neo4j, or other key-value stores like Riak, Cassandra, or Voldemort?
• Android development. Gentoo is perfect as a development environment. We should be pushing it hard for mobile development, especially Android given its Linux base. There’s a couple of halfhearted wiki pages but that does not an effort make. If the SDKs and related packages are there, the docs need to be there too.

Where does Gentoo shine? As a platform for developers, as a platform for flexibility, as a platform to eke every last drop of performance out of a system. All of the above use cases are relevant to at least one of those areas.

I’m writing this post because I would love it if anyone else who wants to help Gentoo be more awesome would chip in with packaging in these specific areas. Let me know!

Update: Michael Stahnke suggested I point to some resources on Gentoo packaging, for anyone interested, so take a look at the Gentoo Development Guide. The Developer Handbook contains some further details on policy as well as info on how to get commit access by becoming a Gentoo developer.

make ARCH=arm USE_LIMARE=1

cg_drawfps 1
timedemo 1
set demodone  "quit"
set demoloop1 "demo four; set nextdemo vstr demodone"
vstr demoloop1

In the Porter/Duff compositing algebra, images are equipped with an alpha channel that determines on a per-pixel basis whether the image is there or not. When the alpha channel is 1, the image is fully there, when it is 0, the image isn’t there at all, and when it is in between, the image is partially there. In other words, the alpha channel describes the shape of the image, it does not describe opacity. The way to think of images with an alpha channel is as irregularly shaped pieces of cardboard, not as colored glass. Consider these two images:

When we combine them, each pixel of the result can be divided into four regions:

One region where only the source is present, one where only the destination is present, one where both are present, and one where neither is present.

By deciding on what happens in each of the four regions, various effects can be generated. For example, if the destination-only region is treated as blank, the source-only region is filled with the source color, and the ‘both’ region is filled with the destination color like this:

The effect is as if the destination image is trimmed to match the source image, and then held up in front of it:

The Porter/Duff operator that does this is called “Dest Atop”.

There are twelve of these operators, each one characterized by its behavior in the three regions: source, destination and both. The ‘neither’ region is always blank. The source and destination regions can either be blank or filled with the source or destination colors respectively.

The formula for the operators is a linear combination of the contents of the four regions, where the weights are the areas of each region:

$A_\text{src} \cdot [s] + A_\text{dest} \cdot [d] + A_\text{both} \cdot [b]$

Where $[s]$ is either 0 or the color of the source pixel, $[d]$ either 0 or the color of the destination pixel, and $[b]$ is either 0, the color of the source pixel, or the color of the destination pixel. With the alpha channel being interpreted as coverage, the areas are given by these formulas:

$A_\text{src} = \alpha_\text{s} \cdot (1 - \alpha_\text{d})$
$A_\text{dst} = \alpha_\text{d} \cdot (1 - \alpha_\text{s})$
$A_\text{both} = \alpha_\text{s} \cdot \alpha_\text{d}$

The alpha channel of the result is computed in a similar way:

$A_\text{src} \cdot [\text{as}] + A_\text{dest} \cdot [\text{ad}] + A_\text{both} \cdot [\text{ab}]$

where $[\text{as}]$ and $[\text{ad}]$ are either 0 or 1 depending on whether the source and destination regions are present, and where $[\text{ab}]$ is 0 when the ‘both’ region is blank, and 1 otherwise.

Here is a table of all the Porter/Duff operators:

And here is how they look:

Despite being referred to as alpha blending and despite alpha often being used to model opacity, in concept Porter/Duff is not a way to blend the source and destination shapes. It is way to overlay, combine and trim them as if they were pieces of cardboard. The only places where source and destination pixels are actually blended is where the antialiased edges meet.

Blending
Photoshop and the Gimp have a concept of layers which are images stacked on top of each other. In Porter/Duff, stacking images on top of each other is done with the “Over” operator, which is also what Photoshop/Gimp use by default to composite layers:

Conceptually, two pieces of cardboard are held up with one in front of the other. Neither shape is trimmed, and in places where both are present, only the top layer is visible.

A layer in these programs also has an associated Blend Mode which can be used to modify what happens in places where both are visible. For example, the ‘Color Dodge’ blend mode computes a mix of source and destination according to this formula:

$ \begin{equation*} B(s,d)= \begin{cases} 0 & \text{if \(d=0\),} \\ 1 & \text{if \(d \ge (1 - s)\),} \\ d / (1 - s) & \text{otherwise} \end{cases} \end{equation*} $

The result is this:

Unlike with the regular Over operator, in this case there is a substantial chunk of the output where the result is actually a mix of the source and destination.

Layers in Photoshop and Gimp are not tailored to each other (except for layer masks, which we will ignore here), so the compositing of the layer stack is done with the source-only and destination-only region set to source and destination respectively. However, there is nothing in principle stopping us from setting the source-only and destination-only regions to blank, but keeping the blend mode in the ‘both’ region, so that tailoring could be supported alongside blending. For example, we could set the ‘source’ region to blank, the ‘destination’ region to the destination color, and the ‘both’ region to ColorDodge:

Here are the four combinations that involve a ColorDodge blend mode:

In this model the original twelve Porter/Duff operators can be viewed as the results of three simple blend modes:

In this generalization of Porter/Duff the blend mode is chosen from a large set of formulas, and each formula gives rise to four new compositing operators characterized by whether the source and destination are blank or contain the corresponding pixel color.

Here is a table of the operators that are generated by various blend modes:

The general formula is still an area weighted average:

$A_\text{src} \cdot [s] + A_\text{dest} \cdot [d] + A_\text{both}\cdot B(s, d)$

where [s] and [d] are the source and destination colors respectively or 0, but where $B(s, d)$ is no longer restricted to one of $0$, $s$, and $d$, but can instead be chosen from a large set of formulas.

The output of the alpha channel is the same as before:

$A_\text{src} \cdot [\text{as}] + A_\text{dest} \cdot [\text{ad}] + A_\text{both} \cdot [\text{ab}]$

except that [ab] is now determined by the blend mode. For the Zero blend mode there is no coverage in the both region, so [ab] is 0; for most others, there is full coverage, so [ab] is 1.

Disclaimer

First and foremost: Everything documented below is included in the Intel public documentation. Anything I say which offends you are my own words and not those of Intel. (Anything I say which is of monetary value is the opposite).

Daniel asked me to write about hardware contexts on Intel GPUs some time ago. Hardware contexts aren’t terribly interesting. However, the role of contexts has been steadily expanding, and as such, you need to start somewhere in the explanation. So let’s consider this a precursor to really interesting stuff in the future (I hope).

I use the term, “GPU client” quite a bit. If it helps you, you can think of a specific GPU client:

• mesa
• libva
• X device dependant 2d driver (ddx)

I worked pretty hard to make all SVG images so you can zoom in on your own. Please send me feedback if this didn’t work out for you. I know that the arrows turned out stupid, but it works in inkscape, I swear!

Finally, I’ve yet read Daniel’s GEM docs, so maybe some of this is a repeat.

Batchbuffers

I want to take a brief diversion into GPU work. The Direct Rendering Interface enables GPU clients, ie. userspace libraries such as mesa, to directly submit work to the GPU. Since this flies in the face of every security architect’s idea of how to design a system, there was one indirection introduced called a “Batchbuffer.” The batchbuffer is nothing more than a set of instructions to be read by the GPU for setting up state, and to a much smaller extent, instructions telling the GPU how to act upon that state. It’s special because we can apply some security flags to it, and it’s not directly executed by HW. It’s indirectly executed via the batchbuffer. The i915 driver has to tell the hardware to execute the batchbuffer, but the i915 driver never modifies the contents of the batchbuffer. Make no mistake, all GPU commands are assembled into this batchbuffer by userspace and they are executed by the GPU. The kernel driver mostly acts as a proxy. Understanding the basic principal of what a batchbuffer does is important in order to understand why hardware contexts are interesting. Batchbuffers, batches, command buffers, and execbuffers, are used interchangeably, and are all the same thing (at least on Intel GPUs, can’t speak for others).

Because the system may have many GPU clients, all with the ability to directly render on the HW, it was therefore implicit [historically] that any client which had work to be done by the GPU could not assume any of it’s previous GPU state would be preserved upon it’s next run. To put it another way, if a GPU client submitted two batchbuffers back to back, it couldn’t, and still can’t assume that those two run in sequence. For older fixed function hardware, this really didn’t matter too much, but that’s changing…

To elaborate a bit more about batchbuffers, a gross oversimplification of how it works: there are 3 (well, 2.5) types of commands,

1. Here is some state, inline with the command (direct state)
2. I have some state, it’s located over there (indirect state)
3. Go do something with all that state I gave you (3D_ commands)

Many modern GPU clients build the  buffers like a stack/heap. Indirect state goes on the heap, and direct state + commands go on the stack. The commands that have indirect state then point within it’s own buffer. This isn’t strictly necessary, but it makes the allocation easier.

Hardware Contexts

Hardware contexts are a feature which have existed on Intel GEN graphics hardware since Ironlake (GEN5). The Linux kernel’s i915 driver has supported it since 3.6 courtesy of yours truly . Much the same as CPUs, it allows GPUs to save off certain state and restore that state at a later time. This is interesting primarily for two reasons:

1. It potentially reduces the overhead on clients having to restore their context.

2. It allows GPU clients to save their context ie. recover GPU state from a previous batch. I’ll explain both in more detail.

Reducing overhead

Reducing overhead is a fairly straightforward thing, and like many simple optimizations, this one doesn’t buy us much. GPU batchbuffers ( AKA execbufs, or command buffers) do not need to emit as many commands to the GPU since some of it’s state was saved from a previous set of commands. This both reduces the amount of DMA traffic, as well as the number of commands the GPU has to execute (the latter is arguable because it’s HW implementation specific).

Some pictures! The first picture shows without contexts, and the second, with contexts. The only thing to observe  is that with contexts, the batch buffer size is slightly smaller.

Since much of the state setup must happen on every batch for reasons which are mostly out of scope, we don’t get a huge savings here – though nobody has actually measured it.

Preserving State

The second reason for contexts is the ability for a GPU client to have its context restored. This featured is actually required in order to implement modern OpenGL.

Recall earlier where I stated that a GPU client can never depend on two batches executing consecutively. That matters if the second operation relied on results from GPU operations in the first batch. If mesa submits some vertices to be processed, and then the DDX for example goes and wipes out all the GPU state, how can mesa then depend on any GPU state. It’s already been flushed through the GPU pipeline by whatever the DDX was doing.

Client 1 gets client 2′s vertex information

As demonstrated above, badly, client 1 requests some work to be done from the GPU and the intermediate pipeline output is stored somewhere in the GPUs internal state. Well, once client 2 executes, it uses the same pipeline, and so that info is lost. So in step 3. when client 1 again submits a batch to query some info from the pipeline state, it would read back client 2′s state.

Transform feedback was the first immediate need for HW contexts’ ability to preserve state, and is still currently the best example of this. GEN hardware implement the feedback buffer via a pointer (to the buffer) and an offset denoting the next free space within the buffer (within the buffer). The offset can be reset, but not arbitrarily moved by software. As information is dumped into the feedback buffer, the offset is incremented by the hardware. That offset is saved within the HW context state. Keeping the offset around is important because you don’t want to destroy the previous feedback data. To date, this is actually achievable simply by the fact that we do not implement Geometry Shaders. The number of vertices, and therefore the amount of vertex information (and thus, the offset) can be accurately predicted and accounted for. On the next batch you set the pointer to where the calculated offset would have been, reset the offset, and away you go.

Geometry Shaders create an unpredictable number of new vertices, and because of that, the above hack will no longer work.

Kernel Implementation

The kernel implementation isn’t very interesting IMO, and I can write a separate article if anyone requests one. There are a couple of gotchas in the implementation, like the initial state, as an example, but the concept is very straight forward. As I mentioned earlier in regards to Direct Rendering, GPU clients do not submit their work directly. The clients submit work to the kernel, which then tells the GPU to consume that work. This gives the kernel the opportunity to insert the proper save and restore commands to the GPU without the clients knowing. For example, if we have two clients X, and Y it might be something like:

... <gotchas>
client X submits batch
i915 emits X's batch to hardware
client Y submits batch
i915 (via HW context) saves client X's state, and restore Y's state
i915 emits Y's batch to hardware

So I assume most of you have read Jonathan Blanfords blog post about leaving Red Hat and me taking over for him as head of the Red Hat desktop team. First of all I would like to thank Jonathan for both his contributions to GNOME and Red Hat, but also for being a good friend for over a decade now. Luckily Linux is also a major piece of his new job, so I am certain we have not seen the last of Jonathan in the community.

For the outside observer I wouldn’t expect any immediate visible changes to happen as part of this transition. My job is to follow up on the many great initiatives that Jonathan started here together with the rest of the team. One major piece I will be tackling is making sure we in the Red Hat desktop team work even closer with the Fedora community to bring forth some great improvements to Fedora and created an even more integrated and seamless experience for those wanting to use the Fedora desktop. This ranges from working with the Fedora team on a new software installer to working on getting Wayland ready for deployment in Fedora. Apart from that we will of course continue to work with the GNOME community on pushing GNOME 3 forward. I strongly recommend following Matthias Clasens blog to get the latest and greatest news on our efforts around GNOME 3.

I hope to post to my blog more frequently going forward to highlight exciting developments the many great projects the Red Hat Desktop team contribute to, like GNOME, LibreOffice, Firefox, Spice, Evolution, X Window/Wayland and more.

Hi everyone!

Today I make an announcement I thought I would never ever make. But things changed.

Discussion about this has a long history, starting as a non-serious suggestion at DesktopSummit 2011, continued with people on IRC, but it was decided back then that it wouldn’t be worth the effort. This has changed too, and a small team has formed to work on it.

We hereby announce Tanglu, a new Debian-based-Linux distribution.

Why do we need another one? Let me explain the concepts of that distro:

Tanglu will be based on Debian Testing and follow the Debian development closely. It will have a 6-months release-cycle and it’s target audience are Linux desktop users. We will make installing and setting up the distro as easy as possible.

Tanglu will be usable for both developers of upstream software and the average Linux user and Linux newbie. This is possible because in our opinion developers and users don’t have different needs for a desktop system. Both kinds of users like a polished desktop which “just works”. We will, hwever, not apply any kind of fancy modification on upstream software, we will basically just distribute what upstream created, so users can get an almost “pure” GNOME and KDE experience.

Tanglu is designed to be able to solve the issue that Debian is frozen for a long time and Debian Developers can’t make new upstream versions available for testing easily. During a Debian freeze, DDs can upload their software to the current Tanglu development version and later start the new Debian cycle with already tested packages from Tanglu. The delta between Tanglu and Debian should be kept as minimal as possible. However, Tanglu is not meant as experimental distribution for Debian, so please upload experimental stuff to Experimental. Only packages good enough for a release should go into Tanglu.

Ideally, Tanglu and Debian should be working well together in mixed environments, where you for example have Debian servers and multiple Tanglu desktops with the new software, targeted at desktop user. Since the differences between Tanglu and Debian should not be very high, administering both systems should be very easy (if you know Debian).

Tanglu will be an open project, driven by community. At the beginning of each cycle, people can make suggestions for release goals they want to implement (similar to Fedora, but without FESCo). These proposals are discussed in public and are rejected if there are major technical concerns. If consensus about a certain proposal is lacking, a vote is done for it. The proposal can be accepted with absolute majority. If this does not happen, the proposal is postponed for the next release, where people can vote for it again. If nobody wants that function, it is rejected. In general, decisions made by Debian are superior and have to be followed.

We don’t think we know every package and every software better than the original upstream. That’s why it makes much sense to rely on feedback from others and to have a community-based and peer-reviewed distribution, instead of secretly developing stuff and dumping it on the community. Tanglu will have a highly predictable set of features, defined at the beginning of a cycle, so you will know what you can expect from the next release as soon as possible and plan for it.

Tanglu will make it easy to deploy applications for it. It will contain a software-center, similar to what Ubuntu has. We will also try to establish a solution for a “Linux-AppCenter”, a place for Linux applications, which will be open not only for Tanglu, but can be implemented in any other distribution too. Possible income will flow back into development of the platform.

Now, let’s answer the FQA (Future Questions Asked):

Why don’t you contribute to Debian directly and create yet another distribution?

First of all, we contribute to Debian And for me, I can say that I will contribute to Debian even more. The point is that Debian can not cover all possible use-cases, and with Tanglu we want to make a distro which solves this. You might ask why we have to create a new distro for that, instead of creating improvements inside Debian? Creating a new distro allows us to do stuff we can never do in Debian. For example, we will include proprietary firmware in that distro, we will make installations of proprietary stuff possible easily (but don’t ship with it by default) and we will have a time-based release cycle. These are already things which are a no-go for Debian, and that’s fine. We don’t want Debian to support these cases, as it is already a great distribution. We want to offer a distro as close to Debian as possible, but with a few modifications for use-cases which are not covered by Debian itself. Of course we will participate in DEX.

If Debian Developers contribute to Tanglu, freezes will take even longer!

This is an often-heard concern, it comes up on every mailinglist discussion about continuing development while freeze. I would disagree here, packaging new upstream stuff is not slowing down testing and improving of packages in Testing. Also, Tanglu is an offer for Debian developers to participate (we will sync privileges for their packages) – we don’t expect anyone to work on it, but as we think DDs know their packages best, we will make it possible for them to participate without extra barriers. We hope that Tanglu can add value to Debian and that Debian cycles can start with better-tested packages.

You said you are a small team – you cannot develop a whole distribution with it!

Let’s put that to the test! All people working on this are well aware of the issue that the project can not survive without much community-involvement on the long run. But we see a chance that many people are interested in it and that there is a high demand for it.

At the beginning, we will just start with a small set of packages. We will also sync many packages from Ubuntu, to reduce workload. For example, it is planned to use the Ubuntu-Kernel and KDE packaging. By doing this, we keep the workload at the beginning low. We also reduce duplicate work with that.

We even have some possible sponsors for the new distribution. But nothing is set in stone yet, so just wait for it to happen.

Why not participate in Arch, OpenSUSE $other_distro?

These are not Debian . I know, it sounds odd, but if you like the Debian way of doing things, you want to use a Debian-based distribution. There is nothing wrong with OpenSUSE. And Debian has issues too. But we want to be close to Debian and use it’s tools and way of doing things.

I hate you!!! You are doing it wrong!! The project is useless!

Well, that’s fine. But there is no reason for hating us. If you dislike our idea, there are basically two options: First, you hate us but the project is successful. In that case, you have been wrong with hate, as there are definitely people who liked the project and contributed to it. Second, you hate us and we fail. In this case, there is no reason for hate, as the project will just vanish and you don’t have to worry about it. So hating it would’ve been just a big waste of energy.

Also keep in mind that forking is a way to keep development healthy and to adapt software to new use-cases which it didn’t target before. And we are not introducing incompatibilities here (like e.g. writing our own display server could). Instead, we want to stay close to Debian and reuse as much code as possible.

Which desktop will you use?

Everyone can add a new desktop to Tanglu, as long as the desktop-environment is present in Debian. Long term, we will have to offer Linux-newbies a default flavour, probably by setting a default download on the website. But as long as there is a community for a given desktop-environment, the desktop is considered as supported.

At the beginning, we will focus on KDE, as many people have experience with it. But adding vanilla GNOME is planned too.

Can you say something about the software used in Tanglu?

Yes, but this is still in flow, so I can’t promise final decisions here. On the server, side, we will try to use the Dak for repository management, as soon as we have enough server capacity. We will also use the standard Debian repository GUI and basically reuse most of the software there, to diverge less from Debian.

The distribution itself will use a Linux Kernel from Ubuntu and systemd as the primary init system, as well as the login manager logind. It will be based on current Debian Testing with some fresh packages from Unstable and Experimental. We might also use the Ubuntu driver packages and KDE packaging. We expect to have a very rough start with the first release, but there will be enough time to polish Tanglu.

Nice idea! How can I help?

Well, you can help with basically anything at time – from writing manuals, over designing logos and pages to administering a webserver and create packages. We are at an early stage of development at the moment, but we wanted to go public with it as soon as possible, to include the community and receive feedback so we can make that distro community-centric from the beginning. Most of the infrastructure is currently in progress too.

So, if you want to get started with Tanglu, subscribe to our mailinglist tanglu-devel and write a mail to it, intruducing yourself. We can then include you in the loop. Generally, if you want to get access to our machines, a trusted GPG-signature will help a lot.

If you want to talk to us, join #tanglu-devel or Freenode! Most discussions are currently happening there.

And that’s it! Tanglu will be awesome!

Some other projects of mine will develop a bit slower because I am now involved in Tanglu. But nothing will stop, and there is some pretty cool stuff coming for both GNOME and KDE (and I still have to implement DEP-11 for Debian).

End of February devconf.cz took place in Brno, Czech Republic. At the conference Kay Sievers, Harald Hoyer and I did two presentations about our work on systemd and about the systemd Journal. These talks were taped and the recordings are now available online.

First, here's our talk about What Are We Breaking Now?, in which we try to give an overview on what we are working on currently in the systemd context, and what we expect to do in the next few months. We cover Predictable Network Interface Names, the Boot Loader Spec, kdbus, the Apps framework, and more.

And then, I did my second talk about The systemd Journal, with a focus on how to practically make use of journalctl, as a day-to-day tool for administrators (these practical bits start around 28:40). The commands demoed here are all explained in an earlier blog story of mine.

Unfortunately, the audience questions are sometimes hard or impossible to understand from the videos, and sometimes the text on the slides is hard to read, but I still believe that the two talks are quite interesting.

Have you heard about it? openSUSE 12.3 is out!

I did an upgrade earlier today on my main laptop (with a simple zypper dup after having updated the repos configuration, which went surprisingly fast), and this release looks great! But the best part: it comes with OpenStack love!

Enjoy Folsom!

For the first time, an openSUSE release provides a fully working set of OpenStack packages. We had some OpenStack packages in the previous release, but they were not in such a great shape and some components were even missing (although we fixed that later on with packages in the build service).

With 12.3, you can finally enjoy OpenStack with the Folsom release in a very straight-forward way, and therefore you can easily deploy your own cloud. The packages that we provide are built from the stable/folsom branch, and there's an online update coming soon so you can enjoy the stable/folsom code as of end of last week.

To help people who might not want to learn everything needed to properly deploy OpenStack, we also have a small openstack-quickstart package, that comes with a script that can be used to deploy everything locally. It is obviously not recommended to run this on your main computer (I usually run this in a virtual machine), but it gets you quickly to the point where you can play with OpenStack.

Dashboard of an OpenStack cloud running on openSUSE 12.3

Play today with Grizzly!

Of course, Folsom is relatively old at this point and the new version, Grizzly, is to be released in three weeks. But don't be sad! We've been working on Grizzly packages for some time now: you can grab them from the Cloud:OpenStack:Master project in the build service (hey, look, it's even building packages for SLE and openSUSE 12.2! the build service is a rather convenient tool!). I guess we'll properly move them to Cloud:OpenStack:Grizzly once Grizzly is officially released.

Develop with DevStack!

I mentioned a few months ago that I had finished porting DevStack to openSUSE 12.2, and I wrote some small documentation on how to use it. It's really a neat tool, both for playing with OpenStack and for developing for it.

However, I realized earlier this week that I had never double-checked everything was still okay for 12.3. It turns out there's a small issue that completely breaks it, oops ;-) But once the fix is checked in, DevStack will be usable on the latest openSUSE. I'll do some more tests before marking this version of openSUSE as supported in DevStack, but that shouldn't block anyone from using DevStack on 12.3.

Join us!

We're pretty open about how we develop OpenStack in openSUSE. Andreas wrote a post about all this a few days ago. We've opened up (or rather, revived) a mailing list dedicated to the cloud recently, which developers, packagers and users can all use to discuss OpenStack. And unsurprisingly, we also have an #opensuse-cloud channel on Freenode. But most importantly, we've worked on making public the infrastructure we use to build OpenStack for openSUSE.

I think the important bit on this is that everybody is able, and welcome, to join this effort. It's not just about being able to say "see, we have OpenStack"; it's about building a rock-solid experience for OpenStack, and enjoying doing that!

Now, let's celebrate the release: party time! :-)

The main feature making up XI 2.3 is Pointer Barrier events. This feature was initiated by Chris Halse-Rogers and then taken up by Jasper St. Pierre and me, until it became part of XI 2.3.

The usual conditions apply: a client must announce XI2.3 support with the XIQueryVersion() request to utilise any of the below.

Pointer barriers

Pointer Barriers are an XFixes v5.0 additions to restrict pointer movement. They were first available in server 1.11 (released Aug 2011). A pointer barrier is created by a client along a specific horizontal or vertical line of pixels. Relative input devices such as mice or touchpads will be constrained by this barrier, preventing the cursor from moving across the barrier.

For example, GNOME3 creates a vertical barrier on the top-left screen edge. In a multi-monitor screen, flicking the pointer up to the Activities menu will constrain the pointer there even if there is a screen to the left of the menu. This makes the menu much easier to hit.

The client can choose for the barrier to be transparent in specific directions. GNOME3's pointer barrier is permissive in the direction of positive X (i.e. left-to-right). Thus, moving from the left screen to the right is unrestricted, even though the pointer is constrained when moving in the other direction.

A simple client that creates a vertical pointer barrier looks like this:

Display *dpy = XOpenDisplay(NULL);
int fixes_opcode, fixes_event_base, fixes_error_base;
PointerBarrier barrier;

if (!XQueryExtension(dpy, "XFIXES",
                     &fixes_opcode,
                     &fixes_event_base,
                     &fixes_error_base))
    return EXIT_FAILURE;

/* vertical barrier from 20/20 to 20/100 */
barrier = XFixesCreatePointerBarrier(dpy, DefaultRootWindow(dpy),
                                     20, 20,
                                     20, 100,
                                     0, /* block in all directions */
                                     0, NULL); /* no per-device barriers */

mainloop();

Pointer barrier events

As of XI 2.3, pointer barrier events are sent to clients when pointer movement is constrained by a barrier, provided that client created pointer barriers, the clients have the matching event mask set. Two new event types were added, XI_BarrierHit and XI_BarrierLeave. Both events are only sent to clients owning a barrier and are only sent to the window used to create the barrier (the root window in the example above).

unsigned char m[XIMaskLen(XI_BarrierLeave)] = {0};
XIEventMask mask;
mask.deviceid = XIAllMasterDevices;
mask.mask_len = XIMaskLen(XI_BarrierLeave);
mask.mask = m;

XISetMask(mask.mask, XI_BarrierHit);
XISetMask(mask.mask, XI_BarrierLeave);
XISelectEvents(dpy, DefaultRootWindow(dpy), &mask, 1);
XSync(dpy, False);

while (1) {
    XEvent ev;
    XNextEvent(dpy, &ev);
    if (ev.type != GenericEvent || ev.xcookie.extension != xi2_opcode)
       continue;

    XGetEventData(dpy, &ev.xcookie);
    XIBarrierEvent *b = ev.xcookie.data;
    if (b->evtype == XI_BarrierHit)
       printf("Pointer hit the barrier\n");
    else if (b->evtype == XI_BarrierLeave)
       printf("Pointer left the barrier\n");
    
    XFreeEventData(dpy, &ev.xcookie);
}

An XI_BarrierHit event is first sent when the pointer movement is first constrained by a barrier. It includes some information such as the device, the barrier and the window. It also includes coordinate data.

XI_BarrierHit events are sent for each movement of the pointer against this barrier, or along it, until the pointer finally moves away from the barrier again. "Moving away" means a move to different position that is not blocked by the barrier.

• if the barrier is vertical, the pointer moves to a different X axis value, or
• if the barrier is horizontal, the pointer moves to a different Y axis value, or
• the pointer moves past the barrier's end point

Who gets events? Always the client that created the barrier, and only if the window has the event mask set. If the client has a grab on the device with the grab_window being the barrier window, the barrier events follow the usual grab event mask behaviour:

• if the grab event mask has XI_BarrierHit set, the event is delivered
• if the grab event mask does not have XI_BarrierHit set but the window mask does and owner_events is True, the event is delivered
• if owner_events is False and the grab mask does not have XI_BarrierHit set, no event is sent

If the client's grab has a grab_window different to the barrier window, or the device is grabbed by another client, event delivery is as usual. In all cases, if the device is grabbed, the XIBarrierDeviceIsGrabbed flag is set. Clients should use this flag to determine what to do. For example, the barrier that is used to trigger the GNOME overlay should probably not trigger if another client has a grab as it may interfere with drag-and-drop.

Coordinates and time in pointer barrier events

Barrier events contain two sets of x/y coordinates. First, the root coordinates which represent the position of the pointer after being confined by barrier (and screen extents, where applicable). This coordinate is the same you would get from a subsequent XIQueryPointer request.

The second set are the delta coordinates (dx/dy), in screen coordinates, from the last pointer position, had the barrier not constrained it. So you can calculate how much the pointer would have moved and thus derive speed. The dtime field in the event helps you to calculate speed, it provides the time in milliseconds since the last pointer event. The deltas are calculated after taking pointer acceleration into account.

    XIBarrierEvent *b = ev.xcookie.data;
    double dx, dy;
    double speed;
    unsigned int millis;

    dx = b->dx;
    dy = b->dy;
    millis = b->dtime;

    speed = sqrt(dx * dx + dy * dy) / millis * 1000;

    printf("Barrier was hit at %.2f/%.2f at %.2f pixels/sec\n",
           b->root_x, b->root_y,  speed);

Releasing a pointer from barrier constraints

By default, a pointer barrier blocks all movement of relative input devices across a barrier. However, a client can opt to temporarily release the pointer from the barrier constraints with the XIBarrierReleasePointer request.

To do so, the client needs the event ID of the barrier event. Since a pointer may bump against the same barrier multiple times before the client reacts (X is asynchronous, after all), the event ID serves to identify a set of movements against or along the pointer barrier.

An event ID is assigned to the first XI_BarrierHit event, and then it remains the same until the XI_BarrierLeave event. This event is the last event with the current event ID, any future barrier events will have a new event ID. This approach may be familiar to you from dealing with touch events, that use a similar approach (touch IDs start at TouchBegin and are tracked through to the TouchEnd).

To release a pointer and let it pass through the barrier, call XIBarrierReleasePointer().

    XIBarrierEvent *b = ev.xcookie.data;
    ...
    if (speed > 200) {
        printf("Movement exceeds speed limit, allowing pointer to go through\n")
        XIBarrierReleasePointer(dpy, b->deviceid, b->barrier, b->eventid);
        XFlush(dpy);
    }

If the release does succeed and the pointer moves through the barrier, the client gets a XI_BarrierLeave event with the XIBarrierPointerReleased flag.

The pointer barrier hit-box

As mentioned above, a XI_BarrierLeave event is sent when the pointer moves away from the barrier. This would usually require a 1 pixel movement (let's ignore subpixel-movement, our hand's aren't that precise). However, during testing we found that 1 px pointer-movement can still happen even when a user tries hart to move along the barrier, or even when pushing against the barrier. Thus, we implemented a hit-box of 2 pixels. Thus, a 1 px movement along the barrier still counts as hitting the barrier, and the pointer is not treated as having left the barrier until it leaves the hit-box.

Testing the current state

The current code is available in the barriers branch of the following repositories:

git://people.freedesktop.org/~whot/inputproto.git 
git://people.freedesktop.org/~whot/libXi.git 
git://people.freedesktop.org/~whot/xserver.git 

Update March 7 2013: This addition was not merged into XI 2.3, largely because there is no real need for it. XI 1.x' XGetExtensionVersion() returns the server version without locking in a client version and at this point there was no perceived need for getting the already-requested client version back. I'll leave this here for archival purposes but again, this request was not merged into XI 2.3

Original post below

Posting this here too to get a bit more exposure.

XIQueryVersion(3) is the first XI2 request clients should send to the server. The client announces its supported version and in return receives the server version (which is always less or equal to the client, never higher).

As XI 2.1 - 2.3 progressed, we started using this information in the server. Clients are treated slightly differently depending on their announced version. The current differences are:

• XIQueryPointer will not set the button 1 mask for pointer-emulated events if the client supports XI 2.2 or newer.
• XIAllowEvents will allow XIRejectTouch and XIAcceptTouch for clients supporting XI 2.2 or newer.
• Raw event delivery changes if a client supports XI 2.1 or newer.

So far, so good. This works fine as long as the client supports one specific version. However, as toolkits like GTK have come to support XI2, the requirements changed a bit. An application and its toolkit usually look like a single client to the server. However, the client may support XI 2.0, but the toolkit may support XI 2.3. And neither knows of the other's version support. If the client requests XIQueryVersion before the toolkit, the toolkit is locked into the client version. But if the toolkit first requests XIQueryVersion, the client is locked into the version supported by the toolkit. Worst case the client may get a BadValue and quit because it may not be built for this case.

Jasper St. Pierre and Owen Taylor brought this up on #xorg-devel today, and I've send a proposed solution to the mailing list.

A new XIGetSupportedVersion request simply returns the server's major/minor version number. Uncapped, so really what the server supports. And the same request also returns the client version previously announced with XIQueryVersion. Or zero, if the client hasn't called it yet.

This request enables toolkits to query what the client has already set, and of course what the server supports without modifying the client state. The request is currently an RFC, but I do hope we may get this into XI 2.3.

If you're working on XI2-aware clients or toolkits and you have a use-case that requires this or would break by this addition, please speak up now.

Composite and Swap — Getting it Right

Where the author tries to make sure DRI3000 is going to do what we want now and in the future

DRI3000

The basic DRI3000 plan seems pretty straightforward:

1. Have applications allocate buffers full of new window contents, attach pixmap IDs to those buffers and pass them to the X server to get them onto the screen.

2. Provide a mechanism to let applications know when those pixmaps are idle so that they can reuse them instead of creating new ones for every frame.

3. Finally, allow the actual presentation of the contents to be scheduled for a suitable time in the future, generally synchronized with the monitor. Let the client know when this has happened in case they want to synchronize themselves to vblank.

The DRI3 extension provides a way to associate pixmap IDs and buffers, and given the MIT-SHM prototype I’ve already implemented, I think we can safely mark this part as demonstrably implementable.

That leaves us with a smaller problem, that of taking pixmap contents and presenting them on the screen at a suitable time and telling applications about the progress of that activity.

In the absence of compositing, I’m pretty sure the initial Swap extension design would do this job just fine, and should resolve some of the known DRI2 limitations related to buffer management. And, I think that goal is sufficient motivation to go and implement that. However, I wanted to write up some further ideas to see if the DRI3000 plan can be made to do precisely what we want in a composited world.

The Composited Goal

To make sure we’re all on the same page, here’s what I expect from the Swap extension in a composited world:

1. Application calls Swap with new window pixmap

2. Compositor hears about the new pixmap and uses that to construct a new screen pixmap

3. Compositor calls Swap with new screen pixmap

4. Vertical retrace happens, executing the pending swap operation

5. Compositor hears about the swap completion for the screen

6. Application hears about the swap completion for its window

In particular, applications should not hear that their swap operations are complete until the contents appear on the screen. This allows for applications to throttle themselves to the screen rate, either doing double or triple buffering as they choose.

I didn’t add steps here indicating buffers going idle or being allocated, because I think that should all happen ‘behind the scenes’ from the application’s perspective. Many applications won’t care about the swap completion notification either, but some will and so that needs to be visible.

Redirected Swaps?

Owen Taylor suggested that one way of getting the compositor involved would be to have it somehow ‘redirect’ Swap operations, much like we do with window management operations today. I think that idea may be a good direction to try:

1. Application calls Swap with new window pixmap

2. Swap is redirected to compositor, passing along the new window pixmap

3. Compositor constructs a new screen pixmap using the new window pixmap

4. Compositor calls Swap on the screen and the window, passing the new screen pixmap and the new window pixmap. When the screen update occurs, the screen and the window both receive swap completion events.

This has the added benefit that the X server knows when the compositor is expecting window pixmaps to change like this — the compositor has to explicitly request Swap redirection.

Window Pixmap Names and GEM Buffer Handles

One issue that swapping window pixmaps around like this brings up is how to manage existing names for the window pixmap. Right now, applications expect that window pixmaps will only change when the window is resized. If the Swap extension is going to actually replace the window pixmap when running with a suitable compositor, then we need to figure out what the old names will reference.

Are there non-compositor applications using NameWindowPixmap that matter to us? How about non-compositor applications using TextureFromPixmap to get a GEM handle for a window pixmap? For now, I’m very tempted to just break stuff and see who complains, but knowing what we’re breaking might be nice beforehand.

Idling Pixmaps

When an application is done drawing to a window pixmap and has passed it off to the X server for presentation, we’d like for that pixmap to be automatically marked as discardable as soon as possible. This way, when memory is tight, the kernel can come steal those pages for something critical. Of course, applications may not want to let the server mark the pixmap as idle after being used, so a flag to the Swap call would be needed.

Ideally, the pixmap would become idle immediately after the pixmap contents have been extracted. In the absence of a compositor, that would probably be when the Swap operation completes. With a compositor running, we’d need explicit instruction from the compositor telling us that the window pixmap was now ‘idle’:

┌───
    SwapIdle
    drawable: Drawable
    pixmap: Pixmap
      ▶
└───

Furthermore, the application needs to know that the pixmap is in fact idle. I think that we’ll need a synchronous X request that marks a buffer as ‘no longer idle’ and have that return whether the buffer was discarded while idle. It doesn’t seem sufficient to use events here as the application will need to completely reconstruct the pixmap contents in this case. This reply could also contain information about precisely what contents the pixmap does contain.

┌───
    SwapReuse
    drawable: Drawable
    pixmap: Pixmap
      ▶
    valid: BOOL
    swap-hi: CARD32
    swap-lo: CARD32
└───

Pixmap Lifetimes and Triple Buffered Applications

If we redirect the Swap operation and send the original application window pixmap ID to the compositor, what happens when the application frees that pixmap before the compositor gets around to using the contents?

Surely the Compositor must handle such cases, and not just crash. However, I’m fine with requiring that the application not free the pixmap until told by the compositor.

Some weeks ago I have been trying to figure out how to decode yEnc-encoded files one word at a time since decoding them byte-by-byte is both slow and lame ;)

Let's first look at how the naive implementation of a yEnc decoder looks like:

uint8_t input[...]; // Input bytes, NUL terminated.
uint8_t output[...]; // Output bytes
uint8_t escape = 0x3d;
int i = 0, j = 0;

while (input[i]) {
    uint8_t c = input[i++];

    // Escape character?
    if (c == escape)
        c = input[i++] - 64;

    output[j++] = c - 42;
}

And this is what I've come up with. Note that this code only works if 0x3d itself is never escaped, ie there will be no pairs 3d3d in the input stream. Thus we will have at most two escaped bytes per word. Yes, this is cheating, but it seems to work well enough with the encoders that are in commong use today.

On the ARMv5 CPU powering my GuruPlug, this code is ~10% faster than the naive approach.

So here's the (optimized) code. The explanations in there aren't nearly as good as they should be, but they are all you'll get for now ;p

void decode_line()
{
    uint8_t input[...];
    uint8_t output[...];
    size_t length = number of input bytes to process;

    /* Align to a multiple of 4. */
    size_t orig_length = length;
    length = (length + 3) & ~3;
    ssize_t faked = length - orig_length;

    uint32_t state = 0;
    uint32_t *wp = (uint32_t *) input;
    int j = 0;

    for (; length; length -= 4) {
        uint32_t word = *wp++;
        uint32_t decoded, ndecoded;

        decoded = decode (&state, word, &ndecoded);

        output[j++] = decoded;

        decoded >>= 8;
        output[j++] = decoded;

        ndecoded--;

        if (--ndecoded) {
            decoded >>= 8;
            output[j++] = decoded;

            if (--ndecoded) {
                decoded >>= 8;
                output[j++] = decoded;
            }
        }
    }

    /* We cheated before when we aligned 'length'.
     * Adjust for that now, so j will contain the
     * actual number of output bytes written.
     */
    j -= faked;
}

/**
 * @param state Used to remember whether the first byte was meant to be escaped.
 * @param word The word to decode.
 * @param ndecoded Will store the number of decoded bytes (ranging from 2 to 4).
 * @return The decoded word.
 */
uint32_t decode (uint32_t *state, uint32_t word, uint32_t *ndecoded)
{
    uint32_t b64 = *state >> 25;
    uint32_t add_magic = 0xd6d6d6d6 & ~b64; /* 256 - 42 = 214 = 0xd6 */
    uint32_t mask, result;

    /* Find the 0x3d bytes. */
    mask = find_eq (word | b64);

    /* From the mask, we can deduce how many bytes we will decode. */
    *ndecoded = count00 (mask);

    /* Perform the actual decoding.
     * Since we need to apply the offset to the bytes _following_ the
     * escape bytes, we're shifting the mask.
     *
     * We're using add instead of sub because the former needs
     * fewer instructions in ARM machine code.
     */
    result = add (word, add_magic & ~(mask << 7));

    /* Check for MSB to see if we had 0x3d in the leftmost byte. */
    *state = mask;

    /* Kick out bytes that originated from 0x3d. */
    return compress (result, mask);
}

/* Turns bytes that are 0x3d in x into 0x80 and all others into 0x00
 * (known as the Mycroft test).
 */
static inline uint32_t
find_eq (uint32_t x)
{
    x ^= 0x3d3d3d3d;

    return (x - 0x01010101) & ~x & 0x80808080;
}

/* Multibyte add, from Hacker's Delight. */
uint32_t add (uint32_t x, uint32_t y)
{
    uint32_t a, b;

    a = (x & 0x7f7f7f7f) + (y & 0x7f7f7f7f);
    b = (x ^ y) & ~0x7f7f7f7f;

    return a ^ b;
}

/* Exchange bits m in x that are k bits apart
 * (from Hacker's Delight).
 */
uint32_t exch (uint32_t x, uint32_t m, uint32_t k)
{
    uint32_t t = (x ^ (x >> k)) & m;

    return x ^ t ^ (t << k);
}

uint32_t compress (uint32_t x, uint32_t m)
{
    if (m) {
        x = exch (x, 0x0000ff, (m & 0x008000) >> 12);
        x = exch (x, 0xff0000, (m & 0x800000) >> 20);

        if (m & 0x008080)
            x >>= 8;
    }

    return x;
}

/* Count number of zero bytes in x. */
uint32_t count00 (uint32_t x)
{
    /* In x, zero bytes are the bytes that were !0x3d
     * in the input word.
     * We know there can be at most two 0x3d bytes,
     * thus we will have 2-4 zero bytes in x.
     */
    uint32_t count = 4;

    if (x)
        count--;

    x = x & (x - 1);

    if (x)
        count--;

    return count;
}

Disclaimer: This post just sums up a concept for a new distribution which matches certain ideals. It is not the announcement of a new distribution. These are just abstract ideas. (However, if there is high interest in a project like this, it might of course develop into something real…)

I have been involved in Debian and Ubuntu for a long time now. When Ubuntu started, I was a Debian Testing user, and I immediately switched to Ubuntu when it started, because I liked the idea of a short-release-cycle, user-centric company-supported Debian based Linux distribution. However, I am now back to Debian for a long time, because of many reasons which nearly all had to do with Canonical policy. But this is not a post to criticise Ubuntu, so I’ll leave out most of that part. I am highly disappointed on how Ubuntu develops – not only the technical decisions are at least questionable, but also the social and community part is not that great anymore. There is a high asymetry in the relation between Canonical and other developers, Ubuntu mailinglists basically don’t create meaningful results, they sometimes even mutate to a Canonical Q/A session. The community does not seem to have a large influence on decisions about core services, and it can’t have it if there are things developed behind closed doors. (This is all, of course my subjective impression)

But really nobody can argue against the basic idea of Ubuntu and the great things Ubuntu created Also, many of the processes Ubuntu uses to develop the distribution are just great and well-working, as well as there is a highly active community around it. As you simply cannot argue with Canonical to change their policy (they are a company and have hidden plans, also they have every right to apply whatever policy they want), the natural way in any OSS project would be to fork it. But doing that blindly would just create another distribution, which would almost certainly vanish again soon, since there are already many Ubuntu derivatives which cover many use-cases using an Ubuntu base.

I discussed this stuff with Daniel some time ago, and we did some kind of brainstorming about what a perfect distribution would look like, from the perspective of a developer who wants to use a Debian-based distribution.

Here is a list of points which would define such a project:

• Every available package complies with the DFSG and Debian policy.
• Packages of DISTRO stay in close sync with Debian packages, changes are preferrably applied in Debian. DISTRO might work as a playground for new technology while Debian is in freeze.
• DISTRO stays as close to upstream as possible. It applies as less patches as possible, to deploy desktop environments which look like the thing upstream intended it to look like. Changes for DISTRO are developed upstream and only applied downstream if doing that doesn’t make sense or changes are distribution-specific and can’t be abstracted.
• All desktop environments are treated equally. There is no preferred DE.
• DISTRO stays in sync with release cycles of KDE and GNOME, to provide developers the latest stable development environment and users a recent version of fresh upstream software.
• DISTRO is user-centric. It tries to make the distribution work on as many hardware as possible and to fix any usability quirks which are found. But work on that should go upstream, of course.
• At the beginning of a release cycle, packages are merged from Debian and selectively merged from Ubuntu, where it makes sense.
• All Debian developers automatically have upload access to DISTRO, so they can upload versions of their packages to the cutting-edge DISTRO and also maintain the Debian things.
• All DISTRO developers can upload any package, but are responsible for everything they might break. Also, changes should be discussed with the original maintainer at Debian – this is the person who knows the package best.
• DISTRO is not a testing environment for Debian, it is developed like a short-cycle-less-stable-but-usable Debian. DISTRO recommends Debian Stable as “LTS” version.
• No CLA is enforced on anyone. People are free to contribute, as well as companies. The project structure is meritocratic.
• Features are discussed in the open. If there is disagreement about the direction of the project or about any technical issue, a public voting on this feature is created, where every project member can vote. The vote result is final (but can be changed in future, of course).
• Create a well-maintained core, make it possible to install the newest applications on that distribution (by using native packages, Listaller or Glick) Think about rolling releases, where applications are constantly updated and the core stuff is refreshed once a year/every 6 months. With this users have new apps immediately, but don’t live in fear that some core API or a desktop workflow changes immediately. (Also not desired for desktops in schools, universities and companies – maintaining a moving target in these larger environments is a pain)
• Create an Application-Center and application-ecosystem based on AppStream and Listaller around it. Encourage upstream developers to publish fresh applications there. Make it possible for every distribution to use this solution and help others to adapt it. Don’t try to lock people on that one platform by providing stuff exclusively for DISTRO or making it harder to use it.
• Encourage using additional commercial stuff like Amazon searches, magazine stores etc., but make these things available on a separate repository and make them entirely opt-in. Never enable stuff like this by default, but add simple instructions how to use these things for people who want to use them.

This is basically what we would like to have in a new distribution. If you take a closer look, you will see that an effort like this would basically create a close-to-upstream, user-centric short-cycle Debian-based and close-to-Debian distribution, which would cover many use cases, including fixing the “use experimental for new packages during freeze” issue at Debian (DISTRO could be used as environment to run cutting-edge technology, which is generally stable enough, but not yet “Debian-Stable”-stable). Something like this does not exist at the moment. If you take a second look at the list above, you will also see that I mixed technical aspects with organizational aspects. This is intentional. This is just brainstorming, because it is good to know what you would like to have, instead of complaing about the status quo of other projects.

But maybe there will be a distribution which matches some of the above points, to create an upstream-friendly entirely community based Ubuntu.

Let’s forget for a second about video drivers, whether it has acceleration or not, and all the related issues with hardware support on Wayland. This is all solved. Let’s talk about the user interface (UI) and ways to customize it all over the computing continuum — from phones, tablets and TV box to desktop PCs, Invehicle Infotainment (IVI), aeroplane systems, among others.

(I’ve made a cheat sheet here also — Creative Commons Legal Code Attribution 2.0. for both figures)

On customization, the shell plugin comes first: changes in there will impact directly which UI paradigm will be used. Specifically, one implementing the plugin protocol will be defining whether the UI is meant for phones, IVI, desktops, etc.

Probably the most important characteristic of the shell plugin is to give ”roles” for surfaces, i.e. define where and how they will be mapped on the screen. For example, if a client wants its surface mapped as a top-level window, or say to resize the dimensions of it, then it’s up to the shell to expose these different surfaces roles, all according the UI paradigm the shell itself is providing.

Worth to note that the shell plugin doesn’t need to rely on any drawing library or graphics toolkit because it doesn’t tackle directly drawing aspects. Also, conceptually it’s mandatory to give roles for surfaces and therefore a shell plugin is a must (or at least a simple implementation of surface::configure).

An special shell client through an special “private” protocol can be used for setting up basic UI elements that require special treatment. For example in the desktop UI, widget elements such as panel, dock, lockscreen and cursors will need special treatments for their positioning, grabbing semantics and so forth.

On customization, different shell clients, exposing different UI elements can be implemented using *the* *same* shell plugin. Some architectures will rather be using one overlay simple client that will take care of spawning and controlling other UI basics applications also.

The special client will probably want to rely on graphics toolkits.

Wayland clients use the Wayland core protocol and the protocol that shell plugin has defined. The corollary is that one client will always know the UI paradigm (due shell plugin) and will *not* work across different paradigms. Though, that doesn’t mean applications will need to know their paradigm necessarily but only the middleware software is connecting to Wayland (like the graphics toolkits).

A footnote about Canonical’s Mir

Canonical announced their new display manager yesterday. There’s a section ”Why Not Wayland / Weston?” where they claim:

“we consider the shell integration parts of the protocol as privileged and we’d rather avoid having any sort of shell behavior defined in the client facing protocol.”

and something similar was written here also:

” Wayland .. exposes privileged sections like the shell integration that we planned to handle differently, both for security reasons and as we wanted to decouple the way the shell works on top of the display server from the application-facing protocol”

so they would rather have:

“An outer-shell together with a frontend-firewall that allow us to port our display server to arbitrary graphics stacks and bind it to multiple protocols.”

First of all, there’s nothing privileged about the shell protocol Wayland is exposing. wl_shell and wl_shell_surface (the “shell protocols”) are part of the Wayland core protocol, yes, but as I’ve explained on this post, it’s all customizable for whatever UI needs. Nevertheless, their usage is completely optional and anyone can build a different shell and stack with the rest of Wayland, just like tablet-shell protocol for instance does. Still, this will be Wayland and use the shiny libwayland for IPC.

Therefore I don’t think Canonical should justify their new project because Wayland “does not fulfill .. requirements completely”. There are no technical reasons Ubuntu cannot use Wayland in principle. What they wrote there is a very very mean excuse instead.

The Ceilometer team is pleased to announce that tomorrow Tuesday 5th March 2013 will be the second bug squash day for Ceilometer.

We wrote an extensive page about how you can contribute to Ceilometer, from updating the documentation, to fixing bugs. There's a lot you can do. We've good support for Ceilometer built into Devstack, so installing a development platform is really easy.

The main goal for this bug day will be to put Ceilometer in the best possible shape before the grizzly-rc1 release arrives (14th March 2013). This version of Ceilometer should be the last one before the final Grizzly release, so it's a pretty important one.

We'll be hanging out on the #openstack-metering IRC channel on Freenode, as usual, so feel free to come by and join us!

I'm using a lot of branches. Almost one per feature or bug, and they add up quickly. Why I'm doing this doesn't matter for this post, but I found it to be a good workflow. The problem with that is of course that after a while I forget which branch was for what, or what branch I worked on three weeks ago. So I started hacking up some git helpers.

I pushed them to https://github.com/whot/git-branch-tools today, feel free to use them or improve on them.

Archiving branches

Some branches are not actively developed anymore but should still be preserved for posterity. These branches are clogging up the branch view.

git archive-branch mybranch

  ...
  archive/2013/touch-test-libtool-linker-issues
  archive/2013/two-screen-coordinates
  archive/2013/wrong-signal-logging-merge
  archive/2013/xi2-protocol-tests
  archive/2013/xi21-confine-to
  archive/2013/xorg-conf-init-cleanup
  attic
  bugfix/xts-segfault
  devel
  fedora-17-branch
  fedora-rawhide-branch
  for-keith
  high-keycodes
  master
  memleak
* next
  ...

Showing recent branches

Working on many branches can mean you forget which branch you worked on last week, or the week before.

git recent-branches

next                                 4 hours ago    last commit 6 days ago
server-1.13-branch                   4 hours ago    last commit 2 weeks ago
touch-grab-race-condition-56578-v2   3 days ago     last commit 3 days ago
touch-grab-race-condition-56578      4 days ago     last commit 6 days ago      †
bug/xts-segfaults                    6 days ago     last commit 6 days ago      †
master                               6 days ago     last commit 3 weeks ago
for-keith                            10 days ago    last commit 2 weeks ago
memleak                              13 days ago    last commit 2 weeks ago

Better branch descriptions

Can't remember what branch "fix-race-condition" was? Me neither. That's what

git branch-description [branchname] [upstream]

:: whot@yabbi:~/xorg/xserver (next)> git-branch-description touch-grab-race-condition-56578-v2
Branch       touch-grab-race-condition-56578-v2
Branched:    Thu Feb 14 11:05:48 2013 -0800
Last commit: Fri Mar 1 16:37:49 2013 +1000

Fixes for https://bugs.freedesktop.org/show_bug.cgi?id=56578, second attempt

============================ Unmerged commits =============================
Commits on touch-grab-race-condition-56578-v2 not in next:
68b937046f278d53de14b586dbf7fd5aa7367f59 Xi: return !Success from DeliverTouchEmulatedEvent if we didn't deliver
f8baab8ac32e5abb31bcd1bb4f74e82d40208221 Xi: use a temp variable for the new listener
9cbb956765c7b4f1572ab2100f46504bf6313330 dix: don't set non-exisiting flags on touch events
2a5b3f2f2293f4a428142fffdb1b6e8ffbbb5db0 dix: fix a comment
76e8756545951d7f13ca84a4bd24fe5f367c5de2 Xi: compress two if statements with the same body
61b06226a43839ed75126f9c54d47bc440285e21 dix: update coords for touch events in PlayReleasedEvents
bd1a5423bbb02a349991a52f4997e830a0dc1992 Xi: add a comment to make a condition a bit clearer
78b26498085a7589e1f4d9ac3c21b69dc3227f87 Xi: not having an ownership mask does not mean automatic acceptance
c7271c7e05cdbeb35a3558223f9c2d6544504c4c dix: don't prepend an activated passive grab to the listeners
71ee72c97e459ef76984e6da64e5dab0ce6e4465 Xi: if we delivered a TouchEnd to a passive grab, end it
9b6966187fd0e6fb7ad3c2c1073456d96e3adab0 Xi: if a pointer grabbing listener gets the touch end, the touch is over
5afef18196ce70faec3e94379c3e6d3767660c4a FIXME: Xi: fix lookup in ActivateEarlyAccept
3784283be1f482a0f039f2eb790c0c8c2cc4bedb Xi: update the core listener state if we delivered the event
3570ef1244c87aef92db97df6e2b921529ffb75a Xi: if a passive async grab is activated from an emulated touch, accept
9bef901d8e28d48f43da3167219b02ad1dba27d8 Xi: save state for early acceptance
7d51022becd5af124896817030a10eedf7f1783a Xi: when punting to a new owner, always create TouchEnd events
4775cdb0d9a2513edcf27a9c4c1916e8213c397b Xi: use public.processInputProc to replay the touch history
431b128b9138af7a208b63d4eb5b917d94c08129 Xi: Don't emit a TouchEnd event to a frozen device
4126d64f6a40d5568b2d1412d519325c02786c9a dix: AllowSome is equivalent to TouchAccept
33421e91a52be91d7121c7c2146ff7bb53bea638 dix: move EmitTouchEnd to touch.c
54f8884aef275b15f2c42e3350e2b4968124af01 dix: XAllowEvents() on a touch event means accepting it

Commits on touch-grab-race-condition-56578-v2 already merged to next:

================================= Activity =================================
e7b4b83 HEAD@{5 hours ago}: checkout: moving from touch-grab-race-condition-56578-v2 to server-1.13-branch
9cbb956 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 9cbb956
d58ddeb HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to d58ddeb
7c3968b HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 7c3968b
a354dd8 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to a354dd8
fdf4869 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to fdf4869
82be6b2 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 82be6b2
82be6b2 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 82be6b2
68b9370 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 68b9370
151eff1 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 151eff1
57fa0b9 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 57fa0b9
b43e866 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to b43e866
ef6a120 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to ef6a120
9064294 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 90642948cc78834d95f7a3bddaac7ff77b68ed7e
9064294 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 90642948cc78834d95f7a3bddaac7ff77b68ed7e
6513e0e HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 6513e0e
0d60ba6 HEAD@{3 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 0d60ba6
dd23302 HEAD@{4 days ago}: checkout: moving from f21354da571dcd39ae1423388298d5c61d3e736d to touch-grab-race-condition-56578-v2
f21354d HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to f21354da571dcd39ae1423388298d5c61d3e736d
0d60ba6 HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 0d60ba6
ae2cac9 HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to ae2cac99a75917d6c4d34b8aa4aeaec0b5d32da7
c2b3d37 HEAD@{4 days ago}: checkout: moving from d75925b9fb8b24c8134b5082294e82abf83294af to touch-grab-race-condition-56578-v2
0d60ba6 HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 0d60ba683e7e95049c01ac5dba48a2f5fd80d9b9
c2b3d37 HEAD@{4 days ago}: checkout: moving from 0d60ba683e7e95049c01ac5dba48a2f5fd80d9b9 to touch-grab-race-condition-56578-v2
0d60ba6 HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 0d60ba683e7e95049c01ac5dba48a2f5fd80d9b9
0d60ba6 HEAD@{4 days ago}: checkout: moving from 90642948cc78834d95f7a3bddaac7ff77b68ed7e to touch-grab-race-condition-56578-v2
9064294 HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 90642948cc78834d95f7a3bddaac7ff77b68ed7e
8e5adb4 HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578-v2 to 8e5adb4ef8bafa2a3188e69409e2908f80288311
033b932 HEAD@{4 days ago}: checkout: moving from touch-grab-race-condition-56578 to touch-grab-race-condition-56578-v2

After giving a talk at FOSDEM last month on this topic, I finally finished my work on GLX_MESA_query_renderer... at least finished enough to send out for review by a wider audience.

The extension is modeled after Apple's CGLDescribeRenderer interface. Before even embarking on this project, several ISVs told me that they really liked that interface. There are some small changes, but it's still pretty similar.

The glXQueryRendererIntegerMESA and glXQueryRendererStringMESA functions allow applications query a bunch of aspects about the driver and hardware before creating a context:

• Driver version
• Driver vendor
• Hardware vendor (may be different from driver vendor!)
• Hardware chipset (PCI ID)
• Available memory
• UMA vs. non-UMA
• Supported and prefered (by the driver) GL profiles

The last one is really important to know before creating a context because it may influence what sort of context that app creates.

The initial branch, with the extension spec, is available at:

http://cgit.freedesktop.org/~idr/mesa/log/?h=query-renderer

x-on-resize: a simple display configuration daemon

I like things to be automated as much as possible, and having abandoned Gnome to their own fate and switched to xfce, I missed the automatic display reconfiguration stuff. I decided to write something as simple as possible that did just what I needed. I did this a few months ago, and when Carl Worth asked what I was using, I decided to pack it up and make it available.

Automatic configuration with a shell script

I’ve had a shell script around that I used to bind to a key press which I’d hit when I plugged or unplugged a monitor. So, all I really need to do is get this script run when something happens.

The missing tool here was something to wait for a change to happen and automatically invoke the script I’d already written.

Resize vs Configure

The first version of x-on-resize just listened for ConfigureNotify events on the root window. These get sent every time anything happens with the screen configuration, from hot-plug to notification when someone runs xrandr. That was as simple as possible; the application was a few lines of code to select for ConfigureNotify events, and invoke a program provided on the command line.

However, it was a bit too simple as it would also respond to manual invocations of xrandr and call the script then as well. So, as long as I was content to accept whatever the script did, things were fine. And, with a laptop that had a DisplayPort connector for my external desktop monitor, and a separate VGA connector for projectors at conferences, the script always did something useful.

Then I got this silly laptop that has only DisplayPort, and for which a dongle is required to get to VGA for projectors. I probably could write something fancy to figure out the difference between a desktop DisplayPort monitor and DisplayPort to VGA dongle, but I decided that solving the simpler problem of only invoking the script on actual hotplug events would be better.

So, I left the current invoke-on-resize behavior intact and added new code that watched the list of available outputs and invoked a new ‘config’ script when that set changed.

The final program, x-on-resize, is available via git at

git://people.freedesktop.org/~keithp/x-on-resize

I even wrote a manual page. Enjoy!

The OpenStack Technical Committee have voted these last weeks about graduation of Heat and Ceilometer, to change their status from incubation to integrated.

The details of the discussion can be found in the TC IRC meetings logs for the brave. The results are:

Approve graduation of Heat (to be integrated in common Havana release)?

yes: 10, abstain: 1, no: 1

Approve graduation of Ceilometer (to be integrated in common Havana release)?

yes: 11, abstain: 1

Therefore both projects have been graduated from Incubation to Integrated status. That means that Heat and Ceilometer will be released as part as OpenStack for the next release cycle Havana, due in Autumn 2013.

For people being curious, we the Ceilometer team put up a nice wiki page about our status and what we think we were ready to jump. For the curious, The OpenStack Technical Committee charter has some explanations about the incubation and integration process.

What about Grizzly?

Both projects will be released with Grizzly too, obviously, since they already follow the release process of OpenStack.

What about core?

The question that has been raised several times to me is if that means the projects are becoming Core projects. The answer is no, because how to become a Core project is still under discussion and is more a matter for the Board of Directors than the Technical Committee. But this is definitely a step in this direction.

Anyway, from a technical point of view, this means both projects are now onboard with other OpenStack components so you can enjoy them!

Kernel setup

root@localhost:~> modprobe elo
root@localhost:~> inputattach -elo /dev/ttyS0 --daemon

udev setup

$> cat /lib/udev/rules.d/99-elographics.rules
ACTION=="add|change", SUBSYSTEM=="tty|pnp", KERNEL=="ttyS4", \
        RUN+="/sbin/modprobe elo", \
        RUN+="/sbin/inputattach -elo /dev/%k --daemon"

$> cat /lib/udev/rules.d/99-wacom-w8001.rules
ACTION=="add|change", SUBSYSTEM=="tty|pnp", KERNEL=="ttyS[0-9]*", \
        ATTRS{id}=="WACf*", \ 
        RUN+="/sbin/modprobe wacom_w8001", \
        RUN+="/sbin/inputattach -w8001 /dev/%k --daemon"

X.Org configuration

$> cat /lib/udev/rules.d/99-elographics.rules
ACTION=="add|change", SUBSYSTEM=="tty|pnp", KERNEL=="ttyS4", \
        RUN+="/sbin/modprobe elo", \
        RUN+="/sbin/inputattach -elo /dev/%k --daemon", \
        ENV{ID_INPUT}="1", ENV{ID_INPUT_TOUCHSCREEN}="1"

$> cat /etc/X11/xorg.conf.d/99-elographics.conf
Section "InputClass"
  Identifier "elographics config"
  MatchProduct "Elo Serial TouchScreen"
  Option "EmulateThirdButton" "on"
EndSection

X.Org configuration on Red Hat Enterprise Linux (RHEL) 6.x

$> cat /etc/hal/fdi/policy/10-elographics.fdi
<?xml version="1.0" encoding="ISO-8859-1"?>
<deviceinfo version="0.2">
  <device>
    <match key="info.category" contains="input">
      <match key="info.product" contains="Elo Serial TouchScreen">
        <merge key="input.x11_driver" type="string">evdev</merge>
        <!-- next line is equivalent to an xorg.conf.d statement of
                Option "Calibration "40 4000 40 4000" -->
        <merge key="input.x11_options.calibration" type="string">40 4000
40 4000</merge>
      </match>
    </match>
  </device>
</deviceinfo>