planet.freedesktop.org

R_AMD64_PC32 error? There, I Fixed It!

2012-02-11T17:38:36+00:00

I try to fairly regularly build recent git checkouts of all the upstream modules from X.Org (at least all those listed in the current build.sh) on Solaris. Normally I do this in 32-bit mode on x86 machines using the Sun compilers on the latest Solaris 11 internal development build, but I also occasionally do it in 64-bit mode, or with gcc compilers, or on a SPARC machine. This helps me catch issues that would break our builds when we integrate the new releases before those releases happen. (Ideally I'd set up a Solaris client of the X.Org tinderbox, but I've not gotten around to that.)

Anyways, recently I finally decided to track down an error that only shows up in the 64-bit builds of the xscope protocol monitor/decoder for X11 on Solaris. The builds run fine up until the final link stage, which fails with:

ld: fatal: relocation error: R_AMD64_PC32: file audio.o: symbol littleEndian: value 0x8086c355 does not fit
ld: fatal: relocation error: R_AMD64_PC32: file audio.o: symbol ServerHostName: value 0x8086b4fe does not fit
ld: fatal: relocation error: R_AMD64_PC32: file decode11.o: symbol LBXEvent: value 0x808664c3 does not fit
(and over 150 more symbols that didn't fit)

A google search turned up some forum posts, a blog post, and an article on the AMD64 ABI support in the Sun Studio compilers. And indeed, the solutions they offered did work - building with -Kpic did allow the program to link.

But is that really the best answer? xscope is a simple program, and shouldn't be overflowing the normal memory model. Once it linked, looking at the resulting binary was a bit shocking:

% /usr/gnu/bin/size  xscope
   text	   data	    bss	    dec	    hex	filename
 416753	   5256	2155921980	2156343989	808732b5	xscope

% /usr/bin/size -f xscope

23(.interp) + 32(.SUNW_cap) + 5860(.eh_frame_hdr) + 27200(.eh_frame)
 + 2964(.SUNW_syminfo) + 5944(.hash) + 4224(.SUNW_ldynsym)
 + 17784(.dynsym) + 14703(.dynstr) + 192(.SUNW_version)
 + 1482(.SUNW_versym) + 3168(.SUNW_dynsymsort) + 96(.SUNW_reloc)
 + 1944(.rela.plt) + 1312(.plt) + 291018(.text) + 33(.init) + 33(.fini)
 + 280(.rodata) + 38461(.rodata1) + 1376(.got) + 784(.dynamic)
 + 1952(.data) + 0(.bssf) + 1144(.picdata) + 0(.tdata) + 0(.tbss)
 + 2155921980(.bss) = 2156343989

% pmap -x `pgrep xscope`
26151:	./xscope
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
0000000000400000        408        408          -          - r-x--  xscope
0000000000476000          8          8          8          - rw---  xscope
0000000000478000    2105388       1064       1064          - rw---  xscope
0000000080C83000         52         52         52          - rw---    [ heap ]
[....]
FFFFFD7FFFDF8000         32         32         32          - rw---    [ stack ]
---------------- ---------- ---------- ---------- ----------
        total Kb    2108668       3204       1300          -

Two gigabytes of .bss space allocated!?!?! That can't be right. Looking through the output of the elfdump and nm programs a single symbol stood out:

Symbol Table Section:  .SUNW_ldynsym
     index    value              size              type bind oth ver shndx          name
[...]
      [89]  0x00000000009ff280 0x0000000080280000  OBJT GLOB  D    1 .bss           FDinfo

[Index]   Value                Size                Type  Bind  Other Shndx   Name
[...]
[528]   |            10482304|          2150105088|OBJT |GLOB |0    |28     |FDinfo

Unfortunately, that wasn't one of the ones listed in the linker errors, since it's starting address fit inside the normal memory model, but everything that came after it was out of range.

So what is this giant static allocation for? It's defined in scope.h:

#define BUFFER_SIZE (1024 \* 32)

struct fdinfo
{
  Boolean Server;
  long    ClientNumber;
  FD      pair;
  unsigned char   buffer[BUFFER_SIZE];
  int     bufcount;
  int     bufstart;
  int     buflimit;     /\* limited writes \*/
  int     bufdelivered; /\* total bytes delivered \*/
  Boolean writeblocked;
};

extern struct fdinfo   FDinfo[StaticMaxFD];

So it allocates a 32k buffer for up to StaticMaxFD file descriptors. How many is that? For that we need to look in xscope's fd.h:

/\* need to change the MaxFD to allow larger number of fd's \*/
#define StaticMaxFD FD_SETSIZE

and from there to the Solaris system headers, which define FD_SETSIZE in <sys/select.h>:

/\*
 \* Select uses bit masks of file descriptors in longs.
 \* These macros manipulate such bit fields.
 \* FD_SETSIZE may be defined by the user, but the default here
 \* should be >= NOFILE (param.h).
 \*/
#ifndef FD_SETSIZE
#ifdef _LP64
#define FD_SETSIZE      65536
#else
#define FD_SETSIZE      1024
#endif  /\* _LP64 \*/

So this makes the buffer fields alone in FDinfo become 65536 \* 32 \* 1024 bytes, aka 2 gigabytes.

Thus in this case, while compiler flags like -Kpic allow the code to link, using -DFD_SETSIZE=256 instead, builds code that's a little bit saner, fits in the normal memory model, and is less likely to fail with out of memory errors when you need it most:

% /usr/gnu/bin/size -f xscope
   text	   data	    bss	    dec	    hex	filename
 409388	   3352	8449804	8862544	 873b50	xscope

% pmap -x `pgrep xscope`
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
0000000000400000        404        404          -          - r-x--  xscope
0000000000475000          4          4          4          - rw---  xscope
0000000000476000       8248         20         20          - rw---  xscope
0000000000C84000         52         52         52          - rw---    [ heap ]
[...]
FFFFFD7FFFDFD000         12         12         12          - rw---    [ stack ]
---------------- ---------- ---------- ---------- ----------
        total Kb      11500       2136        232          -

Of course that assumes that xscope is not going to be monitoring more than about 120 clients at a time (since it opens two file descriptors for each client, one connected to the client and one to the real X server), and still wastes many page mappings if you're only monitoring one client. The real fix being worked on for the next upstream release is to make the buffer allocation be dynamic, and allocate just enough for the number of clients we actually are monitoring.

The moral of this story? Just because you can make it build doesn't mean you've fixed it well, and sometimes it's useful to understand why the linker is giving you a hard time.

Wolfram|Alpha Pro: data science made easy

2012-02-10T15:59:35+00:00

Over at my RedMonk blog, I wrote about how a new tool called Wolfram|Alpha Pro makes it easy for anyone to do basic data science. If you have any interest in data analysis, doing data science, or anything along those lines, check it out.

FOSDEM Aftermath.

2012-02-10T12:56:44+00:00

FOSDEM was awesome this year. We had an overbooked schedule for our DevRoom, we inaugurated the beautiful and fantastic K building, and i got to present the lima driver.

First off, i would like to thank the FOSDEM organizers and the ULB. The already unique event that is FOSDEM just keeps getting better and better. Pascal & friends: congratulations, like every year, you've outdone yourselves.

Secondly, i would like to thank all the speakers in my devroom. It is clear by now why the first-come-first-serve algorithm has to be used, and it is also clear that it is working. But thank you all for making this a successful event (even Chris, who couldn't make it due to a train derailment). I hope you guys had a lot of fun too, both during your talk and with the rest of FOSDEM.

Lastly, to all those who attended my talk (and those who couldn't get in anymore as well): Thank you all for your very positive feedback. No matter what happens with lima in future, this talk will be the most memorable moment. (oh, and a big thanks to Will Stephenson, from SuSE and KDE, for getting a webcam up that quickly). To whoever shouted something along the lines of "we don't see that, it looks like a perfect cube to us" when the caching went off in the rotating cube hack: this is the open source spirit in its most tangible form. Thank you very much.

To end this post, let me plug the lima website again. We also have a mailinglist and the #lima channel on freenode. The limare code has been available since yesterday night. Heise and lwn posted the story already, and the videos from the FOSDEM talk should soon hit phoronix as well.

Doom3 demo benchmarking made easy

2012-02-09T20:31:29+00:00

So i decided to give another shoot at using doom3 as benchmark for my GPU work. As i don't have the full game (shame on me) i have to use the demo but the demo is not usefull for recording and playing back anything. But now things are opensource yeah ! So with git://git.iodoom.org/iodoom3/iodoom3.git and http://people.freedesktop.org/~glisse/0001-doom3-fix-demo-allow-demo-to-replay-demo-file.patch i was able to build doom3 demo on linux :) (scons . TARGET_DEMO=1 NOCURL=1).

Then just uncompress doom3-linux-1.1.1286-demo.x86.run replace gamex86.so by one from iodoom gamex86-demo.so. At that point you can launch the demo start a game and when ever you think it's good time recorddemo mystuff in doom3 console to start recording then stoprecording once you think you have enough frames. To replay your demo doom3-demo +timedemoquit mystuff.

Note you need 1.1.1286 as iodoom have a checksum for this file hardcoded (i haven't try with older demo but skipping the checksum check might work).

So now i have something a little more recent and different from other benchmark i often use (openarena, nexuiz). I wish their were some free advance GL demo with nice content we could use...

Time for some news

2012-02-09T20:06:58+00:00

Time flies by quickly, and another couple of weeks has past. So time for your regular news from the Intel Linux Graphics land!

Starting with the hot topic of the past years, Wayland 0.85 has just been released! This release marks the first milestone in the road to a stable 1.0 Wayland and Weston release. After it, we’ll have a 0.90 release, which would be the one to mark API freeze and will begin the preparation to the final 1.0 release. Of course, additional releases could happen meanwhile. Wayland has evolved a lot for the past years, so make sure to check it out.

Besides Wayland, lots of activities have been going on with all the projects.

On kernel side, Daniel Vetter has sent out his new drm-intel-next tree, which features lots of fixes from Ben, Eric and Chris, improvements to swizzling handling, fence accounting improvements, VTd workarounds for Ironlake, simplified debugfs handling code and an improved i915_error_state logging.

Besides those improvements, lots of attention was directed towards interlaced modes support on Intel GPUs, which resulted in a large series of patches from Paulo Zanoni and Daniel Vetter that should improve support for interlaced and non-interlaced modes and made them, well, work correctly . If you have been experiencing any sort of problems with such modes, please, try the patches and verify if they solve the issues. Chances are that they will, but if they won’t, please, let us know!

Among other interesting patches in kernel for the past few weeks, Ivy Bridge hard-hang fixes are among the most notable ones. Those patches toggle a couple of work-arounds for issues which randomly affected some of the Ivy Bridge machines – and resulted in complete system hangs. Due to their random nature, and specially due to the fact that they mostly affected low-resolution modes (chances to hang your machine in a 320×240 resolutions were noticeable much higher than in any other resolution), we had hard time to track them previously. In fact, we only managed to reproduce them consistently in the last week – and luckily we were able to come with patches which apparently fixed most of the problems. The patches will probably be included into 3.3 kernel via the drm-intel-fixes branch, and if they won’t cause any other issues we’ll backport them to 3.2 kernel as well.

Another interesting patch is the one which applies the missed IRQ fix (a.k.a., the voodoo patch ) to Sandy Bridge platform as well. So far, we had a couple of reports which mentioned that a very similar issue to the one which was fixed on Ivy Bridge last month was happening on Sandy Bridge as well. If you are suffering from this issue, please, give it a try and let us know if it solved the issue to you or transformed it into anything else (like a GPU hang which happened on one of the machines instead of missed IRQ. Those results would be even more interesting, as they could tell us what we are missing and how to fix the problem for good).

Additionally, lots of development was targeted on improving the semaphores issues detection and avoidance, and for better debugging support in the i915_error_state log. Usually, this log is all that remains of a GPU hang – so the more information we can get from there, the better.

And finally, aiming at finally discovering what root-causes all the rc6 issues out there, I sent out a patch which allows to hopefully isolate the issue. Leann Ogasawara from Canonical was also kind enough to pre-build Ubuntu kernels with this patch – so if you are affected by a RC6 issue, and are running Ubuntu – please, give them a try!!

On Mesa side, things are moving very quickly towards the Mesa 8.0 release. The 8.0-RC2 version was already released, and final 8.0 version is expected to happen in a very near future. Who knows, perhaps by the time you’ll be reading this line, it will already be among us . Mesa 8.0 release looks very exciting so far – besides the GL 3.0 extensions and GLSL 1.30 support, it also brings amazing performance and stability improvements. Hopefully you’ll be able to see it with your own eyes when the next stable release will be out – or, if you don’t want to wait that long, or want to use your very last chance to report any major show-stopper bugs in the 8.0 branch – now is the time.

On the xf86-video-intel side, as usual, hundreds of patches were committed by Chris Wilson, mostly targeting the SNA backend, and also the Glamor integration. Since the beginning of the year, git log already shows 380 new patches. Out of those, 368 are SNA-specific for now. Phoronix has run yet another round of SNA testing in the past weeks with some interesting results. Things are getting very interested with this backend.

And finally, moving to the intel-gpu-tools project, two new tools were proposed in the past few days – one for testing different panel fitter settings (intel_panel_fitter, from Paulo Zanoni); and another to demonstrate the sprite features, available for SNB/IVB platform, which uses 3 new IOCTLs included in the latest drm subsystem: GETPLANERESOURCES, GETPLANE, and SETPLANE. Planes and sprites support are ones of the latest additions to the core drm subsystem, and in my opinion they certainly are among the most interesting items there. I believe that more and more applications will be using them at some point (like the 1st one mentioned in this post – yes, the one which starts with ‘W‘), which is great.

Multitouch in X - Multitouch-touchpads

2012-02-07T12:27:19+00:00

This post is part of a series on multi-touch support in the X.Org X server.

In this post, I'll outline the handling of touch events for multitouch-capable touchpads. Multi-touch touchpads that are supported are those that provide position information for more than one finger. The current version of the synaptics X driver does some tricks to pretend two-finger interaction on single-finger touchpads - such devices are not applicable here.

Touchpads are primarily pointer devices and any multi-touch interaction is usually a gesture. In the protocol, such devices are of the type XIDependentDevice and the server does adjust touch event delivery. I've already hinted at this here but this time I'll give a more detailed explanation.

Touch event delivery

Unlike for direct touch devices such as touchscreens, dependent devices have a different picking mechanism in the server. We assume that all gestures are semantically associated with the cursor position. For example, for scrolling, you would move the cursor on top of the window to be scrolled, then you would start scrolling. The server thus adjusts event delivery accordingly. Whereas for direct touch devices the touch events are delivered to whichever window is at the position of the touch, touch events from dependent devices are always delivered to the window underneath the pointer (grab semantics are adjusted to follow the same rules). So if you start a gesture in the top-left corner of the touchpad, the window underneath the cursor gets the events with the top-left coordinates. Note that the event and root coordinates always reflect the pointer position.

The average multi-touch touchpad has two modes of operation: single-finger operation where the purpose is to move the visible cursor and multi-finger operation which is usually interpreted into a gesture on the client. These two modes are important, as they too affect event delivery. The protocol specifies that any interaction with the device that serves to move the visible cursor only should not generate touch events, and that touch events will start once that interaction becomes a true multi-touch interaction. This leaves the drivers a little bit of leeway, but the current implementation in the synaptics driver is the following:

A user places one finger on the touchpad and moves. The client will receive regular pointer events.
The user places a second finger on the touchpad. The client will now receive a TouchBegin event for the first and the second touch, at their respective current positions in device coordinate range.
Movement of either finger now will generate touch events, but no pointer events.
Any other fingers will generate touch events only.
When one of two remaining fingers on the touchpoint ceases the touch, a TouchEnd is sent for both the terminating touch and the remaining touch. The remaining finger will revert to sending pointer events.

Legacy in-driver gestures

As you are likely aware, the synaptics driver currently supports several pseudo gestures such as tap-to-click or two-finger scrolling. These gestures are interpreted in the driver, thus the server and client never see the raw data for them.

With proper multi-touch support these gestures are now somewhat misplaced. On the one hand, we want the clients to interpret multitouch, on the other hand we want the gestures to be handled in the same manner in all applications. (Coincidentally, this is also a problem that we need to solve for Wayland).

We toyed with ideas of marking emulated events so clients can filter but since we do need to be compatible to the core and XI 1.x behaviours, we only found one solution: any in-driver gestures that alter event deliver must not generate touch events. Thus, if the driver is set to handle two-finger scrolling, the clients will only see the pointer events and scroll events, they will not see touch events from two-fingers. To get two-finger scrolling handled by the client, the in-driver gesture must be disabled. The obvious side-effect of that is that you then cannot scroll in applications that don't support the gestures. Oh well, it's the price we have to pay for having integrated gesture support in the wrong place.

Japanese Standing Cat

2012-01-31T11:42:00+00:00

I recently purchased and installed a standing desk. While I normally don't blog about things in my personal life, I figured that this was permissible since it directly affects my ability to write code.

This standing desk is a Fredrik "computer work station," but let's be honest, here: it's a standing desk. It's pretty sturdy and removes the need to hack together various Ikea desks to produce reasonably-scaled tabletops. Pictured here is my desk, in its natural habitat. My workstation and hacked-apart AGP box both have their own monitor, and there is plenty of tabletop room for anything I need to have at my hands.

For those in the audience who aren't yet aware of standing desks, and don't want to read Wikipedia on standing desks, here are the abridged notes:

Pros

Reduces back stress and pain after a few months

Reduces risk of DVT

When monitors are elevated, reduces neck stress and pain

Cons

Slightly more expensive than sitting desks

Hard to find

Causes foot and ankle stress for the first few weeks

Turns one into a hipster programmer

So, on the bright side, this desk has definitely improved my back pain. But, if I start writing Node.js code soon, we'll all know who to blame.

STOP ACTA...

2012-01-30T19:20:22+00:00

I suspect that I'll be receiving quite a few more hits here given the recent retweets. Unfortunately I haven't had much time to update my blog... In fact, I may not have any time in the future.

On a more positive note, I am much more active on Twitter, so you can follow me there (@omcfadde)

Of course the topics we're discussing, SOPA, ACTA, etc are definitely not pleasant or positive. Let's show them we mean business and that we will not accept these agreements!

STOP ACTA

Updates (possibly) coming in the future.

Epäilen, että tulen saa melkoisesti lisää osumia tässä äskettäisten retweets. Valitettavasti minulla ei ole ollut paljon aikaa päivittää blogiin ... Itse en ehkä ole mitään aikaa jatkossa.

Myönteistä huomata, olen paljon enemmän aktiivisia Twitterissä, jotta voit seurata minua sinne (@omcfadde)

Tietenkin aiheita olemme keskustelleet, SOPA, ACTA jne. eivät varmasti miellyttävä tai positiivinen. Katsotaanpa näytä heille että olemme tosissamme ja että me ei hyväksy näitä sopimuksia!

STOP ACTA (In Englanti - Lue!)

Päivitykset (mahdollisesti) tulossa tulevaisuudessa.

Summary of GStreamer Hackfest

2012-01-29T17:17:39+00:00

So as I talked about in my last blog post we had a great GStreamer hackfest. A lot of things got done and quite a few applications got an initial port over to 0.11. For instance Edward Hervey ended up working on porting the Totem video player, or rather trying to come up with a more optimized design for the Clutter-gst as the basis port was already done.

Another cool effort was by Philippe Normand from Igalia who put a lot of effort into porting WebKit to use 0.11. His efforts where rewarded with success as you can see in this screenshot.

Jonathan Matthew had flown up all the way from Australia and made great progress in porting Rhythmbox over to the 0.11 API, a port which became hugely more useful after Wim Taymans and Tim-Phillip Muller fixed a bug that caused mp3 playback not to work .

Peteris Krisjanis made huge strides in porting Jokosher to 0.11. Although like Jason DeRose from Novacut and myself on Transmageddon he did end up spending a lot of time on debugging issues related to gobject-introspection. The challenge for non-C applications like Jokosher, Novacut, Transmageddon and PiTiVi is a combination of the API having changed quite significantly due to the switch to gobject-introspection generated bindings, some general immaturity challenges with the gobject-introspection library and finally missing or wrong annotations in the GStreamer codebase. So once all these issues are sorted things should look a lot brighter for language bindings, but as we discovered there is a lot of heavy lifting to get there. For instance I thought I had Transmageddon running quite smoothly before I suddenly triggered this gobject-introspection bug.

There was a lot of activity around PiTiVi too, with Jean-François Fortin Tam, Thibault Saunier and Antigoni Papantoni working hard on porting PiTiVi to 0.11 and the GStreamer Editing Services library. And knowing Jean-François Fortin I am sure there will soon be a blog with a lot more details about that .

Thomas Vander Stichele, who also wrote a nice blog entry about the event, was working with Andoni Morales Alastruey, both from Flumotion, on porting Flumotion to 0.11, but due to some of the plugins needed not having been ported yet most of their effort ended up being on porting the needed plugins in GStreamer and not so much application porting, but for those of you using plugins such as multifdsink, this effort will be of great value and Andoni also got started on porting some of the non-linux plugins, like the directsoundsink for Windows.

Josep Torra from Fluendo ended up working with Edward Hervey on hammering out the design for the clutter-gst sink at the conference, but he also found some time to do a port of his nice little tuner tool as you can see from the screenshot below.

Tuner tool for GStreamer 0.11

George Kiagiadakis kept hammering away at the qtGStreamer bindings, working both on a new release of the bindings for the GStreamer 0.10 series, but also some preparatory work for 0.11.

In addition to the application work, Wim Taymans, Tim-Phillip Müller and Sebastian Dröge from Collabora did a lot of core GStreamer clean ups and improvements in addition to providing a lot of assistance, bugfixing and advice for the people doing application porting. All critical items are now sorted in 0.11 although there are some nice to have’s still on the radar, and Wim plans on putting out some new releases next week, to kickstart the countdown to the first 1.0 release.

As for my own little pet project Transmageddon, it is quite far along now, with both manually configured re-encodes and profile re-encodes working. Still debugging remuxing though and I am also waiting for the deinterlacer to get ported to re-enable deinterlacing in the new version. For a screenshot take a look at the one I posted in my previous blogpost.

Updates from the Intel Linux Graphics land

2012-01-29T15:11:34+00:00

The real life and work issues had eaten most of my time for the past 2 weeks, but things seem to be geting back to normal now. So time for semi-regular updates from the Intel Linux Graphics land.

Starting with kernel, many notable changes happened in the past weeks. To make our -next merge window process go faster, a new process was adopted for the kernel patch and branch handling process. Let me elaborate a bit more about how this is going to work.

For the past months, we had the drm-intel-next branch (containing patches intended to go into the next merge window – e.g., when the 3.3 kernel is being developed, this is the branch which contains patches intended to be included in the 3.4 kernel version), and also drm-intel-fixes branch (containing patches for the currently developed kernel – e.g., patches which fix issues which were added during the past merge window. So for instance, those are the patches which go into 3.3-rcN kernel versions after the merge window). Both of those patches were maintained by Keith Packard.

However, the amount of patches combined with other tasks resulted in a somewhat slower patch queuing process. So Daniel Vetter has volunteered himself to maintain the drm-intel-next branch from now on, reviewing, queuing and managing the patches intended to get into the next kernel merge window (for now, those are targeting the 3.4 kernel version).

At the same time, Keith Packard will continue to maintain the drm-intel-fixes branch, managing and carrying stability patches for the current kernel.

And finally, to improve the testing and availability of newer features for older kernel versions, I’ve started my own drm-intel-backports set of branches, for 3.0, 3.1 and 3.2 kernel versions – containing all the latest and greatest patches from the latest kernel releases. Of course, those patches do not follow the usual kernel stable development model, so they introduce new features, capabilities and possible issues. But for the brave souls out there willing to see and test the absolutely latest development in our drivers on top of their kernels, it should be a great chance to do so.

(I assume that I was not that fast with the drm-intel-backports patch merging process in the past 2 weeks, so for now it only contains patches from the 3.2 kernel branch. I intend to backport the 3.3 and 3.4-next patches for all those versions in the coming days. Also, apparently those backported patches do not work correctly with pre-Ironlake chipsets on 3.0 kernel, and there are some display-related issues on top of 3.1 one – I’ll look into it as well. Sorry for the delay – but the real life interfered with my patching capability for the past weeks…)

Besides those changes, we had a large number of notable kernel advances in the past weeks:

One of the most interesting features is the support for interlaced modes in the Intel Linux Graphics cards. A great work by Peter Ross, Daniel Vetter and Paulo Zanoni resulted in a series of patches which provided support for interlaced modes. These patches will be queued for inclusion in the 3.4 kernel – but when I’ll get back to my -backports series of patches, I’ll certainly include them into the 3.[012] series as well.
More Ivy bridge 3-display pipes issues were fixed, such as the hibernation problem with 3rd pipe being active.
Stabilizing the almost-ready-to-launch Ivy bridge platform, the hopefully final forcewake-related fixes were included in both drm-intel-next and drm-intel-fixes branches, and were also cherry-picked by Greg to be included in the stable 3.2 kernel as well. This should solve the remaining missing interrupts problems we were experiencing.
Initial patches to support the hardware context switching were sent by Ben Widawsky, and are accessible in his own freedesktop.org repository for testing. Hardware-supported context switching allows to save the hardware registers and state for each process, so it could essentially help the GL_EXT_transform_feedback extension and advanced geometry shading state passing between different stages. And, of course, it can also improve the stability – hardware would ensure that different processes would not interfere with each other in a harmful way.
And almost a hundred pending patches were picked by Daniel Vetter for his new drm-intel-next branch. The next merge window certainly looks interesting for the Intel Linux Graphics support in this sense…

On Mesa side, the 8.0 release is getting really close. After the 8.0-rc1 release a couple of weeks ago, lots of patches were picked into the 8.0 branch, improving the stability, performance, and fixing many rendering issues – especially on the Ivy Bridge platform. More excitingly, the communication between Mesa and Unigine developers resulted in better support for open-source drivers in the just released Oil Rush game. It is really great to see such cooperation – and Unigine-based game certainly is one of the most advanced graphical releases of past years on Linux.

Speaking about the Mesa development, this final stabilization phase prior to the 8.0 release is a great chance for you to report issues, bugs and problems before the release which is expected in a few days from now. So if you observe any last-minute problems and regressions with Mesa 8.0 branch – please, let us know! This next release looks really impressive from the technical side, but your help in additional testing is more than welcome.

On Wayland side, lots of different patches and features were received in the past weeks. Wayland and Weston development goes on with great pace – and the ones of you to attend FOSDEM next week will have some really interesting stuff to gaze upon. But I won’t spoil the surprise .

On Xserver development, the 1.12 RC2 and 1.11.4 releases were unleashed last week, together with a new glproto, util-macros and xkeyboard-config versions. The development continues non-stop, with hundreds of patches and discussions happening in the mailing lists.

And finally, the phoronix feedback I got from you over the past weeks was amazing. Thank you all for all your questions and suggestions – it was really nice to hear what you have in mind, what you like (and dislike) in our drivers and what direction we should be heading. I’ll try to answer the last remaining questions in the next few day, and will summarize everything on a dedicated blog post.

Getting conned: eBay/Paypal fun

2012-01-28T16:19:00+00:00

After seeing, this article about "How secure is Paypal for eBay sellers" in this morning's Guardian, I'll share my personal experience with you.

In October, I sold my first generation MacBook Air on eBay, and got a buyer within a day for the £500 "Buy It Now" price. "Buy It Now" requires using Paypal, and the £500 (minus commission) appeared in my Paypal account¹. After a bit of to and fro, the buyer got in contact, and suggested that he come and pick it up. Saving about £30 of shipping, and sorting out the sale faster, strike me as good ideas.

The "buyer" said he couldn't come, sent one of his "employees". A very courteous man came to pick the laptop. In hindsight, he seemed slightly uncomfortable, and looked like he was very happy to see how easy it was going to be.

The spooky thing is that within 40 minutes -- note, not 3 hours, not a day after, not the day before) -- within 40 minutes of the laptop getting picked up, the holder of the eBay and Paypal accounts submitted an "unauthorised account activity claim", leading to "payment reversal" (me owing £500 to Paypal²).

During my call to eBay's customer support, I was told that "I had nothing to worry about" (I'm guessing that would be the case as long as I repaid the £500). Paypal promptly sent a mail mentioning they needed my help, but with very little possibilities from my side ("no courier tracking number? Give us the money now").

Surrey Police failed to find the culprits, with the 2 mobile numbers associated with the con only being pay-as-you-go phones (topped up in a little convenience store in North London that only keeps a day's worth of CCTV).

So my advices:

If you sell anything via eBay using Paypal, send it recorded, and keep the receipt.
If you bought a MacBook Air first generation with the serial W88500DJ12G, it's stolen, send me an e-mail.

And as opposed to Mssrs Lodge and Reakes, Paypal didn't reimburse me anything, and I'm £500 out of pocket.

¹: I'll pass you the details on eBay referring to a closed Paypal account that meant I got conned two days later than the "buyers" anticipated.
²: On an account that was already closed, see ¹.

Update: Added mention of eBay's ludicrously bad customer service.

Wacom tablets in GNOME 3.4

2012-01-27T13:07:00+00:00

Working from designs.

The cool stuff first

Cosimo Cecchi presents the updated Wacom settings

Go to YouTube directly if you can't see the video here.

A new arrival

As mentioned by Cosimo, we have a new library to help us implement the settings you saw: libwacom.

libwacom is there to give us metadata about tablets, whether or not they are connected to your system, the list of styli it supports, as well as information about the styli themselves. As you can see from the UI, it's pretty important that we know:

whether the tablet is builtin (so we know whether you can calibrate it)
which form factor it has
the list of styli it supports
for each stylus, its full name, the number of buttons, what it looks like

In the past, all this information was only available within the drivers (as comments), exported in different ways (sysfs attributes), non-machine readable in public documentation, or, worst of all, hidden in Wacom's internal drivers for OS X or Windows.

So if you have a Wacom tablet, send us a definition file for your tablet, so you can configure it with the impression that the software actually knows about your device.

Where's that configuration again

After knowing what each tablet had to offer, we had to have a way to match the definitions to XInput devices, assign settings per-tablet, and importantly, switch stylus configuration when the user switches stylus. This is done using the new GsdWacomDevice and GsdWacomStylus objects, shared between gnome-settings-daemon (which will apply the configuration) and gnome-control-center (which will set the configuration).

This also means we have a few debugging applications, such as list-wacom in gnome-settings-daemon, to show you the attached GsdWacomDevices, or test-wacom in gnome-control-center, to test display of particular tablets if you don't own them (this is the place where I spend a lot of time).

What's next

Peter Hutterer, my input buddy at Red Hat, who made the original Wacom panel for GNOME 3.2, and the first version of libwacom, is currently spending a lot of time on Multi-Touch, and fixing bugs I report in the Wacom driver.

Jason Gerecke, from Wacom, who did most of the initial work on calibration support, is working on the related display-mapping. This will allow choosing whether a tablet's working area is the whole desktop, or a single monitor when in multiple monitors are used.

For my part, after fixing the layout bugs that so annoy me in the settings panel, I'll be starting work on tablet button mapping. I look forward to making the LEDs on the tablet match up with the selected keyboard shortcut!

Many thanks to Cosimo and Monty for helping out with presenting the work, and doing the video.

Planet GNOME: Changes in planet membership policy

2012-01-27T02:56:51+00:00

Hello Planet GNOME readers and GNOME community members,

The board of directors has agreed that from now on, to be part of Planet GNOME, it is mandatory to be a member of the GNOME Foundation. In three weeks we will proceed to remove all blogs from people that are not foundation members. This policy change means a few things:

If your blog is in Planet GNOME and you are not a member of the GNOME Foundation, you have three weeks to become a member!
New planets additions from now on will take this policy into account.
This doesn't mean that your blog automatically gets into Planet GNOME if you are a member, the same blog review process will be applied to requests for addition.
There will be a slight exception with GSoC/GOPW members we will encourage them to join in the meantime, but being able to use the planet to report progress to the community is an important part of their work as interns.
We will delete the feed syndication from the planet if your membership expires.

The rationale behind this new policy is simple, we want to increase the value of becoming a foundation member. Think of this as the blogging equivalent of rocking an @gnome.org e-mail address.

Update: I have clarified the language behind the deletion of the feed. Someone in the comments thought I was talking about removing their blogs from blogs.gnome.org. This is not the case, if your membership expires your feed won't show up in the Planet, but that's it, if you have a blog in blogs.gnome.org none of your posts will be deleted.

Foundationally yours,
the Planet GNOME editors team.

The Case for the /usr Merge

2012-01-26T21:29:00+00:00

One of the features of Fedora 17 is the /usr merge, put forward by Harald Hoyer and Kay Sievers^[1]. In the time since this feature has been proposed repetitive discussions took place all over the various Free Software communities, and usually the same questions were asked: what the reasons behind this feature were, and whether it makes sense to adopt the same scheme for distribution XYZ, too.

Especially in the Non-Fedora world it appears to be socially unacceptable to actually have a look at the Fedora feature page (where many of the questions are already brought up and answered) which is very unfortunate. To improve the situation I spent some time today to summarize the reasons for the /usr merge independently. I'd hence like to direct you to this new page I put up which tries to summarize the reasons for this, with an emphasis on the compatibility point of view:

The Case for the /usr Merge

Note that even though this page is in the systemd wiki, what it covers is mostly orthogonal to systemd. systemd supports both systems with a merged /usr and with a split /usr, and the /usr merge should be interesting for non-systemd distributions as well.

Primarily I put this together to have a nice place to point all those folks who continue to write me annoyed emails, even though I am actually not even working on all of this...

Enjoy the read!

Footnotes:

[1] And not actually by me, I am just a supportive spectator and am not doing any work on it. Unfortunately some tech press folks created the false impression I was behind this. But credit where credit is due, this is all Harald's and Kay's work.

GStreamer Hackfest in Malaga update

2012-01-26T14:12:42+00:00

Things have been going really well here at the GStreamer Hackfest in Malaga. Thanks to the help of Ara and Yaiza from Nido Malaga, we have a great venue in downtown Malaga and they have also helped us greatly with sorting out food.
We have a great group of people here and are making great progress, and by tomorrow I hope we will have screenshots of quite a few applications running with GStreamer 0.11, for instance both Rhythmbox and Jokosher for instance is already screen shootable, if not fully functional

GStreamer Hackfest Malaga 2012

Also making good progress on Transmageddon, even if the move to GObject Introspection bindings are making things a bit more complicated. Screenshot below of the progress so far.

Transmageddon at Hackfest in Malaga 2012

Also a big thanks to Fluendo who is sponsoring the lunches at the hackfest and Collabora who is sponsoring tonight’s dinner. Ensuring that no hacker is left hungry during this hackfest.

Update: Yaiza took these photos from the hackfest

Interview with Arun Raghavan about PulseAudio

2012-01-25T15:45:33+00:00

With all the talk generated by Arun Raghavans blog post comparing PulseAudio and Audioflinger I thought it would be good to follow up with an interview with Arun about the latest developments in PulseAudio and the way forward for the project. You can find the PulseAudio interview here. I also made a new page listing all the Collabora developer interviews done so far. Enjoy

GStreamer Hackfest in Malaga

2012-01-23T09:39:45+00:00

Tomorrow I will be heading off to attend the GStreamer Application Porting Hackfest in Malaga, Spain. I think we have managed to pull together an absolutely incredible group of people for this event and I have great hopes that by next weekend we will have squashed a ton of bugs in GStreamer 0.11/1.0 and also have initial ports of a long range of important applications and bindings. This is the first time in GStreamer history that we are trying to hold a hackfest focused on application developers, but hopefully it will be the first of many and that they can become a good way for the core GStreamer community and the application development community to interact and collaborate more closely.

Also want to say a special thanks to the community members attending the event on their own and also to the companies sending their employees to the hackfest; Collabora, Fluendo, Flumotion and Igalia and finally a special thanks to the GNOME Foundation for sponsoring some of the attendees.

Hopefully I will be able to post some screenshots of a fully functional GStreamer 1.0 Transmageddon next weekend

A Little Bit is Better Than Nada

2012-01-22T12:00:00+00:00

It is a running joke in the X.Org community that X12 development will commence as soon as all of the X11 bugs in the FreeDesktop.org bug tracker have been closed. When I first heard this, my reaction was, of course, "Challenge accepted!" But where to start?

There were about 10,000 bugs open on the tracker at that point, a few years ago. We're definitely improving; there are about 9180 bugs open (at the time of writing) which could be considered relevant to any kind of desktop development. Many of those don't belong to anything in X11, but assuming they did...

Pretend that all of the card-carrying members of X.Org closed 92 bugs. That would be 1% per person. We could be done with X11! So, during XDC 2011, I pulled out my netbook, and started closing bugs. I picked, as my targets, things that nobody would miss, like Xprint.

I closed some Xprint-related bugs, before simply shuttering everything in the Xprint product. I also picked up a couple of stray bugs. Finally, with maintainer permission, I closed everything still assigned to xf86-video-nv.

The results:

24 Xprint-related bugs

37 bugs in Product: xprint

42 bugs in Component: driver/nv

37 bugs in Component: driver/nVidia (open)

#9455, #28657, #29832, #30129

A total of 144 bugs. We can do this, guys!

Plumbers Wishlist, The Third Edition, a.k.a. "The Thank You Edition"

2012-01-20T20:26:00+00:00

Last October we published a wishlist for plumbing related features we'd like to see added to the Linux kernel. Three months later it's time to publish a short update, and explain what has been implemented in the kernel, what people have started working on, and what's still missing.

The full, updated list is available on Google Docs.

In general, I must say that the list turned out to be a great success. It shows how awesome the Open Source community is: Just ask nicely and there's a good chance they'll fulfill your wishes! Thank you very much, Linux community!

We'd like to thank everybody who worked on any of the features on that list: Lucas De Marchi, Andi Kleen, Dan Ballard, Li Zefan, Kirill A. Shutemov, Davidlohr Bueso, Cong Wang, Lennart Poettering, Kay Sievers.

Of the items on the list 5 have been fully implemented and are already part of a released kernel, or already merged for inclusion for the next kernels being released.

For 4 further items patches have been posted, and I am hoping they'll get merged eventually. Davidlohr, Wang, Zefan, Kirill, it would be great if you'd continue working on your patches, as we think they are following the right approach^[1] even if there was some opposition to them on LKML. So, please keep pushing to solve the outstanding issues and thanks for your work so far!

Footnotes

[1] Yes, I still believe that tmpfs quota should be implemented via resource limits, as everything else wouldn't work, as we don't want to implement complex and fragile userspace infrastructure to racily upload complex quota data for all current and future UIDs ever used on the system into each tmpfs mount point at mount time.

XKB breaking grabs - CVE-2012-0064

2012-01-20T02:39:39+00:00

Given that there is a copious amount of misinformation being spread, here is a summary of CVE-2012-0064, straight from a horse's mouth.

Outline of the issue

The bug allows users to work around screen locking (e.g. gnome-screensaver) by hitting Control+Alt+keypad multiply or Control+Alt+keypad divide. This terminates the input grab the screensaver has and thus allows a user to interact with the desktop, skipping the password entry.

Affected versions

Affected is anyone running X server 1.11 or later (or release candidates thereof). So if "Xorg -version" shows something else on your box, stop worrying. I doubt any distribution would have back-ported the patches.

In Fedora/Red Hat land - the only distributions affected are Fedora 16 and current Fedora Rawhide. Both have been fixed, the F16 update is avaialble here. Note that the update is to xkeyboard-config, not to the server itself.

Fedora 15 is not affected. RHEL 4, 5, 6 and thus CentOS 4, 5, 6 and other derivatives are not affected. I believe that most other distributions have now pushed out updates as well, if you want to link to the respective updates, please do so in the comments.

Sergey has also pushed out xkeyboard-config 2.5 today with the fix included.

History of the feature

The X protocol does not allow the server to break grabs. Once a client has a grab, the server must wait for that client to release the grab, terminate, or the grab window to become unviewable. This is an issue when debugging applications - if your client has a keyboard grab, you cannot use the debugger since all key events will go to the client being debugged. To avoid this issue, the X server has had two combinations to break grabs: Control+Alt+Keypad multiply and Control+Alt+Keypad divide. They forced grab termination inside the X server and although against the protocol it made debugging possible. The option required explicit enabling in the xorg.conf.

These options were removed in server 1.4 and disabled since. Which made debugging hard, so last year we merged a patch to bring them back, together with some other features. They are triggered by XKB actions (as they used to be). The plan was to remove the XKB actions from the default keymap so that the action is available on request but not enabled by default. This is where a miscommunication happened, the removal from the default keymap never happened. So server 1.11 and vanilla xkeyboard-config ship with both the actions available and present in the current keymap. As a result, any user can break a grab from any application and thus get around screen locking.

Outline of the fix

To shoot yourself in the foot, you need two items: a gun and a trigger. We have removed the trigger. The fix we've now pushed into xkeyboard-config removes the actions from the default keymap and into an XKB option instead. So the fix does not remove the gun, but it requires the user to screw the trigger in themselves before trying to hurt themselves. In a default configuration, it is thus no longer possible to break the grab of your screensaver.

To re-enable grab debugging, run setxkbmap with "-option grab:break_actions" or enable "Allow breaking grabs with keyboard actions (warning: security risk)" in the "Miscellaneous compatibility options" in your keyboard layout configuration tool of choice.

systemd for Administrators, Part XII

2012-01-20T01:26:00+00:00

Here's the twelfth installment of my ongoing series on systemd for Administrators:

Securing Your Services

One of the core features of Unix systems is the idea of privilege separation between the different components of the OS. Many system services run under their own user IDs thus limiting what they can do, and hence the impact they may have on the OS in case they get exploited.

This kind of privilege separation only provides very basic protection however, since in general system services run this way can still do at least as much as a normal local users, though not as much as root. For security purposes it is however very interesting to limit even further what services can do, and shut them off a couple of things that normal users are allowed to do.

A great way to limit the impact of services is by employing MAC technologies such as SELinux. If you are interested to secure down your server, running SELinux is a very good idea. systemd enables developers and administrators to apply additional restrictions to local services independently of a MAC. Thus, regardless whether you are able to make use of SELinux you may still enforce certain security limits on your services.

In this iteration of the series we want to focus on a couple of these security features of systemd and how to make use of them in your services. These features take advantage of a couple of Linux-specific technologies that have been available in the kernel for a long time, but never have been exposed in a widely usable fashion. These systemd features have been designed to be as easy to use as possible, in order to make them attractive to administrators and upstream developers:

Isolating services from the network
Service-private /tmp
Making directories appear read-only or inaccessible to services
Taking away capabilities from services
Disallowing forking, limiting file creation for services
Controlling device node access of services

All options described here are documented in systemd's man pages, notably systemd.exec(5). Please consult these man pages for further details.

All these options are available on all systemd systems, regardless if SELinux or any other MAC is enabled, or not.

All these options are relatively cheap, so if in doubt use them. Even if you might think that your service doesn't write to /tmp and hence enabling PrivateTmp=yes (as described below) might not be necessary, due to today's complex software it's still beneficial to enable this feature, simply because libraries you link to (and plug-ins to those libraries) which you do not control might need temporary files after all. Example: you never know what kind of NSS module your local installation has enabled, and what that NSS module does with /tmp.

These options are hopefully interesting both for administrators to secure their local systems, and for upstream developers to ship their services secure by default. We strongly encourage upstream developers to consider using these options by default in their upstream service units. They are very easy to make use of and have major benefits for security.

Isolating Services from the Network

A very simple but powerful configuration option you may use in systemd service definitions is PrivateNetwork=:

...
[Service]
ExecStart=...
PrivateNetwork=yes
...

With this simple switch a service and all the processes it consists of are entirely disconnected from any kind of networking. Network interfaces became unavailable to the processes, the only one they'll see is the loopback device "lo", but it is isolated from the real host loopback. This is a very powerful protection from network attacks.

Caveat: Some services require the network to be operational. Of course, nobody would consider using PrivateNetwork=yes on a network-facing service such as Apache. However even for non-network-facing services network support might be necessary and not always obvious. Example: if the local system is configured for an LDAP-based user database doing glibc name lookups with calls such as getpwnam() might end up resulting in network access. That said, even in those cases it is more often than not OK to use PrivateNetwork=yes since user IDs of system service users are required to be resolvable even without any network around. That means as long as the only user IDs your service needs to resolve are below the magic 1000 boundary using PrivateNetwork=yes should be OK.

Internally, this feature makes use of network namespaces of the kernel. If enabled a new network namespace is opened and only the loopback device configured in it.

Service-Private /tmp

Another very simple but powerful configuration switch is PrivateTmp=:

...
[Service]
ExecStart=...
PrivateTmp=yes
...

If enabled this option will ensure that the /tmp directory the service will see is private and isolated from the host system's /tmp. /tmp traditionally has been a shared space for all local services and users. Over the years it has been a major source of security problems for a multitude of services. Symlink attacks and DoS vulnerabilities due to guessable /tmp temporary files are common. By isolating the service's /tmp from the rest of the host, such vulnerabilities become moot.

For Fedora 17 a feature has been accepted in order to enable this option across a large number of services.

Caveat: Some services actually misuse /tmp as a location for IPC sockets and other communication primitives, even though this is almost always a vulnerability (simply because if you use it for communication you need guessable names, and guessable names make your code vulnerable to DoS and symlink attacks) and /run is the much safer replacement for this, simply because it is not a location writable to unprivileged processes. For example, X11 places it's communication sockets below /tmp (which is actually secure -- though still not ideal -- in this exception since it does so in a safe subdirectory which is created at early boot.) Services which need to communicate via such communication primitives in /tmp are no candidates for PrivateTmp=. Thankfully these days only very few services misusing /tmp like this remain.

Internally, this feature makes use of file system namespaces of the kernel. If enabled a new file system namespace is opened inheritng most of the host hierarchy with the exception of /tmp.

Making Directories Appear Read-Only or Inaccessible to Services

With the ReadOnlyDirectories= and InaccessibleDirectories= options it is possible to make the specified directories inaccessible for writing resp. both reading and writing to the service:

...
[Service]
ExecStart=...
InaccessibleDirectories=/home
ReadOnlyDirectories=/var
...

With these two configuration lines the whole tree below /home becomes inaccessible to the service (i.e. the directory will appear empty and with 000 access mode), and the tree below /var becomes read-only.

Caveat: Note that ReadOnlyDirectories= currently is not recursively applied to submounts of the specified directories (i.e. mounts below /var in the example above stay writable). This is likely to get fixed soon.

Internally, this is also implemented based on file system namspaces.

Taking Away Capabilities From Services

Another very powerful security option in systemd is CapabilityBoundingSet= which allows to limit in a relatively fine grained fashion which kernel capabilities a service started retains:

...
[Service]
ExecStart=...
CapabilityBoundingSet=CAP_CHOWN CAP_KILL
...

In the example above only the CAP_CHOWN and CAP_KILL capabilities are retained by the service, and the service and any processes it might create have no chance to ever acquire any other capabilities again, not even via setuid binaries. The list of currently defined capabilities is available in capabilities(7). Unfortunately some of the defined capabilities are overly generic (such as CAP_SYS_ADMIN), however they are still a very useful tool, in particular for services that otherwise run with full root privileges.

To identify precisely which capabilities are necessary for a service to run cleanly is not always easy and requires a bit of testing. To simplify this process a bit, it is possible to blacklist certain capabilities that are definitely not needed instead of whitelisting all that might be needed. Example: the CAP_SYS_PTRACE is a particularly powerful and security relevant capability needed for the implementation of debuggers, since it allows introspecting and manipulating any local process on the system. A service like Apache obviously has no business in being a debugger for other processes, hence it is safe to remove the capability from it:

...
[Service]
ExecStart=...
CapabilityBoundingSet=~CAP_SYS_PTRACE
...

The ~ character the value assignment here is prefixed with inverts the meaning of the option: instead of listing all capabalities the service will retain you may list the ones it will not retain.

Caveat: Some services might react confused if certain capabilities are made unavailable to them. Thus when determining the right set of capabilities to keep around you need to do this carefully, and it might be a good idea to talk to the upstream maintainers since they should know best which operations a service might need to run successfully.

Caveat 2: Capabilities are not a magic wand. You probably want to combine them and use them in conjunction with other security options in order to make them truly useful.

To easily check which processes on your system retain which capabilities use the pscap tool from the libcap-ng-utils package.

Making use of systemd's CapabilityBoundingSet= option is often a simple, discoverable and cheap replacement for patching all system daemons individually to control the capability bounding set on their own.

Disallowing Forking, Limiting File Creation for Services

Resource Limits may be used to apply certain security limits on services being run. Primarily, resource limits are useful for resource control (as the name suggests...) not so much access control. However, two of them can be useful to disable certain OS features: RLIMIT_NPROC and RLIMIT_FSIZE may be used to disable forking and disable writing of any files with a size > 0:

...
[Service]
ExecStart=...
LimitNPROC=1
LimitFSIZE=0
...

Note that this will work only if the service in question drops privileges and runs under a (non-root) user ID of its own or drops the CAP_SYS_RESOURCE capability, for example via CapabilityBoundingSet= as discussed above. Without that a process could simply increase the resource limit again thus voiding any effect.

Caveat: LimitFSIZE= is pretty brutal. If the service attempts to write a file with a size > 0, it will immeidately be killed with the SIGXFSZ which unless caught terminates the process. Also, creating files with size 0 is still allowed, even if this option is used.

For more information on these and other resource limits, see setrlimit(2).

Controlling Device Node Access of Services

Devices nodes are an important interface to the kernel and its drivers. Since drivers tend to get much less testing and security checking than the core kernel they often are a major entry point for security hacks. systemd allows you to control access to devices individually for each service:

...
[Service]
ExecStart=...
DeviceAllow=/dev/null rw
...

This will limit access to /dev/null and only this device node, disallowing access to any other device nodes.

The feature is implemented on top of the devices cgroup controller.

Other Options

Besides the easy to use options above there are a number of other security relevant options available. However they usually require a bit of preparation in the service itself and hence are probably primarily useful for upstream developers. These options are RootDirectory= (to set up chroot() environments for a service) as well as User= and Group= to drop privileges to the specified user and group. These options are particularly useful to greatly simplify writing daemons, where all the complexities of securely dropping privileges can be left to systemd, and kept out of the daemons themselves.

If you are wondering why these options are not enabled by default: some of them simply break seamntics of traditional Unix, and to maintain compatibility we cannot enable them by default. e.g. since traditional Unix enforced that /tmp was a shared namespace, and processes could use it for IPC we cannot just go and turn that off globally, just because /tmp's role in IPC is now replaced by /run.

And that's it for now. If you are working on unit files for upstream or in your distribution, please consider using one or more of the options listed above. If you service is secure by default by taking advantage of these options this will help not only your users but also make the Internet a safer place.

starting on Wayland development

2012-01-18T13:53:43+00:00

Do you wanna contribute to a funky open-source project? Are you tired of your nerdy and boring community of developers? Are you the one that wants to get rid of X because it’s a giant, old and fat dinosaur in your system? :) Cool, I have a project to solve your problems!

—

While there’s still lot of churn in the protocol, and yet our goal is soon to wrap up all we’ve been doing to a steady and settle-on-the-stone one description, there’s a lot on the implementation side that needs love. And that mainly concern Weston compositor, which is becoming the de facto compositor on several systems targeting Wayland.

In past months I was accumulating a TODO list for Weston and libraries that I think wouldn’t require any exceptional knowledge on core graphics or Wayland internals. These are good items for people that want start hack or just do some contribution on Wayland:

1. log facility (easy)

Back in July, I had already warmed up a discussion how we could log on Wayland. So now we spit everything to stdout but we want to do it similarly as Xorg, i.e. redirecting to its own file. It turned out that we might want only on Weston compositor and implement our own way of logging for sake of simplicity. Ideally it has to be very simple, without verbosity levels probably, etc. This task should be quite easy to finish.

2. launcher for Weston (medium)

Weston is meant to run as a normal user. Now we have to set manually input devices, DRM and tty with root permissions, so Weston can happily be started. Ideally we should have a setuid helper script doing all this tricky, and in fact I started something here. For a real system though, we need to enhance a bit this program with the policykit, specially for dealing with hotplugs. Probably zero understanding on Wayland internals is needed but an overall knowledge of OS is required.

3. libxkbcommon X merge (medium)

Actually that’s not much Wayland work, but it most definitely would help its development. xkbcommon is the library that exposes the keyboard mapping logic to clients, converting keycodes in keysyms. Weston clients are using for evdev convertion. The library is an adaptation of a bunch of X11 modules to fit in a non-X11 world and as such, it still requires xproto, kbproto and libX11. One task would be to untie such dependencies with X and the other to proper merge libxkbcommon with Xorg. I’d classify as medium-size task due the involvement with the community and the hairy understanding of XKB in general, although Daniel, Kristian and other guys already made a nice progress there.

4. xwayland on X (hard)

xwayland is the implementation on Xorg to make it run as a Wayland client. That works well on Intel chipsets and might work as well with other drivers through the shm X driver. In principle, basic X11 applications work with some WM control lacking, input grab as well and things like Xrandr don’t. One would also forward port xwayland and driver to upstream, once the work is shaped up. I bet WM developers would have an ecstasy and delight themselves hacking around.

—

Hopefully you get excited with some of the items. I definitely can give a hand with any, so just poke me on #wayland at freenode.

If you’re in Europe, go to Monki Gras

2012-01-18T06:02:23+00:00

To my European readers: if you care about the impact of social technologies like Git (and GitHub) & how they’re transforming software development, or the impact of social technology on communities, and you enjoy good beer, you need to be at Monki Gras. I just posted over at my RedMonk blog about how the previous conference in the series, Monktoberfest, was the best conference of my life. And I’ve been to many.

Monki Gras is Feb. 1–2 in London. The timing’s perfect to stop by just before FOSDEM (and that’s exactly what I’m doing). Registration is dirt-cheap, speakers are universally top-notch, and you’ll also get some world-class beers in the package.

Tagged: community, development, gentoo

PulseAudio vs. AudioFlinger

2012-01-16T15:31:00+00:00

Arun put an awesome article up, detailing how PulseAudio compares to Android's AudioFlinger in terms of power consumption and suchlike. Suffice to say, PulseAudio rocks, but go and read the whole thing, it's worth it.

Apparently, AudioFlinger is a great choice if you want to shorten your battery life.

Phoronix survey about Intel Linux Graphics

2012-01-14T22:28:08+00:00

For the ones of you who haven’t read Phoronix lately – Michael is running a survey about Intel Linux Graphics drivers for the past few days.

So far it got 10 pages of amazingly interesting comments (which I was trying to answer as time and my fingertips permitted). 95 comments as of now to be fair (sorry, 96 already – my bad ). Make sure to check it out.

In the end, I’ll try to summarize all my replies in the thread here. But it will be a long read by the looks of it.

Bleeding edge i915 for stable kernels

2012-01-11T19:57:23+00:00

One of the most frequent ~~complains~~ questions related to the Intel Linux Graphics drivers I’ve received in the past few months was: “Why Intel devs work only on the most bleeding edge, and do not give enough attention the us stable users”?

Yes, this question affects all the components of the stack – kernel, mesa, libdrm, 2D driver, and so on, but the answer to this is quite simple – this is how software gets developed. New features go into new releases, and stable releases receive bugfixes and stability improvements at most. And this is not much of an issue to the userspace components anyway – with all the LD_LIBRARY_PATH flexibility it is possible to have multiple versions of the libraries installed without any issue. But as for the kernel, indeed, it is not that easy. So most of the times, those questions were directed at the kernel part of our driver – namely, the tiny i915.ko module which is responsible for making the graphical heart of Intel-based GPU beat.

Even for the kernel, this is not exactly true – Greg’s stable trees do include most of the critical fixes for our drivers since always. But it is true to some point – most of the newest development and patching happens within the usual Linux development window – and those patches and features are then merged to the release candidates of a future kernel during the merge window. And for the kernel, it is not that easy to have multiple kernels around at the same time without having to reboot between them here and again.

So therefore, I thought on giving a small gift to the users who are not still ready to jump into the latest and greatest kernels releases – but still want to enjoy the goodies brought with the latest versions of the graphics driver. So I prepared two kernel branches on my freedesktop.org kernel repository:

3.0-drm-intel-backports — latest 3.0.x kernel with all the i915 patches from latest kernel release. Right now, it is 3.0.16 + 195 backported patches. Diffstat reports 78 files changed, 4988 insertions(+), 2176 deletions(-).
3.1-drm-intel-backports — latest 3.1.x kernel with all the i915 patches from latest kernel release. Right now, it is 3.1.8 + 104 backported patches. Diffstat reports 74 files changed, 3147 insertions(+), 1589 deletions(-).

Besides driver backports, the patches which affect multiple drivers (like, i915, nouveau and r128) were also backported in full in those branches. And all the required build and API dependencies (like HDMI/DP ELD or drm/gem API changes) are also included in full.

I’ll try to keep those branches maintained on a semi-periodic basis and synchronized with the latest updates both from -stable branches, and from newest kernel releases.

Of course, let me put an obligatory support and stability statement:

Those branches are not supported officially, are experimental and can be unstable or even broken sometimes (in which case, if you’d be kind enough to let me know about that, I’d try to fix them ASAP). So use them at your own risk.

Other than that, have fun .

New hosts, cgit and more

2012-01-06T19:03:50+00:00

A while ago, we got some money from Intel, Google and HP for new machines for freedesktop.org. Due to various logistical problems, it has taken us far too long to actually start using those, but we're now in the process of doing so.

The first service to move to the new infrastructure is cgit. As part of this transition, we'll also be retiring some old services, amongst those are:

gitweb; we have cgit and gitweb does not see much traffic
cvs; nothing still lives in CVS. We will keep the directories around for the foreseeable future
svn; nothing uses this any longer. We will keep the directories around for the foreseeable future

We might also retire ftp.freedesktop.org, but that is not yet decided. The rationale being that HTTP is widespread and works well for handling downloads. For mirroring, rsync is better suited than ftp

Additionally, over the weekend we'll move bugzilla to the new hosts as well, something that should improve performance quite a bit, but it means bugzilla might be unavailable sometimes during the weekend.

Feedback is most welcome

– tfheen

Transmageddon runs with GStreamer 0.11

2012-01-06T18:48:25+00:00

After updating GStreamer and doing a couple of small fixes I managed to make Transmageddon work with the GTK3 and the 0.11 branch of GStreamer. Obligatory screenshot below. As you might guess from looking at the screenshot there are still some issues that needs solving, but
I am happy that I managed to get this far.

Transmageddon running GTK3 and GStreamer 0.11/1.0

Hopefully it is a sign that the upcoming GStreamer hackfest in Malaga will be a great successful everyone who is participating.

I hope the remainder of the porting effort will be relatively simple as I would love to get back to working on real features instead of just updating old functionality to use a new backend to do the same. Having had a need for Transmageddon for a couple of work related tasks recently a couple of items, like batch job programming has moved up my priorities list.

New year, new stuff

2012-01-06T16:21:46+00:00

So, holidays came to an end, and it is time to get started with the 2012 series of “tales from the Intel Linux Graphics land”.

But as the year has just started, it is still time to cover some amazing gifts which Santa ClausDaniel Vetter brought to you all right in time for Christmas. Yes, I am talking about the new intel-gpu-tools 1.1 release!

This new release comes after 2 years from the 1.0.2 release (which came out back in 2009), and accumulates an astonishing amount of changes:

Diffstat reports that 138 files were changed, having 29694 insertions and 3524 deletions.
Lots of new instructions and registers decoding logic was added in intel_error_decode and intel_reg_dumper tools
Many changes in intel_gpu_top, which can now run non-interactively for monitoring GPU usage statistics
Different display and mode setting tests
And, probably the most interesting feature out there, the introduction of a full-feature test suite for Intel and DRM testing.

Let me stop a bit on this latest feature, and explain it in a more detailed form. Starting with the 1.1 version of intel-gpu-toops, the package provides a complete testing suite for most of the known issues and problems that have been caught in our graphics drivers for Linux for the past years. So when you run make test (as root) in the intel-gpu-tools directory, this will start a long pile of test cases, testing most parts of the GPU.

Ideally, all tests should pass – this means that your current kernel and userspace libraries are not affected by any known kernel (or userspace) issue. Some tests can fail – usually this comes with an explanation of why it fails (either on the test output or in dmesg). And some tests can fail in a very evil way, hanging the GPU or the entire machine. A specific category of tests (HANG tests), which are not run by default, are designed to test specific corner cases which should make your machine hang for good; and some tests are written specifically to caught performance regressions and hick-ups.

Still wonder why is this interesting? Well, I can think on one obvious huge bonuses we get from this test suite – if you are experiencing an issue, you can run the make test suite, and verify if it is a new genuine bug, or something that was already caught by a test case. And obviously, this is an easy way to find and track regressions on the kernel and drm level.

So, to sum it all up – if you want to give a quick try and check if your kernel/userspace/distribution configuration is affected by any known issue in the driver, just run the testsuite. If it passes, you are good to go. If it does not passes, well, you already know that something is not right and needs investigation.

Besides intel-gpu-tools, Mesa work continues towards the remaining GL 3.0 features. During last few days of 2011, Eric Anholt sent some patches for adding hardware support to GL_EXT_transform_feedback bits present on the Ivy Bridge architecture. Besides those changes, hundreds of patches and discussions take place on the mesa-dev mailing list – the most user-notable change comes from the fact that XCB is now mandatory. I guess we have a clear winner in the XCB vs XLIB match in 2012..

On Kernel side, with the release of 3.2 version of the kernel, next merge windows has opened and already received some patches for 3.3 kernel series. Besides the usual fixes and new features which I already covered in previous posts, several alternatives were proposed by Chris Wilson and Eric Anholt to solve the mysterious missed IRQ issue which affects Ivy Bridge (and some Sandy Bridge) machines. The winner in this competition, however, was Daniel Vetter, who came with a simple patch that not only had solved the issues in a nice way, but also allowed us to remove several work-arounds which were added in the past. So far, all my Ivy Bridge machines are up and running for some days without any problems – which is totally awesome!

Moving to other projects, notable changes which were introduced in the past weeks were the move of intel_decode facility for decoding GPU batch buffers from intel-gpu-tools to libdrm library by Eric Anholt; patches from Paulo Zanoni to support new screen rotation feature of VPro; additional patches for improved Glamor performance from Zhigang Gong; support for building libdrm on Android from Chad Versace; Kristian Høgsberg created a new W-prefixed project for Wayland (namely, Weston, which is a different town in Massachusetts state. I actually got to get through it once when I was in the US back in 2005, working on a remote boot and software streaming project for mstech, which was further acquired by Ardence, which was further acquired in Citrix.. fun to see how the world goes in circles ).

So I think that’s it for now. For the ones who are still in the new year holidays, enjoy the rest of your holidays; and for the ones who are already back into to the real life – welcome to 2012!

Gustavo Noronha Silva Interview about WebKit

2012-01-04T14:09:54+00:00

Just posted another interview on the Collabora website, this time with Gustavo Noronha Silva talking about WebKit and some of the work he and Collabora are doing around that project. So be sure to check it out if you want to learn more about things like WebKit and Clutter integration and how WebKit impacts the GNOME platform.

Google Calendar notifications using pynotify

2012-01-03T18:55:00+00:00

Multitouch in X - Pointer emulation

2012-01-02T23:56:19+00:00

This post is part of a series on multi-touch support in the X.Org X server.

Multitouch in X - Getting events

In this post, I'll outline how pointer emulation on touch events works. This post assumes basic knowledge of the XI2 Xlib interfaces.

Why pointer emulation?

One of the base requirements of adding multitouch support to the X server was that traditional, non-multitouch applications can still be used. Multitouch should be a transparent addition, available where needed, not required where not supported.

So we do pointer emulation for multitouch events, and it's actually specified in the protocol how we do it. Mainly so it's reliable and predictable for clients.

What is pointer emulation in X

Pointer emulation simply means that for specific touch sequences, we generate pointer events. The conditions for emulation are that the the touch sequence is eligible for pointer emulation (details below) and that no other client has a touch selection on that window/grab.

The second condition is important: if your client selects for both touch and pointer events on a window, you will never see the emulated pointer events. If you are an XI 2.2 client and you select for pointer but not touch events, you will see pointer events. These events are marked with the XIPointerEmulated so that you know they come from an emulated source.

Emulation on direct-touch devices

For direct-touch devices, we emulate pointer events for a touch sequence provided the touch is the first touch on the device, i.e. no other touch sequences were active for this device when the touch started. The touch sequence is emulated until it ends, even if other touches start and end while that sequence is active.

Emulation on dependent-touch devices

Dependent touch devices do not emulate pointer events. Rather, we send the normal mouse movements from the device as regular pointer events.

Button events and button state

Pointer emulation triggers motion events and, more importantly, button events. The button number for touches is hardcoded to 1 (any more specific handling such as long-click for right buttons should be handled by touch-aware clients instead), so the detail field of an emulated button event is 1 (unless the button is logically mapped).

The button state field on emulated pointer events adjusts for pointer emulation as it would for regular button events. The button state is thus (usually) 0x0 for the emulated ButtonPress and 0x100 for the MotionNotify and ButtonRelease events.

Likewise, any request that returns the button state will have the appropriate state set, even if no emulated event actually got sent.

Grab handling works as for regular pointer events, though the interactions between touch grabs and emulated pointer grabs are somewhat complex. I'll get to that in a later post.

The confusing bit

There is one behaviour about the pointer emulation that may be confusing, even though the specs may seem logical and the behaviour is within the specs.

If you put one finger down, it will emulate pointer events. If you then put another finger down, the first finger will continue to emulate pointer events. If you now lift the first finger (keeping the second down) and put the first finger down again, that finger will not generate events. This is noticable mainly in bi-manual or multi-user interaction.

The reason this doesn't work is simple: to the X server, putting the first finger down just looks like another touchpoint appearing when there is already one present. The server does not know that this is the same finger again, it doesn't know that your intention was to emulate again with that finger. Most of the semantics for such interaction is in your head alone and hard to guess. Guessing it wrong can be quite bad, since that new touchpoint may have been part of a two-finger gesture with the second finger and whoops - instead of scrolling you just closed a window, pasted your password somewhere or killed a kitten. So we err on the side of caution, because, well, think of the kittens.

Multitouch in X - Touch grab handling

2012-01-02T23:56:05+00:00

This post is part of a series on multi-touch support in the X.Org X server.

In this post, I'll outline how grabs on touch events work. This post assumes basic knowledge of the XI2 Xlib interfaces.

Passive grabs

The libXi interface has one new passive grab call: XIGrabTouchBegin, which works pretty much like the existing passive grab APIs. As with event selection, you must set all three event masks XI_TouchBegin, XI_TouchUpdate and XI_TouchEnd or a BadValue error occurs. Once a passive grab activates in response to a touch, the client must choose to either accept or reject a touch. Details on that below.

Grabs activate on a TouchBegin event and due to the nature of multitouch, multiple touch grabs may be active at any time - some of them for different clients.

Active grabs

Active grabs do not have a new call, they are handled through the event masks of the existing XIGrabDevice(3) call. If a client has an active touch grab on the device, it is automatically the owner of the touch sequence (ownership is described below). If a client has an active pointer or keyboard grab on the device, it is the owner of the touch sequence for pointer emulated touch events only. Other touch events are unaffected by the grab and are processed normally.

Acceptance and rejection

Pointer grabs provide exclusive access to the device, but to some degree a client can opt to replay the event it received on the next client. We expect that touch sequences will often trigger gesture recognition, and a client may realise after a few events that it doesn't actually want that touch sequence. So we expanded the replay semantics. clients with a touch grab must choose to either accept or reject a touch.

Accepting a touch signals to the server that the touch sequence is meant for this client and no-one else. The server then exclusively delivers to that client until the terminating TouchEnd.

Rejecting a touch sequence signals that the touch sequence is not meant for this client. Once a client rejects a touch sequence, the server sends the TouchEnd event to that client (if the touch is still active) and replays the full touch sequence [1] on the next grab or window. We use the term owner of a touch sequence to talk about the current recipient.

The order of events for two clients Cg and Cw, with Cg having a grab and Cw having a regular touch event selection on a window, is thus:

TouchBegin to Cg    → 
TouchUpdate to Cg   → 
TouchUpdate to Cg   → 
                    ← Cg rejects touch
                    ← Cw becomes new owner
TouchEnd+ to Cg     →
TouchBegin* to Cw   → 
TouchUpdate* to Cw  → 
TouchUpdate* to Cw  → 
#### physical touch ends #### 
TouchEnd to Cw      →

Events with + mark an event created by the server, * mark events replayed by the server

For nested grabs, this sequence simply repeats for each client until either a grabbing client accepts the touch or the client with the event selection becomes the owner.

In the above case, the touch ended after Cg rejected the touch. If the touch ends before the current owner accepted or rejected it, the owner gets the TouchEnd event and the touch is left handing until the owner accepts or rejects it. If accepted, that's it. If rejected, the new owner gets the full sequence in one go, including the TouchEnd event. The sequence is thus:

TouchBegin to Cg    → 
TouchUpdate to Cg   → 
TouchUpdate to Cg   → 
#### physical touch ends #### 
TouchEnd to Cg      →
                    ← Cg rejects touch
                    ← Cw becomes new owner
TouchBegin* to Cw   → 
TouchUpdate* to Cw  → 
TouchUpdate* to Cw  → 
TouchEnd* to Cw     →

Touch ownership handling

One additional event type that XI 2.2 introduces is the XI_TouchOwnership event. Clients selecting for this event signal that they need to receive touch events before they're the owner of the touch sequence. This event type can be selected on both grabs and event selections.

First up: there are specific use-cases where you need this. If you don't fall into them, you're better off just skipping on ownership events, they make everything more complicated. And whether you need ownership events depends not only on you, but also the stack you're running under. On normal touch event selection, touch events are only delivered to the current owner of the touch. With multiple grabs, the delivery is sequential and delivery of touch events may be delayed.

Clients selecting for touch ownership events get the events as they occur, even if they are not the current owner. The XI_TouchOwnership event is delivered if and when they become the current owner. The last part is important: if you select for ownership events, you may receive touch events but you may not become the owner of that sequence. So while you can start reacting to that sequence, anything your app does must be undo-able in case the e.g. window manager claims the touch sequence.

If we look at the same sequence as above with two clients selecting for ownership, the sequence looks like this:

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
                     ← Cg rejects touch
                     ← Cw becomes new owner
TouchEnd+ to Cg      →
TouchOwnership to Cw →
#### physical touch ends #### 
TouchEnd to Cw      →

Note: TouchOwnership events do not correspond to any physical event, they are always generated by the server

If a touch ends before the owner accepts, the current owner gets the TouchEnd, all others get a TouchUpdate event instead. That TouchUpdate has a flag XITouchPendingEnd set, signalling that no more actual events will arrive from this touch but the touch is still waiting for owner acceptance.

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
#### physical touch ends #### 
TouchEnd to Cg       →
TouchUpdate to Cw    →  (XITouchPendingEnd flag set)
                     ← Cg rejects touch
                     ← Cw becomes new owner
TouchOwnership to Cw →
TouchEnd to Cw       →

In both cases, we dealt with a rejecting owner. For an accepting owner, the sequences look like this:

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
                     ← Cg accepts touch
TouchEnd+ to Cw      →
TouchUpdate to Cg    → 
#### physical touch ends #### 
TouchEnd to Cg      →

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
#### physical touch ends #### 
TouchEnd to Cg       →
TouchUpdate to Cw    →  (XITouchPendingEnd flag set)
                     ← Cg accepts touch
TouchEnd* to Cw      →

In the case of multiple grabs, the same strategy applies in order of grab activation. Ownership events may be selected by some clients but not others. In that case, each client is treated as requested, so the event sequence the server deals with may actually look like this:

TouchBegin to C1     → 
TouchBegin to C3     → 
TouchOwnership to C1 →
TouchUpdate to C1    → 
TouchUpdate to C3    → 
TouchUpdate to C1    → 
TouchUpdate to C3    → 
                     ← C1 rejects touch
                     ← C2 becomes new owner
TouchEnd+ to C1      →
TouchBegin* to C2    → 
TouchUpdate* to C2   → 
TouchUpdate* to C2   → 
                     ← C2 rejects touch
                     ← C3 becomes new owner
TouchEnd+ to C2      →
TouchOwnership to C3 →
#### physical touch ends #### 
TouchEnd to C3       →

[1] obviously we need to store these events so "full sequence" really means all events until the buffer was full

Happy 2012 to you all

2012-01-01T00:26:17+00:00

Yeah, 2011 is coming to the end, and had already switched place with 2012 in most countries already.

Somewhere on the road to curitiba, starting 2011....

For me, 2011 was extremely busy, exhausting but also very fruitful. I started it in a completely different situation from 2010, when I was a mere developer in Mandriva company. And during the course of 2010 and, later, 2011, I got the chance to work as engineering team leader for Conectiva, then development manager and, finally, technical diretor for Mandriva. And, after this experience, I got to move into Intel, working with even more challenging and amazing projects.

So yes, it was a hugely crazy and great experience. Thank you 2011 – despite all the problems and obstacles, you was great after all if we sum up everything.

See you all in 2012!

Happy 2012 to you all

2012-01-01T00:25:23+00:00

Yeah, 2011 is coming to the end, and had already switched place with 2012 in most countries already.

And finishing 2011 in this landscape...

So yes, it was a hugely crazy and great experience. Thank you 2011 – despite all the problems and obstacles, you was great after all if we sum up everything.

See you all in 2012!

Happy 2012 to you all

2012-01-01T00:21:42+00:00

Yeah, 2011 is coming to the end, and had already switched place with 2012 in most countries already.

For me, 2011 was extremely busy, exhausting but also very fruitful. I started it in a completely different situation from 2010, when I was a developer and impossible-mission-solver in Mandriva. And during the course of 2010 and, later, 2011, I got the chance to work as engineering team leader for Conectiva, then development manager and, finally, technical diretor for Mandriva. And, after this experience, I got to move into Intel, working with even more challenging and amazing projects.

And finishing 2011 in this landscape...

So yes, it was a hugely crazy and great experience. Thank you 2011 – despite all the problems and obstacles, you was great after all if we sum up everything.

See you all in 2012!

Using GTK+ stock icons with pynotify

2011-12-27T11:55:00+00:00

Howto: DisplayLink USB Single Monitor on Linux

2011-12-23T23:23:27+00:00

Unfortunately, Linux doesn’t support multiple graphics adapters the way Windows does, which means you can’t just plug in USB graphics adapters and expect them to extend your desktop (the good news is there is progress on this support).

What is possible, however, is running a single DisplayLink adapter, or several with a Xinerama or multiseat configuration — just as long as you don’t expect to use your main GPU at the same time.

The single-display case is relatively easy to set up, and we’ll cover that here.

First, make sure you’re running kernel version 2.6.35 or later (Ubuntu 10.10 or later). For older kernel versions, you’ll need to update udlfb and run a modified fbdev X server (not covered in this post). On these kernel versions, when you plug in your DisplayLink-based USB graphics device, you should get a green screen. This means that at the driver built into the Linux kernel is happy, healthy, and talking to the device.

Second, if you are running Unity Desktop in Ubuntu 11.04 or later, you’ll need to switch back to Classic Mode so you’re running straight X. Here’s how on Ubuntu:

Click on the power button in the upper right corner (mine looks like a light switch) and choose the last option, System Settings. Search for Login Screen, Double-click to display, Choose Unlock and enter your password, Select Ubuntu Classic as default session.

Third, if you’re running kernel versions between 2.6.35 to 3.1, enable the fb_defio option of udlfb. To do this, create or edit a file like
/etc/modprobe.d/50-displaylink.conf

and add the single line

options udlfb fb_defio=1

And reboot (or run “sudo depmod -a” and unplug/replug your adapter). This will turn on defio (page fault change detection) support. This option is already enabled by default in kernels 3.2+.

Lastly, create an X config file called 60-plugable.conf (or similar) with the following contents and place it in /usr/share/X11/xorg.conf.d (on recent distros; on older distros, make this your xorg.conf):

 
Section "Device" 
  Identifier "uga" 
  driver "fbdev" 
  Option "fbdev" "/dev/fb0" 
EndSection 
 
Section "Monitor" 
  Identifier "monitor" 
EndSection 
 
Section "Screen" 
  Identifier "screen" 
  Device "uga" 
  Monitor "monitor" 
EndSection 
 
Section "ServerLayout" 
  Identifier "default" 
  Screen 0 "screen" 0 0 
EndSection

Note: if your main GPU creates a /dev/fb0 even when the USB display is not attached, then your USB display is probably getting assigned to /dev/fb1. In that case, change /dev/fb0 in the “Device” section above to /dev/fb1

Now, on reboot, you should (hopefully!) see your login come up on your DisplayLink USB attached display!

This kind of simple setup is useful for:

Testing or playing with your USB graphics adatper on Linux.
Embedded systems with USB but no GPU.
As a backup method when the main GPU or its driver isn’t available or working.
Systems where a USB graphics adapter enables higher modes (up to 2048×1152) than the main GPU screen.

Please comment if you have any trouble with this single display case. See our past posts for additional information about the DisplayLink Linux kernel driver and some more involved setups.

The instructed here work on all Plugable USB 2.0 graphics adapters and Plugable USB 2.0 docking stations and thin clients (and should also generally work on all DisplayLink based products).

Santa doesn't forget about old cards (R6xx/R7xx HDMI audio)

2011-12-23T19:32:48+00:00

Since ever few radeon HDMI audio testers were reporting problems with their TVs not displaying anything at all. Enabling audio support resulted in invalid signal or similar error.

Thanks to great help from Nirbheek Chauhan, I've finally found the source of this issue. It appears we calculate checksum of HDMI infoframe incorrectly. Some TVs don't care about this, but others really do, refusing to display or play anything.

I've just posted:
[PATCH] drm/radeon/kms: workaround invalid AVI infoframe checksum
with suggestion of backporting this patch to stable releases. Just a one another Christmas gift from me.

Merry Christmas everyone who celebrate it :)

calypso

2011-12-23T06:49:15+00:00

Calypso — CalDAV/CardDAV/WebDAV for Android and Evolution

Ever since I bought my first Palm Pilot in 1997, I’ve relied upon a pocket-able device to carry a copy of my calendar and contacts, and for that same database to be present on my laptop. I went through a long list of Palm-compatible devices, including both the Palm Treo and Palm Centro telephones. I even wrote a number of my own Palm applications. Years ago, it was pretty obvious that I’d have to find a new phone, but I was stuck looking for something that could provide the same ‘hot-sync’ functionality.

SyncML on the Series 40

I bought a Series 40 Nokia phone in Shanghai that promised ‘SyncML’ support. Given that I had seen numerous SyncML implementations for Linux, it seemed like I should be able to get something to work.

SyncML on the Series 40 is a disaster — the phone couldn’t actually store all of my contacts, and it couldn’t hold half of the data fields I used. So, that phone landed in the box and back to the Centro I went.

SyncML on the N900

Nokia kindly sent me an N900 at some point, so I gave SyncML another try. Given that the N900 runs evolution-data-server, and that I’ve had evolution-data-server running on my laptop, it seemed like I should be in business. Well, almost. It took several days of hacking to fix bugs in the evolution SyncML back end, then another several days fussing with opensync.

I ended up abandoning direct synchronization as unworkable — opensync would sit in an infinite loop, or worse, trash my database completely. I finally found ‘syncml-ds-tool’, which is a debugging tool that comes with opensync. This tool simply synchronizes a set of disk files, one per contact or calendar entry, with the phone. That worked for well over a year. And then, a few months ago, Bluetooth on the laptop stopped connecting with the phone’s SyncML server. I’d get ‘ECONNREFUSED’ every time I tried to use it. So much for the N900. DUN still worked, mostly, although it too would get ECONNREFUSED at times, but retrying seemed to make it work.

However, While the N900 SyncML solution worked, I discovered another thing I wanted—contacts and calendar entries stored in individual files and revision controlled with git. This makes it reasonable to delete stale calendar entries and know that they’re never really gone, just left behind in an older version of the calendar. And, if you mess up, you can recover by poking at the database with git directly.

I switched back to the venerable Palm Centro; it turns out that calendar and contacts are more important to me than being able to surf the web on my phone. Alas, my Centro went ‘swimming’ in August and has passed on to the great electronics recycling house in the sky. I pulled the SIM out and switched back to the N900.

I got my contacts imported on the N900 by copying files over the net work; not a long-term strategy, but at least I had phone numbers again. There was no hope for my calendar. I started looking for a solution in earnest.

How about Android?

At this point in my story, I’m sure you’re asking why I didn’t just use one of the numerous Android phones that came through my hands. The answer is simple — my calendar and contacts are probably some of my most personal data, and I’m not willing to store them outside of my direct control, for reasons similar to those which are driving the development of the FreedomBox.

When Android first came out, it could only talk to Google services, which didn’t meet my hard requirement for personal data storage.

One of my co-workers had his Google account suspended for violating the terms-of-service; he asked what he had done, but they wouldn’t say. He asked if he could get his data back, and they said “no”. They invited him to create a new account, but it would not ever get any of the old data. A few days later, he got a nice apologetic email letting him know that they’d made a mistake and that he hadn’t, in fact, violated any of the terms-of-service, and that his old account was restored with all of his data intact. Wasn’t that nice of Google?

WebOS?

I got a WebOS phone over the summer and discovered that while it had multiple contact/calendar back ends, none of them used standard protocols and so you only had the choice between multiple corporate data centers, which isn’t actually a choice at all.

Furthermore, the WebOS phone refused route PAN packets over the phone network, even though I have a data plan which allows this. It’s not that it couldn’t support it, it’s that it refused.

A couple of weeks after the WebOS phone arrived, HP canceled all of their WebOS hardware products, which made me less interested in trying to solve this problem.

Android recovers

About the same time the WebOS phone arrived, I discovered that Google had published enough information about the calendar/contacts internals for Marten Gajda to write CalDAV-Sync and CardDAV-Sync. And then, Andrew McMillan wrote aCal, which is a complete replacement for the built-in calendar and contacts applications and supports CalDAV and CardDAV.

With two different standards-compliant solutions available, it seemed like it might be time to try Android again.

I’d love for CardDAV-Sync and CalDAV-Sync to become free software like aCal is. Andrew makes money from aCal by offering it for sale via the Android Market, while still publishing the sources for those who want to build their own copy.

CalDAV/CardDAV on Linux

I think the most widely known CalDAV server for Linux is probably DAViCal, a huge pile of PHP and SQL sitting on top of Apache. I’m sure it’s suitable for running on a server and being accessed over the internet, but I’m not interested in that, nor am I interested in having my laptop run Apache and PostgreSQL.

I found a tiny little CalDAV server, Radicale, which seemed like a lot better fit. It’s written in Python and uses the usual Python HTTP server infrastructure, which provides SSL and authentication support along with some fairly convenient APIs for parsing and generating HTML.

Before long, I discovered that Radicale was actually too simple for my needs. It stores the whole calendar in a single file, re-parsing it whenever a request is made, so a calendar with just hundreds of entries caused the server to slow down enough that evolution would time-out when talking to it.

Also, Radicale doesn’t actually parse the calendar entries completely, it has some ad-hoc code that finds various pieces of data, but without dealing with the whole syntax.

I started hacking at Radicale to see how far I could get. I changed the storage code to store one event per file, then added hooks to use git for change management. Then, I found a full vcalendar/vcard parsing library in python, vobject, which I used to replace the ad-hoc parsing code. Finally, I added support for VCARD entries as well, allowing the system to store both calendar and contact information.

Introducing Calypso, a CardDAV/CalDAV server

With this much divergence from the original project, I’ve figured I’d best rename things to avoid confusion, so I decided to call it ‘calypso’, after a brief trip through the dictionary looking for names starting with ‘ca’.

Calypso works with evolution, iceowl and the Android CalDAV/CardDAV plugins. It does not yet work with aCal; for some reason aCal cannot find any calendars on the server.

Calypso also supports importing calendar changes from the command line, allowing you to integrate support into a text-based email application like notmuch or mutt.

Calypso is available via git from git://keithp.com/git/calypso and is distributed under the GPL (v3 or later). I still consider it a work derived from Radicale, and so the code retains all of the Radicale copyrights along with my own.

Using Calypso

Initial setup

Calypso runs as a regular user, all data are stored in ~/.config/calypso. To initialize calypso:

$ mkdir ~/.config/calypso ~/.config/calypso/calendars
$ cat > ~/.config/calypso/config << EOF
[server]
ssl=true
certificate=/home/keithp/.config/calypso/ssl/server.crt
key=/home/keithp/.config/calypso/ssl/server.key

[acl]
;type=htpasswd
type=fake
filename=/home/keithp/.config/calypso/passwd

Running calypso

Then run calypso:

$ python ./calypso.py

No, I haven’t figured out how to install it…

Creating new calendars

To add a new database:

$ mkdir -p ~/.config/calypso/calendars/private/my_calendar
$ cd ~/.config/calypso/calendars/private/my_calendar
$ git init
$ git commit --allow-empty -m'initialize new calendar'

The new calendar should now be visible as https://localhost:5233/private/my_calendar

You can add files to the directory at any time; calypso will check the directory mtime at each operation and update its internal state from that on disk automatically when the directory changes.

Importing files

Given a set of files with VCALENDAR or VCARD entries, you can import them with:

$ calypso --import private/my_calendar <filenames...>

This will update any changed entries and add any new ones.

ToDo list for calypso

Document the config file contents.
Make it installable
Figure out what aCal wants
Support calendar creation

More Android info

Synker. This provides a desktop widget which starts the sync process running on a list of accounts. This makes it easy to manually synchronize with the laptop when you are connected
Contact Editor. This lets you fully edit contacts synchronized over CardDAV. The built-in contact editor doesn’t let you change anything other than the name for some reason.

I’m running cyanogenmod on my Nexus S as that provides PAN support. With PAN, I can create a network link between laptop and phone which doesn’t depend on any local WiFi infrastructure and which gives both phone and laptop static IP addresses, allowing me to configure the sync URLs statically on the phone. I’d use mDNS, but Android doesn’t bother to support that.

update on hotplug server

2011-12-22T17:29:56+00:00

No new videos yet, need to fix some more rendering bugs so it looks nicer :)

So I've been working towards 3 setups:

a) intel rendering + nouveau offload
b) nouveau rendering + DVI output + intel LVDS output
c) hotplug USB with either intel or nvidia rendering.

Categorisation of devices roles:
I've identified 4 devices roles so far:
preferred master: the device is happy to be master
reluctant master: the device can be a master but would rather not be
offload slave: device can be used as an additional DRI2 renderer for a master
output slave: device can be used an additional output for a master

For the 3 setups above:
a) intel would be preferred master, nvidia would be offload slave
b) nvidia would be preferred master, intel would be output slave
c) usb devices would be output slaves, however if no master exists, usb device would be reluctant master.

I've rebased the prime work[1] on top of the dma-buf upstream work, and worked through most of the lifetime problems. Some locking issues still exist, and I'll have to get back to them. But the code works and doesn't oops randomly which is good.

prime is the kernel magic needed for this work, as it allows sharing of a buffer between two drm drivers, so for (a) it shares the dri2 front pixmap between devices, for (b/c) it shares a pixmap that the rendering gpu copies dirty updates to and the output slaves use as their scanout pixmap.

So I've done nearly all the work to share between intel and nouveau and I've done the kernel driver work for udl, but I haven't done the last piece in userspace for (c), which is to use the shared pixmap as usb scanout via the modesetting driver.

Today I hacked in a switch on the first randr command, so I can start the X server with intel as master and nouveau in offload mode. I can run gears on intel or nouveau, then after the randr command and another randr command to set a mode, the X server migrates everything to the nouveau driver, puts it in master mode, and places the intel driver into output slave mode. It seems to render my xterm + metacity content fine.

So the current short-term TODO is:
fix some issues with my nouveau/exa port rendering
fix some issues with xcompmgr
add usb output slave support.

Medium-term TODO:
worked out how to control this stuff, via randr protocol. How much information do we need to expose to clients about GPUs, and how do we control them. Open issues with atomicity of updates to avoid major uglys. Switching from intel master to nvidia master + intel outputs, means we have to reconfigure the Intel output to point at the new pixmap, but the more steps we put in there for clients to do, the more ugly and flashing we'll see on screen, however we probably want a lot of this to be client driven (i.e. gnome-settings-daemon).

Longer term TODO:
Get GLX_ARB_robustness done, now that Ian has done the context creation stuff, this should be a lot more trivial. (so trivial someone else could do it :)

[1] http://cgit.freedesktop.org/~airlied/linux/log/?h=drm-prime-dmabuf

Multitouch in X - Getting events

2011-12-22T00:30:21+00:00

This post is part of a series on multi-touch support in the X.Org X server. I recommend re-reading Thoughts on Linux multitouch from last year for some higher-level comments.
In this post, I'll outline how to identify touch devices and register for touch events.

This post assumes basic knowledge of the XI2 Xlib interfaces. Code examples should not be scrutinised for language-correctness.

New event types

XI 2.2 defines four new event types: XI_TouchBegin, XI_TouchUpdate, XI_TouchEnd are the standard events that most applications will be using. The fourth event, XI_TouchOwnership is mainly for handling specific situations where reaction speed is at a premium and gesture processing when grabs are active. I won't be covering those in this post.

Identifying touch devices

To use multitouch functionality from a client application, the client must announce support for the X Input Extension version 2.2 through the XIQueryVersion(3) request.

int major = 2, minor = 2;
XIQueryVersion(dpy, &major, &minor);
if (major * 1000 + minor < 2002)
    printf("Server does not support XI 2.2\n");

Once announced, an XIQueryDevice(3) call may return a new class type, the XITouchClass. If this class is present on a device, the device supports multitouch.The class struct itself is defined like this:

typedef struct
{
    int         type;
    int         sourceid;
    int         mode;
    int         num_touches;
} XITouchClassInfo;

The num_touches field specifies the number of simultaneous touches supported by the device. If the number is 0, we simply don't know (likely) or the device supports an unlimited number of touches (less likely). Regardless of the value expect that some devices lie, so it's best to treat this value as a guide only.

The mode field specifies the type of touch devices. We currently define two types and the server behaviour differs depending on the type:

XIDirectTouch for direct-input touch devices (e.g. your average touchscreen or tablet). For this type of device, the touch events will be delivered to the windows at the of the touch point. Again, similar to what you would expect from a tablet interface - you press top left and the application top-left responds.
XIDependentTouch for a indirect input devices with multi-touch functionality. Touchpads are the prime example here. Touch events on such devices will be sent to the window underneath the cursor and clients are expected to interpret the touchpoints as (semantically) relative to the cursor position. For example, if your cursor is inside a Firefox window and you touch with two fingers on the top-left corner of the touchpad, Firefox will get those events. It can then decide on how to interpret those touchpoints.

A device that has a TouchClass may send touch events, but these events use the same axes as pointer events. Having said that, a touch device may still send pointer events as well - if the physical device generates both.
Your code to identify touch devices could roughly look like this:

XIDeviceInfo *info;
int nevices;

info = XIQueryDevice(display, XIAllDevices, &ndevices);

for (i = 0; i < ndevices; i++)
{
    XIDeviceInfo *dev = &info[i];
    printf("Device name %d\n", dev->name);
    for (j = 0; j < dev->num_classes; j++)
    {
        XIAnyClassInfo *class = dev->classes[j];
        XITouchClassInfo *t = (XITouchClassInfo*)class;

        if (class->type != XITouchClass)
            continue;

        printf("%s touch device, supporting %d touches.\n",
               (t->mode == XIDirectTouch) ?  "direct" : "dependent",
               t->num_touches);
    }
}

Selecting for touch events

Selecting for touch events on a window is mostly identical to pointer events. A client creates an event mask and submits it with XISelectEvents(3). One exception applies: a client must always select for all three touch events [1], XI_TouchBegin, XI_TouchUpdate, XI_TouchEnd. Selecting for one or two only will result in a BadValue error.

As for button events, only one client may select for touch events on any given window and the event delivery attempts traverse from the bottom-most window in the window tree up to the root window. Where a matching event selection is found, the event is delivered and the traversal stops.

Handling touch events

The three event types [1] are XIDeviceEvents like pointer and keyboard events. So from a client's point of view, in essence all we added was new event types.

The detail field of touch events specifies the touch ID, a unique ID for this particular touch for the lifetime of the touch sequence. Each touch sequence consists of a TouchBegin event, zero or more TouchUpdate events and one TouchEnd event. Since multiple touch sequences may be ongoing at any time, keeping track of the ID is important. The server guarantees that the touch ID is unique per device and that it will not be re-used [2]. Note that while touch IDs increase, they increase by an implementation-defined amount. Don't rely on the next touch ID to be the current ID + 1.

The button state in a touch event is the state of the physical buttons only. A TouchUpdate or TouchEnd event will thus usually have a zero button state. [3]

That's pretty much it, otherwise the handling of touch events is identical to pointer or keyboard events. Touch event handling should be straightforward and the significant deviations from the current protocol are in the grab handling, something I'll handle in a future post.

[1] I know, it's four. Good that you're paying attention.
[2] Technically ID collision may occur. For that to happen, you'd need to hold at least one touch down while triggering enough touches to exhaust a 32 bit ID range. And hope that after the wraparound you will get the same ID. There are better ways to spend your weekend.
[3] pointer emulation changes this, but I'll get to that some other time.

Qt on wayland: howto

2011-12-21T09:54:41+00:00

To run Qt applications on Wayland is fairly simple nowadays. Thank to Qt developers, they are following up quite well our last changes on Wayland protocol and updating accordingly on Qt5 code base — by the way, the fresh and just released Qt 4.8 does not ship the latest protocol additions, so that’s not the one I’m referring.

So, today I’ve set up the last bits of Qt environment on my Intel Pine-Trail pretty easy (yeah, I compile on my tablet :-O). You don’t need to clone the whole data base for starting hacking on it. Let’s see.

first, you have to clone qtbase:
# mkdir qt; cd qt # git clone git://gitorious.org/qt/qtbase.git

Before start the compilation steps, I like to set up my testbed in a different directory than what the ordinary system uses, so I include this on my .bashrc:
export QTVER=qt5 export QTDIR=/opt/qt/$QTVER export PATH=$QTDIR/bin/:$PATH export LD_LIBRARY_PATH=$QTDIR/lib/:$LD_LIBRARY_PATH export PKG_CONFIG_PATH=$QTDIR/lib/pkgconfig/:$PKG_CONFIG_PATH export QT_PLUGIN_PATH=$QTDIR/lib/plugins/

and don’t forget to re-run your bashrc! Now, when developing a Wayland application it’s quite useful to check if your app is behaving nicely, so for doing so I like to compare it with Qt’s XCB backend. Then you’ll need the following — I’m under Ubuntu 11.10:
# apt-get install libxcb1 libxcb1-dev libx11-xcb1 libx11-xcb-dev libxcb-keysyms1 libxcb-keysyms1-dev libxcb-image0 libxcb-image0-dev libxcb-shm0 libxcb-shm0-dev libxcb-icccm4 libxcb-icccm4-dev libxcb-sync0 libxcb-sync0-dev libxcb-xfixes0-dev

you should be ready now to go for the configuration:
# ./configure -confirm-license -opensource -no-qt3support -no-multimedia -no-webkit -no-phonon -no-v8 -debug -qpa -xcb -wayland -egl -opengl es2 -nomake examples -prefix ${QTDIR}

and now, the compilation — which took approximately 1hr and half in my system:
# make

install it:
# make install

At this point you got all Qt libraries, and needed tools to compile qtwayland platform. So go back one directory and clone that:
# cd ../ # git clone git://gitorious.org/qt/qtwayland.git

You need Wayland protocol libraries now in your system, and that’s the very late ones, which means the stock packages from your distro won’t work. Get and
build them using the following:

http://wayland.freedesktop.org/building.html

You might want to develop Wayland on the top of X, so before compile you’ll need this:
# apt-get install libxcomposite-dev

Jørgen Lind, updated me with: “the default buffersharing to use is wayland_egl. To be able to use the xcomposite stuff you need to export the environment variable QT_WAYLAND_GL_CONFIG to be either xcomposite_egl or xcomposite_glx”. So in order do to Wayland on X you will need to tweak it as well.

and then:
# cd qtwayland/ # qmake # make && make install

To run now a Qt app, I first set the XDG directory that compositor and clients use to talk under Wayland. You better to watch out, cause some distros already have this set somewhere. Anyway, I put this on my .bashrc:
# export XDG_RUNTIME_DIR=$HOME/.xdg # mkdir $HOME/.xdg

and re-ran .bashrc. Finally the Qt app running and voilà:
# cd ../qtbase/examples/opengl/hellowindow # qmake # make # wayland-compositor & # ./hellowindow -platform wayland
—

Kudos to Thiago Macieira for reviewing the content!

Holidays news from Intel Linux Graphics land

2011-12-20T13:42:16+00:00

Yeah, I admit that my semi-periodic updates about Intel Linux Graphics got more “seldom-periodic” than “truly-periodic” for the past weeks. But have no fear – they are back! And I am still on my self-appointed bi-weekly schedule estimate. This is what’s good about semi-periodic schedule – one never can run too much out of it .

So starting with the coolest news – the Mesa team is getting close to the GL 3.0 milestone! Yeah, with latest GL_ext_transform_feedback patches from Paul Berry, the last major piece of GL 3.0 spec is getting into place. There are still some extensions missing and lots of smaller tasks to be done, but it is possible to say that we are almost there. I think that this is really exciting for both us, and for all the Linux and open-source users in the world – so yeah – we’ve been good boys and girls during the year and Santa Claus gift has materialized itself in form of almost-full GL 3.0 support in Mesa.

Who knows, maybe prior to the Chinese new year we’ll receive the 2nd part of this gift (in other words, mesa 8.0 release ).

On kernel side, the 3.2-rc6 release brought lots of awesome changes to our drivers. Yes, I am talking about everyone’s favorite rc6 and semaphores features. They are on by default on Ivy Bridge architecture, and are also enabled on Sandy Bridge if VTd is disabled. So most of you should enjoy greatly improved battery life, considerable faster performance and also enhanced stability within the i915 driver when Linux 3.2 will be released.

Besides those patches, work has started on collecting patches for the 3.3 merge window. Daniel Vetter sent his pending patches in a form of tiny 43-patches series. Those patches bring PPGTT support, improve debugfs handling, enhance pread/pwrite performance, fix swizzling for SNB/IVB, improve forcewake operations and enhance debugging support for cases when GPU rings get stuck.

Ben has also sent his patches for scheduling/throttling, but they haven’t received much interest except from myself and phoronix . Those patches add support for more fine-grained GPU scheduling and rings load distribution between individual process. I am really interested in this work, and I hope that they will be accepted into the main kernel in the foreseeable future.

Also on kernel, Rodrigo Vivi and Paulo Zanoni sent out some patches which finally fix some corner cases for TV-out and SDVO outputs. This certainly should make many users happy out there just in time for Christmas.

And finally, for the kernel size, Chris Wilson came with a patch which works around the missed IRQs issues on Ivy Bridge platform. With this patch, and with semaphores being enabled on Ivy Bridge by default, I am very happy to say that we don’t have any blocking bugs for Ivy Bridge in our bugzilla. I think that it comes as a nice Christmas gift for all the users out there (the ones who already have an Ivy Bridge machine, and the ones who will get it by its launch – which is still 4 months away). Of course, I won’t talk much about it prior to its official launch, but trust me – Ivy Bridge rocks!. I can’t wait to have an Ultrabook based on this platform for myself…

Besides mesa and kernel, it is worth mentioning that on the 2D side, Zhigang added full Glamor support into the driver. The Glamor acceleration is still considered very experimental and non-stable, but now it is available for the world to take a peek on it and witness how it works with their own eyes.

So I think that this is pretty much it. We have hundreds of patches floating around for all the projects, thousands of emails and millions of users in the world – and we are working hard to make all of them happy with the results of our work. 2011 was extremely productive and rewarding for us – and I hope that the year of 11111011100 (a.k.a., 0x7DC or 2012_base10 for the ones still reading in decimal numbers out there ) will be even more interesting*!

See you!

(*) Assuming the world won’t end in a core dump caused by the Mayan millennium counter overflow bug .

Multitouch patches posted

2011-12-19T22:18:41+00:00

After pulling way too many 12+ hour days, I've finally polished the patchset for native multitouch support in the X.Org server into a reasonable state. The full set of patches is now on the list. And I'm still expecting this to get merged for 1.12 (and thus in time for Fedora 17).

The code is available from the multitouch branches of the following repositories:

  git://people.freedesktop.org/~whot/xserver
  git://people.freedesktop.org/~whot/inputproto
  git://people.freedesktop.org/~whot/xf86-input-evdev
  git://people.freedesktop.org/~whot/libXi

Here's a screencast running Fedora 16 with the modified X server and a little multitouch event debugging application.

Below is a short summary of what multitouch in X actually means, but one thing is important: being the windowing system, X provides multitouch support. That does not mean that every X application now supports multitouch, it merely means that they can now use multitouch if they want to. That also includes gestures, they need application support.

A car analogy: X provides a new road, the applications still have to opt to drive on it.

Multitouch events

XI 2.2 adds three main event types: XI_TouchBegin, XI_TouchUpdate and XI_TouchEnd. These three make up a touch sequence. X clients must subscribe to all three events at once and will then receive the events as they come in from the device (more or less, grabs can interfere here). Each touch event has a unique touch ID so clients can track the touches over time.

We support two device types: XIDirectDevice includes tablets and touchscreens where the events are delivered to the position the touch occurs at. XIDependentDevice includes multitouch-capable touchpads. Such devices still control a normal pointer by default, but for multi-finger gestures are possible. For such devices, the touchpoints are delivered to the window underneath the pointer.

That is pretty much the gist of it. I'll post more information over time as the release gets closer, so stay tuned.

Pointer emulation

Multitouch can be a compelling interaction method but as said above, X only provides support for multitouch. It will take a while for applications to pick it up (Carlos Garnacho is working on GTK3) and some never will. Since we still need to interact with those applications, we provide backwards-compatible pointer emulation. Again, the details are in the protocol but the gist of it is that for the first touchpoint we emulate pointer events.

That's the really nasty bit, because you now have to sync up the grab event semantics of the core, XI 1.x and XI2 protocols and wrap it all around the new grab semantics. So that if you have a multitouch app running under a window manager without multitouch support everything still works as expected.
That framework is now in place too though I expect it to still have bugs, especially in the hairier corner cases.

But other than that, it should work just as intended. I can interact with my GNOME3 desktop quite well and I get multitouch events to my test applications.

[edit Dec 20: typo fix]

Flame on: Graphing hot paths through X server code

2011-12-19T21:34:39+00:00

Last week, Brendan Gregg, who wrote the book on DTrace, published a new tool for visualizing hot sections of code by sampling the process stack every few milliseconds and then graphing which stacks were seen the most during the run: Flame Graphs. This looked neat, but my experience has been that I get the most understanding out of things like this by trying them out myself, so I did.

Fortunately, it's very easy to setup, as Brendan provided the tools as two standalone perl scripts you can download from the Flame Graph github repo. Then the next step is deciding what you want to run it on and capturing the data from a run of that.

It just so turns out that a few days before the Flame Graph release, another mail had crossed my inbox that gave me something to look at. Several years ago, I added some Userland Statically Defined Tracing probes to the Xserver code base, creating an Xserver provider for DTrace. These got integrated first to the Solaris Xservers (both Xsun & Xorg) and then merged upstream to the X.Org community release for Xserver 1.4, where they're available for all platforms with DTrace support.

Earlier this year, Adam Jackson enabled the dtrace hooks in the Fedora Xorg server package, using the SystemTap static probe support for DTrace probe compatibility. He then noticed a performance drop, which isn't supposed to happen, as DTrace probes are simply noops when not being actively traced, and submitted a fix for it upstream.

This was due to two of the probes I'd defined - when the Xserver processes a request from a client, there's a request-start probe just before the request is processed, and a request-done probe right after the request is processed. If you just want to see what requests a client is making you can trace either one, but if you want to measure the time taken to run a request, or determine if something is happening while a request is being processed, you need both of the probes. When they first got integrated, the code was simple:

#ifdef XSERVER_DTRACE

XSERVER_REQUEST_START(GetRequestName(MAJOROP), MAJOROP,

((xReq *)client->requestBuffer)->length,

client->index, client->requestBuffer);

#endif

// skipping over input checking and auditing, to the main event,

// the call to the request handling proc for the specified opcode:

result = (* client->requestVector[MAJOROP])(client);

#ifdef XSERVER_DTRACE

XSERVER_REQUEST_DONE(GetRequestName(MAJOROP), MAJOROP,

client->sequence, client->index, result);

#endif

The compiler sees XSERVER_REQUEST_START and XSERVER_REQUEST_DONE as simple function calls, so it does whatever work is necessary to set up their arguments and then calls them. Later, during the linking process, the actual call instructions are replaced with noops and the addresses recorded so that when a dtrace user enables the probe the call can be activated at that time. In these cases, that's not so bad, just a bunch of register access and memory loads of things that are going to be needed nearby. The one outlier is GetRequestName(MAJOROP) which looks like a function call, but was really just a macro that used the opcode as the index an array of strings and returned the string name for the opcode so that DTrace probes could see the request names, especially for extensions which don't have static opcode mappings. For that the compiler would just load a register with the address of the base of the array and then add the offset of the entry specified by MAJOROP in that array.

All was well and good for a bit, until a later project came along during the Xorg 1.5 development cycle to unify all the different lists of protocol object names in the Xserver, as there were different ones in use by the DTrace probes, the security extensions, and the resource management system. That replaced the simple array lookup macro with a function call. While the function doesn't do a lot more work, it does enough to be noticed, and thus the performance hit was taken in the hot path of request dispatching. Adam's patch to fix this simply uses is-enabled probes to only make those function calls when the probes are actually enabled. x11perf testing showed the win on a Athlon 64 3200+ test system running Solaris 11:

Before: 250000000 trep @   0.0001 msec ( 8500000.0/sec): X protocol NoOperation
After:  300000000 trep @   0.0001 msec (10400000.0/sec): X protocol NoOperation

But numbers are boring, and this gave an excuse to try out Flame Graphs. To capture the data, I took advantage of synchronization features built into xinit and dtrace -c:

# xinit /usr/sbin/dtrace -x ustackframes=100 \
  -n 'profile-997 /execname == "Xorg"/ { @[ustack()] = count(); }' \
  -o out.user_stacks.noop -c "x11perf -noop" \
  -- -config xorg.conf.dummy

To explain this command, I'll start at the end. For xinit, everything after the double dash is a set of arguments to the Xserver it starts, in this case, Xorg is told to look in the normal config paths for xorg.conf.dummy, which it would find is this simple config file in /etc/X11/xorg.conf.dummy setting the driver to the Xvfb-like “dummy” driver, which just uses RAM as a frame buffer to take graphics driver considerations out of this test:

Section "Device"
	Identifier  "Card0"
	Driver      "dummy"
EndSection

Since I'm using a modern Xorg version, that's all the configuration needed, all the unspecified sections are autoconfigured. xinit starts the Xserver, waits for the Xserver to signal that it's finished its start up, and then runs the first half of the command line as a client with the DISPLAY set to the new Xserver. In this case it runs the dtrace command, which sets up the probes based on the examples in the Flame Graphs README, and then runs the command specified as the -c argument, the x11perf benchmark tool. When x11perf exits, dtrace stops the probes, generates its report, and then exits itself, which in turn causes xinit to shut down the X server and exit.

The resulting Flame Graphs are, in their native SVG interactive form:

Before

After

If your browser can't handle those as inline SVG's, or they're scrunched too small to read, try Before (SVG) or Before (PNG), and After (SVG) or After (PNG).

You can see in the first one a bar showing a little over 10% of the time was in stacks involving LookupMajorName, which is completely gone in the second patch. Those who saw Adam's patch series come across the xorg-devel list last week may also notice the presence of XaceHook calls, which Adam optimized in another patch. Unfortunately, while I did build with that patch as well, we don't get the benefits of it since the XC-Security extension is on by default, and those fill in the hooks, so it can't just bypass them as it does when the hooks are empty.

I also took measurements of what Xorg did as gdm started up and a test user logged in, which produced the much larger flame graph you can see in SVG or PNG. As you can see the recursive calls in the font catalogue scanning functions make for some really tall flame graphs. You can also see that, to no one's surprise, xf86SlowBCopy is slow, and a large portion of the time is spent “blitting” bits from one place to another. Some potential areas for improvement stand out - like the 5.7% of time spent rescanning the font path because the Solaris gnome session startup scripts make xset fp calls to add the fonts for the current locale to the legacy font path for old clients that still use it, and another nearly 5% handling the ListFonts and ListFontsWithInfo calls, which dtracing with the request-start probes turned out to be the Java GUI for the Solaris Visual Panels gnome-panel applet.

Now because of the way the data for these is gathered, from looking at them alone you can't tell if a wide bar is one really long call to a function (as it is for the main() function bar in all these) or millions of individual calls (as it was for the ProcNoOperation calls in the x11perf -noop trace), but it does give a quick and easy way to pick out which functions the program is spending most of its time in, as a first pass for figuring out where to dig deeper for potential performance improvements.

Brendan has made these scripts easy to use to generate these graphs, so I encourage you to try them out as well on some sample runs to get familiar with them, so that when you really need them, you know what cases they're good for and how to capture the data and generate the graphs for yourself. Trying really is the best method of learning.

The AI Challenge - a look back

2011-12-19T10:03:00+00:00

For the last four weeks, I have worked on my submission for the Google AI Challenge. The deadline has passed this morning, so it is time to relax while the official final tournament is running on the contest servers. Until yesterday, my submission was ranked quite consistently in the top 20. Then I uploaded my final version (which resets the skill) which was quite consistently better on the unofficial TCP servers, but given that everybody else was doing last minute tweaks, too, it's far too early to call.

I enjoyed the spirit of this contest immensely, and now I would like to document some of my thoughts on how my submission works. I have uploaded the source code to Github (https://github.com/nhaehnle/aiant) so you can peruse it while following this blog entry if you wish.

High level structure

The bot is divided into a number of modules that select high-level goals for the ants it controls. This is done in a very straightforward way. Every ant initially has no direction to go in (the field Ant::assigneddirection is initialized to false each turn). The high-level modules then simply assign directions to ants that do not have one yet, and the order in which the modules are called reflects the relative importance I assign to the various tasks. For example, the HillDefense module will only assign ants that have not been assigned by the FoodSeeker.

There are two modules that fall outside of this basic structure: The Zoc ("zone of control") module does not steer any ants. Instead, it keeps track of how fast my ants vs. enemy ants can reach each square of the map. And the Tactical module overrides the previous decisions if necessary for ants that may be involved in combat.

The strategy modules

The following strategy modules are used by the bot, assigning jobs to ants in the given order:

FoodSeeker: Getting enough food is probably the most important problem of the game, and so this module comes first. It greedily sends the closest ant to each item of food that is visible, using several breadth-first-searches.
HillDefense: When one of the bot's hills can be reached in few turns by the enemy, this module ensures that a few ants (adjusted based on the number of enemy ants in sight) stick close to the hill.

An important tweak of this code is that it does not send ants back to the hill indiscriminately. Instead, it only recalls an ant if it is too far away from the hill relative to the closest enemy. This way, ants are not needlessly wasted on patrol duty. It would probably be a good idea to treat multi-hill maps specially, but this is not done.
OpportunisticAttack: This rather stupid piece of code ensures that the ants move more aggressively towards enemy hills. After all, that is the only way to win the game.
Scout: This module assigns a tiny number of ants to re-explore squares that have not been seen in a long time.

This is needed because the rest of the code uses the Zoc module to understand that an enemy can never come out of a cul-de-sac once it's been secured. So without some re-scouting logic, the bot would simply ignore the food in those secured locations!
Diffusion: This is a very ad-hoc heuristic to spread out my ants better than they would otherwise. There would probably have been some potential for improvement in this part of the code.
Zoc-based movement: any ant that has not been assigned a move up to this point will simply use the Zoc data to move closer to the enemy. This is done in Bot::makeMoves rather than in a separate module.

On a "historical" note, it may be interesting to know that I started out with just the FoodSeeker and the Zoc-based movement. Together with Tactical, this was enough to get into the top 100 a bit more than two weeks before the deadline.

The tactical logic

The idea behind the TacticalSm module is simple:

Carve out small Submaps that contain all the ants that could potentially be involved in combat on the next turn.
Generate potential moves for both my bot and the enemy. The combination of Submap and associated PlayerMoves is called a Theater.
Evaluate those moves using some simple scoring function.
Try to find better moves for both parties.
Repeat until time runs out.
Pick a move using one of two strategies.

Note that my tactical code doesn't understand situations involving enemy ants of more than one enemy player. This is certainly a limitation, but it's hard to tell how bad it really is.

Most of the tactical code is about engineering, and careful consideration of the scoring function. For example, during a major code reorganization halfway through the competition, the tactical code stopped considering proximity to food in the scoring function. That hurt the performance of my bot quite significantly, and my bot's skill recovered when I turned that part of the scoring function back on.

There are really only two clever parts. One is to use a scoring function that, by default, assigns a higher value to the own ants, to avoid costly trades of ants, but to turn on aggressive mode if the own ants overpower the enemy. This switch is done randomly based on the ratio of ants, where the probability to turn on aggressive mode is conceptually a logistic function in the log of the ants ratio.

The second clever part is to use a small bit of rock-paper-scissors logic in the final move selection. My first method to select a move is a simple min-max (or rather, max-min): pick the move that maximizes the minimum scoring value I could possibly get, given all enemy moves under consideration. Since this is a rather conservative choice of moves, especially given that there is absolutely no look-ahead, I implemented max-average as a second strategy: choose the move that maximizes the average scoring value, where the average is over all enemy moves, weighted by their max-min value.

Of course, this strategy may sometimes be too aggressive. What's more, the best strategy may depend on the opponent. Therefore, my bot uses a variant of the randomized weighted majority algorithm to pick the move-selection strategy. At the beginning of each turn, my bot determines what the best strategy would have been in hindsight to update the weights of the strategy. One important tweak is that the weights depend both on which enemy player my bot is facing, and on the number of ants in play.

Discarded experiments

I experimented with adding more move selection strategies, but the results were not convincing at all, perhaps because it takes longer for the bot to learn which strategy to choose, so I scrapped that again.

I also implemented map symmetry detection to guess unseen enemy hills and a corresponding module for a coordinated offense against enemy hills. The code is still there, but I have disabled it. The simple implementation I have is far too aggressive and wasteful, and I didn't feel like trying to tweak it to a point where it becomes useful.

I also did some experiments with an alternative move generation method for my tactical system, as well as a very simple implementation of the sampling-based combat system described by a1k0n on the challenge forums. This simple method performed worse than my tactical code; I have some ideas for improving it (just like a1k0n obviously did), but ultimately did not have the time to try them out. Those experiments can still be found in the Git history if you feel like digging through it.

I tried some automatic tuning of parameters using meta-heuristics (see the test/tune.py script), but somehow that didn't yield very convincing results either.

Code quality?

I have to admit that the code is not very well documented, and some of it is rather hackish. I hope you'll forgive me for this type of one-off code.

There is one thing that I did consistently that I would never do in another type of project, and that is keeping STL containers around as long as possible to avoid re-allocation. I intensively rely on the fact that std::vector::clear does not free the allocated memory, at least in the default implementation of g++. By keeping those vectors around, I want to avoid unpleasant surprises in the performance of memory management.

I don't think this is strictly necessary, given that the bot actually uses surprisingly little memory, but it didn't hurt in this case. It reduces the maintainability of the code, which is why I wouldn't do it on other projects, but maintainability was obviously never a goal for a competition like this one.

A neat randomization trick

"When in doubt, throw a coin" is perhaps the most important lesson that Theoretical Compute Science teaches. When there are several directions to go in that all score the same, which one should you choose? The best approach is to choose randomly.

To facilitate this, I pre-generated all 24 permutations of the 4 directions, and all 120 permutation of the 5 possible moves of an ant, and I randomly choose one of those permutation whenever I have a loop through directions to choose from.

Sometimes, however, I have a loop through variable-sized vectors, for example, when choosing ants for an offensive move in the tactical logic. I would like to have a random permutation for those cases as well, and of course there are algorithms to generate them. But they take time, and the benefit of a perfectly uniform distribution is not that clear.

So here's a little trick I use to permute variable-sized vector of size N. I pick a random prime p out of a decently sized pre-generated list of large primes (larger than any N the code is ever going to see), as well as a random offset ofs and loop through the numbers 0..N-1. But instead of using these numbers i as an index, I use (p*i + ofs) % N as index. Since p is prime different from N, it is invertible modulo N, and therefore the map i -> p*i + ofs is a bijection, aka a permutation. Of course, this is far from a uniformly distributed permutation: there are N! potential permutations, out of which this method can generate at most N * φ(N). But hey: it's good enough for what I need.

My OpenStack work

2011-12-16T17:34:00+00:00

GStreamer 1.0 Application porting hackfest/code sprint

2011-12-13T11:09:16+00:00

We are trying to put together a hackfest to help developers using GStreamer to port their applications to GStreamer 1.0. The GStreamer Hackfest/Code Sprint 2012 will take place in Malaga, Spain, between 25th to 27th of January (3 full days). I would ask all developers out there interested in attending to add your name to the wiki as quickly as possible, just so we can estimate interest. If you are interested, but don’t know for sure if you can make it there, it would still be good if you added your name with maybe a comment mentioning that you need to verify before you are certain.

We will be offering some travel sponsorship, so even if you are short on cash we hope to have you attend, more details on the hackfest&codesprint wiki page.

For those wondering why we choose Malaga, well the reason is that Wim Taymans, the GStreamer 1.0 designer and lead developer lives there, and at a previous hackfest we tried to do in Oslo he ended up in Helsinki instead. So this time we are taking no risks, but instead we are taking the hackfest to him

Lawrence Lessig interviews Jack Abramoff

2011-12-11T21:42:48+00:00

Last week I went to the first in a new series of events called "In the Dock" -- Harvard Law professor Lawrence Lessig interviewed Jack Abramoff on the topic of Abramoff's illegal lobbying in specific, and the state of corruption in US politics in general. I took some photos (CC-BY-SA 3.0), you can watch a video of the talk here, and my friend Ben Schwartz also has a write-up including some great quotes.

I went in willing to detest what Abramoff had done, and him by extension -- and it's not that I ended up liking or trusting him, or that I was unexpecting his "Look, you put a few guys in jail, it doesn't mean the problem's been solved and everything's okay now" argument for why assigning a disproportionate amount of blame to him is unproductive, but he did manage to convince me that he's uniquely placed to be an ally to help fix the system; that he's someone who knows all of its intricacies and is sincere about achieving a sense of redemption by working on stopping other people from doing exactly what he did.

His argument for how his book tour isn't a cynical attempt to make money is convincing, too: to the small extent that he's able to make money from books and speaking, he's forced to use it to repay a $40m restitution order to the government and his victims.

The most surprising thing I learned was that the crimes he went to jail for were not the particularly objectionable democracy-perverting forms of corruption that we ascribe to him -- those are totally legal (even more so than before, thanks to Citizens United) and still happening today across Washington's 30,000 lobbyists -- but instead mostly unrelated charges, like mail fraud. He thinks that the only way to stop bribery in Congress is to ban political contributions from anyone who stands to benefit from public funds (which Lessig criticized as being far too ambiguous and broad: who doesn't stand to benefit from public funds?), and to ban lawmakers and their staff from later working for lobbyists for the rest of their lives. He described how, before his downfall, he would agree to hire a lawmaker's staffers later while they were still working for the lawmaker, and would then have control over them from that point onwards, even though no money had changed hands -- not only is this movement from being congressional staff to becoming a lobbyist still legal, it's daily routine.

I hope this talk series continues. I can't think of many other examples of powerful figures being brave enough to open themselves up and engage in an extended ethical (rather than legal or technical) critique and cross-examination by their peers and the public, and it was powerful to watch and learn from.

(This is reposted from my Google+ stream.)

XDC 2012: Nuremberg!

2011-12-09T04:32:10+00:00

Yes! The board has decided! XDC comes to Nuremberg!

For 2012 we (Egbert Eich, Professor Hopf, and I) will be hosting the annual X conference in Nuremberg!

Egbert will try to get the main SuSE conference room, or, failing that, Matthias will try to get us a university aula, so the venue itself will work itself out beautifully in one way or another. Then... Nuremberg is one of those places which is perfect for large crowds who need food and some liquids in the evening (frankonian/bavarian beergarten culture), so it is the perfect (and highly affordable) conference area from that point of view. And, the best part, even though Nuremberg is not the international hub that Frankfurt is, or the european hub that Munich is, it is halfway between the two, and travel is relatively easy from either of those points, either you take the plane, or you take a much more comfortable train from either airport, and get to Nuremberg in pretty much the same time. You can really make a big save comparing those two airports when flying inside european aerospace, and this for no time difference. One insider tip though: you get to ride the ICE at full speed (300+km/h!) when traveling from Munich (you do have to endure the rather pedestrian S-bahn for 45 minutes though).

Anyway, the main action item now is that Egbert can start to poke SuSE to see when their main conference room is available for 3 days in september 2012 (working network and enough power sockets are a given then!). I doubt that we will get an answer still in the three remaining weeks of this year.

The actual proposal e-mail sent to the board is sadly only available to X.org foundation members, but a wiki page will soon be created which recreates most of that information. But rest assured, we will get close to the wonderful experiences of XDC Toulouse (thank you Matthieu!) and XDC chicago (thank you Michael!) indeed!

(oh, and btw, we have a FOSDEM DevRoom this year, which is rapidly getting its schedule filled! If you are coming, get your talk in right now: first come, first serve!)

Evergreen HDMI audio ready!

2011-12-07T22:31:36+00:00

Few days ago I've managed to hack radeon to play audio over HDMI on my Evergreen card. At first I just forced radeon to use single hardcoded HDMI block I've discovered on my card.
Then I started looking for universal (general) solution. I was trying to do HDMI block mapping based on crtc offset, but it failed. Trying to use encoder type (which is really a transmitter in Radeon hardware) also didn't work out.

Finally, after analyzing dumps I've received from people, I found out the real solution. HDMI block mapping has to be based on DIG encoder.

I want to help the community for helping me with the dumps. Without them I wouldn't write really-working code so fast. Sometimes you don't have to known ANSI C and hardware programming to help in development :)

The patches will be released in minutes, I hope every Evergreen card will be supported now. They hopefully will hit Dave's tree soon and kernel 3.3 as the result.

P.S.
List of patches:
[PATCH] drm/radeon/kms: minor HDMI audio cleanups
[PATCH 1/3] drm/radeon/kms: one more HDMI cleanup (in disabling code)
[PATCH 2/3] drm/radeon/kms: support for audio on Evergreen
[PATCH V2 3/3] drm/radeon/kms: enable HDMI mode on Evergreen encoders
[PATCH V2] drm/radeon/kms: do not force DVI mode on DCE4 if audio is on

New job, new blog

2011-12-07T13:47:00+00:00

Are you in Malaga, Spain? Or know someone there?

2011-12-07T11:52:15+00:00

I am desperately looking for someone familiar with Malaga, Spain, maybe a local LUG or similar. The reason is that we are interested in doing a GStreamer 1.0 application porting hackfest there in late January. So getting hold of someone with local knowledge who would be able to help us find a suitable venue would be great. Please get in touch with me on christian.schaller(at)gmail.com if you are in the area and willing to help or know someone would probably could help us.

why you don't actually want dpi

2011-12-07T04:35:00+00:00

Inspired by a discussion in #wayland today, here are snippets from three people explaining why X declares its DPI as 96, and why a single 'DPI' sledgehammer isn't actually what basically anyone* wants. Please read them. Thanks.

Adam Jackson:

I am clearly going to have to explain this one more time, forever. Let's see if I can't write it authoritatively once and simply answer with a URL from here out.

Matthew Garrett:

But what about the single monitor case? Let's go back to your Vaio. It's got a high DPI screen, so let's adjust to that. Now you're happy. Right up until you plug in an external monitor and now when you run any applications on the external display your fonts are twice the size they should be. WOOHOO GO TEAM of course that won't make us look like amateurs at all. So you need another heuristic to handle that, and of course "heuristic" is an ancient african word meaning "maybe bonghits will make this problem more tractable".

Federico Mena-Quintero:

People who know a bit of typography may know a few factoids:
- Printed books generally use fonts which can be from about 9 to about 12 points in size.
- A point is roughly 1/72 of an inch. For people in civilized countries, this translates to "I have no idea what the fuck a quarter pounder is".

*: Yes, I know you need to have actual point equivalence, and you've had all your displays and printer colour-calibrated for the past ten years too. You're doing all this in the GIMP or some other kind of design tool, so please yell at them to use the display size information that XRandR gives you right now, already, today.

A short update on multitouch

2011-12-06T22:05:32+00:00

For the last couple of weeks I've been pretty much working full-time on getting multitouch/XI 2.2 ready for the merge (well, I was on holidays for a bit too). So first of all - sorry if I've been ignoring bugs or emails, I'm working to a few deadlines here. Anyway, here's a bit of a status update.

Right now, it looks like touch event delivery is working, including nested grabs.Chase Douglas started on the pointer emulation while I was away and we're now at the point where emulation works, except that pointer grabs on top of multitouch clients aren't handled yet. I'm still rather optimistic to get this into 1.12, though it's getting a bit unwieldly. Carlos Garnacho has already sent me some patches, so he's testing the lot against the GTK branches.

However, since touch support cannot simply be bolted on top and needs to be integrated properly, this has triggered some extra rewrites here and there. I'm currently some 200 commits ahead of master sync-point. I'm planning to get this number down to something sane before merging but meanwhile, sorry, I'll have to keep ignoring you until this is done.

HTML5, Video editing and the Collabora website

2011-12-06T16:42:17+00:00

So I did a bit of work last week to convert the Collabora website to HTML5. The actual porting was quite simple, mostly replacing the DOCTYPE tag to the new HTML5 one. Found a few other issues through the W3C validator, but nothing major. Today I took the next (small) step in the process by actual adding some real HTML5 content to the site. Actually I only sort of did. Instead of hosting the video locally and using the new video tag I ended up uploading it to youtube and embedding the WebM video in our page. The small video clip I added is demonstrating the HTML5 video editing demo we released recently. (For those of you who missed it I recommend the HTML5 video editing blog entry by Mateu Batle, which explains the whole thing in detail and also links to the code).

The demo, while simple, is quite cool, showing off our HTML5 based touch screen interface all built on top of Webkit and the GStreamer Editing Services. The embedded video is to be found on the GStreamer Editing Services page.

As a sidenote, to make this I actually relied on the GNOME 3 built in screen capture support, which I have to say worked like a charm Always felt screen casting to be a pain before, but this time it worked very well for me.

New Intel graphics stack release

2011-12-06T12:16:26+00:00

As many of you have already seen, we have just released a new version of the Intel Linux Graphics stack, composed by Kernel 3.1 (or 3.1.x in practice), Mesa 7.11.2, Libdrm 2.4.27, vaapi and vaapi-driver-intel 1.0.15 and xf86-video-intel 2.17.

The focus of this release was on performance and stability improvements in the drivers, and I think we managed to get the job done according to what we expected. Thanks to LLC caching and improvements in both Kernel and Mesa, 3D performance was improved by a large margin on Sandy Bridge generation of GPUs onwards, and many annoying issues were fixed as well (with some of the fixes already queued for Kernel 3.2 release). And I’d also like to highlight lots of work towards GLES compliance on Sandy Bridge architecture, and many and many different stability fixes in all the projects.

For statistical purposes, my saved bugzilla searches count 253 bug reports which were closed after the previous release, and we had 594 bug tickets in total which had any activity since that release. Of course, this does not represents what was done by an accurate number, so please, just use those numbers as some sort of bugzilla pondering references.

So to sum it all up, I’d like to thank all the developers, QA, users, testers, and all the community we have around Intel Linux Graphics project for your amazing work. Yes, the Intel Linux Graphics project has not only a team within Intel – but also a huge community and users all around the world, following the Open-Source road from Kernel up to the UI components. Thank you all guys!

Enjoy!

Farstream and libnice, an interview with Youness Alaoui

2011-12-05T14:04:34+00:00

After the success of the GStreamer interview with Wim Taymans I thought we follow up with another great interview with a Collabora developer.

This time we are talking with Youness Alaoui who is one of the maintainers of Farstream, the audio and video conferencing framework built on top of GStreamer. We also cover another of Youness Alaoui projects, libnice, the NAT traversal library. So if you want to know what is happening with audio and video conferencing on Linux be sure to read the full interview with Youness Alaoui here.

Stalk^W Following designs, the easy way

2011-12-05T10:13:00+00:00

If you want to follow all the new designs from the GNOME Design team, including work-in-progress mockups, gathering of relevant art, etc. be sure to subscribe yourself to the pages that interest you in the various sections of the GNOME Wiki.

A nice trick is using our Wiki's notification, with regex support. Head onto your notification settings page, and add those lines to the "Subscribed wiki pages":

GnomeLive:Design*
GnomeLive:GnomeShell/Design*
GnomeLive:GnomeOS/Design*