
| January 27, 2012 | |
Hello Planet GNOME readers and GNOME community members,
The board of directors has agreed that from now on, to be part of Planet GNOME, it is mandatory to be a member of the GNOME Foundation. In three weeks we will proceed to remove all blogs from people that are not foundation members. This policy change means a few things:
The rationale behind this new policy is simple, we want to increase the value of becoming a foundation member. Think of this as the blogging equivalent of rocking an @gnome.org e-mail address.
Update: I have clarified the language behind the deletion of the feed. Someone in the comments thought I was talking about removing their blogs from blogs.gnome.org. This is not the case, if your membership expires your feed won't show up in the Planet, but that's it, if you have a blog in blogs.gnome.org none of your posts will be deleted.
Foundationally yours,
the Planet GNOME editors team.
| January 26, 2012 | |
One of the features of Fedora 17 is the /usr merge, put forward by Harald Hoyer and Kay Sievers[1]. In the time since this feature has been proposed repetitive discussions took place all over the various Free Software communities, and usually the same questions were asked: what the reasons behind this feature were, and whether it makes sense to adopt the same scheme for distribution XYZ, too.
Especially in the Non-Fedora world it appears to be socially unacceptable to actually have a look at the Fedora feature page (where many of the questions are already brought up and answered) which is very unfortunate. To improve the situation I spent some time today to summarize the reasons for the /usr merge independently. I'd hence like to direct you to this new page I put up which tries to summarize the reasons for this, with an emphasis on the compatibility point of view:
Note that even though this page is in the systemd wiki, what it covers is mostly orthogonal to systemd. systemd supports both systems with a merged /usr and with a split /usr, and the /usr merge should be interesting for non-systemd distributions as well.
Primarily I put this together to have a nice place to point all those folks who continue to write me annoyed emails, even though I am actually not even working on all of this...
Enjoy the read!
Footnotes:
[1] And not actually by me, I am just a supportive spectator and am not doing any work on it. Unfortunately some tech press folks created the false impression I was behind this. But credit where credit is due, this is all Harald's and Kay's work.
Things have been going really well here at the GStreamer Hackfest in Malaga. Thanks to the help of Ara and Yaiza from Nido Malaga, we have a great venue in downtown Malaga and they have also helped us greatly with sorting out food.
We have a great group of people here and are making great progress, and by tomorrow I hope we will have screenshots of quite a few applications running with GStreamer 0.11, for instance both Rhythmbox and Jokosher for instance is already screen shootable, if not fully functional 
Also making good progress on Transmageddon, even if the move to GObject Introspection bindings are making things a bit more complicated. Screenshot below of the progress so far.
Also a big thanks to Fluendo who is sponsoring the lunches at the hackfest and Collabora who is sponsoring tonight’s dinner. Ensuring that no hacker is left hungry during this hackfest.
Update: Yaiza took these photos from the hackfest
| January 25, 2012 | |
With all the talk generated by Arun Raghavans blog post comparing PulseAudio and Audioflinger I thought it would be good to follow up with an interview with Arun about the latest developments in PulseAudio and the way forward for the project. You can find the PulseAudio interview here. I also made a new page listing all the Collabora developer interviews done so far. Enjoy
| January 23, 2012 | |
Tomorrow I will be heading off to attend the GStreamer Application Porting Hackfest in Malaga, Spain. I think we have managed to pull together an absolutely incredible group of people for this event and I have great hopes that by next weekend we will have squashed a ton of bugs in GStreamer 0.11/1.0 and also have initial ports of a long range of important applications and bindings. This is the first time in GStreamer history that we are trying to hold a hackfest focused on application developers, but hopefully it will be the first of many and that they can become a good way for the core GStreamer community and the application development community to interact and collaborate more closely.
Also want to say a special thanks to the community members attending the event on their own and also to the companies sending their employees to the hackfest; Collabora, Fluendo, Flumotion and Igalia and finally a special thanks to the GNOME Foundation for sponsoring some of the attendees.
Hopefully I will be able to post some screenshots of a fully functional GStreamer 1.0 Transmageddon next weekend
| January 22, 2012 | |
It is a running joke in the X.Org community that X12 development will commence as soon as all of the X11 bugs in the FreeDesktop.org bug tracker have been closed. When I first heard this, my reaction was, of course, "Challenge accepted!" But where to start?
There were about 10,000 bugs open on the tracker at that point, a few years ago. We're definitely improving; there are about 9180 bugs open (at the time of writing) which could be considered relevant to any kind of desktop development. Many of those don't belong to anything in X11, but assuming they did...
Pretend that all of the card-carrying members of X.Org closed 92 bugs. That would be 1% per person. We could be done with X11! So, during XDC 2011, I pulled out my netbook, and started closing bugs. I picked, as my targets, things that nobody would miss, like Xprint.
I closed some Xprint-related bugs, before simply shuttering everything in the Xprint product. I also picked up a couple of stray bugs. Finally, with maintainer permission, I closed everything still assigned to xf86-video-nv.
The results:
- 24 Xprint-related bugs
- 37 bugs in Product: xprint
- 42 bugs in Component: driver/nv
- 37 bugs in Component: driver/nVidia (open)
- #9455, #28657, #29832, #30129
A total of 144 bugs. We can do this, guys!
| January 20, 2012 | |
Last October we published a wishlist for plumbing related features we'd like to see added to the Linux kernel. Three months later it's time to publish a short update, and explain what has been implemented in the kernel, what people have started working on, and what's still missing.
The full, updated list is available on Google Docs.
In general, I must say that the list turned out to be a great success. It shows how awesome the Open Source community is: Just ask nicely and there's a good chance they'll fulfill your wishes! Thank you very much, Linux community!
We'd like to thank everybody who worked on any of the features on that list: Lucas De Marchi, Andi Kleen, Dan Ballard, Li Zefan, Kirill A. Shutemov, Davidlohr Bueso, Cong Wang, Lennart Poettering, Kay Sievers.
Of the items on the list 5 have been fully implemented and are already part of a released kernel, or already merged for inclusion for the next kernels being released.
For 4 further items patches have been posted, and I am hoping they'll get merged eventually. Davidlohr, Wang, Zefan, Kirill, it would be great if you'd continue working on your patches, as we think they are following the right approach[1] even if there was some opposition to them on LKML. So, please keep pushing to solve the outstanding issues and thanks for your work so far!
Footnotes
[1] Yes, I still believe that tmpfs quota should be implemented via resource limits, as everything else wouldn't work, as we don't want to implement complex and fragile userspace infrastructure to racily upload complex quota data for all current and future UIDs ever used on the system into each tmpfs mount point at mount time.
Here's the twelfth installment of my ongoing series on systemd for Administrators:
One of the core features of Unix systems is the idea of privilege separation between the different components of the OS. Many system services run under their own user IDs thus limiting what they can do, and hence the impact they may have on the OS in case they get exploited.
This kind of privilege separation only provides very basic protection however, since in general system services run this way can still do at least as much as a normal local users, though not as much as root. For security purposes it is however very interesting to limit even further what services can do, and shut them off a couple of things that normal users are allowed to do.
A great way to limit the impact of services is by employing MAC technologies such as SELinux. If you are interested to secure down your server, running SELinux is a very good idea. systemd enables developers and administrators to apply additional restrictions to local services independently of a MAC. Thus, regardless whether you are able to make use of SELinux you may still enforce certain security limits on your services.
In this iteration of the series we want to focus on a couple of these security features of systemd and how to make use of them in your services. These features take advantage of a couple of Linux-specific technologies that have been available in the kernel for a long time, but never have been exposed in a widely usable fashion. These systemd features have been designed to be as easy to use as possible, in order to make them attractive to administrators and upstream developers:
All options described here are documented in systemd's man pages, notably systemd.exec(5). Please consult these man pages for further details.
All these options are available on all systemd systems, regardless if SELinux or any other MAC is enabled, or not.
All these options are relatively cheap, so if in doubt use them. Even if you might think that your service doesn't write to /tmp and hence enabling PrivateTmp=yes (as described below) might not be necessary, due to today's complex software it's still beneficial to enable this feature, simply because libraries you link to (and plug-ins to those libraries) which you do not control might need temporary files after all. Example: you never know what kind of NSS module your local installation has enabled, and what that NSS module does with /tmp.
These options are hopefully interesting both for administrators to secure their local systems, and for upstream developers to ship their services secure by default. We strongly encourage upstream developers to consider using these options by default in their upstream service units. They are very easy to make use of and have major benefits for security.
A very simple but powerful configuration option you may use in systemd service definitions is PrivateNetwork=:
... [Service] ExecStart=... PrivateNetwork=yes ...
With this simple switch a service and all the processes it consists of are entirely disconnected from any kind of networking. Network interfaces became unavailable to the processes, the only one they'll see is the loopback device "lo", but it is isolated from the real host loopback. This is a very powerful protection from network attacks.
Caveat: Some services require the network to be operational. Of course, nobody would consider using PrivateNetwork=yes on a network-facing service such as Apache. However even for non-network-facing services network support might be necessary and not always obvious. Example: if the local system is configured for an LDAP-based user database doing glibc name lookups with calls such as getpwnam() might end up resulting in network access. That said, even in those cases it is more often than not OK to use PrivateNetwork=yes since user IDs of system service users are required to be resolvable even without any network around. That means as long as the only user IDs your service needs to resolve are below the magic 1000 boundary using PrivateNetwork=yes should be OK.
Internally, this feature makes use of network namespaces of the kernel. If enabled a new network namespace is opened and only the loopback device configured in it.
Another very simple but powerful configuration switch is PrivateTmp=:
... [Service] ExecStart=... PrivateTmp=yes ...
If enabled this option will ensure that the /tmp directory the service will see is private and isolated from the host system's /tmp. /tmp traditionally has been a shared space for all local services and users. Over the years it has been a major source of security problems for a multitude of services. Symlink attacks and DoS vulnerabilities due to guessable /tmp temporary files are common. By isolating the service's /tmp from the rest of the host, such vulnerabilities become moot.
For Fedora 17 a feature has been accepted in order to enable this option across a large number of services.
Caveat: Some services actually misuse /tmp as a location for IPC sockets and other communication primitives, even though this is almost always a vulnerability (simply because if you use it for communication you need guessable names, and guessable names make your code vulnerable to DoS and symlink attacks) and /run is the much safer replacement for this, simply because it is not a location writable to unprivileged processes. For example, X11 places it's communication sockets below /tmp (which is actually secure -- though still not ideal -- in this exception since it does so in a safe subdirectory which is created at early boot.) Services which need to communicate via such communication primitives in /tmp are no candidates for PrivateTmp=. Thankfully these days only very few services misusing /tmp like this remain.
Internally, this feature makes use of file system namespaces of the kernel. If enabled a new file system namespace is opened inheritng most of the host hierarchy with the exception of /tmp.
With the ReadOnlyDirectories= and InaccessibleDirectories= options it is possible to make the specified directories inaccessible for writing resp. both reading and writing to the service:
... [Service] ExecStart=... InaccessibleDirectories=/home ReadOnlyDirectories=/var ...
With these two configuration lines the whole tree below /home becomes inaccessible to the service (i.e. the directory will appear empty and with 000 access mode), and the tree below /var becomes read-only.
Caveat: Note that ReadOnlyDirectories= currently is not recursively applied to submounts of the specified directories (i.e. mounts below /var in the example above stay writable). This is likely to get fixed soon.
Internally, this is also implemented based on file system namspaces.
Another very powerful security option in systemd is CapabilityBoundingSet= which allows to limit in a relatively fine grained fashion which kernel capabilities a service started retains:
... [Service] ExecStart=... CapabilityBoundingSet=CAP_CHOWN CAP_KILL ...
In the example above only the CAP_CHOWN and CAP_KILL capabilities are retained by the service, and the service and any processes it might create have no chance to ever acquire any other capabilities again, not even via setuid binaries. The list of currently defined capabilities is available in capabilities(7). Unfortunately some of the defined capabilities are overly generic (such as CAP_SYS_ADMIN), however they are still a very useful tool, in particular for services that otherwise run with full root privileges.
To identify precisely which capabilities are necessary for a service to run cleanly is not always easy and requires a bit of testing. To simplify this process a bit, it is possible to blacklist certain capabilities that are definitely not needed instead of whitelisting all that might be needed. Example: the CAP_SYS_PTRACE is a particularly powerful and security relevant capability needed for the implementation of debuggers, since it allows introspecting and manipulating any local process on the system. A service like Apache obviously has no business in being a debugger for other processes, hence it is safe to remove the capability from it:
... [Service] ExecStart=... CapabilityBoundingSet=~CAP_SYS_PTRACE ...
The ~ character the value assignment here is prefixed with inverts the meaning of the option: instead of listing all capabalities the service will retain you may list the ones it will not retain.
Caveat: Some services might react confused if certain capabilities are made unavailable to them. Thus when determining the right set of capabilities to keep around you need to do this carefully, and it might be a good idea to talk to the upstream maintainers since they should know best which operations a service might need to run successfully.
Caveat 2: Capabilities are not a magic wand. You probably want to combine them and use them in conjunction with other security options in order to make them truly useful.
To easily check which processes on your system retain which capabilities use the pscap tool from the libcap-ng-utils package.
Making use of systemd's CapabilityBoundingSet= option is often a simple, discoverable and cheap replacement for patching all system daemons individually to control the capability bounding set on their own.
Resource Limits may be used to apply certain security limits on services being run. Primarily, resource limits are useful for resource control (as the name suggests...) not so much access control. However, two of them can be useful to disable certain OS features: RLIMIT_NPROC and RLIMIT_FSIZE may be used to disable forking and disable writing of any files with a size > 0:
... [Service] ExecStart=... LimitNPROC=1 LimitFSIZE=0 ...
Note that this will work only if the service in question drops privileges and runs under a (non-root) user ID of its own or drops the CAP_SYS_RESOURCE capability, for example via CapabilityBoundingSet= as discussed above. Without that a process could simply increase the resource limit again thus voiding any effect.
Caveat: LimitFSIZE= is pretty brutal. If the service attempts to write a file with a size > 0, it will immeidately be killed with the SIGXFSZ which unless caught terminates the process. Also, creating files with size 0 is still allowed, even if this option is used.
For more information on these and other resource limits, see setrlimit(2).
Devices nodes are an important interface to the kernel and its drivers. Since drivers tend to get much less testing and security checking than the core kernel they often are a major entry point for security hacks. systemd allows you to control access to devices individually for each service:
... [Service] ExecStart=... DeviceAllow=/dev/null rw ...
This will limit access to /dev/null and only this device node, disallowing access to any other device nodes.
The feature is implemented on top of the devices cgroup controller.
Besides the easy to use options above there are a number of other security relevant options available. However they usually require a bit of preparation in the service itself and hence are probably primarily useful for upstream developers. These options are RootDirectory= (to set up chroot() environments for a service) as well as User= and Group= to drop privileges to the specified user and group. These options are particularly useful to greatly simplify writing daemons, where all the complexities of securely dropping privileges can be left to systemd, and kept out of the daemons themselves.
If you are wondering why these options are not enabled by default: some of them simply break seamntics of traditional Unix, and to maintain compatibility we cannot enable them by default. e.g. since traditional Unix enforced that /tmp was a shared namespace, and processes could use it for IPC we cannot just go and turn that off globally, just because /tmp's role in IPC is now replaced by /run.
And that's it for now. If you are working on unit files for upstream or in your distribution, please consider using one or more of the options listed above. If you service is secure by default by taking advantage of these options this will help not only your users but also make the Internet a safer place.
| January 18, 2012 | |
Do you wanna contribute to a funky open-source project? Are you tired of your nerdy and boring community of developers? Are you the one that wants to get rid of X because it’s a giant, old and fat dinosaur in your system? :) Cool, I have a project to solve your problems!
—
While there’s still lot of churn in the protocol, and yet our goal is soon to wrap up all we’ve been doing to a steady and settle-on-the-stone one description, there’s a lot on the implementation side that needs love. And that mainly concern Weston compositor, which is becoming the de facto compositor on several systems targeting Wayland.
In past months I was accumulating a TODO list for Weston and libraries that I think wouldn’t require any exceptional knowledge on core graphics or Wayland internals. These are good items for people that want start hack or just do some contribution on Wayland:
1. log facility (easy)
Back in July, I had already warmed up a discussion how we could log on Wayland. So now we spit everything to stdout but we want to do it similarly as Xorg, i.e. redirecting to its own file. It turned out that we might want only on Weston compositor and implement our own way of logging for sake of simplicity. Ideally it has to be very simple, without verbosity levels probably, etc. This task should be quite easy to finish.
2. launcher for Weston (medium)
Weston is meant to run as a normal user. Now we have to set manually input devices, DRM and tty with root permissions, so Weston can happily be started. Ideally we should have a setuid helper script doing all this tricky, and in fact I started something here. For a real system though, we need to enhance a bit this program with the policykit, specially for dealing with hotplugs. Probably zero understanding on Wayland internals is needed but an overall knowledge of OS is required.
3. libxkbcommon X merge (medium)
Actually that’s not much Wayland work, but it most definitely would help its development. xkbcommon is the library that exposes the keyboard mapping logic to clients, converting keycodes in keysyms. Weston clients are using for evdev convertion. The library is an adaptation of a bunch of X11 modules to fit in a non-X11 world and as such, it still requires xproto, kbproto and libX11. One task would be to untie such dependencies with X and the other to proper merge libxkbcommon with Xorg. I’d classify as medium-size task due the involvement with the community and the hairy understanding of XKB in general, although Daniel, Kristian and other guys already made a nice progress there.
4. xwayland on X (hard)
xwayland is the implementation on Xorg to make it run as a Wayland client. That works well on Intel chipsets and might work as well with other drivers through the shm X driver. In principle, basic X11 applications work with some WM control lacking, input grab as well and things like Xrandr don’t. One would also forward port xwayland and driver to upstream, once the work is shaped up. I bet WM developers would have an ecstasy and delight themselves hacking around.
—
Hopefully you get excited with some of the items. I definitely can give a hand with any, so just poke me on #wayland at freenode.
To my European readers: if you care about the impact of social technologies like Git (and GitHub) & how they’re transforming software development, or the impact of social technology on communities, and you enjoy good beer, you need to be at Monki Gras. I just posted over at my RedMonk blog about how the previous conference in the series, Monktoberfest, was the best conference of my life. And I’ve been to many.
Monki Gras is Feb. 1–2 in London. The timing’s perfect to stop by just before FOSDEM (and that’s exactly what I’m doing). Registration is dirt-cheap, speakers are universally top-notch, and you’ll also get some world-class beers in the package.
| January 16, 2012 | |
Arun put an awesome article up, detailing how PulseAudio compares to Android's AudioFlinger in terms of power consumption and suchlike. Suffice to say, PulseAudio rocks, but go and read the whole thing, it's worth it.
Apparently, AudioFlinger is a great choice if you want to shorten your battery life.
| January 14, 2012 | |
For the ones of you who haven’t read Phoronix lately – Michael is running a survey about Intel Linux Graphics drivers for the past few days.
So far it got 10 pages of amazingly interesting comments (which I was trying to answer as time and my fingertips permitted). 95 comments as of now to be fair (sorry, 96 already – my bad
). Make sure to check it out.
In the end, I’ll try to summarize all my replies in the thread here. But it will be a long read by the looks of it.
| January 11, 2012 | |
One of the most frequent complains questions related to the Intel Linux Graphics drivers I’ve received in the past few months was: “Why Intel devs work only on the most bleeding edge, and do not give enough attention the us stable users”?
Yes, this question affects all the components of the stack – kernel, mesa, libdrm, 2D driver, and so on, but the answer to this is quite simple – this is how software gets developed. New features go into new releases, and stable releases receive bugfixes and stability improvements at most. And this is not much of an issue to the userspace components anyway – with all the LD_LIBRARY_PATH flexibility it is possible to have multiple versions of the libraries installed without any issue. But as for the kernel, indeed, it is not that easy. So most of the times, those questions were directed at the kernel part of our driver – namely, the tiny i915.ko module which is responsible for making the graphical heart of Intel-based GPU beat.
Even for the kernel, this is not exactly true – Greg’s stable trees do include most of the critical fixes for our drivers since always. But it is true to some point – most of the newest development and patching happens within the usual Linux development window – and those patches and features are then merged to the release candidates of a future kernel during the merge window. And for the kernel, it is not that easy to have multiple kernels around at the same time without having to reboot between them here and again.
So therefore, I thought on giving a small gift to the users who are not still ready to jump into the latest and greatest kernels releases – but still want to enjoy the goodies brought with the latest versions of the graphics driver. So I prepared two kernel branches on my freedesktop.org kernel repository:
Besides driver backports, the patches which affect multiple drivers (like, i915, nouveau and r128) were also backported in full in those branches. And all the required build and API dependencies (like HDMI/DP ELD or drm/gem API changes) are also included in full.
I’ll try to keep those branches maintained on a semi-periodic basis and synchronized with the latest updates both from -stable branches, and from newest kernel releases.
Of course, let me put an obligatory support and stability statement:
Those branches are not supported officially, are experimental and can be unstable or even broken sometimes (in which case, if you’d be kind enough to let me know about that, I’d try to fix them ASAP). So use them at your own risk.
Other than that, have fun
.
| January 06, 2012 | |
A while ago, we got some money from Intel, Google and HP for new machines for freedesktop.org. Due to various logistical problems, it has taken us far too long to actually start using those, but we're now in the process of doing so.
The first service to move to the new infrastructure is cgit. As part of this transition, we'll also be retiring some old services, amongst those are:
We might also retire ftp.freedesktop.org, but that is not yet decided. The rationale being that HTTP is widespread and works well for handling downloads. For mirroring, rsync is better suited than ftp
Additionally, over the weekend we'll move bugzilla to the new hosts as well, something that should improve performance quite a bit, but it means bugzilla might be unavailable sometimes during the weekend.
Feedback is most welcome
– tfheenAfter updating GStreamer and doing a couple of small fixes I managed to make Transmageddon work with the GTK3 and the 0.11 branch of GStreamer. Obligatory screenshot below. As you might guess from looking at the screenshot there are still some issues that needs solving, but
I am happy that I managed to get this far.
Hopefully it is a sign that the upcoming GStreamer hackfest in Malaga will be a great successful everyone who is participating.
I hope the remainder of the porting effort will be relatively simple as I would love to get back to working on real features instead of just updating old functionality to use a new backend to do the same. Having had a need for Transmageddon for a couple of work related tasks recently a couple of items, like batch job programming has moved up my priorities list.
So, holidays came to an end, and it is time to get started with the 2012 series of “tales from the Intel Linux Graphics land”.
But as the year has just started, it is still time to cover some amazing gifts which Santa ClausDaniel Vetter brought to you all right in time for Christmas. Yes, I am talking about the new intel-gpu-tools 1.1 release!
This new release comes after 2 years from the 1.0.2 release (which came out back in 2009), and accumulates an astonishing amount of changes:
Let me stop a bit on this latest feature, and explain it in a more detailed form. Starting with the 1.1 version of intel-gpu-toops, the package provides a complete testing suite for most of the known issues and problems that have been caught in our graphics drivers for Linux for the past years. So when you run make test (as root) in the intel-gpu-tools directory, this will start a long pile of test cases, testing most parts of the GPU.
Ideally, all tests should pass – this means that your current kernel and userspace libraries are not affected by any known kernel (or userspace) issue. Some tests can fail – usually this comes with an explanation of why it fails (either on the test output or in dmesg). And some tests can fail in a very evil way, hanging the GPU or the entire machine. A specific category of tests (HANG tests), which are not run by default, are designed to test specific corner cases which should make your machine hang for good; and some tests are written specifically to caught performance regressions and hick-ups.
Still wonder why is this interesting? Well, I can think on one obvious huge bonuses we get from this test suite – if you are experiencing an issue, you can run the make test suite, and verify if it is a new genuine bug, or something that was already caught by a test case. And obviously, this is an easy way to find and track regressions on the kernel and drm level.
So, to sum it all up – if you want to give a quick try and check if your kernel/userspace/distribution configuration is affected by any known issue in the driver, just run the testsuite. If it passes, you are good to go. If it does not passes, well, you already know that something is not right and needs investigation.
Besides intel-gpu-tools, Mesa work continues towards the remaining GL 3.0 features. During last few days of 2011, Eric Anholt sent some patches for adding hardware support to GL_EXT_transform_feedback bits present on the Ivy Bridge architecture. Besides those changes, hundreds of patches and discussions take place on the mesa-dev mailing list – the most user-notable change comes from the fact that XCB is now mandatory. I guess we have a clear winner in the XCB vs XLIB match in 2012..
On Kernel side, with the release of 3.2 version of the kernel, next merge windows has opened and already received some patches for 3.3 kernel series. Besides the usual fixes and new features which I already covered in previous posts, several alternatives were proposed by Chris Wilson and Eric Anholt to solve the mysterious missed IRQ issue which affects Ivy Bridge (and some Sandy Bridge) machines. The winner in this competition, however, was Daniel Vetter, who came with a simple patch that not only had solved the issues in a nice way, but also allowed us to remove several work-arounds which were added in the past. So far, all my Ivy Bridge machines are up and running for some days without any problems – which is totally awesome!
Moving to other projects, notable changes which were introduced in the past weeks were the move of intel_decode facility for decoding GPU batch buffers from intel-gpu-tools to libdrm library by Eric Anholt; patches from Paulo Zanoni to support new screen rotation feature of VPro; additional patches for improved Glamor performance from Zhigang Gong; support for building libdrm on Android from Chad Versace; Kristian Høgsberg created a new W-prefixed project for Wayland (namely, Weston, which is a different town in Massachusetts state. I actually got to get through it once when I was in the US back in 2005, working on a remote boot and software streaming project for mstech, which was further acquired by Ardence, which was further acquired in Citrix.. fun to see how the world goes in circles
).
So I think that’s it for now. For the ones who are still in the new year holidays, enjoy the rest of your holidays; and for the ones who are already back into to the real life – welcome to 2012!
| January 04, 2012 | |
Just posted another interview on the Collabora website, this time with Gustavo Noronha Silva talking about WebKit and some of the work he and Collabora are doing around that project. So be sure to check it out if you want to learn more about things like WebKit and Clutter integration and how WebKit impacts the GNOME platform.
| January 03, 2012 | |
I use Google Calendar to manage my calendars, and I really missed something to warn me whenever I have an appointment with an alert set.
So here is an example of a Python program to do such a thing. It is written using the Google Data APIs Python client library and pynotify.
I'll detail the code here, so you can build your own and adapt it to your needs.
First, we need to import GTK+ and pynotify, and initialize it.
import gtk import pynotify pynotify.init(sys.argv[0])
Then, we need to import gdata Calendar API and connect to the calendar. I'll use the simple email/password way to login, which is clearly not the best, but it's also the simplest. Feel free to use OAuth 2.0. :-)
calendar_service = gdata.calendar.service.CalendarService() calendar_service.email = 'mygooglelogin' calendar_service.password = 'mygooglepassword' calendar_service.ProgrammaticLogin()
Now we're ready to request stuff and notify! First, request the events from the default calendar.
feed = calendar_service.GetCalendarEventFeed()
Now we can iterate over feed and do various checks.
for event in feed.entry:
# If the event status is not confirmed, go to the next event.
if event.event_status.value != "CONFIRMED":
continue
# Now iterate over all the event dates (usually it has one)
for when in event.when:
# Parse start and end time
try:
start_time = datetime.datetime.strptime(when.start_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
end_time = datetime.datetime.strptime(when.end_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
except ValueError:
# ValueError happens on parsing error. Parsing errors
# usually happen for "all day" events since they have
# not time, but we do not care about this events.
continue
now = datetime.datetime.now()
# Check that the event hasn't already ended
if end_time > now:
# Check each alert
for reminder in when.reminder:
# We handle only reminders with method "alert"
# and whose start time minus the reminder delay has passed
if reminder.method == "alert" \
and start_time - datetime.timedelta(0, 60 * int(reminder.minutes)) < now:
# Build the notification
notification = pynotify.Notification(summary=event.title.text,
message=event.content.text)
# Set an icon from the GTK+ stock icons
notification.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_DIALOG_INFO,
gtk.ICON_SIZE_LARGE_TOOLBAR))
notification.set_timeout(0)
# Show the notification
notification.show()
Running this program, you should see a notification if an appointment has an alert to be raised at that time.
This should be enough to start to build something.
If you don't want to program this into Python, you might want to take a look at gcalcli.
| January 02, 2012 | |
TouchBegin to Cg →
TouchUpdate to Cg →
TouchUpdate to Cg →
← Cg rejects touch
← Cw becomes new owner
TouchEnd+ to Cg →
TouchBegin* to Cw →
TouchUpdate* to Cw →
TouchUpdate* to Cw →
#### physical touch ends ####
TouchEnd to Cw →
Events with + mark an event created by the server, * mark events replayed by the server
TouchBegin to Cg →
TouchUpdate to Cg →
TouchUpdate to Cg →
#### physical touch ends ####
TouchEnd to Cg →
← Cg rejects touch
← Cw becomes new owner
TouchBegin* to Cw →
TouchUpdate* to Cw →
TouchUpdate* to Cw →
TouchEnd* to Cw →
TouchBegin to Cg →
TouchBegin to Cw →
TouchOwnership to Cg →
TouchUpdate to Cg →
TouchUpdate to Cw →
TouchUpdate to Cg →
TouchUpdate to Cw →
← Cg rejects touch
← Cw becomes new owner
TouchEnd+ to Cg →
TouchOwnership to Cw →
#### physical touch ends ####
TouchEnd to Cw →
Note: TouchOwnership events do not correspond to any physical event, they are always generated by the serverTouchBegin to Cg →
TouchBegin to Cw →
TouchOwnership to Cg →
TouchUpdate to Cg →
TouchUpdate to Cw →
TouchUpdate to Cg →
TouchUpdate to Cw →
#### physical touch ends ####
TouchEnd to Cg →
TouchUpdate to Cw → (XITouchPendingEnd flag set)
← Cg rejects touch
← Cw becomes new owner
TouchOwnership to Cw →
TouchEnd to Cw →
In both cases, we dealt with a rejecting owner. For an accepting owner, the
sequences look like this:
TouchBegin to Cg →
TouchBegin to Cw →
TouchOwnership to Cg →
TouchUpdate to Cg →
TouchUpdate to Cw →
TouchUpdate to Cg →
TouchUpdate to Cw →
← Cg accepts touch
TouchEnd+ to Cw →
TouchUpdate to Cg →
#### physical touch ends ####
TouchEnd to Cg →
or
TouchBegin to Cg →
TouchBegin to Cw →
TouchOwnership to Cg →
TouchUpdate to Cg →
TouchUpdate to Cw →
TouchUpdate to Cg →
TouchUpdate to Cw →
#### physical touch ends ####
TouchEnd to Cg →
TouchUpdate to Cw → (XITouchPendingEnd flag set)
← Cg accepts touch
TouchEnd* to Cw →
In the case of multiple grabs, the same strategy applies in order of grab
activation. Ownership events may be selected by some clients but not others.
In that case, each client is treated as requested, so the event sequence the
server deals with may actually look like this:
TouchBegin to C1 →
TouchBegin to C3 →
TouchOwnership to C1 →
TouchUpdate to C1 →
TouchUpdate to C3 →
TouchUpdate to C1 →
TouchUpdate to C3 →
← C1 rejects touch
← C2 becomes new owner
TouchEnd+ to C1 →
TouchBegin* to C2 →
TouchUpdate* to C2 →
TouchUpdate* to C2 →
← C2 rejects touch
← C3 becomes new owner
TouchEnd+ to C2 →
TouchOwnership to C3 →
#### physical touch ends ####
TouchEnd to C3 →
| January 01, 2012 | |
Yeah, 2011 is coming to the end, and had already switched place with 2012 in most countries already.

Somewhere on the road to curitiba, starting 2011....
For me, 2011 was extremely busy, exhausting but also very fruitful. I started it in a completely different situation from 2010, when I was a mere developer in Mandriva company. And during the course of 2010 and, later, 2011, I got the chance to work as engineering team leader for Conectiva, then development manager and, finally, technical diretor for Mandriva. And, after this experience, I got to move into Intel, working with even more challenging and amazing projects.
So yes, it was a hugely crazy and great experience. Thank you 2011 – despite all the problems and obstacles, you was great after all if we sum up everything.
See you all in 2012!
Yeah, 2011 is coming to the end, and had already switched place with 2012 in most countries already.
For me, 2011 was extremely busy, exhausting but also very fruitful. I started it in a completely different situation from 2010, when I was a mere developer in Mandriva company. And during the course of 2010 and, later, 2011, I got the chance to work as engineering team leader for Conectiva, then development manager and, finally, technical diretor for Mandriva. And, after this experience, I got to move into Intel, working with even more challenging and amazing projects.

And finishing 2011 in this landscape...
So yes, it was a hugely crazy and great experience. Thank you 2011 – despite all the problems and obstacles, you was great after all if we sum up everything.
See you all in 2012!
Yeah, 2011 is coming to the end, and had already switched place with 2012 in most countries already.
For me, 2011 was extremely busy, exhausting but also very fruitful. I started it in a completely different situation from 2010, when I was a developer and impossible-mission-solver in Mandriva. And during the course of 2010 and, later, 2011, I got the chance to work as engineering team leader for Conectiva, then development manager and, finally, technical diretor for Mandriva. And, after this experience, I got to move into Intel, working with even more challenging and amazing projects.

And finishing 2011 in this landscape...
So yes, it was a hugely crazy and great experience. Thank you 2011 – despite all the problems and obstacles, you was great after all if we sum up everything.
See you all in 2012!
| December 27, 2011 | |
It took me a while to find this, so I'm just blogging it so other people will be able to find it.
I wanted to send a desktop notification using pynotify, but using a GTK+ stock icons.
With the following snippet, I managed to do it.
import pynotify
pynotify.init("myapp")
import gtk
n = pynotify.Notification(summary="Summary", message="Message!")
n.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_HARDDISK, gtk.ICON_SIZE_LARGE_TOOLBAR))
n.show()
Note that the use of a Label is just to have a widget instanciated to use the render_icon() method. It could be any widget type as far as I understand.
| December 23, 2011 | |
Unfortunately, Linux doesn’t support multiple graphics adapters the way Windows does, which means you can’t just plug in USB graphics adapters and expect them to extend your desktop (the good news is there is progress on this support).
What is possible, however, is running a single DisplayLink adapter, or several with a Xinerama or multiseat configuration — just as long as you don’t expect to use your main GPU at the same time.
The single-display case is relatively easy to set up, and we’ll cover that here.
First, make sure you’re running kernel version 2.6.35 or later (Ubuntu 10.10 or later). For older kernel versions, you’ll need to update udlfb and run a modified fbdev X server (not covered in this post). On these kernel versions, when you plug in your DisplayLink-based USB graphics device, you should get a green screen. This means that at the driver built into the Linux kernel is happy, healthy, and talking to the device.
Second, if you are running Unity Desktop in Ubuntu 11.04 or later, you’ll need to switch back to Classic Mode so you’re running straight X. Here’s how on Ubuntu:
Click on the power button in the upper right corner (mine looks like a light switch) and choose the last option, System Settings. Search for Login Screen, Double-click to display, Choose Unlock and enter your password, Select Ubuntu Classic as default session.
Third, if you’re running kernel versions between 2.6.35 to 3.1, enable the fb_defio option of udlfb. To do this, create or edit a file like
/etc/modprobe.d/50-displaylink.conf
and add the single line
options udlfb fb_defio=1
And reboot (or run “sudo depmod -a” and unplug/replug your adapter). This will turn on defio (page fault change detection) support. This option is already enabled by default in kernels 3.2+.
Lastly, create an X config file called 60-plugable.conf (or similar) with the following contents and place it in /usr/share/X11/xorg.conf.d (on recent distros; on older distros, make this your xorg.conf):
Section "Device" Identifier "uga" driver "fbdev" Option "fbdev" "/dev/fb0" EndSection Section "Monitor" Identifier "monitor" EndSection Section "Screen" Identifier "screen" Device "uga" Monitor "monitor" EndSection Section "ServerLayout" Identifier "default" Screen 0 "screen" 0 0 EndSection
Note: if your main GPU creates a /dev/fb0 even when the USB display is not attached, then your USB display is probably getting assigned to /dev/fb1. In that case, change /dev/fb0 in the “Device” section above to /dev/fb1
Now, on reboot, you should (hopefully!) see your login come up on your DisplayLink USB attached display!
This kind of simple setup is useful for:
Please comment if you have any trouble with this single display case. See our past posts for additional information about the DisplayLink Linux kernel driver and some more involved setups.
The instructed here work on all Plugable USB 2.0 graphics adapters and Plugable USB 2.0 docking stations and thin clients (and should also generally work on all DisplayLink based products).
Ever since I bought my first Palm Pilot in 1997, I’ve relied upon a pocket-able device to carry a copy of my calendar and contacts, and for that same database to be present on my laptop. I went through a long list of Palm-compatible devices, including both the Palm Treo and Palm Centro telephones. I even wrote a number of my own Palm applications. Years ago, it was pretty obvious that I’d have to find a new phone, but I was stuck looking for something that could provide the same ‘hot-sync’ functionality.
I bought a Series 40 Nokia phone in Shanghai that promised ‘SyncML’ support. Given that I had seen numerous SyncML implementations for Linux, it seemed like I should be able to get something to work.
SyncML on the Series 40 is a disaster — the phone couldn’t actually store all of my contacts, and it couldn’t hold half of the data fields I used. So, that phone landed in the box and back to the Centro I went.
Nokia kindly sent me an N900 at some point, so I gave SyncML another try. Given that the N900 runs evolution-data-server, and that I’ve had evolution-data-server running on my laptop, it seemed like I should be in business. Well, almost. It took several days of hacking to fix bugs in the evolution SyncML back end, then another several days fussing with opensync.
I ended up abandoning direct synchronization as unworkable — opensync would sit in an infinite loop, or worse, trash my database completely. I finally found ‘syncml-ds-tool’, which is a debugging tool that comes with opensync. This tool simply synchronizes a set of disk files, one per contact or calendar entry, with the phone. That worked for well over a year. And then, a few months ago, Bluetooth on the laptop stopped connecting with the phone’s SyncML server. I’d get ‘ECONNREFUSED’ every time I tried to use it. So much for the N900. DUN still worked, mostly, although it too would get ECONNREFUSED at times, but retrying seemed to make it work.
However, While the N900 SyncML solution worked, I discovered another thing I wanted—contacts and calendar entries stored in individual files and revision controlled with git. This makes it reasonable to delete stale calendar entries and know that they’re never really gone, just left behind in an older version of the calendar. And, if you mess up, you can recover by poking at the database with git directly.
I switched back to the venerable Palm Centro; it turns out that calendar and contacts are more important to me than being able to surf the web on my phone. Alas, my Centro went ‘swimming’ in August and has passed on to the great electronics recycling house in the sky. I pulled the SIM out and switched back to the N900.
I got my contacts imported on the N900 by copying files over the net work; not a long-term strategy, but at least I had phone numbers again. There was no hope for my calendar. I started looking for a solution in earnest.
At this point in my story, I’m sure you’re asking why I didn’t just use one of the numerous Android phones that came through my hands. The answer is simple — my calendar and contacts are probably some of my most personal data, and I’m not willing to store them outside of my direct control, for reasons similar to those which are driving the development of the FreedomBox.
When Android first came out, it could only talk to Google services, which didn’t meet my hard requirement for personal data storage.
One of my co-workers had his Google account suspended for violating the terms-of-service; he asked what he had done, but they wouldn’t say. He asked if he could get his data back, and they said “no”. They invited him to create a new account, but it would not ever get any of the old data. A few days later, he got a nice apologetic email letting him know that they’d made a mistake and that he hadn’t, in fact, violated any of the terms-of-service, and that his old account was restored with all of his data intact. Wasn’t that nice of Google?
I got a WebOS phone over the summer and discovered that while it had multiple contact/calendar back ends, none of them used standard protocols and so you only had the choice between multiple corporate data centers, which isn’t actually a choice at all.
Furthermore, the WebOS phone refused route PAN packets over the phone network, even though I have a data plan which allows this. It’s not that it couldn’t support it, it’s that it refused.
A couple of weeks after the WebOS phone arrived, HP canceled all of their WebOS hardware products, which made me less interested in trying to solve this problem.
About the same time the WebOS phone arrived, I discovered that Google had published enough information about the calendar/contacts internals for Marten Gajda to write CalDAV-Sync and CardDAV-Sync. And then, Andrew McMillan wrote aCal, which is a complete replacement for the built-in calendar and contacts applications and supports CalDAV and CardDAV.
With two different standards-compliant solutions available, it seemed like it might be time to try Android again.
I’d love for CardDAV-Sync and CalDAV-Sync to become free software like aCal is. Andrew makes money from aCal by offering it for sale via the Android Market, while still publishing the sources for those who want to build their own copy.
I think the most widely known CalDAV server for Linux is probably DAViCal, a huge pile of PHP and SQL sitting on top of Apache. I’m sure it’s suitable for running on a server and being accessed over the internet, but I’m not interested in that, nor am I interested in having my laptop run Apache and PostgreSQL.
I found a tiny little CalDAV server, Radicale, which seemed like a lot better fit. It’s written in Python and uses the usual Python HTTP server infrastructure, which provides SSL and authentication support along with some fairly convenient APIs for parsing and generating HTML.
Before long, I discovered that Radicale was actually too simple for my needs. It stores the whole calendar in a single file, re-parsing it whenever a request is made, so a calendar with just hundreds of entries caused the server to slow down enough that evolution would time-out when talking to it.
Also, Radicale doesn’t actually parse the calendar entries completely, it has some ad-hoc code that finds various pieces of data, but without dealing with the whole syntax.
I started hacking at Radicale to see how far I could get. I changed the storage code to store one event per file, then added hooks to use git for change management. Then, I found a full vcalendar/vcard parsing library in python, vobject, which I used to replace the ad-hoc parsing code. Finally, I added support for VCARD entries as well, allowing the system to store both calendar and contact information.
With this much divergence from the original project, I’ve figured I’d best rename things to avoid confusion, so I decided to call it ‘calypso’, after a brief trip through the dictionary looking for names starting with ‘ca’.
Calypso works with evolution, iceowl and the Android CalDAV/CardDAV plugins. It does not yet work with aCal; for some reason aCal cannot find any calendars on the server.
Calypso also supports importing calendar changes from the command line, allowing you to integrate support into a text-based email application like notmuch or mutt.
Calypso is available via git from git://keithp.com/git/calypso and is distributed under the GPL (v3 or later). I still consider it a work derived from Radicale, and so the code retains all of the Radicale copyrights along with my own.
Calypso runs as a regular user, all data are stored in ~/.config/calypso. To initialize calypso:
$ mkdir ~/.config/calypso ~/.config/calypso/calendars
$ cat > ~/.config/calypso/config << EOF
[server]
ssl=true
certificate=/home/keithp/.config/calypso/ssl/server.crt
key=/home/keithp/.config/calypso/ssl/server.key
[acl]
;type=htpasswd
type=fake
filename=/home/keithp/.config/calypso/passwd
Then run calypso:
$ python ./calypso.py
No, I haven’t figured out how to install it…
To add a new database:
$ mkdir -p ~/.config/calypso/calendars/private/my_calendar
$ cd ~/.config/calypso/calendars/private/my_calendar
$ git init
$ git commit --allow-empty -m'initialize new calendar'
The new calendar should now be visible as
https://localhost:5233/private/my_calendar
You can add files to the directory at any time; calypso will check the directory mtime at each operation and update its internal state from that on disk automatically when the directory changes.
Given a set of files with VCALENDAR or VCARD entries, you can import them with:
$ calypso --import private/my_calendar <filenames...>
This will update any changed entries and add any new ones.
Synker. This provides a desktop widget which starts the sync process running on a list of accounts. This makes it easy to manually synchronize with the laptop when you are connected
Contact Editor. This lets you fully edit contacts synchronized over CardDAV. The built-in contact editor doesn’t let you change anything other than the name for some reason.
I’m running cyanogenmod on my Nexus S as that provides PAN support. With PAN, I can create a network link between laptop and phone which doesn’t depend on any local WiFi infrastructure and which gives both phone and laptop static IP addresses, allowing me to configure the sync URLs statically on the phone. I’d use mDNS, but Android doesn’t bother to support that.
| December 22, 2011 | |
int major = 2, minor = 2;
XIQueryVersion(dpy, &major, &minor);
if (major * 1000 + minor < 2002)
printf("Server does not support XI 2.2\n");
Once announced, an XIQueryDevice(3) call may return a new class type, the XITouchClass. If this class is present on a device, the device supports multitouch.The class struct itself is defined like this:typedef struct
{
int type;
int sourceid;
int mode;
int num_touches;
} XITouchClassInfo;
The num_touches field specifies the number of simultaneous touches supported by the device. If the number is 0, we simply don't know (likely) or the device supports an unlimited number of touches (less likely). Regardless of the value expect that some devices lie, so it's best to treat this value as a guide only.
XIDeviceInfo *info;
int nevices;
info = XIQueryDevice(display, XIAllDevices, &ndevices);
for (i = 0; i < ndevices; i++)
{
XIDeviceInfo *dev = &info[i];
printf("Device name %d\n", dev->name);
for (j = 0; j < dev->num_classes; j++)
{
XIAnyClassInfo *class = dev->classes[j];
XITouchClassInfo *t = (XITouchClassInfo*)class;
if (class->type != XITouchClass)
continue;
printf("%s touch device, supporting %d touches.\n",
(t->mode == XIDirectTouch) ? "direct" : "dependent",
t->num_touches);
}
}
| December 21, 2011 | |
To run Qt applications on Wayland is fairly simple nowadays. Thank to Qt developers, they are following up quite well our last changes on Wayland protocol and updating accordingly on Qt5 code base — by the way, the fresh and just released Qt 4.8 does not ship the latest protocol additions, so that’s not the one I’m referring.
So, today I’ve set up the last bits of Qt environment on my Intel Pine-Trail pretty easy (yeah, I compile on my tablet :-O). You don’t need to clone the whole data base for starting hacking on it. Let’s see.
first, you have to clone qtbase:
# mkdir qt; cd qt
# git clone git://gitorious.org/qt/qtbase.git
Before start the compilation steps, I like to set up my testbed in a different directory than what the ordinary system uses, so I include this on my .bashrc:
export QTVER=qt5
export QTDIR=/opt/qt/$QTVER
export PATH=$QTDIR/bin/:$PATH
export LD_LIBRARY_PATH=$QTDIR/lib/:$LD_LIBRARY_PATH
export PKG_CONFIG_PATH=$QTDIR/lib/pkgconfig/:$PKG_CONFIG_PATH
export QT_PLUGIN_PATH=$QTDIR/lib/plugins/
and don’t forget to re-run your bashrc! Now, when developing a Wayland application it’s quite useful to check if your app is behaving nicely, so for doing so I like to compare it with Qt’s XCB backend. Then you’ll need the following — I’m under Ubuntu 11.10:
# apt-get install libxcb1 libxcb1-dev libx11-xcb1 libx11-xcb-dev libxcb-keysyms1 libxcb-keysyms1-dev libxcb-image0 libxcb-image0-dev libxcb-shm0 libxcb-shm0-dev libxcb-icccm4 libxcb-icccm4-dev libxcb-sync0 libxcb-sync0-dev libxcb-xfixes0-dev
you should be ready now to go for the configuration:
# ./configure -confirm-license -opensource -no-qt3support -no-multimedia -no-webkit -no-phonon -no-v8 -debug -qpa -xcb -wayland -egl -opengl es2 -nomake examples -prefix ${QTDIR}
and now, the compilation — which took approximately 1hr and half in my system:
# make
install it:
# make install
At this point you got all Qt libraries, and needed tools to compile qtwayland platform. So go back one directory and clone that:
# cd ../
# git clone git://gitorious.org/qt/qtwayland.git
You need Wayland protocol libraries now in your system, and that’s the very late ones, which means the stock packages from your distro won’t work. Get and
build them using the following:
http://wayland.freedesktop.org/building.html
You might want to develop Wayland on the top of X, so before compile you’ll need this:
# apt-get install libxcomposite-dev
Jørgen Lind, updated me with: “the default buffersharing to use is wayland_egl. To be able to use the xcomposite stuff you need to export the environment variable QT_WAYLAND_GL_CONFIG to be either xcomposite_egl or xcomposite_glx”. So in order do to Wayland on X you will need to tweak it as well.
and then:
# cd qtwayland/
# qmake
# make && make install
To run now a Qt app, I first set the XDG directory that compositor and clients use to talk under Wayland. You better to watch out, cause some distros already have this set somewhere. Anyway, I put this on my .bashrc:
# export XDG_RUNTIME_DIR=$HOME/.xdg
# mkdir $HOME/.xdg
and re-ran .bashrc. Finally the Qt app running and voilà:
# cd ../qtbase/examples/opengl/hellowindow
# qmake
# make
# wayland-compositor &
# ./hellowindow -platform wayland
—
Kudos to Thiago Macieira for reviewing the content!
| December 20, 2011 | |
Yeah, I admit that my semi-periodic updates about Intel Linux Graphics got more “seldom-periodic” than “truly-periodic” for the past weeks. But have no fear – they are back! And I am still on my self-appointed bi-weekly schedule estimate. This is what’s good about semi-periodic schedule – one never can run too much out of it
.
So starting with the coolest news – the Mesa team is getting close to the GL 3.0 milestone! Yeah, with latest GL_ext_transform_feedback patches from Paul Berry, the last major piece of GL 3.0 spec is getting into place. There are still some extensions missing and lots of smaller tasks to be done, but it is possible to say that we are almost there. I think that this is really exciting for both us, and for all the Linux and open-source users in the world – so yeah – we’ve been good boys and girls during the year and Santa Claus gift has materialized itself in form of almost-full GL 3.0 support in Mesa.
Who knows, maybe prior to the Chinese new year we’ll receive the 2nd part of this gift (in other words, mesa 8.0 release
).
On kernel side, the 3.2-rc6 release brought lots of awesome changes to our drivers. Yes, I am talking about everyone’s favorite rc6 and semaphores features. They are on by default on Ivy Bridge architecture, and are also enabled on Sandy Bridge if VTd is disabled. So most of you should enjoy greatly improved battery life, considerable faster performance and also enhanced stability within the i915 driver when Linux 3.2 will be released.
Besides those patches, work has started on collecting patches for the 3.3 merge window. Daniel Vetter sent his pending patches in a form of tiny 43-patches series. Those patches bring PPGTT support, improve debugfs handling, enhance pread/pwrite performance, fix swizzling for SNB/IVB, improve forcewake operations and enhance debugging support for cases when GPU rings get stuck.
Ben has also sent his patches for scheduling/throttling, but they haven’t received much interest except from myself and phoronix
. Those patches add support for more fine-grained GPU scheduling and rings load distribution between individual process. I am really interested in this work, and I hope that they will be accepted into the main kernel in the foreseeable future.
Also on kernel, Rodrigo Vivi and Paulo Zanoni sent out some patches which finally fix some corner cases for TV-out and SDVO outputs. This certainly should make many users happy out there just in time for Christmas.
And finally, for the kernel size, Chris Wilson came with a patch which works around the missed IRQs issues on Ivy Bridge platform. With this patch, and with semaphores being enabled on Ivy Bridge by default, I am very happy to say that we don’t have any blocking bugs for Ivy Bridge in our bugzilla. I think that it comes as a nice Christmas gift for all the users out there (the ones who already have an Ivy Bridge machine, and the ones who will get it by its launch – which is still 4 months away). Of course, I won’t talk much about it prior to its official launch, but trust me – Ivy Bridge rocks!. I can’t wait to have an Ultrabook based on this platform for myself…
Besides mesa and kernel, it is worth mentioning that on the 2D side, Zhigang added full Glamor support into the driver. The Glamor acceleration is still considered very experimental and non-stable, but now it is available for the world to take a peek on it and witness how it works with their own eyes.
So I think that this is pretty much it. We have hundreds of patches floating around for all the projects, thousands of emails and millions of users in the world – and we are working hard to make all of them happy with the results of our work. 2011 was extremely productive and rewarding for us – and I hope that the year of 11111011100 (a.k.a., 0x7DC or 2012_base10 for the ones still reading in decimal numbers out there
) will be even more interesting*!
See you!
(*) Assuming the world won’t end in a core dump caused by the Mayan millennium counter overflow bug
.
| December 19, 2011 | |
git://people.freedesktop.org/~whot/xserver git://people.freedesktop.org/~whot/inputproto git://people.freedesktop.org/~whot/xf86-input-evdev git://people.freedesktop.org/~whot/libXiHere's a screencast running Fedora 16 with the modified X server and a little multitouch event debugging application.
Last week, Brendan Gregg, who wrote the book on DTrace, published a new tool for visualizing hot sections of code by sampling the process stack every few milliseconds and then graphing which stacks were seen the most during the run: Flame Graphs. This looked neat, but my experience has been that I get the most understanding out of things like this by trying them out myself, so I did.
Fortunately, it's very easy to setup, as Brendan provided the tools as two standalone perl scripts you can download from the Flame Graph github repo. Then the next step is deciding what you want to run it on and capturing the data from a run of that.
It just so turns out that a few days before the Flame Graph release, another mail had crossed my inbox that gave me something to look at. Several years ago, I added some Userland Statically Defined Tracing probes to the Xserver code base, creating an Xserver provider for DTrace. These got integrated first to the Solaris Xservers (both Xsun & Xorg) and then merged upstream to the X.Org community release for Xserver 1.4, where they're available for all platforms with DTrace support.
Earlier this year, Adam Jackson enabled the dtrace hooks in the Fedora Xorg server package, using the SystemTap static probe support for DTrace probe compatibility. He then noticed a performance drop, which isn't supposed to happen, as DTrace probes are simply noops when not being actively traced, and submitted a fix for it upstream.
This was due to two of the probes I'd defined - when the Xserver processes a request from a client, there's a request-start probe just before the request is processed, and a request-done probe right after the request is processed. If you just want to see what requests a client is making you can trace either one, but if you want to measure the time taken to run a request, or determine if something is happening while a request is being processed, you need both of the probes. When they first got integrated, the code was simple:
The compiler sees XSERVER_REQUEST_START and XSERVER_REQUEST_DONE as simple function calls, so it does whatever work is necessary to set up their arguments and then calls them. Later, during the linking process, the actual call instructions are replaced with noops and the addresses recorded so that when a dtrace user enables the probe the call can be activated at that time. In these cases, that's not so bad, just a bunch of register access and memory loads of things that are going to be needed nearby. The one outlier is GetRequestName(MAJOROP) which looks like a function call, but was really just a macro that used the opcode as the index an array of strings and returned the string name for the opcode so that DTrace probes could see the request names, especially for extensions which don't have static opcode mappings. For that the compiler would just load a register with the address of the base of the array and then add the offset of the entry specified by MAJOROP in that array.
All was well and good for a bit, until a later project came along during the Xorg 1.5 development cycle to unify all the different lists of protocol object names in the Xserver, as there were different ones in use by the DTrace probes, the security extensions, and the resource management system. That replaced the simple array lookup macro with a function call. While the function doesn't do a lot more work, it does enough to be noticed, and thus the performance hit was taken in the hot path of request dispatching. Adam's patch to fix this simply uses is-enabled probes to only make those function calls when the probes are actually enabled. x11perf testing showed the win on a Athlon 64 3200+ test system running Solaris 11:
Before: 250000000 trep @ 0.0001 msec ( 8500000.0/sec): X protocol NoOperation After: 300000000 trep @ 0.0001 msec (10400000.0/sec): X protocol NoOperation
But numbers are boring, and this gave an excuse to try out Flame Graphs. To capture the data, I took advantage of synchronization features built into xinit and dtrace -c:
# xinit /usr/sbin/dtrace -x ustackframes=100 \
-n 'profile-997 /execname == "Xorg"/ { @[ustack()] = count(); }' \
-o out.user_stacks.noop -c "x11perf -noop" \
-- -config xorg.conf.dummy
To explain this command, I'll start at the end. For xinit, everything after the double dash is a set of arguments to the Xserver it starts, in this case, Xorg is told to look in the normal config paths for xorg.conf.dummy, which it would find is this simple config file in /etc/X11/xorg.conf.dummy setting the driver to the Xvfb-like “dummy” driver, which just uses RAM as a frame buffer to take graphics driver considerations out of this test:
Section "Device" Identifier "Card0" Driver "dummy" EndSectionSince I'm using a modern Xorg version, that's all the configuration needed, all the unspecified sections are autoconfigured. xinit starts the Xserver, waits for the Xserver to signal that it's finished its start up, and then runs the first half of the command line as a client with the DISPLAY set to the new Xserver. In this case it runs the dtrace command, which sets up the probes based on the examples in the Flame Graphs README, and then runs the command specified as the -c argument, the x11perf benchmark tool. When x11perf exits, dtrace stops the probes, generates its report, and then exits itself, which in turn causes xinit to shut down the X server and exit.
The resulting Flame Graphs are, in their native SVG interactive form:
You can see in the first one a bar showing a little over 10% of the time was in stacks involving LookupMajorName, which is completely gone in the second patch. Those who saw Adam's patch series come across the xorg-devel list last week may also notice the presence of XaceHook calls, which Adam optimized in another patch. Unfortunately, while I did build with that patch as well, we don't get the benefits of it since the XC-Security extension is on by default, and those fill in the hooks, so it can't just bypass them as it does when the hooks are empty.
I also took measurements of what Xorg did as gdm started up and a test user logged in, which produced the much larger flame graph you can see in SVG or PNG. As you can see the recursive calls in the font catalogue scanning functions make for some really tall flame graphs. You can also see that, to no one's surprise, xf86SlowBCopy is slow, and a large portion of the time is spent “blitting” bits from one place to another. Some potential areas for improvement stand out - like the 5.7% of time spent rescanning the font path because the Solaris gnome session startup scripts make xset fp calls to add the fonts for the current locale to the legacy font path for old clients that still use it, and another nearly 5% handling the ListFonts and ListFontsWithInfo calls, which dtracing with the request-start probes turned out to be the Java GUI for the Solaris Visual Panels gnome-panel applet.
Now because of the way the data for these is gathered, from looking at them alone you can't tell if a wide bar is one really long call to a function (as it is for the main() function bar in all these) or millions of individual calls (as it was for the ProcNoOperation calls in the x11perf -noop trace), but it does give a quick and easy way to pick out which functions the program is spending most of its time in, as a first pass for figuring out where to dig deeper for potential performance improvements.
Brendan has made these scripts easy to use to generate these graphs, so I encourage you to try them out as well on some sample runs to get familiar with them, so that when you really need them, you know what cases they're good for and how to capture the data and generate the graphs for yourself. Trying really is the best method of learning.
Ant::assigneddirection is initialized to false each turn). The high-level modules then simply assign directions to ants that do not have one yet, and the order in which the modules are called reflects the relative importance I assign to the various tasks. For example, the HillDefense module will only assign ants that have not been assigned by the FoodSeeker.Zoc ("zone of control") module does not steer any ants. Instead, it keeps track of how fast my ants vs. enemy ants can reach each square of the map. And the Tactical module overrides the previous decisions if necessary for ants that may be involved in combat.FoodSeeker: Getting enough food is probably the most important problem of the game, and so this module comes first. It greedily sends the closest ant to each item of food that is visible, using several breadth-first-searches.HillDefense: When one of the bot's hills can be reached in few turns by the enemy, this module ensures that a few ants (adjusted based on the number of enemy ants in sight) stick close to the hill.OpportunisticAttack: This rather stupid piece of code ensures that the ants move more aggressively towards enemy hills. After all, that is the only way to win the game.Scout: This module assigns a tiny number of ants to re-explore squares that have not been seen in a long time.Zoc module to understand that an enemy can never come out of a cul-de-sac once it's been secured. So without some re-scouting logic, the bot would simply ignore the food in those secured locations!Diffusion: This is a very ad-hoc heuristic to spread out my ants better than they would otherwise. There would probably have been some potential for improvement in this part of the code.Zoc data to move closer to the enemy. This is done in Bot::makeMoves rather than in a separate module.FoodSeeker and the Zoc-based movement. Together with Tactical, this was enough to get into the top 100 a bit more than two weeks before the deadline.TacticalSm module is simple:Submaps that contain all the ants that could potentially be involved in combat on the next turn.Submap and associated PlayerMoves is called a Theater.aggressive mode if the own ants overpower the enemy. This switch is done randomly based on the ratio of ants, where the probability to turn on aggressive mode is conceptually a logistic function in the log of the ants ratio.test/tune.py script), but somehow that didn't yield very convincing results either.std::vector::clear does not free the allocated memory, at least in the default implementation of g++. By keeping those vectors around, I want to avoid unpleasant surprises in the performance of memory management.N. I pick a random prime p out of a decently sized pre-generated list of large primes (larger than any N the code is ever going to see), as well as a random offset ofs and loop through the numbers 0..N-1. But instead of using these numbers i as an index, I use (p*i + ofs) % N as index. Since p is prime different from N, it is invertible modulo N, and therefore the map i -> p*i + ofs is a bijection, aka a permutation. Of course, this is far from a uniformly distributed permutation: there are N! potential permutations, out of which this method can generate at most N * φ(N). But hey: it's good enough for what I need. | December 16, 2011 | |
Like I already wrote here last week, I've been heavily working on OpenStack for the last weeks.
My first assignment was to package OpenStack for Debian. The packages already present in unstable were mainly done by Thomas Goirand, who based its work on the one done in Ubuntu. Therefore, the packages where not in a very good shape for Debian.
Today Ghe Rivero and I (members of the OpenStack Debian packaging team) managed to push the OpenStack Essex 2 milestone into unstable with great success. You can now test and deploy OpenStack Essex 2 very easily!
Packaging OpenStack made me write several patches, mainly related to packaging, patches which were all accepted and merged by upstream. This is nice because most of the OpenStack Debian packages lost their debian/patches directories now!
Finally, I've finished to implement one blueprint I really missed: the ability to boot from an ISO image using libvirt. The code still needs a review, but it should be included in the Essex 3 milestone if everything's right.
| December 13, 2011 | |
We are trying to put together a hackfest to help developers using GStreamer to port their applications to GStreamer 1.0. The GStreamer Hackfest/Code Sprint 2012 will take place in Malaga, Spain, between 25th to 27th of January (3 full days). I would ask all developers out there interested in attending to add your name to the wiki as quickly as possible, just so we can estimate interest. If you are interested, but don’t know for sure if you can make it there, it would still be good if you added your name with maybe a comment mentioning that you need to verify before you are certain.
We will be offering some travel sponsorship, so even if you are short on cash we hope to have you attend, more details on the hackfest&codesprint wiki page.
For those wondering why we choose Malaga, well the reason is that Wim Taymans, the GStreamer 1.0 designer and lead developer lives there, and at a previous hackfest we tried to do in Oslo he ended up in Helsinki instead. So this time we are taking no risks, but instead we are taking the hackfest to him
| December 11, 2011 | |
Last week I went to the first in a new series of events called "In the Dock" -- Harvard Law professor Lawrence Lessig interviewed Jack Abramoff on the topic of Abramoff's illegal lobbying in specific, and the state of corruption in US politics in general. I took some photos (CC-BY-SA 3.0), you can watch a video of the talk here, and my friend Ben Schwartz also has a write-up including some great quotes.
I went in willing to detest what Abramoff had done, and him by extension -- and it's not that I ended up liking or trusting him, or that I was unexpecting his "Look, you put a few guys in jail, it doesn't mean the problem's been solved and everything's okay now" argument for why assigning a disproportionate amount of blame to him is unproductive, but he did manage to convince me that he's uniquely placed to be an ally to help fix the system; that he's someone who knows all of its intricacies and is sincere about achieving a sense of redemption by working on stopping other people from doing exactly what he did.
His argument for how his book tour isn't a cynical attempt to make money is convincing, too: to the small extent that he's able to make money from books and speaking, he's forced to use it to repay a $40m restitution order to the government and his victims.
The most surprising thing I learned was that the crimes he went to jail for were not the particularly objectionable democracy-perverting forms of corruption that we ascribe to him -- those are totally legal (even more so than before, thanks to Citizens United) and still happening today across Washington's 30,000 lobbyists -- but instead mostly unrelated charges, like mail fraud. He thinks that the only way to stop bribery in Congress is to ban political contributions from anyone who stands to benefit from public funds (which Lessig criticized as being far too ambiguous and broad: who doesn't stand to benefit from public funds?), and to ban lawmakers and their staff from later working for lobbyists for the rest of their lives. He described how, before his downfall, he would agree to hire a lawmaker's staffers later while they were still working for the lawmaker, and would then have control over them from that point onwards, even though no money had changed hands -- not only is this movement from being congressional staff to becoming a lobbyist still legal, it's daily routine.
I hope this talk series continues. I can't think of many other examples of powerful figures being brave enough to open themselves up and engage in an extended ethical (rather than legal or technical) critique and cross-examination by their peers and the public, and it was powerful to watch and learn from.
(This is reposted from my Google+ stream.)
| December 09, 2011 | |
| December 07, 2011 | |
It has been a while since I blogged but I've been very busy, with my new job and this new blog!
I quitted my job last September, and found another one that I started in October. I'm now the lead developer of eNovance Labs, where I work on the OpenStack project. So far, this allowed me to contribute heavily to the Debian packaging of OpenStack.
In the meantime, I took some time to redesign my personal homepage and this blog, which is now using Hyde, the Python equivalent of Jekyll, which is in Ruby. Since I dislike Ruby (sorry), I preferred to use a Python based generator, and I admit Hyde is really cool. Since I really suck at Web design, this one is obviously based on Twitter's bootstrap
I am desperately looking for someone familiar with Malaga, Spain, maybe a local LUG or similar. The reason is that we are interested in doing a GStreamer 1.0 application porting hackfest there in late January. So getting hold of someone with local knowledge who would be able to help us find a suitable venue would be great. Please get in touch with me on christian.schaller(at)gmail.com if you are in the area and willing to help or know someone would probably could help us.
Inspired by a discussion in #wayland today, here are snippets from three people explaining why X declares its DPI as 96, and why a single 'DPI' sledgehammer isn't actually what basically anyone* wants. Please read them. Thanks.
I am clearly going to have to explain this one more time, forever. Let's see if I can't write it authoritatively once and simply answer with a URL from here out.
But what about the single monitor case? Let's go back to your Vaio. It's got a high DPI screen, so let's adjust to that. Now you're happy. Right up until you plug in an external monitor and now when you run any applications on the external display your fonts are twice the size they should be. WOOHOO GO TEAM of course that won't make us look like amateurs at all. So you need another heuristic to handle that, and of course "heuristic" is an ancient african word meaning "maybe bonghits will make this problem more tractable".
People who know a bit of typography may know a few factoids:
- Printed books generally use fonts which can be from about 9 to about 12 points in size.
- A point is roughly 1/72 of an inch. For people in civilized countries, this translates to "I have no idea what the fuck a quarter pounder is".
*: Yes, I know you need to have actual point equivalence, and you've had all your displays and printer colour-calibrated for the past ten years too. You're doing all this in the GIMP or some other kind of design tool, so please yell at them to use the display size information that XRandR gives you right now, already, today.
| December 06, 2011 | |
So I did a bit of work last week to convert the Collabora website to HTML5. The actual porting was quite simple, mostly replacing the DOCTYPE tag to the new HTML5 one. Found a few other issues through the W3C validator, but nothing major. Today I took the next (small) step in the process by actual adding some real HTML5 content to the site. Actually I only sort of did. Instead of hosting the video locally and using the new video tag I ended up uploading it to youtube and embedding the WebM video in our page. The small video clip I added is demonstrating the HTML5 video editing demo we released recently. (For those of you who missed it I recommend the HTML5 video editing blog entry by Mateu Batle, which explains the whole thing in detail and also links to the code).
The demo, while simple, is quite cool, showing off our HTML5 based touch screen interface all built on top of Webkit and the GStreamer Editing Services. The embedded video is to be found on the GStreamer Editing Services page.
As a sidenote, to make this I actually relied on the GNOME 3 built in screen capture support, which I have to say worked like a charm
Always felt screen casting to be a pain before, but this time it worked very well for me.
As many of you have already seen, we have just released a new version of the Intel Linux Graphics stack, composed by Kernel 3.1 (or 3.1.x in practice), Mesa 7.11.2, Libdrm 2.4.27, vaapi and vaapi-driver-intel 1.0.15 and xf86-video-intel 2.17.
The focus of this release was on performance and stability improvements in the drivers, and I think we managed to get the job done according to what we expected. Thanks to LLC caching and improvements in both Kernel and Mesa, 3D performance was improved by a large margin on Sandy Bridge generation of GPUs onwards, and many annoying issues were fixed as well (with some of the fixes already queued for Kernel 3.2 release). And I’d also like to highlight lots of work towards GLES compliance on Sandy Bridge architecture, and many and many different stability fixes in all the projects.
For statistical purposes, my saved bugzilla searches count 253 bug reports which were closed after the previous release, and we had 594 bug tickets in total which had any activity since that release. Of course, this does not represents what was done by an accurate number, so please, just use those numbers as some sort of bugzilla pondering references.
So to sum it all up, I’d like to thank all the developers, QA, users, testers, and all the community we have around Intel Linux Graphics project for your amazing work. Yes, the Intel Linux Graphics project has not only a team within Intel – but also a huge community and users all around the world, following the Open-Source road from Kernel up to the UI components. Thank you all guys!
Enjoy!
| December 05, 2011 | |
After the success of the GStreamer interview with Wim Taymans I thought we follow up with another great interview with a Collabora developer.
This time we are talking with Youness Alaoui who is one of the maintainers of Farstream, the audio and video conferencing framework built on top of GStreamer. We also cover another of Youness Alaoui projects, libnice, the NAT traversal library. So if you want to know what is happening with audio and video conferencing on Linux be sure to read the full interview with Youness Alaoui here.
| December 04, 2011 | |



[read more]
| December 03, 2011 | |
| December 02, 2011 | |
extern void foo(struct foo *f,
Bool check_device,
int max_devices,
Bool emulate);
foo(mystruct, TRUE, 0, FALSE);
foo(mystruct, !dev->invalid, 0, list_is_first_entry(dev->list));
Bool check_device = !dev->invalid;
Bool emulate_pointer = list_is_first_entry(dev->list));
foo(mystruct, check_device, 0, emulate_pointer);
| December 01, 2011 | |
Development goes on, on all the fronts, and time has come for some news about Intel Linux Graphics project.
For the Kernel side, some nice patch series have arrived to the list:
Moving up the stack to the 3D driver, some major news worth highlighting are:
And finally, on the xserver side, discussions were raised on the mailing list on the release dates for the next xserver release, and the Xinput 2.2 state in it. It is almost there, and will probably get merged until Christmas. Also, Xorg-server 1.11.2.901 (a.k.a., 1.11.3-RC2) was released.
Oops, only three blog posts in the last year. I've mostly been posting over at Google+, wherein I met a bunch of photographers and picked up a fledgling photography obsession of my own. I'll try to write a "what I did in the last year" blog post at some point.
This post isn't about that, though -- like last year, this year Madeleine and I are again donating N% of our joint pre-tax income to effective charities for each year that we've been married, and this year N is equal to 6. Mad has a post outlining why we're doing this and which groups she's chosen, and here's a writeup of who I decided to donate my half of the 6% to:
40% to Schistosomiasis Control Initiative
It's very common to think about international aid in terms of "lives saved", but it makes more sense to talk about something like "number of disability-adjusted life years increased" (DALYs). GiveWell thinks that this charity -- which concentrates on the "Neglected Tropical Diseases" which are usually worms/parasites -- offers an extremely effective intervention at improving DALYs; because these infections are readily treatable using very inexpensive drugs, yet often come with debilitating symptoms that don't quite kill the "host".
25% to Give Directly
Give Directly is a fascinating project. The charity simply finds the poorest people in an area (currently they're working in Kenya) and transfers money to them via mobile phone. This leaves the charity itself with very little overhead -- all the charity has to do is identify who the poorest people are, which they often do by looking at what kind of place they live in. The claim is that this outperforms many other attempts at aid; there's nowhere for the effect of the money to get diluted or misappropriated along the way.
I should be clear that I don't think this is the best possible aid intervention. But, as GiveWell points out, it should be the intervention that we treat as the baseline that other interventions are measured against -- if you think you have a better idea, then you should be able to prove it by comparing outcomes against this method. Give Directly has a commitment to measuring the quantitative effects of its work; I want to support finding out how well this intervention works, even though the optimist in me hopes we can do much better!
15% to GiveWell
GiveWell has dramatically changed how I think about and evaluate charitable giving. This year I've been pleased to see them doing things like exposing errors in commonly referenced DALY calculations, and generally acting as the quantitative sanity-checker for development charities.
15% to the Tor Project
Tor is a technology that helps its users achieve anonymous access to the Internet over a connection that may be being monitored; as a side-effect of this, it allows its users to get around filtering of their connections. I think this pairs up nicely with Madeleine's choice of donating to the Wikimedia Foundation -- it's important to have the world's knowledge available to everyone, not just the people who are lucky enough to have an unfiltered and unmonitored connection. I increased my donation to Tor this year after seeing how effective the Internet has been as a pro-democracy tool this year, and how many regimes tried to filter communication using it when it was being used by citizens to coordinate with each other.
5% to the EFF
While Tor works on "exporting" the Internet that we use to regimes that wish to block or filter it, the EFF is helping to keep the network itself safe from becoming controlled by groups like governments or media companies; attempting to preserve the freedoms that the net provides today.
| November 30, 2011 | |
| planet.fd.o | ||
|
planet.freedesktop.org is powered by Venus,
and the freedesktop.org community.
|
||
| Planetarium | ||
|
Planet Debian Planet Fedora Planet Gentoo Planet GNOME Planet GStreamer Planet IM Planet Jabber Planet KDE Planet Kernel Planet Mozilla Planet Document Foundation Planet SuSE Planet Xiph |
||