In this week’s article for my ongoing Google Summer of Code (GSoC) project I planned on writing about the basic idea behind the project, but I reconsidered and decided to first give an overview on how Xwayland functions on a high-level and in the next week take a look at its inner workings in detail. The reason for that is, that there is not much Xwayland documentation available right now. So these two articles are meant to fill this void in order to give interested beginners a helping hand. And in two weeks I’ll catch up on explaining the project’s idea.

As we go high level this week the first question is, what is Xwayland supposed to achieve at all? You may know this. It’s something in a Wayland session ensuring that applications, which don’t support Wayland but only the old Xserver still function normally, i.e. it ensures backwards compatibility. But how does it do this? Before we go into this, there is one more thing to talk about, since I called Xwayland only something before. What is Xwayland exactly? How does it look to you on your Linux system? We’ll see in the next week that it’s not as easy to answer as the following simple explanation makes it appear, but for now this is enough: It’s a single binary containing an Xserver with a special backend written to communicate with the Wayland compositor active on your system - for example with KWin in a Plasma Wayland session.

To make it more tangible let’s take a look at Debian: There is a package called Xwayland and it consists of basically only the aforementioned binary file. This binary gets copied to /usr/bin/Xwayland. Compare this to the normal Xserver provided by X.org, which in Debian you can find in the package xserver-xorg-core. The respective binary gets put into /usr/bin/Xorg together with a symlink /usr/bin/X pointing to it.

While the latter is the central building block in an X session and therefore gets launched before anything else with graphical output, the Xserver in the Xwayland binary works differently: It is embedded in a Wayland session. And in a Wayland session the Wayland compositor is the central building block. This means in particular that the Wayland compositor also takes up the role of being the server, who talks to Wayland native applications with graphical output as its clients. They send request to it in order to present their painted stuff on the screen. The Xserver in the Xwayland binary is only a necessary link between applications, which are only able to speak to an Xserver, and the Wayland compositor/server. Therefore the Xwayland binary gets launched later on by the compositor or some other process in the workspace. In Plasma it’s launched by KWin after the compositor has initialized the rendering pipeline. You find the relevant code here.

Although in this case KWin also establishes some communication channels with the newly created Xwayland process, in general the communication between Xwayland and a Wayland server is done by the normal Wayland protocoll in the same way other native Wayland applications talk to the compositor/server. This means the windows requested by possibly several X based applications and provided by Xwayland acting as an Xserver are translated at the same time by Xwayland to Wayland compatible objects and, acting as a native Wayland client, send to the Wayland compositor via the Wayland protocol. These windows look to the Wayland compositor just like the windows - in Wayland terminology surfaces - of every other Wayland native application. When reading this keep in mind, that an application in Wayland is not limited to using only one window/surface but can create multiple at the same time, so Xwayland as a native Wayland client can do the same for all the windows created for all of its X clients.

In the second part next week we’ll have a close look at the Xwayland code to see how Xwayland fills its role as an Xserver in regards to its X based clients and at the same time acts as a Wayland client when facing the Wayland compositor.

Felt it been to long since I did another Fedora Workstation update. We spend a lot of time trying to figure out how we can best spend our resources to produce the best desktop possible for our users, because even though Red Hat invests more into the Linux desktop than any other company by quite a margin, our resources are still far from limitless. So we have a continuous effort of asking ourselves if each of the areas we are investing in are the right ones that give our users the things they need the most, so below is a sampling of the things we are working on.

Improving integration of the NVidia binary driver
This has been ongoing for quite a while, but things have started to land now. Hans de Goede and Simone Caronni has been collaboring, building on the work by NVidia and Adam Jackson around glvnd. So if you set up Simones NVidia repository hosted on negativo17 you will be able to install the Nvidia driver without any conflicts with the Mesa stack and due to Hans work you should be fairly sure that even if the NVidia driver stops working with a given kernel update you will smoothly transition back to the open source Nouveau driver. I been testing it on my own Lenovo P70 system for the last week and it seems to work well under X. That said once you install the binary NVidia driver that is what your running on, which is of course not the behaviour you want from a hybrid graphics system. Fixing that last issue requires further collaboration between us and NVidia.
Related to this Adam Jackson is currently working on a project he calls glxmux. glxmux will allow you to have more than one GLX implementation on the system, so that you can switch between Mesa GLX for the Intel integrated graphics card and NVidia GLX for the binary driver. While we can make no promises we hope to have the framework in place for Fedora Workstation 27. Having that in place should allow us to create a solution where you only use the NVidia driver when you want the extra graphics power which will of course require significant work from Nvidia to enable it on their side so I can’t give a definite timeline for when all the puzzle pieces are in place. Just be assured we are working on it and talking regularly to NVidia about it. I will let you know here as soon as things come together.

On the Wayland side the Jonas Ådahl is working on putting the final touches on Hybrid Graphics support

Fleet Commander ready for take-off
Another major project we been working on for a long time in Fleet Commander. Fleet Commander is a tool to allow you to manage Fedora and RHEL desktops centrally. This is a tool targeted at for instance Universities or companies with tens, hundreds or thousands of workstation installation. It gives you a graphical browser based UI (accessible through Cockpit) to create configuration profiles and deploy across your organization. Currently it allows you to control anything that has a gsetting associated with it like enabling/disabling extensions and setting configuration settings in GTK+ and GNOME applications. It allows you to configure Network Manager settings so if your updating the company VPN or proxy settings you can easily push those changes out to all user in the organization. Or quickly migrate Evolution email settings to a new email server. The tool also allows you to control recommended applications in the Software Center and set bookmarks in Firefox. There is also support for controlling settings inside LibreOffice.

All this features can be set and controlled on either a user level or a group level or organization wide due to the close integration we have with FreeIPA suite of tools. The data is stored inside your organizations LDAP server alongside other user information so you don’t need to have the clients connect to a new service for this, and while it is not there in this initial release we will in the future also support Active Directory.

The initial release and Fleet Commander website will be out alongside Fedora Workstation 26.

PipeWire
I talked about PipeWire before, when it was still called Pinos, but the scope and ambition for the project has significantly changed since then. Last time when I spoke about it the goal was just to create something that could be considered a video equivalent of pulseaudio. Wim Taymans, who you might know co-created GStreamer and who has been a major PulseAudio contributor, has since expanded the scope and PipeWire now aims at unifying linux Audio and Video. The long term the goal is for PipeWire to not only provide handling of video streams, but also handle all kinds of audio. Due to this Wim has been spending a lot of time making sure PipeWire can handle audio in a way that not only address the PulseAudio usecases, but also the ones handled by Jack today. A big part of the motivation for this is that we want to make Fedora Workstation the best place to create content and we want the pro-audio crowd to be first class citizens of our desktop.

At the same time we don’t want to make this another painful subsystem transition so PipeWire so we will need to ensure that PulseAudio applications can still be run without modification.

We expect to start shipping PipeWire with Fedora Workstation 27, but at that point only have it handle video as we need this to both enable good video handling for Flatpak applications through a video portal, but also to provide an API for applications that want to do screen capture under Wayland, like web browser applications offering screen sharing. We will the bring the audio features onboard in subsequent releases as we also try to work with the Jack and PulseAudio communities to make this a joint effort. We are also working now on a proper website for PipeWire.

Red Hat developer integration
A feature we are quite excited about is the integration of support for the Red Hat developer account system into Fedora. This means that you should be able to create a Red Hat developer account through GNOME Online accounts and once you have that account set up you should be able to easily create Red Hat Enterprise Linux virtual machines or containers on your Fedora system. This is a crucial piece for the developer focus that we want the workstation to have and one that we think will make a lot of developers life easier. We where originally hoping to have this ready for Fedora Workstaton 26, but atm it looks more likely to hit Fedora Workstation 27, but we will keep you up to date as this progresses.

Fractional scaling for HiDPI systems
Fedora Workstation has been leading the charge in supporting HiDPI on Linux and we hope to build on that with the current work to enable fractional scaling support. Since we introduced HiDPI support we have been improving it step by step, for instance last year we introduced support for dealing with different DPI levels per monitor for Wayland applications. The fractional scaling work will take this a step further. The biggest problem it will resolve is that for certain monitor sizes the current scaling options either left things to small or to big. With the fractional scaling support we will introduce intermediate steps, so that you can scale your interface 1.5x times instead of having to go all the way to 2. The set of technologies we are developing for handling fractional scaling should also allow us to provide better scaling for XWayland applications as it provides us with methods for scaling that doesn’t need direct support from the windowing system or toolkit.

GNOME Shell performance
Carlos Garnacho has been doing some great work recently improving the general performance of GNOME Shell. This comes on top of his earlier performance work that was very well received. How fast/slow GNOME shell is often a subjective thing, but reducing overhead where we can is never a bad thing.

Flatpak building
Owen Taylor has been working hard on putting the pieces in place for start large scale Flatpak building in Fedora. You might see a couple of test flatpaks appear in a Fedora Workstation 26 timeframe, but the goal is to have a huge Flatpak catalog ready in time for Fedora Workstation 27. Essentially what we are doing is making it very simple for a Fedora maintainer to build a Flatpak of the application they maintain through the Fedora package building infrastructure and push that Flatpak into a central Flatpak registry. And while of course this is mainly meant to be to the benefit of Fedora users there is of course nothing stopping other distributions from offering these Flatpak packaged applications to their users also.

Atomic Workstation
Another effort that is marching forward is what we call Atomic Workstation. The idea here is to have an immutable OS image kinda like what you see on for instance Android devices. The advantage to this is that the core of the operating system gets tested and deployed as a unit and the chance of users ending with broken systems decrease significantly as we don’t need to rely on packages getting applied in the correct order or scripts executing as expected on each individual workstation out there. This effort is largely based on the Project Atomic effort, and the end goal here is to have a image based OS install and Flatpak based applications on top of it. If you are very adventerous and/or want to help out with this effort you can get the ISO image installer for Atomic Workstation here.

Firmware handling
Our Linux Firmware project is still going strong with new features being added and new vendors signing on. As Richard Hughes recently blogged about the latest vendor joining the effort is Logitech who now will upload their firmware into the service so that you can keep your Logitech peripherals updated through it. It is worthwhile pointing out here how we worked with Logitech to make this happen, with Richard working on the special tooling needed and thus reducing the threshold for Logitech to start offering their firmware through the service. We have other vendors we are having similar discussions and collaborations with so expect to see more. At this point I tend to recommend people get a Dell to run Linux, due to their strong support for efforts such as the Linux Firmware Service, but other major vendors are in the final stages of testing so expect more major vendors starting to push firmware updates soon.

High Dynamic Range
The next big thing in the display technology field is HDR (High Dynamic Range). HDR allows for deeper more vibrant colours and is a feature seen on a lot of new TVs these days and game consoles like the Playstation 4 support it. Computer monitors are appearing on the market too now with this feature, for instance the Dell UP2718Q. We want to ensure Fedora and Linux is a leader here, for the benefit of video and graphics artists using Fedora and Red Hat Enterprise Linux. We are thus kicking of an effort to make sure this technology mature as quickly as possible and be fully supported. We are not the only ones interested in this so we will hopefully be collaborating with our friends at Intel, AMD and NVidia on this. We hope to have the first monitors delivered to our office within a few weeks.

Codecs
While playback these days have moved to streaming where locally installed codecs are of less importance for the consumption usecase, having a wide selection of codecs available is still important for media editing and creation usecases, so we want you to be able to load a varity of old media files into you video editor for instance. Luckily we are at a crossroads now where a lot of widely used codecs have their essential patents expire (mp3, ac3 and more) while at the same time the industry focus seems to have moved to royalty free codec development moving forward (Opus, VP9, Alliance for Open Media). We have been spending a lot of time with the Red Hat legal team trying to clear these codecs, which resulted in mp3 and AC3 now shipping in Fedora Workstation. We have more codecs on the way though, so this effort is in no way over. My goal is that over the course of this year the situation of software patents being a huge issue when dealing with audio and video codecs on Linux will be considered a thing of the past. I would like to thank the Red Hat legal team for their support on this issue as they have had to spend significant time on it as a big company like Red Hat do need to do our own due diligence when it comes to these things, we can’t just trust statements from random people on the internet that these codecs are now free to ship.

Battery life
We been looking at this for a while now and hope to be able to start sharing information with users on which laptops they should get that will have good battery life under Fedora. Christian Kellner is now our point man on battery life and he has taken up improving the Battery Bench tool that Owen Taylor wrote some time ago.

QtGNOME platform
We will have a new version of the QtGNOME platform in Fedora 26. For those of you who have not yet heard of this effort it is a set of themes and tools to ensure that Qt applications runs without any major issues under GNOME 3. With the new version the theming expands to include the accessibility and dark themes in Adwaita, meaning that if you switch to one of these themes under GNOME shell it will also switch your Qt applications over. We are also making sure things like cut’n paste and drag and drop works well. The version in Fedora Workstation 26 is a big step forward for this effort and should hopefully make Qt applications be first class citizens under your Fedora Workstation desktop.

Wayland polish
Ever since we switched the default to Wayland we have kept the pressure up and kept fixing bugs and finding solutions for corner cases. The result should be an improved Wayland experience in Fedora Workstation 26. A big thanks for Olivier Fourdan, Jonas Ådahl and the whole Wayland community for their continued efforts here. Two major items Jonas is working on for instance is improving fractional scaling, to ensure that your desktop scales to an optimal size on HiDPI displays of various sizes. What we currently have is limited to 1x or 2x, which is either to small or to big for some screens, but with this work you can also do 1.5x scaling. He is also working on preparing an API that will allow screen sharing under Wayland, so that for instance sharing your slides over video conferencing can work under Wayland.

Introducing casync

In the past months I have been working on a new project: casync. casync takes inspiration from the popular rsync file synchronization tool as well as the probably even more popular git revision control system. It combines the idea of the rsync algorithm with the idea of git-style content-addressable file systems, and creates a new system for efficiently storing and delivering file system images, optimized for high-frequency update cycles over the Internet. Its current focus is on delivering IoT, container, VM, application, portable service or OS images, but I hope to extend it later in a generic fashion to become useful for backups and home directory synchronization as well (but more about that later).

The basic technological building blocks casync is built from are neither new nor particularly innovative (at least not anymore), however the way casync combines them is different from existing tools, and that's what makes it useful for a variety of use-cases that other tools can't cover that well.

Why?

I created casync after studying how today's popular tools store and deliver file system images. To briefly name a few: Docker has a layered tarball approach, OSTree serves the individual files directly via HTTP and maintains packed deltas to speed up updates, while other systems operate on the block layer and place raw squashfs images (or other archival file systems, such as IS09660) for download on HTTP shares (in the better cases combined with zsync data).

Neither of these approaches appeared fully convincing to me when used in high-frequency update cycle systems. In such systems, it is important to optimize towards a couple of goals:

1. Most importantly, make updates cheap traffic-wise (for this most tools use image deltas of some form)
2. Put boundaries on disk space usage on servers (keeping deltas between all version combinations clients might want to run updates between, would suggest keeping an exponentially growing amount of deltas on servers)
3. Put boundaries on disk space usage on clients
4. Be friendly to Content Delivery Networks (CDNs), i.e. serve neither too many small nor too many overly large files, and only require the most basic form of HTTP. Provide the repository administrator with high-level knobs to tune the average file size delivered.
5. Simplicity to use for users, repository administrators and developers

I don't think any of the tools mentioned above are really good on more than a small subset of these points.

Specifically: Docker's layered tarball approach dumps the "delta" question onto the feet of the image creators: the best way to make your image downloads minimal is basing your work on an existing image clients might already have, and inherit its resources, maintaining full history. Here, revision control (a tool for the developer) is intermingled with update management (a concept for optimizing production delivery). As container histories grow individual deltas are likely to stay small, but on the other hand a brand-new deployment usually requires downloading the full history onto the deployment system, even though there's no use for it there, and likely requires substantially more disk space and download sizes.

OSTree's serving of individual files is unfriendly to CDNs (as many small files in file trees cause an explosion of HTTP GET requests). To counter that OSTree supports placing pre-calculated delta images between selected revisions on the delivery servers, which means a certain amount of revision management, that leaks into the clients.

Delivering direct squashfs (or other file system) images is almost beautifully simple, but of course means every update requires a full download of the newest image, which is both bad for disk usage and generated traffic. Enhancing it with zsync makes this a much better option, as it can reduce generated traffic substantially at very little cost of history/meta-data (no explicit deltas between a large number of versions need to be prepared server side). On the other hand server requirements in disk space and functionality (HTTP Range requests) are minus points for the use-case I am interested in.

(Note: all the mentioned systems have great properties, and it's not my intention to badmouth them. They only point I am trying to make is that for the use case I care about — file system image delivery with high high frequency update-cycles — each system comes with certain drawbacks.)

Security & Reproducibility

Besides the issues pointed out above I wasn't happy with the security and reproducibility properties of these systems. In today's world where security breaches involving hacking and breaking into connected systems happen every day, an image delivery system that cannot make strong guarantees regarding data integrity is out of date. Specifically, the tarball format is famously nondeterministic: the very same file tree can result in any number of different valid serializations depending on the tool used, its version and the underlying OS and file system. Some tar implementations attempt to correct that by guaranteeing that each file tree maps to exactly one valid serialization, but such a property is always only specific to the tool used. I strongly believe that any good update system must guarantee on every single link of the chain that there's only one valid representation of the data to deliver, that can easily be verified.

What casync Is

So much about the background why I created casync. Now, let's have a look what casync actually is like, and what it does. Here's the brief technical overview:

Encoding: Let's take a large linear data stream, split it into variable-sized chunks (the size of each being a function of the chunk's contents), and store these chunks in individual, compressed files in some directory, each file named after a strong hash value of its contents, so that the hash value may be used to as key for retrieving the full chunk data. Let's call this directory a "chunk store". At the same time, generate a "chunk index" file that lists these chunk hash values plus their respective chunk sizes in a simple linear array. The chunking algorithm is supposed to create variable, but similarly sized chunks from the data stream, and do so in a way that the same data results in the same chunks even if placed at varying offsets. For more information see this blog story.

Decoding: Let's take the chunk index file, and reassemble the large linear data stream by concatenating the uncompressed chunks retrieved from the chunk store, keyed by the listed chunk hash values.

As an extra twist, we introduce a well-defined, reproducible, random-access serialization format for file trees (think: a more modern tar), to permit efficient, stable storage of complete file trees in the system, simply by serializing them and then passing them into the encoding step explained above.

Finally, let's put all this on the network: for each image you want to deliver, generate a chunk index file and place it on an HTTP server. Do the same with the chunk store, and share it between the various index files you intend to deliver.

Why bother with all of this? Streams with similar contents will result in mostly the same chunk files in the chunk store. This means it is very efficient to store many related versions of a data stream in the same chunk store, thus minimizing disk usage. Moreover, when transferring linear data streams chunks already known on the receiving side can be made use of, thus minimizing network traffic.

Why is this different from rsync or OSTree, or similar tools? Well, one major difference between casync and those tools is that we remove file boundaries before chunking things up. This means that small files are lumped together with their siblings and large files are chopped into pieces, which permits us to recognize similarities in files and directories beyond file boundaries, and makes sure our chunk sizes are pretty evenly distributed, without the file boundaries affecting them.

The "chunking" algorithm is based on a the buzhash rolling hash function. SHA256 is used as strong hash function to generate digests of the chunks. xz is used to compress the individual chunks.

Here's a diagram, hopefully explaining a bit how the encoding process works, wasn't it for my crappy drawing skills:

The diagram shows the encoding process from top to bottom. It starts with a block device or a file tree, which is then serialized and chunked up into variable sized blocks. The compressed chunks are then placed in the chunk store, while a chunk index file is written listing the chunk hashes in order. (The original SVG of this graphic may be found here.)

Details

Note that casync operates on two different layers, depending on the use-case of the user:

1. You may use it on the block layer. In this case the raw block data on disk is taken as-is, read directly from the block device, split into chunks as described above, compressed, stored and delivered.

2. You may use it on the file system layer. In this case, the file tree serialization format mentioned above comes into play: the file tree is serialized depth-first (much like tar would do it) and then split into chunks, compressed, stored and delivered.

The fact that it may be used on both the block and file system layer opens it up for a variety of different use-cases. In the VM and IoT ecosystems shipping images as block-level serializations is more common, while in the container and application world file-system-level serializations are more typically used.

Chunk index files referring to block-layer serializations carry the .caibx suffix, while chunk index files referring to file system serializations carry the .caidx suffix. Note that you may also use casync as direct tar replacement, i.e. without the chunking, just generating the plain linear file tree serialization. Such files carry the .catar suffix. Internally .caibx are identical to .caidx files, the only difference is semantical: .caidx files describe a .catar file, while .caibx files may describe any other blob. Finally, chunk stores are directories carrying the .castr suffix.

Features

Here are a couple of other features casync has:

1. When downloading a new image you may use casync's --seed= feature: each block device, file, or directory specified is processed using the same chunking logic described above, and is used as preferred source when putting together the downloaded image locally, avoiding network transfer of it. This of course is useful whenever updating an image: simply specify one or more old versions as seed and only download the chunks that truly changed since then. Note that using seeds requires no history relationship between seed and the new image to download. This has major benefits: you can even use it to speed up downloads of relatively foreign and unrelated data. For example, when downloading a container image built using Ubuntu you can use your Fedora host OS tree in /usr as seed, and casync will automatically use whatever it can from that tree, for example timezone and locale data that tends to be identical between distributions. Example: casync extract http://example.com/myimage.caibx --seed=/dev/sda1 /dev/sda2. This will place the block-layer image described by the indicated URL in the /dev/sda2 partition, using the existing /dev/sda1 data as seeding source. An invocation like this could be typically used by IoT systems with an A/B partition setup. Example 2: casync extract http://example.com/mycontainer-v3.caidx --seed=/srv/container-v1 --seed=/srv/container-v2 /src/container-v3, is very similar but operates on the file system layer, and uses two old container versions to seed the new version.

2. When operating on the file system level, the user has fine-grained control on the meta-data included in the serialization. This is relevant since different use-cases tend to require a different set of saved/restored meta-data. For example, when shipping OS images, file access bits/ACLs and ownership matter, while file modification times hurt. When doing personal backups OTOH file ownership matters little but file modification times are important. Moreover different backing file systems support different feature sets, and storing more information than necessary might make it impossible to validate a tree against an image if the meta-data cannot be replayed in full. Due to this, casync provides a set of --with= and --without= parameters that allow fine-grained control of the data stored in the file tree serialization, including the granularity of modification times and more. The precise set of selected meta-data features is also always part of the serialization, so that seeding can work correctly and automatically.

3. casync tries to be as accurate as possible when storing file system meta-data. This means that besides the usual baseline of file meta-data (file ownership and access bits), and more advanced features (extended attributes, ACLs, file capabilities) a number of more exotic data is stored as well, including Linux chattr(1) file attributes, as well as FAT file attributes (you may wonder why the latter? — EFI is FAT, and /efi is part of the comprehensive serialization of any host). In the future I intend to extend this further, for example storing btrfs sub-volume information where available. Note that as described above every single type of meta-data may be turned off and on individually, hence if you don't need FAT file bits (and I figure it's pretty likely you don't), then they won't be stored.

4. The user creating .caidx or .caibx files may control the desired average chunk length (before compression) freely, using the --chunk-size= parameter. Smaller chunks increase the number of generated files in the chunk store and increase HTTP GET load on the server, but also ensure that sharing between similar images is improved, as identical patterns in the images stored are more likely to be recognized. By default casync will use a 64K average chunk size. Tweaking this can be particularly useful when adapting the system to specific CDNs, or when delivering compressed disk images such as squashfs (see below).

5. Emphasis is placed on making all invocations reproducible, well-defined and strictly deterministic. As mentioned above this is a requirement to reach the intended security guarantees, but is also useful for many other use-cases. For example, the casync digest command may be used to calculate a hash value identifying a specific directory in all desired detail (use --with= and --without to pick the desired detail). Moreover the casync mtree command may be used to generate a BSD mtree(5) compatible manifest of a directory tree, .caidx or .catar file.

6. The file system serialization format is nicely composable. By this I mean that the serialization of a file tree is the concatenation of the serializations of all files and file sub-trees located at the top of the tree, with zero meta-data references from any of these serializations into the others. This property is essential to ensure maximum reuse of chunks when similar trees are serialized.

7. When extracting file trees or disk image files, casync will automatically create reflinks from any specified seeds if the underlying file system supports it (such as btrfs, ocfs, and future xfs). After all, instead of copying the desired data from the seed, we can just tell the file system to link up the relevant blocks. This works both when extracting .caidx and .caibx files — the latter of course only when the extracted disk image is placed in a regular raw image file on disk, rather than directly on a plain block device, as plain block devices do not know the concept of reflinks.

8. Optionally, when extracting file trees, casync can create traditional UNIX hard-links for identical files in specified seeds (--hardlink=yes). This works on all UNIX file systems, and can save substantial amounts of disk space. However, this only works for very specific use-cases where disk images are considered read-only after extraction, as any changes made to one tree will propagate to all other trees sharing the same hard-linked files, as that's the nature of hard-links. In this mode, casync exposes OSTree-like behavior, which is built heavily around read-only hard-link trees.

9. casync tries to be smart when choosing what to include in file system images. Implicitly, file systems such as procfs and sysfs are excluded from serialization, as they expose API objects, not real files. Moreover, the "nodump" (+d) chattr(1) flag is honored by default, permitting users to mark files to exclude from serialization.

10. When creating and extracting file trees casync may apply an automatic or explicit UID/GID shift. This is particularly useful when transferring container image for use with Linux user name-spacing.

11. In addition to local operation, casync currently supports HTTP, HTTPS, FTP and ssh natively for downloading chunk index files and chunks (the ssh mode requires installing casync on the remote host, though, but an sftp mode not requiring that should be easy to add). When creating index files or chunks, only ssh is supported as remote back-end.

12. When operating on block-layer images, you may expose locally or remotely stored images as local block devices. Example: casync mkdev http://example.com/myimage.caibx exposes the disk image described by the indicated URL as local block device in /dev, which you then may use the usual block device tools on, such as mount or fdisk (only read-only though). Chunks are downloaded on access with high priority, and at low priority when idle in the background. Note that in this mode, casync also plays a role similar to "dm-verity", as all blocks are validated against the strong digests in the chunk index file before passing them on to the kernel's block layer. This feature is implemented though Linux' NBD kernel facility.

13. Similar, when operating on file-system-layer images, you may mount locally or remotely stored images as regular file systems. Example: casync mount http://example.com/mytree.caidx /srv/mytree mounts the file tree image described by the indicated URL as a local directory /srv/mytree. This feature is implemented though Linux' FUSE kernel facility. Note that special care is taken that the images exposed this way can be packed up again with casync make and are guaranteed to return the bit-by-bit exact same serialization again that it was mounted from. No data is lost or changed while passing things through FUSE (OK, strictly speaking this is a lie, we do lose ACLs, but that's hopefully just a temporary gap to be fixed soon).

14. In IoT A/B fixed size partition setups the file systems placed in the two partitions are usually much shorter than the partition size, in order to keep some room for later, larger updates. casync is able to analyze the super-block of a number of common file systems in order to determine the actual size of a file system stored on a block device, so that writing a file system to such a partition and reading it back again will result in reproducible data. Moreover this speeds up the seeding process, as there's little point in seeding the white-space after the file system within the partition.

Example Command Lines

Here's how to use casync, explained with a few examples:

$ casync make foobar.caidx /some/directory

This will create a chunk index file foobar.caidx in the local directory, and populate the chunk store directory default.castr located next to it with the chunks of the serialization (you can change the name for the store directory with --store= if you like). This command operates on the file-system level. A similar command operating on the block level:

$ casync make foobar.caibx /dev/sda1

This command creates a chunk index file foobar.caibx in the local directory describing the current contents of the /dev/sda1 block device, and populates default.castr in the same way as above. Note that you may as well read a raw disk image from a file instead of a block device:

$ casync make foobar.caibx myimage.raw

To reconstruct the original file tree from the .caidx file and the chunk store of the first command, use:

$ casync extract foobar.caidx /some/other/directory

And similar for the block-layer version:

$ casync extract foobar.caibx /dev/sdb1

or, to extract the block-layer version into a raw disk image:

$ casync extract foobar.caibx myotherimage.raw

The above are the most basic commands, operating on local data only. Now let's make this more interesting, and reference remote resources:

$ casync extract http://example.com/images/foobar.caidx /some/other/directory

This extracts the specified .caidx onto a local directory. This of course assumes that foobar.caidx was uploaded to the HTTP server in the first place, along with the chunk store. You can use any command you like to accomplish that, for example scp or rsync. Alternatively, you can let casync do this directly when generating the chunk index:

$ casync make ssh.example.com:images/foobar.caidx /some/directory

This will use ssh to connect to the ssh.example.com server, and then places the .caidx file and the chunks on it. Note that this mode of operation is "smart": this scheme will only upload chunks currently missing on the server side, and not re-transmit what already is available.

Note that you can always configure the precise path or URL of the chunk store via the --store= option. If you do not do that, then the store path is automatically derived from the path or URL: the last component of the path or URL is replaced by default.castr.

Of course, when extracting .caidx or .caibx files from remote sources, using a local seed is advisable:

$ casync extract http://example.com/images/foobar.caidx --seed=/some/exising/directory /some/other/directory

Or on the block layer:

$ casync extract http://example.com/images/foobar.caibx --seed=/dev/sda1 /dev/sdb2

When creating chunk indexes on the file system layer casync will by default store meta-data as accurately as possible. Let's create a chunk index with reduced meta-data:

$ casync make foobar.caidx --with=sec-time --with=symlinks --with=read-only /some/dir

This command will create a chunk index for a file tree serialization that has three features above the absolute baseline supported: 1s granularity time-stamps, symbolic links and a single read-only bit. In this mode, all the other meta-data bits are not stored, including nanosecond time-stamps, full UNIX permission bits, file ownership or even ACLs or extended attributes.

Now let's make a .caidx file available locally as a mounted file system, without extracting it:

$ casync mount http://example.comf/images/foobar.caidx /mnt/foobar

And similar, let's make a .caibx file available locally as a block device:

$ casync mkdev http://example.comf/images/foobar.caibx

This will create a block device in /dev and print the used device node path to STDOUT.

As mentioned, casync is big about reproducibility. Let's make use of that to calculate the a digest identifying a very specific version of a file tree:

$ casync digest .

This digest will include all meta-data bits casync and the underlying file system know about. Usually, to make this useful you want to configure exactly what meta-data to include:

$ casync digest --with=unix .

This makes use of the --with=unix shortcut for selecting meta-data fields. Specifying --with-unix= selects all meta-data that traditional UNIX file systems support. It is a shortcut for writing out: --with=16bit-uids --with=permissions --with=sec-time --with=symlinks --with=device-nodes --with=fifos --with=sockets.

Note that when calculating digests or creating chunk indexes you may also use the negative --without= option to remove specific features but start from the most precise:

$ casync digest --without=flag-immutable

This generates a digest with the most accurate meta-data, but leaves one feature out: chattr(1)'s immutable (+i) file flag.

To list the contents of a .caidx file use a command like the following:

$ casync list http://example.com/images/foobar.caidx

or

$ casync mtree http://example.com/images/foobar.caidx

The former command will generate a brief list of files and directories, not too different from tar t or ls -al in its output. The latter command will generate a BSD mtree(5) compatible manifest. Note that casync actually stores substantially more file meta-data than mtree files can express, though.

What casync isn't

1. casync is not an attempt to minimize serialization and downloaded deltas to the extreme. Instead, the tool is supposed to find a good middle ground, that is good on traffic and disk space, but not at the price of convenience or requiring explicit revision control. If you care about updates that are absolutely minimal, there are binary delta systems around that might be an option for you, such as Google's Courgette.

2. casync is not a replacement for rsync, or git or zsync or anything like that. They have very different use-cases and semantics. For example, rsync permits you to directly synchronize two file trees remotely. casync just cannot do that, and it is unlikely it every will.

Where next?

casync is supposed to be a generic synchronization tool. Its primary focus for now is delivery of OS images, but I'd like to make it useful for a couple other use-cases, too. Specifically:

1. To make the tool useful for backups, encryption is missing. I have pretty concrete plans how to add that. When implemented, the tool might become an alternative to restic, BorgBackup or tarsnap.

2. Right now, if you want to deploy casync in real-life, you still need to validate the downloaded .caidx or .caibx file yourself, for example with some gpg signature. It is my intention to integrate with gpg in a minimal way so that signing and verifying chunk index files is done automatically.

3. In the longer run, I'd like to build an automatic synchronizer for $HOME between systems from this. Each $HOME instance would be stored automatically in regular intervals in the cloud using casync, and conflicts would be resolved locally.

4. casync is written in a shared library style, but it is not yet built as one. Specifically this means that almost all of casync's functionality is supposed to be available as C API soon, and applications can process casync files on every level. It is my intention to make this library useful enough so that it will be easy to write a module for GNOME's gvfs subsystem in order to make remote or local .caidx files directly available to applications (as an alternative to casync mount). In fact the idea is to make this all flexible enough that even the remoting back-ends can be replaced easily, for example to replace casync's default HTTP/HTTPS back-ends built on CURL with GNOME's own HTTP implementation, in order to share cookies, certificates, … There's also an alternative method to integrate with casync in place already: simply invoke casync as a sub-process. casync will inform you about a certain set of state changes using a mechanism compatible with sd_notify(3). In future it will also propagate progress data this way and more.

5. I intend to a add a new seeding back-end that sources chunks from the local network. After downloading the new .caidx file off the Internet casync would then search for the listed chunks on the local network first before retrieving them from the Internet. This should speed things up on all installations that have multiple similar systems deployed in the same network.

Further plans are listed tersely in the TODO file.

FAQ:

1. Is this a systemd project? — casync is hosted under the github systemd umbrella, and the projects share the same coding style. However, the code-bases are distinct and without interdependencies, and casync works fine both on systemd systems and systems without it.

2. Is casync portable? — At the moment: no. I only run Linux and that's what I code for. That said, I am open to accepting portability patches (unlike for systemd, which doesn't really make sense on non-Linux systems), as long as they don't interfere too much with the way casync works. Specifically this means that I am not too enthusiastic about merging portability patches for OSes lacking the openat(2) family of APIs.

3. Does casync require reflink-capable file systems to work, such as btrfs? — No it doesn't. The reflink magic in casync is employed when the file system permits it, and it's good to have it, but it's not a requirement, and casync will implicitly fall back to copying when it isn't available. Note that casync supports a number of file system features on a variety of file systems that aren't available everywhere, for example FAT's system/hidden file flags or xfs's projinherit file flag.

4. Is casync stable? — I just tagged the first, initial release. While I have been working on it since quite some time and it is quite featureful, this is the first time I advertise it publicly, and it hence received very little testing outside of its own test suite. I am also not fully ready to commit to the stability of the current serialization or chunk index format. I don't see any breakages coming for it though. casync is pretty light on documentation right now, and does not even have a man page. I also intend to correct that soon.

5. Are the .caidx/.caibx and .catar file formats open and documented? — casync is Open Source, so if you want to know the precise format, have a look at the sources for now. It's definitely my intention to add comprehensive docs for both formats however. Don't forget this is just the initial version right now.

6. casync is just like $SOMEOTHERTOOL! Why are you reinventing the wheel (again)? — Well, because casync isn't "just like" some other tool. I am pretty sure I did my homework, and that there is no tool just like casync right now. The tools coming closest are probably rsync, zsync, tarsnap, restic, but they are quite different beasts each.

7. Why did you invent your own serialization format for file trees? Why don't you just use tar? — That's a good question, and other systems — most prominently tarsnap — do that. However, as mentioned above tar doesn't enforce reproducibility. It also doesn't really do random access: if you want to access some specific file you need to read every single byte stored before it in the tar archive to find it, which is of course very expensive. The serialization casync implements places a focus on reproducibility, random access, and meta-data control. Much like traditional tar it can still be generated and extracted in a stream fashion though.

8. Does casync save/restore SELinux/SMACK file labels? — At the moment not. That's not because I wouldn't want it to, but simply because I am not a guru of either of these systems, and didn't want to implement something I do not fully grok nor can test. If you look at the sources you'll find that there's already some definitions in place that keep room for them though. I'd be delighted to accept a patch implementing this fully.

9. What about delivering squashfs images? How well does chunking work on compressed serializations? – That's a very good point! Usually, if you apply the a chunking algorithm to a compressed data stream (let's say a tar.gz file), then changing a single bit at the front will propagate into the entire remainder of the file, so that minimal changes will explode into major changes. Thankfully this doesn't apply that strictly to squashfs images, as it provides random access to files and directories and thus breaks up the compression streams in regular intervals to make seeking easy. This fact is beneficial for systems employing chunking, such as casync as this means single bit changes might affect their vicinity but will not explode in an unbounded fashion. In order achieve best results when delivering squashfs images through casync the block sizes of squashfs and the chunks sizes of casync should be matched up (using casync's --chunk-size= option). How precisely to choose both values is left a research subject for the user, for now.

10. What does the name casync mean? – It's a synchronizing tool, hence the -sync suffix, following rsync's naming. It makes use of the content-addressable concept of git hence the ca- prefix.

11. Where can I get this stuff? Is it already packaged? – Check out the sources on GitHub. I just tagged the first version. Martin Pitt has packaged casync for Ubuntu. There is also an ArchLinux package. Zbigniew Jędrzejewski-Szmek has prepared a Fedora RPM that hopefully will soon be included in the distribution.

Should you care? Is this a tool for you?

Well, that's up to you really. If you are involved with projects that need to deliver IoT, VM, container, application or OS images, then maybe this is a great tool for you — but other options exist, some of which are linked above.

Note that casync is an Open Source project: if it doesn't do exactly what you need, prepare a patch that adds what you need, and we'll consider it.

If you are interested in the project and would like to talk about this in person, I'll be presenting casync soon at Kinvolk's Linux Technologies Meetup in Berlin, Germany. You are invited. I also intend to talk about it at All Systems Go!, also in Berlin.

The All Systems Go! 2017 Call for Participation is Now Open!

We’d like to invite presentation proposals for All Systems Go! 2017!

All Systems Go! is an Open Source community conference focused on the projects and technologies at the foundation of modern Linux systems — specifically low-level user-space technologies. Its goal is to provide a friendly and collaborative gathering place for individuals and communities working to push these technologies forward.

All Systems Go! 2017 takes place in Berlin, Germany on October 21st+22nd.

All Systems Go! is a 2-day event with 2-3 talks happening in parallel. Full presentation slots are 30-45 minutes in length and lightning talk slots are 5-10 minutes.

We are now accepting submissions for presentation proposals. In particular, we are looking for sessions including, but not limited to, the following topics:

• Low-level container executors and infrastructure
• IoT and embedded OS infrastructure
• OS, container, IoT image delivery and updating
• Building Linux devices and applications
• Low-level desktop technologies
• Networking
• System and service management
• Tracing and performance measuring
• IPC and RPC systems
• Security and Sandboxing

While our focus is definitely more on the user-space side of things, talks about kernel projects are welcome too, as long as they have a clear and direct relevance for user-space.

Please submit your proposals by September 3rd. Notification of acceptance will be sent out 1-2 weeks later.

To submit your proposal now please visit our CFP submission web site.

For further information about All Systems Go! visit our conference web site.

systemd.conf will not take place this year in lieu of All Systems Go!. All Systems Go! welcomes all projects that contribute to Linux user space, which, of course, includes systemd. Thus, anything you think was appropriate for submission to systemd.conf is also fitting for All Systems Go!

I just released the first release candidate for libinput 1.8. Aside from the build system switch to meson one of the more visible things is that the helper tools have switched from a "libinput-some-tool" to the "libinput some-tool" approach (note the space). This is similar to what git does so it won't take a lot of adjustment for most developers. The actual tools are now hiding in /usr/libexec/libinput. This gives us a lot more flexibility in writing testing and debugging tools and shipping them to the users without cluttering up the default PATH.

There are two potential breakages here, one is that the two existing tools libinput-debug-events and libinput-list-devices have been renamed too. We currently ship compatibility wrappers for those but expect those wrappers to go away with future releases. The second breakage is of lesser impact: typing "man libinput" used to bring up the man page for the xf86-input-libinput driver. Now it brings up the man page for the libinput tool which then points to the man pages of the various features. That's probably a good thing, it puts the documentation a bit closer to the user. For the driver, you now have to type "man 4 libinput" though.

Last week I got review on the new interface for tiling for vc4, and landed it in the kernel. I’ve submitted Mesa patches to the list, landed the kernel backport in Raspbian, and have sent Simon (the package maintainer at the Foundation) the Mesa patchset to backport all of my work from the last 6 months (unfortunately Raspbian generally doesn’t update Mesa past the Debian stable release, so graphics work for them has long delays). I also worked with Dom at the Foundation to get an interface for the firmwarekms mode to do tiled scanout, so it will be as fast as the open source driver.

I also reworked the official 7” touchscreen panel driver again. I had initially written it as a DSI panel driver, then got told it should be a bridge driver plus a panel driver, then the bridge maintainers told me it should be only one panel driver but attach to I2C. Maybe this version will stick, who knows.

My Mesa side for pl111 is now in master, so I’ve got the second platform for the vc4 driver basically done. I can’t wait to do another one.

I got a question about whether VC4 could do EGL_ANDROID_native_fence, to allow using Android’s hwcomposer (the display server that puts surfaces in KMS planes if it can). I spent a while looking into it, and found that while code had landed in Mesa for other drivers, the piglit tests had been left around untouched for the last half a year. I reviewed them and hopefully they can land soon. Unfortunately, they don’t cover the important functionality of the extension, so I can’t use them to actually write the driver code (which doesn’t look very hard) until I build some more tests.

On the non-vc4 kernel side, I prepared and submitted pull requests for 4.13. We’ve got USB OTG support for the Pi Zero, SDHOST enabled by default finally, thermal enabled by default finally, and the RPi3 DT being installed for 32-bit kernels. Three of these took months longer than they should have, because the Linux kernel has the worst software development process I’ve worked with since XFree86.

There is a saying that persistence is the key to success. Not that I’m always following this advice, but I did luckily earlier this year when I had to decide if I wanted to apply again for a Google Summer of Code (GSoC) spot after my application in the last year was rejected by my organisation of choice back then. I talked about it in one of my last posts. Anyway, thanks to being persistent this year one of my project ideas got accepted and I now have the opportunity to work on a very interesting project concerning Xwayland, this time for the X.Org Foundation.

The project is kind of difficult though. At least it feels this way to me right now, after I’ve already spent quite some time on digging into it and getting a feel for it.

The main reason for the difficulty is, that various components of the XServer are in play and documentation for them is most often missing. Would I be on my own, I could basically only work with the code and try to comprehend the steps done one after the other. I’m not on my own though! With Daniel Stone I have one of the core developers of several key parts of the Linux graphic stack as my mentor, who seems to really want to help me to understand the difficult stuff on a large scale and still gives concrete advice on what to do next. He even drew a diagram for me!

I plan on publishing in future blog posts resources like this otherwise clearly underused diagram, since such material to understand the XServer code is otherwise only sparsely to find on the web. Additionally I’ll try to explain the idea of my project and the general structure of the code I’m dealing with in my own words. Regarding these future posts a nasty surprise for me was this week, that it’s expected from me to publish one blog post per week about my project. This somewhat spoils my plans for this blog, where I wanted to publish very few but in comparison more sophisticated posts. Nevertheless you can therefore expect to have frequent weekly posts until August. Next week I want to explain the basic idea of my project.

• LCD panel not working with the i915 driver on some devices


• Battery monitoring not working on most Bay and Cherry devices (battery monitoring did not work on any device using the very popular AXP288 PMIC)


• Backlight control not working on many devices (you need to builtin the pwm-lpss drivers, or add them to your initrd for this fix to work)


• Volume and power buttons not working on Cherry Trail devices shipping with Win10


• The microsd slot on the Asus Transformer T100TA not working


• Preliminary (incomplete) support Cherry Trail devices with a Whiskey Cove PMIC, some bits to get e.g. battery monitoring working are still missing


• Added the rtl8723bs driver to staging, enabling ootb support for wifi on many devices

Back in 2003, the pressure range value for Wacom pen tablets was set to 2048. That's a #define in the driver, but it shouldn't matter because we also advertise this range as part of the device description in the X Input protocol. Clients should be using that advertised min/max range and scale appropriately, in the same way as they should be doing this for the x/y axis ranges.

Fast-forward to 2017 and we changed the pressure range. New Wacom devices now use ~8000 levels, but we opted to just #define it in the driver to 65536 and be done with it. We now scale all models into that range, with varying granularity based on the physical hardware. It shouldn't matter because it's not tied to a reliable physical property anyway and the only thing that matters is the percentage of the max value (which is why libinput just gives you a [0, 1] range. Hindsight is a bliss.).

Slow-forward to 2017-and-a-bit and we received complaints that pressure handling is now broken. Turns out that some applications hardcoded the old 2048 range and now get confused, because virtually any pressure will now hit that maximum. Since those applications are largely proprietary ones and cannot be updated easily, we needed a workaround to this. Jason Gerecke from Wacom got busy and we now have a "Pressure2K" option available in the driver. If set, this option will scale everything into the 2048 range to make sure those applications still work. To get this to work, the following xorg.conf.d snippet is recommended:

Section "InputClass"
    Identifier "Wacom pressure compatibility"
    MatchDriver "wacom"
    Option "Pressure2K" "true"
EndSection

$>  xinput list "Wacom Intuos4 6x9 Pen stylus"
Wacom Intuos4 6x9 Pen stylus                    id=25   [slave  pointer  (2)]
        Reporting 8 classes:
                Class originated from: 25. Type: XIButtonClass
                Buttons supported: 9
                Button labels: None None None None None None None None None
                Button state:
                Class originated from: 25. Type: XIKeyClass
                Keycodes supported: 248
                Class originated from: 25. Type: XIValuatorClass
                Detail for Valuator 0:
                  Label: Abs X
                  Range: 0.000000 - 44704.000000
                  Resolution: 200000 units/m
                  Mode: absolute
                  Current value: 22340.000000
                Class originated from: 25. Type: XIValuatorClass
                Detail for Valuator 1:
                  Label: Abs Y
                  Range: 0.000000 - 27940.000000
                  Resolution: 200000 units/m
                  Mode: absolute
                  Current value: 13970.000000
                Class originated from: 25. Type: XIValuatorClass
                Detail for Valuator 2:
                  Label: Abs Pressure
                  Range: 0.000000 - 2048.000000
                  Resolution: 1 units/m
                  Mode: absolute
                  Current value: 0.000000
                Class originated from: 25. Type: XIValuatorClass
                Detail for Valuator 3:
                  Label: Abs Tilt X
                  Range: -64.000000 - 63.000000
                  Resolution: 57 units/m
                  Mode: absolute
                  Current value: 0.000000
                Class originated from: 25. Type: XIValuatorClass
                Detail for Valuator 4:
                  Label: Abs Tilt Y
                  Range: -64.000000 - 63.000000
                  Resolution: 57 units/m
                  Mode: absolute
                  Current value: 0.000000
                Class originated from: 25. Type: XIValuatorClass
                Detail for Valuator 5:
                  Label: Abs Wheel
                  Range: -900.000000 - 899.000000
                  Resolution: 1 units/m
                  Mode: absolute
                  Current value: 0.000000

This is an application bug, but this workaround will make sure new versions of the driver can be used until those applications have been fixed. The option will be available in the soon-to-be-released xf86-input-wacom 0.35. The upstream commit is d958ab79d21b57141415650daac88f9369a1c861.

Edit 02/06/2017: wacom devices have 8k pressure levels now.

It’s been a busy month. Part of that is that I took a week off to go to our Burning Man regional event, which threw a serious wrench into my status updates.

Hans Verkuil now has HDMI CEC working, and is polishing the patches up for inclusion in the kernel. Unfortunatelly we set his work back a bit with Boris’s new HDMI power management code, and that’s getting resolved now. He also tripped over a bug in my new dma-buf reservation code that I helped track down.

Boris has the transposer working and has reviewed the core writeback code, so hopefully we can land it soon. I’m really excited to use this for testing tiled display support (see below).

My work for the last couple of weeks has been in figuring out what to do about the fact that the new Raspbian display stack (vc4 firmwarekms, glamor, compton compositor) takes way more memory than the old display stack (fbdev on the firmware, no glamor, no compositing). It’s been pretty easy for users to hit the 256MB CMA limit, and we aren’t even using 256MB of CMA on the Pi0/1 due to its only having 512MB of system memory.

One of the ideas would be to page GPU allocations out to system memory. That only gets us from 256MB to <1GB of memory available. It’s not like you want to be swapping to SD cards. If we’re hitting 256MB this easily, we need to figure out how to fix that.

The first step was to figure out where our memory was going. I wrote a hack that had the kernel give names to the BOs it allocated, and added a new UAPI that let userspace attach more descriptive names to BOs. With that, I added a little hack to Mesa to attach some descriptions. I then made /debug/dri/0/bo_stats print out the stats on count/size per name, rather than just total. Here’s the output of LXDE with compton as I was dragging a terminal around:

 scanout resource 1920x1080@32/0:  48600kb BOs (6)
    tiling shadow 1920x1080:  32640kb BOs (4)
		   overflow:  16384kb BOs (1)
    resource 1920x1080@32/0:  16320kb BOs (2)
    resource 1024x1024@32/0:   4096kb BOs (1)
      tiling shadow 661x411:   2184kb BOs (2)
      resource 657x386@32/0:   1092kb BOs (1)
  scanout resource 661x411@32/0:   1068kb BOs (1)
     resource 1048576x1@8/0:   1024kb BOs (1)
     resource 1024x1024@8/0:   1024kb BOs (1)
		   resource:    888kb BOs (20)
      tiling shadow 1920x26:    240kb BOs (1)
      resource 1920x26@32/0:    240kb BOs (1)
		     shader:    236kb BOs (56)
  scanout resource 1920x26@32/0:    196kb BOs (1)
		 mesa cache:    152kb BOs (5)
       resource 659x20@32/0:     84kb BOs (1)
       resource 65536x1@8/0:     64kb BOs (1)
      resource 128x128@32/0:     64kb BOs (1)
	  resource 1x1@32/0:     56kb BOs (14)
	      dumb 64x64@32:     48kb BOs (3)
	resource 659x1@32/0:     36kb BOs (3)
		      cache:     28kb BOs (4)
			RCL:     20kb BOs (2)
	 resource 26x1@32/0:     16kb BOs (4)
	resource 1x386@32/0:     16kb BOs (2)
	resource 1x361@32/0:     16kb BOs (2)
	 resource 1x25@32/0:     16kb BOs (4)
	resource 609x1@32/0:     12kb BOs (1)
	resource 607x1@32/0:     12kb BOs (1)
	resource 12x18@32/0:      4kb BOs (1)
			BCL:      4kb BOs (1)

This horrifying. First, this is already after a fix I’ve landed that kept us from leaking an extra overflow BO, that cost 16MB at all times. Next, 6 full-screen scanout resources? I expected 3, certainly, due to compton’s pageflipping (though for a compositor, keeping 3 is bad: I should have GPU idle time built into my pipeline anyway, as presumably GPU work is being done by other clients to generate the frames I’m compositing). Our non-pageflipping scanout buffer is probably another one (can we free the screen pixmap’s contents during pageflipping? That would sure be nice). Maybe the file manager is drawing into a window rather than the root window, to make the another one. There’s still one more I haven’t accounted for, though.

More embarassing is the tiling shadow allocations. We do this in vc4 because it can’t texture from linear BOs, so we allocate a shadow resource that we blit a tiled copy into. The cost of those tile blits are actually why we’re running a compositor in Raspbian in the first place, because window movement is too expensive otherwise.

My crazy idea was to make a new mode for glamor that just keeps all pixmaps in system memory until they need to be on the GPU (because they’re either scanned out by the GPU or they have an exposed reference to them for DRI). My previous reworks for linear vs tiled buffers in glamor already made this look easy. You end up with software fallbacks all the time (The old fbturbo driver was all software fallbacks, though), and with the new gbm_bo_map() interface we should be able to keep the cost of software fallbacks down to just the cost of the rendering performed, not a 1920x1080 readback per drawing operation.

However, 20 patches in and without having gotten to the point of gbm_bo_map() yet, this is looking like a mess. On Friday I had breakfast with keithp and talked over what I was doing, and he convinced me to go back to an older, simpler plan: Just make all of our pixmaps tiled, so we don’t have the shadow copies at all. If we don’t have the shadow copies, then we don’t need the compositor to get window-dragging performance. At least 64MB of the allocations above would go away.

The new plan means we eat the memory bandwidth cost of scanning out from tiled, but it will be easy enough to register a timer to go make a linear copy and flip to that. We’re not the only platform that would like that, I’m sure.

I’ve now written the kernel interface and debugging it is the plan for this week.

In other news, my panel-bridge v3 is now in drm-misc-next, along with most of the associated code deletion. I also helped the team trying to integrate my PL111 code with a bug in panning support, which their current 3D stack relies on. In the process of debugging the pl111+vc4 code (a full screen X11 fallback would reallocate the BO when it shouldn’t), I added debugfs register dumping for pl111, which should be useful for other developers using the code.

I have the X Server building in Travis CI, at long last. The meson build system was critical for this, as autotools was just too slow. The other key part was switching to Docker for bundling our dependencies, since 20 autotools invocations was unreasonable, and I won’t have the time to convert those 20 things to meson. Next steps on CI are to get the X Test suite running in the meson build system (the docker image already has it present).

Finally, earlier this month I submitted my Mesa register allocation series (let the driver choose the color during graph coloring), which should be good for about a 2% performance improvement to vc4 3D.

With modifier support added to Mesa and gbm_gralloc, it is now possible to boot Android on iMX6 platforms using no proprietary blobs at all. This makes iMX6 one of the very few embedded SOCs that needs no blobs at all to run a full graphics stack.

Not only is that a great win for Open Source in general, but it also makes the iMX6 more attractive as a platform. A further positive point is that this lays the groundwork for the iMX8 platform, and supporting it will come much easier.

What are modifiers used for?

Modifiers are used to represent different properties of buffers. These properties can cover a range of different information about a buffer, for example compression and tiling.

For the case of …

With modifier support added to Mesa and gbm_gralloc, it is now possible to boot Android on iMX6 platforms using no proprietary blobs at all. This makes iMX6 one of the very few embedded SOCs that needs no blobs at all to run. Not only is that a great win for Open Source in general, but it also makes the iMX6 more attractive as a platform.

Currently the modifiers work is in the process of being upstreamed, but in the meantime it can be found here. If you'd like to test this out yourself I maintain a how-to.

What are modifiers used for?

Modifiers are used to represent different properties of buffers. These properties can cover a range of different information about a buffer, for example …

DRM leasing part three (Vulkan)

With the kernel APIs off for review, and the X RandR bits looking like they're in reasonable shape, I finally found some time to sit down and figure out how I wanted to integrate this into Vulkan.

Avoiding two DRM file descriptors

Given that a DRM lease is represented by a DRM master file descriptor, we want to use that for all of the operations in the driver, including rendering and mode setting. Using the vulkan driver render node and the lease master node together would require passing buffer objects between the kernel contexts using even more file descriptors.

The Mesa Vulkan drivers open the device nodes while enumerating devices, not when they are created. This seems a bit early to me, but it makes sure that the devices being enumerated are actually available for use, and not just present in the system. To replace the render node fd with the lease master fd means hooking into the system early enough that the enumeration code can see the lease fd. And that means creating an instance extension as the instance gets created before devices are enumerated.

The VK_KEITHP_kms_display instance extension

This simple instance extension provides the necessary hooks to get the lease information from the application down into the driver before the DRM node is opened. In the first implementation, I added a function that could be called before the devices were enumerated to save the information in the Vulkan loader. That worked, but required quite a bit of study of the Vulkan loader and its XML description of the full Vulkan API.

Mark Young suggested that a simpler plan would be to chain the information into the VkInstanceCreateInfo pNext field; with no new APIs added to Vulkan, there shouldn't be any need to change the Vulkan loader -- the device driver would advertise the new instance extension and the application could find it.

That would have worked great, except the Vulkan loader 'helpfully' elides all instance extensions it doesn't know about before returning the list to the application. I'd say this was a bug and should be fixed, but for now, I've gone ahead and added the few necessary definitions to the loader to make it work.

In the application, it's a simple matter of searching for this extension, constructing the VkKmsDisplayInfoKEITHP structure, chaining that into the VkInstanceCreateInfo pNext list and passing that in to the vkCreateInstance call.

typedef struct VkKmsDisplayInfoKEITHP {
    VkStructureType         sType;  /* VK_STRUCTURE_TYPE_KMS_DISPLAY_INFO_KEITHP */
    const void*             pNext;
    int                     fd;
    uint32_t                crtc_id;
    uint32_t                *connector_ids;
    int                     connector_count;
    drmModeModeInfoPtr      mode;
} VkKmsDisplayInfoKEITHP;

As you can see, this includes the master file descriptor along with all of the information necessary to set the desired video mode using the specified resources.

The driver just walks the pNext list from the VkInstanceCreateInfo structure looking for any provided VkKmsDisplayInfoKEITHP structure and pulls the data out.

To avoid questions about file descriptor lifetimes, the driver dup's the provided fd. The application is expected to close their copy at a suitable time.

The VK_KHR_display extension

Vulkan already has an API for directly accessing the raw device, including code for exposing video modes and everything. As tempting as it may be to just go do something simpler, there's a lot to be said for using existing APIs.

This extension doesn't provide any direct APIs for acquiring display resources, relying on the VK_EXT_acquire_xlib_display extension for that part. And that takes a VkPhysicalDisplay parameter, which is only available after the device is opened, which is why I created the VK_KEITHP_kms_display extension instead of just using the VK_EXT_acquire_xlib_display extension -- we can't increase the capabilities of the render node opened by the driver, and we don't want to keep two file descriptors around.

With the information provided by the VK_KEITHP_kms_display extension, we can implement all of the VK_KHR_display extension APIs, including enumerating planes and modes and creating the necessary display surface. Of course, there's only one plane and one mode, so some of the implementation is pretty simplistic.

The big piece of work was to create the swap chain structure and associated frame buffers.

A working example

I've taken the 'cube' example from the Vulkan loader and hacked it up to use XCB to construct a DRM lease, the VK_KEITHP_kms_display extension to pass that lease into the Vulkan driver. The existing support for the VK_KHR_display extension "just worked", which was pretty satisfying.

It's a bit of a mess

I'm not satisfied with the mesa code at this point; there's a bunch of code in the radeon driver which should be in the vulkan WSI bits, and the vulkan WSI bits should probably not have the KMS interfaces wired in. I'll ask around and see what other Mesa developers think I should do before restructuring it further; I'll probably have to rewrite it all at least one more time before it's ready to upstream.

Seeing the code

I'll be cleaning the code up a bit before sending it out for review, but it's already visible in my own repositories:

• mesa: https://cgit.freedesktop.org/~keithp/mesa/log/?h=drm-leases

• Vulkan Loader: https://cgit.freedesktop.org/~keithp/Vulkan-LoaderAndValidationLayers/log/?h=drm-leases

• demo: https://cgit.freedesktop.org/~keithp/jesse-cube/log/

TLDR: If you see devices like "xwayland-pointer" show up in your xinput list output, then you are running under a Wayland compositor and debugging/configuration with xinput will not work.

For many years, the xinput tool has been a useful tool to debug configuration issues (it's not a configuration UI btw). It works by listing the various devices detected by the X server. So a typical output from xinput list under X could look like this:

:: whot@jelly:~> xinput list
⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPad                id=22   [slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                     id=23   [slave  pointer  (2)]
⎜   ↳ ELAN Touchscreen                          id=20   [slave  pointer  (2)]
⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
    ↳ Power Button                              id=6    [slave  keyboard (3)]
    ↳ Video Bus                                 id=7    [slave  keyboard (3)]
    ↳ Lid Switch                                id=8    [slave  keyboard (3)]
    ↳ Sleep Button                              id=9    [slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                    id=24   [slave  keyboard (3)]

$ xinput list
⎡ Virtual core pointer                     id=2 [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer               id=4 [slave  pointer  (2)]
⎜   ↳ xwayland-pointer:13                      id=6 [slave  pointer  (2)]
⎜   ↳ xwayland-relative-pointer:13             id=7 [slave  pointer  (2)]
⎣ Virtual core keyboard                    id=3 [master keyboard (2)]
    ↳ Virtual core XTEST keyboard              id=5 [slave  keyboard (3)]
    ↳ xwayland-keyboard:13                     id=8 [slave  keyboard (3)]

This usually doesn't matter, but when it comes to debugging or configuring devices with xinput we run into a few issues. First, configuration via xinput usually means changing driver-specific properties but in the XWayland case there is no driver involved - it's all handled by libinput inside the compositor. Second, debugging via xinput only shows what the wayland protocol sends to XWayland and what XWayland then passes on to the client. For low-level issues with devices, this is all but useless.

The takeaway here is that if you see devices like "xwayland-pointer" show up in your xinput list output, then you are running under a Wayland compositor and debugging with xinput will not work. If you're trying to configure a device, use the compositor's configuration system (e.g. gsettings). If you are debugging a device, use libinput-debug-events. Or compare the behaviour between the Wayland session and the X session to narrow down where the failure point is.

Not only, but to a large extent I worked in the last few months on foundational improvements to KWin’s DRM backend, which is a central building block of KWin’s Wayland session. The idea in the beginninng was to directly expand upon my past Atomic Mode Setting (AMS) work from last year. We’re talking about direct scanout of graphic buffers for fullscreen applications and later layered compositing. Indeed this was my Season of KDE project with Martin Flöser as my mentor, but in the end relative to the initial goal it was unsuccessful.

The reason for the missed goal wasn’t a lack of work or enthusiasm from my side, but the realization that I need to go back and first rework the foundations, which were in some kind of disarray, mostly because of mistakes I did when I first worked on AMS last year, partly because of changes Daniel Stone made to his work-in-progress patches for AMS support in Weston, which I used as an example throughout my work on AMS in KWin, and also because of some small flaws introduced to our DRM backend before I started working on it.

The result of this rework are three seperate patches depending on each other and all of them got merged last week. They will be part of the 5.10 release. The reason for doing three patches instead of only one, was to ease the review process.

The first patch dealt with the query of important kernel display objects, which represent real hardware, the CRTCs and Connectors. KWin didn’t remember these objects in the past, although they are static while the system is running. This meant for example that KWin requeried all of them on a hot plugging event and had no prolonged knowledge about their state after a display was disconnected again. The last point made it in particular difficult to do a proper cleanup of the associated memory after a disconnect. So changing this in a way that the kernel objects are only queried once in the beginning made sense. Also from my past work I already had created a generic class for kernel object with the necessary subclasses, which could be used in this context. But still to me this patch was the most “controversial” one of the three, which means it was the one I was most worried about being somehow “wrong”, not just in details, but in general, especially since it didn’t solve any observable specific misbehaviour, which it could be benchmarked against. Of course I did my research, but there is always the anxiety of overlooking something crucial. Too bad the other patches depended on it. But the patch was accepted and to my relief everything seems to work well on the current master and the beta branch for the upcoming release as well.

The second patch restructured the DrmBuffer class. We support KWin builds with or without Generic Buffer Manager (GBM). It made therefore sense to split off the GBM dependent part of DrmBuffer into a seperate file, which gets only included when GBM is available. Martin had this idea and, although the patch is still quite large because of all the moved around code and renamed classes, the change was straight forward. I still managed to introduce a build breaking regression, which was quickly discovered and easily to solve. This patch was also meant as a preperation for the future direct scanout of buffers, which will then be done by a new subclass of DrmBuffer, also depending on GBM.

The last patch finally directly tackled all the issues I experienced when trying to use the before that rather underwhelming code path for AMS. Yes, you saw the picture on the screen, the buffer flipping worked, but basic functionality like hot plugging or display suspending were not working at all or led to unpredictable behaviour. Basically a complete rewrite later with many, many manual in and out pluggings of external monitors to test the bahaviour the problems have been solved to the point I consider the AMS code path now to be ready for daily use. For Plasma 5.11 I therefore plan to make it the new default. That means that it will be available on Intel graphics automatically from Linux kernel 4.12 onwards, when on the kernel side the Intel driver also defaults to it. If you want to test my code on Plasma 5.10 you need to set the environment variable KWIN_DRM_AMS and on kernels older than 4.12 you need to add the boot parameter i915.nuclear_pageflip. If you use Nvidia with the open source Nouveau driver, AMS should be available to you since kernel 4.10. In this case you should only need to set the environment variable above on 5.10, if you want to test it. Since I only tested AMS with Intel graphics until now, some reports back how it works with Nouveau would be great.

That’s it for now. But of course there is more to come. I haven’t given up on the direct scanout and at some point in the future I want to finish it. I already had a working prototype and mainly waited for my three patches to land. But for now I’ll postpone further work on direct scanout and layered compositing. Instead the last weeks I worked on something special for our Wayland session in 5.11. I call it Night Color, and with this name you can probably guess what it will be. And did I mention, that I was accepted as a Google Summer of Code student for the X.org foundation with my project to implement multi buffered present support in XWayland? Nah, I didn’t. Sorry for rhetorical asking in this smug way, but I’m just very happy and also a bit proud of having learned so much in basically only one year to the point of now being able to start work on an X.org project directly. I’ll write about it in another blog post in the near future.

Early last year, Oracle announced it would be shutting down the general project hosting on java.net & kenai.com at the end of April 2017.

We'd been hosting various Solaris content on there since opensolaris.org shut down in 2013, and have worked to move most of it off now. We've requested some redirects be put in place where possible, but they only offer one per subproject, so we can't make all pages redirect to the best spot for that particular content.

If you referenced anything under http://solaris.java.net or its sub projects, here's a guide to where to find that content now:

FOSS Source Code:

• Userland & Desktop: https://github.com/oracle/solaris-userland/
• X Window System: https://github.com/oracle/solaris-xorg/
• IPS: https://github.com/oracle/solaris-ips/

Discussion:

The java.net mailing lists were not heavily used, so we've not set up new replacement mailing lists, but suggested discussion migrate to the forums at https://community.oracle.com/community/server_&_storage_systems instead.

Information pages:

• CDDL license: https://oss.oracle.com/licenses/
• OpenStack: http://www.oracle.com/technetwork/server-storage/solaris11/technologies/openstack-2135773.html
• Zones: http://www.oracle.com/technetwork/server-storage/solaris11/technologies/virtualization-306056.html
• ZFS: http://www.oracle.com/technetwork/server-storage/solaris11/technologies/data-management-2237949.html

And of course, much of it was archived in the Internet Archive for historical reference.

I’ve been procrastinating on writing up status, so this update’s a big one.

Boris Brezillon has been hacking away at implementing the VC4 transposer module as a DRM writeback connector, and has things running in test apps. My immediate goal with his work is to enable IGT testing of VC4’s HVS (hardware compositor) features, so that we can implement plane tiling and formats with confidence. However, there’s also a long term goal to extend X11’s modesetting driver to do Present into DRM planes, and only collapse planes down to the primary framebuffer once an atomic check fails or the display goes idle. This is something that other compositors like Raspberry Pi’s DispmanX firmware, Android HWC2, or Weston can all do, so we should fix X11 to do it as well.

Hans Verkuil has started on a CEC implementation for vc4’s HDMI module. The hardware looks fairly simple – unlike many other HDMI CEC implementations, ours is right there in the HDMI module, so there are no complicated inter-kernel-module dependencies to implement. I’ve set him up with register definitions and answered some hardware questions, and I suspect he’ll have results soon.

On my end, I’m excited to announce that the ARM pl111 module DRM KMS driver for Broadcom’s Cygnus is now merged to drm-misc-next, along with V3D support. This has been a more or less pleasant process: I got useful review feedback from Linus Walleij (who’s being involved in the fbdev version of this driver), some cleanups on the V3D code from Florian, and I got the code landed in a reasonable amount of time. The drm-misc process may not be perfect, but it’s the best I’ve experienced in the Linux kernel yet.

Next steps on Cygnus: I need to land a reset controller driver and the associated DT (right now if you lock up the GPU, it never resets). On Raspberry Pi reset works by us turning the power domain off and on, which causes the firmware to go through the reset process, but on Cygnus we don’t have a separate power domain for V3D. I also need to finish the panel driver and submit it (I’m using a stub for now, and have a proper panel driver in progress). And I’m working on using gallium’s “renderonly” helper functions to allow you to bring up GBM with GL accelerated on vc4 on a pl111 DRM node. This should get X, glmark, and kmscube (and any other GBM applications), working on Cygnus with no userspace changes.

I’ve also been reworking Raspberry Pi’s DSI panel support. I got really discouraged on this after trying to submit to the DRM panel tree back in December. Based on that feedback, I’m moving most of the code to a DRM bridge driver, with just the panel timing in the DRM panel tree. However, writing drivers that talk to DRM panels sucks, so I’ve written the “panel-bridge” helper that cuts about 100 lines of boilerplate from most DRM drivers that talk to a panel, by wrapping the panel in a DRM bridge structure. I submitted an early version for review last week, which was met with some skepticism. I need to re-submit my updated version that converts more drivers – probably when the patch series removes 400 lines of code from DRM, it’ll get a warmer reception.

I’ve also been working on cleaning up our GLES2 conformance test results. I’ve landed 4 patches to the GLSL compiler, and one more Mesa patch is in the queue. The most interesting problem for conformance is that some of the GLES2 conformance tests fail in our register allocator: vc4 can’t spill registers, so we need to be really good at register allocation to make arbitrary programs render successfully.

My current goal for allocation is to implement Sethi-Ullman based instruction scheduling at the QIR stage to try to keep register pressure down. However, register pressure is weighted at a middle level in the QIR instruction scheduler’s heuristics, because otherwise later on we end up too constrained for the QPU instruction scheduler and introduce stalls. To fix the QPU scheduling, I wrote a patch series that gives the driver a chance to choose among available registers during register coloring. With that, the driver can prioritize the accumulators while rotating through the available registers. Initial performance results look excellent, so I should be able to bump register pressure up in priority next and do the Sethi-Ullman work. Interestingly, this callback could resolve a longstanding issue that Intel’s pre-Sandybridge driver had in register allocation as well!

In the X Server, I’ve landed the meson build system. I’m not quite dogfooding it yet, because I can successfully run my desktop with startx but not with gdm (it looks like something is going wrong in the systemd integration). I’ve received patches from a few other X developers fixing up the meson build system for themselves, so I think we’re on track toward general adoption. I’ve made some more progress on implementing Travis CI for the X Server now, and on integrating the current X Server unit tests into meson.

Since the hardware very much matters this is going to be divided into a few parts, the common steps and the hardware specific ones.

This post is a bit of a living document and will be changed over time, and if you have any questions about it, please reach out through email (robert.foss at collabora.com) or irc (tomeu or robertfoss on #dri-devel on freenode).

Changelog:
2017-04-27: build_android.sh, setup_sdcard.sh: Added -b [device] support to build_android.sh and setup_sdcard.sh
2017-05-02: build_android.sh: Don't write to SD-card without -b option
2017-05-04: Switch git repo urls to shared repository
2017-05-09: Add compiler installation to apt-get
2017-05-09: Re-ordered some instructions
2017 …

So we have a new job available for someone interested in joing our team and work on improving the Linux graphics stack. The focus of this job will be on GPU compute related work, but you should also expect to be spending time on improving the graphics driver stack in general. We are looking for someone at the Principal Engineer level, but I do recommend that even if you don’t feel you are quite at that level yet you should apply because to be fair the amount of people with the kind of experience we are looking for are few and far between, so in the end there is a chance we will hire two more junior developers instead if we have candidates with the right profile.

We are quite flexible on working location for this job, so for the right candidate working remotely is definitely a possibility. And of course if you are interested in joining us at one of our offices that is an option too, for instance we have existing team members working out of our Boston (USA), Brno(Czech Republic), Brisbane (Australia) and Munich (Germany) offices.

GPU Compute is rapidly growing in importance and use so this is your chance to be in the middle of it and work for what I personally think is one of the best companies in the world to work for.

So be sure to submit an application though the Red Hat hiring portal.

Since the hardware very much matters this is going to be divided into a few parts, the common steps and the hardware specific ones.

Common steps

mkdir /opt/android
repo init -u https://android.googlesource.com/platform/manifest -b android-7.1.1_r28
cd /opt/android/.repo
git clone git://git.collabora.com/git/user/robertfoss/android_manifest.git local_manifests -b etnaviv-android
repo sync -j75

mkdir /opt/imx6_android
cp /opt/imx6_android
git clone git://git.collabora.com/git/user/robertfoss/linux.git -b imx_rdu2_v4.11-rc3

# The mkimage tool is used even if you're not
# using u-boot it as a bootloader
sudo apt install u-boot-tools

# Fetch Kconfig, bootloaders and some scripts
git clone git://git.collabora.com/git/user/robertfoss/rdu2.git .

# This will destroy all data …

Mesa EGL with Wayland, and simplified X as comparison.

• There are two different EGL platforms in play: one for the server, and one for the clients.
• A Wayland server does not contain any graphics hardware or driver specific code, it is all in the generic libraries of DRM, EGL and GL (libdrm and Mesa).
• Everything about wl_drm is an implementation detail internal to the EGL library in use.

The pre-history

The middle age

The new age

The global thermonuclear war (named after a git branch by Rob Clark)

And then there shall be ponies (at the end of the rainbow)

How protocol objects are created

How protocol objects are destroyed

Enter the boogeyman: races

The boogeyman's brother

Concluding with recommendations

• Always make sure there is a guaranteed way to destroy all objects. This may sound obvious, but we have fixed several cases in the Wayland core protocol where there was no way to destroy a created protocol object such, that all resources on both server and client side could be freed. And there are still some cases not fixed.
  
• Always define a destructor request. If you have any doubt whether your new interface needs a destructor request, just put it there. It is more awkward to add later than normal requests. If you do not have one, the client cannot tell the server to free those protocol object resources.
• Do not use destructor events. They are hard to design right, and extending the interface later will be a bitch. The client cannot tell the server to free the resources, so objects with destructor events should be short-lived, and the destruction must be guaranteed.
• Do not use server-side created objects without a serious thought. Designing the destruction sequence such that it never leaks nor explodes is tricky.

The old algorithm

Figure 1. The old algorithm, the client paints as response to frame callbacks.

Figure 2. The old algorithm, the client paints as response to Presentation feedback events.

Figure 3. The old algorithm, client posts a frame while the compositor is idle.

The modification to the algorithm

The new algorithm

Figure 4. The new algorithm, the client paints as response to frame callback.

Figure 5. The new algorithm, the client paints as response to Presentation feedback.

Discussion

An image and a pixel

A pixel in memory

Figure 1. Three equivalent pixel formats with 8 bits for each channel. The writing convention here is to list channels from highest to lowest bits in a unit. That is, abgr8888 has r in bits 0-7, g in bits 8-15, etc.

An image in memory

Figure 2. The usual layout of pixels of an image in memory.

Stride or pitch

Addressing a pixel

pixel_address = data_begin + y * stride_bytes + x * bytes_per_pixel.

uint32_t *p = data_begin;
p += y * stride_bytes / sizeof(uint32_t);
p += x;

uint32_t v = *p;
uint8_t r = (v >> 16) & 0xff;

Figure 3. How to compute the address of a pixel.

Not everyone has the "right" way up

Multi-planar formats

Tiled formats

Click to see the video!

• wayland
• weston
• pixman
• mtdev
• cairo
• libxkbcommon, and its dependencies xproto, kbproto, and xkeyboard-config
• libffi
• mouse cursors

• an image you can flash into your device, if you have the right tools: android-4.0.1_r1.2-b.tar.gz (106 MB) (sha1: 1bd52cef8b53574452b4e2feac76c5191e815884)
• instructions for building a full Android OS with Weston: README

• The source code you can build yourself, see README. Use the module wayland where anagrman is invoked.
• A snapshot image you can flash into a Samsung Galaxy Nexus: wayland-on-android-20120924.zip (105 MB) (sha1: b75cb4e325bc7a2960464128ae7f569c21741471). You can flash it with the command: $ fastboot update wayland-on-android-20120924.zip

• # setprop service.compositor surfaceflinger
• # setprop service.compositor weston

• the client side support for graphics hardware acceleration
• the compositor's support for hardware accelerated clients
• the compositor's own rendering or compositing, and output to screen

• Do such handles exist at all?
• How do you create a buffer and a handle?
• How do you direct GL ES rendering into that buffer?
• What is the handle? Does it contain integers, open file descriptors, or opaque pointers? Integers and file descriptors are not a problem, but you cannot pass (host virtual) pointers from one process to another.
• How do you create something usable, like an EGLImageKHR or a GL texture, from the handle?

bluetoothd[280]: Refusing input device connect: Operation already in progress (114)
bluetoothd[280]: Refusing connection from ##:##:##:##:##:##: setup in progress
(bluetoothd:280): GLib-WARNING **: Invalid file descriptor.

• There are hooks for surface create and destroy, so you can track per-surface renderer private state.
• The attach hook is called when a new buffer is committed to a surface.
• The flush_damage hook is called only for wl_shm buffers, when the compositor is preparing to composite a surface. That is where e.g. GL texture updates happen in the GL renderer, and not on every commit, just in case the surface is not on screen right now.
• The surface_set_color callback informs the renderer that this surface will not be getting a buffer, but instead it must be painted with the given color. This is used for effects, like desktop fade-in and fade-out, by having a black full-screen solid color surface whose alpha channel is changed.
• The repaint_output is the workhorse of a renderer. In Weston core, weston_output_repaint() is called for each output when the output needs to be repainted. That calls into the backend's output repaint callback, which then calls the renderer's hook. The renderer then iterates over all surfaces in a list, painting them according to their state as needed.
• Finally, the read_pixels hook is for screen capturing.

Introduction

The problem

Enter sub-surfaces

Atomicity

Input handling considerations

Independent application sub-modules

Multiple input handlers

void pointer_enter(void *data, struct wl_pointer *wl_pointer, uint32_t serial, struct wl_surface *surface, wl_fixed_t surface_x, wl_fixed_t surface_y)

• Always be prepared to receive an unknown wl_surface on enter and similar events.
• When writing sub-modules and plugin interfaces, specify whether input is allowed, and whose responsibility is to set the input region to empty.

Out of scope

• Embedding content from other Wayland clients. The sub-surface extension does not implement any "foreign surface" interfaces, or anything like what X allows by just taking the Window XID and passing it to another client to use. The current consensus seems to be that this should be solved by implementing a mini-compositor in the hosting application.
• Clipping or scaling. The buffer you attach to a sub-surface will decide the size of the sub-surface. There is another extension coming for clipping and scaling.
• Any kind of message passing between application components. That is better solved in application specific ways.

Summary

Since the hardware very much matters this is going to be divided into a few parts, the common steps and the hardware specific ones.

Common steps

mkdir /opt/android
repo init -u https://android.googlesource.com/platform/manifest -b android-7.1.1_r28
cd /opt/android/.repo
git clone git://git.collabora.com/git/user/robertfoss/android_manifest.git local_manifests -b etnaviv-android
repo sync -j75

mkdir /opt/imx6_android
cp /opt/imx6_android
git clone git://git.collabora.com/git/user/robertfoss/linux.git -b imx_rdu2_v4.11-rc3

# The mkimage tool is used even if you're not
# using u-boot it as a bootloader
sudo apt install u-boot-tools

# Fetch Kconfig, bootloaders and some scripts
git clone git://git.collabora.com/git/user/robertfoss/rdu2.git .

# This will destroy all data …

There exists official documentation for how to create a custom boot animation, but unfortunately it is lacking in actual examples.

So this guide is a bit more hands on.

Structure of bootanimation.zip

Without covering too much of the same gound as the documentation, let's have a quick look at what is in a simple bootanimation.zip.

$ ls -la bootanimation
total 28
drwxr-xr-x 4 hottuna hottuna 4096 Apr 19 22:39 .
drwxr-xr-x 8 hottuna hottuna 4096 Apr 19 22:39 ..
-rw-r--r-- 1 hottuna hottuna   92 Apr 19 15:21 desc.txt
drwxr-xr-x 2 hottuna hottuna 4096 Apr 19 12:44 part0
drwxr-xr-x 2 hottuna hottuna 4096 Apr 19 12:45 part1

$ cat bootanimation/desc.txt 
1920 1080 30         # WIDTH HEIGHT FPS
c 5 15 part0 …

We, at Igalia, have been involved in enabling ARB_gpu_shader_fp64 extension to different Intel generations: first Broadwell and later, then Haswell. Now IvyBridge support is finished and landed Mesa’s master branch.

This feature was the last one to expose OpenGL 4.0 in Intel IvyBridge with the open-source Mesa driver. This is a big achievement for an old hardware generation (IvyBridge was released in 2012), which allows users to run OpenGL 4.0 games/apps on GNU/Linux without overriding the supported version with a Mesa-specific environment variable.

More goods news… ARB_vertex_attrib64 support has landed too, meaning that we are exposing OpenGL 4.2 on Intel Ivybrige!

Technical details

Diving a little bit into technical details (skip this if you are not interested on those)…

This work is standing on top of the shoulders of Intel Haswell support for ARB_gpu_shader_fp64. The latter introduced support of double floating-point (DF) data types on both scalar and vec4 backends which is, in general, very similar to Ivybridge. If you are interested in the technical details about adding ARB_gpu_shader_fp64 to Intel GPUs, see Iago’s talk at last XDC (slides here).

Ivybridge was the first Intel generation that supported double floating-point data types natively. The most important difference bettwen Ivybridge and Haswell is that both execution size and regioning parameters (stride and width) are in terms of 32-bits, so we need to double both regioning parameters and execution size at DF instructions’ emission.

But this is not the only annoyance, there are others quite relevant like:

• We emit an scalar DF instruction with a maximum execution size of 4 (doubled later to 8) to avoid hitting the gen7’s decompression instruction bug -present also in Haswell- that makes the hardware to read 2 consecutive GRFs regardless the vertical stride. This is specially annoying when reading DF scalars, because the stride is zero -we just want to read data from one GRF- and this bug would make us to read next GRF too; furthermore the hardware applies the same channel enable signals to both halves of the compressed instruction which will be just wrong under non-uniform control flow if force_writemask_all is disabled. However, there is also a physical limitation related to this when using Align16 access mode: SIMD8 is not allowed for DF operations. Also, in order to make DF instructions work under non-uniform control flow, we use NibCtrl to choose the proper flags of the execution mask.

• 32-bit data types to double (and vice-versa) conversions are quite special. Each 32-bit source data should be aligned 64-bits, so we need to apply an stride on the original data in order to keep this aligment. This is because the FPU internals cannot do the conversion if the data is not aligned to the size of the bigger one. A similar thing happens on converting doubles to 32-bit data types: the output elements need to be 64-bit aligned too.

• When splitting each DF instruction in two (or more) instructions with an exec_size of 4 as maximum, sometimes it is not so trivial to do and need temporary registers to save the intermediate results before merge them to the real destination.

Due to these things, we needed to improve the d2x lowering pass (now called lower_conversions) which fixes the aforementioned conversions from double floating-point data to 32-bit data types, add some specific fixes in the generator, add code in the validator to detect invalid cases, among other things.

In summary, although Ivybridge is very similar to Haswell, the 64-bit floating point support is quite special and it needs specific code.

Acknowledgements

I would like to thanks Matt Turner and Francisco Jerez from Intel for their insightful reviews, sometimes spotting problems that we did not foresee and their contributions to the patch series. Also, I would like to thanks Juan for his contributions to make this support happen and Igalia for allowing me to work on this amazing open-source project.

First, a meta update: I’ve moved away from my ancient Livejournal for hosting this blog. LJ recently updated their TOS with a bunch of moderately scary stuff about following Russian law, including requirements to register as a media outlet if your blog gets too popular. While Russian internet censorship probably wouldn’t ever apply to this blog, and it’s unlikely to become popular enough to be a media outlet, I don’t want to support them after this move.

I landed the VC4 V3D fencing code last week. This allows drivers like tinydrm (for the little SPI-attached panels for Raspberry Pi) or PL111 (for my bcm911360 phone) to correctly synchronize display pageflipping to V3D rendering. In the process of writing my V3D code, I found a bug and my reviewers found a cleanup, which I have also submitted for msm and etnaviv.

I also worked on fixing up the NEON tiling code for Mesa 17.1. With what I had landed recently, we weren’t doing runtime detection of NEON being present – I just used NEON if we were on an ARMv7 build. This works for Debian or Fedora’s builds that would support Pi 2/3, but not for Raspbian’s ARMv6 builds that support Raspberry Pi 0-3. I’ve got runtime detection now, and I’m just working on cleaning up all the ARMv6 and ARMv8 cases.

Boris submitted a patch for HDMI runtime power management. This will reduce power consumption (by an amount we haven’t measured) when the HDMI display is off on VC4, but will also result in being able to switch from HDMI to SDTV and back, which was previously broken. I’ll be testing and hopefully landing it this week.

I received an entertaining Mesa patch from the developers of Exagear: They wrote SSE code for V3D’s tiling format, to improve the performance of texture uploads in their x86 emulation environment on Raspberry Pi. We’re still doing patch review (their SSE code doesn’t runtime SSE detection yet), but it should be ready soon. Has anyone experimented with qemu-user for x86 emulation on Raspberry Pis?

Other projects last week: I reviewed and landed the STM32 DRM display driver, tested Meson upstream fixes for the X Server’s build system, reviewed a small performance fix for Mesa’s GL name lookups, and reviewed Gustavo’s patch series to clean up cursor handling on drivers like vc4.

Finally, something that’s not quite vc4-related but that I wanted to highlight: Daniel Vetter posted a patch to document the new code of conduct that applies to DRM kernel development. Many have been pushing for better behavior on our mailing lists (a process in which I’ve been an occasional participant), but until now it’s always been in the form of polite requests. The new patch gives us a bit of a stick for when people refuse to change their behavior. The response was clear, with 22 Reviewed-bys and Acked-bys flooding in in the next few days. This is a huge step forward for our community, and hopefully the patch will land this week.

The pl111 driver might be the most satisfying driver submission yet. Not because I’m desperate to see it in tree; I don’t actually have that hardware. Not because it’s bringing really exciting new capabilities. But, from Eric’s v5 submission:

v2: Nearly complete rewrite by anholt, cutting ²⁄₃ of the code thanks to DRM core’s excellent new helpers.

and a follow-up review:

I must say the driver is really slim and readable with all the new helpers from DRM, good job all who refactored the DRM support for simple framebuffer systems.

The DRM community really has come a long, long, way. Great to see it so thriving and healthy that people are actively dusting off ancient drivers which never got merged, deleting most of them in the process, and getting them in just because the process works so well.

On Friday, after consulting with the other freedesktop.org admins, I pushed a change to the freedesktop.org wiki, adding a Code of Conduct, based on the widely-used Contributor Covenant. In doing this, we join pretty much every other large open source project on the planet, with the exception of the Linux kernel, a magnificent anti-pattern. From this point on, all projects hosted on freedesktop.org are subject to this CoC.

why so broad?

fd.o is a notoriously loose collective of communities, who largely run their own affairs. However, freedesktop.org as a project is responsible for the content on our site: mailing list archives, bug tracking systems, website, etc. We’ve already had to intervene to remove legally problematic content from our hosting platforms; ultimately, it is our responsibility.

The culture of our member projects reflect on us as a wider organisation, and the problems of abusive and bullying behaviour weren’t solving themselves. In some specific cases we looked at, we were told directly by senior figures in the project that the lack of a defined fd.o-wide CoC made it harder for them to enforce it themselves.

In the end, the only course of action was completely clear: that we take the same approach to unacceptable behaviour as we do to legally-unacceptable content. Enforcing it across the platform gives everyone complete clarity of what’s required (i.e. behaving like reasonable human beings).

but the honeytrap / bad code

The notion that codes of conduct are used as a kind of submarine device to saddle communities with terrible code, and run completely excellent people out of the community for literally no reason, has been thoroughly debunked over the years that codes of conduct have been implemented. I don’t plan to give these arguments any time at all.

what now?

The conduct mailing list now exists, for confidential reports of any CoC violations. This is currently only manned by fd.o admins. We have, however, been in touch with some of the larger member projects, inviting them to help deal with conduct enforcement in their own projects. This process will take time, but if you’re interested in doing this for your project, please get in touch.

And hopefully, people continue to build healthy communities producing excellent code.

About a week ago there where 2 articles on LWN, the first coverging memory management patch review and the second covering the trouble with making review happen. The take away from these two articles seems to be that review is hard, there’s a constant lack of capable and willing reviewers, and this has been the state of review since forever. I’d like to counter pose this with our experiences in the graphics subsystem, where we’ve rolled out a well-working review process for the Intel driver, core subsystem and now the co-maintained small driver efforts with success, and not all that much pain.

tl;dr: require review, no exceptions, but document your expectations

Aside: This is written with a kernel focus, from the point of view of a maintainer or group of maintainers trying to establish review within their subsystem. But the principles really work anywhere.

Require Review

When review doesn’t happen, that generally means no one regards it as important enough. You can try to improve the situation by highlighting review work more, and giving less focus for top committer stats. But that only goes so far, in the end when you want to make review happen, the one way to get there is to require it.

Of course if that then results in massive screaming, then maybe you need to start with improving the recognition of review and value it more. Trying to put a new process into place over the persistent resistance is not going to work. But as long as there’s general agreement that review is good, this is the easy part.

No Exceptions

The trouble is that there’s a few really easy way to torpedo reviews before you event started, and they’re all around special priviledges and exceptions. From one of the LWN articles:

… requiring reviews might be fair, but there should be one exception: when developers modify their own code.

Another similar exception is often demanded by maintainers for applying their own patches to code they maintain - in the Linux kernel only about 25% of all maintainer patches have any kind of review tag attached when they land. This is in contrast to other contributors, who always have to get past at least their direct maintainer to get a patch applied.

There’s a few reasons why having exceptions for the original developer of some code, or a maintainer of a subsystem, is a really bad idea:

• Doing review (or at least full review) only for new contributors and people external to the subsystem makes it look like it’s just an elaborate hazing ritual, until you’re welcomed into the inner cabal. That tends to not go down too well with new folks, and not many want to help keep such a system running by actively contributing review. End result is that the pool of reviewers will stay limited to the old guard.

• Forcing even established contributors to go through review is a great opportunity for them to teach the art of a good review to someone new. Even when you write perfect code, eventually you’ll be gone, and then someone else needs to have understand your code. Not subjecting your own patches to review by others and new contributors drops one of the best mentoring opportunities on the floor we have in open source.

• And really, unicorns who always write perfect code don’t exist. At least I haven’t seen them yet …

On the flip side, requiring review from all your main contributors is a really easy way to kickstart a working review economy: Instantly you both have a big demand for review. And capable reviewers who are very much willing to trade a bit of review for getting reviews on their own patches.

Another easy pitfall is maintainers who demand unconditional NAck rights for the code they maintain, sometimes spiced up by claiming they don’t even need to provide reasons for the rejection. Of course more experienced people know more about the pitfalls of a code base, and hence are more likely to find serios defects in a change. But most often these rejections aren’t about clear bugs, but more design dogmas once established (and perhaps no longer valid), or just plain personal style preferences. Again, this is a great way to prevent review from happening:

• Anyone without NAck priviledges can only do second class review, and their review is then of course much less valued. Which means it won’t happen. Note this isn’t about experience, see above, review from new folks can still be useful, but only about who’s been around for longer, or who has the commit powers. Valuing only review from the old guard makes sure you won’t train new reviewers.

• It also curbs a working review economy, since non-maintainers can’t unblock patches. If only maintainers can provide real review, then you can’t trade reviews. Worse, non-maintainer reviewers might direct the author into a direction that the maintainer doesn’t approve of, wasting everyone’s time.

And again, I haven’t seen unicorns who write perfect code yet, neither have I seen someone who’s review feedback was consistently impeccable.

But Document your Expectations

Training reviews through direct mentoring is great, but it doesn’t scale. Document what you expect from a review as much as possible. This includes everything from coding style, to how much and in which detail code correctness should be checked. But also related things like documentation, test-cases, and process details on how exactly, when and where review is happening.

And like always, executable documentation is much better, hence try to script as much as possible. That’s why build-bots, CI bots, coding style bots, and all these things are possible - it frees the carbon-based reviewers from wasting time on the easy things and instead concentrate on the harder parts of review like code design and overall architecture, and how to best get there from the current code base. But please make sure your scripting and automated testing is of high-quality, because if the results need interpretation by someone experienced you haven’t gained anything. The kernel’s checkpatch.pl coding style checker is a pretty bad example here, since it’s widely accepted that it’s too opinionated and it’s suggestions can’t be blindly followed.

As some examples we have the dim ingloriuos maintainer scripts and some fairly extensive documentation on what is expected from reviewers for Intel graphics driver patches. Contrast that to the comparetively lax review guidelines for small drivers in drm-misc. At Intel we’ve also done internal trainings on review best practices and guidelines. Another big thing we’re working on on the automation front is CI crunching through patchwork series to properly regression test new patches before they land.

Audio Shop is a simple script that I cobbled together that gets you started with mangling image data as if it was audio data.

The script wraps 3 individually excellent tools; ffmpeg, ImageMagick and SoX.

The way it works is by first converting an image to a raw format like rgb or yuv. This is done to prevent the audio editor from destroying the structure of (relatively) complex formats like jpg, png or gif.

If converting to a raw format is the first step, the second step is importing the raw image data into the audio editor. To do this in way you can expect good results from, the raw format should use a bit-depth that your image editor can use. For example RGB, where …

Warning! This is not a directly Linux/Tech related blog post (it is not the only thing I care about in this world :).

One thing that has been on my mind for a while is the state of journalism. The quality of journalism seems to have
been declining over the last decade and I think it is clear that the new internet driven expectation that news content
is free for the consumer is a big part of the explanation. We all know that newspapers and TV news teams have seen their
staff cut as advertising revenue has not been strong enough to keep staffing up. And in my opinion advertising is in itself
a horrible way to finance something like news and information as it drivers a lot of unwanted behaviour both in terms of
avoiding critical journalism that might drive away advertisers and an intensive drive for ‘clicks’ per news article that often
comes at the expense of level of accuracy and making news very scandal driven.

So I have come to believe over the last few years that if we want to see quality journalism and a healthy democracy we need
to move away from free news content to accepting that we get what we pay for and start paying for our news again. This is true
both in terms of mainstream media, but also in terms of topical media like tech media.

So as a start I began paying for some of my most use news sites last year. I am now a paying subscriber to the Economist, which is one
of the best sources of quality news in my opinion and on the tech side I am a paying subscriber to Phoronix (I am also a paying subscriber to LWN through work). Anyway, I feel strongly enough about this to write this blog and hope that other people reading it agree with my thinking here and start paying for the content you enjoy, be that through subscriptions or patreons or similar. And maybe we can be part of a process to change the expectation and understanding of the value of well funded independent media. Lets help make news something that is made to help inform us as readers and not something that is made to help someone sell something to us.

So as most of you probably know Mark Shuttleworth just announced that they will be switching to GNOME 3 and Wayland again for Ubuntu. So I would like to on behalf of the Red Hat Desktop and Fedora teams to welcome them and say that we look forward to keep working with great Canonical and Ubuntu people like Allison Lortie and Robert Ancell on projects of shared interest around GNOME, Wayland and hopefully Flatpak.

It is worth mentioning that even as we been competing with Unity and Ubuntu we have also been collaborating with them, most recently on working with them to integrate the features they wanted from GNOME Software like the user reviews, but of course now sharing a bigger set of technologies collaboration will be even easier.

I am personally happy to see this convergence of efforts happening because I have for a long time felt that the general level of investment in the Linux desktop has not been great enough to justify the plethora of Linux desktops out there, so by now having reached a position where Canonical, Endless, Red Hat and Suse again share one desktop technology stack and along with consulting companies like Centricular, CodeThink, Collabora and Igalia helping push parts of the stack forward we are at least all pulling in the same direction.
This change should also make life easier for ISV who now have a more clear target if they want to try to integrate their UI with the Linux desktop as ‘the linux desktop’ becomes a more meaningful term with this change.

And to state the obvious, we will continue our effort around Fedora Workstation to continue to lead on innovation and engineering and setting the direction for where Desktop Linux goes in the future.

It’s time for a long-overdue blogpost about the status of Tanglu. Tanglu is a Debian derivative, started in early 2013 when the systemd debate at Debian was still hot. It was formed by a few people wanting to create a Debian derivative for workstations with a time-based release schedule using and showcasing new technologies (which include systemd, but also bundling systems and other things) and built in the open with a community using the similar infrastructure to Debian. Tanglu is designed explicitly to complement Debian and not to compete with it on all devices.

Tanglu has achieved a lot of great things. We were the first Debian derivative to adopt systemd and with the help of our contributors we could kill a few nasty issues affecting it and Debian before it ended up becoming default in Debian Jessie. We also started to use the Calamares installer relatively early, bringing a modern installation experience additionally to the traditional debian-installer. We performed the usrmerge early, uncovering a few more issues which were fed back into Debian to be resolved (while workarounds were added to Tanglu). We also briefly explored switching from initramfs-tools to Dracut, but this release goal was dropped due to issues (but might be revived later). A lot of other less-impactful changes happened as well, borrowing a lot of useful ideas and code from Ubuntu (kudos to them!).

On the infrastructure side, we set up the Debian Archive Kit (dak), managing to find a couple of issues (mostly hardcoded assumptions about Debian) and reporting them back to make using dak for distributions which aren’t Debian easier. We explored using fedmsg for our infrastructure, went through a long and painful iteration of build systems (buildbot -> Jenkins -> Debile) before finally ending up with Debile, and added a set of own custom tools to collect archive QA information and present it to our developers in an easy to digest way. Except for wanna-build, Tanglu is hosting an almost-complete clone of basic Debian archive management tools.

During the past year however, the project’s progress slowed down significantly. For this, mostly I am to blame. One of the biggest challenges for a young project is to attract new developers and members and keep them engaged. A lot of the people coming to Tanglu and being interested in contributing were unfortunately no packagers and sometimes no developers, and we didn’t have the manpower to individually mentor these people and teach them the necessary skills. People asking for tasks were usually asked where their interests were and what they would like to do to give them a useful task. This sounds great in principle, but in practice it is actually not very helpful. A curated list of “junior jobs” is a much better starting point. We also invested almost zero time in making our project known and create the necessary “buzz” and excitement that’s actually needed to sustain a project like this. Doing more in the advertisement domain and “help newcomers” area is a high priority issue in the Tanglu bugtracker, which to the day is still open. Doing good alone isn’t enough, talking about it is of crucial importance and that is something I knew about, but didn’t realize the impact of for quite a while. As strange as it sounds, investing in the tech only isn’t enough, community building is of equal importance.

Regardless of that, Tanglu has members working on the project, but way too few to manage a project of this magnitude (getting package transitions migrated alone is a large task requiring quite some time while at the same time being incredibly boring :P). A lot of our current developers can only invest small amounts of time into the project because they have a lot of other projects as well.

The other issue why Tanglu has problems is too much stuff being centralized on myself. That is a problem I wanted to rectify for a long time, but as soon as a task wasn’t done in Tanglu because no people were available to do it, I completed it. This essentially increased the project’s dependency on me as single person, giving it a really low bus factor. It not only centralizes power in one person (which actually isn’t a problem as long as that person is available enough to perform tasks if asked for), it also centralizes knowledge on how to run services and how to do things. And if you want to give up power, people will need the knowledge on how to perform the specific task first (which they will never gain if there’s always that one guy doing it). I still haven’t found a great way to solve this – it’s a problem that essentially kills itself as soon as the project is big enough, but until then the only way to counter it slightly is to write lots of documentation.

Last year I had way less time to work on Tanglu than the project deserves. I also started to work for Purism on their PureOS Debian derivative (which is heavily influenced by some of the choices we made for Tanglu, but with different focus – that’s probably something for another blogpost). A lot of the stuff I do for Purism duplicates the work I do on Tanglu, and also takes away time I have for the project. Additionally I need to invest a lot more time into other projects such as AppStream and a lot of random other stuff that just needs continuous maintenance and discussion (especially AppStream eats up a lot of time since it became really popular in a lot of places). There is also my MSc thesis in neuroscience that requires attention (and is actually in focus most of the time). All in all, I can’t split myself and KDE’s cloning machine remains broken, so I can’t even use that ;-). In terms of projects there is also a personal hard limit of how much stuff I can handle, and exceeding it long-term is not very healthy, as in these cases I try to satisfy all projects and in the end do not focus enough on any of them, which makes me end up with a lot of half-baked stuff (which helps nobody, and most importantly makes me loose the fun, energy and interest to work on it).

Good news everyone! (sort of)

So, this sounded overly negative, so where does this leave Tanglu? Fact is, I can not commit the crazy amounts of time for it as I did in 2013. But, I love the project and I actually do have some time I can put into it. My work on Purism has an overlap with Tanglu, so Tanglu can actually benefit from the software I develop for them, maybe creating a synergy effect between PureOS and Tanglu. Tanglu is also important to me as a testing environment for future ideas (be it in infrastructure or in the “make bundling nice!” department).

So, what actually is the way forward? First, maybe I have the chance to find a few people willing to work on tasks in Tanglu. It’s a fun project, and I learned a lot while working on it. Tanglu also possesses some unique properties few other Debian derivatives have, like being built from source completely (allowing us things like swapping core components or compiling with more hardening flags, switching to newer KDE Plasma and GNOME faster, etc.). Second, if we do not have enough manpower, I think converting Tanglu into a rolling-release distribution might be the only viable way to keep the project running. A rolling release scheme creates much less effort for us than making releases (especially time-based ones!). That way, users will have a constantly updated and secure Tanglu system with machines doing most of the background work.

If it turns out that absolutely nothing works and we can’t attract new people to help with Tanglu, it would mean that there generally isn’t much interest from the developer or user side in a project like this, so shutting it down or scaling it down dramatically would be the only option. But I do not think that this is the case, and I believe that having Tanglu around is important. I also have some interesting plans for it which will be fun to implement for testing

The only thing that had to stop is leaving our users in the dark on what is happening.

Sorry for the long post, but there are some subjects which are worth writing more than 140 characters about

If you are interested in contributing to Tanglu, get in touch with us! We have an IRC channel #tanglu-devel on Freenode (go there for quicker responses!), forums and mailinglists,

It looks like I will be at Debconf this year as well, so you can also catch me there! I might even talk about PureOS/Tanglu infrastructure at the conference.

So there is a thread on Hacker News based on a question from a Canonical employee asking for feedback on what people want from the next version of Ubuntu. I always try to read such threads even when they are not about Fedora or Red Hat. I fact I often read such articles and threads about non-Linux systems too to help understand what people are looking for and thus enable us to prioritize what we do with Fedora Workstation even better.

Fedora Workstation

Over the last few years I do feel we managed to nail down what the major pain points are and crossed them out one by one or gotten people assigned to work on them. So a lot of the items people asked for in that thread we already have in Fedora Workstation or have already in our roadmap. So I thought it would be nice to write them up and maybe encourage people to take a look at Fedora Workstation if you haven’t done so already. The list below is my trying to go through the long thread and pick up important and recurring topics, so I hope I got most of them, but if I missed something feel free to add a comment and I will try to answer.

1. Handling of DPI scaling and HiDPI
This has been something we been working on for quite a while. I think we where the first distribution to implemented general HiDPi and put a lot of engineering time into updating Wayland and GNOME to make it happen. That said things are not perfect yet. But we are working on resolving those. Jonas Ådahl and Rui Matos are currently trying to resolve the two main issues we still see. The first item is non-integer UI scaling. Currently we only offer integer scaling meaning that we only offer 2x scaling. This is to much however so we are working on a solution to offer fractional scaling like 1.5 for instance. We are certain to have that ready for Fedora Workstation 27, but there is a small hope we can finalize it already for Fedora Workstation 26. The other item is dealing with applications relying on XWayland because they do not support DPI scaling across two or more monitors, unlike native Wayland applications. We are dealing with that in two ways, one being working with upstreams to get their applications Wayland native like the work we been doing with LibreOffice and Firefox. We are also trying to come up with a scaling solution for XWayland using applications, but we haven’t been able to come up with a solution there yet.

2. Multitouch gestures like 3-finger swipe to change workspace.
This is another item we have put significant effort into. Over the last few years we made sure that we went from almost no touch support in the desktop to now supporting touch throughout the stack. The one big remaining item that was holding items like proper gestures back is that until kernel 4.12 is out the Synaptics touchpads are using PS/2. This causes only 2 touches to be reported, which is not great when you want 3 fingers gestures. With the new kernel, we will be using a different bus for those Synaptics touchpads, and we will have proper 5 fingers support. Benjamin Tissoires on our team has spent 4 years to get this code upstream but it’s finally here. We plan on backport this code to Fedora Workstation 26. Of course application developers will need to make use of the infrastructure in their applications for this feature to be fully realized everywhere.

Multitouch

3. Battery life
This is something we realize is a major issue and it has been on our agenda for a long time. It is a really hard issue to resolve because it is tied into a lot of things outside of control, like hardware used and in some cases third party drivers. That said, as many of you might know we recently set up a Laptop team here inside the bigger Red Hat desktop team and battery life is one of their top priorities. Christian Kellner is our point man on battery life and he has taken over the GNOME Battery bench tool that was originally created by Owen Taylor when we starting looking at battery life. He is currently working on improving GNOME battery bench and talking to hardware vendors to figure out what we can do. We are also actively speaking with NVidia to ensure that we can provide good battery life for hybrid graphics users when the binary NVidia driver is installed. We hope to agree with them on interfaces that should allow us to provide top notch battery life for such systems, but we are beholden to changes in the binary drivers to make that happen so it is also an example of the limits of what we can do on our own here.

GNOME Battery bench

4. UEFI issues
There where people on the Hacker News thread talking about issues with UEFI. Once again this is an area where we have a dedicated engineer assigned to UEFI and making sure it works great. In fact Peter Jones who is our UEFI point man is on the UEFI standards commitee doing ongoing work to ensure the standard is open source friendly and well supported by Linux. It is also worth mentioning that we created the Linux Vendor Firmware Service to make updating UEFI firmware and easy process. So if you see firmware updates offered in GNOME Software for your laptop or other devices that is because of the work we put into this service. We expect to have most of the major vendors signed up by the end of the year, so if your system is currently not supported that is hopefully a temporary thing. So this is both something that works well under Fedora and RHEL due to having someone dedicated to the effort and it is another example us doing the heavy lifting to make things actually happen.

UEFI Firmware updates

5. We got Wayland
A lot of people in the thread asked about Wayland support and well, we got Wayland! And this is another area where we dedicated serious engineering resources to it and are continuing to do so. For instance in addition to the above mentioned multi-DPI system work we are working on items such as HDR (High dynamic range) and next generation hybrid graphics support in Wayland. We are also working with NVidia to ensure their binary driver works well with Wayland.

Wayland Graphics

6. Something like Redshift
Some time ago we picked up on the growing popularity of tools such as Redshift and f.lux and this was another often repeated request in that Hacker News thread. Well we once again invested our resources into this and thus in the newly released GNOME 3.24 there is built in support for this feature, called Night Light. We drove this feature work and it will of course be available alongside GNOME 3.24 in Fedora Workstation 26.

Nightlight

7. Improved GPU driver update
This is another item we be spending significant time and resources on. We have a team dedicated to work on the linux graphics stack, which includes people like the graphics subsystem kernel maintainer and RADV creator Dave Airlie, Nouveau maintainer Ben Skeggs, core X, Mesa and Wayland dev Adam Jackson, Freedreeno creator and maintainer Rob Clark and more. This team is pushing the linux graphics stack forward alongside their colleagues at Intel, AMD and NVidia. The one thing we recently been working on for instance is dealing with the NVidia binary driver which has been a pain for a long time due to the file level conflict with Mesa. We didn’t want to do a workaround or hack, so what we did was work with NVidia on their glvnd proposal to make that a reality. This included supporting glvnd in Mesa in addition to the NVidia driver, but also working with the OS level tools to ensure fallbacks and autodetection worked fine. We got the basics of glvnd support already in Fedora and are polishing it up. Hans de Goede who took over that work from Adam Jackson has recently be working with the fine folks at rpmfusion and negativo17 to make sure we have some good packages available taking full advantage of his work, and thus enabling easy install and upgrade of these drivers. We are also planning to start offering a COPR with the latest and greatest Mesa drivers going forward to ensure you can always have the latest drivers available if you want to test and try them out.

NVidia driver install

8.Improved printer support
Even in this digital work printing is still important and thus we got people dedicated to this task too. So Marek Kasik is working on ensuring we keep CUPS working well and Felipe Borges recently wrote a blog entry talking about the redesigned printer control panel. So this is another area we are spending serious resources on and continuously trying to improve.

New Printer Panel

9. Improve Bluetooth
Bastien Nocera on our team is probably the person who has done the single most to make sure desktop bluetooth is working at all. We decided to boost that effort by having Christian Kellner work on this too, so he has been working on patches for various bluetooth related issues with Bastien providing guidance and code review. We are also working on coming up with some kind of bluetooth testing harness to allow us to catch regressions more easily and verify support on new hardware. Christian focus currently is improving the handling of Bluetooth Audio.

Summary
If you are contemplating giving Fedora a try I think the items above illustrate one thing very strongly and that is how many of these issues we are the primary force behind, so by using Fedora you are not only getting access to them first and at the same time have some assurance that the integration work has been done right, but you are also supporting the effort of moving these technologies forward and also putting yourself in a position to more directly interact with the engineers working on these and a long slew of other important technologies in the desktop and beyond. And our efforts are not just limited to writing code, like for example our current effort to clear the legal hurdles blocking Linux systems from supporting various media codecs. So if you haven’t already I strongly suggest you go to the Get Fedora website and grab our convenient Fedora installer or an ISO image. And as I said initially if you have other pressing items I didn’t cover here, feel free to post a comment and I will be happy to try to answer any questions I get.