planet.freedesktop.org
September 21, 2017

HUION PenTablet devices are graphics tablet devices aimed at artists. These tablets tend to aim for the lower end of the market, driver support is often somewhere between meh and disappointing. The DIGImend project used to take care of them, but with that out of the picture, the bugs bubble up to userspace more often.

The most common bug at the moment is a lack of proximity events. On pen devices like graphics tablets, we expect a BTN_TOOL_PEN event whenever the pen goes in or out of the detectable range of the tablet ('proximity'). On most devices, proximity does not imply touching the surface (that's BTN_TOUCH or a pressure-based threshold), on anything that's not built into a screen proximity without touching the surface is required to position the cursor correctly. libinput relies on proximity events to provide the correct tool state, which again is relied upon by compositors and clients.

The broken HUION devices only send BTN_TOOL_PEN once whenever the pen first goes into proximity and then never again until the device is disconnected. To make things more fun, HUION re-uses USB ids, so we cannot even reliably detect the broken devices and do the usual approach to hardware-quirking. So far, libinput support for HUION devices has thus been spotty. The good news is that libinput git master (and thus libinput 1.9) will have a fix for this. The one thing we can rely on is that tablets keep sending events at the device's scanout frequency. So in libinput we now add a timeout to the tablets and assume proximity-out has happened. libinput fakes a proximity out event and waits for the next event from the tablet - at which point we'll fake a proximity in before processing the events. This is enabled on all HUION devices now (re-using USB IDs, remember?) but not on any other device.

One down, many more broken devices more to go. Yay.

September 20, 2017
It's been a while since I posted about Fedora specific Bluetooth enhancements, and even longer that I posted about PlayStation controllers support.

Let's start with the nice feature.

Dual-Shock 3 and 4 support

We've had support for Dual-Shock 3 (aka Sixaxis, aka PlayStation 3 controllers) for a long while, but I've added a long-standing patchset to the Fedora packages that changes the way devices are setup.

The old way was: plug in your joypad via USB, disconnect it, and press the "P" button on the pad. At this point, and since GNOME 3.12, you would have needed the Bluetooth Settings panel opened for a question to pop up about whether the joypad can connect.

This is broken in a number of ways. If you were trying to just charge the joypad, then it would forget its original "console" and you would need to plug it in again. If you didn't have the Bluetooth panel opened when trying to use it wirelessly, then it just wouldn't have worked.

Set up is now simpler. Open the Bluetooth panel, plug in your device, and answer the question. You just want to charge it? Dismiss the query, or simply don't open the Bluetooth panel, it'll work dandily and won't overwrite the joypad's settings.


And finally, we also made sure that it works with PlayStation 4 controllers.



Note that the PlayStation 4 controller has a button combination that allows it to be visible and pairable, except that if the device trying to connect with it doesn't behave in a particular way (probably the same way the 25€ RRP USB adapter does), it just wouldn't work. And it didn't work for me on a number of different devices.

Cable pairing for the win!

And the boring stuff

Hey, do you know what happened last week? There was a security problem in a package that I glance at sideways sometimes! Yes. Again.

A good way to minimise the problems caused by problems like this one is to lock the program down. In much the same way that you'd want to restrict thumbnailers, or even end-user applications, we can forbid certain functionality from being available when launched via systemd.

We've finally done this in recent fprintd and iio-sensor-proxy upstream releases, as well as for bluez in Fedora Rawhide. If testing goes well, we will integrate this in Fedora 27.
September 19, 2017

In quite a few blog posts I been referencing Pipewire our new Linux infrastructure piece to handle multimedia under Linux better. Well we are finally ready to formally launch pipewire as a project and have created a Pipewire website and logo.Pipewire logo

To give you all some background, Pipewire is the latest creation of GStreamer co-creator Wim Taymans. The original reason it was created was that we realized that as desktop applications would be moving towards primarly being shipped as containerized Flatpaks we would need something for video similar to what PulseAudio was doing for Audio. As part of his job here at Red Hat Wim had already been contributing to PulseAudio for a while, including implementing a new security model for PulseAudio to ensure we could securely have containerized applications output sound through PulseAudio. So he set out to write Pipewire, although initially the name he used was PulseVideo. As he was working on figuring out the core design of PipeWire he came to the conclusion that designing Pipewire to just be able to do video would be a mistake as a major challenge he was familiar with working on GStreamer was how to ensure perfect audio and video syncronisation. If both audio and video could be routed through the same media daemon then ensuring audio and video worked well together would be a lot simpler and frameworks such as GStreamer would need to do a lot less heavy lifting to make it work. So just before we starting sharing the code publicaly we renamed the project to Pinos, named after Pinos de Alhaurín, a small town close to where Wim is living in southern Spain. In retrospect Pinos was probably not the worlds best name :)

Anyway as work progressed Wim decided to also take a look at Jack, as supporting the pro-audio usecase was an area PulseAudio had never tried to do, yet we felt that if we could ensure Pipewire supported the pro-audio usecase in addition to consumer level audio and video it would improve our multimedia infrastructure significantly and ensure pro-audio became a first class citizen on the Linux desktop. Of course as the scope grew the development time got longer too.

Another major usecase for Pipewire for us was that we knew that with the migration to Wayland we would need a new mechanism to handle screen capture as the way it was done under X was very insecure. So Jonas Ådahl started working on creating an API we could support in the compositor and use Pipewire to output. This is meant to cover both single frame capture like screenshot, to local desktop recording and remoting protocols. It is important to note here that what we have done is not just implement support for a specific protocol like RDP or VNC, but we ensured there is an advaned infrastructure in place to support any protocol on top of. For instance we will be working with the Spice team here at Red Hat to ensure SPICE can take advantage of Pipewire and the new API for instance. We will also ensure Chrome and Firefox supports this so that you can share your Wayland desktop through systems such as Blue Jeans.

Where we are now
So after multiple years of development we are now landing Pipewire in Fedora Workstation 27. This initial version is video only as that is the most urgent thing we need supported for Flatpaks and Wayland. So audio is completely unaffected by this for now and rolling that out will require quite a bit of work as we do not want to risk breaking audio on your system as a result of this change. We know that for many the original rollout of PulseAudio was painful and we do not want a repeat of that history.

So I strongly recommend grabbing the Fedora Workstation 27 beta to test pipewire and check out the new website at Pipewire.org and the initial documentation at the Pipewire wiki. Especially interesting is probably the pages that will eventually outline our plans for handling PulseAudio and JACK usecases.

If you are interested in Pipewire please join us on IRC in #pipewire on freenode. Also if things goes as planned Wim will be on Linux Unplugged tonight talking to Chris Fisher and the Unplugged crew about Pipewire, so tune in!

September 14, 2017
Introduction If you were waiting for a new post to learn something new about Intel GEN graphics, I am sorry. I started moving away from i915 kernel development about one year ago. Although I had worked landed patches in mesa before, it has taken me some time to get to a point where I had […]
My next project for Red Hat is to work on improving Linux laptop battery life. Part of the (hopefully) low hanging fruit here is using kernel tunables to
enable more runtime powermanagement. My first target here is SATA Link Power Management (LPM) which, as Matthew Garrett blogged about 2 years ago, can lead to a significant improvement in battery life.

There is only one small problem, there have been some reports that some disks/SSDs don't play well with Linux' min_power LPM policy and that this may lead to system crashes and even data corruption.

Let me repeat this: Enabling SATA LPM may lead to DATA CORRUPTION. So if you want to help with testing this please make sure you have recent backups! Note this happens only in rare cases (likely only with a coupe of specific SSD models with buggy firmware. But still DATA CORRUPTION may happen make sure you have BACKUPS.

As part of his efforts 2 years ago Matthew found this document which describes the LPM policy the Windows Intel Rapid Storage Technology (IRST) drivers use by default and most laptops ship with these drivers installed.

So based on an old patch from Matthew I've written a patch adding support for a new LPM policy called "med_power_with_dipm" to Linux. This saves
(almost) as much power as the min_power setting and since it matches Windows defaults I hope that it won't trip over any SSD/HDD firmware bugs.

So this is where my call for testers comes in, for Fedora 28 we would like to switch to this new SATA LPM policy by default (on laptops at least), but
we need to know that this is safe to do. So we are looking for people to help test this, if you have a laptop with a SATA drive (not NVME) and would like to help please make BACKUPS and then continue reading :)



First of all on a clean Fedora (no powertop --auto-tune, no TLP) do "sudo dnf install powertop", then close all your apps except for 1 terminal, maximimze that terminal and run "sudo powertop".

Now wait 5 minutes, on some laptops the power measurement is a moving average so this is necessary to get a reliable reading. Now look at the
power consumption shown (e.g. 7.95W), watch it for a couple of refreshes as it sometimes spikes when something wakes up to do some work, write down the lowest value you see, this is our base value for your laptops power consumption.

Next install the new kernel and try the new SATA LPM policy. I've done a scratch-build of the Fedora kernel with this patch added, which
you can download here.

After downloading all the .x86_64.rpm files there into a dir, do from this dir:
sudo rpm -ivh kernel*.x86_64.rpm

Next download a rc.local script applying the new settings from here, copy it to /etc/rc.d/rc.local, and make it executable: "sudo chmod +x /etc/rc.d/rc.local".

Now reboot and do: "cat /sys/class/scsi_host/host0/link_power_management_policy" this should return med_power_with_dipm, if not something is wrong.

Then close all your apps except for 1 terminal, maximimze that terminal and run "sudo powertop" again. Wait 5 minutes as last time, then get a couple of readings and write down the lowest value you see.

After this continue using your laptop as normal, please make sure that you keep running the special kernel with the patch adding the "med_power_with_dipm" policy. If after 2 weeks you've not noticed any bad side effects (or if you do notice bad side effects earlier) send me a mail at hdegoede@redhat.com with:

  • Report of success or bad side effects

  • The idle powerconsumption before and after the changes

  • The brand and model of your laptop

  • The output of the following commands:

  • cat /proc/cpuinfo | grep "model name"

  • cat /sys/class/scsi_device/*/device/model

I will gather the results in a table which will be part of the to-be-created Fedora 28 Changes page for this.

Did I mention already that although the chance is small something will go wrong, it is non zero and you should create backups ?

Thank you for your time.
September 12, 2017

So our graphics team is looking for a new Senior Software Engineer to help with our AMD GPU support, including GPU compute. This is a great opportunity to join a versatile and top notch development team who plays a crucial role in making sure Linux has up-to-date and working graphics support and who are deeply involved with most major new developments in Linux graphics.

Also as a piece of advice when you read the job advertisement remember that it is very rare anyone can tick all the boxes in the requirement list, so don’t hesitate to apply just because you don’t fit the description and requirements perfectly. For example even if you are more junior in terms of years you could still be a great candidate if you for instance participated in GPU related Google Summer of Code projects or just as a community contributor. And for this position we are open to candidates from around the globe interested in working as remotees, although as always if you are willing or interested in joining one of our development offices in either Boston-USA, Brisbane-Australia or Brno-Czech Republic that is a plus of course.

So please check out the job advertisement forSenior Software Engineer and see if it could be your chance to join the worlds premier open source company.

September 10, 2017

Downloads

If you're curios about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of Open Source Summit NA, for hosting a great community event.

September 09, 2017
I've been polishing a patch of Marek to enable out-of-order rasterization on VI+. Assuming it goes through as planned, this will be the first time we're adding driver-specific drirc configuration options that are unfamiliar to the enthusiast community (there's radeonsi_enable_sisched already, but Phoronix has reported on the sisched option often enough). So I thought it makes sense to explain what those options are about.

Background: Out-of-order rasterization

Out-of-order rasterization is an optimization that can be enabled in some cases. Understanding it properly requires some background on how tasks are spread across shader engines (SEs) on Radeon GPUs.

The frontends (vertex processing, including tessellation and geometry shaders) and backends (fragment processing, including rasterization and depth and color buffers) are spread across SEs roughly like this:


(Not shown are the compute units (CUs) in each SE, which is where all shaders are actually executed.)

The input assembler distributes primitives (i.e., triangles) and their vertices across SEs in a mostly round-robin fashion for vertex processing. In the backend, work is distributed across SEs by on-screen location, because that improves cache locality.

This means that once the data of a triangle (vertex position and attributes) is complete, most likely the corresponding rasterization work needs to be distributed to other SEs. This is done by what I'm simplifying as the "crossbar" in the diagram.

OpenGL is very precise about the order in which the fixed-function parts of fragment processing should happen. If one triangle comes after another in a vertex buffer and they overlap, then the fragments of the second triangle better overwrite the corresponding fragments of the first triangle (if they weren't rejected by the depth test, of course). This means that the "crossbar" may have to delay forwarding primitives from a shader engine until all earlier primitives (which were processed in another shader engine) have been forwarded. This only happens rarely, but it's still sad when it does.

There are some cases in which the order of fragments doesn't matter. Depth pre-passes are a typical example: the order in which triangles are written to the depth buffer doesn't matter as long as the "front-most" fragments win in the end. Another example are some operations involved in stencil shadows.

Out-of-order rasterization simply means that the "crossbar" does not delay forwarding triangles. Triangles are instead forwarded immediately, which means that they can be rasterized out-of-order. With the in-progress patches, the driver recognizes cases where this optimization can be enabled safely.

By the way #1: From this explanation, you can immediately deduce that this feature only affects GPUs with multiple SEs. So integrated GPUs are not affected, for example.

By the way #2: Out-of-order rasterization is entirely disabled by setting R600_DEBUG=nooutoforder.


Why the configuration options?

There are some cases where the order of fragments almost doesn't matter. It turns out that the most common and basic type of rendering is one of these cases. This is when you're drawing triangles without blending and with a standard depth function like LEQUAL with depth writes enabled. Basically, this is what you learn to do in every first 3D programming tutorial.

In this case, the order of fragments is mostly irrelevant because of the depth test. However, it might happen that two triangles have the exact same depth value, and then the order matters. This is very unlikely in common scenes though. Setting the option radeonsi_assume_no_z_fights=true makes the driver assume that it indeed never happens, which means out-of-order rasterization can be enabled in the most common rendering mode!

Some other cases occur with blending. Some blending modes (though not the most common ones) are commutative in the sense that from a purely mathematical point of view, the end result of blending two triangles together is the same no matter which order they're blended in. Unfortunately, additive blending (which is one of those modes) involves floating point numbers in a way where changing the order of operations can lead to different rounding, which leads to subtly different results. Using out-of-order rasterization would break some of the guarantees the driver has to give for OpenGL conformance.

The option radeonsi_commutative_blend_add=true tells the driver that you don't care about these subtle errors and will lead to out-of-order rasterization being used in some additional cases (though again, those cases are rarer, and many games probably don't encounter them at all).

tl;dr

Out-of-order rasterization can give a very minor boost on multi-shader engine VI+ GPUs (meaning dGPUs, basically) in many games by default. In most games, you should be able to set radeonsi_assume_no_z_fights=true and radeonsi_commutative_blend_add=true to get an additional very minor boost. Those options aren't enabled by default because they can lead to incorrect results.
September 05, 2017

This week I worked on stabilizing the VC5 DRM. I got the MMU support almost working – rendering triangles, but eventually crashing the kernel, and then decided to simplify my plan a bit. First, a bit of background on DRM memory management:

One of the things I’m excited about for VC5 is having each process have a separate address space. It has always felt wrong to me on older desktop chips that we would let any DRI client read/write the contents of other DRI clients’ memory. We just didn’t have any hardware support to help us protect them, without effectively copying out and zeroing the other clients’ on-GPU memory. With i915 we gained page tables that we could swap out, at least, but we didn’t improve the DRM to do this for a long time (I’m actually not entirely sure if we even do so now). One of our concerns was that it could be a cost increase when switching between clients.

However, one of the benefits of having each process have a separate address space is that then we can give the client addresses for its buffer that it can always rely on. Instead of each reference to a buffer needing to be tracked so that kernel (or userspace, with new i915 ABI) can update them when the buffer address changes, we just keep track of the list of all buffers that have been referenced and need to be at their given offsets. This should be a win for CPU overhead.

What I built at first was each VC5 DRM fd having its own page table and address space that it would bind buffers into. The problem was that the page tables are expensive (up to 4MB of contiguous memory), and so I’d need to keep the page tables separate from the address spaces so that I could allocate them on demand.

After a bit of hacking on that plan, I decided to simplify things for now. Given that today’s boards have less than 4 GB of memory, I could have all the clients share the same page tables for a 4GB address space and use the GMP (an 8kb bitmask for 128kb-granularity access control) to mask each client’s access to the address space. With a shared page table and the GMP disabled I soon got this stable enough to do basic piglit texturing tests reliably. The GMP plan should also reduce our context switch overhead, by replacing the small 8kb GMP memory instead of flushing all the MMU caches.

Somewhere down the line, if we find we need 4GB per client, we can build kernel support to have clients that exhaust the shared table get kicked out to their own page tables.

Next up for the kernel module: GPU reset (so I can start running piglit tests and settling my V3D DT binding), and filling out those GMP bitmasks.

The last couple of days of the week were spent on forking the anv driver to start building a VC5 Vulkan driver (“bcmv”). I’ll talk about it soon (next week is my bike trip, so no TWIV then, and after that is XDC 2017).

August 31, 2017

I would like to thank the wonderful organizers, GDG Berlin Android, for hosting a great community event.

Downloads

If you're curios about the slides, you can download the PDF or the OTP.

August 29, 2017

The All Systems Go! 2017 Call for Participation is Closing on September 3rd!

Please make sure to get your presentation proprosals forAll Systems Go! 2017 in now! The CfP closes on sunday!

In case you haven't heard about All Systems Go! yet, here's a quick reminder what kind of conference it is, and why you should attend and speak there:

All Systems Go! is an Open Source community conference focused on the projects and technologies at the foundation of modern Linux systems — specifically low-level user-space technologies. Its goal is to provide a friendly and collaborative gathering place for individuals and communities working to push these technologies forward. All Systems Go! 2017 takes place in Berlin, Germany on October 21st+22nd. All Systems Go! is a 2-day event with 2-3 talks happening in parallel. Full presentation slots are 30-45 minutes in length and lightning talk slots are 5-10 minutes.

In particular, we are looking for sessions including, but not limited to, the following topics:

  • Low-level container executors and infrastructure
  • IoT and embedded OS infrastructure
  • OS, container, IoT image delivery and updating
  • Building Linux devices and applications
  • Low-level desktop technologies
  • Networking
  • System and service management
  • Tracing and performance measuring
  • IPC and RPC systems
  • Security and Sandboxing

While our focus is definitely more on the user-space side of things, talks about kernel projects are welcome too, as long as they have a clear and direct relevance for user-space.

To submit your proposal now please visit our CFP submission web site.

For further information about All Systems Go! visit our conference web site.

systemd.conf will not take place this year in lieu of All Systems Go!. All Systems Go! welcomes all projects that contribute to Linux user space, which, of course, includes systemd. Thus, anything you think was appropriate for submission to systemd.conf is also fitting for All Systems Go!

August 28, 2017

This week was fairly focused on getting my BCM7268 board up and running and building the kernel module for it. I first got the board booting to an NFS root on an upstream kernel – huge thanks to the STB team for the upstreaming work they’ve done, so that I could just TFTP load an upstream kernel and start working.

The second half of the week was building the DRM module. I started with copying from vc4, since our execution model is very similar. We have an MMU now, so I replaced the CMA helpers with normal shmem file allocation, and started building bits we’ll need to have a page table per DRM file descriptor. I also ripped out the display components – we’re 3D only now, and any display hardware would be in a separate kernel module and communicate through dma-bufs. Finally, I put together a register header, debugfs register decode, and part of the interrupt handlers. Next week it’s going to be getting our first interrupts handled and filling out the MMU, then I should be on to trying to actually paint some pixels!

August 27, 2017
Occasionally the question comes up about why we convert between various IR's (intermediate representations), like glsl to NIR, in the process of compiling a shader.  Wouldn't it be faster if we just skipped a step and went straight from glsl to "the final thing", which would be ir3 (freedreno), codegen (nouveau), or LLVM (radeonsi/radv).  It is a reasonable question, since most people haven't worked on compilers and we probably haven't done a good job at explaining all the various passes involved in compiling a shader or presenting a breakdown of where the time is spent.

So I spent a bit of time this morning with perf to profile a shader-db run (or rather a subset of a full run to keep the perf.data size manageable, see notes at end).
A flamegraph from the shader-db run, since every blog post needs a catchy picture.

Breakdown:

  • parser, into glsl: 9.98%
  • glsl to nir: 1.3%
  • nir opt/lowering passes: 21.4%
    • CSE: 6.9%
    • opt algebraic: 3.5%
    • conversion to SSA: 2.1%
    • DCE: 2.0%
    • copy propagation: 1.3%
    • other lowering passes: 5.6%
  • nir to ir3: 1.5%
  • ir3 passes:  21.5%
    • register allocation: 5.1%
    • sched: 14.3%
    • other: 2.1%
  • assembly (ir3->binary): 0.66%
This is ignoring some of the fixed overheads of shader-db runner, and also doesn't capture individually a bunch of NIR lowering passes.  NIR has ~40 lowering passes, some that are gl related like nir_lower_draw_pixels and nir_lower_wpos_ytransform (because for hysterical reasons textures and therefore FBO's are upside down in gl).  For gallium drivers using NIR, these gl specific passes are called from mesa state-tracker.

The other lowering passes are not gl specific but tend to be specific to general GPU shader features (ie. things that you wouldn't find in a C compiler for a cpu) and things that are needed by multiple different drivers.  Such as, nir_lower_tex which handles sampling from YUV textures, ie. inserting the instructions to do YUV->RGB conversion (since GLES and android strongly assume this is a thing that hardware can always do), lowering RECT textures, or clamping texture coords.  These lowering passes are called from the driver backend so the driver is in control of what lowering pass are needed, including configuration about individual features in passes which handle multiple things, based on what the hardware does not support directly.

These lowering passes are mostly O(n), and lost in the noise.

Also note that freedreno, along with the other drivers that can consume NIR directly, disable a bunch of opt passes that were originally done in glsl, but that NIR (or LLVM) can do more efficiently.  For freedreno, disabling the glsl opt passes shaved ~30% runtime off of a shader-db run, so spending 1.3% to convert into NIR is way more than offset.

For other drivers, the breakdown may be different.  I expect radeonsi/radv skips some of the general opt passes in NIR which have a counterpart in LLVM, but re-uses other lowering passes which do not have a counterpart in LLVM.


Is it still a gallium driver?


This is a related question that comes up sometimes, is it a gallium driver if it doesn't use TGSI?  Yes.

The drivers that can consume NIR and implement the gallium pipe driver interface, freedreno a3xx+, vc4, vc5, and radeonsi (optionally), are gallium drivers.  They still have to accept TGSI for state trackers which do not support NIR, and various built-in shaders (blits, mipmap generation, etc).  Most use the shared tgsi_to_nir pass for TGSI shaders.  Note that currently tgsi_to_nir does not support all the TGSI features, but just features needed by internal shaders, and what is needed for gl3/gles3 (ie. basically what freedreno and vc4 needed before mesa state-tracker grew support for glsl_to_nir).


Notes:


Collected from shader-db run (glamor + supertuxkart + 0ad shaders) with a debug mesa build (to have debug syms and prevent inlining) but with NIR_VALIDATE=0 (otherwise results with debug builds are highly skewed).  A subset of all shader-db shaders was used to keep the perf.data size manageable.
August 25, 2017

In this last week of my GSoC project I aimed at bringing my code into its final form. The goals for that are simple:

  • Test the code, fix bugs and protocol any remaining issues.
  • Remove any coding style misdemeanors.
  • Since in the end my changes to the Xwayland as well as the Present code became pretty extensive, portion them into multiple patches so they are easier to review.

Regarding the remaining issues in my test scenarios there are still some problems with some of my test applications. As long as the underlying problem isn’t identified the question here is always though, if my code is the problem or the application is not working correctly. For example with KWin’s Wayland session there is a segmentation fault at the end of its run time whenever at some point during run time an Xwayland based client did a flip on a sub-surface. But according to GDB the segmentation fault is caused by KWin’s own egl context destruction and happens somewhere in the depths of the EGL DRI2 implementation in MESA.

I tried to find the reasons for some difficile issues like this one in the last few weeks, but at some point for a few of them I had to admit that either the tools I’m using aren’t capable enough to hint me in the right direction or I’m simply not knowledgeable enough to know where to look at. I hope that after I have sent in the patches in the next few days I get some helpful advice from senior developers who know the code better than I do and I can then solve the remaining problems.

The patches I will send to the mailing list in the next days are dependent on each other. As an overview here is how I planned them:

  1. Introduces an internal API to the Xserver’s Present implementation. The goal of this API is to make it easy to add other flipping modes than just full screen flips. While the previous code for full screen flips needs to be adapted to this API this patch doesn’t yet alter any functionality.
  2. Distribute more code onto several new files to improve readability.
  3. Add a window flipping mode to Present using the new internal API.
  4. Let Xwayland use the window flipping mode.

This partitioning will hopefully help the reviewers. The other advantage is that mistakes I may have made in one of these patches and have been overlooked in the review process might be easier to get found afterwards. Splitting my code changes up into smaller units also gives me the opportunity to look into the structure of my code from a different perspective one more time and fix details I may have overlooked until now.

I hope being able to send the patches in tomorrow. I’m not sure if I’m supposed to write one more post next week after the GSoC project has officially ended. But in any case I plan on writing one more post at some point in the future about the reaction to my patches and if and how they got merged in the end.

August 23, 2017
This is the last entry in the Google Summer of Code series that I have been writing weekly for the last three months. It is different from the usual updates in that I won’t be discussing development progress: rather, this will be the submission report for the project as a whole. I’ll be discussing the why? behind the project, the plan that my mentor and I came up with to execute the project, the work I have done over the summer including a video of the result, the things that are left to work on, what I’ve learned during the project and finally, the links to the code that I have written for the actual submission.
Well, this really is the last week of my Summer of Code! This will be a short update on the final changes made since my last update on Friday, which will be followed by a proper blog post for my final submission later this week. Last week I mentioned a few items that were work-in-progress. The search field for the button mapping dialog has been merged, including its revamp of the dialog’s UI.
August 21, 2017

This week I finished fixing regressions from the VC5 QPU instruction scheduler, and polished the vc5 series up so I could post it for review in Mesa. I landed a few initial bits that wouldn’t affect anyone else.

I also took another look at my DSI series. I had previously tried to work around the boot time dependency circle by letting the DSI device be created before the DSI host showed up. That proved to be fragile as well (in particular it would have had issues if the host had to -EPROBE_DEFER for some other reason), so I went back and made VC4 advertise a DSI host before any of the rest of the DSI encoder was ready. This proved to be not very hard, and I’m hoping once again that this is the final version of the series.

Other bits this week:

  • Fixed tile layout for 64bpp textures on VC5.
  • Fixed DSI transactions sleeping in the IRQ handler.
  • Did some investigation of HW behavior for upstreaming Pi3 BT
  • Sent a second set of PRs for bcm2835 features in 4.14
  • Landed NEON build support for ARMv6 and Jonas’s ARM64 port
  • Landed HDMI EDID leak fix
  • Landed X Server scissoring optimizations for VC4.
  • Fixed and resubmitted GL_OES_required_internalformat support (part of vc4 GLES2 DEQP fixes)
August 18, 2017

The last week of GSoC 2017 is about to begin. My project is in a pretty good state I would say: I have created a big solution for the Xwayland Present support, which is integrated firmly and not just attached to the main code path like an afterthought. But there are still some issues to sort out. Especially the correct cleanup of objects is difficult. That’s only a problem with sub-surfaces though. So, if I’m not able to solve these issues in the next few days I’ll just allow full window flips. This would still include all full screen windows and for example also the Steam client as it’s directly rendering its full windows without usage of the compositor.

I still hope though to solve the last problems with the sub-surfaces, since this would mean that we can in all direct rendering cases on Wayland use buffer flips, which would be a huge improvement compared to native X.

In any case at first I’ll send the final patch for the Present extension to the xorg-devel mailing list. This patch will add a separate mode for flips per window to the extension code. After that I’ll send the patches for Xwayland, either with or without sub-surface support.

That’s already all for now as a last update before the end and with still a big decision to be made. In one week I can report back on what I chose and how the final code looks like.

This week I have been solving more issues to make sure that Piper offers a pleasent user experience, doesn’t crash and runs smoothly. My mentor and cvuchener from the libratbag project have been testing Piper the last week, and together with a handful of users attracted to Piper they have opened a bunch of issues for me to solve. Let’s run through the most visible ones! Solving global warming Probably the most fun issue to resolve this week was the one reported by my mentor: Piper contributes to global warming (you won’t believe what happened next!
August 16, 2017
The indoor garden at Collège de Maisonneuve, the DebConf 17 venue
Decorative photo of the indoor garden

I'm currently at DebConf 17 in Montréal, back at DebConf for the first time in 10 years (last time was DebConf 7 in Edinburgh). It's great to put names to faces and meet more of my co-developers in person!

On Monday I gave a talk entitled “A Debian maintainer's guide to Flatpak”, aiming to introduce Debian developers to Flatpak, and show how Flatpak and Debian (and Debian derivatives like SteamOS) can help each other. It seems to have been quite well received, with people generally positive about the idea of using Flatpak to deliver backports and faster-moving leaf packages (games!) onto the stable base platform that Debian is so good at providing.

A video of the talk is available from the Debian Meetings Archive. I've also put up my slides in the DebConf git-annex repository, with some small edits to link to more source code: A Debian maintainer's guide to Flatpak. Source code for the slides is also available from Collabora's git server.

The next step is to take my proof-of-concept for building Flatpak runtimes and apps from Debian and SteamOS packages, flatdeb, get it a bit more production-ready, and perhaps start publishing some sample runtimes from a cron job on a Debian or Collabora server. (By the way, if you downloaded that source right after my talk, please update - I've now pushed some late changes that were necessary to fix the 3D drivers for my OpenArena demo.)

I don't think Debian will be going quite as far as Endless any time soon: as Cosimo outlined in the talk right before mine, they deploy their Debian derivative as an immutable base OS with libOSTree, with all the user-installable modules above that coming from Flatpak. That model is certainly an interesting thing to think about for Debian derivatives, though: at Collabora we work on a lot of appliance-like embedded Debian derivatives, with a lot of flexibility during development but very limited state on deployed systems, and Endless' approach seems a perfect fit for those situations.

[Edited 2017-08-16 to fix the link for the slides, and add links for the video]

August 14, 2017
I recently acquired an r7 360 (BONAIRE) and spent some time getting radv stable and passing the same set of conformance tests that VI and Polaris pass.

The main missing thing was 10-bit integer format clamping for a bug in the SI/CIK fragment shader output hardware, where it truncates instead of clamps. The other missing piece was code for handling f16->f32 conversions according to the vulkan spec that I'd previously fixed for VI.

I also looked at a trace from amdgpu-pro and noticed it was using a ds_swizzle for the derivative calculations which avoided accessing LDS memory. I wrote support to use this path for radv/radeonsi since LLVM supported the intrinsic for a while now.

With these fixed CIK is pretty much in the same place as VI/Polaris.

I then plugged in my SI (Tahiti), and got lots of GPU hangs and crashes. I fixed a number of SI specific bugs (tiling and MSAA handling, stencil tiling). However even with those fixed I was getting random hangs, and a bunch of people on a bugzilla had noticed the same thing. I eventually discovered adding a shader pipeline and cache flush at the end of every command buffer (this took a few days to narrow down exactly). We aren't 100% sure why this is required on SI only, it may be a kernel bug, or a command processor bug, but it does mean radv on SI now can run games without hanging.

There are still a few CTS tests outstanding on SI only, and I'll probably get to them eventually, however I also got an RX Vega and once I get a newer BIOS for it from AMD I shall be spending some time fixing the radv support for it.

This week was spent almost entirely on the VC5 QPU instruction scheduler. This is what packs together the ADD and MUL instruction components and the signal flags (LDUNIF, LDVPM, LDTMU, THRSW, etc.) into the final sequence of 64-bit instructions.

I based it on the VC4 scheduler, which in turn is based on i965’s. Being the 5th of these I’ve worked on, it would sure be nice if it felt like less copy and paste, but it’s almost all very machine-dependent code and I haven’t come up with a way to reduce the duplication.

The initial results are great:

instructions in affected programs:     116269 -> 71677 (-38.35%)

but I’ve still got a handful of regressions to fix.

Next up for scheduling is to fill the thrsw and branch delay slots. I also need to pack more than 2 things into one QPU instruction – right we pick two of ADD, MUL, LDUNIF, and LDVPM, but we could do an ADD, MUL, LDVPM, and LDUNIF all together. That should be a small change from here.

Other bits this week:

  • Fixed a leak of the EDID data from the HDMI connector (~256 bytes per xrandr -q invocation)
  • Reviewed modifier blob support from daniels for better atomic-modesetting weston on vc4
  • Pushed the RCL walk order DRM support.
  • Reworked patches for turning on vc4 NEON support in Raspbian builds, to fix Android builds.
August 11, 2017

One more time I decided to start from the beginning and try another even more radical approach to my Xwayland GSoC project than the last time. I have now basically written a full API inside the Present extension, with which modes of presentation can be added. There are of course only two modes right now: The default screen presenting mode as how it worked until now and the new one for Xwayland to present on individual windows and without the need of them being full screen. While this was also possible with the version from last week, the code is now substantially better structured.

I’m currently still in a phase of testing so I won’t write much more about it for now. Instead I want to talk about one very persistent bug, which popped up seemingly from one day to the other in my KWin session and hindered my work immensely.

This is also a tale of how difficult it can be to find the exact reason for a bug. Especially when there is not much information to work with: As said the problem came out of nowhere. I had used Neverball to test my Xwayland code in the past all the time. But starting a few days ago whenever I selected a level and the camera was to pan down to the level the whole KWin session blocked and I could only hard reboot the whole computer or SIGKILL the KWin process via SSH. The image of the level was frozen and keyboard inputs didn’t work anymore. That said I still heard the Neverball music playing in the background, so the application wasn’t the problem. And Xwayland or KWin didn’t quit with a segfault, they just stopped doing stuff.

So I began the search. Of course I first suspected my own code to be the problem. But when I tried the Xwayland master branch I experienced the same problem. But please, what was that? Why suddenly didn’t Neverball work at all anymore? I had used it in the past all the time, but now everything blocks? So I tried first to roll back commits in the master branches of Xwayland, KWin, KWayland in the last few weeks, thinking that the problem must have been introduced at that point in time because I could still use Neverball just recently without problems, right?

But the problem was still there. So I went further back. It worked with my distribution’s Xwayland and when manually testing through the input related commits to Xwayland starting from somewhere at the beginning of the year I finally found the responsible commit, or so I thought. But yeah, before that commit no blocking, with it blocking, so there has to be an error with this particular commit, right? But on the other side why could I use Neverball just one week ago without blocking and this commit is several months old?

Nevertheless I studied the Xwayland input code thoroughly. The documentation for this stuff is non-existent and the function patterns confusing, but with time I understood it well enough to decide that this couldn’t be the cause of the problem. Another indicator for that was, that Weston worked. The problem had to be in KWin or KWayland somewhere. After lots of time I also finally understood somewhat why I still could use Neverball a few days ago but now not at all anymore: I always started KWin from the terminal before that without launching a full Wayland Plasma session. But after here everything worked fine I switched to testing it in the Plasma session and apparently missed that from now on the problem existed. So was it Plasma itself? But this wasn’t really possible, since Plasma is a different process.

I didn’t want to give up and so I looked through the KWayland and KWin code related to pointer locking and confinement, which is a lot. Hours later I finally found the root cause: KWin creates small on screen notifications when a pointer is locked or confined to a window. Most of the time this works without problem, but with the above patch to Xwayland the client sends in quick succession the pointer confine and lock requests to KWin and for some reason when trying to show both notifications at the same time KWin or maybe the QML engine for the notification can’t process any further. Without the patch Xwayland always only sent the confinement request and nothing blocked. I don’t know how Martin would like to have this issue solved so I created a bug report for now. It’s weird that it was such a petty cause in the end with such huge consequences, but that’s how it goes.

Last week I shared the news that all large features had been implemented; all that was left were issues raised by GNOME contributors when my mentor demoed my progress over GUADEC. This week I’ve just been chugging away on those issues; let me show you. Obvious save button is.. not obvious? Hadess noted that the save button wasn’t obvious when my mentor demoed Piper to him; I already mentioned this last week: the icon is a save to disk icon and it’s not obvious at first glance that you have to press it to write the changes you made to the device.
August 09, 2017

The All Systems Go! 2017 Headline Speakers Announced!

Don't forget to send in your submissions to the All Systems Go! 2017 CfP! Proposals are accepted until September 3rd!

A couple of headline speakers have been announced now:

  • Alban Crequy (Kinvolk)
  • Brian "Redbeard" Harrington (CoreOS)
  • Gianluca Borello (Sysdig)
  • Jon Boulle (NStack/CoreOS)
  • Martin Pitt (Debian)
  • Thomas Graf (covalent.io/Cilium)
  • Vincent Batts (Red Hat/OCI)
  • (and yours truly)

These folks will also review your submissions as part of the papers committee!

All Systems Go! is an Open Source community conference focused on the projects and technologies at the foundation of modern Linux systems — specifically low-level user-space technologies. Its goal is to provide a friendly and collaborative gathering place for individuals and communities working to push these technologies forward.

All Systems Go! 2017 takes place in Berlin, Germany on October 21st+22nd.

To submit your proposal now please visit our CFP submission web site.

For further information about All Systems Go! visit our conference web site.

August 08, 2017

A while back at the awesome maintainerati I chatted with a few great fellow maintainers about how to scale really big open source projects, and how github forces projects into a certain way of scaling. The linux kernel has an entirely different model, which maintainers hosting their projects on github don’t understand, and I think it’s worth explaining why and how it works, and how it’s different.

Another motivation to finally get around to typing this all up is the HN discussion on my “Maintainers Don’t Scale” talk, where the top comment boils down to “… why don’t these dinosaurs use modern dev tooling?”. A few top kernel maintainers vigorously defend mailing lists and patch submissions over something like github pull requests, but at least some folks from the graphics subsystem would love more modern tooling which would be much easier to script. The problem is that github doesn’t support the way the linux kernel scales out to a huge number of contributors, and therefore we can’t simply move, not even just a few subsystems. And this isn’t about just hosting the git data, that part obviously works, but how pull requests, issues and forks work on github.

Scaling, the Github Way

Git is awesome, because everyone can fork and create branches and hack on the code very easily. And eventually you have something good, and you create a pull request for the main repo and get it reviewed, tested and merged. And github is awesome, because it figured out an UI that makes this complex stuff all nice&easy to discover and learn about, and so makes it a lot simpler for new folks to contribute to a project.

But eventually a project becomes a massive success, and no amount of tagging, labelling, sorting, bot-herding and automating will be able to keep on top of all the pull requests and issues in a repository, and it’s time to split things up into more manageable pieces again. More important, with a certain size and age of a project different parts need different rules and processes: The shiny new experimental library has different stability and CI criteria than the main code, and maybe you have some dumpster pile of deprecated plugins that aren’t support, but you can’t yet delete them: You need to split up your humongous project into sub-projects, each with their own flavour of process and merge criteria and their own repo with their own pull request and issue tracking. Generally it takes a few tens to few hundreds of full time contributors until the pain is big enough that such a huge reorganization is necessary.

Almost all projects hosted on github do this by splitting up their monorepo source tree into lots of different projects, each with its distinct set of functionality. Usually that results in a bunch of things that are considered the core, plus piles of plugins and libraries and extensions. All tied together with some kind of plugin or package manager, which in some cases directly fetches stuff from github repos.

Since almost every big project works like this I don’t think it’s necessary to delve on the benefits. But I’d like to highlight some of the issues this is causing:

  • Your community fragments more than necessary. Most contributors just have the code and repos around that they directly contribute to, and ignore everything else. That’s great for them, but makes it much less likely that duplicated effort and parallel solutions between different plugins and libraries get noticed and the efforts shared. And people who want to steward the overall community need to deal with the hassle of tons of repos either managed through some script, or git submodules, or something worse, plus they get drowned in pull requests and issues by being subscribed to everything. Any kind of concern (maybe you have shared build tooling, or documentation, or whatever) that doesn’t neatly align with your repo splits but cuts across the project becomes painful for the maintainers responsible for that.

  • Even once you’ve noticed the need for it, refactoring and code sharing have more bureaucratic hurdles: First you have to release a new version of the core library, then go through all the plugins and update them, and then maybe you can remove the old code in the shared library. But since everything is massively spread around you can forget about that last step.

    Of course it’s not much work to do this, and many projects excel at making this fairly easy. But it is more effort than a simple pull request to the one single monorepo. Very simple refactorings (like just sharing a single new function) will happen less often, and over a long time that compounds and accumulates a lot of debt. Except when you go the node.js way with repos for single functions, but then you essentially replace git with npm as your source control system, and that seems somewhat silly too.

  • The combinatorial explosion of theoretically supported version mixes becomes unsupportable. As a user that means you end up having to do the integration testing. As a project you’ll end up with blessed combinations, or at least de-facto blessed combinations because developers just close bug reports with “please upgrade everything first”. Again that means defacto you have a monorepo, except once more it’s not managed in git. Well, except if you use submodules, and I’m not sure that’s considered git …

  • Reorganizing how you split the overall projects into sub-projects is a pain, since it means you need to reorganize your git repositories and how they’re split up. In a monorepo shifting around maintainership just amounts to updating OWNER or MAINTAINERS files, and if your bots are all good the new maintainers get auto-tagged automatically. But if your way of scaling means splitting git repos into disjoint sets, then any reorg is as painful as the initial step from a monorepo to a group of split up repositories. That means your project will be stuck with a bad organizational structure for too long.

Interlude: Why Pull Requests Exist

The linux kernel is one of the few projects I’m aware of which isn’t split up like this. Before we look at how that works - the kernel is a huge project and simply can’t be run without some sub-project structure - I think it’s interesting to look at why git does pull requests: On github pull request is the one true way for contributors to get their changes merged. But in the kernel changes are submitted as patches sent to mailing lists, even long after git has been widely adopted.

But the very first version of git supported pull requests. The audience of these first, rather rough, releases was kernel maintainers, git was written to solve Linus Torvalds’ maintainer problems. Clearly it was needed and useful, but not to handle changes from individual contributors: Even today, and much more back then, pull requests are used to forward the changes of an entire subsystem, or synchronize code refactoring or similar cross-cutting change across different sub-projects. As an example, the 4.12 network pull request from Dave S. Miller, committed by Linus: It contains 2k+ commits from 600 contributors and a bunch of merges for pull requests from subordinate maintainers. But almost all the patches themselves are committed by maintainers after picking up the patches from mailing lists, not by the authors themselves. This kernel process peculiarity that authors generally don’t commit into shared repositories is also why git tracks the committer and author separately.

Github’s innovation and improvement was then to use pull requests for everything, down to individual contributions. But that wasn’t what they were originally created for.

Scaling, the Linux Kernel Way

At first glance the kernel looks like a monorepo, with everything smashed into one place in Linus’ main repo. But that’s very far from it:

  • Almost no one who’s using linux is running the main repo from Linus Torvalds. If they run something upstream-ish it’s probably one of the stable kernels. But much more likely is that they run a kernel from their distro, which usually has additional patches and backports, and isn’t even hosted on kernel.org, so would be a completely different organization. Or they have a kernel from their hardware vendor (for SoC and pretty much anything Android), which often have considerable deltas compared to anything hosted in one of the “main” repositories.

  • No one (except Linus himself) is developing stuff on top of Linus’ repository. Every subsystem, and often even big drivers, have their own git repositories, with their own mailing lists to track submissions and discuss issues completely separate from everyone else.

  • Cross-subsystem work is done on top of the linux-next integration tree, which contains a few hundred git branches from about as many different git repositories.

  • All this madness is managed through the MAINTAINERS file and the get_maintainers.pl script, which for any given snippet of code can tell you who’s the maintainer, who should review this, where the right git repo is, which mailing lists to use and how and where to report bugs. And it’s not just strictly based on file locations, it also catches code patterns to make sure that cross-subsystem topics like device-tree handling, or the kobject hierarchy are handled by the right experts.

At first this just looks like a complicated way to fill everyone’s disk space with lots of stuff they don’t care about, but there’s a pile of compounding minor benefits that add up:

  • It’s dead easy to reorganize how you split things into sub-project, just update the MAINTAINERS file and you’re done. It’s a bit more work than it really needs to be, since you might need to create a new repo, new mailing lists and a new bugzilla. That’s just an UI problem that github solved with this neat little fork button.

  • It’s really really easy to reassign discussions on pull requests and issues between sub-projects, you simply adjust the Cc: list on your reply. Similarly, doing cross-subsystem work is much easier to coordinate, since the same pull request can be submitted to multiple sub-projects, and there’s just one overall discussions (since the Msg-Ids: tags used for mailing list threading are the same for everyone), despite that the mails are archived in a pile of different mailing list archives, go through different mailing lists and land in a few thousand different inboxes. Making it easier to discuss topics and code across sub-projects avoids fragmentation and makes it much easier to spot where code sharing and refactoring would be beneficial.

  • Cross-subsystem work doesn’t need any kind of release dance. You simply change the code, which is all in your one single repository. Note that this is strictly more powerful than what a split repo setup allows you: For really invasive refactorings you can still space out the work over multiple releases, e.g. when there’s so many users that you can just change them all at once without causing too big coordination pains.

    A huge benefit of making refactoring and code sharing easier is that you don’t have to carry around so much legacy gunk. That’s explained at length in the kernel’s no stable api nonsense document.

  • It doesn’t prevent you from creating your own experimental additions, which is one of the key benefits of the multi-repo setup. Add your code in your own fork and leave it at that - no one ever forces you to push the code back, or push it into the one single repo or even to the main organization, because there simply is no central repositories. This works really well, maybe too well, as evidenced by the millions of code lines which are out-of-tree in the various Android hardware vendor repositories.

In short, I think this is a strictly more powerful model, since you can always fall back to doing things exactly like you would with multiple disjoint repositories. Heck there’s even kernel drivers which are in their own repository, disjoint from the main kernel tree, like the proprietary Nvidia driver. Well it’s just a bit of source code glue around a blob, but since it can’t contain anything from the kernel for legal reasons it is the perfect example.

This looks like a monorepo horror show!

Yes and no.

At first glance the linux kernel looks like a monorepo because it contains everything. And lots of people learned that monorepos are really painful, because past a certain size they just stop scaling.

But looking closer, it’s very, very far away from a single git repository. Just looking at the upstream subsystem and driver repositories gives you a few hundred. If you look at the entire ecosystem, including hardware vendors, distributions, other linux-based OS and individual products, you easily have a few thousand major repositories, and many, many more in total. Not counting any git repo that’s just for private use by individual contributors.

The crucial distinction is that linux has one single file hierarchy as the shared namespace across everything, but lots and lots of different repos for all the different pieces and concerns. It’s a monotree with multiple repositories, not a monorepo.

Examples, please!

Before I go into explaining why github cannot currently support this workflow, at least if you want to retain the benefits of the github UI and integration, we need some examples of how this works in practice. The short summary is that it’s all done with git pull requests between maintainers.

The simple case is percolating changes up the maintainer hierarchy, until it eventually lands in a tree somewhere that is shipped. This is easy, because the pull request only ever goes from one repository to the next, and so could be done already using the current github UI.

Much more fun are cross-subsystem changes, because then the pull request flow stops being an acyclic graph and morphs into a mesh. The first step is to get the changes reviewed and tested by all the involved subsystems and their maintainers. In the github flow this would be a pull request submitted to multiple repositories simultaneously, with the one single discussion stream shared among them all. Since this is the kernel, this step is done through patch submission with a pile of different mailing lists and maintainers as recipients.

The way it’s reviewed is usually not the way it’s merged, instead one of the subsystems is selected as the leading one and takes the pull requests, as long as all other maintainers agree to that merge path. Usually it’s the subsystem most affected by a set of changes, but sometimes also the one that already has some other work in-flight which conflicts with the pull request. Sometimes also an entirely new repository and maintainer crew is created, this often happens for functionality which spans the entire tree and isn’t neatly contained to a few files and directories in one place. A recent example is the DMA mapping tree, which tries to consolidate work that thus far has been spread across drivers, platform maintainers and architecture support groups.

But sometimes there’s multiple subsystems which would both conflict with a set of changes, and which would all need to resolve some non-trivial merge conflict. In that case the patches aren’t just directly applied (a rebasing pull request on github), but instead the pull request with just the necessary patches, based on a commit common to all subsystems, is merged into all subsystem trees. The common baseline is important to avoid polluting a subsystem tree with unrelated changes. Since the pull is for a specific topic only, these branches are commonly called topic branches.

One example I was involved with added code for audio-over-HDMI support, which spanned both the graphics and sound driver subsystems. The same commits from the same pull request where both merged into the Intel graphics driver and also merged into the sound subsystem.

An entirely different example that this isn’t insane is the only other relevant general purpose large scale OS project in the world also decided to have a monotree, with a commit flow modelled similar to what’s going on in linux. I’m talking about the folks with such a huge tree that they had to write an entire new GVFS virtual filesystem provider to support it …

Dear Github

Unfortunately github doesn’t support this workflow, at least not natively in the github UI. It can of course be done with just plain git tooling, but then you’re back to patches on mailing lists and pull requests over email, applied manually. In my opinion that’s the single one reason why the kernel community cannot benefit from moving to github. There’s also the minor issue of a few top maintainers being extremely outspoken against github in general, but that’s a not really a technical issue. And it’s not just the linux kernel, it’s all huge projects on github in general which struggle with scaling, because github doesn’t really give them the option to scale to multiple repositories, while sticking to with a monotree.

In short, I have one simple feature request to github:

Please support pull requests and issue tracking spanning different repos of a monotree.

Simple idea, huge implications.

Repositories and Organizations

First, it needs to be possible to have multiple forks of the same repo in one organization. Just look at git.kernel.org, most of these repositories are not personal. And even if you might have different organizations for e.g. different subsystems, requiring an organization for each repo is silly amounts of overkill and just makes access and user managed unnecessarily painful. In graphics for example we’d have 1 repo each for the userspace test suite, the shared userspace library, and a common set of tools and scripts used by maintainers and developers, which would work in github. But then we’d have the overall subsystem repo, plus a repository for core subsystem work and additional repositories for each big drivers. Those would all be forks, which github doesn’t do. And each of these repos has a bunch of branches, at least one for feature work, and another one for bugfixes for the current release cycle.

Combining all branches into one repository wouldn’t do, since the point of splitting repos is that pull requests and issues are separated, too.

Related, it needs to be possible to establish the fork relationship after the fact. For new projects who’ve always been on github this isn’t a big deal. But linux will be able to move at most a subsystem at a time, and there’s already tons of linux repositories on github which aren’t proper github forks of each another.

Pull Requests

Pull request need to be attached to multiple repos at the same time, while keeping one unified discussion stream. You can already reassign a pull request to a different branch of repo, but not at multiple repositories at the same time. Reassigning pull requests is really important, since new contributors will just create pull requests against what they think is the main repo. Bots can then shuffle those around to all the repos listed in e.g. a MAINTAINERS file for a given set of files and changes a pull request contains. When I chatted with githubbers I originally suggested they’d implement this directly. But I think as long as it’s all scriptable that’s better left to individual projects, since there’s no real standard.

There’s a pretty funky UI challenge here since the patch list might be different depending upon the branch the pull request is against. But that’s not always a user error, one repo might simple have merged a few patches already.

Also, the pull request status needs to be different for each repo. One maintainer might close it without merging, since they agreed that the other subsystem will pull it in, while the other maintainer will merge and close the pull. Another tree might even close the pull request as invalid, since it doesn’t apply to that older version or vendor fork. Even more fun, a pull request might get merged multiple times, in each subsystem with a different merge commit.

Issues

Like pull requests, issues can be relevant for multiple repos, and might need to be moved around. An example would be a bug that’s first reported against a distribution’s kernel repository. After triage it’s clear it’s a driver bug still present in the latest development branch and hence also relevant for that repo, plus the main upstream branch and maybe a few more.

Status should again be separate, since once push to one repo the bugfix isn’t instantly available in all of them. It might even need additional work to get backported to older kernels or distributions, and some might even decide that’s not worth it and close it as WONTFIX, even thought the it’s marked as successfully resolved in the relevant subsystem repository.

Summary: Monotree, not Monorepo

The Linux Kernel is not going to move to github. But moving the Linux way of scaling with a monotree, but mutliple repos, to github as a concept will be really beneficial for all the huge projects already there: It’ll give them a new, and in my opinion, more powerful way to handle their unique challenges.

August 07, 2017

The biggest news this week is that I’ve resolved the window movement performance on Raspbian. After all my work on improving vc4 performance for overlapping CopyArea to parity with SW rendering, the end-user experience was still awful. It turned out that openbox just had a really naive event loop for window movement, so each pixel of motion of the mouse that got reported resulted in a window movement, even if there were 100 of those queued to the window manager in a stream.

I first fixed openbox to consume all the motion events it could and then set the window to the last position it knew of. This didn’t quite work, because thanks to it was only seeing one motion event read in at a time, and queuing up a reposition in between each one.

Next, I made openbox do a round-trip to the server after each copy, so that we get all the motion collected at once per copy that happens on the server side. Now, window dragging keeps up with the mouse as fast as you can wiggle it.

I also worked on backporting the NEON assembly for texture upload/download to the ARMv6 build. Jonas Pfeil started on this, but I want to rework the detection logic a bit to match what Mesa does for other asm support. That should land next week, so that we can have NEON on Raspbian.

On the VC5 front, the next big step is QPU instruction scheduling. So far I’ve removed a duplicate list of QPU instructions, and spent time thinking about how I want to structure the input to the scheduler (it would be good to have instructions reading uniforms be able to be reordered, but we can’t do that if I have the ldunif signal separate from the usage like I do today).

Finally, when I was reviewing daniels’s X11 Present modifiers protocol, I complained that he was representing 64-bit values as pairs of 32-bit values, when we (finally!) recently started using 64-bit values on the wire and in our code. He replied that he had tried that, but the SYNC extension was #defining CARD64 to an XSyncValue and there were conflicts over CARD64. CARD* is X11’s custom not-quite-stdint.h types for sized unsigned values, and XSyncValue is a struct with 2 32-bit values to represent a signed 64-bit value.

My solution was to just remove the CARD64 usage in the SYNC extension, packing and unpacking int64_ts from the wire protocol and then using normal 64-bit math. On 32-bit processors this should expand to the same code as before, but on 64-bit we will probably get actual 64-bit math that we didn’t before (unless your compiler is pretty clever). It also cleaned up some gross code for making XSyncValues of small values like “4” and “0”.

However, since I modified the code, I wanted to run the tests, and I found no tests. Now that we have the meson build system and simple-xinit, it was trivial to write some XCB touch-testing of the protocol and verify that my changes at least didn’t regress the protocol I’ve tested. Hopefully these tests can be a model for future automated protocol testing under “make check”. I didn’t port the test to automake, because automake makes testing awful.

Also this week:

  • Fixed a kernel crashing regression when vc4 fails to load completely since the BO labeling debug work.
  • Started merging a bunch of previous vc4 kernel patches thanks to review from Boris
  • Wrote a piglit test for the overlapping copy extension
  • Fixed up xserver’s Travis CI to run tests that start an X Server.
August 04, 2017

I reworked this week huge parts of my code and I have a feeling that I’m on the right track. I wrote a second mail to the xorg-devel mailing list and the feedback I got back was also way more positive than on the first try. Thanks again to Michel Dänzer for taking the time for this.

In the mail I described my current approach in detail. Basically I created a second code path in the Present DIX for DDX with the possibility to flip windows individually, like Xwayland is capable of doing it. In this code path I also changed the Present extension so that it will only ever signal the client that a certain used pixmap is able to be reused when the DDX has also given it free. In the Xwayland case it does this of course when the Wayland buffer associated with the pixmap has been released by the Wayland server.

What’s still somewhat fragile, but one of my personal darlings is flipping the pixmaps of child windows inside another window geometry. I had the idea to use Wayland sub-surfaces for that and at least on KWin it worked quite nicely. I like it, because it reaches deeply into the X rendering pipeline being a combination of very different internal objects of X and Wayland, which fit together in an amazingly smooth way.

As said though the sub-surface part is still somewhat fragile. I often had problems with dangling pointers because of all the different objects in play, in particular with the RRCrtcs provided by the DDX. Of course you can always go back and guard the code against it when you’ve encountered such a situation while testing. But the risk of missing edge cases is high.

For example everything seemed to work flawlessly at some point a few days ago with my two main test applications VLC and Neverball, but when I tried the Steam desktop client I got segfaults all the time when opening or closing the small drop down menus in the upper half of the window. I came to the conclusion, that this happened because the drop down menus as windows were reparented in between, the old parent was destroyed and by that the child window suddenly had a dangling pointer to the RRCrtc of its former parent. And yeah, I fixed this problem with some guards, but there might be more of these issues I just haven’t yet encountered, right?

What I’m considering instead right now in order to improve the situation in a more general sense is making the CRTCs and windows structs independent. Currently we make an implicit assumption that one window corresponds over its lifetime to exactly one RRCrtc. In Xwayland I created just one fake CRTC per window for that. Conceptually this makes sense, because we want to flip the windows individually and don’t only flip display front buffers as represented by their CRTCs, but the crux is that Present might try to access a CRTC object after the window has been deleted long ago. Making the objects independent again, we could directly test in Xwayland if a RRCrtc is still available as an xwl_output in the output list of the xwl_screen struct, just the way like it’s normally done.

Making CRTCs and windows independent would also bring back the Present code paths to being more similar again from a structural point of view, but it still might mean that the two code paths need to be dissociated more strongly. I’ll see this weekend how it works out and will report on my findings in the article next week.

When I proposed the project with my mentor, we worked out a bunch of features that the new Piper should have. These are listed in the Redesign Wiki, but here’s a high-level summary: A welcome screen, presenting a list of connected and supported devices. An error screen, presenting any problems in a user-friendly manner. The main screen, presenting the configuration pages: A page to configure resolutions (see part 7) A page to configure button mappings (see parts 9 and 10) A page to configure LEDs (see part 8) Support for device profiles (see part 9 for the start on this).
August 01, 2017

This week I worked on building an implementation of the overlapping blit extension for VC4. My first extension spec effectively said “unscaled overlapping BlitFramebuffer now gives you the answer you obviously want.” I had plumbed it all the way through to the backend, written spec text, and was getting ready to submit it for review, when I realized it wasn’t the spec I wanted.

The problem for X11, and specifically Raspbian’s desktop, is that we’re doing a CopyArea with a non-rectangular region. If we implement overlapping blits using BlitFramebuffer, we have a blit per rectangle in the region, each of which dispatches a rendering job to the kernel. Since rendering jobs only operate on tiles, those single-scanline rectangles at the top and bottom of your rounded window have 64x the cost they should, not even counting job dispatch overhead.

So, I went back to the drawing board and wrote a new spec. What I’m offering now is control over the basic HW functionality: You can flip a bit that says “Instead of tile raster order being HW-defined, you have two bits for left-to-right or right-to-left, and top-to-bottom or bottom-to-top.” Combined with GL_ARB_texture_barrier (where I provide the definition for what guarantees the tile raster order provides you), we can submit drawing for all the rectangles in the copy in a single job.

So far, review in Mesa is going well, though the kernel has silence so far. Jason Ekstrand at Intel noted that while they can’t do tile raster order, they could do overlapping 1:1 blits in most cases, and could break things down in the others to a series of valid operations with cache flushes in between. However, his description of the breakdown process is already something that Glamor could do using GL_ARB_texture_barrier:

while (dest_region is nonempty) {
    src_region = dest_region + src copy offset
    this_region = dest_region - src_region
    copy this region;
    glTextureBarrier();
    dest_region -= this_region
}

While I don’t need to do this for VC4, since we have a better option with my extension, it still seems like a good thing to implement.

For VC4, I have x11perf -copywinwin500 performing the same as the previous SW rasterization that Raspbian had. Next week: fixing the interactivity now that the raw performance is in place.

Other news tidbits this week:

  • Landed the CLIF-style debug dumping for VC4
  • Landed the bank-aware register selection optimization for VC4, at last.
  • Debugged an assertion failure regression on VC4 from Marek’s CPU optimization work
  • Tested and landed Hans Verkuil’s HDMI CEC driver for VC4.
  • Landed kernel-side BO labeling (debugfs memory allocation descriptions) support for VC4
  • Did some review on the DRI3 plane modifier patch series from Collabora (a great start, but underspecified in some critical ways).
July 30, 2017

In the previous post we talked about the Phong lighting model as a means to represent light in a scene. Once we have light, we can think about implementing shadows, which are the parts of the scene that are not directly exposed to light sources. Shadow mapping is a well known technique used to render shadows in a scene from one or multiple light sources. In this post we will start discussing how to implement this, specifically, how to render the shadow map image, and the next post will cover how to use the shadow map to render shadows in the scene.

Note: although the code samples in this post are for Vulkan, it should be easy for the reader to replicate the implementation in OpenGL. Also, my OpenGL terrain renderer demo implements shadow mapping and can also be used as a source code reference for OpenGL.

Algorithm overview

Shadow mapping involves two passes, the first pass renders the scene from te point of view of the light with depth testing enabled and records depth information for each fragment. The resulting depth image (the shadow map) contains depth information for the fragments that are visible from the light source, and therefore, are occluders for any other fragment behind them from the point of view of the light. In other words, these represent the only fragments in the scene that receive direct light, every other fragment is in the shade. In the second pass we render the scene normally to the render target from the point of view of the camera, then for each fragment we need to compute the distance to the light source and compare it against the depth information recorded in the previous pass to decice if the fragment is behind a light occluder or not. If it is, then we remove the diffuse and specular components for the fragment, making it look shadowed.

In this post I will cover the first pass: generation of the shadow map.

Producing the shadow map image

Note: those looking for OpenGL code can have a look at this file ter-shadow-renderer.cpp from my OpenGL terrain renderer demo, which contains the shadow map renderer that generates the shadow map for the sun light in that demo.

Creating a depth image suitable for shadow mapping

The shadow map is a regular depth image were we will record depth information for fragments in light space. This image will be rendered into and sampled from. In Vulkan we can create it like this:

...
VkImageCreateInfo image_info = {};
image_info.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO;
image_info.pNext = NULL;
image_info.imageType = VK_IMAGE_TYPE_2D;
image_info.format = VK_FORMAT_D32_SFLOAT;
image_info.extent.width = SHADOW_MAP_WIDTH;
image_info.extent.height = SHADOW_MAP_HEIGHT;
image_info.extent.depth = 1;
image_info.mipLevels = 1;
image_info.arrayLayers = 1;
image_info.samples = VK_SAMPLE_COUNT_1_BIT;
image_info.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
image_info.usage = VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT |
                   VK_IMAGE_USAGE_SAMPLED_BIT;
image_info.queueFamilyIndexCount = 0;
image_info.pQueueFamilyIndices = NULL;
image_info.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
image_info.flags = 0;

VkImage image;
vkCreateImage(device, &image_info, NULL, &image);
...

The code above creates a 2D image with a 32-bit float depth format. The shadow map’s width and height determine the resolution of the depth image: larger sizes produce higher quality shadows but of course this comes with an additional computing cost, so you will probably need to balance quality and performance for your particular target. In the first pass of the algorithm we need to render to this depth image, so we include the VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT usage flag, while in the second pass we will sample the shadow map from the fragment shader to decide if each fragment is in the shade or not, so we also include the VK_IMAGE_USAGE_SAMPLED_BIT.

One more tip: when we allocate and bind memory for the image, we probably want to request device local memory too (VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) for optimal performance, since we won’t need to map the shadow map memory in the host for anything.

Since we are going to render to this image in the first pass of the process we also need to create a suitable image view that we can use to create a framebuffer. There are no special requirements here, we just create a view with the same format as the image and with a depth aspect:

...
VkImageViewCreateInfo view_info = {};
view_info.sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO;
view_info.pNext = NULL;
view_info.image = image;
view_info.format = VK_FORMAT_D32_SFLOAT;
view_info.components.r = VK_COMPONENT_SWIZZLE_R;
view_info.components.g = VK_COMPONENT_SWIZZLE_G;
view_info.components.b = VK_COMPONENT_SWIZZLE_B;
view_info.components.a = VK_COMPONENT_SWIZZLE_A;
view_info.subresourceRange.aspectMask = VK_IMAGE_ASPECT_DEPTH_BIT;
view_info.subresourceRange.baseMipLevel = 0;
view_info.subresourceRange.levelCount = 1;
view_info.subresourceRange.baseArrayLayer = 0;
view_info.subresourceRange.layerCount = 1;
view_info.viewType = VK_IMAGE_VIEW_TYPE_2D;
view_info.flags = 0;

VkImageView shadow_map_view;
vkCreateImageView(device, &view_info, NULL, &view);
...

Rendering the shadow map

In order to generate the shadow map image we need to render the scene from the point of view of the light, so first, we need to compute the corresponding View and Projection matrices. How we calculate these matrices depends on the type of light we are using. As described in the previous post, we can consider 3 types of lights: spotlights, positional lights and directional lights.

Spotlights are the easiest for shadow mapping, since with these we use regular perspective projection.

Positional lights work similar to spotlights in the sense that they also use perspective projection, however, because these are omnidirectional, they see the entire scene around them. This means that we need to render a shadow map that contains scene objects in all directions around the light. We can do this by using a cube texture for the shadow map instead of a regular 2D texture and render the scene 6 times adjusting the View matrix to capture scene objects in front of the light, behind it, to its left, to its right, above and below. In this case we want to use a field of view of 45º with the projection matrix so that the set of 6 images captures the full scene around the light source with no image overlaps.

Finally, we have directional lights. In the previous post I mentioned that these lights model light sources which rays are parallel and because of this feature they cast regular shadows (that is, shadows that are not perspective projected). Thus, to render shadow maps for directional lights we want to use orthographic projection instead of perspective projection.

Projected shadow from a point light source
Regular shadow from a directional light source

In this post I will focus on creating a shadow map for a spotlight source only. I might write follow up posts in the future covering other light sources, but for the time being, you can have a look at my OpenGL terrain renderer demo if you are interested in directional lights.

So, for a spotlight source, we just define a regular perspective projection, like this:

glm::mat4 clip = glm::mat4(1.0f, 0.0f, 0.0f, 0.0f,
                           0.0f,-1.0f, 0.0f, 0.0f,
                           0.0f, 0.0f, 0.5f, 0.0f,
                           0.0f, 0.0f, 0.5f, 1.0f);

glm::mat4 light_projection = clip *
      glm::perspective(glm::radians(45.0f),
                       (float) SHADOW_MAP_WIDTH / SHADOW_MAP_HEIGHT,
                       LIGHT_NEAR, LIGHT_FAR);

The code above generates a regular perspective projection with a field of view of 45º. We should adjust the light’s near and far planes to make them as tight as possible to reduce artifacts when we use the shadow map to render the shadows in the scene (I will go deeper into this in a later post). In order to do this we should consider that the near plane can be increased to reflect the closest that an object can be to the light (that might depend on the scene, of course) and the far plane can be decreased to match the light’s area of influence (determined by its attenuation factors, as explained in the previous post).

The clip matrix is not specific to shadow mapping, it just makes it so that the resulting projection considers the particularities of how the Vulkan coordinate system is defined (Y axis is inversed, Z range is halved).

As usual, the projection matrix provides us with a projection frustrum, but we still need to point that frustum in the direction in which our spotlight is facing, so we also need to compute the view matrix transform of our spotlight. One way to define the direction in which our spotlight is facing is by having the rotation angles of spotlight on each axis, similarly to what we would do to compute the view matrix of our camera:

glm::mat4
compute_view_matrix_for_rotation(glm::vec3 origin, glm::vec3 rot)
{
   glm::mat4 mat(1.0);
   float rx = DEG_TO_RAD(rot.x);
   float ry = DEG_TO_RAD(rot.y);
   float rz = DEG_TO_RAD(rot.z);
   mat = glm::rotate(mat, -rx, glm::vec3(1, 0, 0));
   mat = glm::rotate(mat, -ry, glm::vec3(0, 1, 0));
   mat = glm::rotate(mat, -rz, glm::vec3(0, 0, 1));
   mat = glm::translate(mat, -origin);
   return mat;
}

Here, origin is the position of the light source in world space, and rot represents the rotation angles of the light source on each axis, representing the direction in which the spotlight is facing.

Now that we have the View and Projection matrices that define our light space we can go on and render the shadow map. For this we need to render scene as we normally would but instead of using our camera’s View and Projection matrices, we use the light’s. Let’s have a look at the shadow map rendering code:

Render pass

static VkRenderPass
create_shadow_map_render_pass(VkDevice device)
{
   VkAttachmentDescription attachments[2];

   // Depth attachment (shadow map)
   attachments[0].format = VK_FORMAT_D32_SFLOAT;
   attachments[0].samples = VK_SAMPLE_COUNT_1_BIT;
   attachments[0].loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
   attachments[0].storeOp = VK_ATTACHMENT_STORE_OP_STORE;
   attachments[0].stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
   attachments[0].stencilStoreOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
   attachments[0].initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
   attachments[0].finalLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
   attachments[0].flags = 0;

   // Attachment references from subpasses
   VkAttachmentReference depth_ref;
   depth_ref.attachment = 0;
   depth_ref.layout = VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL;

   // Subpass 0: shadow map rendering
   VkSubpassDescription subpass[1];
   subpass[0].pipelineBindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS;
   subpass[0].flags = 0;
   subpass[0].inputAttachmentCount = 0;
   subpass[0].pInputAttachments = NULL;
   subpass[0].colorAttachmentCount = 0;
   subpass[0].pColorAttachments = NULL;
   subpass[0].pResolveAttachments = NULL;
   subpass[0].pDepthStencilAttachment = &depth_ref;
   subpass[0].preserveAttachmentCount = 0;
   subpass[0].pPreserveAttachments = NULL;

   // Create render pass
   VkRenderPassCreateInfo rp_info;
   rp_info.sType = VK_STRUCTURE_TYPE_RENDER_PASS_CREATE_INFO;
   rp_info.pNext = NULL;
   rp_info.attachmentCount = 1;
   rp_info.pAttachments = attachments;
   rp_info.subpassCount = 1;
   rp_info.pSubpasses = subpass;
   rp_info.dependencyCount = 0;
   rp_info.pDependencies = NULL;
   rp_info.flags = 0;

   VkRenderPass render_pass;
   VK_CHECK(vkCreateRenderPass(device, &rp_info, NULL, &render_pass));

   return render_pass;
}

The render pass is simple enough: we only have one attachment with the depth image and one subpass that renders to the shadow map target. We will start the render pass by clearing the shadow map and by the time we are done we want to store it and transition it to layout VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL so we can sample from it later when we render the scene with shadows. Notice that because we only care about depth information, the render pass doesn’t include any color attachments.

Framebuffer

Every rendering job needs a target framebuffer, so we need to create one for our shadow map. For this we will use the image view we created from the shadow map image. We link this framebuffer target to the shadow map render pass description we have just defined:

VkFramebufferCreateInfo fb_info;
fb_info.sType = VK_STRUCTURE_TYPE_FRAMEBUFFER_CREATE_INFO;
fb_info.pNext = NULL;
fb_info.renderPass = shadow_map_render_pass;
fb_info.attachmentCount = 1;
fb_info.pAttachments = &shadow_map_view;
fb_info.width = SHADOW_MAP_WIDTH;
fb_info.height = SHADOW_MAP_HEIGHT;
fb_info.layers = 1;
fb_info.flags = 0;

VkFramebuffer shadow_map_fb;
vkCreateFramebuffer(device, &fb_info, NULL, &shadow_map_fb);

Pipeline description

The pipeline we use to render the shadow map also has some particularities:

Because we only care about recording depth information, we can typically skip any vertex attributes other than the positions of the vertices in the scene:

...
VkVertexInputBindingDescription vi_binding[1];
VkVertexInputAttributeDescription vi_attribs[1];

// Vertex attribute binding 0, location 0: position
vi_binding[0].binding = 0;
vi_binding[0].inputRate = VK_VERTEX_INPUT_RATE_VERTEX;
vi_binding[0].stride = 2 * sizeof(glm::vec3);

vi_attribs[0].binding = 0;
vi_attribs[0].location = 0;
vi_attribs[0].format = VK_FORMAT_R32G32B32_SFLOAT;
vi_attribs[0].offset = 0;

VkPipelineVertexInputStateCreateInfo vi;
vi.sType = VK_STRUCTURE_TYPE_PIPELINE_VERTEX_INPUT_STATE_CREATE_INFO;
vi.pNext = NULL;
vi.flags = 0;
vi.vertexBindingDescriptionCount = 1;
vi.pVertexBindingDescriptions = vi_binding;
vi.vertexAttributeDescriptionCount = 1;
vi.pVertexAttributeDescriptions = vi_attribs;
...
pipeline_info.pVertexInputState = &vi;
...

The code above defines a single vertex attribute for the position, but assumes that we read this from a vertex buffer that packs interleaved positions and normals for each vertex (each being a vec3) so we use the binding’s stride to jump over the normal values in the buffer. This is because in this particular example, we have a single vertex buffer that we reuse for both shadow map rendering and normal scene rendering (which requires vertex normals for lighting computations).

Again, because we do not produce color data, we can skip the fragment shader and our vertex shader is a simple passthough instead of the normal vertex shader we use with the scene:

....
VkPipelineShaderStageCreateInfo shader_stages[1];
shader_stages[0].sType =
   VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO;
shader_stages[0].pNext = NULL;
shader_stages[0].pSpecializationInfo = NULL;
shader_stages[0].flags = 0;
shader_stages[0].stage = VK_SHADER_STAGE_VERTEX_BIT;
shader_stages[0].pName = "main";
shader_stages[0].module = create_shader_module("shadowmap.vert.spv", ...);
...
pipeline_info.pStages = shader_stages;
pipeline_info.stageCount = 1;
...

This is how the shadow map vertex shader (shadowmap.vert) looks like in GLSL:

#version 400

#extension GL_ARB_separate_shader_objects : enable
#extension GL_ARB_shading_language_420pack : enable

layout(std140, set = 0, binding = 0) uniform vp_ubo {
    mat4 ViewProjection;
} VP;

layout(std140, set = 0, binding = 1) uniform m_ubo {
     mat4 Model[16];
} M;

layout(location = 0) in vec3 in_position;

void main()
{
   vec4 pos = vec4(in_position.x, in_position.y, in_position.z, 1.0);
   vec4 world_pos = M.Model[gl_InstanceIndex] * pos;
   gl_Position = VP.ViewProjection * world_pos;
}

The shader takes the ViewProjection matrix of the light (we have already multiplied both together in the host) and a UBO with the Model matrices of each object in the scene as external resources (we use instanced rendering in this particular example) as well as a single vec3 input attribute with the vertex position. The only job of the vertex shader is to compute the position of the vertex in the transformed space (the light space, since we are passing the ViewProjection matrix of the light), nothing else is done here.

Command buffer

The command buffer is pretty similar to the one we use with the scene, only that we render to the shadow map image instead of the usual render target. In the shadow map render pass description we have indicated that we will clear it, so we need to include a depth clear value. We also need to make sure that we set the viewport and sccissor to match the shadow map dimensions:

...
VkClearValue clear_values[1];
clear_values[0].depthStencil.depth = 1.0f;
clear_values[0].depthStencil.stencil = 0;

VkRenderPassBeginInfo rp_begin;
rp_begin.sType = VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO;
rp_begin.pNext = NULL;
rp_begin.renderPass = shadow_map_render_pass;
rp_begin.framebuffer = shadow_map_framebuffer;
rp_begin.renderArea.offset.x = 0;
rp_begin.renderArea.offset.y = 0;
rp_begin.renderArea.extent.width = SHADOW_MAP_WIDTH;
rp_begin.renderArea.extent.height = SHADOW_MAP_HEIGHT;
rp_begin.clearValueCount = 1;
rp_begin.pClearValues = clear_values;

vkCmdBeginRenderPass(shadow_map_cmd_buf,
                     &rp_begin,
                     VK_SUBPASS_CONTENTS_INLINE);

VkViewport viewport;
viewport.height = SHADOW_MAP_HEIGHT;
viewport.width = SHADOW_MAP_WIDTH;
viewport.minDepth = 0.0f;
viewport.maxDepth = 1.0f;
viewport.x = 0;
viewport.y = 0;
vkCmdSetViewport(shadow_map_cmd_buf, 0, 1, &viewport);

VkRect2D scissor;
scissor.extent.width = SHADOW_MAP_WIDTH;
scissor.extent.height = SHADOW_MAP_HEIGHT;
scissor.offset.x = 0;
scissor.offset.y = 0;
vkCmdSetScissor(shadow_map_cmd_buf, 0, 1, &scissor);
...

Next, we bind the shadow map pipeline we created above, bind the vertex buffer and descriptor sets as usual and draw the scene geometry.

...
vkCmdBindPipeline(shadow_map_cmd_buf,
                  VK_PIPELINE_BIND_POINT_GRAPHICS,
                  shadow_map_pipeline);

const VkDeviceSize offsets[1] = { 0 };
vkCmdBindVertexBuffers(shadow_cmd_buf, 0, 1, vertex_buf, offsets);

vkCmdBindDescriptorSets(shadow_map_cmd_buf,
                        VK_PIPELINE_BIND_POINT_GRAPHICS,
                        shadow_map_pipeline_layout,
                        0, 1,
                        shadow_map_descriptor_set,
                        0, NULL);

vkCmdDraw(shadow_map_cmd_buf, ...);

vkCmdEndRenderPass(shadow_map_cmd_buf);
...

Notice that the shadow map pipeline layout will be different from the one used with the scene too. Specifically, during scene rendering we will at least need to bind the shadow map for sampling and we will probably also bind additional resources to access light information, surface materials, etc that we don’t need to render the shadow map, where we only need the View and Projection matrices of the light plus the UBO with the model matrices of the objects in the scene.

We are almost there, now we only need to submit the command buffer for execution to render the shadow map:

...
VkPipelineStageFlags shadow_map_wait_stages = 0;
VkSubmitInfo submit_info = { };
submit_info.pNext = NULL;
submit_info.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO;
submit_info.waitSemaphoreCount = 0;
submit_info.pWaitSemaphores = NULL;
submit_info.signalSemaphoreCount = 1;
submit_info.pSignalSemaphores = &signal_sem;
submit_info.pWaitDstStageMask = 0;
submit_info.commandBufferCount = 1;
submit_info.pCommandBuffers = &shadow_map_cmd_buf;

vkQueueSubmit(queue, 1, &submit_info, NULL);
...

Because the next pass of the algorithm will need to sample the shadow map during the final scene rendering,we use a semaphore to ensure that we complete this work before we start using it in the next pass of the algorithm.

In most scenarios, we will want to render the shadow map on every frame to account for dynamic objects that move in the area of effect of the light or even moving lights, however, if we can ensure that no objects have altered their positions inside the area of effect of the light and that the light’s description (position/direction) hasn’t changed, we may not need need to regenerate the shadow map and save some precious rendering time.

Visualizing the shadow map

After executing the shadow map rendering job our shadow map image contains the depth information of the scene from the point of view of the light. Before we go on and start using this as input to produce shadows in our scene, we should probably try to visualize the shadow map to verify that it is correct. For this we just need to submit a follow-up job that takes the shadow map image as a texture input and renders it to a quad on the screen. There is one caveat though: when we use perspective projection, Z values in the depth buffer are not linear, instead precission is larger at distances closer to the near plane and drops as we get closer to the far place in order to improve accuracy in areas closer to the observer and avoid Z-fighting artifacts. This means that we probably want to linearize our shadow map values when we sample from the texture so that we can actually see things, otherwise most things that are not close enough to the light source will be barely visible:

#version 400

#extension GL_ARB_separate_shader_objects : enable
#extension GL_ARB_shading_language_420pack : enable

layout(std140, set = 0, binding = 0) uniform mvp_ubo {
    mat4 mvp;
} MVP;

layout(location = 0) in vec2 in_pos;
layout(location = 1) in vec2 in_uv;

layout(location = 0) out vec2 out_uv;

void main()
{
   gl_Position = MVP.mvp * vec4(in_pos.x, in_pos.y, 0.0, 1.0);
   out_uv = in_uv;
}
#version 400

#extension GL_ARB_separate_shader_objects : enable
#extension GL_ARB_shading_language_420pack : enable

layout (set = 1, binding = 0) uniform sampler2D image;

layout(location = 0) in vec2 in_uv;

layout(location = 0) out vec4 out_color;

void main()
{
   float depth = texture(image, in_uv).r;
   out_color = vec4(1.0 - (1.0 - depth) * 100.0);
}

We can use the vertex and fragment shaders above to render the contents of the shadow map image on to a quad. The vertex shader takes the quad’s vertex positions and texture coordinates as attributes and passes them to the fragment shader, while the fragment shader samples the shadow map at the provided texture coordinates and then “linearizes” the depth value so that we can see better. The code in the shader doesn’t properly linearize the depth values we read from the shadow map (that requires to pass the Z-near and Z-far values used in the projection), but for debugging purposes this works well enough for me, if you use different Z clipping planes you may need to alter the ‘100.0’ value to get good results (or you might as well do a proper conversion considering your actual Z-near and Z-far values).

Visualizing the shadow map

The image shows the shadow map on top of the scene. Darker colors represent smaller depth values, so these are fragments closer to the light source. Notice that we are not rendering the floor geometry to the shadow map since it can’t cast shadows on any other objects in the scene.

Conclusions

In this post we have described the shadow mapping technique as a combination of two passes: the first pass renders a depth image (the shadow map) with the scene geometry from the point of view of the light source. To achieve this, we need a passthrough vertex shader that only transforms the scene vertex positions (using the view and projection transforms from the light) and we can skip the fragment shader completely since we do not care for color output. The second pass, which we will cover in the next post, takes the shadow map as input and uses it to render shadows in the final scene.

July 29, 2017

Alex Larsson talks about Flathub at GUADEC 2017At the Gtk+ hackfest in London earlier this year, we stole an afternoon from the toolkit folks (sorry!) to talk about Flatpak, and how we could establish a “critical mass” behind the Flatpak format. Bringing Linux container and sandboxing technology together with ostree, we’ve got a technology which solves real world distribution, technical and security problems which have arguably held back the Linux desktop space and frustrated ISVs and app developers for nearly 20 years. The problem we need to solve, like any ecosystem, is one of users and developers – without stuff you can easily get in Flatpak format, there won’t be many users, and without many users, we won’t have a strong or compelling incentive for developers to take their precious time to understand a new format and a new technology.

As Alex Larsson said in his GUADEC talk yesterday: Decentralisation is good. Flatpak is a tool that is totally agnostic of who is publishing the software and where it comes from. For software freedom, that’s an important thing because we want technology to empower users, rather than tell them what they can or can’t do. Unfortunately, decentralisation makes for a terrible user experience. At present, the Flatpak webpage has a manually curated list of links to 10s of places where you can find different Flatpaks and add them to your system. You can’t easily search and browse to find apps to try out – so it’s clear that if the current situation remains we’re not going to be able to get a critical mass of users and developers around Flatpak.

Enter Flathub. The idea is that by creating an obvious “center of gravity” for the Flatpak community to contribute and build their apps, users will have one place to go and find the best that the Linux app ecosystem has to offer. We can take care of the boring stuff like running a build service and empower Linux application developers to choose how and when their app gets out to their users. After the London hackfest we sketched out a minimum viable system – Github, Buildbot and a few workers – and got it going over the past few months, culminating in a mini-fundraiser to pay for the hosting of a production-ready setup. Thanks to the 20 individuals who supported our fundraiser, to Mythic Beasts who provided a server along with management, monitoring and heaps of bandwidth, and to Codethink and Scaleway who provide our ARM and Intel workers respectively.

We inherit our core principles from the Flatpak project – we want the Flatpak technology to succeed at alleviating the issues faced by app developers in targeting a diverse set of Linux platforms. None of this stops you from building and hosting your own Flatpak repos and we look forward to this being a wide and open playing field. We care about the success of the Linux desktop as a platform, so we are open to proprietary applications through Flatpak’s “extra data” feature where the client machine downloads 3rd party binaries. They are correctly labeled as such in the AppStream, so will only be shown if you or your OS has configured GNOME Software to show you apps with proprietary licenses, respecting the user’s preference.

The new infrastructure is up and running and I put it into production on Thursday. We rebuilt the whole repository on the new system over the course of the week, signing everything with our new 4096-bit key stored on a Yubikey smartcard USB key. We have 66 apps at the moment, although Alex is working on bringing in the GNOME apps at present – we hope those will be joined soon by the KDE apps, and Endless is planning to move over as many of our 3rd party Flatpaks as possible over the coming months.

So, thanks again to Alex and the whole Flatpak community, and the individuals and the companies who supported making this a reality. You can add the repository and get downloading right away. Welcome to Flathub! Go forth and flatten… 🙂

Flathub logo

Like all good blog posts, this one starts with an apology about not blogging for ages – in my case it looks like it’s been about 7 years which is definitely a new personal best (perhaps the equally or more remarkable thing is that I have diligently kept WordPress running in the meantime). In that time, as you might expect, a few things have happened, like I met a wonderful woman and fell in love and we have two wonderful children. I also decided to part ways with my “first baby” and leave my role as CTO & co-founder of Collabora. This was obviously a very tough decision – it’s a fantastic team where I met and made many life-long friends, and they are still going strong and doing awesome things with Open Source. However, shortly after that, in February last year, I was lucky enough to land what is basically a dream job working at Endless Computers as the VP of Deployment.

As I’m sure most readers know, Endless is building an OS to bring personal computing to millions of new users across the world. It’s based on Free Software like GNOME, Debian, ostree and Flatpak, and the more successful Endless is, the more people who get access to education, technology and opportunity – and the more FOSS users and developers there will be in the world. But in my role, I get to help define the product, understand our users and how technology might help them, take Open Source out to new people, solve commercial problems to get Endless OS out into the world, manage a fantastic team and work with bright people, learn from great managers and mentors, and still find time to squash bugs, review and write more code than I used to. Like any startup, we have a lot to do and not enough time to do it, so although there aren’t quite enough days in the week, I’m really happy!

In any case, the main point of this blog post is that I’m at GUADEC in Manchester right now, and I’d like to blog about Flathub, but I thought it would be weird to just show up and say that after 7 years of silence without saying hello again. 🙂

July 28, 2017

What is the worst thing, that can happen to someone’s work? Probably showing it to someone and he isn’t impressed at all by it. Guess, what happened to me! I said in the last post, that I wanted to reach out to the good people from the xorg-devel mailing list for some feedback on my work so far. I did, but the response wasn’t that reassuring.

Basically several fundamental parts of my code were questioned. OK, the problem of timings with the frame callback was easily to solve - just don’t do it for now, it’s not really necessary. But the problems pointed out by Michel had a more profound impact. How I should deal with the Present extension and what was requested by Michel couldn’t easily be integrated in the code I had written.

But I didn’t lose my motivation because of this setback and of course I’m deeply thankful for the voiced critique by Pekka and Michel, since I can only learn from their feedback and for sure better rework the code now than later. I did exactly that in the last few days. And in my opinion with great success.

I interpreted Michel’s mail, such that I’m allowed to make larger changes to the Present extension code. Otherwise the rootless flipping wouldn’t be possible for sure. So I added a “Rootless mode” to the extension, which of course currently only is used by the Xwayland DDX, but could be used by other DDX in the future with similar functionality. It allows to flip windows individually and without the need to be full screen. This is a huge functionality boost in comparison to the normal X behavior, where a window needs to be full screen on the whole _Screen, and this in particular means that it won’t work with a full screen window on a single display in a RandR environment.

Of course some big change like this never works directly. For example I forgot to increase the reference counter on the pixmap remembered for restoring the window and only after tedious bug search found that this was the culprit for sporadic segfaults when exiting the full screen mode of the VLC player.

And yes, in order to be able to flip the pixmap not only in full screen mode but even inside a window with other sub-windows being composited around it, more was necessary. The solution was in the end to use a sub-surface for the flipping pixmap. And I’m not sure if it will be viable in the end, but I’m damn proud that it works so far.

Sometimes there are still visual artifacts I experience on window movements. The compositor doesn’t seem to get informed that the sub-surface has been moved away and still paints its content where it was before. But maybe the culprit is in this case KWin not handling the sub-surface correctly. I’ll need to cross-check this problem with some other compositor like Weston.

And I plan on writing another mail this weekend to the mailing list. Let’s see how this radical new beginning will resonate.

I can start this blog post by sharing the news that I have passed for my second evaluation! This means that I’m now on the last sprint towards the finish line, with ahead of me profile support and the welcome screen. Yes; you’ve read that correctly: I didn’t mention the button page. I managed to fix that this week and it’s now pending review and some final issues (more about that below).
July 27, 2017

A few days ago, I pushed code for button debouncing into libinput, scheduled for libinput 1.9. What is button debouncing you ask? Well, I'm glad you asked, because otherwise typing this blog post would've been a waste of time :)

Over in Utopia, when you press the button on a device, you get a press event from the hardware. When you release said button, you get a release event from the hardware. Together, they form the button click interaction we have come to learn and love over the last couple of decades. Life is generally merry and the sunshine to rainbow to lollipop ratio is good. Meanwhile, over here in the real world, buttons can be quite dodgy, don't always work like they're supposed to, lollipops are unhealthy and boy, have you seen that sunburn the sunshine gave me? One way how buttons may not work is that they can lose contact for a fraction of a second and send release events even though the button is being held down. The device usually detects that the button is still being down in the next hardware cycle (~8ms on most devices) and thus sends another button press.

For us, there are not a lot of hints that this is bad hardware besides the timestamps. It's not possible for a user to really release and press a button within 8ms, so we can take this as a signal for dodgy hardware. But at least that's someting. All we need to do is ignore the release event (and subsequent button event) and only release when the button is released properly. This requires timeouts and delays of the event, something we generally want to avoid unless absolutely necessary. So the behaviour libinput has now is enabled but inactive button debouncing on all devices. We monitor button release and button press timestamps, but otherwise leave the events as-is, so no delays are introduced. Only if a device sends release/press combos with unfeasably short timeouts, activate button debouncing. Once active, we filter all button release events and instead set a timer. Once the timer expires, we send the button release event. But if at any time before then another button press is detected, the scheduled release is discarded, the button press is filtered and no event is sent. Thus, we paper over the release/press combo the hardware gives us and to anyone using libinput, it will look like the button was held down for the whole time.

There's one downside with this approach - the very first button debounce to happen on a device will still trigger an erroneous button release event. It remains to be seen whether this is a problem in real-world scenarios. That's the cost of having it as an auto-enabling feature rather than an explicit configuration option.

If you do have a mouse that suffers from button bouncing, I recommend you try libinput's master branch and file any issues if the debouncing doesn't work as it should. Might as well get any issues fixed before we have a release.

July 26, 2017
A quick update, as we've touched upon Evince recently.

I mentioned that we switched from using external tools for decompression to using libarchive. That's not the whole truth, as we switched to using libarchive for CBZ, CB7 and the infamous CBT, but used a copy/paste version of unarr to support RAR files, as libarchive support lacks some needed features.

We hope to eventually remove the internal copy of unarr, but, as a stop-gap, that allowed us to start supporting CBR comics out of the box, and it's always a good thing when you have one less non-free package to grab from somewhere to access your media.

The second new format is really two formats, from either side of the 2-digit-year divide: PostScript-based Adobe Illustrator and PDF-based Adobe Illustrator. Evince now declares to support "the format" if both of the backends are built and supported. It only took 12 years, and somebody stumbling upon the feature request while doing bug triaging. The nooks and crannies of free software where the easy feature requests get lost :)


Both features will appear in GNOME 3.26, the out-of-the-box CBR support is however available now in an update for the just released Fedora 26.

I know many of you have wanted to test running Wayland on NVidia. The work on this continues between Jonas Ådahl, Adam Jackson and various developers at NVidia. It is not ready for primetime yet as we are still working on the server side glvnd piece we need for XWayland. That said with both Adam Jackson looking at this from our side and Kyle Brenneman looking at it from NVidia I am sure we will be able to hash out the remaining open questions and get that done.

In the meantime Miguel A. Vico from NVidia has set up a Copr to let people start testing using EGLStreams under Wayland. I haven’t tested it myself yet, but if you do and have trouble make sure to let Miguel and Jonas know.

As a sidenote, I am heading off to GUADEC in Manchester tomorrow and we do plan to discuss efforts like these there. We have team members like Jonas Ådahl flying in from Taiwan and Peter Hutterer flying in from Australia, so it will be a great chance to meet core developers who are far away from us in terms of timezone and geographical distance. GUADEC this year should be a lot of fun and from what I hear we are going to have record level attendance this year based early registration numbers, so if you can make it Manchester I strongly recommend joining us as I think this years event will have a lot of energy and a lot of interesting discussions on what the next steps are for GNOME.

July 24, 2017

This week’s VC5 progress included:

  • Major fixes to texture tiling and layout
  • Major fixes to branching in shaders (the uniform stream didn’t branch the same as the PC)
  • Implementing textureSize()
  • Implementing texelFetch()
  • Implementing textureQueryLevels()
  • Partially implementing transform feedback
  • Fixing gl_VertexID/gl_InstanceID usage.
  • Fixing some NaN behavior
  • Fixing assertion failures with PointSize set when drawing non-points.
  • Fixing use-after-free with Z or stencil (but not both) clears.

VC5’s GLES3 correctness is reaching the point where I need to start building the 7268 kernel driver, and then do some of the basic performance work (shader threading, QPU instruction scheduling)

I made another small fix to the X11 copy performance series I have, that improves things when windows have rounded corners, and spent a while planning out next week’s work on the overlapping blit extension.

I sent out another version of the DSI series (effectively v6). The bridge driver maintainers seem happy with my core DSI support changes, so hopefully the panel maintainer will be OK with them as well.

July 21, 2017

I planned on writing about the Present extension this week, but I’ll postpone this since I’m currently strongly absorbed into finding the last rough edges of a first patch I can show off. I then hope to get some feedback on this from other developers in the xorg-devel mailing list.

Another reason is that I stalled my work on the Present extension for now and try to get first my Xwayland code working. My mentor Daniel recommended that to me since the approach I pursued in my work on Present might be more difficult than I first assessed. At least it is something similar to what other way more experienced developers than myself tried in the past and weren’t able to do according to Daniel. My idea was to make Present flip per CRTC only, but this would clash with Pixmaps being linked to the whole screen only. There are no Pixmaps only for CRTCs in X.

On the other hand when accepting the restriction of only being able to flip one window at a time my code already works quite good. The flipping is smooth and at least in a short test also improved the frame rate. But the main problem I had and still to some degree have, is that stopping the flipping can fail. The reason seems to be that the Present extension sets always the Screen Pixmap on flips. But when I test my work with KWin, it drives Xwayland in rootless mode, i.e. without a Screen Pixmap and only the Window Pixmaps. I’m currently looking into how to circumvent this in Xwayland. I think it’s possible, but I need to look very carefully on how to change the process in order to not forget necessary cleanups on the flipped Pixmaps. I hope though that I’m able to solve these issues already this weekend and then get some feedback on the xorg-devel mailing list.

As always you can find my latest work on my working branch on GitHub.

@GodTributes took over my title, soz.

Dude, where's my maintainer?

Last year, probably as a distraction from doing anything else, or maybe because I was asked, I started reviewing bugs filed as a result of automated flaw discovery tools (from Coverity to UBSan via fuzzers) being run on gdk-pixbuf.

Apart from the security implications of a good number of those problems, there was also the annoyance of having a busted image file bring down your file manager, your desktop, or even an app that opened a file chooser either because it was broken, or because the image loader for that format didn't check for the sanity of memory allocations.

(I could have added links to Bugzilla entries for each one of the problems above, but that would just make it harder to read)

Two big things happened in gdk-pixbuf 2.36.1, which was used in GNOME 3.24:

  • the removal of GdkPixdata as a stand-alone image format loader. We really don't want to load GdkPixdata files from sources other than generated sources or embedded data structures, and removing that loader closed off those avenues. We still ended up fixing a fair number of naive assumptions in helper functions though.
  • the addition of a thumbnailer for gdk-pixbuf supported images. Images would not be special-cased any more in gnome-desktop's thumbnailing code, making the file manager, the file chooser and anything else navigating directories full of broken and huge images more reliable.
But that's just the start. gdk-pixbuf continues getting bug fixes, and we carry on checking for overflows, underflows and just flows, breaks and beats in general.

Programmatic Thumbellina portrait-maker

Picture, if you will, a website making you download garbage files from the Internet, the ROM dump of a NES cartridge that wasn't properly blown on and digital comic books that you definitely definitely paid for.

That's a nice summary of the security bugs foisted upon GNOME in past year or so, even if, thankfully, we were ahead of the curve in terms of fixing those issues (the GStreamer NSF decoder bug was removed in 2013, the comics backend in evince was rewritten over a period of 2 years and committed in March 2017).

Still, 2 pieces of code were running on pretty much every file downloaded, on purpose or not, from the Internet: Tracker's indexers and the file manager's thumbnailers.

Tracker started protecting itself not long after the NSF vulnerability, even if recent versions of GStreamer weren't vulnerable, as we mentioned.

That left the thumbnailers. Some of those are first party, like the gdk-pixbuf, and those offered by core applications (Evince, Videos), written by GNOME developers (yours truly for both epub/mobi and Nintendo DS).

They're all good quality code I'd vouch for (having written or maintained quite a few of them), but they can rely on third-party libraries (say GStreamer, poppler, or libarchive), have naive or insufficiently defensive code (gdk-pixbuf loaders,  GStreamer plugins) or, worst of all: THIRD-PARTY EXTENSIONS.

There are external plugins and extensions for image formats in gdk-pixbuf, for video and audio formats in GStreamer, and for thumbnailers pretty much anywhere. We can't control those, but the least we can do when they explode in a wet mess is make sure that the toilet door is closed.

Not even Nicholas Cage can handle this Alcatraz

For GNOME 3.26 (and today in git master), the thumbnailer stall will be doubly bolted by a Bubblewrap sandbox and a seccomp blacklist.

This closes a whole vector of attack for the GNOME Desktop, but doesn't mean we're completely out of the woods. We'll need to carry on maintaining and fixing security bugs in those libraries and tools we depend on, as GStreamer plugin bugs still affect Videos, gdk-pixbuf bugs still affect Photos and Eye Of Gnome, etc.

And there are limits to what those 2 changes can achieve. The sandboxing and syscall blacklisting avoids those thumbnailers writing anything but an image file in PNG format in a specific directory. There's no network, the filename of the original file is hidden and sanitised, but the thumbnailer could still create a crafted PNG file, and the sandbox doesn't work inside a sandbox! So no protection if the application running the thumbnailer is inside Flatpak.

In fine

GNOME 3.26 will have better security for thumbnailers, so you won't "need to delete GNOME Files".

But you'll probably want to be careful with desktops that forked our thumbnailing code, namely Cinnamon and MATE, which don't implement those security features.

The next step for the thumbnailers will be beefing up our protection against greedy thumbnailers (in terms of CPU and memory usage), and sharing the code better between thumbnailers.

Note for later, more images of cute animals.
When I discussed this project with my mentor before GSoC, he told me that the button mappings were going to be the most complicated piece. This week I’ve been working on precisely that and, well, let’s just say he wasn’t wrong 😉 If you’ve been following along on GitHub, you’re probably thinking that it was a slow week. Indeed, there hasn’t been that much activity this week as in previous weeks.
July 20, 2017

Since the last post a lot work has gone into upstreaming and stabilizing the etnaviv on Android ecosystem. This has involved Android, kernel and Mesa changes. Many of which are available upstream now. A How-To for getting you up and running on an iMX6 dev board is available here.

Improvements

Modifiers support

Modifiers support has been accepted into Mesa, GBM and gbm_gralloc. Modifiers were mentioned in a previous post.

Etnaviv driver support for Android

Patches enabling the etnaviv Mesa driver being built for Android have now landed upstream.

Stability on Android

A number for small stability issues present while running Android on i.MX6 hardware have now been fixed, and the platform is now relatively stable.

Performance diagnostics

We have a decent understanding that the …

July 17, 2017

Video of my casync Presentation @ kinvolk

The great folks at kinvolk have uploaded a video of my casync presentation at their offices last week.

The slides are available as well.

Enjoy!

This week’s VC5 progress primarily involved rewriting the CLIF-style debug dumps using Intel’s gen_decode.c code. Instead of code-generating a bunch of C functions that print out a struct’s contents, I now have a little bit of C code that parses a compressed version of the XML at runtime to pick apart the struct and dump it. I’ve implemented this on VC4 and VC5, and started the Android build debugging process for it.

I also finished fixing the regressions from my VC5 QIR redesign. We now operate on just QPU instructions with sideband information for register allocation, instead of a higher-level IR (that’s what NIR is for).

For Raspbian performance, I’ve been talking with keithp and others about my window-dragging performance issue. My current plan is to implement a little GL extension that gives glBlitFramebuffer() defined behavior for 1:1 overlapping copies, and then use that from X11 to avoid the temporary. That should cut the cost of the window movement in half (not counting the cost of the drawing caused by the expose events).

On the KMS front, I’ve fixed a regression in dual-display support from my previous tiling work. In the process, I’ve also written a fix for panning, which was broken even before the tiling work. I’ve pushed a fix from Boris for a warning on CRTC enable. I’ve also worked on handling the review feedback from the last DSI series, and started on review of Hans’s VC4 CEC support.

July 14, 2017

I provided in the past few weeks some general information about my project and hopefully helpful documentation for the multiple components I’m working with, but I have not yet talked about the work I’m doing on the code itself. Let’s change this today.

You can find my work branch on GitHub. It’s basically just a personal repository so I can sync my work between my devices, so be warned: The commits are messy as nothing is cleaned up and debug lines as well as temporary TODOs are all over the place. And to be honest up until yesterday my changes didn’t accumulate to much. For some reason no picture was displayed in my two test applications, which are Neverball and VLC.

Then on the weekend suddenly the KWin Wayland session wouldn’t even launch anymore. Well, at least this issue I was able to fix pretty quickly. But there was still no picture, it seemed the presenting just halted after the first buffer was sent to KWin and without any further messages. Neither the Xwayland server nor the client were unresponsive though. Only today I finally could solve the problem thanks to Daniel’s help. The reason for the halt was that I waited on a frame callback from KWin in order to present the next frame. But this never arrived since I hadn’t set any damage in the previous frame and KWin then wouldn’t signal a new frame. I fixed it by adding a generic damage request for now. After that the picture was depicted and moving nicely.

This is definitively the first big milestone with this project. Until now all I achieved was increasing my own knowledge by reading documentation and poking into the code with debug lines. Ok, I also added some code I hoped would make sense, but besides the compilation there was no feedback through a working prototype to see if my code was going in the right direction or if it was utter bollocks. But after today I can say that my buffer flipping and committing code at least produces a picture. And when looking at the FPS counter in Neverball I would even say, that the buffer flipping replacing all the buffer copies already improved the frame rate.

But to test this I first had to solve another problem: The frame rate was always limited to the 60 Hz of my display. The reason was simple: I called present_event_notify only on the frame callback, but in the Xwayland case we can call it directly after the buffer has been sent to the compositor. The only problem I see with this is that the Present extension assumes, that after a new Pixmap has been flipped the old one can be instantly set ready to be used again for new rendering content. But if the last Pixmap’s buffer is still used by the compositor in some way this can lead to tearing.

This hints to a fundamental issue with our approach of using the Present extension in Xwayland. The extension was written with hardware in mind. It assumes a flip happens directly on a screen. There is no intermediate link like a Wayland compositor and if a flip has happened the old buffer is not on the screen anymore. Why do we still try to leverage the Present extension support in Xwayland then? There are two important features of a Wayland compositor we want to have with Xwayland: A tear-free experience for the user and the ability to output a buffer rendered by a direct rendering client on a hardware plane without any copies in between. Every frame is perfect should also remain valid when using some legacy application and that we want no unnecessary copies is simply a question of performance improvements. This is especially important for many of the more demanding games out there, which won’t be Wayland native in the short term and some of them maybe never. Both features need the the full Present extension support in the Xwayland DDX. Without it a direct rendering application would still use the Present extension but only with its fallback code path of copying the Pixmap’s content. And for a tear-free experience we would at least need to sync these copies to the frame events sent by the Wayland compositor or better directly allow multiple buffers, otherwise we would limit our frame rate. In both cases this means again to increase the Present extension support.

I plan on writing about the Present extension in detail in the next week. So if you didn’t fully understand some of the concepts I talked about in this post it could be a good idea to check back.

Originally the plan for this week was to start working on the button mappings, however together with my mentor I decided that it’s better to do the LEDs first. This is because I was sure I could finish this in a few days, and button mappings is definitely going to take much longer than that. So, this week I’ll run you through the implementation of the LED stack page, and the coming weeks I’ll be working on button mappings, profile support and a proper welcome screen, in that order.
July 13, 2017

We managed to get Fedora Workstation 26 out the door this week which I am very happy about. In some ways it was far from our most splashy release as it mostly was about us improving on already released features, like improving the Wayland support and improving the Flatpak support in GNOME Software and improving the Qt integration into GNOME through the QtGNOME platform.

One major thing that is fully functional now though and that I have been testing myself extensively is being able to easily install the NVidia binary driver. If you set up the repository from Negativo17 you should be able to go install the Nvidia driver either using dnf on the command line or by searching for NVidia in GNOME Software, and just install it without any further work thanks to all the effort we and NVidia have been putting into things like glvnd. If you have a workstation with an NVidia card I would say that you have a fully functional system at this point without any hacks or file conflicts with Mesa.

For hybrid graphics laptops this also just works, with the only caveat being that your NVidia card will be engaged at all times once you do this, which is not great for your battery life. We are working to improve this, but it will take some time as it both requires us re-architecting some older parts of the stack and get the Nvidia driver updated to support the new solution.

We do plan on listing the NVidia driver in GNOME Software soon without having to manually setup the repository, so soon we will have a very smooth experience where the Nvidia driver is just a click in the Software store away for our users.

Another item of interest here for the discerning user is that if you are on the NVidia binary driver you will be using X and not Wayland. The reason for this as I have stated in previous blog posts too is that we still have some major gaps on the Wayland side when it comes to dealing with the binary NVidia driver. The biggest one here is that XWayland OpenGL applications doesn’t work, something the team is hard at work trying to resolve. Also the general infrastructure for dealing with hybrid graphics under Wayland is not there yet, but we are working on that too. We have a top notch team looking at the issues here, including Adam Jackson, Jonas Ådahl and Olivier Fourdan, so I am sure we will close this gap as soon as techically possible.

The other big item we have for Fedora Workstation 26 is going to be the formal launch of the Fleet Commander project, with a fully functional release and proper website. We hope to get that set up for next week, so I will blog more about it then. It is a really cool piece of technology which should make deploying Fedora and RHEL in large orgainzations a lot simpler.

As a sidenote, we received our first HDR capable monitor in the office this week, a Dell Ultrasharp UP2718Q. We have another one already ordered and we should be bringing in more in the next Months. This means we can finally seriously kick off figuring out the plumbing work and update the userland stack to have full HDR support under Linux for both media creation and consumption.

July 10, 2017
A little while back I took to wondering why one particular demo from the Sascha Willems vulkan demos was a lot slower on radv compared to amdgpu-pro. Like half the speed slow.

I internally titled this my "no fps left behind" project.

The deferred demo, does an offscreen rendering to 3 2048x2048 color attachments and one 2048x2048 D32S8 depth attachment. It then does a rendering using those down to as 1280x720 screen image.

Bas identifed the first cause was probably the fact we were doing clear color eliminations on the offscreen surfaces when we didn't need to. AMD GPU have a delta-color compression feature, and with certain clear values you don't need to do the clear color eliminations step. This brought me back from about 1/2 the FPS to about 3/4, however it took me quite a while to figure out where the rest of the FPS were hiding.

I took a few diversions in my testing, I pulled in some experimental patches to allow the depth buffer to be texture cache compatible, so could bypass the depth decompression pass, however this didn't seem to budge the number too much.

I found a bunch of registers we were setting different values from -pro, nothing too much came of these.

I found some places we were using a compute shader to fill some DCC or htile surfaces to a value, then doing a clear and overwriting the values, not much help.

I noticed the vertex descriptions and buffer attachments on amdgpu-pro were done quite different to how radv does it. With vulkan you have vertex descriptors and bindings, with radv we generate a set of hw descriptors from the combination of both descriptors and bindings. The pro driver uses typed buffer loads in the shader to embed the descriptor contents in the shader, then it only updates the hw descriptors for the buffer bindings. This seems like it might be more efficient, guess what, no help. (LLVM just grew support for typed buffer loads, so we could probably move to this scheme if we wished now).

I dug out some patches that inline all the push constants and some descriptors so our shaders had less overhead, (really helps our meta shaders have less impact), no helps.

I noticed they export the shader results in a different order from the fragment shader, and always at the end. (no help). The vertex shader emits pos first, (no help). The vertex shader uses off exports for unused channels, (no help).

I went on holidays for a week and came back to stare at the traces again, when I my brain finally noticed something I'd missed. When binding the 3 color buffers, the addresses given as the base address were unusual. A surface has a 40-bit address, normally for alignment and tiling the bottom 16-bits are 0, and we shift 8 of those off completely before writing them. This leaves the bottom 8 bits of the base address has should be 0, and the CIK docs from AMD say that. However the pro traces didn't have these at 0. It appears from earlier evergreen/cayman documents these register control some tiling offset bits. After writing a hacky patch to set the values, I managed to get back the rest of the FPS I was missing in the deferred demo. I discussed with AMD developers, and we worked out the addrlib library has an API for working out these values, and it seems that it allows better memory bandwidth utilisation. I've written a patch to try and use these values correctly and sent it out along with the DCC avoidance patch.

Now I'm not sure this will help any real apps, we may not be hitting limitations in that area, and I'm never happy with the benchmarks I run myself. I thought I saw some FPS difference with some madmax scenes, but I might be lying to myself. Once the patches land in mesa I'm sure others will run benchmarks and we can see if there is any use case where they have an effect. The AMD radeonsi OpenGL driver can also do the same tweaks so hopefully there as well there will be some benefit.

Otherwise I can just write this off as making deferred run at equality and removing at least one of the deltas that radv has compared to the pro driver. Some of the other differences I discovered along the way might also have some promise in other scenarios, so I'll keep an eye on them.

Thanks to Bas, Marek and Christian for looking into what the magic meant!
Due to lots of people telling me LJ is bad, mm'kay, I've migrated to blogspot.

New blog is/will be here: https://airlied.blogspot.com
I'm moving my blog from LJ to blogspot, because people keep telling me LJ is up to no go, like hacking DNC servers and interfering in elections.

Last week I got permission to open source my work on a Mesa-based VC5 3D driver for BCM7268. You can see the announcement here which I won’t replicate on this blog. TWIVC4 is going to be a lot of TWIVC5 from here on out!

I spent the rest of the week working on fixing performance regressions when Raspbian switches from software rendering to to using the vc4 driver without a compositor enabled. The current concern is that window dragging gets slower, and in the worst case can end up with seconds of window dragging queued up behind the motion of the mouse cursor.

Past debugging of mine into how we end up with seconds of window movement queued was fruitless. I suspect it’s “each mouse position is streamed out to the window manager, and the window manager naively queues up a window move for each position it gets, rather than reading through all the position events it gets at once and sending a single move for the last one”. Instead, I worked on just seeing if we can speed up enough that we don’t care.

X11 opaque window dragging is a tough case, because unlike compositors, the contents of the window are stored on the screen (saving gobs of memory, which is important on the Raspberry Pi). When you drag the window, the src and dst regions usually overlap, so we have to be careful to not overwrite src pixels before they’ve been copied to the dst. In software rasterization, we just arrange the memcpy to happen in the correct order. For GL, we have no such control.

What glamor does instead is make a temporary copy of the src pixels, and then copy from the temp to the dst. This creates dependencies between the screen-to-temp and temp-to-screen jobs, so we flush the rendering job at least twice per copy of the window, not counting any flushes that happen for the rendering of the exposed contents from whatever was underneath the window’s old position.

In my tracing, I found that the jobs being generated during window dragging were saying that they could modify any tile on the screen, not just the tiles being affected by the copy (so we read and write each tile on the screen for those jobs). In many paths in glamor we use glScissor() to limit our rendering to some subset of the screen, and this lets the GL keep our jobs trimmed to the appropriate size. However, copies and rectangle fills were scissoring only to the destination drawable’s area, which for LXDE was everything but the global menu bar.

I made a small series that looks at small drawing operations and uses glScissor() to clip to their bounds to help tiled renderers like vc4. I was careful to try to limit the impact of these changes on non-vc4 – fast desktop renderers don’t want to spend the CPU to compute the bounds of the operations when they don’t use the bounds.

It hasn’t completely fixed RPi window dragging, but things are a lot smoother. We may find more paths that need this treatment as more people switch to using vc4 for X11 drawing.

July 07, 2017

After the two-part series on the fundamentals of Xwayland, I want to briefly introduce the basic idea for my Google Summer of Code (GSoC) project for X.Org. This means I’ll talk about how Xwayland currently handles the graphic buffers of its applications, why this leads to tearing and how we plan to change that.

The project has its origin in my work on KWin. In fact there is some connection to my unsuccessful GSoC application from last year on atomic mode setting and layered compositing in KWin. You can read up on these notions and the previous application in some of my older posts, but the relevant part of it to this year’s project is in short the transfer of application graphic buffers directly onto the screen without the Wayland server compositing them into a global scene before that. This can be done by putting the buffers on some overlay planes and let the hardware do the compositing of these planes into a background provided by the compositor or in the simpler case by putting a single buffer of a full screen application directly onto the primary plane.

At the beginning of the year I was working on enabling this simpler case in KWin. In a first working prototype I was pretty sure I got the basic implementation right, but my test, a full screen video in VLC, showed massive tearing. Of course I suspected at first my own code to be the problem, but in this case it wasn’t. Only after I wrote a second test application, which was a simple QML application playing the same video in full screen and showing no tearing, I had the suspicion that the problem wasn’t my code but Xwayland, since VLC was running on Xwayland while my test application was Wayland native.

Indeed the Wayland protocol should prevent tearing overall, as long as the client respects the compositor’s messages. It works like this: After committing a newly drawn buffer to the server, the client is not allowed to touch it anymore and only after the compositor has sent the release event, the client is again allowed to repaint or delete it. If the client needs to repaint in the meantime it is supposed to allocate a different buffer object. But this is exactly, what Xwayland based applications are not doing, as Daniel Stone was quick to tell me after I asked for help from him for the tearing issues I experienced.

Under Xwayland an app only ever uses one buffer at all and repaints are always done into this one buffer. This means that the buffer is given to the compositor at some point but the application doesn’t stop repainting into it. So in my case the buffer content changed, although at the same time it was presented to the user on the primary plane. The consequence is tearing. Other developers noticed that as well the same time around as documented in this bug report.

The proposed solution is to bolster the Present extension support in Xwayland. In theory with that extension an X based application should be able to paint into more than one Pixmap, which then translate to different Wayland buffers. On the other side Xwayland notifies the app through the Present extension when it can reuse one of its Pixmaps based on the associated Wayland buffer event. The Present extension is a relatively new extension to the Xserver, but is already supported by most of the more interesting applications. It was written by Keith Packard, and you can read more about it on his blog. In theory it should only be necessary to add support for the extension to the Xwayland DDX. But there are some issues in the DIX side of the extension code, which first need to be ironed out. I plan on writing more about the Present extension in general and the limitations we encounter in our Xwayland use case in the next articles.

This week was a good week with loads of progress. I’ll just go through each item one by one, as usual. MouseMap merged Last Friday I opened the pull request to merge the MouseMap widget. After a bit of a discussion around preferences, my mentor and I decided to merge ongoing work in a rewrite branch. This keeps the master branch functional (while still receiving improvements, such as adding new functionality to the ratbagd bindings or a Gtk.
July 06, 2017
While my colleagues are working on mice that shine in all kinds of different colours, I went towards the old school.

For around 10 units of currency, you should be able to find the uDraw tablet for the PlayStation 3, the drawing tablet that brought down a company.



The device contains a large touchpad which can report one or two touches, for right-clicking (as long as the fingers aren't too close), a pen interface which will make the cheapest of the cheapest Wacom tablets feel like a professional tool from 30 years in the future, a 4-button joypad (plus Start/Select/PS) with the controls either side of the device, and an accelerometer to play Marble Madness with.

The driver landed in kernel 4.10. Note that it only supports the PlayStation 3 version of the tablet, as the Wii and XBox 360 versions require receivers that aren't part of the package. Here, a USB dongle should be provided.

Recommended for: point'n'click adventure games, set-top box menu navigation.

The second driver landed in kernel 4.12, and is a primer for more work to be done. This driver adds support for the Retrode 2's joypad adapters.

The Retrode is a USB console cartridge reader which makes Sega Mega Drive (aka Genesis) and Super Nintendo (aka Super Famicom) cartridges show up as files on a mass storage devices in your computer.



It also has 4 connectors for original joypads which the aforementioned driver now splits up and labels, so you know which is which, as well as making the mouse work out of the box. I'd still recommend picking up the newer optical model of that mouse, from Hyperkin. Moving a mouse with a ball in it is like weighing a mobile phone from that same era.

I will let you inspect the add-ons for the device, like support for additional Nintendo 64 pads and cartridges, and Game Boy/GB Color/GB Advance, and Sega Master System adapters.

Recommended for: cartridge-based retro games, obviously.

Integrated firmware updates, and better integration with Games is in the plans.

I'll leave you with this video, which shows how you could combine GNOME Games, a Retrode, this driver, a SNES mouse, and a cartridge of Mario Paint. Let's get creative :)

Some time ago I promised to write a bit more about how shadow mapping works. It has taken me a while to bring myself to actually deliver on that front, but I have finally decided to put together some posts on this topic, this being the first one. However, before we cover shadow mapping itself we need to cover some lighting basics first. After all, without light there can’t be shadows, right?

This post will introdcuce the popular Phong reflection model as the basis for our lighting model. A lighting model provides a simplified representation of how light works in the natural world that allows us to simulate light in virtual scenes at reasonable computing costs. So let’s dive into it:

Light in the natural world

In the real world, the light that reaches an object is a combination of both direct and indirect light. Direct light is that which comes straight from a light source, while indirect light is the result of light rays hitting other surfaces in the scene, bouncing off of them and eventually reaching the object as a result, maybe after multiple reflections from other objects. Because each time a ray of light hits a surface it loses part of its energy, indirect light reflection is less bright than direct light reflection and its color might have been altered. The contrast between surfaces that are directly hit by the light source and surfaces that only receive indirect light is what creates shadows. A shadow isn’t but the part of a scene that doesn’t receive direct light but might still receive some amount of (less intense) indirect light.

Direct vs Indirect light

Light in the digital world

Unfortunately, implementing realistic light behavior like that is too expensive, specially for real-time applications, so instead we use simplifications that can produce similar results with much lower computing requirements. The Phong reflection model in particular describes the light reflected from surfaces or emitted by light sources as the combination of 3 components: diffuse, ambient and specular. The model also requires information about the direction in which a particular surface is facing, provided via vectors called surface normals. Let’s introduce each of these concepts:

Surface normals

When we study the behavior of light, we notice that the direction in which surfaces reflect incoming light affects our perception of the surface. For example, if we lit a shiny surface (such as a piece of metal) using a strong light shource so that incoming light is reflected off the surface in the exact opposite direction in which we are looking at it, we will see a strong reflection in the form of highlights. If we move around so that we look at the same surface from a different angle, then we will see the reflection get dimmer and the highlights will eventually disappear. In order to model this behavior we need to know the direction in which the surfaces we render reflect incoming light. The way to do this is by associating vectors called normals with the surfaces we render so that shaders can use that information to produce lighting calculations akin to what we see in the natural world.

Usually, modeling programs can compute normal vectors for us, even model loading libraries can do this work automatically, but some times, for example when we define vertex meshes programatically, we need to define them manually. I won’t covere here how to do this in the general, you can see this article from Khronos if you’re interested in specific algorithms, but I’ll point out something relevant: given a plane, we can compute normal vectors in two opposite directions, one is correct for the front face of the plane/polygon and the other one is correct for the back face, so make sure that if you compute normals manually, you use the correct direction for each face, otherwise you won’t be reflecting light in the correct direction and results won’t be as you expect.

Light reflected using correct normal vector for the front face of the triangle

In most scenarios, we only render the front faces of the polygons (by enabling back face culling) and thus, we only care about one of the normal vectors (the one for the front face).

Another thing to notice about normal vectors is that they need to be transformed with the model to be correct for transformed models: if we rotate a model we need to rotate the normals too, since the faces they represent are now rotated and thus, their normal directions have rotated too. Scaling also affects normals, specifically if we don’t use regular scaling, since in that case the orientation of the surfaces may change and affect the direction of the normal vector. Because normal vectors represent directions, their position in world space is irrelevant, so for the purpose of lighting calculations, a normal vector such as (1, 0, 0) defined for a surface placed at (0, 0, 0) is still valid to represent the same surface at any other position in the world; in other words, we do not need to apply translation transforms to our normal vectors.

In practice, the above means that we want to apply the rotation and scale transforms from our models to their normal vectors, but we can skip the translation transform. The matrix representing these transforms is usually called the normal matrix. We can compute the normal matrix from our model matrix by computing the transpose of the inverse of the 3×3 submatrix of the model matrix. Usually, we’d want to compute this matrix in the application and feed it to our vertex shader like we do with our model matrix, but for reference, here is how this can be achieved in the shader code itself, plus how to use this matrix to transform the original normal vectors:

mat3 NormalMatrix = transpose(inverse(mat3(ModelMatrix)));
vec3 out_normal = normalize(NormalMatrix * in_normal);

Notice that the code above normalizes the resulting normal before it is fed to the fragment shader. This is important because the rasterizer will compute normals for all fragments in the surface automatically, and for that it will interpolate between the normals for each vertex we emit. For the interpolated normals to be correct, all vertex normals we output in the vertex shader must have the same length, otherwise the larger normals will deviate the direction of the interpolated vectors towards them because their larger size will increase their weight in the interpolation computations.

Finally, even if we emit normalized vectors in the vertex shader stage, we should note that the interpolated vectors that arrive to the fragment shader are not guaranteed to be normalized. Think for example of the normal vectors (1, 0, 0) and (0, 1, 0) being assigned to the two vertices in a line primitive. At the half-way point in between these two vertices, the interpolator will compute a normal vector of (0.5, 0.5, 0), which is not unit-sized. This means that in the general case, input normals in the fragment shader will need to be normalized again even if have normalized vertex normals at the vertex shader stage.

Diffuse reflection

The diffuse component represents the reflection produced from direct light. It is important to notice that the intensity of the diffuse reflection is affected by the angle between the light coming from the source and the normal of the surface that receives the light. This makes a surface looking straight at the light source be the brightest, with reflection intensity dropping as the angle increases:

Diffuse light (spotlight source)

In order to compute the diffuse component for a fragment we need its normal vector (the direction in which the surface is facing), the vector from the fragment’s position to the light source, the diffuse component of the light and the diffuse reflection of the fragment’s material:

vec3 normal = normalize(surface_normal);
vec3 pos_to_light_norm = normalize(pos_to_light);
float dp_reflection = max(0.0, dot(normal, pos_to_light_norm));
vec3 diffuse = material.diffuse * light.diffuse * dp_reflection;

Basically, we multiply the diffuse component of the incoming light with the diffuse reflection of the fragment’s material to produce the diffuse component of the light reflected by the fragment. The diffuse component of the surface tells how the object absorbs and reflects incoming light. For example, a pure yellow object (diffuse material vec3(1,1,0)) would absorb the blue component and reflect 100% of the red and green components of the incoming light. If the light is a pure white light (diffuse vec3(1,1,1)), then the observer would see a yellow object. However, if we are using a red light instead (diffuse vec3(1,0,0)), then the light reflected from the surface of the object would only contain the red component (since the light isn’t emitting a green component at all) and we would see it red.

As we said before though, the intensity of the reflection depends on the angle between the incoming light and the direction of the reflection. We account for this with the dot product between the normal at the fragment (surface_normal) and the direction of the light (or rather, the vector pointing from the fragment to the light source). Notice that because the vectors that we use to compute the dot product are normalized, dp_reflection is exactly the cosine of the angle between these two vectors. At an angle of 0º the surface is facing straight at the light source, and the intensity of the diffuse reflection is at its peak, since cosine(0º)=1. At an angle of 90º (or larger) the cosine will be 0 or smaller and will be clamped to 0, meaning that no light is effectively being reflected by the surface (the computed diffuse component will be 0).

Ambient reflection

Computing all possible reflections and bounces of all rays of light from each light source in a scene is way too expensive. Instead, the Phong model approximates this by making indirect reflection from a light source constant across the scene. In other words: it assumes that the amount of indirect light received by any surface in the scene is the same. This eliminates all the complexity while still producing reasonable results in most scenarios. We call this constant factor ambient light.

Ambient light

Adding ambient light to the fragment is then as simple as multiplying the light source’s ambient light by the material’s ambient reflection. The meaning of this product is exactly the same as in the case of the diffuse light, only that it affects the indirect light received by the fragment:

vec3 ambient = material.ambient * light.ambient;

Specular reflection

Very sharp, smooth surfaces such as metal are known to produce specular highlights, which are those bright spots that we can see on shiny objects. Specular reflection depends on the angle between the observer’s view direction and the direction in which the light is reflected off the surface. Specifically, the specular reflection is strongest when the observer is facing exactly in the opposite direction in which the light is reflected. Depending on the properties of the surface, the specular reflection can be more or less focused, affecting how the specular component scatters after being reflected. This property of the material is usually referred to as its shininess.

Specular light

Implementing specular reflection requires a bit more of work:

vec3 specular = vec3(0);
vec3 light_dir_norm = normalize(vec3(light.direction));
if (dot(normal, -light_dir_norm) >= 0.0) {
   vec3 reflection_dir = reflect(light_dir_norm, normal);
   float shine_factor = dot(reflection_dir, normalize(in_view_dir));
   specular = light.specular.xyz * material.specular.xyz *
         pow(max(0.0, shine_factor), material.shininess.x);
}

Basically, the code above checks if there is any specular reflection at all by computing the cosine of the angle between the fragment’s normal and the direction of the light (notice that, once again, both vectors are normalized prio to using them in the call to dot()). If there is specular reflection, then we compute how shiny the reflection is perceived by the viewer based on the angle between the vector from this fragment to the observer (in_view_dir) and the direction of the light reflected off the fragment’s surface (reflection_dir). The smaller the angle, the more parallel the directions are, meaning that the camera is receiving more reflection and the specular component received is stronger. Finally, we modulate the result based on the shininess of the fragment. We can compute in_view_dir in the vertex shader using the inverse of the View matrix like this:

mat4 ViewInv = inverse(View);
out_view_dir =
   normalize(vec3(ViewInv * vec4(0.0, 0.0, 0.0, 1.0) - world_pos));

The code above takes advantage of the fact that camera transformations are an illusion created by applying the transforms to everything else we render. For example, if we want to create the illusion that the camera is moving to the right, we just apply a translation to everything we render so they show up a bit to the left. This is what our View matrix achieves. From the point of view of GL or Vulkan, the camera is always fixed at (0,0,0). Taking advantage of this, we can compute the position of the virtual observer (the camera) in world space coordinates by applying the inverse of our camera transform to its fixed location (0,0,0). This is what the code above does, where world_pos is the position of this vertex in world space and View is the camera’s view matrix.

In order to produce the final look of the scene according to the Phong reflection model, we need to compute these 3 components for each fragment and add them together:

out_color  = vec4(diffuse + ambient + specular, 1.0)
Diffuse + Ambient + Specular (spotlight source)

Attenuation

In most scenarios, light intensity isn’t constant across the scene. Instead, it is brightest at its source and gets dimmer with distance. We can easily model this by adding an attenuation factor that is multiplied by the distance from the fragment to the light source. Typically, the intensity of the light decreases quite fast with distance, so a linear attenuation factor alone may not produce the best results and a quadratic function is preferred:

float attenuation = 1.0 /
    (light.attenuation.constant +
     light.attenuation.linear * dist +
     light.attenuation.quadratic * dist * dist);

diffuse = diffuse * attenuation;
ambient = ambient * attenuation;
specular = specular * attenuation;

Of course, we may decide not to apply attenuation to the ambient component at all if we really want to make it look like it is constant across the scene, however, do notice that when multiple light sources are present, the ambient factors from each source will accumulate and may produce too much ambient light unless they are attenuated.

Types of lights

When we model a light source we also need to consider the kind of light we are manipulating:

Directional lights

These are light sources that emit rays that travel along a specific direction so that all are parallel to each other. We typically use this model to represent bright, distant light sources that produce constant light across the scene. An example would be the sun light. Because the distance to the light source is so large compared to distances in the scene, the attenuation factor is irrelevant and can be discarded. Another particularity of directional light sources is that because the light rays are parallel, shadows casted from them are regular (we will talk more about this once we cover shadow mapping in future posts).

Directional light

If we had used a directional light in the scene, it would look like this:

Scene with a directional light

Notice how the brightness of the scene doesn’t lower with the distance to the light source.

Point lights

These are light sources for which light originates at a specific position and spreads outwards in all directions. Shadows casted by point lights are not regular, instead they are projected. An example would be the light produced by a light bulb. The attenuation code I showed above would be appropriate to represent point lights.

Point light

Here is how the scene would look like with a point light:

Scene with a point light

In this case, we can see how attenuation plays a factor and brightness lowers as we walk away from the light source (which is close to the blue cuboid).

Spotlights

This is the light source I used to illustrate the diffuse, ambient and specular components. They are similar to point lights, is the sense that light originates from a specific point in space and spreads outwards, however, instead of scattering in all directions, rays scatter forming a cone with the tip at the origin of the light. The angle formed by the lights’s direction and the sides of the cone is usually called the cutoff angle, because not light is casted outside its limits. Flashlights are a good example of this type of light.

Spotlight

In order to create spotlights we need to consider the cutoff angle of the light and make sure that no diffuse or specular component is reflected by a fragment which is beyond the cutoff threshold:

vec3 light_to_pos_norm = -pos_to_light_norm;
float dp = dot(light_to_pos_norm, light_dir_norm);
if (dp <= light.cutoff) {
   diffuse = vec3(0);
   specular = vec3(0);
}

In the code above we compute the cosine of the angle between the light’s direction and the vector from the light to the fragment (dp). Here, light.cutoff represents the cosine of the spotlight’s cutoff angle too, so when dp is smaller it means that the fragment is outside the light cone emitted by the spotlight and we remove its diffuse and specular reflections completely.

Multiple lights

Handling multiple lights is easy enough: we only need to compute the color contribution for each light separately and then add all of them together for each fragment (pseudocode):

vec3 fragColor = vec3(0);
foreach light in lights
    fragColor += compute_color_for_light(light, ...);
...

Of course, light attenuation plays a vital role here to limit the area of influence of each light so that scenes where we have multiple lights don’t get too bright.

An important thing to notice above the pseudocode above is that this process involves looping through costy per-fragment light computations for each light source, which can lead to important performance hits as the number of lights in the scene increases. This shading model, as described here, is called forward rendering and it has the benefit that it is very simple to implement but its downside is that we may incur in many costy lighting computations for fragments that, eventually, won’t be visible in the screen (due to them being occluded by other fragments). This is particularly important when the number of lights in the scene is quite large and its complexity makes it so that there are many occluded fragments. Another technique that may be more suitable for these situations is called deferred rendering, which postpones costy shader computations to a later stage (hence the word deferred) in which we only evaluate them for fragments that are known to be visible, but that is a topic for another day, in this series we will focus on forward rendering only.

Lights and shadows

For the purpose of shadow mapping in particular we should note that objects that are directly lit by the light source reflect all 3 of the light components, while objects in the shadow only reflect the ambient component. Because objects that only reflect ambient light are less bright, they appear shadowed, in similar fashion as they would in the real world. We will see the details how this is done in the next post, but for the time being, keep this in mind.

Source code

The scene images in this post were obtained from a simple shadow mapping demo I wrote in Vulkan. The source code for that is available here, and it includes also the shadow mapping implementation that I’ll cover in the next post. Specifically relevant to this post are the scene vertex and fragment shaders where lighting calculations take place.

Conclusions

In order to represent shadows we first need a means to represent light. In this post we discussed the Phong reflection model as a simple, yet effective way to model light reflection in a scene as the addition of three separate components: diffuse, ambient and specular. Once we have a representation of light we can start discussing shadows, which are parts of the scene that only receive ambient light because other objects occlude the diffuse and specular components of the light source.

July 04, 2017

I (finally!) merged a patchset to detect palms based on pressure into libinput. This should remove a lot of issues that our users have seen with accidental pointer movement. Palm detection in libinput previously used two approaches: disable-while-typing and an edge-based approach. The former simply ignores touchpad events while keyboard events are detected, the latter ignores touches that happen in the edge zones of the touchpad where real interaction is unlikely. Both approaches have the obvious disadvantages: they're timeout- and location-dependent, causing erroneous pointer movements. But their big advantage is that they work even on old touchpads where a lot of other information is unreliable. Touchpads are getting better, so it's time to make use of that.

The new feature is relatively simple: libinput looks at per-touch pressure and if that pressure hits a given threshold, the touch is regarded as palm. Once a palm, that touch will be ignored until touch up. The threshold is intended to be high enough that it cannot easily be hit. At least on the touchpads I have available for testing, I have to go through quite some effort to trigger palm detection with my finger.

Pressure on touchpads is unfortunately hardware-dependent and we can expect most laptops to have different pressure thresholds. For our users this means that the feature won't immediately work perfectly, it will require a lot of hwdb entries. libinput now ships a libinput measure touchpad-pressure tool to experiment with the various pressure thresholds. This makes it easy to figure out the right pressure threshold and submit a bug report (or patch) for libinput to get the pressure threshold updated. The documentation for this tool is available as part of libinput's online documentation.

TLDR: if libinput seems to misdetect touches as palms, figure out the right threshold with libinput measure touchpad-pressure and file a bug report so we can merge this into our hwdb.

July 01, 2017

DRM leasing part three (vblank)

The last couple of weeks have been consumed by getting frame sequence numbers and events handled within the leasing environment (and Vulkan) correctly.

Vulkan EXT_display_control extension

This little extension provides the bits necessary for applications to track the display of frames to the user.

VkResult
vkGetSwapchainCounterEXT(VkDevice           device,
             VkSwapchainKHR         swapchain,
             VkSurfaceCounterFlagBitsEXT    counter,
             uint64_t           *pCounterValue);

This function just retrieves the current frame count from the display associated with swapchain.

VkResult
vkRegisterDisplayEventEXT(VkDevice          device,
              VkDisplayKHR          display,
              const VkDisplayEventInfoEXT   *pDisplayEventInfo,
              const VkAllocationCallbacks   *pAllocator,
              VkFence           *pFence);

This function creates a fence that will be signaled when the specified event happens. Right now, the only event supported is when the first pixel of the next display refresh cycle leaves the display engine for the display. If you want something fancier (like two frames from now), you get to do that on your own using this basic function.

drmWaitVBlank

drmWaitVBlank is the existing interface for all things sequence related and has three modes (always nice to have one function do three things, I think). It can:

  1. Query the current vblank number
  2. Block until a specified vblank number
  3. Queue an event to be delivered at a specific vblank number

This interface has a few issues:

  • It has been kludged into supporting multiple CRTCs by taking bits from the 'type' parameter to hold a 'pipe' number, which is the index in the kernel into the array of CRTCs.

  • It has a random selection of 'int' and 'long' datatypes in the interface, making it need special helpers for 32-bit apps running on a 64-bit kernel.

  • Times are in microseconds, frame counts are 32 bits. Vulkan does everything in nanoseconds and wants 64-bits of frame counts.

For leases, figuring out the index into the kernel list of crtcs is pretty tricky -- our lease has a subset of those crtcs, so we can't actually compute the global crtc index.

drmCrtcGetSequence

int drmCrtcGetSequence(int fd, uint32_t crtcId,
               uint64_t *sequence, uint64_t *ns);

Here's a simple new function — hand it a crtc ID and it provides the current frame sequence number and the time when that frame started (in nanoseconds).

drmCrtcQueueSequence

int drmCrtcQueueSequence(int fd, uint32_t crtcId,
                 uint32_t flags, uint64_t sequence,
             uint64_t user_data);

struct drm_event_crtc_sequence {
    struct drm_event    base;
    __u64           user_data;
    __u64           time_ns;
    __u64           sequence;
};

This will cause a CRTC_SEQUENCE event to be delivered at the start of the specified frame sequence. That event will include the frame when the event was actually generated (in case it's late), along with the time (in nanoseconds) when that frame was started. The event also includes a 64-bit user_data value, which can be used to hold a pointer to whatever data the application wants to see in the event handler.

The 'flags' argument contains a combination of:

#define DRM_CRTC_SEQUENCE_RELATIVE      0x00000001  /* sequence is relative to current */
#define DRM_CRTC_SEQUENCE_NEXT_ON_MISS      0x00000002  /* Use next sequence if we've missed */
#define DRM_CRTC_SEQUENCE_FIRST_PIXEL_OUT   0x00000004  /* Signal when first pixel is displayed */

These are similar to the values provided for the drmWaitVBlank function, except I've added a selector for when the event should be delivered to align with potential future additions to Vulkan. Right now, the only time you can ask for is first-pixel-out, which says that the event should correspond to the display of the first pixel on the screen.

DRM events → Vulkan fences

With the kernel able to deliver a suitable event at the next frame, all the Vulkan code needed was a to create a fence and hook it up to such an event. The existing fence code only deals with rendering fences, so I added window system interface (WSI) fencing infrastructure and extended the radv driver to be able to handle both kinds of fences within that code.

Multiple waiting threads

I've now got three places which can be waiting for a DRM event to appear:

  1. Frame sequence fences.

  2. Wait for an idle image. Necessary when you want an image to draw the next frame to.

  3. Wait for the previous flip to complete. The kernel can only queue one flip at a time, so we have to make sure the previous flip is complete before queuing another one.

Vulkan allows these to be run from separate threads, so I needed to deal with multiple threads waiting for a specific DRM event at the same time.

XCB has the same problem and goes to great lengths to manage this with a set of locking and signaling primitives so that only one thread is ever doing poll or read from the socket at time. If another thread wants to read at the same time, it will block on a condition variable which is then signaled by the original reader thread at the appropriate time. It's all very complicated, and it didn't work reliably for a number of years.

I decided to punt and just create a separate thread for processing all DRM events. It blocks using poll(2) until some events are readable, processes those and then broadcasts to a condition variable to notify any waiting threads that 'something' has happened. Each waiting thread simply checks for the desired condition and if not satisfied, blocks on that condition variable. It's all very simple looking, and seems to work just fine.

Code Complete, Validation Remains

At this point, all of the necessary pieces are in place for the VR application to take advantage of an HMD using only existing Vulkan extensions. Those will be automatically mapped into DRM leases and DRM events as appropriate.

The VR compositor application is working pretty well; tests with Dota 2 show occasional jerky behavior in complex scenes, so there's clearly more work to be done somewhere. I need to go write a pile of tests to independently verify that my code is working. I wonder if I'll need to wire up some kind of light sensor so I can actually tell when frames get displayed as it's pretty easy to get consistent-but-wrong answers in this environment.

Source Code

  • Linux. This is based off of a reasonably current drm-next branch from Dave Airlie. 965 commits past 4.12 RC3.

    git://people.freedesktop.org/~keithp/linux drm-lease-v3

  • X server (which includes xf86-video-modesetting). This is pretty close to master.

    git://people.freedesktop.org/~keithp/xserver drm-lease

  • RandR protocol changes

    git://people.freedesktop.org/~keithp/randrproto drm-lease

  • xcb proto (no changes to libxcb sources, but it will need to be rebuilt)

    git://people.freedesktop.org/~keithp/xcb/proto drm-lease

  • DRM library. About a dozen patches behind master.

    git://people.freedesktop.org/~keithp/drm drm-lease

  • Mesa. Branched early this month (4 June), this is pretty far from master.

    git://people.freedesktop.org/~keithp/mesa drm-lease

June 30, 2017

Last week in part one of this two part series about the fundamentals of Xwayland, we treated Xwayland like a black box. We stated what its purpose is and gave a rough overview on how it connects to its environment, notably its clients and the Wayland compositor. In a sense this was only a teaser, since we didn’t yet look at Xwayland’s inner workings. So welcome to part two, where we do a deep dive into its code base!

You can find the Xwayland code base here. Maybe to your surprise this is just the code of X.org’s Xserver, which we will just refer to as the Xserver in the rest of this text. But as a reminder from part one: Xwayland is only a normal Xserver “with a special backend written to communicate with the Wayland compositor active on your system.” This backend is located in /hw/xwayland. To understand why we find this special backend here and what I mean with an Xserver backend at all, we have to first learn some Xserver fundamentals.

DIX and DDX

The hw subdirectory is the Device Dependent X (DDX) part of the Xserver. All other directories in the source tree form the Device Independent X (DIX) part. This structuring is an important abstraction in the Xserver. Like the names suggest the DIX part is supposed to be generic enough to be the same on every imaginable hardware platform. The word hardware hereby should be understood in an abstract way as being some sort of environment the Xserver works in and has to talk to, which could be the kernel with its DRM subsystem and hardware drivers or as we already know a Wayland compositor. On the other side all code, that is potentially different with respect to the environment the Xserver is compiled for is bundled into the DDX part. Since this code is by its very definition mostly responsible for establishing and maintaining the required communication channels with the environment, we can indeed call the platform specific code paths in DDX the Xserver’s backends.

I want to emphasize that the Xserver is compiled for different environments, because we are now able to understand how the Xorg and Xwayland binaries we talked about in part one and that both implement a full Xserver come into existence: Autotools, the build system of the Xserver, is told by configuration parameters before compilation what the intended target platforms are. It then will use for each enabled target platform the respective subdirectory in hw to compile a binary with this platform’s appropriate DDX plus the generic DIX from the other top level directories. For example to compile only the Xwayland binary, you can use this command from the root of the source tree:

./autogen.sh --prefix=/usr --disable-docs --disable-devel-docs \
  --enable-xwayland --disable-xorg --disable-xvfb --disable-xnest \
  --disable-xquartz --disable-xwin

Coming back to the functionality let’s look at two examples in order to better understand the DIX and DDX divide and how the two parts interact with each other. Take first the concept of regions: A region specifies a certain portion of the view displayed to the user. It is defined by values for its width, height and position in some coordinate system. How regions work is therefore completely independent on the choice of hardware the Xserver runs on. That allowed the Xserver creators to put all the region code in the DIX part of the server.

Talking about regions in a view we think directly of the screen this view is displayed on. That’s the second example. We can always assume that there is some sort of real or emulated screen or even multiple of them to display our view. But how these screens and their properties are retrieved is dependent on the environment. So there needs to be some “screen code” in DDX, but on the other hand we want to move as much logic as possible in the DIX to avoid rewriting shared functionality for different platforms.

The Xserver is equipped with tools to facilitate this dichotomy. In our example about screens DIX represents the generic part of such a screen in its _Screen struct. But the struct features also the void pointer field devPrivate, which can be set by the DDX part to some struct, that then provides the device dependent information for the screen. When DIX then calls DDX to do something concerning the screen, DIX also hands over a _Screen pointer and DDX can retrieve these information through the devPrivate pointer. The private resource pointer is a tool featured in several core objects of the Xserver. For example we can also find it in the _Window struct for windows.

Besides this information sharing between DIX and DDX there are of course also procedures triggered in one part and reaching into the other one. And these procedures run according to the main event loop. We will learn more about them when we now finally analyze the Xwayland DDX code itself.

The Xwayland DDX

The names of the source files in the /hw/xwayland directory already indicate what they are supposed to do. Luckily there are not many of them and most of the files are rather compact. It’s quite a feat that the creators of Xwayland were able to provide X backward compatibility in a Wayland session with only that few lines of code added to the generic part of a normal Xserver. This is of course only possible thanks to the abstractions described above.

But coming back to the files here’s a table of all the files with short descriptions:

Files Description
xwayland.h
xwayland.c
Basically the entry point to everything else, define and implement the most central structs and functions of the Xwayland DDX.
xwayland-output.c Provides a representation of a display/output. All its data is of course received from the Wayland server.
xwayland-cvt.c Supports the output creation by generating a display mode calculated from available information.
xwayland-input.c Deals with inputs provided by mice and other input devices. As you can see by its size, it’s not the most straight forward area to work on.
xwayland-cursor.c Makes a cursor appear. That is in a graphic pipeline often treated as a special case to reduce repaints.
xwayland-glamor.c
xwayland-shm.c
Provide two different ways for allocating graphic buffers.
xwayland-glamor-xv.c
xwayland-vidmode.c
Support for hardware accelerated video playback and older games, what is in parts not yet fully functional.

In the following we will restrict our analysis to the xwayland.* files, in order to keep the growing length of this article in check.

Some basic structs and functions also shared with the other source files are defined in the header file xwayland.h. A good first point to remember is, that all structs and functions with names starting on xwl_ are only known to the Xwayland DDX and won’t be called from anywhere else. But at the beginning of the xwayland.c file we find some methods without the prefix. They are only defined in the DIX and their implementation is required to make Xwayland a fully functional DDX.

Scrolling down to the end of the file we see the main entry point to the DDX on server startup, the InitOutput method. If you look closely you will notice a call to AddScreen, where we also hook up an Xwayland internal screen init function as one of its arguments. But it’s only called once! So what about multiple screens? The explanation is, that Xwayland uses the RandR extension for its screen management and here only asks for the creation of one screen struct as a dummy, which holds on runtime some global information about the Wayland environment. We looked at this particular screen struct in the previous chapter as an example for information sharing between DIX and DDX through void pointers and that these pointers are set by the DDX.

Although it’s only a dummy, we can still follow this now live in action in the hooked up init function xwl_screen_init. Here we set with the help of some DIX methods a hash key to later identify the data field again and then set the data, which is an xwl_screen struct with static information about the Wayland environment the Xwayland server is deployed in.

In the hooked up init function the later manipulation of the function pointers RealizeWindow, UnrealizeWindow and so on is also quite interesting. I asked Daniel about it, because I didn’t understand at all the steps done here as well as similar ones later in the involved functions xwl_realize_window, xwl_unrealize_window and so on. Daniel explained the mechanism well to me and it is quite nifty indeed. Basically thanks to this trick, called wrapping, Xwayland and other DDX can intercept DIX calls to a procedure like RealizeWindow, execute their own code, and then go on with the procedure looking to the DIX like it never happened.

In the case of RealizeWindow, which is called when a window was created and is now ready to be displayed, we intercept it with xwl_realize_window, where an Xwayland internal representation of type struct xwl_window is allocated with all the Xwayland specific additional information, in particular a Wayland surface. At the end the request to create the surface is sent to the Wayland server via the Wayland protocol. You can probably imagine what UnrealizeWindow and the wrapped xwl_unrealize_window is supposed to do and that it does this in a very similar way.

As a last point let’s look at the event loop and the buffer dispatch of possibly new or changed graphical content. We have block_handler, which was registered in xwl_screen_init to the DIX, and gets called continuously throughout the event loop. From here we call into a global damage posting function and from there for each window into xwl_window_post_damage. If we’re lucky we get a buffer with hardware acceleration from the implementation in xwayland-glamor.c or otherwise without acceleration from the one in xwayland-shm.c, attach it to the surface and fire it away. In the next event loop we play the same game.

Forcing an end to this article, what we ignored in total is input handling in Xwayland and we also only touched the graphics buffer in the end. But at least the graphic buffers we’ll discuss in the coming weeks exhaustively, since my Google Summer of Code project is all about these little guys.

This week Piper saw some progress again! Today I opened the pull request for the MouseMap that I’ve been working on for the past two and a half weeks now. I’ll discuss the changes made since the last blog later; first, I want to highlight the other work I did the past week. A major milestone this week is the merging of ratbagd and libratbag. While Piper shouldn’t notice any of this (it talks to ratbagd over DBus), it’s still a highlight I want to mention.