Blog on Erik “kusma” Faye-Lund

Zink brings conformant OpenGL on Imagination GPUs

kusmabite@gmail.com (Erik Faye-Lund) — Thu, 06 Jul 2023 21:15:00 +0200

Today, Imagination Technologies announced some very exciting news: they are now using Zink for full OpenGL 4.6 support! Collabora had the pleasure of working together with engineers from Imagination to make this a reality, and it’s very rewarding to now be able show the results to the world!

More importantly, this is the first time we’ve seen a hardware vendor trust the OpenGL-on-Vulkan Mesa driver enough to completely side-step a native OpenGL driver and use it in a shipping product. It’s wonderful to see that Zink can realistically be used as a work-horse, especially in a high-performance graphics setting.

Zink started out as a small R&D project at Collabora, but has since grown to be a full-on community project. None of this would have been possible without the awesome work done by Mike and the other Zink contributors!

Conformance

One small detail from Imagination’s post that I think is important to highlight is that the solution is officially conformant. This is the first product to be officially conformant using Zink, but it’s not going to be the last! In fact, we only need one more conformant implementation before Zink itself is conformant as a generic layered implementation, according to the Khronos Conformant Product Criteria.

An Open Source graphics future

In the not too distant future, we should be able to combine Zink with the in-progress open source driver from Imagination, and that’s when things will really start to shine for the open source graphics stack on Imagination hardware. So there’s plenty more to look forward to here!

Zink: Summer 2021 Update

kusmabite@gmail.com (Erik Faye-Lund) — Mon, 14 Jun 2021 20:00:00 +0100

There’s a lot that has happened in the world of Zink since my last update, so let’s see if I can bring you up to date on the most important stuff.

Upstream development

Gosh, when I last blogged about Zink, it hadn’t even landed upstream in Mesa yet! Well, by now it’s been upstream for quite a while, and most development has moved there.

At the time of writing, we have merged 606 merge-requests labeled “zink”. The current tip of mesa’s main branch is totaling 1717 commits touching the src/gallium/drivers/zink/ sub-folder, written by 42 different contributors. That’s pretty awesome in my eyes, Zink has truly become a community project!

Another noteworthy change is that Mike Blumenkrantz has come aboard the project, and has churned out an incredible amount of improvements to Zink! He got hired by Valve to work on Zink (among other things), and is now the most prolific contributor, with more than twice the amount of commits than I have written.

If you want a job in Open Source graphics, Zink has a proven track-record as a job-creator! 😄

In addition to Mike, there’s some other awesome people who have been helping out lately.

Half-Life 2 running with Zink.

OpenGL 4.6 support

Thanks to a lot of hard work by Mike assisted by Dave Airlie and Adam Jackson, both of RedHat, Zink is now able to expose the OpenGL 4.6 (Core Profile) feature set, given enough Vulkan features! 🎉

Please note that this doesn’t mean that Zink is yet a conformant implementation, there’s some details left to be ironed out before we can claim that. In particular, we need to pass the conformance tests, and submit a conformance report to Khronos. We’re not there yet.

I’m also happy to see that Zink is currently at the top of MesaMatrix (together with LLVMpipe, i965 and RadeonSI), reporting a total of 160 OpenGL extensions at the time of writing!

In theory, that means you can run any OpenGL application you can think of on top of Zink. Mike is hard at work testing the entire Steam game library, and things are working pretty well.

Is this the end of the line for Zink? Are we done now? Not at all! 😆

OpenGL compabibility profile

We’re still stuck at OpenGL 3.0 for compatibility contexts, mainly due to lack of testing. There’s a lot of features that need to work together in relatively complicated ways for this to work for us.

Note that this only matters for applications that rely on legacy OpenGL features. Modern OpenGL programs gets OpenGL 4.6 support, as mentioned previously.

I don’t think this is going to be a big deal to enable, but I haven’t spent time on it.

OpenGL ES 3.1 support

Similar to the OpenGL 4.6 support, we’re now able to expose the OpenGL ES 3.1 feature set. This is again thanks to a lot of hard work by Mike and the gang.

Why not OpenGL ES 3.2? This comes down to the GL_KHR_blend_equation_advanced feature. Mike blogged about the issue a while ago.

Lavapipe and continuous integration

To prevent regressions, we’ve started testing Zink on the Mesa CI system for every change. This is made possible thanks to Lavapipe, a Vulkan software implementation in Mesa that reuse the rasterizer from LLVMpipe.

This means we can run tests on virtual cloud machines without having to depend on unreliable hardware. 🤖

At the time of writing, we’re only exposing OpenGL 4.1 on top of Lavapipe, due to some lacking features. But we have patches in the works to bring this up to OpenGL 4.5, and OpenGL 4.6 probably won’t be far off when that lands.

Windows support

Basic support for Zink on Microsoft Windows has landed. This isn’t particularly useful at the moment, because we need better window-system integration to get anywhere near reasonable performance. But it’s there.

macOS support

Thanks to work by Duncan Hopkins of The Foundry, there’s also some support for macOS. This uses MoltenVK as the Vulkan implementation, meaning that we also support the Vulkan Portability Extension to some degree.

This support isn’t quite as drop-in as on other platforms, because it’s completely lacking window-system integration. But it seems to work for the use-cases they have at The Foundry, so it’s worth mentioning as well.

Driver support

Beyond this, Igalia has brought up Zink on the V3DV driver, and I’ve heard some whispers that there’s some people running Zink on top of Turnip, an open-source Vulkan driver for recent Qualcomm Adreno GPUs.

I’ve heard some people have some success getting things running on NVIDIA, but there’s a few obvious problems in the way there due to the lack of proper DRI support… Which brings us to:

Window System Integration

Another awesome new development is that Adam is working on Penny. So, what’s Penny?

Penny is another way of bringing up Zink, on systems without DRI support. It works as a dedicated GLX integration that uses the VK_KHR_swapchain extension to integrate properly with the native Vulkan driver’s window-system integration instead of Mesa baking its own.

This solves a lot of small, nasty issues in the DRI code-path. I’ll say the magic “implicit synchronization” word, and hope that scares away anyone wondering what it’s about.

Performance

A lot more has happened on the performance front as well, again all thanks to Mike. However, much of this is still out-of-tree, and waiting in Mike’s zink-wip branch.

So instead, I suggest you check out Mike’s blog for the latest performance information (and much more up-to-date info on Zink). There’s been a lot going on, and I’m sure there’s even more to come!

Closing words

I think this should cover the most interesting bits of development.

On a personal note, I recently became a dad for the first time, and as a result I’ll be away for a while on paternity leave, starting early this fall. Luckily, Zink is in good hands with Mike and the rest of the upstream community taking care of things.

I would like to again plug Mike’s blog as a great source of Zink-related news, if you’re not already following it. He posts a lot more frequent than I do, and he’s also an epic meme master, so it’s all great fun!

OpenGL on DirectX: Conformance & upstreaming of the D3D12 driver

kusmabite@gmail.com (Erik Faye-Lund) — Wed, 10 Mar 2021 20:30:28 +0100

Over the last year and then so, we at Collabora have been working with Microsoft on their D3D12 mapping layer, which I announced in my previous blog post. In July, Louis-Francis Ratté-Boulianne wrote an update on the status on the Collabora blog, but a lot has happened since then, so it’s time for another update.

There’s two major things that has happened since then; we have passed the OpenGL 3.3 conformance tests, and we have upstreamed the code in Mesa 3D.

Photoshop support

It might not be a big surprise, but one of the motivation for this work was to be able to run applications like Photoshop on Windows devices without full OpenGL support.

I’m happy to report that Microsoft has released their compatibility pack that uses our work to provide OpenGL (and OpenCL) support, Photoshop can now run on Windows on ARM CPUs! This is pretty exciting to see high-profile applications like that benefit from our work!

OpenGL 3.3 Conformance Test Suite

First of all, I would like to point out that having passed the OpenGL CTS isn’t necessarily the same as being formally conformant. There’s some details about how to formally becoming conformant on layered implementations that are complicated, and I’ll leave the question about formal conformance up to Microsoft and Khronos.

Instead, I want to talk a bit about passing the OpenGL CTS.

Challenges for layered implementations

A problem we face with layered implementations is that we are subject to a few more sources of issues, some of which are entirely out of our control. A normal OpenGL, non-layered, implementation have two primary sources of issues:

The OpenGL driver
The hardware

Issues with the OpenGL driver itself that leads to tests failing is always required to be fixed before results are submitted. Issues with the hardware generally requires software workarounds, but this is not always feasible, so Khronos have a system where a vendor can file a waiver for a hardware issue, and if approved they can mark test-failures as waived, and the appropriate failures will be ignored.

So far so good.

But for our layered implementations, our world looks a bit different. We don’t really see the hardware, but instead we see D3D12 and the D3D12 driver. This means our sources of issues are:

The OpenGL driver
The D3D12 run-time
The D3D12 vendor-driver
The hardware

For the OpenGL driver, the story is the same as for a non-layered implementation, but from there on things start changing.

Problems in the D3D12 run-time must also be fixed before submitting results. We work together with Microsoft to get these issues fixed as appropriate. Such fixes can take a while to trickle all the way into a Windows build and to end-users, but they will eventually show up.

But for the D3D12 vendor-driver and below, things gets complicated. First of all, it’s not always possible for us to tell vendor-driver issues and hardware issues apart. And worse, as these are developed by third party companies, we have little insight there. We can’t affect their priorities, so it’s hard to know when or even if an issue gets resolved.

It’s also not really a good idea to work around such issues, because if they turn out to be fixable software problems, we don’t know when they will be fixed, so we can’t really tell when to disable the work-around. We also don’t know exactly what combination of hardware and software these issues apply to.

But there’s one case where we have full insight, and that’s when the D3D12 vendor-driver is WARP, a high-performance software rasterizer. Because that component is developed by Microsoft, and we have channels to report issues and even make sure they get resolved!

Bugs, bugs, bugs

When developing something new, there’s always going to be bugs. But usually also when using something existing in a new way. We encountered a lot of bugs on our way, and here’s a quick overview over some of them. This is in no way exhaustive, and most of our own bugs are not that interesting. So this is mostly about problems unique to layered implementations.

64-bit shifts

It turned out early on that the DXIL validator in D3D12 had a requirement when parsing the LLVM bitcode that required that the amounts were always 32-bit values. While this seems fine by itself, LLVM itself requires that all operands to binops have the same bit-size. This obviously meant that only 32-bit shifts could pass the validator.

Microsoft quickly removed this requirement once we figured out what was going on.

Aligned block-compressed textures

In D3D12, one requirement for block-compressed textures is that the base-level is aligned to ther block-size. This requirement does not apply to mip-levels, and OpenGL has no such requirement. This isn’t technically speaking a bug, but a documented limitation in DirectX.

It turns out that this limitation was an artificial historical left-over, and after a bunch of testing (and fixing of WARP), we got this limitation lifted. Great :)

D3D12 vendor-driver bugs

Something that has been much more frustrating is bugs in the vendor-drivers. The problem here is that even though we have channels to file bugs, we don’t have any influence or even insight into their prioritization and release schedule.

I think it suffice to say that there’s been several reported bugs to all vendors we’ve actively been running the OpenGL CTS on top of. We believe fixes are underway for at least some of this, but we can’t make any promises here.

Current status

Right now, the only configurations we’re cleanly passing the OpenGL 3.3 CTS on are WARP (which became conformant on November 24th, 2020), and NVIDIA (which became conformant on February 26th, 2021).

Having these multiple independent implementations of DirectX drivers passing in conjunction with the Mesa/D3D12 layer shows that we are able to implement GLon12 in a vendor-neutral way, which allowed us to bring the layer to conformance. Many thanks to Khronos for their assistance through this process.

We’ve also submitted results on top of an Intel GPU, but that submission has been halted due to failures, and will as far as I know be updated as soon as Intel publish updated drivers.

The conformance tests have been run against our downstream fork, which is no longer actively maintained, because:

Upstreaming

The D3D12 driver was upstreamed in Mesa in Merge-Request 7477, and the OpenCL compiler followed in Merge-Request 7565. There’s been a lot more merge-requests since then, and even more is expected in the future.

The process of upstreaming the driver into Mesa3D went relatively smoothly, but there were quite a lot of regressions that happened quickly after we upstreamed the code, so to avoid this from becoming a big problem we’ve added the D3D12 driver to Mesa’s set of GitLab CI tests. We now build and test the D3D12 driver on top of WARP on CI, as well as running some basic sanity-tests for the OpenCL compiler.

All in all, this seems to work very well right now, and we’re looking forward to the future. Next step, WSL support!

I’m no longer working full-time on this project, instead I’m trying to take some of the lessons learned and apply them to Zink. I’m sure there’s even more room for code-reuse than what we currently have, but it will probably take some time to figure out.

Introducing OpenCL™ and OpenGL® on DirectX

kusmabite@gmail.com (Erik Faye-Lund) — Tue, 24 Mar 2020 13:46:55 +0100

For the last few months, we have been working on two exciting new projects at Collabora, and it’s finally time to share some information about them with the world:

We are partnering with Microsoft DirectX engineers to build OpenCL and OpenGL mapping layers, in order to bring OpenCL 1.2 and OpenGL 3.3 support to all Windows and DirectX 12 enabled devices out there!

This work builds on a lot of previous work. First and foremost, we are building this by using Mesa 3D, with the Gallium interface as the base for the OpenGL layer, and NIR as the base for the OpenCL compiler. We are also using LLVM and the SPIRV-LLVM-Translator from Khronos as the compiler front-end.

In addition, we are taking advantage of Microsoft’s experience in creating their D3D12 Translation Layer, as well as our own experience from developing Zink.

What is Mesa 3D?

Mesa 3D is an open source implementation of several graphics technologies, including OpenCL and OpenGL. The OpenGL implementation in Mesa is robust and is used as the base for several industry-strength OpenGL drivers from multiple GPU vendors.

Among other things, Mesa consists of several API implementations (called state-trackers) as well as the Gallium low-level driver interface. The Gallium interface hides a lot of the legacy OpenGL details and translates OpenGL calls into something that looks more like modern GPU primitives.

Why translate APIs?

Not all Windows-powered devices have consistent support for hardware-accelerated OpenCL and OpenGL. So in order to improve application compatibility, we are building a generic solution to the problem. This means that a GPU vendor only has to implement a D3D12 driver for their hardware in order to support all three APIs.

This mapping layer is also expected to serve as a starting point in porting older OpenCL and OpenGL applications over to D3D12.

In addition, we believe this is good for the wider open source community. A lot of the problems we are solving here are shared with other drivers and translation layers, and we hope that the code will be useful beyond the use cases listed above.

Implementation

The work is largely split into three parts: an OpenCL compiler, an OpenCL runtime, and a Gallium driver that builds and executes command-buffers on the GPU using the D3D12 API.

In addition, there is a shared NIR-to-DXIL shader compiler that both components use. For those not familiar with NIR, it is Mesa’s internal representation for GPU shaders. Similarly, DXIL is Microsoft’s internal representation, which D3D12 drivers will consume and translate into hardware-specific shaders.

OpenCL compiler

The OpenCL compiler uses LLVM and the SPIRV-LLVM-Translator to generate SPIR-V representations of OpenCL kernels. These, in turn, are passed to Mesa’s SPIR-V to NIR translator, where some optimizations and semantical translations are done. Then the NIR representation is finally passed to NIR-to-DXIL, which produces a DXIL compute shader and the needed metadata so it can be executed on the GPU by the runtime using D3D12.

Here’s a diagram of the complete process, including NIR-to-DXIL, which will be described below:

OpenCL Compiler Overview

OpenCL runtime

While Mesa provides an OpenCL implementation called Clover, we are not using it for this project. Instead, we have a new OpenCL runtime that does a more direct translation to the DirectX 12 API.

NIR-to-DXIL

DXIL is essentially LLVM 3.7 bitcode with some extra metadata and validation. This was a technical choice that made sense for Microsoft because all the major driver vendors already used LLVM in their compiler toolchain. Using an older version of the LLVM bitcode format gives good compatibility with drivers because the LLVM bitcode format is backwards compatible.

Because we depend on a much more recent version of LLVM for the compiler front-end, we sadly cannot easily use the DirectX Shader Compiler as a compiler back-end. The DirectX Shader Compiler is effectively a fork of LLVM 3.7, and we are currently using LLVM 10.0 for the compiler front-end. Using DirectX Shader Compiler as that would require us to link two different versions of LLVM into the same binary, which would have led to problems.

We also cannot easily use LLVM itself to generate the bitcode. While the LLVM bitcode format is backwards compatible, LLVM itself is not forward compatible. This means that newer versions of LLVM cannot produce a bitcode format that is understood by older versions. This makes sense from LLVM’s point of view because it was never meant as a general interchange format.

So instead, we have decided to implement our own DXIL emitter. This is quite a bit harder than it looks because LLVM bitcode goes to great lengths to try to make the format as dense as possible. For instance, LLVM does not store its bitcode as a sequence of bytes and words, but rather as variable-width bitfields in a long sequence of bits.

There are a lot of tricky details to get right, but in the end we have a compiler that works.

D3D12 Gallium driver

The D3D12 Gallium driver is the last piece of the puzzle. Essentially, it takes OpenGL commands and, with the help of the NIR to DXIL translator, turns them into D3D12 command-buffers, which it executes on the GPU using the D3D12 driver.

There are a lot of interesting details that makes this tricky as well, but I will save those details for later.

But to not leave you empty-handed, here’s a screenshot of the Windows version of the famous glxgears, wglgears:

wglgears on DirectX12

Source code

In the short term, the source code can be found here. We intend on upstreaming this work into the main Mesa repository shortly, so it is not a permanent home.

Next steps

This is just the announcement, and a whole lot of work is left to be done. We have something that works in some cases right now, but we are just starting to scratch the surface.

First of all, we need to get up to the feature-level that we target. Our goals at the moment is to pass conformance tests for OpenCL 1.2 and OpenGL 3.3. We have a long way to go, but with some hard work and sweat, I am sure we will get there.

Secondly, we need to work on application compatibility. For now we will be focusing on productivity applications.

We also want to upstream this in Mesa. This way we can keep up with fixes and new features in Mesa, and other drivers can benefit from what we are doing as well.

Acknowledgments

It is also important to point out that I am not the only one working on this. Our team consists of five additional Collabora engineers (Boris Brezillon, Daniel Stone, Elie Tournier, Gert Wollny, Louis-Francis Ratté-Boulianne) and two Microsoft DirectX engineers (Bill Kristiansen, Jesse Natalie).

Zink: Fall Update

kusmabite@gmail.com (Erik Faye-Lund) — Thu, 24 Oct 2019 15:22:00 +0100

I recently went to XDC 2019, where I gave yet another talk about Zink. I kinda forgot to write a blog-post about it, so here’s me trying to make up for it… or something like that. I’ll also go into some more recent developments as well.

My presentation was somewhat similar to the talk I did at SIGGRAPH this year, but with a bit more emphasis on the technical aspect, as the XDC audience is more familiar with Mesa internals.

If you’re interested, you can find the slides for the talk here. The talk goes through the motivation and basic approach. I don’t think I need to go through this again, as I’ve already covered that before.

As for the status, Zink currently supports OpenGL 2.1 and OpenGL ES 2.0. And there’s no immediate plans on working on OpenGL 3.0 and OpenGL ES 3.0 until Zink is upstream.

Which gets us to the more interesting bit; that I started working on upstreaming Zink. So let’s talk about that for a bit.

Upstreaming Zink

So, my current goal is to get Zink upstream in Mesa. The plan outline in my XDC talk is slightly outdated by now, so here I’ll instead say what’s actually happened so far, and what I hope will follow.

Before I could add the driver itself, I decided to send all changes outside of the driver as a set of merge-requests:

The last one is probably the most interesting one, as it moves a lot of fixed-function operations into the state-tracker, so individual drivers won’t have to deal with them. Unless they choose to, of course.

But all of these has already been merged, and there’s just one final merge-request left:

Introduce Zink: OpenGL on Vulkan

This merge-request adds the driver in its current state. It consists of 163 commits at the time of writing, so it’s not a thing of beauty. But new drivers usually aren’t, so I’m not too worried.

When this is merged, Zink will finally be a “normal” part of Mesa… Well, sort of anyway. I don’t think we’ll enable Zink to be built by default for a while. But that’ll just be a matter of adding zink to the -Dgallium-drivers meson-option.

Testing on CI

The current branch only adds building of Zink to the CI. There’s no testing being done yet. The reasons for this is two-fold:

We need to get a running environment on CI. Rather of bringing up some hardware-enabled test-runner, I intend to try to set up SwiftShader as a software rasterizer instead, as that supports Vulkan 1.1 these days.
We need some tests to run. Zink currently only supports OpenGL 2.1, and sadly the conformance test suite doesn’t have any tests for OpenGL versions older than 3.0. Piglit has some, but a full piglit-run takes significantly more time, which makes it tricky for CI. So right now, it seems the OpenGL ES 2.0 conformance tests are our best bet. We’ll of course add more test-suites as we add more features.

So, there’s some work to be done here, but it seems like we should be able to get something working without too much hassle.

Next steps

After Zink is upstream, it will be maintained similarly to other Mesa drivers. Practically speaking, this means that patches are sent to the upstream repo rather than my fork. It shouldn’t make a huge difference for most users.

The good thing is that if I go away on vacation, or are for some other reason unavailable, other people can still merge patches, so we’d slightly reduce the technical bus-factor.

I’m not stopping developing Zink at all, but I have other things going on in my life that means I might be busy with other things at times. As is the case for everyone! 😉

In fact, I’m very excited to start working on OpenGL 3.x and 4.x level features; we still have a few patches for some 3.x features in some older branches.

The future is bright! ☀️

Zink: Summer Update and SIGGRAPH 2019

kusmabite@gmail.com (Erik Faye-Lund) — Thu, 25 Jul 2019 21:38:36 +0200

Here’s an overview of the recent changes in Zink since the previous post, as well as an exciting announcement!

What’s new in the world of Zink?

OK, so I haven’t been as good at making frequent updates on as I was hoping, but let’s try to make up for it:

Since last time, there’s quite a lot of things that has been resolved:

We now do proper control-flow. This means things like if-statements, for-loops etc. There might be some control-flow primitives missing still, but that’s because I haven’t encountered any use yet.
Alpha testing has been implemented.
Client-defined clip-planes has been implemented.
Support for gl_FrontFacing has been implemented.
Lowering of glPointSize() to gl_PointSize has been implemented. This means you can use glPointSize() to specify sizes instead of having to write the gl_PointSize-output from the vertex shader.
Support for gl_FragDepth has been implemented.
Two-sided lighting has been implemented.
Shadow-samplers has been implemented.
Support for 8-bit primitive indices has been implemented.
Occlusion queries has been implemented correctly across command buffers. This includes the ability to pause / restore queries.
The compiler has been ported to C.
The compiler no longer lowers IO, but instead process derefs directly.
The compiler now handles booleans properly. It’s no longer lowering them to floats.
David Airlie has contributed lowering of glShadeModel(GL_FLAT) to flat interpolation qualifiers. This still doesn’t give us the right result, because Vulkan only supports the first vertex as the provoking vertex, and OpenGL defaults to the last one. To resolve this in a reasonable way, we need to inject a geometry shader that reorders the vertices, but this hasn’t been implemented yet. You can read more about this in this ticket
… and a boat-load of smaller fixes. There’s just too many to mention, really.

In addition to this, there’s been a pretty significant rewrite, changing the overall design of Zink. The reason for this, was that I made some early design-mistakes, and after having piled a bit too many features on top of this, I decided that it would be better to get the fundamentals right first.

Sadly, not all features have been brought forward since the rewrite, so we’re currently back to OpenGL 2.1 support. Fixing this is on my list of things I want to do, but I suspect that cleaning things up and upstreaming will take presedence over OpenGL 3.0 support.

And just to give you something neat to look at, here’s Blender running using Zink:

Blender on Zink

Khronos Vulkan BoF at SIGGRAPH 2019

Khronos has been nice enough to give me a slot in their Vulkan Sessions at the Khronos BoFs during SIGGRAPH 2019!

The talk will be a slightly less tech-heavy introduction to Zink, what it does and what the future holds. It will focus more on the motivation and use cases than the underlying technical difficulties compared to my previous talks.

So, if you’re in the area please drop by! A conference pass is not required to attend the BoF, as it’s not part of the official SIGGRAPH program.

Introducing Zink: OpenGL on Vulkan

kusmabite@gmail.com (Erik Faye-Lund) — Wed, 31 Oct 2018 17:52:21 +0100

For the last month or so, I’ve been playing with a new project during my work at Collabora, and as I’ve already briefly talked about at XDC 2018, it’s about time to talk about it to a wider audience.

What is Zink?

Zink is an OpenGL implementation on top of Vulkan. Or to be a bit more specific, Zink is a Mesa Gallium driver that leverage the existing OpenGL implementation in Mesa to provide hardware accelerated OpenGL when only a Vulkan driver is available.

glxgears on Zink

Here’s an overview of how this fits into the Mesa architecture, for those unfamiliar with it:

Architectural overview

Why implement OpenGL on top of Vulkan?

There’s several motivation behind this project, but let’s list a few:

Simplify the graphics stack
Lessen the work-load for future GPU drivers
Enable more integration
Support application porting to Vulkan

I’ll go through each of these points in more detail below.

But there’s another, less concrete reason; someone had to do this. I was waiting for someone else to do it before me, but nobody seemed to actually go ahead. At least as long as you don’t count solutions who only implement some variation of OpenGL ES (which in my opinion doesn’t solve the problem; we need full OpenGL for this to be really valuable).

1. Simplifying the graphics stack

One problem is that OpenGL is a big API with a lot of legacy stuff that has accumulated since its initial release in 1992. OpenGL is well-established as a requirement for applications and desktop compositors.

But since the very successful release of Vulkan, we now have two main-stream APIs for essentially the same hardware functionality.

It’s not looking like neither OpenGL nor Vulkan is going away, and the software-world is now hard at work implementing Vulkan support everywhere, which is great. But this leads to complexity. So my hope is that we can simplify things here, by only require things like desktop compositors to support one API down the road. We’re not there yet, though; not all hardware has a Vulkan-driver, and some older hardware can’t even support it. But at some point in the not too far future, we’ll probably get there.

This means there might be a future where OpenGL’s role could purely be one of legacy application compatibility. Perhaps Zink can help making that future a bit closer?

2. Lessen the work-load for future GPU drivers

The amount of drivers to maintain is only growing, and we want the amount of code to maintain for legacy hardware to be as little as possible. And since Vulkan is a requirement already, maybe we can get good enough performance through emulation?

Besides, in the Open Source world, there’s even new drivers being written for old hardware, and if the hardware is capable of supporting Vulkan, it could make sense to only support Vulkan “natively”, and do OpenGL through Zink.

It all comes down to the economics here. There aren’t infinite programmers out there that can maintain every GPU driver forever. But if we can make it easier and cheaper, maybe we can get better driver-support in the long run?

3. Enable more integration

Because Zink is implemented as a Gallium driver in Mesa, there’s some interesting side-benefits that comes “for free”. For instance, projects like Gallium Nine or Clover could in theory work on top of the i965 Vulkan driver through Zink. Please note that this hasn’t really been tested, though.

It should also be possible to run Zink on top of a closed-source Vulkan driver, and still get proper window system integration. Not that I promote the idea of using a closed-source Vulkan driver.

4. Support application porting to Vulkan

This might sound a bit strange, but it might be possible to extend Zink in ways where it can act as a cooperation-layer between OpenGL and Vulkan code in the same application.

The thing is, big CAD applications etc won’t realistically rewrite all of their rendering-code to Vulkan in a wave of a hand. So if they can for instance prototype some Vulkan-code inside an OpenGL application, it might be easier to figure out if Vulkan is worth it or not for them.

What does Zink require?

Zink currently requires a Vulkan 1.0 implementation, with the following extensions (there’s a few more, due to extensions requiring other extensions, but I’ve decided to omit those for simplicity):

VK_KHR_maintenance1: This is required for the viewport flipping. It’s also possible to do without this extension, and we have some experimental patches for that. I would certainly love to require as few extensions as possible.
VK_KHR_external_memory_fd: This is required as a way of getting the rendered result on screen. This isn’t technically a hard requirement, as we also have a copy-based approach, but that’s almost unusably slow. And I’m not sure if we’ll bother keeping it around.

Zink has to my knowledge only been tested on Linux. I don’t think there’s any major reasons why it wouldn’t run on any other operating system supporting Vulkan, apart from the fact that some window-system integration code might have to be written.

What does Zink support?

Right now, it’s not super-impressive: we implement OpenGL 2.1, and OpenGL ES 1.1 and 2.0 plus some extensions. Please note that the list of extensions might depend on the Vulkan implementation backing this, as we forward capabilities from that.

The list of extensions is too long to include here in a sane way, but here’s a link to the output of glxinfo as of today on top of i965.

Here’s some screenshots of applications and games we’ve tested that renders more or less correctly:

OpenArena on Zink

Weston on Zink

Quake 3 on Zink

Extreme Tux Racer on Zink

What doesn’t work?

Yeah, so when I say OpenGL 2.1, I’m ignoring some features that we simply do not support yet:

glPointSize() is currently not supported. Writing to gl_PointSize from the vertex shader does work. We need to write some code to plumb this through the vertex shader to make it work.
Texture borders are currently always black. This will also need some emulation code, due to Vulkan’s lack of arbitrary border-color support. Since a lot of hardware actually support this, perhaps we can introduce some extension to add it back to the API?
No control-flow is supported in the shaders at the moment. This is just because of lacking implementation for those opcodes. It’s coming.
No GL_ALPHA_TEST support yet. There’s some support code in NIR for this, we just need to start using it. This will depend on control-flow, though.
glShadeModel(GL_FLAT) isn’t supported yet. This isn’t particularly hard or anything, but we currently emit the SPIR-V before knowing the drawing-state. We should probably change this. Another alternative is to patch in a flat-decoration on the fly.
Different settings for glPolygonMode(GL_FRONT, ...) and glPolygonMode(GL_BACK, ...). This one is tricky to do correct, at least if we want to support newer shader-stages like geometry and tessellation at the same time. It’s also hard to do performant, even without these shader-stages, as we need to draw these primitives in the same order as they were specified but with different primitive types. Luckily, Vulkan can do pretty fast geometry submission, so there might be some hope for some compromise-solution, at least. It might also be possible to combine stream-out and a geometry-shader or something here if we really end up caring about this use-case.

And most importantly, we are not a conformant OpenGL implementation. I’m not saying we will never be, but as it currently stands, we do not do conformance testing, and as such we neither submit conformance results to Khronos.

It’s also worth noting that at this point, we tend to care more about applications than theoretical use-cases and synthetic tests. That of course doesn’t mean we do not care about correctness at all, it just means that we have plenty of work ahead of us, and the work that gets us most real-world benefit tends to take precedence. If you think otherwise, please send some patches! 😉

What’s the performance-hit compared to a “native” OpenGL driver?

One thing should be very clear; a “native” OpenGL driver will always have a better performance-potential, simply because anything clever we do, they can do as well. So I don’t expect to beat any serious OpenGL drivers on performance any time soon.

But the performance loss is already kinda less than I feared, especially since we haven’t done anything particularly fancy with performance yet.

I don’t yet have any systematic benchmark-numbers, and we currently have some kinda stupid bottlenecks that should be very possible to solve. So I’m reluctant to spend much time on benchmarking until those are fixed. Let’s just say that I can play Quake 3 at tolerable frame rates right now ;)

But OK, I will say this: I currently get around 475 FPS on glxgears on top of Zink on my system. The i965 driver gives me around 1750 FPS. Don’t read too much into those results, though; glxgears isn’t a good benchmark. But for that particular workload, we’re about a quarter of the performance. As I said, I don’t think glxgears is a very good benchmark, but it’s the only thing somewhat reproducible that I’ve run so far, so it’s the only numbers I have. I’ll certainly be doing some proper benchmarking in the future.

In the end, I suspect that the pipeline-caching is going to be the big hot-spot. There’s a lot of state to hash, and finally compare once a hit has been found. We have some decent ideas on how to speed it up, but there’s probably going to be some point where we simply can’t get it any better.

But even then, perhaps we could introduce some OpenGL extension that allows an application to “freeze” the render-state into some objects, similar to Vertex Array Objects, and that way completely bypass this problem for applications willing to do a bit of support-code? The future will tell…

All in all, I’m not too worried about this yet. We’re still early in the project, and I don’t see any major, impenetrable walls.

How to use Zink

Zink is only available as source code at the moment. No distro-packages exits yet.

Requirements

In order to build Zink, you need the following:

Git
Build dependencies to compile Mesa
Vulkan headers and libraries
Meson and Ninja

Building

The code currently lives in the zink-branch in my Mesa fork.

The first thing you have to do, is to clone the repository and build the zink-branch. Even though Mesa has an autotools build-system, Zink only supports the Meson build-system. Remember to enable the zink gallium-driver (-Dgallium-drivers=zink) when configuring the build.

Install the driver somewhere appropriate, and use the $MESA_LOADER_DRIVER_OVERRIDE environment variable to force the zink-driver. From here you should be able to run many OpenGL applications using Zink.

Here’s a rough recipe:

$ git clone https://gitlab.freedesktop.org/kusma/mesa.git mesa-zink
Cloning into 'mesa-zink'...
...
Checking out files: 100% (5982/5982), done.
$ cd mesa-zink
$ git checkout zink
Branch 'zink' set up to track remote branch 'zink' from 'origin'.
Switched to a new branch 'zink'
$ meson --prefix=/tmp/zink -Dgallium-drivers=zink build-zink
The Meson build system
...
Found ninja-X.Y.Z at /usr/bin/ninja
$ ninja -C build-zink install
ninja: Entering directory `build-zink'
...
installing /home/kusma/temp/mesa-zink/build-zink/src/gallium/targets/dri/libgallium_dri.so to /tmp/zink/lib64/dri/zink_dri.so
$ LIBGL_DRIVERS_PATH=/tmp/zink/lib64/dri/ MESA_LOADER_DRIVER_OVERRIDE=zink glxgears -info
GL_RENDERER   = zink (Intel(R) UHD Graphics 620 (Kabylake GT2))
GL_VERSION    = 2.1 Mesa 18.3.0-devel (git-395b12c2d7)
GL_VENDOR     = Collabora Ltd
GL_EXTENSIONS = GL_ARB_multisample GL_EXT_abgr ...

Submitting patches

Currently, the development happens on #dri-devel on Freenode. Ping me (my handle is kusma) with a link your branch, and I’ll take a look.

Where do we go from here?

Well, I think “forwards” is the only way to move 😉. I’m currently working 1-2 days per week on this at Collabora, so things will keep moving forward on my end. In addition, Dave Airlie seems to have a high momentum at the moment also. He has a work-in-progress branch that hints at GL 3.3 being around the corner!

I also don’t think there’s any fundamental reason why we shouldn’t be able to get to full OpenGL 4.6 eventually.

Besides the features, I also want to try to get this upstream in Mesa in some not-too-distant future. I think we’re already beyond the point where Zink is useful.

I also would like to point out that David Airlie of RedHat has contributed a lot of great patches, greatly advancing Zink from what it was before his help! At this point, he has implemented at least as many features as I have. So this is very much his accomplishment as well.

Working at Collabora

kusmabite@gmail.com (Erik Faye-Lund) — Tue, 31 Jul 2018 12:01:00 +0200

In the Fuse Open post, I mentioned that I would no longer be working at Fuse. I didn’t mention what I was going to do next, and now that it’s been a while I guess it’s time to let the cat out of the bag: I’ve started working at Collabora.

I’ve been working here for 1.5 months now, and I’m really enjoying it so far! I get to work on things I really enjoy, and I get a lot of freedom! 😄 🎉

What is Collabora

Collabora is an Open Source consultancy, specializing in a few industries. Most of what Collabora does is centered around things like Linux, automotive, embedded systems, and multimedia. You can read more about Collabora here.

The word “consultant” brings out quite a lot of stereotypes in my mind. Luckily, we’re not that kind of consultants. I haven’t worn a tie a single day at work yet!

When I got approached by Collabora, I was immediately attracted by the prospect of working more or less exclusively on Open Source Software. Collabora has “Open First” as their motto, and this fits my ideology very well! And trust me, Collabora really means it! 😀

What will I be doing?

I’m hired as a Principal Engineer on the Graphics team. This obviously means I’ll be working on graphics technology.

So far, I’ve been working a bit on some R&D tasks about Vulkan, but mostly on Virgil 3D (“VirGL”). If you don’t know what VirGL is, the very short explanation is that it’s GPU virtualization for virtual machines. I’ve been working on adding/fixing support for OpenGL 4.3 as well as OpenGL ES 3.1. The work isn’t complete but it’s getting close, and patches are being upstreamed as I write this.

I’m also working on Mesa. Currently mostly through Virgil, probably through other projects in the future as well. Apart from that, things depend heavily on customer needs.

Working Remotely

A big change from my previous jobs, is that I now work from home Instead of a shared office with my coworkers. This is because Collabora doesn’t have an Oslo office, as it’s largely a distributed team.

I’ve been doing this for around 1.5 months already, and it works a lot better than I feared. In fact, this was one of my biggest worries with taking this job, but so far it hasn’t been a problem at all! 🎉

But who knows, maybe all work and no play will make Jack a dull boy in the end? 🔪

Jokes aside, if this turns out to be a problem in the long term, I’ll look into getting a desk at some co-working space. There’s tons of them nearby.

Working as a Contractor

Another effect of Collabora not having an Oslo office means that I have to formally work as a contractor. This is mostly a formality (Collabora seems to treat people the same regardless if they are normal employees or contractors), but there’s quite a lot of legal challenges on my end due to this.

I would definitely have preferred normal employment, but I guess I don’t get to choose all the details ;)

Closing

So, this is what I’m doing now. I’m happy with my choice and I have a lot of really great colleagues! I also get to work with a huge community, and as part of that I’ll be going to more conferences going forward (next up: XDC)!

Fuse Open

kusmabite@gmail.com (Erik Faye-Lund) — Mon, 14 May 2018 22:52:00 +0200

OK, so the announcement that I hinted at in my previous post is now public, so I can finally talk about this.

What

I’m very happy to tell you that Fuse is now fully open source software! 🎉

For those who don’t know what I’m talking about: Fuse is a mobile app development environment, developed by Fusetools. You can read more about it here.

This work represents more than 3.5 years of my life (and of course lots of other people, but you know, this is my blog 😏), and I’ve been pushing hard inside the company my whole time there to open source it. We already open sourced fuselibs about a year ago, and as far as I’m concerned, that was a success.

Now, there’s some sad news that comes with this as well. Fusetools will no longer be working on Fuse. That’s now up to the community. The company is switching focus to an app-as-a-service business model.

This also marks the end of my time at Fuse. I’ll wrap up my end of Fuse 1.9 and then take a few weeks of vacation, before I start my next job. What I’ll be doing next is very exciting, but deserve its own post.

Why

There’s many reasons why we ended up where we are now. I’ll try to explain what I believe are the major reasons below.

Please note that the opinions stated here are my personal ones, and not those of my employer. I’m writing this article to shed some light on what I believe is the reasons we’ve come to where we are, not to blame anyone, but to try to learn from our mistakes.

First and foremost, we failed to build a viable business on selling development tools. Developing a full platform, comprising of compilers, standard libraries, editor-plugins, debugging tools, IDEs etc is a big undertaking, and cost a lot of money, while the market is full of free tools that does a pretty decent job. To convince someone to pay for something they can get for free somewhere else, you need to be a lot better.

Secondly, we’ve been taking a lot of time-consuming detours. We’ve started projects that, in my opinion, should never have been started. We should have kept our main focus enabling our users to do more, not to do more for them.

Third, I sadly think we failed to be as good as we needed to. This one is a bit touchy, but I’ll try to explain. I’m not saying I think we did bad work, far from it. But to make up for the first point, we would have needed to be vastly better than the competition. But due to the second point, the competition largely caught up to us.

What could we have done differently?

I think the problem we set out to solve was a hard one, so the short answer is “beats me”… 😕

However, I believe that our chances would at least have been better if we open sourced the platform from the beginning.

The reason is that we might not have had to build the platform all by ourselves. If we would for instance made our package-manager usable for shipping 3rd-party packages, we could have leveraged the community more at more of the high-level work. Our company could have focused on functionality that enabled the community. Instead we ended up doing a lot of very high-level work that didn’t end up benefiting a lot of users.

I also believe that if we made the Uno compiler a stand-alone C# to C++ transpiler, and did all the graphics functionality and app-bootstrapping as library functionality instead of compiler-internals, the compiler might have been useful for other C#-oriented projects, and we could have aligned closer with the C# community than we ended up. Instead, we treated Uno as an internal tool that app-developers weren’t really supposed to care about.

I’m far from sure that those changes together would have been enough, though. This was a hard problem to solve, and that was exactly why I was interested in this gig to begin with.

The future

Well, I hope that the future of Fuse is bright, but different than before. Thanks to Fusetools releasing all of the code, I do believe that the project will live on, with a better future that isn’t meddled with short-term wins that hurt in the long run.

I plan on continuing at least some of my work on Uno / Fuselibs, and I’m not the only one. In addition, there’s also still some commercial interest for Fuse, just not from Fusetools. Anything in this area is still not announced, but expect to hear more.

I still believe the technology is good, and hopefully now that it’s finally all out there for anyone to play with, we can get some great work done! Fuse has a unique tech-stack, with a declarative UI engine that is built around preview / live-refresh, and a comprehensive toolkit. The turn-around time for experimenting is still miles beyond the competition. ✨

Hello World!

kusmabite@gmail.com (Erik Faye-Lund) — Sun, 13 May 2018 11:36:58 +0200

Welcome to my new blog, where I’ll be writing about my open source development.

This is just a short announcement post, to introduce this blog.

If you don’t already know who I am, I am Erik “kusma” Faye-Lund, a Norwegian graphics programmer.

I’m currently working on the following projects, which I expect to be posting about:

Fuselibs: This is the core UI engine in Fuse, where I currently work.
Rocket: This is a tool for tweaking synchronization of audio and visuals in demoscene productions.
Grate: This is a reverse-engineered GPU driver for the GeForce ULP in the NVIDIA Tegra 2/3/4 SoCs (code-name: AR20 / Aurora).
Mesa 3D: This is what Grate is being built on top, and the goal of Grate is ultimately to upstream a driver into Mesa.

In addition to this, there’s some cool new announcements coming up very soon! So stay tuned!