This Week in V3D

2019-08-02

Fri, 02 Aug 2019 00:30:00 +0000

Recently the Raspberry Pi Foundation released the Raspberry Pi 4, which shipped with the V3D driver I wrote as its GLES driver.

I’m pretty proud of the work I did on the project. I was a solo developer building a GLES3 graphics driver based on Mesa, splitting my time between the new V3D and maintaining VC4, while also fixing issues in the X server and building a kernel driver. I didn’t finish everything (the hardware should be able to do GLES 3.2, and I almost made it to CTS-complete on 3.1 before shipping), but I feel like this is clear proof of how productive graphics driver developers can be working on the Mesa stack.

Now, I’m working at Google on the freedreno graphics driver and Mesa in general, as part of the Chrome OS graphics team. The folks at Igalia are taking over from me on V3D (I’ve already done a bunch of review of perf improvement and bugfix patches they’ve made), and Bootlin is going to be continuing to work on getting open source display to parity with the closed source stack. The future of open source Raspberry Pi graphics is still looking good, even if I’m not driving it any more.

2018-12-10

Mon, 10 Dec 2018 00:30:00 +0000

For V3D last week, I resurrected my old GLES 3.1 series with SSBO and shader imgae support, rebuilt it for V3D 4.1 (shader images no longer need manual tiling), and wrote indirect draw support and started on compute shaders. As of this weekend, dEQP-GLES31 is passing 1387/1567 of tests with “compute” in the name on the simulator. I have a fix needed for barrier(), then it’s time to build the kernel interface. In the process, I ended up fixing several job flushing bugs, plugging memory leaks, improving our shader disassembly debug dumps, and reducing memory consumption and CPU overhead.

The TFU series is now completely merged, and the kernel cache management series from last week is now also merged. I also fixed a bug in the core GPU scheduler that would cause use-after-frees when tracing.

For vc4, Boris completed and merged the H/V flipping work after fixing display of flipped/offset SAND images. More progress has been made on chamelium testing, so we’re nearing having automated regression tests for parts of our display stack. My PM driver has the acks we needed, but my co-maintainer has tacked a new request on in v3 so it’ll be delayed.

2018-12-03

Mon, 03 Dec 2018 00:30:00 +0000

In V3D land, I built kernel support for the Texture Formatting Unit (TFU) and started using it for glGenerateMipmaps() support. This unit will be also be important for reformatting linear dmabufs (such as from X11 with a linear-only scanout engine) or media decode output in SAND format into the UIF format that V3D can render from. The kernel side is in drm-misc-next, and I’ll land the Mesa side once it hits drm-next.

I also rewrote the V3D cache invalidation in the kernel thanks to feedback from Dave Emett on the hardware team. In the process, this fixed a 3ms(!) CPU-side wait on every job submission, which improved throughput by 4-10x. Now I know why my fancy new hardware felt so slow! I also landed new tracepoints for anyone else debugging execution (aka I could write good sysprof visualization now).

Now that cache management is less broken, I’ve started on resurrecting my old SSBO and shader image branches so that I can write CSD (compute shader) support and get the driver to GLES 3.1.

In the process of writing new kernel code, I found multiple regressions in DRM core during CTS runs. I rebased my old V3D IGT support and got it merged, and built new tests for replicating one of the failures (GPU hang recovery was broken).

The tinydrm work from the previous month also continued. I’ve polished kmsro some more (fewer driver symlinks, more loader knowledge of kmsro), and will be resubmitting soon I hope.

Since I had kmsro nearly working, I gave it a shot for a V3D testing idea from last month. Right now to get X11 running to do conformance runs on it, I’ve got a stub KMS implementation in a branch. If I could do buffer sharing between VKMS and V3D, then kmsro could bind the two together so that we didn’t need out-of-tree patches for V3D testing. It turned out that VKMS didn’t have PRIME support, so I used Noralf’s proposed shmem helpers to add it. I got X11 up and running and looking plausible, but tests were failing left and right. This may be due to the shmem helpers using cached CPU mappings, when we want WC for interaction with most GPUs. Because of issues like this, the shmem helpers for VKMS are stalled at the moment but I may land V3D support in kmsro anyway.

In vc4 land, the Bootlin folks have been knocking off more of the HVS TODO entries. Boris respun underscan support, and hopefully after the next revision we’ll be able to land it. He’s also implemented H/V flipping of planes, which will be useful in some cases for 180-degree rotated displays. (For 90 degree rotations, we would need panel support for it as the HVS can’t go that way). Paul is taking over Boris’s load tracker work and is building tests for it to determine how accurate it is in predicting underflows.

Finally, for vc4 I got permission to write a driver for the PM block that controls power and reset. So far the only thing Linux uses the block for directly is the watchdog timer (which also performs reboots by letting the watchdog timer fire quickly). For power domains, we’ve been using my raspberrypi-power driver to talk to the Raspberry Pi firmware. However, we should be able to do things more reliably, and implement GPU reset properly, if we expose power domains and a reset controller from the PM block. I wrote a series that moves the PM block to a new MFD driver, binds the watchdog driver to it, and adds a new soc driver for power domains and reset. So far, I’ve only tested this driver for V3D power domains and reset, but it looks good and gets more of the Raspberry Pi platform supported in fully open source code.

2018-10-30

Tue, 30 Oct 2018 00:30:00 +0000

This last week, I worked on the oldest issue I had in my rpi kernel repo: Adding support for tinydrm displays (it actually predates tinydrm). I wrote a tinydrm driver for the panel I had on hand and got it upstreamed. I wrote a Mesa series enabling vc4 to talk to this new hx8357d tinydrm driver. Once the Mesa side is agreed on, then we can extend it to the other tinydrm drivers and have a solution for X with 3D with limited updates (as opposed to full screen refreshing like fbdev did) on these panels.

I also fixed a longstanding vc4 bug that slightly rotated SDL2 output. They were running all of their coordinates through a rotation matrix they built on each VS invocation, and in the common case of no rotation the cos(0)/sin(0) was inaccurate enough to rotate things by a pixel near the window edges. The sin()/cos() in Mesa is now more accurate so this problem shouldn’t happen with old SDL2, we have a piglit test to make sure other drivers don’t break with SDL2, and upstream SDL2 has stopped building a rotation matrix in the shader. (yay!)

Small stuff from the V3D driver since my last post:

Fixed depth writes (or depth/color non-writes) on V3D 4.2.
Reduced shader recompiles on V3D 4.2.
Reduced VPM consumption by vertex shaders using shared NIR optimizations.
Took advantage of hardware half-float pack/unpack operations.
Switched from FLUSH_ALL_STATE to FLUSH (less overhead, tested by HW team).
Merged a fix for use-after-free of sched fences.
Merged a fix for debugfs oops on V3D 4.1
Merged the kernel V3D clock measurement debugfs entry.
Merged more documentation of the SUBMIT_CL.

Also in V3D, I’ve been experimenting with other ways of running the driver using the software simulator. Right now I have a frankenstein mode in vc4 and v3d where the driver sits on top of the i915 GEM driver, and does BO mapping and interactions with the window system on i915 and then copies those BOs into the simulator to run vc4/v3d ioctls on it. This is gross, and it’s only really usable on my desktop with the i915 driver.

Talking with one of the HW engineers who’s interested in running Mesa against the simulator, there are a few ways we could do simulation. One would be to have the actual kernel driver call back up into a userspace daemon linking the simulator, and provide mappings of BOs and proxied register writes to that userspace dameon. Another would be to implement an actual V3D device in QEMU by linking in the simulator, but we can’t do that due to the GPL. Another would be to use my current simulation code, but let it work on top of vkms or vgem (so that a CI system wouldn’t need i915). This one seems easy. However, I started on a project mimicing what Intel did for their software-only CI setup: intercept libc calls to simulate having a V3D driver that’s not actually present in the kernel. I’ve got the code to the point of trying to submit CLs, at which point the simulator hangs for reasons I haven’t debugged yet. I don’t know if I’ll want to do this long term, but it seems like potentially useful code to other driver developers who have a software simulator.

Finally, in the X world, this month I’ve reviewed and merged more patches than I normally would (thanks, gitlab!), and wrote an extensible testcase for the damage extension due to a bug report/fix that a user had been submitted.

2018-09-07

Fri, 07 Sep 2018 00:30:00 +0000

TWIV3D has been inactive for a bit, because I’ve been inactive for a bit. On August 15th, we added Moss Anholt to our family:

So most of my time has been spent laying around with them and keeping us fed instead of writing software.

Here and there I’ve managed to write some patches. I’ve been working on meeting the current GLES3 CTS’s requirement for a 565 window system buffer being available. We don’t expose those on X11 today, because why would you? Your screen is 8888, so all you could do with 565 is render to a pixmap or buffer. Rendering to a pixmap is useless, becasue EGL doesn’t tell you whether R is in the top or bottom bits, so the X11 client doesn’t know how to interpret the rendered contents other than glReadPixels() (at which point, just use an FBO). My solution for this bad requirement from the CTS has been to add support for 565 pbuffers, which is pointless but is the minimum. This led to a patch to the xserver and a fix to V3D for blending against RTs without alpha channels.

I’m particularly proud of a small series to create a default case handler function for gallium pipe caps. Previously, every addition of a PIPE_CAP (which happens probably at least 10 times a year) has to touch every single driver, or that driver will start failing. It’s easy to forget a driver, particularly when rebasing a series across the addition of a new driver. And for driver authors, you’re faced with this list of 100 values which you don’t know the correct state of when starting to write a driver. The new helper lets CAP creators set a default value (usually whatever the hardcoded value was before their new CAP), and lets driver authors specify a much smaller set of required caps to get started.

Just before Moss arrived, I had landed some patches to VC4 to improve texture upload/download performance. Non-GL SDL apps on glamor were really slow, because the window wouldn’t be aligned to tiles and we’d download the tile-aligned area from the framebuffer to a raster-order temporary, store our new contents into it, then reupload that to the screen. The download from write-combined memory is brutally slow. My new code (mostly a rebase of patches I wrote in January 2017!) got the following results:

Improves x11perf -putimage10 performance by 139.777% +/- 2.83464% (n=5)
Improves x11perf -putimage100 performance by 383.908% +/- 22.6297% (n=11)
Improves x11perf -getimage10 performance by 2.75731% +/- 0.585054% (n=145)

I also wrote a followup patch to avoid the copy from the user’s buffer into the temporary in the driver for texture uploads, for another 12.1586% +/- 1.38155% (n=145)

2018-07-30

Mon, 30 Jul 2018 00:30:00 +0000

I made some progress on GMP this week, getting jobs to run successfully if I enable regions in the GMP but never disable them. I need to do some more work on BO lifetime management if I want to clear disallowed regions back out of the GMP.

I did some more conformance debug, fixing one of the intermittent issues in GLES3 (The same rendering-versus-VS-texturing synchronization bug that broke glmark2 terrain on vc4). GLES3 conformance is now at GLES2 status plus one new set of intermittent issues. Another fix I made (not misusing HW semaphores) unfortunately doesn’t seem to have improved anything.

I had a few performance ideas left over from last week, and built them on Monday (fixed extra flushing due to glClear() after drawing, or the GFHX-1461 workaround, avoid storing buffers that had all their drawing masked out). Performance of simple apps was still astoundingly slow, so I did some more poking around and determined that my V3D HW is actually running at 37 Mhz. My other board runs at 27Mhz. Now I know why my conformance runs have been taking so long! Unfortunately, Broadcom STB doesn’t support clock control from Linux, so I won’t really be able to do performance work on this driver until I can get a SW stack under me that can set the right clock speeds.

Finally, I cleaned up and merged my CLIF dumping code, so V3D can now generate simple traces that can be replayed by the software simulator or on FPGAs to debug lockups. It’s going to take a bit more work to make it so that BO addresses in uniforms work, which is needed for texturing, UBOs, and spilling.

2018-07-23

Mon, 23 Jul 2018 00:30:00 +0000

Two more weeks of conformance debug, and I’m now down to 1 group of failures in GLES2 and 1 group in EGL, both of which I believe are bugs in the tests (regressions in recent CTS releases, which is why Intel hadn’t caught them already).

Fixes have included:

Various bugs in fence handling (one of which caused throttling of frames to fail, leading to unlimited userspace BO cache usage and 2 days of debug flailing to track down).
Allow EGLImages to be created of textures/renderbuffers that we can’t do EGL_MESA_image_dma_buf_export on.
Fixed EGLImage import for cubemap images.
Landed the UIF format modifier support.
Fixed uniform lowering to not cause absurd register spilling.
Implemented more of the GFXH-1461 workaround to fix missing Z/S clears
Implemented a hack in gallium’s blitter to make the awful dEQP-GLES3.functional.fbo.blit.rect.nearest_consistency_* tests pass.
Fixed incorrect reallocation of PERSISTENT buffers (also a vc4 bug!)
Fixed 1D_ARRAY texture upload/download.
Fixed drawing to MRT without independent blending
Fixed MSAA mapping bugs in gallium’s transfer helper.

I also spent a day knocking out some quick performance improvements:

Rotate through the registers to allow the backend scheduler to pair up QPU instructions more frequently.
Allow reading from the phys regfile in the following instruction.
Use the SFU instructions instead of magic writes on V3D 4.x.
Skip texture parameter P2 if it’s just the default value.
Start using small immediates (especially useful for VS input/outputs on 4.x).
Fix extra scene flush when doing the GFXH-1461 workaround.

After this I should probably work on the GMP support again, then get back to fixing GLES3.

2018-07-09

Mon, 09 Jul 2018 00:30:00 +0000

Now that my V3D GPU hangs are cleared up, I made a lot of progress on conformance:

Fixes from last week:

Don’t reset the GPU until it stops making progress on the job (instead of just a fixed timeout).
Fixed noperspective interpolation
Fixed leaks of default attributes and spill BOs.
Fixed ARB_color_buffer_float clamping of gl_FragData[]
Fixed support for GL_EXT_draw_buffers2’s independent blend enables.
Fixed GL_SAMPLE_ALPHA_TO_ONE.

Fail/(Pass+Fail) ratios after my weekend run:

EGL: 96/2143
GLES2: 7/16985
GLES3: 25/42549
GLES3 multisample: crash!

I’ve debugged the first MSAA crash, but there are some MSAA resolve bugs after that which I need to sort out (some of which are probably the cause of those EGL failures).

On the VC4 front, Boris landed the writeback changes, and I submitted the DT for it in my ARM pull request.

2018-07-02

Mon, 02 Jul 2018 00:30:00 +0000

For V3D, last week I spent mostly building up .CLIF dumping support so that I could talk to the HW team about the GPU hangs I’ve been experiencing. This involved reformatting my dump output, naming some missing (0-filled) fields of the GPU packets I emit, and restructuring the dumper so that I could dump the contents of each BO all at once (whereas before my dumps didn’t really care about BOs and would just dump each memory area as it was found in the worklist).

The final result was a CLIF file I sent to the HW team of a simple transform feedback test that was locking up the GPU. They reported that I was missing a wait packet that was required, beyond what the HW specs said. (I had actually earlier tried emitting the wait, but typoed the condition for enabling it!) With that, I think I’ve cleared up all of the GPU hangs in piglit runs of khr_gles3.

On the VC4 front, the DSI panel enable/disable sequencing patch is in, which should enable other people to successfully build DSI panel drivers for Raspberry Pi. Boris has resubmitted the VC4 writeback (TXP) series now that the core writeback support is in, and there were just a couple of cleanups and it should now be ready to land.

2018-06-25

Mon, 25 Jun 2018 00:30:00 +0000

For V3D, last week included:

Created the DRM fourcc for talking about the new UIF tiling between processes.
Fixed offsets of buffers shared between processes.
Reduced CPU overhead and binary size of V3D and VC4 release builds.
Fixed min/mag determination in non-mipmapped texture filtering modes.
Implemented ALPHA_TO_COVERAGE.
Fixed flushing of jobs writing to separate stencil buffers.
Fixed the return status of ClientWaitSync.
Fixed blits from linear winsys BOs.
Wrote a workaround for broken transform feedback setup with gallium NIR.

For VC4, I fixed a regression in display initialization in drm-misc-next. I’ve also been doing some study of the HVS to help Boris with the T-format scanout offset patch. Hopefully with what I figured out today, he can get it all working. I also respun my DSI enable/disable sequencing patch to not need any changes in the core.

Finally, I put together the branches for bcm2835 maintainership for kernel 4.19. This week I should PR them.