Fedora

First a definition: a trackstick is also called trackpoint, pointing stick, or "that red knob between G, H, and B". I'll be using trackstick here, because why not.

This post is the continuation of libinput and the Lenovo T450 and T460 series touchpads where we focused on a stalling pointer when moving the finger really slowly. Turns out the T460s at least, possibly others in the *60 series have another bug that caused a behaviour that is much worse but we didn't notice for ages as we were focusing on the high-precision cursor movement. Specifically, the pointer would just randomly stop moving for a short while (spoiler alert: 300ms), regardless of the movement speed.

libinput has built-in palm detection and one of the things it does is to disable the touchpad when the trackstick is in use. It's not uncommon to rest the hand near or on the touchpad while using the trackstick and any detected touch would cause interference with the pointer motion. So events from the touchpad are ignored whenever the trackpoint sends events. [1]

On (some of) the T460s the trackpoint sends spurious events. In the recording I have we have random events at 9s, then again 3.5s later, then 14s later, then 2s later, etc. Each time, our palm detection could would assume the trackpoint was in use and disable the touchpad for 300ms. If you were using the touchpad while this was happening, the touchpad would suddenly stop moving for 300ms and then continue as normal. Depending on how often these spurious events come in and the user's current caffeination state, this was somewhere between odd, annoying and infuriating.

The good news is: this is fixed in libinput now. libinput 1.5 and the upcoming 1.4.3 releases will have a fix that ignores these spurious events and makes the touchpad stalls a footnote of history. Hooray.

[1] we still allow touchpad physical button presses, and trackpoint button clicks won't disable the touchpad

This post explains how the evdev protocol works. After reading this post you should understand what evdev is and how to interpret evdev event dumps to understand what your device is doing. The post is aimed mainly at users having to debug a device, I will thus leave out or simplify some of the technical details. I'll be using the output from evemu-record as example because that is the primary debugging tool for evdev.

What is evdev?

evdev is a Linux-only generic protocol that the kernel uses to forward information and events about input devices to userspace. It's not just for mice and keyboards but any device that has any sort of axis, key or button, including things like webcams and remote controls. Each device is represented as a device node in the form of /dev/input/event0, with the trailing number increasing as you add more devices. The node numbers are re-used after you unplug a device, so don't hardcode the device node into a script. The device nodes are also only readable by root, thus you need to run any debugging tools as root too.

evdev is the primary way to talk to input devices on Linux. All X.Org drivers on Linux use evdev as protocol and libinput as well. Note that "evdev" is also the shortcut used for xf86-input-evdev, the X.Org driver to handle generic evdev devices, so watch out for context when you read "evdev" on a mailing list.

Communicating with evdev devices

Communicating with a device is simple: open the device node and read from it. Any data coming out is a struct input_event, defined in /usr/include/linux/input.h:

struct input_event {
 struct timeval time;
 __u16 type;
 __u16 code;
 __s32 value;
};

I'll describe the contents later, but you can see that it's a very simple struct.

Static information about the device such as its name and capabilities can be queried with a set of ioctls. Note that you should always use libevdevto interact with a device, it blunts the few sharp edges evdev has. See the libevdev documentation for usage examples.

evemu-record, our primary debugging tool for anything evdev is very simple. It reads the static information about the device, prints it and then simply reads and prints all events as they come in. The output is in machine-readable format but it's annotated with human-readable comments (starting with #). You can always ignore the non-comment bits. There's a second command, evemu-describe, that only prints the description and exits without waiting for events.

Relative devices and keyboards

The top part of an evemu-record output is the device description. This is a list of static properties that tells us what the device is capable of. For example, the USB mouse I have plugged in here prints:

# Input device name: "PIXART USB OPTICAL MOUSE"
# Input device ID: bus 0x03 vendor 0x93a product 0x2510 version 0x110
# Supported events:
#   Event type 0 (EV_SYN)
#     Event code 0 (SYN_REPORT)
#     Event code 1 (SYN_CONFIG)
#     Event code 2 (SYN_MT_REPORT)
#     Event code 3 (SYN_DROPPED)
#     Event code 4 ((null))
#     Event code 5 ((null))
#     Event code 6 ((null))
#     Event code 7 ((null))
#     Event code 8 ((null))
#     Event code 9 ((null))
#     Event code 10 ((null))
#     Event code 11 ((null))
#     Event code 12 ((null))
#     Event code 13 ((null))
#     Event code 14 ((null))
#   Event type 1 (EV_KEY)
#     Event code 272 (BTN_LEFT)
#     Event code 273 (BTN_RIGHT)
#     Event code 274 (BTN_MIDDLE)
#   Event type 2 (EV_REL)
#     Event code 0 (REL_X)
#     Event code 1 (REL_Y)
#     Event code 8 (REL_WHEEL)
#   Event type 4 (EV_MSC)
#     Event code 4 (MSC_SCAN)
# Properties:

The device name is the one (usually) set by the manufacturer and so are the vendor and product IDs. The bus is one of the "BUS_USB" and similar constants defined in /usr/include/linux/input.h. The version is often quite arbitrary, only a few devices have something meaningful here.

We also have a set of supported events, categorised by "event type" and "event code" (note how type and code are also part of the struct input_event). The type is a general category, and /usr/include/linux/input-event-codes.h defines quite a few of those. The most important types are EV_KEY (keys and buttons), EV_REL (relative axes) and EV_ABS (absolute axes). In the output above we can see that we have EV_KEY and EV_REL set.

As a subitem of each type we have the event code. The event codes for this device are self-explanatory: BTN_LEFT, BTN_RIGHT and BTN_MIDDLE are the left, right and middle button. The axes are a relative x axis, a relative y axis and a wheel axis (i.e. a mouse wheel). EV_MSC/MSC_SCAN is used for raw scancodes and you can usually ignore it. And finally we have the EV_SYN bits but let's ignore those, they are always set for all devices.

Note that an event code cannot be on its own, it must be a tuple of (type, code). For example, REL_X and ABS_X have the same numerical value and without the type you won't know which one is which.

That's pretty much it. A keyboard will have a lot of EV_KEY bits set and the EV_REL axes are obviously missing (but not always...). Instead of BTN_LEFT, a keyboard would have e.g. KEY_ESC, KEY_A, KEY_B, etc. 90% of device debugging is looking at the event codes and figuring out which ones are missing or shouldn't be there.

Exercise: You should now be able to read a evemu-record description from any mouse or keyboard device connected to your computer and understand what it means. This also applies to most special devices such as remotes - the only thing that changes are the names for the keys/buttons. Just run sudo evemu-describe and pick any device in the list.

The events from relative devices and keyboards

evdev is a serialised protocol. It sends a series of events and then a synchronisation event to notify us that the preceeding events all belong together. This synchronisation event is EV_SYN SYN_REPORT, is generated by the kernel, not the device and hence all EV_SYN codes are always available on all devices.

Let's have a look at a mouse movement. As explained above, half the line is machine-readable but we can ignore that bit and look at the human-readable output on the right.

E: 0.335996 0002 0000 0001      # EV_REL / REL_X                1
E: 0.335996 0002 0001 -002      # EV_REL / REL_Y                -2
E: 0.335996 0000 0000 0000      # ------------ SYN_REPORT (0) ----------

This means that within one hardware event, we've moved 1 device unit to the right (x axis) and two device units up (y axis). Note how all events have the same timestamp (0.335996).

Let's have a look at a button press:

E: 0.656004 0004 0004 589825    # EV_MSC / MSC_SCAN             589825
E: 0.656004 0001 0110 0001      # EV_KEY / BTN_LEFT             1
E: 0.656004 0000 0000 0000      # ------------ SYN_REPORT (0) ----------
E: 0.727002 0004 0004 589825    # EV_MSC / MSC_SCAN             589825
E: 0.727002 0001 0110 0000      # EV_KEY / BTN_LEFT             0
E: 0.727002 0000 0000 0000      # ------------ SYN_REPORT (0) ----------

For button events, the value 1 signals button pressed, button 0 signals button released.

And key events look like this:

E: 0.000000 0004 0004 458792    # EV_MSC / MSC_SCAN             458792
E: 0.000000 0001 001c 0000      # EV_KEY / KEY_ENTER            0
E: 0.000000 0000 0000 0000      # ------------ SYN_REPORT (0) ----------
E: 0.560004 0004 0004 458976    # EV_MSC / MSC_SCAN             458976
E: 0.560004 0001 001d 0001      # EV_KEY / KEY_LEFTCTRL         1
E: 0.560004 0000 0000 0000      # ------------ SYN_REPORT (0) ----------
[....]
E: 1.172732 0001 001d 0002      # EV_KEY / KEY_LEFTCTRL         2
E: 1.172732 0000 0000 0001      # ------------ SYN_REPORT (1) ----------
E: 1.200004 0004 0004 458758    # EV_MSC / MSC_SCAN             458758
E: 1.200004 0001 002e 0001      # EV_KEY / KEY_C                1
E: 1.200004 0000 0000 0000      # ------------ SYN_REPORT (0) ----------

Mostly the same as button events. But wait, there is one difference: we have a value of 2 as well. For key events, a value 2 means "key repeat". If you're on the tty, then this is what generates repeat keys for you. In X and Wayland we ignore these repeat events and instead use XKB-based key repeat.

Now look at the keyboard events again and see if you can make sense of the sequence. We have an Enter release (but no press), then ctrl down (and repeat), followed by a 'c' press - but no release. The explanation is simple - as soon as I hit enter in the terminal, evemu-record started recording so it captured the enter release too. And it stopped recording as soon as ctrl+c was down because that's when it was cancelled by the terminal. One important takeaway here: the evdev protocol is not guaranteed to be balanced. You may see a release for a key you've never seen the press for, and you may be missing a release for a key/button you've seen the press for (this happens when you stop recording). Oh, and there's one danger: if you record your keyboard and you type your password, the keys will show up in the output. Security experts generally reocmmend not publishing event logs with your password in it.

Exercise: You should now be able to read a evemu-record events list from any mouse or keyboard device connected to your computer and understand the event sequence.This also applies to most special devices such as remotes - the only thing that changes are the names for the keys/buttons. Just run sudo evemu-record and pick any device listed.

Absolute devices

Things get a bit more complicated when we look at absolute input devices like a touchscreen or a touchpad. Yes, touchpads are absolute devices in hardware and the conversion to relative events is done in userspace by e.g. libinput. The output of my touchpad is below. Note that I've manually removed a few bits to make it easier to grasp, they will appear later in the multitouch discussion.

# Input device name: "SynPS/2 Synaptics TouchPad"
# Input device ID: bus 0x11 vendor 0x02 product 0x07 version 0x1b1
# Supported events:
#   Event type 0 (EV_SYN)
#     Event code 0 (SYN_REPORT)
#     Event code 1 (SYN_CONFIG)
#     Event code 2 (SYN_MT_REPORT)
#     Event code 3 (SYN_DROPPED)
#     Event code 4 ((null))
#     Event code 5 ((null))
#     Event code 6 ((null))
#     Event code 7 ((null))
#     Event code 8 ((null))
#     Event code 9 ((null))
#     Event code 10 ((null))
#     Event code 11 ((null))
#     Event code 12 ((null))
#     Event code 13 ((null))
#     Event code 14 ((null))
#   Event type 1 (EV_KEY)
#     Event code 272 (BTN_LEFT)
#     Event code 325 (BTN_TOOL_FINGER)
#     Event code 328 (BTN_TOOL_QUINTTAP)
#     Event code 330 (BTN_TOUCH)
#     Event code 333 (BTN_TOOL_DOUBLETAP)
#     Event code 334 (BTN_TOOL_TRIPLETAP)
#     Event code 335 (BTN_TOOL_QUADTAP)
#   Event type 3 (EV_ABS)
#     Event code 0 (ABS_X)
#       Value   2919
#       Min     1024
#       Max     5112
#       Fuzz       0
#       Flat       0
#       Resolution 42
#     Event code 1 (ABS_Y)
#       Value   3711
#       Min     2024
#       Max     4832
#       Fuzz       0
#       Flat       0
#       Resolution 42
#     Event code 24 (ABS_PRESSURE)
#       Value      0
#       Min        0
#       Max      255
#       Fuzz       0
#       Flat       0
#       Resolution 0
#     Event code 28 (ABS_TOOL_WIDTH)
#       Value      0
#       Min        0
#       Max       15
#       Fuzz       0
#       Flat       0
#       Resolution 0
# Properties:
#   Property  type 0 (INPUT_PROP_POINTER)
#   Property  type 2 (INPUT_PROP_BUTTONPAD)
#   Property  type 4 (INPUT_PROP_TOPBUTTONPAD)

We have a BTN_LEFT again and a set of other buttons that I'll explain in a second. But first we look at the EV_ABS output. We have the same naming system as above. ABS_X and ABS_Y are the x and y axis on the device, ABS_PRESSURE is an (arbitrary) ranged pressure value.

Absolute axes have a bit more state than just a simple bit. Specifically, they have a minimum and maximum (not all hardware has the top-left sensor position on 0/0, it can be an arbitrary position, specified by the minimum). Notable here is that the axis ranges are simply the ones announced by the device - there is no guarantee that the values fall within this range and indeed a lot of touchpad devices tend to send values slightly outside that range. Fuzz and flat can be safely ignored, but resolution is interesting. It is given in units per millimeter and thus tells us the size of the device. in the above case: (5112 - 1024)/42 means the device is 97mm wide. The resolution is quite commonly wrong, a lot of axis overrides need the resolution changed to the correct value.

The axis description also has a current value listed. The kernel only sends events when the value changes, so even if the actual hardware keeps sending events, you may never see them in the output if the value remains the same. In other words, holding a finger perfectly still on a touchpad creates plenty of hardware events, but you won't see anything coming out of the event node.

Finally, we have properties on this device. These are used to indicate general information about the device that's not otherwise obvious. In this case INPUT_PROP_POINTER tells us that we need a pointer for this device (it is a touchpad after all, a touchscreen would instead have INPUT_PROP_DIRECT set). INPUT_PROP_BUTTONPAD means that this is a so-called clickpad, it does not have separate physical buttons but instead the whole touchpad clicks. Ignore INPUT_PROP_TOPBUTTONPAD because it only applies to the Lenovo *40 series of devices.

Ok, back to the buttons: aside from BTN_LEFT, we have BTN_TOUCH. This one signals that the user is touching the surface of the touchpad (with some in-kernel defined minimum pressure value). It's not just for finger-touches, it's also used for graphics tablet stylus touchpes (so really, it's more "contact" than "touch" but meh).

The BTN_TOOL_FINGER event tells us that a finger is in detectable range. This gives us two bits of information: first, we have a finger (a tablet would have e.g. BTN_TOOL_PEN) and second, we may have a finger in proximity without touching. On many touchpads, BTN_TOOL_FINGER and BTN_TOUCH come in the same event, but others can detect a finger hovering over the touchpad too (in which case you'd also hope for ABS_DISTANCE being available on the touchpad).

Finally, the BTN_TOOL_DOUBLETAP up to BTN_TOOL_QUINTTAP tell us whether the device can detect 2 through to 5 fingers on the touchpad. This doesn't actually track the fingers, it merely tells you "3 fingers down" in the case of BTN_TOOL_TRIPLETAP.

Exercise: Look at your touchpad's description and figure out if the size of the touchpad is correct based on the axis information [1]. Check how many fingers your touchpad can detect and whether it can do pressure or distance detection.

The events from absolute devices

Events from absolute axes are not really any different than events from relative devices which we already covered. The same type/code combination with a value and a timestamp, all framed by EV_SYN SYN_REPORT events. Here's an example of me touching the touchpad:

E: 0.000001 0001 014a 0001      # EV_KEY / BTN_TOUCH            1
E: 0.000001 0003 0000 3335      # EV_ABS / ABS_X                3335
E: 0.000001 0003 0001 3308      # EV_ABS / ABS_Y                3308
E: 0.000001 0003 0018 0069      # EV_ABS / ABS_PRESSURE         69
E: 0.000001 0001 0145 0001      # EV_KEY / BTN_TOOL_FINGER      1
E: 0.000001 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +0ms
E: 0.021751 0003 0018 0070      # EV_ABS / ABS_PRESSURE         70
E: 0.021751 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +21ms
E: 0.043908 0003 0000 3334      # EV_ABS / ABS_X                3334
E: 0.043908 0003 0001 3309      # EV_ABS / ABS_Y                3309
E: 0.043908 0003 0018 0065      # EV_ABS / ABS_PRESSURE         65
E: 0.043908 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +22ms
E: 0.052469 0001 014a 0000      # EV_KEY / BTN_TOUCH            0
E: 0.052469 0003 0018 0000      # EV_ABS / ABS_PRESSURE         0
E: 0.052469 0001 0145 0000      # EV_KEY / BTN_TOOL_FINGER      0
E: 0.052469 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +9ms

In the first event you see BTN_TOOL_FINGER and BTN_TOUCH set (this touchpad doesn't detect hovering fingers). An x/y coordinate pair and a pressure value. The pressure changes in the second event, the third event changes pressure and location. Finally, we have BTN_TOOL_FINGER and BTN_TOUCH released on finger up, and the pressure value goes back to 0. Notice how the second event didn't contain any x/y coordinates? As I said above, the kernel only sends updates on absolute axes when the value changed.

Ok, let's look at a three-finger tap (again, minus the ABS_MT_ bits):

E: 0.000001 0001 014a 0001      # EV_KEY / BTN_TOUCH            1
E: 0.000001 0003 0000 2149      # EV_ABS / ABS_X                2149
E: 0.000001 0003 0001 3747      # EV_ABS / ABS_Y                3747
E: 0.000001 0003 0018 0066      # EV_ABS / ABS_PRESSURE         66
E: 0.000001 0001 014e 0001      # EV_KEY / BTN_TOOL_TRIPLETAP   1
E: 0.000001 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +0ms
E: 0.034209 0003 0000 2148      # EV_ABS / ABS_X                2148
E: 0.034209 0003 0018 0064      # EV_ABS / ABS_PRESSURE         64
E: 0.034209 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +34ms
[...]
E: 0.138510 0003 0000 4286      # EV_ABS / ABS_X                4286
E: 0.138510 0003 0001 3350      # EV_ABS / ABS_Y                3350
E: 0.138510 0003 0018 0055      # EV_ABS / ABS_PRESSURE         55
E: 0.138510 0001 0145 0001      # EV_KEY / BTN_TOOL_FINGER      1
E: 0.138510 0001 014e 0000      # EV_KEY / BTN_TOOL_TRIPLETAP   0
E: 0.138510 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +23ms
E: 0.147834 0003 0000 4287      # EV_ABS / ABS_X                4287
E: 0.147834 0003 0001 3351      # EV_ABS / ABS_Y                3351
E: 0.147834 0003 0018 0037      # EV_ABS / ABS_PRESSURE         37
E: 0.147834 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +9ms
E: 0.157151 0001 014a 0000      # EV_KEY / BTN_TOUCH            0
E: 0.157151 0003 0018 0000      # EV_ABS / ABS_PRESSURE         0
E: 0.157151 0001 0145 0000      # EV_KEY / BTN_TOOL_FINGER      0
E: 0.157151 0000 0000 0000      # ------------ SYN_REPORT (0) ---------- +10ms

In the first event, the touchpad detected all three fingers at the same time. So get BTN_TOUCH, x/y/pressure and BTN_TOOL_TRIPLETAP set. Note that the various BTN_TOOL_* bits are mutually exclusive. BTN_TOOL_FINGER means "exactly 1 finger down" and you can't have exactly 1 finger down when you have three fingers down. In the second event x and pressure update (y has no event, it stayed the same).

In the event after the break, we switch from three fingers to one finger. BTN_TOOL_TRIPLETAP is released, BTN_TOOL_FINGER is set. That's very common. Humans aren't robots, you can't release all fingers at exactly the same time, so depending on the hardware scanout rate you have intermediate states where one finger has left already, others are still down. In this case I released two fingers between scanouts, one was still down. It's not uncommon to see a full cycle from BTN_TOOL_FINGER to BTN_TOOL_DOUBLETAP to BTN_TOOL_TRIPLETAP on finger down or the reverse on finger up.

Exercise: test out the pressure values on your touchpad and see how close you can get to the actual announced range. Check how accurate the multifinger detection is by tapping with two, three, four and five fingers. (In both cases, you'll likely find that it's very much hit and miss).

Multitouch and slots

Now we're at the most complicated topic regarding evdev devices. In the case of multitouch devices, we need to send multiple touches on the same axes. So we need an additional dimension and that is called multitouch slots (there is another, older multitouch protocol that doesn't use slots but it is so rare now that you don't need to bother).

First: all axes that are multitouch-capable are repeated as ABS_MT_foo axis. So if you have ABS_X, you also get ABS_MT_POSITION_X and both axes have the same axis ranges and resolutions. The reason here is backwards-compatibility: if a device only sends multitouch events, older programs only listening to the ABS_X etc. events won't work. Some axes may only be available for single-touch (ABS_MT_TOOL_WIDTH in this case).

Let's have a look at my touchpad, this time without the axes removed:

# Input device name: "SynPS/2 Synaptics TouchPad"
# Input device ID: bus 0x11 vendor 0x02 product 0x07 version 0x1b1
# Supported events:
#   Event type 0 (EV_SYN)
#     Event code 0 (SYN_REPORT)
#     Event code 1 (SYN_CONFIG)
#     Event code 2 (SYN_MT_REPORT)
#     Event code 3 (SYN_DROPPED)
#     Event code 4 ((null))
#     Event code 5 ((null))
#     Event code 6 ((null))
#     Event code 7 ((null))
#     Event code 8 ((null))
#     Event code 9 ((null))
#     Event code 10 ((null))
#     Event code 11 ((null))
#     Event code 12 ((null))
#     Event code 13 ((null))
#     Event code 14 ((null))
#   Event type 1 (EV_KEY)
#     Event code 272 (BTN_LEFT)
#     Event code 325 (BTN_TOOL_FINGER)
#     Event code 328 (BTN_TOOL_QUINTTAP)
#     Event code 330 (BTN_TOUCH)
#     Event code 333 (BTN_TOOL_DOUBLETAP)
#     Event code 334 (BTN_TOOL_TRIPLETAP)
#     Event code 335 (BTN_TOOL_QUADTAP)
#   Event type 3 (EV_ABS)
#     Event code 0 (ABS_X)
#       Value   5112
#       Min     1024
#       Max     5112
#       Fuzz       0
#       Flat       0
#       Resolution 41
#     Event code 1 (ABS_Y)
#       Value   2930
#       Min     2024
#       Max     4832
#       Fuzz       0
#       Flat       0
#       Resolution 37
#     Event code 24 (ABS_PRESSURE)
#       Value      0
#       Min        0
#       Max      255
#       Fuzz       0
#       Flat       0
#       Resolution 0
#     Event code 28 (ABS_TOOL_WIDTH)
#       Value      0
#       Min        0
#       Max       15
#       Fuzz       0
#       Flat       0
#       Resolution 0
#     Event code 47 (ABS_MT_SLOT)
#       Value      0
#       Min        0
#       Max        1
#       Fuzz       0
#       Flat       0
#       Resolution 0
#     Event code 53 (ABS_MT_POSITION_X)
#       Value      0
#       Min     1024
#       Max     5112
#       Fuzz       8
#       Flat       0
#       Resolution 41
#     Event code 54 (ABS_MT_POSITION_Y)
#       Value      0
#       Min     2024
#       Max     4832
#       Fuzz       8
#       Flat       0
#       Resolution 37
#     Event code 57 (ABS_MT_TRACKING_ID)
#       Value      0
#       Min        0
#       Max    65535
#       Fuzz       0
#       Flat       0
#       Resolution 0
#     Event code 58 (ABS_MT_PRESSURE)
#       Value      0
#       Min        0
#       Max      255
#       Fuzz       0
#       Flat       0
#       Resolution 0
# Properties:
#   Property  type 0 (INPUT_PROP_POINTER)
#   Property  type 2 (INPUT_PROP_BUTTONPAD)
#   Property  type 4 (INPUT_PROP_TOPBUTTONPAD)

We have an x and y position for multitouch as well as a pressure axis. There are also two special multitouch axes that aren't really axes. ABS_MT_SLOT and ABS_MT_TRACKING_ID. The former specifies which slot is currently active, the latter is used to track touch points.

Slots are a static property of a device. My touchpad, as you can see above ony supports 2 slots (min 0, max 1) and thus can track 2 fingers at a time. Whenever the first finger is set down it's coordinates will be tracked in slot 0, the second finger will be tracked in slot 1. When the finger in slot 0 is lifted, the second finger continues to be tracked in slot 1, and if a new finger is set down, it will be tracked in slot 0. Sounds more complicated than it is, think of it as an array of possible touchpoints.

The tracking ID is an incrementing number that lets us tell touch points apart and also tells us when a touch starts and when it ends. The two values are either -1 or a positive number. Any positive number means "new touch" and -1 means "touch ended". So when you put two fingers down and lift them again, you'll get a tracking ID of 1 in slot 0, a tracking ID of 2 in slot 1, then a tracking ID of -1 in both slots to signal they ended. The tracking ID value itself is meaningless, it simply increases as touches are created.

Let's look at a single tap:

E: 0.000001 0003 0039 0387 # EV_ABS / ABS_MT_TRACKING_ID   387
E: 0.000001 0003 0035 2560 # EV_ABS / ABS_MT_POSITION_X    2560
E: 0.000001 0003 0036 2905 # EV_ABS / ABS_MT_POSITION_Y    2905
E: 0.000001 0003 003a 0059 # EV_ABS / ABS_MT_PRESSURE      59
E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH            1
E: 0.000001 0003 0000 2560 # EV_ABS / ABS_X                2560
E: 0.000001 0003 0001 2905 # EV_ABS / ABS_Y                2905
E: 0.000001 0003 0018 0059 # EV_ABS / ABS_PRESSURE         59
E: 0.000001 0001 0145 0001 # EV_KEY / BTN_TOOL_FINGER      1
E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
E: 0.021690 0003 003a 0067 # EV_ABS / ABS_MT_PRESSURE      67
E: 0.021690 0003 0018 0067 # EV_ABS / ABS_PRESSURE         67
E: 0.021690 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +21ms
E: 0.033482 0003 003a 0068 # EV_ABS / ABS_MT_PRESSURE      68
E: 0.033482 0003 0018 0068 # EV_ABS / ABS_PRESSURE         68
E: 0.033482 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +12ms
E: 0.044268 0003 0035 2561 # EV_ABS / ABS_MT_POSITION_X    2561
E: 0.044268 0003 0000 2561 # EV_ABS / ABS_X                2561
E: 0.044268 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +11ms
E: 0.054093 0003 0035 2562 # EV_ABS / ABS_MT_POSITION_X    2562
E: 0.054093 0003 003a 0067 # EV_ABS / ABS_MT_PRESSURE      67
E: 0.054093 0003 0000 2562 # EV_ABS / ABS_X                2562
E: 0.054093 0003 0018 0067 # EV_ABS / ABS_PRESSURE         67
E: 0.054093 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +10ms
E: 0.064891 0003 0035 2569 # EV_ABS / ABS_MT_POSITION_X    2569
E: 0.064891 0003 0036 2903 # EV_ABS / ABS_MT_POSITION_Y    2903
E: 0.064891 0003 003a 0059 # EV_ABS / ABS_MT_PRESSURE      59
E: 0.064891 0003 0000 2569 # EV_ABS / ABS_X                2569
E: 0.064891 0003 0001 2903 # EV_ABS / ABS_Y                2903
E: 0.064891 0003 0018 0059 # EV_ABS / ABS_PRESSURE         59
E: 0.064891 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +10ms
E: 0.073634 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID   -1
E: 0.073634 0001 014a 0000 # EV_KEY / BTN_TOUCH            0
E: 0.073634 0003 0018 0000 # EV_ABS / ABS_PRESSURE         0
E: 0.073634 0001 0145 0000 # EV_KEY / BTN_TOOL_FINGER      0
E: 0.073634 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +9ms

We have a tracking ID (387) signalling finger down, as well as a position plus pressure. then some updates and eventually a tracking ID of -1 (signalling finger up). Notice how there is no ABS_MT_SLOT here - the kernel buffers those too so while you stay in the same slot (0 in this case) you don't see any events for it. Also notice how you get both single-finger as well as multitouch in the same event stream. This is for backwards compatibility [2]

Ok, time for a two-finger tap:

E: 0.000001 0003 0039 0496 # EV_ABS / ABS_MT_TRACKING_ID   496
E: 0.000001 0003 0035 2609 # EV_ABS / ABS_MT_POSITION_X    2609
E: 0.000001 0003 0036 3791 # EV_ABS / ABS_MT_POSITION_Y    3791
E: 0.000001 0003 003a 0054 # EV_ABS / ABS_MT_PRESSURE      54
E: 0.000001 0003 002f 0001 # EV_ABS / ABS_MT_SLOT          1
E: 0.000001 0003 0039 0497 # EV_ABS / ABS_MT_TRACKING_ID   497
E: 0.000001 0003 0035 3012 # EV_ABS / ABS_MT_POSITION_X    3012
E: 0.000001 0003 0036 3088 # EV_ABS / ABS_MT_POSITION_Y    3088
E: 0.000001 0003 003a 0056 # EV_ABS / ABS_MT_PRESSURE      56
E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH            1
E: 0.000001 0003 0000 2609 # EV_ABS / ABS_X                2609
E: 0.000001 0003 0001 3791 # EV_ABS / ABS_Y                3791
E: 0.000001 0003 0018 0054 # EV_ABS / ABS_PRESSURE         54
E: 0.000001 0001 014d 0001 # EV_KEY / BTN_TOOL_DOUBLETAP   1
E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
E: 0.012909 0003 002f 0000 # EV_ABS / ABS_MT_SLOT          0
E: 0.012909 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID   -1
E: 0.012909 0003 002f 0001 # EV_ABS / ABS_MT_SLOT          1
E: 0.012909 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID   -1
E: 0.012909 0001 014a 0000 # EV_KEY / BTN_TOUCH            0
E: 0.012909 0003 0018 0000 # EV_ABS / ABS_PRESSURE         0
E: 0.012909 0001 014d 0000 # EV_KEY / BTN_TOOL_DOUBLETAP   0
E: 0.012909 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +12ms

This was a really quick two-finger tap that illustrates the tracking IDs nicely. In the first event we get a touch down, then an ABS_MT_SLOT event. This tells us that subsequent events belong to the other slot, so it's the other finger. There too we get a tracking ID + position. In the next event we get an ABS_MT_SLOT to switch back to slot 0. Tracking ID of -1 means that touch ended, and then we see the touch in slot 1 ended too.

Time for a two-finger scroll:

E: 0.000001 0003 0039 0557 # EV_ABS / ABS_MT_TRACKING_ID   557
E: 0.000001 0003 0035 2589 # EV_ABS / ABS_MT_POSITION_X    2589
E: 0.000001 0003 0036 3363 # EV_ABS / ABS_MT_POSITION_Y    3363
E: 0.000001 0003 003a 0048 # EV_ABS / ABS_MT_PRESSURE      48
E: 0.000001 0003 002f 0001 # EV_ABS / ABS_MT_SLOT          1
E: 0.000001 0003 0039 0558 # EV_ABS / ABS_MT_TRACKING_ID   558
E: 0.000001 0003 0035 3512 # EV_ABS / ABS_MT_POSITION_X    3512
E: 0.000001 0003 0036 3028 # EV_ABS / ABS_MT_POSITION_Y    3028
E: 0.000001 0003 003a 0044 # EV_ABS / ABS_MT_PRESSURE      44
E: 0.000001 0001 014a 0001 # EV_KEY / BTN_TOUCH            1
E: 0.000001 0003 0000 2589 # EV_ABS / ABS_X                2589
E: 0.000001 0003 0001 3363 # EV_ABS / ABS_Y                3363
E: 0.000001 0003 0018 0048 # EV_ABS / ABS_PRESSURE         48
E: 0.000001 0001 014d 0001 # EV_KEY / BTN_TOOL_DOUBLETAP   1
E: 0.000001 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +0ms
E: 0.027960 0003 002f 0000 # EV_ABS / ABS_MT_SLOT          0
E: 0.027960 0003 0035 2590 # EV_ABS / ABS_MT_POSITION_X    2590
E: 0.027960 0003 0036 3395 # EV_ABS / ABS_MT_POSITION_Y    3395
E: 0.027960 0003 003a 0046 # EV_ABS / ABS_MT_PRESSURE      46
E: 0.027960 0003 002f 0001 # EV_ABS / ABS_MT_SLOT          1
E: 0.027960 0003 0035 3511 # EV_ABS / ABS_MT_POSITION_X    3511
E: 0.027960 0003 0036 3052 # EV_ABS / ABS_MT_POSITION_Y    3052
E: 0.027960 0003 0000 2590 # EV_ABS / ABS_X                2590
E: 0.027960 0003 0001 3395 # EV_ABS / ABS_Y                3395
E: 0.027960 0003 0018 0046 # EV_ABS / ABS_PRESSURE         46
E: 0.027960 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +27ms
E: 0.051720 0003 002f 0000 # EV_ABS / ABS_MT_SLOT          0
E: 0.051720 0003 0035 2609 # EV_ABS / ABS_MT_POSITION_X    2609
E: 0.051720 0003 0036 3447 # EV_ABS / ABS_MT_POSITION_Y    3447
E: 0.051720 0003 002f 0001 # EV_ABS / ABS_MT_SLOT          1
E: 0.051720 0003 0036 3080 # EV_ABS / ABS_MT_POSITION_Y    3080
E: 0.051720 0003 0000 2609 # EV_ABS / ABS_X                2609
E: 0.051720 0003 0001 3447 # EV_ABS / ABS_Y                3447
E: 0.051720 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +24ms
[...]
E: 0.272034 0003 002f 0000 # EV_ABS / ABS_MT_SLOT          0
E: 0.272034 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID   -1
E: 0.272034 0003 002f 0001 # EV_ABS / ABS_MT_SLOT          1
E: 0.272034 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID   -1
E: 0.272034 0001 014a 0000 # EV_KEY / BTN_TOUCH            0
E: 0.272034 0003 0018 0000 # EV_ABS / ABS_PRESSURE         0
E: 0.272034 0001 014d 0000 # EV_KEY / BTN_TOOL_DOUBLETAP   0
E: 0.272034 0000 0000 0000 # ------------ SYN_REPORT (0) ---------- +30ms

Note that "scroll" is something handled in userspace, so what you see here is just a two-finger move. Everything in there i something we've already seen, but pay attention to the two middle events: as updates come in for each finger, the ABS_MT_SLOT changes before the upates are sent. The kernel filter for identical events is still in effect, so in the third event we don't get an update for the X position on slot 1. The filtering is per-touchpoint, so in this case this means that slot 1 position x is still on 3511, just as it was in the previous event.

That's all you have to remember, really. If you think of evdev as a serialised way of sending an array of touchpoints, with the slots as the indices then it should be fairly clear. The rest is then just about actually looking at the touch positions and making sense of them.

Exercise: do a pinch gesture on your touchpad. See if you can track the two fingers moving closer together. Then do the same but only move one finger. See how the non-moving finger gets less updates.

That's it. There are a few more details to evdev but much of that is just more event types and codes. The few details you really have to worry about when processing events are either documented in libevdev or abstracted away completely. The above should be enough to understand what your device does, and what goes wrong when your device isn't working. Good luck.

[1] If not, file a bug against systemd's hwdb and CC me so we can put corrections in
[2] We treat some MT-capable touchpads as single-touch devices in libinput because the MT data is garbage