Accessibility profiles

The 🌳 umi profiles (Unified Multimodal Input, unified multimodal input) generate a workstation usable without keyboard or mouse. It is controlled by several input methods coordinated with each other: touch, voice, gaze… The goal is accessibility, especially for people with motor disabilities.

They consist of two parts: a host profile umi which declares the hardware and session, and a user profile umi which sets the user’s environment.

The principle

No single input method replaces keyboard and mouse on its own: each has its limits (gaze precision, voice fatigue, gesture range). The UMI workstation combines them so they complement each other, and adjusts to the user’s abilities.

Input method	Tool	Usage
Touch	Touchscreen, Onboard on-screen keyboard	Point, click, enter text
Voice	Talon	Voice commands
Gaze	Tobii Eye Tracker 5, driven by Talon	Move the pointer

Coordination example: gaze places the pointer, voice or dwell-click confirms, on-screen keyboard enters text.

Hardware

Each device serves one input method : only equip the workstation with those suited to the user.

Item	Requirement
Eye tracker	Tobii Eye Tracker 5 (USB, manufacturer ID 2104)
Architecture	x86_64-linux only (Talon provides no other binary)
Microphone	Needed for voice commands
Speakers	Audio feedback for visual keyboard keystrokes
Screen	A good touchscreen for controlling the desktop and applications

First boot under Windows

The Tobii 5 firmware must be initialized once on a Windows machine, using the Tobii Experience app. Without this step, the tracker stays silent under Linux.

Setting up a workstation

1. Declare the host and its user in etc/config.yaml, both with the umi profile :

   etc/config.yaml

   users:
     alice:
       uid: 1010
       name: "Alice"
       profile: "umi"
       groups: ["global"]
   
   hosts:
     - hostname: "umi-desktop"
       name: "Poste accessible"
       zone: "maison:2.3"
       profile: "umi"
       users: ["alice"]

   On the umi host, the user profile adds the account to the input and uinput groups, without which Talon cannot inject events.

2. Enable automatic login in the machine configuration, if the user cannot use a keyboard to log in :

   usr/machines/umi-desktop/configuration.nix

   darkone.host.umi.autoLoginUser = "alice";

3. Generate and deploy :

   just generate
   just apply umi-desktop

4. Test and configure

   Eye tracker: calibrate on first launch

   Unplug then replug the tracker (udev rules only apply on connection), restart Talon, then calibrate gaze.

Multiple users on the same machine

Only the account listed in autoLoginUser logs in on its own. Any account on the host with the umi profile opens the Cinnamon session, imposed at every boot. Ordinary accounts keep their usual GNOME Wayland session.

Configure the host

Options of darkone.host.umi, to set in usr/machines/<host>/configuration.nix.

Option	Default	Role
enable	enabled by the profile	Activates the host profile
autoLoginUser	null	umi account started at boot
package	pkgs.talon	Talon package, or null to install it manually

Talon absent or unwanted

With package = null, everything else is configured (udev, uinput, accessibility) and Talon can be launched manually from the upstream archive, which preserves its automatic update.

Configuring the user

Options of darkone.home.umi, set in usr/users/<login>/default.nix.

Option	Default	Role
enable	enabled by the profile	Enables the user profile
enableTalonAutostart	true	Starts Talon at login
communityScripts	null	talonhub/community tree deployed in ~/.talon/user/community
enableDwell	false	Permanent dwell clicking
enableTrackerAuto	true	Gaze and click follow the tracker
gazeFilterSpeed	1.0	Gaze responsiveness (Talon’s)
cursorFilterSpeed	4.0	Pointer responsiveness (Talon : 2.0)
enableGazeControl	true	The pointer follows the gaze
enableHeadControl	true	The head refines the pointer placed by the gaze
headSpeed	5.0	Head sensitivity (Talon : 1)
enableHeadJump	false	A head movement brings the pointer back to the gaze
enableMouseJump	false	Moving a mouse toward the gaze brings the pointer there
enableGazeFocus	false	The gazed window takes focus (experimental)
dwellTime	1.6	Dwell click delay, in seconds
panelHeight	64	Cinnamon panel height, in pixels
textScaling	1.5	Text magnification factor (Cinnamon)
enablePanel	true	Imposed panel, rewritten at each deployment
enableMaximize	true	Application windows open maximized (Cinnamon)
panelLaunchers	Files, Firefox	Panel launchers (.desktop files)
menuFavorites	Calibration, Calculator, Calendar, Settings	Menu favorites, rewritten at deployment
enableRofiMenu	true	Full-screen application grid
rofiGridResolution	{ columns = 6; rows = 4; }	Tiles per grid page
hiddenApps	Terminals, duplicates, NixOS manual	Hidden from the grid and menu
enableStkWithMouse	false	SuperTuxKart with the mouse

Dwell clicking follows the tracker

Under Cinnamon, enableTrackerAuto enables pointer control by gaze and dwell clicking when the tracker is plugged in, and turns off that click when it is unplugged : with a mouse, it clicks on everything the pointer hovers over. enableDwell is then only useful under GNOME.

Eye Tracking menu modes

enableGazeControl, enableHeadControl, enableHeadJump, enableMouseJump and enableGazeFocus are re-imposed at each Talon start. A change made from the menu lasts until the next start.

Community contributions

The community scripts talonhub/community bring mouse zoom and tongue-click clicking. They are declared reproducibly :

usr/users/alice/default.nix

{ pkgs, ... }:
{
  darkone.home.umi.communityScripts = pkgs.fetchFromGitHub {
    owner = "talonhub";
    repo = "community";
    rev = "…";
    hash = "…";
  };
}

SuperTuxKart with the mouse

With enableStkWithMouse, the user drives SuperTuxKart 🡕 with their mouse alone, for example an adapted joystick recognized as a mouse (n-ABLER…). umi host only. The game is installed, its launcher also starts the dnf-stk-mouse converter.

Gesture	Action in the game
Joystick forward	Accelerate
Joystick backward	Brake, reverse
Joystick left, right	Turn
Left click held	Drift
Double left click	Use item (weapon)
Right click held	Nitro

The pointer stays active : the game menus are driven with the mouse, the race ignores it. The gestures only act when the game window is active, elsewhere the mouse stays a mouse.

To validate with an n-ABLER

Thresholds and release delay tuned without the real device. The game keys must stay the default ones (arrows, V, Space, N).

What the profile changes

The workstation is deliberately stripped of everything that assumes a keyboard or mouse.

Change	Reason
Cinnamon session on X11	Talon does not (unfortunately) run under Wayland
Automatic login	No password can be typed
Lock and sleep disabled	A locked screen is a dead end
Keyring without password	Avoids a prompt that cannot be satisfied
Accessibility active from login	No activation prompt (Onboard)
Cursor, text, panel and icons enlarged	Limited gaze precision (a few tens of pixels) and finger precision
Application windows maximized	Large targets, panel always reachable
Open with a single click (Nemo)	A dwell double click would require changing modes
Visual keyboard anchored at the bottom	Large targets, word prediction
Visual keyboard hidden at startup	Uncluttered screen, shown by its button
Dark theme by default (Cinnamon)	Like GNOME sessions

The workstation is not protected at boot

Automatic login, no screen lock : anyone who powers on the machine is in the session. The only useful bulwark is disk encryption, to plan from installation if the workstation is mobile.

Facial-recognition login is on the TODO list.

Keyring without password

Only the automatically logged-in account is concerned, encrypted workstation or not : without a password, its keyring opens without a prompt. On an encrypted workstation, LUKS protects its secrets at rest.

No password reaches it, password login included : GNOME would re-encrypt with it. A password-protected keyring is deleted at the next update or boot, its secrets lost.

The panel

Under Cinnamon, enablePanel organizes the bottom panel for the gaze.

Position	Content
Left	Menu, grid, launchers, open windows
Right	Click types, pause zone, gaze
Then	Accessibility, visual keyboard
Finally	Status icons (Talon, Nextcloud…), clock

No Bluetooth icon and no corner bar (show desktop). Pairing goes through the Bluetooth Manager in the menu.

The click types (single, one-shot, double, drag, secondary) and the pause zone only appear if dwell clicking is active. The pause zone lets the gaze explore the screen without triggering anything ; the click types hide during the pause.

The one-shot single click (orange “1” badge) clicks once, then pauses dwell clicking. On resume, the simple click repeats.

Action on the pause zone	Effect
Click, clicks active	Suspends dwell clicking
Pointer resting 1.5 s, paused	Reactivates clicks (green bar)
Mouse click, paused	Reactivates clicks

The gaze button (eye) turns off pointer control by gaze : the eyes no longer move anything, and dwell clicking pauses to leave the hand to the mouse. Turning it back on with the mouse, touch, or the Talon action user.umi_gaze also reactivates clicks. It is turned back on at every start.

The application grid

With enableRofiMenu, the first launcher on the panel opens all applications full screen, in pages of large tiles, with no scrollbar.

Target	Effect
Hovered tile	Highlighted
Click a tile	Launches the application, closes the grid
◀ ▶ (screen edges)	Previous, next page
Close	Closes the grid

Settings panels stay in the Cinnamon menu. hiddenApps hides other applications (.desktop files) from both the grid and the menu. rofiGridResolution sets the number of tiles per page.

To validate with the tracker

Dwell click in the grid not yet tested with the Tobii. The grid captures the mouse: the pause zone, click types and on-screen keyboard stop responding while it is open.

Calibrating gaze

The Calibrate gaze favorite, at the top of the Cinnamon menu, launches Talon’s calibration, without going through its menu in the notification area. It appears with enableTrackerAuto.

Recalibrate after a change of screen, tracker position or user, or lighting.

The tracker mounts under the main screen, the one where Talon calibrates. The Tobii 5 supports a 16:9 screen up to 27”, at 45-95 cm : beyond that (TV…), calibration stops at the first point.

Tuning the pointer live

With enableHeadControl, the gaze brings the pointer into the target zone and the head refines it. Above 1, headSpeed makes the head more sensitive ; below, it tames overly large head movements. These settings can be tried without redeploying, from Talon’s console (REPL), and are lost when Talon restarts.

1. Open the console of the user’s Talon, from an administrator account on the host :

   sudo -u alice env HOME=/home/alice /home/alice/.talon/.venv/bin/repl

2. Read the applied settings :

   print(actions.user.umi_tuning())

3. Try values, the user at the workstation :

   Command	Effect
   actions.tracking.control_head_toggle(True)	Enables the head
   actions.user.umi_head_speed(5)	Head 5 times more sensitive (default)
   actions.user.umi_head_speed(0.3)	Head 3 times less sensitive
   actions.user.umi_head_speed(1)	Talon’s original sensitivity
   actions.user.umi_filter_speed(1, 4)	Gaze and pointer responsiveness (default)

4. Record the chosen values in usr/users/<login>/default.nix (enableHeadControl, headSpeed, gazeFilterSpeed, cursorFilterSpeed), then deploy.

Default values

Measured on a Tobii 5. headSpeed acts on the head acceleration curve, an internal Talon setting.

Other accounts on the host

A regular account keeps its GNOME Wayland session, with a few differences inherited from the host :

• Startup: the machine opens the session for the UMI account. Close that session or switch users to open your own.
• Seahorse: not present, as it is unusable without a keyboard.
• su: does not transmit screen access, and a prompt (keyring…) would block the UMI account. Administer via SSH or from your own session.
• Shutdown: each service has 5 seconds to stop.

Troubleshooting

Symptom	Lead
Talon does not see the tracker	Replug the tracker, then restart Talon (udev rule applied on connection)
Talon icon missing from the panel	Reopen the session
Firmware not initialized	Run Tobii Experience once under Windows
Session opened under GNOME	Check the account’s umi profile and its presence in the host’s users, then reboot
The pointer clicks by itself	enableDwell enabled without a tracker, disable it
“Hover Click” window, clicks always simple, wheel unusable	A mousetweaks daemon is running in the session, started before the update : reopen the session
The pointer always aims beside the gaze	Recalibrate
Calibration stops at the first point	Screen larger than 27”, or too far away
The pointer stops before the gazed point	Increase gazeFilterSpeed and cursorFilterSpeed
The pointer overshoots or jitters	Decrease them
The head moves the pointer too much	Lower headSpeed, or set enableHeadControl to false
The kart does not respond to the mouse	Restore the game’s default keys, read the converter’s messages (journalctl -b -g 'dnf-stk-mouse:')
The joystick is not in the mice read (reading …)	Absent from lsusb : plug it into another USB port (connect-debounce failed in dmesg)

Expected kernel messages

The tracker exposes two video interfaces that uvcvideo cannot handle, hence the messages Unknown video format, No supported video formats found and probe with driver uvcvideo failed with error -22. They are normal and desirable : no kernel driver claims the interfaces, leaving them free for Talon.

FAQ

Can a UMI account choose GNOME?

Not on a umi host: the Cinnamon session is re-imposed at every boot, because Talon requires X11.

And on an ordinary workstation?

The account opens the host’s GNOME Wayland session, with the GNOME on-screen keyboard, the enlarged cursor and text. Talon and Onboard, which require X11, stay inactive.