Rokid, Client: Midea
A dual-input AR workflow for assembly line sorting, where a three-button ring and voice commands drive the same interface. Built on a mapping model that let physical geometry set the rules for the screens.
ROLE
UX Designer
2 UX Designers, 2 Engineers, 1 PM
Jun-Sept 2021
Rokid Glasses 2, Rokid OS
The sorting workflow runs on either input, screen for screen, rather than dividing the two by function. Left, confirm, and right on the ring do the same work as "last one", "confirm", and "next one" spoken aloud, and each screen is laid out to show both, so an operator can drop voice in a noisy bay or drop the ring when both hands are in a parts bin without learning a second model.
Voice-only was the shipped interaction model, in daily use by Midea's sorters, and the feedback came back below what the team expected. Research at Rokid HQ and on-site at Midea traced it to the environment and to the loop: noise and masks fought the input itself, while a workflow built from a few commands repeated all shift turned remembering them into its own workload. Efficient for one task, tiring across a shift.
Two inputs with nothing physically in common had to point at the same interface, so I mapped each one to the screen by a different kind of similarity. The asymmetry is what mattered: voice tolerates any layout, the ring tolerates only layouts shaped like itself. The hardware's geometry became the constraint the screens were redrawn against.
Engineers had built a rapid prototype under time and hardware constraints: two buttons for navigation, one for confirmation, no defined semantics, and no defined placement on the hand. I settled both from comfort studies with in-house employees and on-site assembly workers. Comfort decided it rather than speed, because the ring shares a gloved hand that is already doing the actual work.
Each change below is the ring's shape stated in layout.
Screen composition: mixed → symmetrical, around a center the way the ring is
Call-to-action buttons: vertical stack → horizontal pair, the way the thumb moves
List navigation: four-direction grid → two-direction carousel
Midea adopted the approach, and it reached hundreds of sorting operators.
Operators worked through a sorting task longer before tiring, once the input stopped being the tiring part.
In on-site testing, sorters found the ring and voice interface more comfortable than voice alone.
Engineering built the ring, and nobody had decided what it should mean on screen. I stepped into that gap: took the hardware as given, worked out the interaction model its three buttons implied, and redrew the interface to match it. I had expected to design the screens first and ask hardware to fit them, and it ran the other way. Once the model was clear most of the screen changes followed from it, and the only hard calls were the places where the model had to lose.
