 |
| © Provided by Atlantic Media, Inc. Using Microsoft HoloLens |
By Cassie Werber, Quartz
There will come a time when sitting in front of a screen, tapping
keys with your fingers, will seem impossibly clunky and laughably
old-fashioned. Instead, researchers say, the most likely next phase of
human-computer interaction will be typing in “thin air” while gazing—for
example—at the ocean.
As computers become smaller and—ultimately—largely virtual,
researchers are having to think of other ways for us to interact with
them. The current generation of smartphone users may be willing to learn
to type with just two thumbs; some of us have carried around bluetooth
keyboards for train journeys and conferences. But no one wants to
connect a keyboard to a smart watch, and while voice recognition
software is improving,
users are highly resistant to talking to computers in public. If the
future of screens is augmented reality—as Tim Cook, Apple’s CEO has predicted—
a virtual space where the user sees both what’s really in front of them
and the “objects” on the screen, how are we going to interact with it?
The question is not just a philosophical one, but has major commercial applications. Per
Ola Kristensson, a professor in Interactive Systems Engineering at the
University of Cambridge, says desks in the future will likely be almost
entirely free from hardware: No laptop, no phone, and certainly no
keyboard. A move away from restrictive un-ergonomic
keyboards would help those affected by problems like carpal tunnel
syndrome, repetitive strain injury, back pain, and eye strain, which
have become some of the modern worker’s most troublesome physical
problems, and can be debilitating.
There are two big
challenges to designing human-computer interaction in this reality,
Kristensson explained. Designers no longer control what the user is
looking at (what he calls the “pixel-space” of the traditional screen);
and if someone’s work takes place in a space without traditional
limits—no buttons being pressed or screens swiped—there’s an added
challenge of working out what it is the user wants to do.
Kristensson presents a possible solution to this challenge in a forthcoming research paper.
The researchers used a head-mounted piece of hardware to project a
virtual screen and a virtual keyboard, on which users “typed.” The
fingers don’t touch anything; their movement was detected by a
depth-sensor, which tracked both where the fingers typed, and how ‘deep’
they pressed into the virtual surface.
The problem with the
resultant data was that the imprecision of hitting a virtual key rather
than a real one meant lots of errors. To correct for this, the
researchers created a model of what humans were most likely to type.
Working from a dataset of billions of words from Twitter, blogs, and
other online sources, they used machine learning to train a program to
recognize the most-common letter combinations. (The technology, which
currently exists in English and German, is similar to that used in
predictive text on a traditional smartphone.)
The speed at which
users learned to type virtually was fascinating. After users had
practiced on the visible keyboard, they removed all the letters. Users’
typing speed barely altered. “It turns out people remember the
QWERTY keyboard very well,” Kristensson says. “So you can actually not
show the keys at all. It can be completely blank, and people can still
type.” Rather than screensavers of beautiful views, we could be staring
at the views themselves.
But wait: If we’re removing the keyboard, shouldn’t we be removing the need to move the fingers as if they’re touching a keyboard? Indeed, technology allowing people to type simply by looking at the letters does exist. Early
eye-gaze typing, developed for adults with cognitive function but
reduced mobility, was once slow and straining, but has come a long way.
At a recent artificial intelligence conference in Cambridge, Kristensson
presented a method for typing which combines eye-tracking with
prediction, meaning that a user can slide their gaze from letter to
letter, not dwelling, and not worrying if their gaze ‘touches’
intervening letters. (It’s somewhat similar to the Android keyboards
that allow a user to slide their finger from letter to letter; Kristensson was one of the inventors of that technology back in 2002.)
The
Tobii-Dynavox, a robust tablet-like computer that can be controlled
with only the gaze, is one example of “dwell-free” eye technology being
marketed to people with decreased motor function.
All these solutions, of course, maintain at least the idea
of a keyboard. Is it necessary? Interacting with a computer via thought
alone is a tantalizing prospect. In fact, thought-only typing is
technically possible, but as previous experiments have shown, it’s still
a laborious process.
Facebook
was for a time seriously researching brain-only interaction, though the
head of its moonshot division at the time, Regina Dugan, has since
left. Mind-typing, Kristensson says, had engendered a lot of interest, but relies on signals too faint and imprecise to lead to real outcomes—or as he puts it: “the equivalent of being outside a concert house and trying to listen for the flute.” To make it actually efficient, he says, you’d need to drill a port in your head.
There are those who want to take that step. Elon Musk has said he sees the future of AI as necessarily involving the implantation of hardware in the human brain, and has suggested a fine mesh of electrodes that can knit, over time, with brain cells.
How
soon until it’s normal to see someone lying on a comfortable sofa,
gazing at the ceiling, and using a pair of glasses or nearby eye-tracker
to compose an email or write a novel? Tech companies are secretive, but
“everybody’s racing to figure out how to do this and how to make it deployable,” Kristensson says.
COMMENTS