Towards a General Video-based Keystroke Inference Attack
Zhuolin Yang
Yuxin Chen
Zain Sarwar
Hadleigh Schwartz
Haitao Zheng
Ben Y. Zhao
Proceedings of 32th USENIX Security Symposium (USENIX Security 2023)
[Full Text in PDF Format, 3.4MB]
Paper Abstract
A large collection of research literature has identified the privacy risks of keystroke inference attacks
that use statistical models to extract content typed onto a keyboard. Yet existing attacks cannot operate
in realistic settings, and rely on strong assumptions of labeled training data, knowledge of keyboard
layout, carefully placed sensors or data from other side-channels. This paper describes experiences
developing and evaluating a general, video-based keystroke inference attack that operates in common
public settings using a single commodity camera phone, with no pretraining, no keyboard knowledge, no
local sensors, and no side-channels. We show that using a self-supervised approach, noisy finger tracking
data from a video can be processed, labeled and filtered to train DNN keystroke inference models that
operate accurately on the same video. Using IRB approved user studies, we validate attack efficacy across
a variety of environments, keyboards, and content, and users with different typing behaviors and
abilities. Our project website is here.