AR timewarping: A temporal synchronization framework for real-Time sensor fusion in head-mounted displays

Peter Kim, Jason Orlosky, Kiyoshi Kiyokawa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

A significant issue associated with the use of head-mounted displays for augmented reality is the presence of latency between the real world and the augmented images displayed to the headset, which, if uncompensated for, results in registration error that may limit the effectiveness of the augmented information as well as bring about user discomfort. In addition, further temporal discrepancies arise in the case when fusing information from multiple cameras of different capture frequencies to construct the augmented image as, in addition to temporal misalignment between the real world and the augmented image, there is desynchronization among the different sensors that may also lead to registration mismatch. In order to address these temporal inconsistencies, we present AR Timewarping, a novel temporal synchronization framework that is particularly adapted for video see-Through (VST) head-mounted displays and consists of two main algorithms, one for head motion and one for scene motion, that together act to temporally warp and merge information from multiple sensors to significantly improve registration in the augmented image. System tests of our algorithms show that we can reduce registration error between two unsynchronized video streams by 87.04% and 81.64% for registration error arising from head motion and scene motion, respectively. Results from a user experiment show that subjects' abilities to track a moving object were significantly improved by our algorithms, with a 32.14% average reduction in angular tracking error, and furthermore subjects rated the combined algorithms better overall than the base case in terms of both image clarity and user comfort.

Original languageEnglish (US)
Title of host publicationProceedings of the 9th Augmented Human International Conference, AH 2018
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450354158
DOIs
StatePublished - Feb 6 2018
Externally publishedYes
Event9th Augmented Human International Conference, AH 2018 - Seoul, Korea, Republic of
Duration: Feb 7 2018Feb 9 2018

Publication series

NameACM International Conference Proceeding Series

Conference

Conference9th Augmented Human International Conference, AH 2018
Country/TerritoryKorea, Republic of
CitySeoul
Period2/7/182/9/18

Keywords

  • Augmented Reality
  • Delay Compensation
  • Sensor Fusion
  • Vision Augmentation

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'AR timewarping: A temporal synchronization framework for real-Time sensor fusion in head-mounted displays'. Together they form a unique fingerprint.

Cite this