PianoHandSync: An Alignment-based Hand Pose Discrepancy Visualization System for Piano Learning

CHI 2023 LATE-BREAKING WORK (VIDEO FIGURE)
PianoHandSync: An Alignment-based Hand Pose Discrepancy Visualization System for Piano Learning
Ruofan Liu, Erwin Wu, Chen-Chieh Liao, Hayato Nishioka, Shinichi Furuya, Hideki Koike

ABSTRACT
Video-based lessons are becoming a popular way for distance piano education. However, limited by the fixed camera angle, a video is difficult to tell precise 3D hand posture, which is one of the most essential factors for learning piano. This paper presents a visualization system providing the intuitive discrepancy of hand postures in two piano performance videos. Through a motion capture system, the estimated 3D postures are visualized and discrepancies based on distinct metrics are displayed, integrated with modular functions assisting skill acquisition. A pilot study proves that the proposed visualization can be a supplementary means for only video-based lessons in terms of correcting hand postures and fingering.

Видео PianoHandSync: An Alignment-based Hand Pose Discrepancy Visualization System for Piano Learning канала RUOFAN LIU