VioPose – pose estimation made for the violin

A research model that estimates the 3D pose of violinists over time (“4D”) from video and sound together. It is trained on violin playing, where general pose models struggle with the small, fast movements of bow and fingers.

What it does

Audiovisual: the sound helps to resolve movements the camera sees poorly.
Hierarchical: estimates pose, velocity and acceleration in stages, so that the motion stays physically plausible.
Comes with the dataset VioDat: violin performances recorded with several smartphones and 3D motion capture as reference.

Using it

VioPose is research code in Python and runs locally – no upload. Installation needs some care (a separate Python environment and a few adjustments on a Mac); Healthy Strings keeps its installation notes in the team area. Compared with the SInES pose tracker it is less convenient but specialised for the violin; both can be compared on the same recordings.

Literature

Yoo, S. J., Shrestha, S., Muresanu, I., & Fermüller, C. (2025). VioPose: Violin performance 4D pose estimation by hierarchical audiovisual inference. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 4923–4934). https://doi.org/10.1109/WACV61041.2025.00482