SPD – String Performance Dataset

A large multi-camera dataset of violin and cello performances with synchronous sound and 3D annotations – and a method in which the audio signal improves markerless motion capture of string players.

What the dataset contains

Violin and cello, players from students to professionals, more than three hours of music.
Up to 23 synchronised cameras (2656 × 2300 pixels, 30 fps) and studio sound.
3D annotations of body, hands, instrument and bow – recorded without markers, so the players are not disturbed.

The method: audio matters too

The authors combine several detectors: a whole-body pose model with 133 keypoints (DWPose), point tracking for the instrument (TAPNet) and a trained detector for frog and tip of the bow (YOLOv8). The sound adds what cameras miss: the pitch tells where the fingers must be on the string, and bow and string contact can be inferred. Fusing the views gives detailed 3D motion of player, instrument and bow.

How Healthy Strings uses it

Survey clips of three SPD violinists are part of the expert video ratings (30R and 90R views).
A multi-camera viewer in the team area shows all views of a recording with synchronised sound (SPD viewer, login).
The many camera angles make it possible to test which views a markerless analysis really needs – a question behind our two-camera protocol.

Use within Healthy Strings is internal until the licence of the data is clarified. The article itself is open access (CC BY 4.0).

Links and literature

Jin, Y., Qiu, Z., Shi, Y., Sun, S., Wang, C., Pan, D., Zhao, J., Liang, Z., Wang, Y., Li, X., Yu, F., Yu, T., & Dai, Q. (2024). Audio matters too! Enhancing markerless motion capture with audio signals for string performance capture. ACM Transactions on Graphics, 43(4), Article 90. https://doi.org/10.1145/3658235
Dataset and code: github.com/Yitongishere/string_performance