Open tools for movement and sound

Start with what you have – a video, a sound recording or a data table – and find a suitable tool. This section brings together the browser-based SInES Tools (Vienna), PyEyesWeb (Genoa), the String Performance Dataset, VioPose and other pose systems, with a step-by-step guide for students.

Start here

From the recording to the result in eight steps – for Master’s and PhD students, with the most common mistakes.

Learning path →

I have a video

Posture, arm and hand movement, points on the bow, overall motion.

Pose and motion tracking →

I have a sound recording

Loudness, brightness, intonation, vibrato, onsets and tempo.

Sound analysis →

I have data tables

Check tracking data, merge time series, statistics and plots.

Data tables →

Which tool for which question?

A first orientation. Every tool is described on its section page, with input, output and pitfalls.

You have …	You want …	Tool	Then
Video, one person	posture, arms, head	2D/3D Pose Tracker (BlazePose)	3D Data Analyzer → R/JASP
Close-up of a hand	bow hold, fingers	Hand Tracker (21 points)	Data Analyzer
Video	points on bow or instrument	Point Tracker	Excel Synchroniser
Video, several persons	ensemble, lesson	Multiple Person Pose Tracker	2D Data Analyzer
Video	overall motion, no skeleton	Video Motion Analysis	Time Series Analyser
Phone video (mov)	mp4 and the sound	Movie to MP3/MP4	pose and sound tools
Sound	intonation, vibrato	Pitch Estimator, VIP, Modulation Transfer	Excel Synchroniser
Sound	loudness, brightness, noise	Signal Analysis I, Harmonicity Analyser	Columns Correlator
Several tables	one time axis	Excel Synchroniser	statistics
Ratings of several experts	agreement	Interrater Calculator (Fleiss’ kappa); ICC in R	report
Movement data	smoothness, symmetry, synchronisation	PyEyesWeb (Python)	R or Python
Need exact 3D values	validation	reference system (Qualisys, Vicon, Noraxon) or several cameras – see other pose systems	–

When to move from the browser to R or Python: many files, results that must be exactly reproducible, mixed models, and movement-quality features from PyEyesWeb. The browser tools are ideal for exploring, checking and teaching.

Partners and projects

Vienna · University of ViennaSInES Tools
Browser-based lab with about 120 tools for audio analysis, AI estimation, synthesis, body, hand, face and point tracking, visualisers and statistics (Christoph Reuter and team). Healthy Strings uses about forty of them: pose · video · sound · tables.
Genoa · Casa Paganini – InfoMusPyEyesWeb
Open-source Python toolkit for movement quality: smoothness, synchronisation, symmetry, contraction and expansion, equilibrium. More →
BeijingString Performance Dataset (SPD)
Violin and cello, up to 23 cameras, synchronous sound and 3D annotations – and a method in which audio improves markerless motion capture. More →
MarylandVioPose
Audiovisual 4D pose estimation made for the violin. More →
Pisa · Scuola Superiore Sant’AnnaThe BioRobotics Institute
Robotics and bioengineering, artificial touch, wearable sensors and neuroprosthetics; within Healthy Strings biorobotics, sensor technology, motion capture and functional movement analysis.

AI that combines data and expert knowledge

Machine learning can find patterns in multimodal data – body landmarks, audio features, sensor signals. In Healthy Strings these patterns are always related to expert knowledge: the assessments of physiotherapists and violin teachers are the human reference, and their concepts guide which features are worth computing. Methods range from interpretable movement features (PyEyesWeb) to trajectory analysis and machine learning (IMT Lucca, Giorgio Gnecco).

Beyond the camera

Cameras do not see everything. Sensorised instruments – for example a chin rest, fingerboard or bow with force sensors – can measure contact forces that are invisible in video, and EMG shows muscle activity. Healthy Strings therefore combines markerless video analysis with such measurements in selected sub-studies, and checks the results against laboratory motion capture.

Further open resources and literature

Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., & Grundmann, M. (2020). BlazePose: On-device real-time body pose tracking [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2006.10204
Hamilton, R. I., Glavcheva-Laleva, Z., Haque Milon, M. I., Anil, Y., Williams, J., Bishop, P., & Holt, C. (2024). Comparison of computational pose estimation models for joint angles with 3D motion capture. Journal of Bodywork and Movement Therapies, 40, 315–319. https://doi.org/10.1016/j.jbmt.2024.04.033
Dill, S., Ahmadi, A., Grimmer, M., Haufe, D., Rohr, M., Zhao, Y., Sharbafi, M., & Hoog Antink, C. (2024). Accuracy evaluation of 3D pose reconstruction algorithms through stereo camera information fusion for physical exercises with MediaPipe Pose. Sensors, 24(23), 7772. https://doi.org/10.3390/s24237772
Grosshauser, T., & Tröster, G. (2013). Finger position and pressure sensing techniques for string and keyboard instruments. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME) (pp. 479–484). https://doi.org/10.5281/zenodo.1178538

SInES Tools: Christoph Reuter and the SInES team, Department of Musicology, University of Vienna – free, MIT licence, everything runs in your browser, no data leaves your computer. Full catalogue (about 120 tools): sinestools.univie.ac.at. Cite: Reuter, C., Czedik-Eysenberg, I., Cui, A.-X., Ambros, S., Klooss, F., Feller, G., Schachner, I., Roos, M., Lindenbauer, H., Li, H., & Bertsch, M. (2026). SInES Tools, helpers, visualizers as your scientific realizers. In Fortschritte der Akustik – DAGA 2026 (pp. 1002–1005). https://doi.org/10.71568/daga2026.094