For my thesis, I taught a machine to dance. I used LSTM RNNs (Recurrent Neural Networks) to “learn” the relationship between human joint movements and music. With the help of Google‘s open-source pose estimation API, I was able to extract “dance data” from solely videos of my colleagues having a boogie. Previous state-of-the-art AI dance has required very expensive motion capture studio set ups to collect this dance data - my method negates this requirement and hence makes this early-stage art form more accessible to researchers, artist and engineers. 
1. Create an entirely open-source method of AI dance synthesis, without the need of expensive motion capture data.
2. Explore and test engagement factors associated with AI dance in VR concert environments. 

Motion Capture: An expensive method of dance data collection.

Contemporary VR concert environment

Dance Data Collection
Mediapipe, Google's open source human pose estimation library, was used an alternative method of dance data collection to Motion Capture. This powerful library allowed the 3D coordinate tracking of all human joints. This drastically reduced the cost of cost of data collection - as expensive motion capture studios were no longer required. 
Music Data Collection
Essentia, an open source c++, library was used to extract audio features from the music participants were dancing to during dance data collection. For every frame of dance, 69 audio features were extracted and paired with the joint movement data.
The Machine Learning Process  
Once music and dance data was obtained and currently formatted, thee machine learning models were developed for comparison. The main purpose of the model is to learn the relationship between music and dance patterns, with the purpose of generating dance when given music it has not heard before. The three models were variations of LSTM-RNN (Recurrent Neural Networks), exploring the effects different dimension reduction techniques on dance output. 

The models learns the relationship between music and dance.

Model Comparison 
Engagement Testing... 
Back to Top