Fingerspelling with Machine Learning
Experiences,Products
Type:
Experiences
Client:
J.Views
Deliverables:
Development, UI, UX, Concept, Illustration
Following The DNA Project, J.Views and Hello Monday teamed up once again to create a new kind of musical experience. The world’s first interactive music video that you, the viewer, can create with every beat of your heart.
Grammy-nominated artist J.Views, embarked on a mission to document the creative journey of making an album — including the pitfalls and setbacks experienced along the way. One such moment, feeling lost in his song-writing process and not feeling present, led him to, "Almost forgot the heart”. As the lead single to his album 401 days, we wanted to create a unique digital experience that would force us to be fully in the moment.
By placing your finger on the back camera of your device, we are able to detect slight changes in your skin colour that occur each time your heart pumps blood through your body - a process known as photoplethysmograph. We created an algorithm that would detect this phenomenon, and adjust the song’s tempo according to the users heart rate, whilst also producing synchronized visuals.
The faster your heart-rate is, the faster the song plays. The song is made with a flexible BPM in mind so the song can change its tempo based on your heart rate.
Your heart rate is represented by a small illustrated creature as it travels through an increasingly colorful landscape changing the longer you stay present.