New Research: ABLE Music Interactive Platform

A demonstration of the MetaMotion R+ Bluetooth wearable sensor by MbientLab controlling an audio visual interactive application created by Pulse Lab.

The ABLE Music platform is a co-creation project in collaboration with older adults with dementia and their caregivers, to provide unique pair or group interactions and intergenerational play by transforming movement into art experiences (digital painting and musical creation) in order to enhance well-ness (physical, mood, cognitive).

The ABLE Music platform advances current research on dementia that tends to, a) not be interactive, b) not exploit digital tools, and c) doesn’t take advantage of the opportunity to engage families and caregivers, reduce their stress, and restore identity, dignity and relationships. These advancements are built on research that demonstrates the benefits of: bright colour palettes’ ability to stimulate older adults with dementia, music and painting experiences that reflect the memories and preferences of older adults, and intergenerational gaming that will allow younger children and adults to teach older adults digital gaming skills.

Art combined with movement has a synergistic effect – it has the power to enhance mood, physical health and cognition. Reducing depression makes us less susceptible to cognitive & memory impairment, due to depression ageing the brain. Arts-based and Montessori- based approaches increase communication, episodic memory, and relationships between People with Dementia (PwD) and their family members and caregivers. Crip approaches to design can help researchers center PwD and caregiver perspectives in treatment plans, in addition to rethinking how power can influence the research process.

The ABLE Music platform will connect to a computer using an MbientLab MetaMotion R+ wearable sensor. The sensor will connect via Bluetooth (BLE) to the host computer to translate the user’s gestures into motion-based sound and art on the screen. The user can clip on the sensor, housed in a silicone case, to their sleeve, a glove or wear it like a watch. The user can then move their hand or extremity that the sensor is attached to in 3D space to create art and music in an application on the host computer.

Machine learning algorithms will aid in the recognition of gestures made by the user. This can aid in recognizing patterns that can help improve the enjoyability of the interaction and aid in the recalling of memories for the user and their caregivers.

MbientLab MetaMotion R+ wearable sensor
MbientLab MetaMotion R+ wearable sensor wristband

Future research to be conducted between Pulse Lab and older adults is the application of wireless gesture based control to art, gaming and music based applications. Our initial research is designing a paint by numbers style art experience that can be based on photographs from the users’ own collection and music that is augmented by the users’ progress in completing the art work.

The paint-by-numbers experience is fully digital. The users input with a wearable sensor controls the paint brush in order to complete the composition, filling in each cell of colour as they move the sensor around.

The beginning of a paint-by-numbers experience.
The finished fully painted art work.