Goldsmiths develops audio system for children with multiple learning difficulties

Goldsmiths KAP system

Designed for users with autism spectrum conditions, mobility, learning or speech difficulties, the Kinect Audio Project (KAP), created by MSc Computing student Patricia Afari, is a gestural instrument intended to encourage exercise, understanding of movement and association, and to address the key issue of independent participation.

Afari said: “Children with profound learning difficulties are unable to perform typical activities in the usual way. By enabling children to interact and recognise their own movements, we hope to encourage independence.  KAP enhances children’s own abilities and through engagement with the technology they can better communicate with teachers and other pupils.”

Based on AV cues, KAP displays a camera image of the user and replicates the user’s hand movements on-screen through oversized, animated gloved hands. The user can move their hands to interact with on-screen visuals, which in turn trigger sounds.

Afari added: “Children learn through repetition, so it was important not to incorporate randomised audio responses. Instead, each visual has been assigned a particular note or instrument.”

KAP incorporates hardware and software from gaming technologies, namely Microsoft’s Kinect controller and Firelight Technologies’ FMOD Ex Programmer’s API. It is currently being trialled at South Downs Community Special School, Eastbourne under the supervision of Tom Smurthwaite, an interaction designer who has 12 years’ experience working with people with profound and multiple learning disabilities.

Smurthwaite explained: “Children with mobility problems often fall at the first hurdle of playing a musical instrument, being unable to hold or operate it. Gesture recognition gives them the opportunity to join in. Being based on budget consumer technology and free downloadable software, KAP has the potential to be a required tool in special education. “