Goldsmiths develops audio system for children with multiple learning difficulties
Paul Milligan, October 2, 2012
An audio system aimed at helping children with profound and multiple learning difficulties to interact through natural gestures has been developed by a student at Goldsmiths, University of London.
Designed for users with autism spectrum conditions, mobility, learning or speech difficulties, the Kinect Audio Project (KAP), created by MSc Computing student Patricia Afari, is a gestural instrument intended to encourage exercise, understanding of movement and association, and to address the key issue of independent participation.
Afari said: “Children with profound learning difficulties are unable to perform typical activities in the usual way. By enabling children to interact and recognise their own movements, we hope to encourage independence. KAP enhances children’s own abilities and through engagement with the technology they can better communicate with teachers and other pupils.”
Based on AV cues, KAP displays a camera image of the user and replicates the user’s hand movements on-screen through oversized, animated gloved hands. The user can move their hands to interact with on-screen visuals, which in turn trigger sounds.
Afari added: “Children learn through repetition, so it was important not to incorporate randomised audio responses. Instead, each visual has been assigned a particular note or instrument.”
KAP incorporates hardware and software from gaming technologies, namely Microsoft’s Kinect controller and Firelight Technologies’ FMOD Ex Programmer’s API. It is currently being trialled at South Downs Community Special School, Eastbourne under the supervision of Tom Smurthwaite, an interaction designer who has 12 years’ experience working with people with profound and multiple learning disabilities.
Smurthwaite explained: “Children with mobility problems often fall at the first hurdle of playing a musical instrument, being unable to hold or operate it. Gesture recognition gives them the opportunity to join in. Being based on budget consumer technology and free downloadable software, KAP has the potential to be a required tool in special education. “