In my last post, I took you behind the scenes of my recent AI project: a GPT-3 enhanced communication device for adolescents with speech impairments. After countless hours of coding and testing, I'm thrilled to share the results of this experiment that earned me a spot at the California State Science Fair!
But first, a refresher. As the head tutor for the nonprofit organization Inspired to Learn, I've seen both the great potential and challenges that come with AAC devices. Because many of the kids I mentor find using their AACs cumbersome, I wanted to see if AI could remedy the issue. I developed a prototype of an AAC that used OpenAI's GPT-3 API to generate full sentences from the words that the kids tapped. I then tested the novel device virtually on 6 adolescent participants with speech impairments.
To conduct my experiment, I set up eight 10-minute virtual sessions with each participant individually. In each session, I had a conversation with them about topics of their interest. The participants used a control device (identical, but without the AI feature) in the first four sessions and the prototype in the final four sessions.
I measured and averaged the "conversation turns" each participant exhibited, which I defined as the number of times they "spoke" a complete sentence or phrase using the device. This way, I was able to determine whether or not the AI-based AAC increased the participants' rate of communication.
The graph above shows that participants consistently communicated at a much higher rate when they used the AI-powered AAC. On average, there was a 104% increase in overall communication turns when they switched from the control to the prototype. Awesome, right?
Not quite yet. On closer inspection of the session transcripts, I found that many of the AI-generated responses didn't make much sense in the context of the conversations. I counted these "nonsensical" conversation turns and compared their frequency in both the control device phase and the experimental phase:
In this graph, you can see that there is a significantly higher proportion of conversation turns classified as "non-answers" when participants used the prototype than when they used the baseline control.
In other words, GPT-3 sometimes outputted wacky sentences when fed a few words as input and instructed to form a full sentence from them. Nevertheless, there's no denying the great increase in conversation turns and communication rate from the prototype, suggesting the potential of AI to streamline AAC communication. In addition, the participants seemed more engaged in the conversations when they used the prototype with the AI feature. In a follow-up questionnaire, they reported that the AI-powered AAC was "more fun than the [control]" to use and that it's "a good idea for the future."
AI-enhanced AAC devices will need many improvements to become practical for everyday use. For example, future researchers could try to implement an AI language model that continuously learns from users' speaking styles to generate more personalized sentences.
With the current incredible growth of innovations in AI, it's undoubtedly likely that Large Language Models will advance enough to make communication devices more efficient. It would be a game-changer for the millions of people who rely on AACs, helping them express themselves more effectively.
Stay tuned for more AI project updates!
Comentários