We last left our intrepid Strand 1 researchers on听the precipice of the incoming classroom data听from our cross-strand data collection team.听As this new data comes in, the Strand 1 team听spent the quarter听facing a new challenge鈥攈ow exactly will听we use this data to define the capabilities of our AI Partner?
What Can Our AI Partner Hear & Understand?
Before our future AI Partner can help facilitate small group听classroom collaborations, it鈥檒l need to recognize what students are saying. Strand 1鈥檚 Speech Recognition team was听hard at work this quarter to听help make this a possibility. This听group of researchers selected five sessions of datasets collected from various Denver Public Schools and St. Vrain Valley听Schools middle school classrooms. The team manually transcribed and annotated mark times when each student was听speaking (to the nearest 0.1 sec). These data are being used to听evaluate both background filter performance and diarization
performance using Automatic Speech Recognition (ASR) 鈥斕齮he process of automatically transcribing audio input.
To facilitate this work, the team installed the recently released SpeechBrain system and provided samples of speech听from each target student. The team then used the novel听approach of having the system filter out from these sample听segments that do not contain speech from any of the target听speakers.
The group will continue the current work on filtering background noise and tuning Google ASR for optimum performance, including providing language model context. They will听also be augmenting the existing hardware testing pipeline and听building a similar pipeline compatible with cameras.
In conjunction with the Speech Recognition team, the Annotation Working Group and the Content Analysis team are also听piloting other types of annotation of the students鈥� utterances. These include a听straightforward on-topic/off-topic classification, more Abstract Meaning Representations (AMRs),听and a layer of Rhetorical Structure Theory (RST). In addition,听Strand 1 is engaged in developing a new听AMR parsing model听that will encompass this new, preliminary RST (RST) annotation. Experimental results are expected soon! To parse the听student dialogues from our classroom datasets, the team听is also听working on a few shots learning model鈥攁n object听categorization machine learning model that aims to learn information about object categories from a handful, rather than听hundreds or thousands, of听training samples. The teams鈥� best听AMR parsing results so far have been an 81.4% accuracy rate听on a medium sized model, and they hope to test larger AMR听parser models in the coming weeks.听
Recent Strand 1 advances in AI capabilities
Our Dialogue Management team was also hard at work听helping our future AI Partner understand student speech. This听quarter, the team worked on two dialogue systems surveys:听one that describes current听dialogue models and architecture,听and another focused on designing human-centered dialogue听systems. The latter explicitly recognizes the different stake-holders in the dialogue process and focuses on the areas of听end-user needs, design values, and data collection as well as听an evaluation of these systems.
What Can Our AI Partner Understand听& Do?
As several teams work toward creating the ears and eyes for听the Partner, other Strand 1 teams鈥攚ith the help of Strands 2听and 3鈥攁re working to help the Partner understand what to听do with this information.听The Annotation team worked toward creating annotations听that will help our AI Partner detect听which students are working toward the assigned task and which students are off task.
On-task students might be talking about the content of the听class, for example, or they might be talking through a group听assignment. The group also听hopes to annotate 鈥渁ttentiveness鈥澨齛nd 鈥渕ind-wandering,鈥� as well as students who are moving听the conversation forward. As their work progresses, the听Annotation team will work closely with Strand 3 for guidance on听socially relevant annotations, such as inclusive听speech versus听harmful speech.听
While the Annotation group strives to help our AI Partner听understand what constitutes on-topic and off-topic behavior, the Reinforcement Learning (RL)听鈥� a machine learning听paradigm that allows AI agents to learn how to take actions听(i.e., learn policies) in an environment by interacting with it or听by听observing others鈥� interactions 鈥� team focuses on enabling听the AI Partner to learn and improve discourse policies from听real interactions in the听classroom. The team works closely听with our Strand 2 experts on Collaborative Problem Solving听skills to understand which policies will help the AI听Partner听learn. During this quarter, the RL team developed a new RL听approach that enables them to teach a single agent to learn a听basket of policies听instead of only one. In the coming quarters,听the team hopes to adapt their approach to the classroom听with the goal of seeing if they can tune the AI听Partner鈥檚 behavior on the fly to match teachers鈥� changing objectives.
听