A Neurophysiological Investigation of Listening Effort in Normal Hearing Adults Using fNIRS and Pupillometry
Date of Award
Doctor of Philosophy (PhD)
Speech and Hearing Science
Mark Hedrick, PhD
Aaron T. Buss, PhD Patti Johnstone, PhD Patrick Plyler, PhD Caglar Tas, PhD
fNIRS, listening effort, neuroimaging, Pupillometry, semantic information, Speech Recognition
Real-world conversations are often accompanied by some sort of interference that challenges the clarity of the speaker’s message, causing listeners to exert more effort to understand speech. Previous research has demonstrated that when listening to speech becomes difficult, various regions of the brain are recruited beyond those which engage during optimal listening conditions. However, the neural correlates that underly listening effort are not fully understood. Importantly, the pupillary response can be used to index listening effort, such that pupil size increases with increasing cognitive demand. I proposed that pupillometry can be used to characterize the cortical response, such that changes in pupil size would be associated with neural activation that directly relates to effortful processing. The primary goal of this study was to investigate the physiological mechanisms underlying recognition of realistic cochlear implant (CI) speech simulations (i.e., vocoded speech) and examine how effort might be alleviated with access to semantic information. To achieve this, I implemented a speech recognition task and manipulated the semantic content of the sentence (Predictability), spectral degradation (Speech Quality), and the way participants reported their response (Task Mode). Concurrent measurements of speech recognition, pupillary responses, and cortical activation via functional near infrared spectroscopy (fNIRS) were recorded from 41 normal hearing (NH) adults. As expected, challenging vocoded speech resulted in larger pupil sizes than did non-vocoded speech, and use of semantic information reduced listening effort. Interestingly, the largest exertion of listening effort was produced after the sentence presentation ended. Neural data revealed Predictability effects in frontal, temporal, and parietal cortices, such that activation in frontal regions responded to higher-level semantic representations of speech, and posterior brain regions were more sensitive to Task Mode which modulated the cortical mechanisms used to resolve Predictability. Positive correlations between the strength of the neural response and peak pupil response were observed in the Middle Frontal Gyrus (MFG) and Inferior Parietal Cortex (IPC), revealing cortical regions related to listening effort. Surprisingly, pupil size just prior to the sentence onset positively predicted individual speech recognition score, suggesting that the degree to which participants were engaged (or prepared) at the start of the trial predicted their overall speech recognition performance. On the other hand, pupil size right after sentence offset was negatively associated individual capacity to exploit semantic information, suggesting that listeners with smaller improvement from Low- to High-Predictability conditions had engaged more effort after the sentence was presented. Overall, the findings presented here indicate that the MFG and IPC carry out cortical mechanisms related to effortful processing, and listeners with a stronger MFG response were more impacted by the degraded listening conditions as they exhibited more effort (larger pupil size) and lower benefit in performance from semantic information. Therefore, it appears that the effect of listening effort on communication (i.e., speech recognition score and semantic capacity) is determined by the individual physiological response of the listener.
Defenderfer, Jessica (https://orcid.org/0000-0002-2830-7333), "A Neurophysiological Investigation of Listening Effort in Normal Hearing Adults Using fNIRS and Pupillometry" (2022). Theses and Dissertations (ETD). Paper 585. http://dx.doi.org/10.21007/etd.cghs.2022.0594.
Investigative Techniques Commons, Medical Neurobiology Commons, Medical Physiology Commons, Musculoskeletal, Neural, and Ocular Physiology Commons, Neurosciences Commons, Physiological Processes Commons, Speech and Hearing Science Commons, Speech Pathology and Audiology Commons