Skip to main content

Computer Science PhD Defense , Ruixue Liu " Decoding Cognitive States from fNIRS Neuroimaging Data Using Machine Learning"

Wednesday, December 02, 2020
4:00 pm to 5:00 pm

Committee members: 

Prof. Erin Solovey, WPI - Computer Science (Advisor)

Prof. Rodica Neamtu, WPI - Computer Science

Prof. Ali Yousefi, WPI - Computer Science

Prof. Kate Arrington, Lehigh University (External Member)


Date and time: December 2, 4 pm-5 pm


Recently, the use of functional near-infrared spectroscopy (fNIRS) neuroimaging has received focus because of its promise for monitoring an individual's cognitive state in more ecologically valid studies. This dissertation focuses on improving and expanding the usability of fNIRS for brain-computer interaction research. Particularly, we investigated the feasibility of using fNIRS to identify several user states that occur frequently in human-computer interaction, and that could inform adaptive user interfaces, but that are difficult to detect. We accomplished this goal by designing and conducting three human subjects experiments, collecting and curating fNIRS datasets, as well as developing and applying novel machine learning methods appropriate for the particular classification problem and that are tuned to the characteristics of fNIRS data. We distinguish mind-wandering state from on-task states, classify different levels of driver cognitive load, and differentiate positive and negative cognitive processes during learning. 


First, to explore mind wandering detection, we conducted a study using fNIRS during the Sustained Attention to Response Task (SART) to collect fNIRS data associated with mind-wandering state and on-task state. To incorporate individuals' differences in fNIRS data, we proposed an individual-based novel window selection algorithm. We show the proposed algorithm can significantly improve the accuracy for detecting mind-wandering using fNIRS. Secondly, to explore driver cognitive load, we conducted a study using fNIRS in a driving simulator with the n-back task used as a secondary task to impart structured cognitive load on drivers. To extract important spatial and temporal patterns from fNIRS data, we explored the application of artificial neural networks, including Convolutional Neural Networks (CNNs), multivariate Long Short Term Memory Fully Convolutional Networks (LSTM-FCNs), and Echo State Networks (ESNs). We show the proposed ESN method achieves state-of-art accuracy for cognitive load classification. Lastly, we explored cognitive processes associated with positive and negative learning outcomes. To do this, we conducted a study using fNIRS with a rule-learning task. To validate the proposed ESN model's generalizability across tasks, we applied the ESN model for classifying successful and unsuccessful rule learning processes. Our results suggest that the ESN model can effectively extract significant temporal patterns from fNIRS data. The results from this research serve as a foundation for future work that integrates fNIRS into brain-computer interfaces that can automatically adapt to an individual's changing cognitive states, with important implications in many domains, such as gaming, driving, and learning.