Thesis

Towards brain-computer interface-based information systems : leveraging machine learning for cognitive interaction

Creator
Rights statement
Awarding institution
  • University of Strathclyde
Date of award
  • 2025
Thesis identifier
  • T17350
Person Identifier (Local)
  • 202076919
Qualification Level
Qualification Name
Department, School or Faculty
Abstract
  • The field of computational neuroscience has seen significant growth, driven by thedevelopment of sophisticated machine learning algorithms. These advancements allowfor detailed analysis of brain signals, helping researchers to discover new properties andphenomena within the human brain. Progress in machine learning, especially with theintroduction of the transformer architecture and Large Language Models (LLMs), hasrevolutionized Natural Language Processing (NLP) by achieving unprecedented results.The continuous evolution of these methods highlights the ongoing advancements in NLPtechnologies and applications.Beyond NLP, machine learning models like Data2Vec and Wav2Vec2 have broadened possibilities in image and video processing by integrating multiple data modalities.These models improve multi-modal frameworks, enhancing the capability to interpretvarious inputs, such as combining textual and visual data in question-answering systems. This integration signals a paradigm shift in machine learning, exemplified byadvancements like voice-activated assistants and Google Lens, which offer innovativeinteraction methods. This thesis explores the potential of brain-computer powered interfaces for user interaction through cognitive processes as an alternative way of interacting with a computer system, recognizing the need to address foundational challengesto advance this cutting-edge field. Due to the early stages of this innovative discipline,substantial groundwork is required to identify and systematically resolve the multifaceted challenges intrinsic to the development of such advanced interaction systems,thereby establishing a robust foundation for future advancements in this intriguingdomain.In the Chapter 1 of this thesis, a thoroughly developed introduction is presented This section fulfils several critical functions: it outlines the structure of the thesis, providing readers with a coherent roadmap of the thesis’s contents and the trajectory ofthe forthcoming discussion. Additionally, it concisely summarizes the research achievements to date, offering a retrospective overview of the progress attained during theinvestigation. The chapter also examines the foundational motivation driving the research effort, clarifying the reasoning behind the development of the proposed system.Central to this section is the expression of the core research questions that the thesisaims to explore, which are essential in steering the scholarly inquiry and contributingto the broader academic dialogue.Within Chapter 2 of the present thesis, a comprehensive literature review has beenrigorously executed to encompass a broad spectrum of the most esteemed brain imagingmodalities, along with their recent advancements within the multidisciplinary sphereof neuroscience. This extensive analysis also highlights certain contemporary developments in machine learning that are pertinent to the field of neuroscience. Moreover, itpresents particular tools, such as Data2Vec, which primarily are not being specificallyintended for interaction with neural data, bear potential utility in the conceptualizationand design of such a complex system.In Chapter 3, the proposed system is introduced in detail. This chapter offers anexhaustive description of the system, articulating all of its essential components andanticipating potential challenges that may arise during its development. A schematichigh-level design of the system is presented. It is crucial to emphasize that the fullimplementation of such an ambitious system lies beyond the scope of this thesis. Chapters 4 through 6 provide a thorough investigation into three critical components of thiscomplex system. Each component is meticulously examined to clarify the challengesencountered, the recent progress made, and the subsequent improvements achieved tointegrate these components seamlessly into the comprehensive system framework.In Chapter 4, we employ a rigorous analytical methodology to evaluate the congruence of advanced natural language processing models with empirical neuroscientificdata. This evaluation is imperative for the prospective implementation of models capable of interpreting human cognitive processes. Our findings validate that while certain models do not achieve complete congruence with brain data, they demonstrate a significant level of alignment, thereby affirming the proficiency of current state-of-the-artmodels in acquiring intricate representations.Chapter 5 embarks on a comprehensive investigation into the domain of text generation derived from neurological data. This investigation is motivated by two primaryconsiderations. Firstly, text serves as one of the most prevalent means through whichindividuals interact with computational systems. A variety of mechanisms have beendeveloped to facilitate this interaction with both precision and security. Adhering tothe principle of eschewing the duplication of existing mechanisms, it was hypothesizedthat if a model could be developed to generate text from neural signals, it could beseamlessly integrated with existing systems. The brain’s capacity to provide precise lexical terms could, therefore, enhance search engine inputs by mitigating issues relatedto query disambiguation.Secondly, within the domain of Neuroscience, the ambition to generate text frombrain activity, particularly through non-invasive neuroimaging techniques, has persistedas a longstanding intellectual venture among many researchers. As presented in Chapter 5, the preliminary attempt to actualize this type of generation involved utilizingtransformer models to synthesize brain features and subsequently applying a large language model to produce text. Diverse activation functions were employed to optimizethe performance of the processing pipeline. This approach was based on the observation that conventional activation functions predominantly assume linearity withinthe data at some stage of the training process, as demonstrated by their graphicalrepresentations.Nonetheless, empirical studies have indicated that the aforementioned assumptionfails to hold in real-world data situations, notably within meteorological datasets. Theempirical assessment of various activation functions identified that those based on polynomials, along with polynomial functions featuring adjustable constants fine-tuned during the training phase, emerged as the most effective. These results represent a significant advancement in the neural data-to-text transformation framework, presenting anovel dimension to the fields of computational and neuroscientific research.In the course of advancing this thesis, particularly in the forthcoming Chapter 6, anexploration is undertaken into the development of a novel brain encoder. This encoderseeks to establish a comprehensive and generalized modelling framework capable ofeffectively learning and encapsulating general features pertinent to the cognitive processes by which the human brain interprets language. The impetus for this researchtrajectory stems from the findings of Chapter 5, wherein, despite surpassing existingbaseline results, there remained a significant gap in achieving authentic brain-to-textdecoding capabilities. The hypothesis informing this investigation posits that the limitation does not reside in the incorporation of specific neural features derived from thetransformer encoder model. Chapter 6 further explores how advanced methodologiessuch as Data2Vec and Wav2Vec2 might be harnessed to formulate such holistic brain’embeddings.’ Previous efforts in this domain have predominantly focused on utilizinggeneric embeddings for mental state classification, with minimal focus on their potential in the generation domain. Consequently, Chapter 6 articulates and implementsa systematic pipeline designed to construct these generic embeddings, systematicallyapplying them to the complex domain of brain-to-text conversion processes, therebyoffering a novel perspective and making a significant contribution to the field.In Chapter 7, we present the design of a novel interface that offers a dual contribution to both the Neuroscience and Machine Learning communities. The primaryobjective of this thesis was to develop a software system that enables users to interactthrough brain activity. We have designed and implemented a chatbot interface capable of concurrently capturing brain data while functioning as a conventional chatbot.Furthermore, this chatbot is engineered to be adaptable and highly customizable withmillisecond precision, allowing it to serve as a bridge between machine learning andneuroscience, as well as a platform for further neuroscience-focused data collection.In the final Chapter 8 of this thesis, a comprehensive and detailed synthesis of theresearch findings is meticulously articulated, with particular emphasis on the systematicaddressing of each research question. Additionally, this chapter scrupulously delineatesthe current limitations that hinder the development of such an innovative system. Athorough and comprehensive report is subsequently presented, providing robust andpractical guidelines for leveraging the insights derived from this research. Such a reportis crucial for constructing a framework upon which future scholarly research can buildto ultimately achieve the completion and realization of this sophisticated system.
Advisor / supervisor
  • Moshfeghi, Yashar
  • Pollick, Frank
Resource Type
DOI

Relations

Items