Frontiers in Behavioral Neuroscience (Nov 2014)

Fast mental states decoding in mixed reality.

  • Daniele eDe Massari,
  • Daniel ePacheco,
  • Rahim eMalekshahi,
  • Rahim eMalekshahi,
  • Alberto eBetella,
  • Paul F.M.J. Verschure,
  • Paul F.M.J. Verschure,
  • Niels eBirbaumer,
  • Niels eBirbaumer,
  • Andrea eCaria,
  • Andrea eCaria

DOI
https://doi.org/10.3389/fnbeh.2014.00415
Journal volume & issue
Vol. 8

Abstract

Read online

The combination of Brain-Computer Interface technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. In this context, assessing to what extent brain states can be discriminated during mixed reality experience is critical for adapting specific data features to contingent brain activity. In this study we recorded EEG data while participants experienced a mixed reality scenario implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the user's actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in mixed reality, using LDA and SVM classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5 s time-window shifting every 200 ms. Our results indicate overall better performance of LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled mixed reality scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in mixed reality.

Keywords