Tom M. Mitchell, Svetlana V. Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L. Malave, Robert A. Mason, Marcel Adam Just
Predicting Human Brain Activity Associated with the Meanings of Nouns

• authors present a computational model, which predicts the functional magnetic resonance imaging (fMRI) of neural activation associated with words for which no fMRI data are available
• fMRI prediction for a word w is a two-step process:
  1. compute a vector of semantic features over a huge corpus
     • 25 features are defined in terms of co-occurrence of w with forms of 25 manually selected sensory-motor verbs
  2. predict neural fMRI activation as a weighted sum of semantic features
     • weights for every voxel (3D pixel) and feature are estimated using multiple regression
• fMRI data
  • they created 60 representative fMRI images
    • word - picture combination from 12 semantic categories
    • they measured brain activation of 9 participants after being exposed to all 60 word - picture combinations
• evaluation
  • they carried out “leave-two-out” cross validation with all 60 examples
    • 58 of them served as a training data
    • for two of them fMRI images were predicted and compared with observed images. On the basis of cosine similarity measure a matching was determined. If the predicted image for the first word matches with its corresponding observed image, one positive point is scored - aggregated over all folds, it forms an accuracy measure
• experiments and results
  • quantitative measurments
    • matching two unseen words to their seen fMRI images
      • 0.77 averaged over all 9 participants - significantly above a chance level
      • higher activation tends to be predicted in the left hemisphere - it is consistent with the generally held view that left hemisphere is more responsible for semantic representation than the right one
    • prediction for a word in a new semantic category
      • in training stage, they excluded all examples from the same semantic category as either of the two tested words
      • 0.70 averaged over all 9 participants - still above the chance level
    • prediction, when two tested words belong to the same category
      • hard to distinguish
      • 0.62 averaged over all participants - slightly above the chance level
    • ability to distinguish in even more diverse range of words
      • model trained on 59 examples and tested on a remaining example + another 1000 highly frequent words
      • it ranked predicted fMRI images for 1001 words with respect to its similarity to observed fMRI image of the testing example
      • 0.72 over 9 participants
  • examination of learnt basis set of fMRI signatures for 25 verb-based signatures
    • 'eat' predicts strong activation in gustatory cortex involved in the sense of taste, 'push' in a part of brain involved in the planning of complex coordinated movements, 'run' in the part involved in perception of biological motion
    • though for other verbs these correspondences between the function and brain regions is not present across all participants
  • convenience of selected verbs as a basis for features
    • they generated 115 random sets of 25 features constructed from 5000 highly frequent words (excluding 25 verbs used in an original setting) in corpus and trained system using these feature sets
    • accuracy of prediction fMRI ranged from 0.46 to 0.68 (with mean equal to 0.60) ⇒ compared with 0.77 in setting using

• authors selected 25 sensory-motor verbs as a basis for their co-occurence features. But they did not sufficiently explain what led them to pick exactly these ones.

Written by Michal Novák