Tom M. Mitchell, Svetlana V. Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L. Malave, Robert A. Mason, Marcel Adam Just
Predicting Human Brain Activity Associated with the Meanings of Nouns

• authors present a computational model, which predicts the functional magnetic resonance imaging (fMRI) of neural activation associated with words for which no fMRI data are available
• fMRI prediction for a word w is a two-step process:
  1. compute a vector of semantic features over a huge corpus
     • 25 features are defined in terms of co-occurrence of w with forms of 25 manually selected sensory-motor verbs
  2. predict neural fMRI activation as a weighted sum of semantic features
     • weights for every voxel (3D pixel) and feature are estimated using multiple regression
• fMRI data
  • they created 60 representative fMRI images
    • word - picture combination from 12 semantic categories
    • they measured brain activation of 9 participants after being exposed to all 60 word - picture combinations
• evaluation
  • they carried out “leave-two-out” cross validation with all 60 examples
    • 58 of them served as a training data
    • for two of them fMRI images were predicted and compared with observed images. On the basis of cosine similarity measure a matching was determined. If the predicted image for the first word matches with its corresponding observed image, one positive point is scored - aggregated over all folds, it forms an accuracy measure
• experiments and results
  • quantitative measurment
    • matching two unseen words to their seen fMRI images
      • 0.77 averaged over all 9 participants - significantly above a chance level
      • higher activation tends to be predicted in the left hemisphere - it is consistent with the generally held view that left hemisphere is more responsible for semantic representation than the right one
    • prediction for a word in a new semantic category
      • in training stage, they excluded all examples from the same semantic category as either of the two tested words
      • 0.70 averaged over all 9 participants - still above the chance level
    • prediction, when two tested words belong to the same category
      • hard to distinguish
      • 0.62 averaged over all participants - slightly above the chance level
    • ability to distinguish in even more diverse range of words
      • model trained on 59 examples and tested on a remaining example + another 1000 highly frequent words
      • it ranked predicted fMRI images for 1001 words with respect to its similarity to observed fMRI image of the testing example
      • 0.72 over 9 participants

• authors selected 25 sensory-motor verbs as a basis for their co-occurence features. But they did not sufficiently explain what led them to pick exactly these ones.

Written by Michal Novák