MC-ViViT: Multi-branch classifier-ViViT to detect mild cognitive impairment in older adults using facial videos

Keywords:

Deep learning, Facial expression features, Inter- and intra-class imbalance, Mild Cognitive Impairment, Multi-branch Classifier, Transformer, ViViT.

Contributions:

• We propose MC-ViViT to detect MCI from the interview videos provided by the I-CONECT Study.
• We design the MC module to enrich the extracted spatio-temporal features. Its multi-branch structure helps ViViT to capture the visual features from different perspectives.
• We develop the HP loss by combining Focal loss and AD-CORRE loss. The HP loss addresses the inter- and intra-class imbalanced issues and helps the model pay attention to classes with less samples and subjects with short video lengths.

Production: MC-ViViT, Multi-branch Classifier (MC), HP loss

Used Framework:

BibTex:

@article{sun2024mc,
  title={MC-ViViT: Multi-branch Classifier-ViViT to detect Mild Cognitive Impairment in older adults using facial videos},
  author={Sun, Jian and Dodge, Hiroko H and Mahoor, Mohammad H},
  journal={Expert Systems with Applications},
  volume={238},
  pages={121929},
  year={2024},
  publisher={Elsevier}
}

Recommended citation: Sun, J., Dodge, H. H., & Mahoor, M. H. (2024). MC-ViViT: Multi-branch classifier-ViViT to detect mild cognitive impairment in older adults using facial videos. Expert Systems with Applications, 238, 121929.
Download Paper | Download Slides