GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals

Qianyun Lin; Annie Cheung

doi:10.63313/JCSFT.9083

Authors

Qianyun Lin Viridien Group, Houston, USA Author
Annie Cheung University of Michigan, Ann Arbor, MI, USA Author

DOI:

https://doi.org/10.63313/JCSFT.9083

Keywords:

Multimodal deep learning, Gastrointestinal disease, Endoscopy, Transformer, Physiological signals, Medical image analysis

Abstract

Accurate and efficient diagnosis of gastrointestinal (GI) diseases is essential for timely clinical decision-making, yet traditional diagnostic workflows rely heavily on manual interpretation of endoscopic images and disconnected physiological measurements. These limitations often lead to variability in diagnosis and increased clinician workload. To address these challenges, we propose GIMeT (GastroIntestinal Multimodal Transformer), a novel deep learning framework that integrates endoscopic imaging, physiological time-series data, and optional clinical text information for automated GI disease classification and lesion segmentation. GIMeT consists of three modality-specific encoders—a CNN-based image encoder, a temporal encoder for physiological signals, and a BERT-based text encoder—followed by a Cross-Modal Transformer (CMT) to achieve fine-grained feature alignment and joint representation learning. A FiLM-conditioned U-Net decoder further enhances pixel-level lesion segmentation using multimodal cues. We evaluate GIMeT on a multi-center dataset comprising endoscopic images, electro-physiological measurements, and symptom descriptions. Experimental results show that GIMeT consistently surpasses all unimodal and multimodal baselines across both classification and segmentation tasks. The model achieves improvements of approximately 3-4% in accuracy and F1-score, and a 3% gain in lesion Dice coefficient. Ablation studies further validate the contribution of multimodal fusion and cross-modal attention, demonstrating GIMeT’s robustness and strong potential for computer-assisted gastrointestinal diagnosis.

References

[1] Hirasawa, Toshiaki, et al. "Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images." Gastric cancer 21.4 (2018): 653-660.

[2] Urban, Gregor, et al. "Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy." Gastroenterology 155.4 (2018): 1069-1078.

[3] El-Gammal E M, El-Shafai W, Taha T E, et al. A survey of artificial intelligence models for wireless capsule endoscopy videos for superior automatic diagnosis: problems and solutions[J]. Multimedia Tools and Applications, 2025, 84(33): 40555-40589.

[4] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.

[5] Liu J, Cen X, Yi C, et al. Challenges in AI-driven biomedical multimodal data fusion and analysis[J]. Genomics, proteomics & bioinformatics, 2025, 23(1): qzaf011.

[6] Rajpurkar, Pranav, et al. "Cardiologist-level arrhythmia detection with convolutional neural networks." arXiv preprint arXiv:1707.01836 (2017).

[7] Geng D, Wang Q, Zheng W, et al. ECG-and HRV-Based Hybrid Architecture—Early Detection of Alzheimer’s Disease and Mild Cognitive Impairment[J]. Applied Sciences, 2025, 15(23): 12555.

[8] Geldof, H., and E. J. Van der Schee. "Electrogastrography: clinical applications." Scandinavian Journal of Gastroenterology 24.sup171 (1989): 75-82.

[9] Hailemariam M A, Dufera B D, Ahmed H Y. Lung Disease Detection Using Multimodal Machine Learning with Structured Clinical Data and Medical Images[J]. 2026.

[10] Adahada E, Sassoon I, Hone K, et al. A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports[J]. arXiv preprint arXiv:2508.13796, 2025.

[11] Du H, Dong Z, Wu L, et al. A deep-learning based system using multi-modal data for diagnosing gastric neoplasms in real-time (with video)[J]. Gastric Cancer, 2023, 26(2): 275-285.

[12] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

[13] Tsai Y H H, Bai S, Liang P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]//Proceedings of the 57th annual meeting of the association for computational linguistics. 2019: 6558-6569.

[14] Wang Y. Zynq SoC-Based Acceleration of Retinal Blood Vessel Diameter Measurement[J]. Archives of Advanced Engineering Science, 2025: 1-9.

[15] Yao Y, Li Y, Zheng S, et al. Deep Learning Models for Multi-class Pneumonia Detection in Chest X-rays: A Comparative Study of VGG16, MobileNet, and ResNet152[J]. Computer Simulation in Application, 2025, 3(1): 66-73.

[16] Xu S, Jiang L, Gu B. Design and Validation of a Smart Neuromorphic System Architecture for Algorithmic Trading[C]//Proceedings of the 2nd International Symposium on Integrated Circuit Design and Integrated Systems. 2025: 127-136.

[17] Yao Y, Li Y, Zheng S, et al. MSA-TransUNet: A Multi-Scale Attention Enhanced Transformer-UNet Architecture for Accurate Vessel Segmentation and Visualization in Medical Imaging[J]. Journal of Medicine and Life Sciences, 2025, 1(4): 48-57.

[18] Ou Y, Zhu X. Intelligent Signal Analysis and Fault Diagnosis System for Integrated Circuits Using Multi-Model Machine Learning[C]//Proceedings of the 2nd International Symposium on Integrated Circuit Design and Integrated Systems. 2025: 251-256.

[19] Wang Y. Low-power design of advanced image processing algorithms under fpga in real-time applications[C]//2024 IEEE 4th International Conference on Power, Electronics and Computer Applications (ICPECA). IEEE, 2024: 1080-1084.

[20] Sun Q, Zhao X, Lin X. Design of a Hardware-Software Co-designed Real-Time Machine Learning System for Big Data Streams[C]//Proceedings of the 2nd International Symposium on Integrated Circuit Design and Integrated Systems. 2025: 265-271.