Tri-Modal Bone Cancer Intelligence: Late-Fusion and Cross-Modal Attention over Radiographs, WSIs, and Omics

• M.C. Shanker

  Department of Biomedical Engineering, Vel Tech Multi Tech Dr. Rangarajan, Dr. Sakunthala Engineering College, Chennai, Tamil Nadu, India

• V. Gokula Krishnan

  Department of CSE, Easwari Engineering College, Chennai, Tamil Nadu, India

• D. Arul Kumar

  Department of ECE, Panimalar Engineering College, Chennai, Tamil Nadu, India

• M. Bhuvaneswari

  Department of CSE-CS, Easwari Engineering College, Chennai, Tamil Nadu, India

• K. Sathyamoorthy

  Department of CSE, Vel Tech Rangarajan Dr Sagunthala R&D Institute of Science and Technology, Avadi, Tamil Nadu, India

• B. Prathusha Laxmi

  Department of AIDS, R.M.K. College of Engineering and Technology, Kavaraipettai, Tamil Nadu, India

Bone cancers have diverse radiologic and histopathologic characteristics, and visual ambiguity frequently constrains single-modality AI. To create a tri-modal system that learns from (i) radiographs in the Bone Cancer Detection (Kaggle) dataset, (ii) whole-slide images (WSIs) with weak labels through multiple-instance learning (MIL), and (iii) RNA-seq±mutation profiles (TARGET-OS) represented by a compact variational/Multi Layer Perceptron (MLP) bottleneck. To align modality latents with Maximum Mean Discrepancy (MMD) and Information Noise-Contrastive Estimation (InfoNCE) contrast, and then to employ reliability-aware late-fusion with optional cross-modal co-attention to combine them. Temperature scaling adjusts the chances. The fused model gets Accuracy of 0.926±0.016, Macro-averaged F1 score of 0.914±0.018, Area Under the Receiver Operating Characteristic Curve (AUROC)of 0.965±0.010, Area Under the Precision–Recall Curve (AUPRC)of 0.958±0.011, Brier score of 0.067±0.008, and Expected Calibration Error (ECE)of 0.018±0.006 on tests that were held out. Single streams do worse (Radiograph AUROC 0.940; WSI-MIL 0.918; Omics 0.902), while two-stream combinations close much of the gap. Ablations show that alignment and co-attention are quite important (for example, taking out MMD lowers AUROC by 0.012 and raises ECE by 0.006). Robustness experiments demonstrate elegant decline in the presence of X-ray blur/jitter, stain jitter, and omics batch shifts; the external-site AUROC remains robust at 0.958 (fused). An adjusted operating point θ* gives Coverage 94.1%, Sensitivity 0.936, Specificity 0.903, Positive Predictive Value (PPV) 0.907, Negative Predictive Value (NPV) 0.933, and the best utility when costs are high for false negatives. This study shows that combining morphologic and molecular signals with an awareness of uncertainty leads to accurate, well-calibrated, and more generalizable predictions of bone cancer. This is true even when only radiographs are available.

[1] Sampath K, Rajagopal S, Chintanpalli A. A comparative analysis of CNN-based deep learning architectures for early diagnosis of bone cancer using CT images. Scientific Reports 2024; 14(1): 2144.

[2] Aarthy R, Muthupriya V, Balaji GN. Detection of bone cancer based on a four-phase framework generative deep belief neural network in deep learning. Alexandria Engineering Journal 2024; 109: 394-407.

[3] Shrivastava A, Nag MK. Enhancing bone cancer diagnosis through image extraction and machine learning: a state-of-the-art approach. Surgical Innovation 2024; 31(1): 58-70.

[4] Murugan S, Patil SS, Reddy RK, KN SK. A recent survey on bone cancer detection using deep learning techniques. In: Proc. of the Second International Conference on Advances in Information Technology (ICAIT). IEEE 2024;pp. 1-6.

[5] Gunanithi S, Ilavarasan S, Karthika RN. Revolutionizing osteosarcoma diagnosis: a comparative analysis of deep learning models for precise bone cancer detection using multi-modal medical imaging. In: Int. Conf. Summit on Artificial Intelligence. Singapore: Springer2024;pp. 197-207.

[6] Deng S, Huang Y, Li C, Qian J, Wang X. Auxiliary diagnosis of primary bone tumors based on machine learning model. Journal of Bone Oncology 2024; 49: 100648.

[7] Zhao X, Dong YH, Xu LY, Shen YY, Qin G, Zhang ZB. Deep bone oncology diagnostics: computed tomography based machine learning for detection of bone tumors from breast cancer metastasis. Journal of Bone Oncology 2024; 48: 100638.

[8] Baswaraju S, Thirumalraj A, Manjunatha B. Unlocking the potential of deep learning in knee bone cancer diagnosis using MSCSA-Net segmentation and MLGC-LTNet classification. In: Sustainable Development Using Private AI. CRC Press2024;pp. 190-213.

[9] Singh J, Patel P, Ingole BS, Inaganti R, Ramineni V, Krishnappa MS, et al. Advanced computational methods for pelvic bone cancer detection: efficacy comparison of convolutional neural networks. In: Proc. of IEEE 17th Int. Symp. on Embedded Multicore/Many-core Systems-on-Chip (MCSoC). IEEE2024;pp. 287-293.

[10] Srividya K, Reddy GV, Bakki V, Adilakshmi T. AI-based bone cancer detection using image processing and CNN. In: Int. Conf. on Cognitive Computing and Cyber Physical Systems. Cham: Springer Nature Switzerland2024;pp. 283-304.

[11] Wang H, He Y, Wan L, Li C, Li Z, Li Z, et al. Deep learning models in classifying primary bone tumors and bone infections based on radiographs. NPJ Precision Oncology 2025; 9(1): 72.

[12] Xie W, Wang X, Liu M, Mai L, Shangguan H, Pan X, et al. A novel two-step classification approach for differentiating bone metastases from benign bone lesions in SPECT/CT imaging. Academic Radiology 2025.

[13] Curto-Vilalta A, Schlossmacher B, Valle C, Gersing A, Neumann J, von Eisenhart-Rothe R, et al. Semi-supervised label generation for 3D multi-modal MRI bone tumor segmentation. Journal of Imaging Informatics in Medicine 2025.

[14] Wang K, Han Y, Ye Y, Chen Y, Zhu D, Huang Y, et al. Mixed reality infrastructure based on deep learning medical image segmentation and 3D visualization for bone tumors using DCU-Net. Journal of Bone Oncology 2025; 50: 100654

[15] Li B, Xu D, Lin H, Wu R, Wu S, Shao J, et al. Domain adaptive detection framework for multi-center bone tumor detection on radiographs. Computerized Medical Imaging and Graphics 2025;123: 102522.

[16] Kaggle. Bone cancer detection dataset. Available from: https://www.kaggle.com/datasets/ziya07/bone-cancer-detection-dataset

• PDF
• Suppl

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.