Mitigating Shortcut Learning in
Brain Tumour MRI Classification

Pathology-guided ResNet50V2–RViT hybrids for evidence-aware four-class classification

Author

Affiliation

Department of Computer Science, University of Hertfordshire

Published

Aug. 13, 2026

DOI

10.5281/zenodo.21903347

PDF

TL;DR

High held-out accuracy did not settle the reliability question. Across four ResNet50V2–RViT variants, Ablation B was the strongest classifier, reaching 99.22% test accuracy, 0.9920 macro-F1, and 10 errors in 1,284 test images. Yet the stronger guided model, Hybrid B, produced the most tumour-centred Grad-CAM++ evidence in the three qualitative cases reported in the FYP.

The central result is therefore a trade-off, not a victory claim: pathology guidance changed where the system appeared to look more clearly than it improved what class it predicted.

Study factVerified value
Raw images discovered7,023
Exact duplicates removed297
Unique images retained6,726
Held-out test images1,284
Best test accuracy99.22%
Best macro-F10.9920
Fewest test errors10
Evidence boundary

This is a BSc Artificial Intelligence project, not a peer-reviewed clinical study or a certified medical device. It contains no external hospital validation, tumour-mask evaluation, or formal calibration analysis. The page makes no definitive state-of-the-art or clinical-generalisation claim.

Why Shortcut Learning Matters

A brain MRI classifier can be accurate for the wrong reason. Instead of relying on pathology, a model may exploit background, skull boundaries, annotations, acquisition patterns, or other dataset-specific regularities. This is the shortcut-learning problem: benchmark performance can remain high even when the learned evidence is brittle or clinically irrelevant (Geirhos et al., 2020; Lin et al., 2024).

The risk is particularly important in medical imaging because seemingly minor differences in preprocessing, patient populations, scanners, and dataset construction can create strong non-causal signals. Methodological reviews therefore emphasise leakage control, transparent split policies, held-out evaluation, and external validation (Mongan et al., 2020; Varoquaux & Cheplygina, 2022).

Can pathology-guided CNN–Transformer hybrids maintain strong four-class performance while shifting the evidence pathway toward tumour-related regions?

Project research question

What was tested

The study compares two guided systems with two same-family ablations:

ModelCNN pathwayTransformer pathwayGuidance intervention
Hybrid AResNet50V2 descriptor49 CNN feature tokensPFD mask guides transformer-facing tokens
Ablation AMatched A-family backboneSame tokenisationPFD–GSTE removed
Hybrid BPFD-gated ResNet50V2 descriptor196 raw-image patch tokens before optional poolingPFD gates CNN evidence and guides patch tokens
Ablation BMatched B-family backboneSame tokenisationPFD–GSTE removed

What the work contributes

  1. A leakage-aware benchmark built after byte-level SHA1 duplicate removal.
  2. Two pathology-guided ResNet50V2–RViT designs with explicit same-family ablations.
  3. Branch-specific explainability using Grad-CAM++ and Attention Rollout.
  4. MC Dropout analysis over 20 stochastic passes.
  5. Reproducible training, evaluation, visualisation, checkpoints, a browser demo, and reusable PFD–GSTE modules.

These are contributions within the project’s experimental setting. They do not establish that shortcut learning has been eliminated.

  1. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665–673. 10.1038/s42256-020-00257-z
  2. Lin, M., Weng, N., Mikolaj, K., Bashir, Z., Svendsen, M. B. S., Tolsgaard, M. G., Christensen, A. N., & Feragen, A. (2024). Shortcut Learning in Medical Image Segmentation. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, 623–633. 10.1007/978-3-031-72111-3_59
  3. Mongan, J., Moy, L., & Kahn, C. E. (2020). Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiology: Artificial Intelligence, 2(2), e200029. 10.1148/ryai.2020200029
  4. Varoquaux, G., & Cheplygina, V. (2022). Machine learning for medical imaging: methodological failures and recommendations for the future. Npj Digital Medicine, 5, 48. 10.1038/s41746-022-00592-y

The PFD–GSTE Method

The model family combines a ResNet50V2 convolutional branch with a rotation-aware Vision Transformer branch. Residual CNNs provide strong local feature extraction (He et al., 2016), while Vision Transformers model long-range relationships through token interactions (Dosovitskiy et al., 2021). The transformer design follows rotation-aware ideas where applicable from Krishnan et al. (Krishnan et al., 2024).

The project adds two experimental modules:

The terminology is project-specific. The design is motivated by pathology-emphasis, hybrid CNN–Transformer, and token-selection literature (Rao et al., 2021; Xia et al., 2020); it should not be read as a claim that an established clinical concept has been reproduced.

Stage 1: soft spatial gating

Let F∈RC×H×WF \in \mathbb{R}^{C \times H \times W} denote the CNN feature map and let g(⋅)g(\cdot) be a learned mask head. PFD computes

m=σ(g(F)),F~=F⊙m,m = \sigma(g(F)), \qquad \widetilde{F} = F \odot m,

where m∈[0,1]1×H×Wm \in [0,1]^{1 \times H \times W} is a differentiable spatial gate and ⊙\odot denotes broadcast element-wise multiplication.

Plain-language interpretation: the network learns a continuous spatial preference map rather than receiving a manual tumour segmentation mask.

Stage 2: mean-normalised token guidance

The mask is resized to the token grid and normalised to preserve its mean scale:

α=Resize⁡(m)mean⁡(Resize⁡(m))+ε,T~=T⊙α.\alpha = \frac{\operatorname{Resize}(m)} {\operatorname{mean}(\operatorname{Resize}(m)) + \varepsilon}, \qquad \widetilde{T} = T \odot \alpha.

Variant A applies this mechanism to a fixed 7×77\times7 CNN feature grid, giving 49 tokens. Variant B begins with 14×1414\times14 raw-image patches, giving 196 tokens, and can use concentration-aware weighted pooling toward a minimum 7×77\times7 spatial side.

Stage 3: late fusion

The CNN and transformer summaries remain separate until the classifier:

z=[zcnn;zvit],h=ReLU⁡(Wfz+bf),y^=Wo ⁣(Dropout⁡(h))+bo.z = [z_{\mathrm{cnn}};z_{\mathrm{vit}}], \qquad h = \operatorname{ReLU}(W_f z + b_f), \qquad \widehat{y} = W_o\!\left(\operatorname{Dropout}(h)\right)+b_o.

This equation is the late-fusion formulation reported in the FYP.

Two Guided Variants

Hybrid A architecture showing ResNet50V2, PFD guidance, 49 CNN feature tokens, RViT processing, and late fusion.
Hybrid A. The learned mask guides the transformer-facing feature tokens; the CNN descriptor remains ungated.
Hybrid B architecture showing PFD-gated CNN evidence, 196 raw-image patch tokens, GSTE guidance, optional dynamic weighted pooling, RViT processing, and late fusion.
Hybrid B. The stronger intervention gates the CNN descriptor and guides raw-image patch tokens.
PropertyHybrid AHybrid B
Initial transformer tokens49 CNN feature tokens196 raw-image patch tokens
CNN descriptorUngatedPFD-gated
Guidance reachTransformer-facing pathwayCNN and transformer pathways
Token countFixedOptional concentration-aware pooling
Trainable parameters26.68M26.58M
Interpretation discipline

Because A and B differ in token source and guidance reach, cross-family comparisons are not clean estimates of one isolated design change. The strongest causal comparisons are Hybrid A versus Ablation A and Hybrid B versus Ablation B.

  1. Dosovitskiy, A., Beyer, L., Kolesnikov, A., & others. (2021). An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations. https://openreview.net/forum?id=YicbFdNTTy
  2. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity Mappings in Deep Residual Networks. Computer Vision – ECCV 2016, 630–645. 10.1007/978-3-319-46493-0_38
  3. Krishnan, P. T., Krishnadoss, P., Khandelwal, M., Gupta, D., Nihaal, A., & Kumar, T. S. (2024). Enhancing brain tumor detection in MRI with a rotation invariant Vision Transformer. Frontiers in Neuroinformatics, 18, 1414925. 10.3389/fninf.2024.1414925
  4. Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., & Hsieh, C.-J. (2021). DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. Advances in Neural Information Processing Systems, 34. https://arxiv.org/abs/2106.02034
  5. Xia, T., Chartsias, A., & Tsaftaris, S. A. (2020). Pseudo-healthy synthesis with pathology disentanglement and adversarial learning. Medical Image Analysis, 64, 101719. 10.1016/j.media.2020.101719

Leakage-Safe Benchmark

The experiment uses the public four-class Brain Tumor MRI Dataset (Nickparvar, 2021). Before training, every discovered file was read as bytes and hashed with SHA1. Exact duplicate hashes were grouped across the original Kaggle directories.

When an exact duplicate crossed the original Training/Testing boundary, the Testing copy was retained and the Training copy removed. This policy protects the held-out boundary instead of allowing visually identical files to appear on both sides of evaluation.

7,023 raw files  −  297 exact duplicates  =  6,726 unique images.7{,}023\ \text{raw files} \; - \; 297\ \text{exact duplicates} \; = \; 6{,}726\ \text{unique images}.

Fixed split composition

The cleaned Kaggle Training partition was divided into stratified train and validation subsets using an 80/20 split. The cleaned Kaggle Testing partition remained untouched as the held-out test set.

SplitGliomaMeningiomaPituitaryNo tumourTotal
Train1,0571,0641,1521,0804,353
Validation2642672882701,089
Test2993043003811,284
Bar charts showing class counts in the final training, validation, and held-out test splits.
Final data split. Class composition after exact deduplication, an 80/20 stratified train-validation split, and preservation of Kaggle Testing as the held-out test set.
Original Kaggle training and testing class counts before duplicate removal.
Original Kaggle split composition.
Class distribution after exact duplicate removal.
Class balance after deduplication.
Representative brain MRI images inspected during the rule-based raw-image audit.
Representative raw-image audit panel. The report recorded zero unreadable, too-dark, too-bright, or low-contrast files under its rule-based checks.

Training and Evaluation Protocol

All four variants used the same core optimisation policy:

ComponentSetting
OptimiserAdamW
CNN learning rate10−410^{-4}
Transformer and fusion learning rate5×10−45\times10^{-4}
Weight decay0.01
ObjectiveCross-entropy with label smoothing 0.05
ScheduleCosine annealing, ηmin⁡=10−6\eta_{\min}=10^{-6}
Batch size32
Maximum epochs100
CNN warm-upFrozen for first 5 epochs
Early stoppingValidation macro-F1, patience 10
Gradient clipping1.0
Random seed42

Checkpoint selection used validation macro-F1 only. Each chosen checkpoint was then evaluated once on the fixed 1,284-image test set.

Training and validation loss, accuracy, and macro-F1 curves for Hybrid A and Hybrid B.
Guided-model learning curves. Hybrid B reached its best validation macro-F1 at epoch 14.
Training and validation loss, accuracy, and macro-F1 curves for Ablation A and Ablation B.
Ablation learning curves. Late fluctuations support validation-based early stopping rather than final-epoch selection.
Reproducibility caveat

B-family training used drop_last=True, whereas A-family training used drop_last=False. cuDNN benchmarking was enabled and strict deterministic execution was not enforced. Exact reruns may therefore vary, and within-family comparisons are the cleanest.

  1. Nickparvar, M. (2021). Brain Tumor MRI Dataset. Kaggle dataset. https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset

Results

All four models performed strongly on the fixed held-out set. Guidance, however, did not produce the best classification score.

Held-out performance explorer
Switch metrics to compare all four variants. Percentage axes are intentionally zoomed and labelled; the error view uses the absolute count from 1,284 test images.
ModelParamsAccuracyMacro-F1LossSpecificityκ\kappaMCCErrorsBest epoch
Hybrid A26.68M0.98750.98750.08030.99590.98330.98331643
Hybrid B26.58M0.98520.98490.07530.99520.98020.98031914
Ablation A26.68M0.98750.98730.08470.99590.98330.98341630
Ablation B26.58M0.99220.99200.06680.99750.98960.98961042

Within-family ablation result

Key finding

Ablation B is the strongest classifier in this evaluation. Any account that presents Hybrid B or PFD–GSTE as the accuracy winner would contradict the reported experiment.

Held-out confusion matrices for Hybrid A, Hybrid B, Ablation A, and Ablation B.
Held-out confusion matrices. Report order: Hybrid A, Hybrid B, Ablation A, Ablation B. Remaining mistakes primarily occur at tumour-subtype boundaries.

Benchmark context, not a state-of-the-art claim

The report found that this model family compares favourably with the reviewed four-class studies in its chosen benchmark setting. That numerical comparison is not equivalent to a definitive state-of-the-art result: prior papers differ in preprocessing, duplicate handling, split policy, augmentation, checkpoint selection, and reporting detail. External validation and repeated-seed analysis would be required before making stronger generalisation claims (Varoquaux & Cheplygina, 2022).

  1. Varoquaux, G., & Cheplygina, V. (2022). Machine learning for medical imaging: methodological failures and recommendations for the future. Npj Digital Medicine, 5, 48. 10.1038/s41746-022-00592-y

Evidence Focus

Classification accuracy and localisation quality answer different questions. The report therefore inspects both the CNN branch with Grad-CAM++ (Chattopadhyay et al., 2018) and the transformer branch with Attention Rollout (Abnar & Zuidema, 2020).

This analysis is deliberately qualitative. Saliency maps can be unstable or visually persuasive without being faithful, so they require cautious interpretation and stronger controls than a few attractive examples (Adebayo et al., 2018; Arun et al., 2021).

Classification–localisation trade-off
Horizontal position is held-out test accuracy. Vertical position is the number of Grad-CAM++ judgements marked “Correct” in the three selected report cases. The vertical axis is qualitative and n=3—not a segmentation score.
CaseHybrid AHybrid BAblation AAblation B
Correct meningiomaGC++ wrong; rollout partially correctGC++ correct; rollout wrongGC++ wrong; rollout partially correctGC++ correct; rollout wrong
Challenging gliomaGC++ wrong; rollout wrongGC++ correct; rollout partially wrongGC++ correct; rollout partially wrongGC++ partially wrong; rollout partially wrong
Pituitary → meningioma errorGC++ wrong; rollout wrongGC++ correct; rollout partially correctGC++ wrong; rollout partially correctGC++ wrong; rollout partially correct

By this report-level visual judgement, Hybrid B recorded 3/3 correct Grad-CAM++ localisations; Hybrid A recorded 0/3; Ablation A and Ablation B each recorded 1/3.

Grad-CAM++ and Attention Rollout comparison for a correctly classified meningioma across all four model variants.
Correct meningioma. All variants classified the image correctly, but the reported evidence alignment differed across branches.
Grad-CAM++ and Attention Rollout comparison for a challenging glioma across all four model variants.
Challenging glioma. Hybrid B and Ablation A were judged tumour-centred under Grad-CAM++; rollout remained partial or wrong.
Grad-CAM++ and Attention Rollout comparison for a pituitary MRI misclassified as meningioma by every model.
Shared subtype error. Every variant predicted meningioma instead of pituitary. Hybrid B still localised the visible region under Grad-CAM++, demonstrating that evidence location and class correctness can diverge.
What this supports—and what it does not

The three selected cases provide qualitative support that the stronger Hybrid B intervention can alter evidence focus. They do not demonstrate full-dataset localisation performance, causal faithfulness, or elimination of shortcut learning. There are no ground-truth tumour masks, Dice scores, IoU scores, or pointing-game results.

Uncertainty Analysis

MC Dropout retained dropout at inference while keeping BatchNorm layers in evaluation mode. Predictive statistics were computed over 20 stochastic forward passes (Gal & Ghahramani, 2016).

In the pituitary image misclassified as meningioma by all four models, confidence remained high:

Confidence on the shared pituitary → meningioma error
Mean confidence over 20 MC Dropout passes. The axis is zoomed to 90–100% and must not be interpreted as a calibration curve.

The example exposes an important failure mode: stochastic uncertainty did not prevent a wrong-but-confident subtype prediction.

Calibration gap

Expected calibration error, Brier score, reliability diagrams, and external calibration were not reported. MC Dropout variance alone does not demonstrate clinically calibrated confidence (Rajaraman et al., 2022).

  1. Abnar, S., & Zuidema, W. (2020). Quantifying Attention Flow in Transformers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4190–4197. 10.18653/v1/2020.acl-main.385
  2. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity Checks for Saliency Maps. Advances in Neural Information Processing Systems, 31. https://arxiv.org/abs/1810.03292
  3. Arun, N., Gaw, N., Singh, P., & others. (2021). Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging. Radiology: Artificial Intelligence, 3(6), e200267. 10.1148/ryai.2021200267
  4. Chattopadhyay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018). Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. 2018 IEEE Winter Conference on Applications of Computer Vision, 839–847. 10.1109/WACV.2018.00097
  5. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on Machine Learning, 48, 1050–1059. https://proceedings.mlr.press/v48/gal16.html
  6. Rajaraman, S., Ganesan, P., & Antani, S. (2022). Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks. PLOS ONE, 17(1), e0262838. 10.1371/journal.pone.0262838

Limitations

The study’s strongest value is that its boundaries are inspectable.

  1. No external validation. One aggregated Kaggle benchmark cannot establish robustness across hospitals, scanners, sequences, patient populations, or clinical workflows.
  2. No ground-truth tumour masks. Localisation is qualitative and case-based; there is no full-test-set overlap metric.
  3. No formal calibration analysis. Twenty MC Dropout passes provide stochastic summaries, not proof of calibrated uncertainty.
  4. Architecture and loader differences remain. A and B differ in tokenisation and guidance reach, while their loaders differ in drop_last.
  5. Single 2D slices. Tumour subtype discrimination may require multi-slice, multi-sequence, volumetric, or histopathological context.
  6. Single seeded training run per variant. Repeated seeds and confidence intervals are needed to separate stable effects from optimisation variance.
  7. No clinical claim. The system is an experimental research prototype and must not be used for diagnosis, patient management, or treatment.
Scientific status

The results support a benchmark-level conclusion: guidance may improve qualitative evidence focus without improving the best classification score. They do not establish clinical utility, universal shortcut robustness, or superiority over all published methods.

Evidence Readiness

The public implementation already supports a strong reproducibility record, but a polished website is not a substitute for missing experiments. The table below separates what is verified from the evidence required before each claim can be strengthened.

DimensionVerified nowCompletion gate
Research integrityExact reported values, negative ablation outcomes, authorship, and limitations are preservedKeep all claims traceable to report, code, or released artefacts
Open artefactsPublic code, checkpoints, model cards, PyPI package, report, and machine-readable article dataPublish this article repository with a versioned release and build check
Statistical robustnessOne seeded training run per variantRun at least five matched seeds per variant; report mean, standard deviation, 95% confidence intervals, and paired within-family deltas
Fair comparisonGuided variants have same-family ablationsUse the same loader policy, preprocessing manifest, checkpoint rule, and deterministic settings in the repeated-seed rerun
CalibrationTwenty-pass MC Dropout summaries for selected examplesArchive validation and test probabilities; report NLL, multiclass Brier score, ECE, reliability diagrams, and validation-fitted temperature scaling
External generalisationA five-image external meningioma sample exists only as a qualitative demoEvaluate a provenance-audited external cohort with compatible labels and no overlap with the development benchmark
XAI validationGrad-CAM++ and Attention Rollout are reviewed on three selected casesEvaluate a preregistered masked subset with Dice, IoU, pointing-game or localisation-error metrics plus saliency sanity checks
Clinical evidenceNone claimedIndependent clinical governance, representative cohorts, and prospective validation would be required before any clinical claim
No empty-result substitution

The completion gates are an experimental roadmap, not unreported results. Confidence intervals, calibration scores, external-cohort performance, and mask-based localisation metrics will be added only after their underlying predictions, annotations, and evaluation artefacts are released.

Reproducibility

The companion implementation records the preprocessing audit, fixed split construction, four training configurations, evaluation scripts, model-specific XAI procedures, checkpoints, and reusable guidance modules.

ResourceAccess
Model and experiment codeGitHub repository
Browser demonstrationHugging Face Space
Reusable modulesPFD–GSTE on PyPI
Machine-readable website dataapp/src/content/assets/data/ in this article repository
Evidence-upgrade protocolVALIDATION_PLAN.md in the article repository
pip install pfd-gste

Conclusion

PFD–GSTE partially supports the project hypothesis. The stronger guidance pathway changed qualitative evidence focus most clearly in Hybrid B, but it did not improve the best held-out classification score. Ablation B remained the strongest classifier; Hybrid B remained the strongest qualitative Grad-CAM++ localiser among the three reported cases.

The next decisive experiment is external, multi-centre validation with tumour masks, quantitative localisation, calibration testing, repeated seeds, and multi-sequence or volumetric MRI. Until then, accuracy and evidence quality should be reported as complementary—not interchangeable—properties.

Supervision and project context

This work was completed as a BSc Artificial Intelligence project in the Department of Computer Science, University of Hertfordshire, under the supervision of Dr Kheng Lee Koay. The report is dated 26 March 2026.