Medistillation implements and compares multiple knowledge-distillation methods for medical question answering, focusing on transferring capability from large teachers (e.g. Meditron-scale) into compact students (e.g. Qwen2-1.5B).
The PyTorch/Hugging Face codebase covers methods such as logit KD, adaptive KD, CoT distillation, FitNets, attention transfer, and RL-style variants, with evaluation across medical QA benchmarks and fidelity metrics.