zuraiqi/neural-organ-transplantation-checkpoints

🧬 Neural Organ Transplantation (NOT) - Pre-trained Donor Checkpoints

Pre-trained donor layer checkpoints for Neural Organ Transplantation (NOT) — a checkpoint-based modular adaptation framework for transformer models.

Overview

Neural Organ Transplantation enables trained transformer layers to function as reusable, transferable checkpoints for domain adaptation. These checkpoints can be transplanted into compatible recipient models without access to the original training data.

Available Checkpoints

Model	Folder	Size	Layers	Perplexity	Best Position
GPT-2 (124M)	gpt2/	81 MB	4, 5, 6	17.33	1
TinyLlama (1.1B)	tinyllama/	252 MB	7, 8, 9	54.15	3
GPT-OSS (20B)	gptoss/	4.6 GB	8, 9, 10	34.56	4

Performance vs Baselines

Model	Donor (Ours)	LoRA	Full FT	Improvement
GPT-2	17.33	668.40	1352.05	38.6× vs LoRA
TinyLlama	54.15	460.41	788.36	8.5× vs LoRA
GPT-OSS	34.56	98.37	—	2.8× vs LoRA

Key Finding: Donor transplantation achieves 2.8–38.6× better perplexity than LoRA while training 2–28× faster on decoder-only models.

Quick Start

Download and Use a Checkpoint

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM

# Download GPT-2 donor checkpoint
checkpoint_path = hf_hub_download(
    repo_id="zuraiqi/neural-organ-transplantation-checkpoints",
    filename="gpt2/donor_layers.pt"
)

# Load checkpoint
checkpoint = torch.load(checkpoint_path, map_location="cpu")

# Load base model
model = AutoModelForCausalLM.from_pretrained("gpt2")

# See full transplantation code at GitHub repo

Download TinyLlama Checkpoint

from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="zuraiqi/neural-organ-transplantation-checkpoints",
    filename="tinyllama/donor_layers.pt"
)

Download GPT-OSS Checkpoint

from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="zuraiqi/neural-organ-transplantation-checkpoints",
    filename="gptoss/donor_layers.pt"
)

Checkpoint Contents

Each folder contains:

• donor_layers.pt — Trained donor layer weights
• donor_metadata.json — Training metadata (layer indices, perplexity, etc.)

Metadata Example (GPT-2)

{
  "layer_indices": [4, 5, 6],
  "perplexity": 17.33,
  "train_time": 12.83,
  "method": "standalone_wrapper_training"
}

Training Details

• Dataset: WikiText (1,000 samples)
• Epochs: 5
• Optimizer: AdamW (lr=1e-4)
• Method: Standalone wrapper training with frozen embeddings

Usage Guidelines

1. Best Insertion Position: Use early positions (first quarter of layers)
2. Bridge Fine-tuning: Brief recovery fine-tuning improves results
3. Architecture: Works best with decoder-only models (GPT, LLaMA families)

Links

• 📄 Paper: Neural Organ Transplantation (NOT): Checkpoint-Based Modular Adaptation for Transformer Models
• 💻 Code: github.com/zuraiqi/neural-organ-transplant

Citation

@article{alzuraiqi2026neural, title={Neural Organ Transplantation (NOT): Checkpoint-Based Modular Adaptation for Transformer Models}, author={Al-Zuraiqi, Ahmad}, journal={arXiv preprint arXiv:2601.13580}, year={2026} }

License

MIT License