SST: Structural Self-Teaching

Getting the pieces—and their relationships—right.

Objects, attributes, and spatial relations need to work together.
Explore the same prompt across the base model, LoRA fine-tuning, and SST.

BLIP-3o 4B

“a penguin and three pink cows”

BLIP-3o 4B SST output for a penguin and three pink cows — SST (Ours)

The base model entangles the requested animals. SST separates the penguin from the pink cows.

Representative examples from the paper. Click an image to view it at full size.

Abstract

Unified multimodal models can localize concepts in images even when they struggle to generate those concepts together. We introduce Structural Self-Teaching (SST), which uses the model’s understanding path as an internal source of phrase-level structural supervision. Phrase-conditioned attention maps become soft support masks, used in two ways: phrase-local contrastive alignment aligns generated and reference region representations, while a structural scaffold provides spatial guidance during training. The scaffold is progressively dropped and removed at inference. Across BLIP-3o, Show-o2, and BAGEL, SST improves compositional fidelity over matched LoRA fine-tuning, using no external structural annotations and preserving the original generation interface. Our current instantiation targets diffusion and AR–diffusion hybrid models with spatially organized intermediate states.

Models can localize concepts they struggle to generate.

The understanding path provides spatially grounded phrase information.
SST reuses this internal structure to supervise the generation path.

Two compositional prompts with a reference image, phrase-wise attention maps, base model generation failures, and SST outputs — Reference images reveal phrase-wise grounding in the understanding path. The base model can miss objects or bind attributes incorrectly when generating from the same text. Reference images and supports are used for diagnosis or training; inference uses the original text-to-image interface.

Explore the diagnostic across four backbones

Compositional scores as prompt atomicity increases

BLIP-3o 4B: oracle-image understanding scores remain relatively stable as prompt-only generation scores decline — BLIP-3o 4B

BLIP-3o 8B understanding and generation diagnostic — BLIP-3o 8B

Show-o2 7B understanding and generation diagnostic — Show-o2 7B

BAGEL understanding and generation diagnostic — BAGEL

Phrase support quality against pseudo-references

BLIP-3o 4B AIR and pointing accuracy across atomicity

BLIP-3o 8B AIR and pointing accuracy across atomicity

Show-o2 AIR and pointing accuracy across atomicity

BAGEL AIR and pointing accuracy across atomicity

Understanding is probed on oracle images; generation is evaluated from text prompts alone. These are different diagnostic conditions. AIR and pointing accuracy use Grounded-SAM pseudo-references and are not comparable to GM as the same metric.

One internal teacher. Two ways to transfer structure.

The same phrase-conditioned supports supervise what a region represents
and guide how phrase-level content is arranged during training.

SST framework: a frozen understanding path extracts phrase supports for contrastive alignment and a training-only spatial scaffold on the generation path — Phrase-conditioned support extraction, phrase-local contrastive alignment, and the attenuated structural scaffold. Click to view the original figure at full resolution.

Representation supervision

Phrase-local alignment

Pool the reference and generated features within each phrase’s support. Align the corresponding representations while distinguishing other phrases in the same prompt.

Spatial supervision

A training-time scaffold

Compose the phrase representations into a spatial canvas and project it into an intermediate generation state. Scheduled dropout gradually removes access to this guidance.

Original inference. No scaffold. Scaffold-on and scaffold-off steps train the same generation parameters. At deployment, the scaffold branch is removed, with no additional structural input or inference-time branch.

More faithful generation across four backbones.

SST improves both overall AM and GM over matched LoRA fine-tuning
on GenEval2’s 800 compositional prompts.

GenEval2 results. Higher is better for every metric.
Backbone	Method	AM ↑	GM ↑	Position ↑
BLIP-3o 4B	LoRA FT	54.56	13.35	36.29
BLIP-3o 4B	SST (Ours)	58.34	16.05	41.57
BLIP-3o 8B	LoRA FT	60.64	14.38	47.63
BLIP-3o 8B	SST (Ours)	61.40	16.89	50.38
Show-o2 7B	LoRA FT	63.56	15.84	43.03
Show-o2 7B	SST (Ours)	64.36	16.55	45.99
BAGEL 14B	LoRA FT	72.97	26.39	52.13
BAGEL 14B	SST (Ours)	74.01	27.21	54.79

Single-run results from the current manuscript. AM averages atom-level scores; GM penalizes missed conditions within a prompt. Additional category and atomicity results are reported in the paper.

Checked against human judgments.

A blinded study independently evaluates whether the prompt’s individual conditions are satisfied.

Improvement in human atom satisfaction, SST minus LoRA FT.
Backbone	Δ Human AM	95% paired CI
BLIP-3o 4B	+4.4 points	[+1.5, +7.1]
BAGEL	+7.0 points	[+2.7, +11.1]

Three non-author annotators evaluated 80 prompts per backbone with model identities hidden. Across 2,448 evaluated atoms, human-majority and Soft-TIFA judgments agree on 89% (Cohen’s κ = 0.76).

Human-study results from the review discussion. The complete-prompt satisfaction rate also increases, but its confidence intervals include zero at this sample size. This study evaluates compositional correctness.

Better composition, with room to improve.

Exact counting and densely composed prompts remain challenging.

View a remaining counting failure on BAGEL

“three checkered cows, and two bicycles behind four pigs”

Base BAGEL output missing the four pigs — Base model

LoRA output also missing the pigs — LoRA fine-tuning

SST restores pigs but generates three instead of the requested four — SST (Ours)

SST restores the missing pigs, but generates three instead of four. Coarse phrase supports do not provide exact instance decomposition.

The current framework is evaluated on diffusion and AR–diffusion hybrid UMMs. Extending the scaffold to purely autoregressive image-token generation remains future work.

BibTeX

@inproceedings{park2026structural,
  title     = {Structural Self-Teaching for Compositional
               Generation in Unified Multimodal Models},
  author    = {Park, Jaeyoo and Han, Bohyung},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}