Research note - Machine learning - June 2026

When CLIP Passes but the Crayon Style Drifts

Which compression and runtime simplification strategies keep object identity and crayon style in a sketch-guided image-generation pipeline?

By The Anh Nguyen - Stable Diffusion 1.5 + ControlNet Scribble + crayon LoRA + fast sampling - controlled evaluation

Style adapter

LoRA

crayon adapter

Style data

training images

Strategies tested

compression, training, pipeline shape

Score-passing rejects

automatic scores failed the grid

Abbreviation guide

CLIP: An image-text scoring model. Here it checks whether an output still looks like the named object and the crayon style.
SVD: Singular value decomposition, a math method for replacing one large table of learned model numbers with smaller pieces.
LoRA: Low-rank adaptation, a small trainable adapter. Here it carries the crayon style without retraining the whole image model.
int4 / int8: 4-bit and 8-bit number formats. They store the model's learned numbers with fewer digits, which makes the model lighter but can change the image.
PTQ: Post-training quantization. Quantize the already-trained model without teaching it again.
QAT: Quantization-aware training. Retrain while simulating the rounding noise from quantization.
VAE: Variational autoencoder, the decoder that turns the model's hidden image representation into pixels.

The first instinct was to make the model smaller. Singular value decomposition, 4-bit quantization, 8-bit quantization, quantization-aware training. All the normal tools were on the table. The catch was the product promise: a child's drawing should come back as the same drawing, rendered in wax crayon.

Evaluation rule. Quality first. A candidate had to preserve object identity, sketch structure, and crayon texture on the fixed grid before any speed, size, or latency claim counted. The report figures use one canonical v1 reference column so the reader can compare candidates against the same baseline.

The baseline was v1: Stable Diffusion 1.5 as the image generator, ControlNet Scribble as the sketch guide, a rank-16 crayon LoRA as the style adapter, and latent consistency model sampling as a shortcut with fewer generation steps. The crayon LoRA was trained on 53 images. It passed the automatic score gates: CLIP style 25.2 and CLIP class 25.5. The visual grid became the real reference. Here is what it is defending.

The Cat Comes Back In Crayon

What good looks like

Kids love colorful art. I do too.

A child draws a cat. Orange, rough, a few guessed-at whiskers. A few seconds after tapping, it comes back in crayon: same cat, same rough shape, now with a cape. Still theirs.

A child's simple orange cat drawn on the app canvas: two ears, two eyes, whiskers. — One tap apart. The same cat, the same pose, rendered in wax crayon on the tablet.

The same cat returned as a vivid crayon rendering with a cape. — One tap apart. The same cat, the same pose, rendered in wax crayon on the tablet.

It happens with an apple too.

Recorded in the app. Left, an apple. Right, the cat above. The same pipeline the report evaluates: Stable Diffusion 1.5, ControlNet Scribble, and the rank-16 crayon LoRA.

I. The Comparison

Five strategies. Does the drawing still come back?

Strategy	What it tried	What happened	Decision
SVDsingular value decomposition	Split attention layers into smaller pieces	CLIP recovered after fine-tuning. The images became muddy and painterly.	Reject
Int44-bit quantization	Make the exported model smaller	The low-level math check passed. The size and final-image checks failed.	Reject
Int8 PTQ8-bit post-training quantization	Quantize without retraining	Best image-preservation result: drift 0.112. It was the cleanest compression result.	Qualified
QATquantization-aware training	Train around 8-bit rounding noise	CLIP passed. The medium shifted from crayon toward pastel.	Reject
Pipeline shape	Change steps and resolution	Matches the v1 baseline directly.	Accept

DoodleCraft strategy grid comparing singular value decomposition, 4-bit quantization, 8-bit post-training quantization, quantization-aware training, and pipeline shape against a canonical v1 baseline column. — Same drawings, same canonical v1 reference column. Candidate columns are rebuilt from the original run artifacts and named to match the comparison table. 8-bit post-training quantization keeps the waxy crayon feel. Quantization-aware training softens into pastel. Singular value decomposition recovery becomes painterly. Pipeline shape matches the canonical v1 column directly.

II. Fine-Tuning Moved The Style

Quantization-aware training moved the style; 8-bit rounding did not

The surprising number was not a CLIP score. It was the image-drift split: 0.019 from 8-bit rounding versus 0.234 from fine-tuning. Rounding barely moved the image. Fine-tuning moved the medium.

Rounding alone was never shippable. It only runs without the int8 math here, to isolate the cost. The real candidate, the one that actually runs end to end in int8, is Section I's Int8 PTQ at drift 0.112.

Drift decomposition comparing the v1 baseline, 8-bit post-training quantization rounding-only output, and quantization-aware training fine-tuning-only output. — The v1 baseline is the reference for the drift split. The PTQ column isolates 8-bit rounding. The QAT fine-tuning-only column isolates the training move before the final 8-bit rounding step. Fine-tuning alone already changes the texture.

That changed the interpretation of the compression ladder. QAT sounded like the careful path: expose the model to quantization noise, let it compensate, then export. In this pipeline, compensation rewrote the style. The class still read correctly. The medium changed.

III. Changing The Run Was Safer Than Changing The Weights

Spend quality headroom, then check the grid

Pipeline shape was less glamorous than compression. Fewer steps, different resolution. But it asked a better question: how much headroom did v1 have before the drawing stopped feeling like itself?

Pipeline shape sweep comparing v1 baseline against lower-step and lower-resolution settings. — The v1 column is the same canonical reference used above. The other columns spend quality headroom by reducing steps or resolution. The sweep did not crown the most aggressive setting. It showed which simplifications the visual grid could still defend.

IV. What Holds

Automatic scores filter. Visual grids decide.

CLIP was useful. It caught obvious failures and kept the comparison cheap. It was not the judge. The judge was the grid, because the promise was not a high CLIP crayon score. The promise was: my drawing came back as crayon art.

Check	Gate	Observed
Baseline quality	CLIP style ≥ 22, CLIP class ≥ 24	v1 passes	PASS
SVD recoverysingular value decomposition	preserve crayon	muddy, painterly	FAIL
Int8 PTQ8-bit post-training quantization	preserve v1 look	drift 0.112; closest compression result	QUALIFIED
QATquantization-aware training	preserve crayon after training	pastel drift	FAIL
Pipeline shape	grid still holds	matches v1 directly; 384 still shows visible drift	PASS

The answer is narrow, and that is why it is useful. In this pipeline, quality-preserving compression was harder than the automatic scores suggested. PTQ preserved style better than QAT. SVD cut the model in the wrong place. Pipeline-shape changes were easier to defend because the grid kept the product promise visible.

Limits. One pipeline. A 53-image style dataset. An 8-sketch fixed evaluation set. Visual grid review, not a formal child study. The result does not say compression cannot work. It says the grid has to say yes too.

Technical record Baseline: Stable Diffusion 1.5 (SD1.5) image generator + ControlNet Scribble sketch guide + latent consistency model (LCM) sampling + rank-16 low-rank adaptation (LoRA) crayon adapter. Strategies tested: singular value decomposition (SVD), 4-bit integer (int4) conversion, 8-bit integer (int8) post-training quantization (PTQ), 8-bit quantization-aware training (QAT), and pipeline-shape sweeps over steps, resolution, and variational autoencoder (VAE) choice.