iamnor

If you cannot name the stain, you can still wash the shirt. Just do not tell the world what color it was.

The most important number this project produced is negative zero point zero five.

Here is how I got it. I had built a probe that generates code from a model suspected of watermarking its output, generates the same code from a control model, and measures the excess structure in the suspect’s version: how much more the suspect’s repeated answers agree with each other, at the spots where free choice exists, than the control’s do. First run: Claude as the suspect, GPT as the control. The probe lit up. Excess of 0.16 on Python, 0.26 on C. For about an hour I had detected a watermark in a frontier model with no key, which would have been a hell of a blog post.

Then I did the thing you are supposed to do to your own best result. I made the control the same model as the suspect, split-half, identical settings. The excess fell to -0.05 on Python and 0.02 on C. There was never a watermark in that measurement. There were two models with two accents, and my probe was a very complex accent detector.

That correction set the tone for everything that followed. I spent the last week building a tool called BleachMark, and most of what I learned, I learned from experiments that died. This post is the autopsy report, with the numbers.

The stain, and why I care

Language models have started marking their text. Not with a disclosure footer. With a statistical fingerprint in the words themselves. A stain, set at the factory, invisible on the shelf.

The stain matters for three reasons, and none of them is catching students.

A multi-bit watermark can carry a model id, and sixteen bits is enough to tell sixty-five thousand users apart. The journalist who drafted a leak with a model, the dissident who cleaned up their English: the model is not the one who gets identified. The person is.

A hidden character is a covert channel. A zero-width space inside a filename, a right-to-left override next to a Latin path: that is not typography, that is a payload, and if the text feeds a tool-runner the payload rides along.

And a provable mark is a censorship handle. It does not prove a machine wrote the document. It proves a machine touched the document, which is all a hostile platform needs to dismiss the whole thing, human edits and all.

One thing BleachMark deliberately is not: an “is this AI-written” detector. BleachMark’s input is model output by definition.

Two stains, not one

My starting hypothesis treated “watermark” as one thing to detect and then remove. Sort the laundry, bleach the stained pile. The first thing the experiment taught me is that there are two completely different stains.

The first stain is dripped ink: extra characters somebody put on the page. For every carrier class I tested, this one is solved, and solved without any AI at all. Walk the string, classify every unusual codepoint, ask the context whether it has a legitimate job (an emoji joiner has one; a zero-width space inside payload.exe does not), strip the rest. On the fixture set BleachMark catches 24 of 24 planted carriers and flags 0 of 28 legitimate pages. My first pass ran at about 0.7 MB/s, which failed my own throughput floor; the rewrite runs at 8.5 MB/s on typical text and 1.9 MB/s on pathological non-ASCII, both above the 1 MB/s floor. Deterministic, offline, done. If you run model text into anything that can act, you should be doing this today.

The second stain is woven in at the loom. Kirchenbauer’s green-list scheme (ICML 2023) secretly paints part of the vocabulary green at each step, seeded by the neighboring tokens, and nudges the model toward green. No character is wrong. No sentence is wrong. The fabric itself is biased, and everything interesting in this project is about that stain and how to remove it.

With the key, it is physics

If you hold the watermark key, detecting the woven stain is not clever at all. It is counting.

$$z=\frac{g-\gamma T}{\sqrt{T\gamma (1-\gamma )}}$$z=Tγ(1−γ)​g−γT​

Green count g, length T, designed green fraction gamma. I watermarked my own text at lengths from 25 to 1600 tokens, 200 samples per length, and fitted the measured curve: z = -0.087 + 1.065·√T, with an R² of 0.99996. The theoretical slope is 1.065. The threshold sits at z = 4, an analytic false-positive rate of about 3 in 100,000; clean text fired 0 times in 1,400 trials, and the wrong key reads zero.

Look at that plot for a second, because the whole rest of the story is about losing it. The signal grows without bound as the text gets longer. Detection at any false-positive rate you like, if you hold the key.

Without the key, it is a coin toss

I do not hold Anthropic’s key, or Google’s. So the operating question was: how much of that math survives keyless?

For a well-built scheme the answer, measured every way I tried, was: none of it.

The accent detector you already met was attempt one. Attempt two was a steal-and-test partition probe: infer the green set from half the corpus, test it on the other half, with a permutation null. In the arena, against ground truth, it separates a keyed watermark (gap 25.6) from flat style (gap 0.08) beautifully. Then I gave it style that varies by position but has no key, the way real code style does, and it produced a gap of 36.7. A false positive bigger than the true positive. The probe cannot tell a watermark from a habit.

Attempt three was scale. The keyed statistic grows as √T, so surely a bigger corpus helps? I ran live prose detection against Claude Opus 5 three times: 7,800 words, 8,200 words with a wider reference baseline including a pre-cutoff Claude that cannot carry the current mark, and 11,000 words. The calibrated false-positive rates came back 0.50, 0.61, 0.50. A rate of 0.50 means the candidate sits dead in the middle of the reference style band; a watermark claim made from that position is as likely to be wrong as right.

The candidate’s signal did grow with corpus size. So did every control’s, at the same rate, because the thing growing was style structure, which every model has. The √T lever belongs to the key holder. Without the key, more data just measures the accent more precisely.

This is not BleachMark being weak. Christ, Gunn, and Zamir proved that a well-constructed watermark is undetectable without its key (arXiv:2306.09194), and my false-positive rates are that theorem showing up in a lab notebook. For a well-built private-key scheme, keyless detection is not a product feature. It is a wish.

One honest exception. If the producer is sloppy and reuses the same green set across every document, a corpus-level estimator can see the overuse. I planted exactly that mistake on a local Mistral: z of 5.16, a false-positive rate near 1 in 10 million, against a control at 0.34. Sloppy tailoring shows. Good tailoring does not.

So you wash every load

Here is where the design flipped. The plan was detect, then bleach. But a detector that fires at chance cannot be a sorting step, and against a well-built scheme no smarter keyless detector is coming, per the theorem. If you cannot sort the stained laundry from the clean, and washing is cheap, the policy writes itself.

Bleach everything.

The bleach does not need a detection hit to work. A blind paraphrase on live local-model prose took a planted green-list z from 6.67 to 1.43 while the meaning score held at 0.926. The same blind pass degraded a 16-bit attribution mark from 87.5 percent of its bits recovered down to 59.4 percent, against a chance floor of 42.2 percent, while the meaning score held at 0.947. Toward chance is not at chance, and I will not caption it “the author is safe.” But every washed document costs the attacker signal, and no oracle had to bless the wash first.

Then I benchmarked the wash cycles properly: five watermark schemes, four bleaches plus an identity baseline, one fixed 24-sample corpus of 400-token texts per scheme, every bleach applied to the same corpus so the comparison is clean.

The strongest wash, and it was not close: translate English to Spanish and back through a bundled dictionary. Several English synonyms collapse to one Spanish word and come home as a different English word. The mark was riding on harbor; the sentence comes back with a port and the green count does not survive the trip. The detectability drop (how much of the detection signal the wash removes, where 1.0 means gone) ran from 0.38 to 0.92 depending on the scheme, at a meaning cost around 0.45. Against the windowed scheme, the detector’s true-positive rate after the wash: 1.0 to 0.0. A dictionary did that, and English through Spanish is the measured pair; another language needs its own dictionary and its own benchmark before it gets to make this claim. The dictionary matters, and not as a cost compromise: a live model translating the return leg could stamp a fresh mark on the way in. You do not wash a shirt in dirty water.

The benchmark also settled how much the bleach must match the fabric. A unigram mark, where green depends only on the token itself, is a dyed thread: rearrange the threads all you want, each one is still dyed, and measured sentence reorder against it removes exactly 0.000. A context-keyed mark is a woven pattern, and moving threads breaks a pattern. So the figure has sentence reorder sitting uselessly on the zero-cost axis against some schemes and helping against others, and the rule falls out: reorder a pattern, re-dye a thread, and never mistake motion for washing.

I also ran the arms race to its end, watermark designer against bleach, each side free to switch strategies until neither wanted to. It settled somewhere boring: the designer plays the unigram mark, the bleach plays 30 percent synonym substitution, and the designer keeps a third of the signal. No clever mixture. Though the designer should not celebrate: the unigram mark that best survives bleaching is also the easiest for anyone to steal from a corpus, which is exactly the sloppy-tailor case above. Winning the bleach game and designing a good watermark are different problems.

The gate that says no

None of this is a defense unless something can refuse the wash, because a wash that destroys the shirt is just vandalism with better branding. Random uniform edits kill watermarks fine, and keep 32 percent of the meaning.

So the meaning gate is a hard constraint, not a score. For prose it is language-matched: word and character n-grams for English at a floor of 0.60, character bigrams for Chinese and Japanese, a floor of 0.52 for Arabic because even genuinely changed Arabic pairs share script n-grams. One floor does not fit all languages: on the calibration pairs, the gate separates preserved from changed meaning cleanly, but only when the metric and the floor match the language of the text. For code, the gate compiles the rewrite and runs the tests: in the live runs, 6 of 8 bleached programs passed, and the 2 that broke were refused at the door.

I learned the hard way why it must be a hard gate. An early version scored meaning as a soft penalty, and the evolutionary search found the obvious exploit: a “winning” defense that maximized its fitness by wrecking the program and eating the penalty. A gate you can outscore is a suggestion.

The failures I am keeping

Two more results died, and each one earned its line.

I tried to hide a mark in code by asking the model to diversify its output. The corpus channel got bigger, every time: a task with zero free slots (a slot being any spot where honest samples of the same program can disagree) opened to 13 slots, and another task’s detection gap grew from 3.3 to 14.0. The keyless corpus detector reads the places samples disagree, so variation is not camouflage. Variation is the channel. The defense that actually closed a channel was a tight generation-time constraint that pinned the code to one form. And no, you cannot get that by asking the model to “write the textbook version” afterward; it opened a closed channel from 0 slots to 7. Determinism closes channels. Prompts do not.

I tried a transcoding bleach that should have been perfect: have the model write with every word’s letters reversed, then invert it deterministically. In the arena, z from 23.5 to -0.14. On a live model, empty output at every useful length. Models cannot write reversed English at editorial length. They can write Spanish. That gap between the arena and the model is why the dictionary round-trip is the cycle that shipped.

The official cross-check

One more result, and this one is a success that looks like a null. Transformers 4.57.3 ships the official SynthID logits processor, so I scored text with official processor-derived g-values against BleachMark’s SynthID stand-in, both directions. What it produced is no cross-detection, which is exactly the point.

Each detector fires only on its own mark. The g-functions are not the same, so BleachMark detects its stand-in and does not detect production SynthID, and now that limitation is a measured fact on the diagonal of a matrix instead of a caveat in a footnote. That is where I want every limitation of this tool to live.

The wash kata

For anyone piping model text into anything that matters, the sequence the measurements support:

1. Strip hidden characters from every document, first, always. Deterministic, offline, 8.5 MB/s. This closes the covert channel before any model sees the text.
2. Never use a keyless watermark score as a gate. Against a well-built scheme it cannot tell you which files to clean, and a bigger corpus does not fix it.
3. Bleach the whole stream, blind. The round-trip through an unmarked Spanish dictionary is the measured winner. State the meaning cost, about 0.45, out loud.
4. Let the meaning gate refuse. Compile the code, run the tests, hold the language-matched floor. A bleach without a gate is vandalism.
5. Keep the keyed test in the drawer for when you hold a key. Then it is physics, and you will want the number.

I started this project planning to name the stain and treat it. The stain declined to be named, in five separate experiments, exactly as the theory said it would. What is left is better than what I planned: you wash every load, you check the shirt still fits, and you stop pretending you know what color the stain was.

The tool is called BleachMark. Every number above lives in its repo, in the results files and the project log, where it can embarrass me later.

• Ron Dilley