What are our first steps?

[sean]

Before we split up to survey the literature for our new project, I think it would be helpful to align on a few foundational questions. Since we are taking on the role of the adversary to systematically break existing watermark and fingerprint techniques, defining our scope and threat model early will save us time later.

Here are four key questions we should clarify:

1. What is our domain scope?
Are we limiting our attacks strictly to the text domain, or do we suspect that the vulnerabilities we find in watermarking strategies will translate to other domains (e.g., image, audio) as well?

2. What is the Defender’s access to our model (Black-box vs. White-box Detection)?
When the defender tries to audit our new setup to find their watermark, what level of access are we assuming they have?

• Black-box Detection: They only have API access to our outputs. This allows us to use modular defenses, like hiding behind a paraphrasing pipeline.
• White-box Detection: They can fully inspect our weights and pipeline. This would render a paraphrasing pipeline useless, as they could just inspect our underlying model directly.

3. What is the adversary’s “budget”?
Should we set a computational or financial budget for the robustness attacks we consider? For example:

• Distillation: Highly effective at eliminating fingerprint-based defenses, but computationally expensive to train.
• Paraphrasing / Knowledge Compression: As I suggested in my last post, this pipeline is cheaper to set up but adds an ongoing computational cost during inference.

4. How should we categorize the defenses?
As per Minh’s point on Friday, the boundary between watermarking and fingerprinting is blurry. Should we adopt the taxonomy from the survey Hanif shared (Liang et al., 2024), or do we need to invent entirely new categories?

I expect our answer to this final question will evolve. Ultimately, our categories should be structured so that our proposed attacks target entire categories rather than just individual methods. This will allow us to conclusively state which categories are inherently fragile and which hold promise for future research.

---

My Initial Thoughts

To get the conversation started, here are my proposed answers to the above questions, along with a confidence score (0-10) for my current stance:

• Q1: Domain Scope: Limit to only the text domain. (Confidence: 8/10)
• Q2: Detection Setting: Start with black-box detection. Assuming we can hide our new setup behind an API is the most realistic deployment scenario for an adversary. Defending against a white-box audit (where they can inspect our weights) severely limits our attack options and makes pipelines like paraphrasing useless. (Confidence: 4/10)
• Q3: Budget: There should be a budget, though I’m unsure of the exact metric. Loosely speaking: if the adversary has to spend more to break the watermark than it would have cost them to train the model from scratch, the attack is likely not viable. (Confidence: 5/10)
• Q4: Categorization: This will definitely evolve. I think we can start with the framing from Liang et al. and adjust as we synthesize the literature ourselves. (Confidence: 2/10)

I’m hoping this sparks some discussion. Please let me know your thoughts, or if there are any other crucial adversary parameters I missed that we should crystallize now!

[minhtg.nguyen]

First of all, fancy forum Sean, thank you for creating this!
1. Domain scope: 90% text, 10% image.
I have read some watermarking work in images, and it was already well-defined since the early 2000s, not only visible watermarking but also invisible watermarking in a pixel-wise way, image gradients, etc. Nowadays, with the explosion of AI-generated images, recent research puts the watermark inside the generative model.

So, before the era of AI image generation, people had already thought of ways to claim ownership of the image itself. If we consider the final product, i.e., images similar to text in LLM, then previously they tried decoding-only watermarking, without touching the model (as they did not have one). Now they put the watermark inside the model’s weights, more like fingerprinting the whole model, so that any output image will carry the watermark.

Meanwhile, this is my guess, backed up by references from ChatGPT: before LLMs, people paid less attention to text watermarking. Anyone can write anything, and your writing style may be similar to someone else’s. Also, a copied text can be easily rewritten in plenty of ways, unlike an image, where only some transformations, e.g., rotating, increasing brightness, etc., can be applied, as not everyone is a professional Photoshopers lol.

“Early work embedded signals into text through syntactic structure, synonym substitution, paraphrasing, or morphosyntactic transformations.” (from my best friend, ChatGPT)

Given this, we should start first with LLMs, for which the most recent papers appeared just after the explosion of Attention (2017). Meanwhile, we should keep an eye on some recent fingerprinting work in image-generation models.

[minhtg.nguyen]

2. What is the Defender’s access to suspect model (Black-box vs. White-box Detection)?
(Bro, “access to our model” sounds like we are thieves lol)

The owner probably only has access to the interface, the generated output, and no more than the logits of each token. I guess this is called a grey-box setting. There are two reasons for this. First, if I steal a model, I would not release it, or I would release only the architecture, not the weights. Second, besides claiming model ownership, another target of this field is to detect AI-generated text. Therefore, the grey-box setting seems more reasonable to me.

One thing I am not sure whether it is convenient or inconvenient is that, if we focus only on generated text, watermarking and fingerprinting can be very close. Since for any watermarked output can be learned by a model, we can define that model as a fingerprinted model, which can generate watermarked outputs.

So we actually narrow down this field by using the grey-box setting.

[minhtg.nguyen]

3. What is the adversary’s “budget”?
I am not sure either, but we should probably try all common attack techniques, right? btw, I just came across MARKLLM, which includes several text-modification attacks.

4. How should we categorize the defenses?
Yes, but let’s wait until we agree on the setting in Section 2 first.

[sean]

Firstly, thanks for the compliment on the forum. It’s called Discourse if you ever want to set up something like this yourself. It’s very easy.

While I agree with you that we should start with the text domain, @han did make a good point on Slack that focusing on purely LLM-based models is becoming increasingly irrelevant. Even the Gemma series is multimodal now.

That being said, I do think it is a good idea to narrow the domain so that we don’t overextend ourselves. And I’m sure that the ideas we learn from the text domain will be translatable if we ever do extend into LVLMs.

[sean]

@minhtg.nguyen

(Bro, “access to our model” sounds like we are thieves lol)

Hahaha true, I’m trying to get into the criminal mindset for this project.

@minhtg.nguyen

no more than the logits of each token

Can you clarify this one? There’s a big difference between providing the full logit distribution and just the top-p log probs. For example, I believe OpenAI’s API only returns the logprops of the top-20 tokens if you request it. Do you mean full logit distribution or something similar to what OpenAI does?

[minhtg.nguyen]

Yes, I mean up to the full logit distribution. So the defender can access attacker’s output (likely top 1 if using greedy decode), top 20 like OpenAI, or all logit distribution

[minhtg.nguyen]

Yes, agreed. We should also stay up to date with image-related work. It would require a clear scoping review to understand where the field stands before deciding on the next steps. I suggest we skim one or two recent surveys in this area

[han]

I just read this article, and I think it might be relevant to our discussion of watermarks.

The interesting part is that the fingerprint here is not an explicit watermark added by the model. It is more like an accidental statistical pattern left by the model’s generation behaviour. They used a simple Markov chain-style analysis to detect and even attribute passwords generated by different LLMs.

So maybe one thing we can consider is separating:

1. intentional watermarks, where the defender adds the signal on purpose, and

2. accidental/statistical fingerprints, where the model leaves detectable patterns without being explicitly watermarked.

Just dropping this here as a possible angle. I am not taking the NLP unit, so you guys probably have a much better view on this than me

[han]

I tried a small experiment based on the GitGuardian article about LLM-generated passwords.

I tested 10 models (actually more, but some failed to generate). For each model, I used the same prompt and generated 1000 API-key-like strings in total. I generated them in batches of 200, so basically 5 rounds per model. Then I ran a few simple statistical checks, with the help of ChatGPT because I was too lazy to manually write all the code

I notice that the generated keys are distinguishable by model (the model attribution test gives around 96.96% macro accuracy across all models). The strongest signal seems to come from short local character patterns, especially around the 3-gram level. Looks like the generated API-key-like strings contain strong model-specific statistical patterns.

The dataset, results, and code are here if anyone wants to take a look: Repo Link

note: my prompt asked the model to “invent a secure 32-character API key format” so the prompt itself may encourage the model to create a repeated format.

[sean]

I think this is a reasonable classification. Is your intuition that the way to break intentional watermarks vs inherit fingerprints will end up being different?

[han]

Yes, I think the way to break intentional watermarks and inherent fingerprints will probably be different, although some attacks may overlap.

For intentional watermarks, the attack can target the watermarking mechanism more directly. For example, with SynthID-Text, the watermark is inserted during token sampling, so an attack can try to disrupt that specific signal or exploit the detector, like the layer inflation idea in this paper. However, attacks like layer inflation may also be easier to defend against once they are known. For example, OpenAI could block the model from suspicious prompts that appear to probe the watermarking system, like this

Inherent fingerprints are not a hidden signal added on purpose, but more like the model’s natural accent or generation habit. For example, in my experiment, the models were distinguishable from short local patterns like 3-grams. A simple paraphraser might reduce some surface style, but it does not fully remove deeper structural habits. To really scrub that kind of fingerprint, the adversary may need a stronger attack, such as regeneration via another model, distillation, or fine tuning, which changes the output distribution more fundamentally.

I would not say these are completely separate, but I think intentional watermark are more mechanism-targeted, while inherent fingerprint are more about the model’s overall writing patterns and statistical habits.

[sean]

So to clarify, would you say that inherent fingerprints likely require a stronger, general attack to scrub, whereas intentional watermarks just require an attack targetting that specific method?

Because of this, I feel like inherent fingerprints would be harder to scrub than intentional watermarks. Would you agree? I would also say that because the signal for both is in the output space, by finding a general attack that can scrub inherent fingerprints, this would also by proxy be able to scrub the watermark signal.

Can we find a counterexample to this? In other words, given an attack that can scrub all fingerprint signals, does there exist a watermark technique that would be able to survive such an attack?

[han]

Yeah, I mostly agree.

Inherent fingerprints probably need a broader attack to scrub, because the signal is not one specific hidden rule. It is more like the model’s overall generation habit. So to remove it properly, the attacker may need to change the output distribution through regeneration, distillation or fine-tuning

For intentional watermarks, the attack can sometimes be more specific to the method. If we know the detector relies on a specific token-level statistic, we can attack that statistic directly, such as the layer inflation attack against the SynthID-Text mean score.

For the counterexample, I think it depends on what we mean by “scrub all fingerprint signals”. If the attack only removes natural model fingerprints, then a watermark with a secret key / semantic-level payload might still survive