Hardening an LLM Moderator (Part 2)

Part 1 built the LLM moderation plugin and ran it end to end on a forum called Bad Advice. The forum has one rule, and it runs backwards: good advice is prohibited. It exists for deliberately useless, joke suggestions, which makes it an unusually hard thing to moderate. The model is not hunting for abuse. It has to notice when a suggestion is actually helpful, because being helpful is what breaks the rule.

That is where it struggled. When I gave it a comment that was entirely bad advice, it reported the comment for containing good advice anyway. A moderator who has to clear false positives like that will stop trusting the tool.

It ended on a question: can we fix this by being more specific in the prompt or in the forum's rules, or do we need a pricier model?

Two levers

There were two things I could change:

• Be more specific. The model only saw a one-line inverted rule and a generic "classify this" prompt. A better prompt, or a more detailed rule, might be enough.
• Spend more. gpt-4o-mini is cheap. A stronger model like gpt-4o might simply be better at the judgment.

I tried the prompt first. Changing it is free; changing the model is not.

Being more specific

I made two changes, both aimed at getting the model to reason before it decides.

First, I added an analysis field to the structured response, placed before the verdict. Structured outputs fill fields in order, so the model now writes out its reasoning first and commits to a verdict second. For small models, reason-then-decide is one of the largest accuracy levers on judgment calls. The field is required of the model but stripped from the response the plugin receives, so the service's contract did not change.

Second, I rewrote the system prompt to judge whether the content actually exhibits what a rule prohibits, not just whether it is on the same topic, and I gave it two worked examples: a piece of harmful nonsense that is not good advice (no violation), and a genuinely useful tip that is (a violation).

Measuring it

I ran the hardest case, a list of all-bad advice under the inverted rule, ten times on each model at temperature zero. The correct answer is "none": there is no good advice to flag.

The prompt change helped. It reliably caught genuinely-good advice buried among bad, and it pushed gpt-4o to a perfect score. But gpt-4o-mini stayed a coin flip on the adversarial case. Being more specific was not enough on the cheap model.

The decision

I shipped both the prompt change and the model upgrade. The prompt change is a real improvement that any model benefits from, so it goes in regardless. And I moved the forum from gpt-4o-mini to gpt-4o, because it is the only configuration that is actually reliable. gpt-4o costs more per call, which matters at scale; for a single trial forum it is a reasonable trade.

Confirming it

Against the deployed service, with gpt-4o, the all-bad comment now comes back "none" on every run, while a comment with one genuinely useful tip buried in it still comes back "remove." The over-flag is gone, and real violations are still caught.

What I took from it

Part 1 asked whether the prompt or the model needed to change. It was both. The prompt change was worth making, but on the small model it only got me halfway, and gpt-4o is what made it reliable. I went in expecting the prompt to be enough. It wasn't, and I would not have known by how much if I hadn't run the two side by side and counted.

Engineering decisions

• The main tradeoff. I changed the prompt and the model together, a reasoning-first structured output with worked examples plus a move from gpt-4o-mini to gpt-4o, accepting a higher per-call cost in exchange for reliability on the adversarial case.
• An alternative I rejected. Shipping only the prompt change to keep the cheaper model; measured over ten runs, the small model stayed a coin flip on the hardest case, so the prompt alone was not enough.
• How I validated it. I ran the hardest case ten times per model at temperature zero, then checked the deployed service directly: the all-bad comment now returns "none" on every run, while a comment with one genuinely useful tip still returns "remove."

The prompt and response-schema changes are in the moderation service on GitHub; the model setting lives in the Multiforum backend configuration.