# Build a Python evidence gate for AI slop

> Build a Python AI slop checker that validates draft sections, evidence links, examples, and generic phrases, then reports gaps for human review.

- URL: https://ferdousazad.com/blog/python-evidence-gate-ai-slop
- Author: Ferdous Azad (Founder @ DataXgen AI)
- Published: 2026-11-08
- Updated: 2026-11-08
- Tags: ai, llm, python, ai-ethics

## 1. Introduction

A detector can’t tell whether a draft contains original thinking. A review gate can check for a clear position, evidence, and a concrete example, then flag gaps for a person to review.

In this tutorial, we’ll build a small Python command-line tool that checks those signals. It won’t claim to determine who wrote the text.

## What you'll learn

You’ll define a lightweight editorial contract for a draft, check its Markdown sections and evidence links with Python’s standard library, and flag a few generic phrases. You’ll also run the check locally, or add it to a development workflow.

## Prerequisites

You’ll need Python 3.11 or later, a terminal, and a Markdown editor. The tool uses no third-party packages.

## Step 1 - Define what a useful draft must contain

A checker needs a contract: a small set of conditions it can verify consistently. For this example, every draft needs three sections named `Position`, `Evidence`, and `Example`.

The `Position` section should state what the author thinks. `Evidence` should include at least one source link. `Example` should give the reader a concrete case. The checker can confirm that the sections contain text and that the evidence section contains a link. It can’t decide whether the position is thoughtful, the source is reliable, or the example is genuinely useful.

That limit is deliberate. A script can flag missing signals; a person still has to judge their quality.

## Step 2 - Create a sample draft

Save this Markdown as `draft.md`. The heading names are part of the contract, so use them as written. You can change the contract later, but the script and drafts need to agree.

```markdown:draft.md
## Position

A review gate is more useful than an authorship detector because it can point
to specific gaps that a reviewer can evaluate.

## Evidence

The source describes the reported result and its context:
[Read the source](https://your-source.example/report)

## Example

For a draft about a product change, explain what changed and show one case where
a user would notice the difference.
```

The link is a placeholder. Replace it with a source relevant to your draft. The sample has a position, a linked source, and an example; that doesn’t make those sections good. It makes them available for review.

## Step 3 - Write the Python checker

Save the following code as `evidence_gate.py`. It looks for the three required headings, checks that each has content, searches the evidence section for a URL or Markdown link, and reports a short list of generic phrases.

```python:evidence_gate.py
import re
import sys
from pathlib import Path

REQUIRED_SECTIONS = ("Position", "Evidence", "Example")
GENERIC_PHRASES = (
    "many people believe",
    "it goes without saying",
    "this speaks for itself",
)
LINK_PATTERN = re.compile(
    r"\[[^\]]+\]\(https?://[^)\s]+\)|https?://\S+",
    re.IGNORECASE,
)


def get_section(markdown: str, wanted: str) -> str:
    lines = markdown.splitlines()
    content = []
    in_section = False

    for line in lines:
        heading = re.match(r"^#{1,6}\s+(.+?)\s*#*\s*$", line)

        if heading:
            if in_section:
                break
            in_section = heading.group(1).strip().casefold() == wanted.casefold()
        elif in_section:
            content.append(line)

    return "\n".join(content).strip()


def check_draft(markdown: str) -> list[str]:
    issues = []

    for name in REQUIRED_SECTIONS:
        section = get_section(markdown, name)
        if not section:
            issues.append(f"missing or empty section: {name}")

    evidence = get_section(markdown, "Evidence")
    if evidence and not LINK_PATTERN.search(evidence):
        issues.append("Evidence section has no link")

    lowered = markdown.casefold()
    for phrase in GENERIC_PHRASES:
        if phrase in lowered:
            issues.append(f"generic phrase to review: {phrase}")

    return issues


def main() -> int:
    if len(sys.argv) != 2:
        print("Usage: python evidence_gate.py <draft.md>", file=sys.stderr)
        return 2

    draft_path = Path(sys.argv[1])
    try:
        markdown = draft_path.read_text(encoding="utf-8")
    except OSError as error:
        print(f"Could not read {draft_path}: {error}", file=sys.stderr)
        return 2

    issues = check_draft(markdown)
    if issues:
        for issue in issues:
            print(f"FAIL: {issue}")
        return 1

    print("PASS: draft meets the configured checks.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
```

The section parser stops collecting text when it reaches the next Markdown heading. That means a section’s content is limited to the area under its heading, rather than accidentally including the rest of the document.

The link pattern accepts either a Markdown link or a plain URL. The phrase list is intentionally short and editable. These phrases are prompts for review, not proof that a person or a model wrote the draft.

## Step 4 - Run the check

Run the script from the directory containing both files:

```bash
python evidence_gate.py draft.md
```

With the sample draft, the output is:

```text
PASS: draft meets the configured checks.
```

Now remove the link from the `Evidence` section or delete the `Example` heading and run the command again. The script prints a failure for the missing check and returns a nonzero exit code. That makes it possible to use the tool in a local workflow or call it from another command that checks the exit status.

A pass only means the configured checks passed. It doesn’t mean the draft’s claims are accurate, the source supports them, or the example adds useful detail.

## Step 5 - Adapt the contract to your workflow

The section names and phrase list are policy choices, not universal rules. If your drafts use different headings, update `REQUIRED_SECTIONS`. If you don’t want phrase matching, remove `GENERIC_PHRASES` and the loop that checks it.

Keep the tool’s job narrow. A useful gate points a reviewer to a missing section, absent link, or phrase worth revisiting. It shouldn’t score originality or label a draft as AI-written. Those judgments require context the script doesn’t have.

One useful next step is to run the checker on a few drafts and adjust the contract when it flags something that isn’t meaningful to your team. The goal is a small, explainable review aid, not a verdict.

## Key takeaways

- A Python evidence gate can check draft structure and links without claiming to identify authorship.
- Make the contract explicit with required headings and clear checks.
- Treat generic-phrase matches as review prompts, not evidence of who wrote the text.
- A passing check confirms only that the configured rules passed; a person still needs to assess the draft.