Methodology · scoring v1

How the slop score works

Slopicide reads a small sample of a public repo and estimates how much that code looks like low-effort AI-generated code. It is a rough read for fun and conversation. It can’t prove who or what wrote anything, and it says nothing about the people behind a project.

1. Sampling the repo

We resolve the branch you linked (or the default branch) to its latest commit, list the file tree through the GitHub API and keep only real source code. Out go lockfiles, minified and generated files, vendored and copied-in code (node_modules, vendor, shadcn-style components/ui…), build output, test fixtures and snapshots, framework scaffolding and config files, images and binaries, files under 500 bytes and files over 150 KB.

From what’s left we pick up to 8 files: the three largest files from different directories, then random picks that favour directories and languages we haven’t covered yet, with at most two test files. The randomness is seeded by the commit SHA, so the same commit always gets the same sample and the same score. Each file is cut to its first 240 lines or 8 KB, after skipping a leading license header.

Pull requests are judged only on what they change. The same rules decide which changed files count; removed files are left out, and so are files where the pull request adds less than 500 bytes of code. From those we pick up to 8 the same way, starting with the files that gain the most new code, and read just the changed parts of each, plus a few lines around them, from the pull request’s latest commit, within the same 8 KB limit.

2. The model and its questions

Each file goes to @cf/cloudflare/clef-flash on Cloudflare Workers AI. It’s a 9B decision model, not a chatbot: it takes the code plus a list of typed questions and returns a probability for each answer in a single pass. It doesn’t generate text, so there are no AI-written explanations here. Everything you see is derived from those probabilities. We ask 18 questions per file, in one call per file with one retry; a file the model still doesn’t answer for is left out of the score and listed on the report:

Slop signals (yes/no)

• Obvious comments ×1.2: “Does this file contain comments like '# Configure logging' or '// Add the item to the list' that just paraphrase the next line?”
• Tutorial voice ×1: “Does this file contain step-by-step narration comments like '// Step 1: ...', '# First, we ...' or '// Now we ...'?”
• Generic naming ×0.8: “Are most names generic, like data, result, item, handleData, processItems, utils or manager?”
• Boilerplate padding ×1: “Could this file do the same job in half the lines without losing clarity?”
• Needless abstraction ×0.8: “Does this file wrap simple logic in classes, managers, factories or interfaces that are used only once?”
• Defensive everything ×1: “Does this file wrap many calls in try/except or try/catch blocks that only log the error and re-raise or return a generic message?”
• README-style docstrings ×1: “Do simple functions here carry docstrings like 'Args: user: The user. Returns: The result.' that only repeat the names?”
• Copy-paste repetition ×0.4: “Does this file contain blocks that are copy-pasted with small edits?”
• Inconsistent style ×0.4: “Does the coding style change noticeably between parts of this file?”
• Unused or phantom imports ×0.4: “Does this file import names it never uses, or call functions that do not seem to exist anywhere?”
• Assistant tells ×1.5: “Does this file contain emoji in logs or comments, or phrases like 'In a real application', 'For simplicity' or 'TODO: implement'?”

Human fingerprints (yes/no)

• Comments explain why ×1: “Do comments here explain why something is done, such as a workaround, a bug reference or a trade-off?”
• Domain-specific naming ×0.8: “Do the names show deep, specific knowledge of this problem domain?”
• Pragmatic shortcuts ×1.2: “Does the author take terse, idiomatic shortcuts that only an experienced developer would?”
• Lived-in code ×1.5: “Does this code look maintained and evolved over years, with quirks and history?”

Holistic reads

• Verdict (5-level rubric, low to high): Clearly handwritten by an experienced developer → Mostly human-written, maybe lightly assisted → Hard to tell; mixed signals → Likely AI-generated with little review → Unmistakable low-effort AI-generated slop.
• Authorship (pick one): An experienced developer wrote it by hand. / A beginner or hobbyist wrote it by hand. / An AI assistant generated it and a developer reworked it. / An AI assistant or app builder generated it and it was committed with little or no review.
• Source check: is this hand-maintained source rather than generated code or data? Files that look generated count for less.

In calibration the model answered short, concrete questions with an example far more reliably than abstract ones, which is why they read the way they do. The two holistic reads separated handwritten from generated code best by a wide margin, so they carry most of the weight; the individual signals explain the score and nudge it.

3. From probabilities to a score

Here authorship is P(AI, unreviewed) + ½ P(AI, reworked) and verdict is the rubric level scaled to 0–1. The hotspot term exists because slop clusters: two pure-slop files among six careful ones shouldn’t average away. On the report, a signal shows as spotted in a file when its probability is at least 30%. Code snippets on reports come from a simple pattern matcher (restating comments, “Step 1” narration, emoji logs…), are labelled as such and never affect the score.

Tiers

• Handcrafted 0–19: Artisanal, small-batch code. A human clearly sweated over this.
• Human-led 20–39: Mostly human fingerprints, with the odd assist.
• Cyborg 40–59: Part human, part autocomplete. Hard to tell where one ends.
• Slop-adjacent 60–79: Plenty of the tell-tale patterns of unreviewed AI output.
• Peak slop 80–100: Reads like a chatbot wrote it and nobody looked back.

4. The pre-AI-era rule

If the scanned commit is older than June 29, 2021 (when GitHub Copilot’s preview opened; ChatGPT came in late 2022), no AI assistant could have written it. Beginner code from that era can look “textbook” to the model, so we cap those scores at 19 and show the uncapped number on the report. Commit dates can be faked, but that seems like a lot of effort for a joke website.

5. Calibration

We tuned the questions and weights on one set of repos and checked them on a held-out set, using well-known handwritten projects, projects that are AI-generated by construction (from an AI app builder, or mass-generated) and pre-2020 hobby projects. Results with scoring v1:

The ranking was clean on this corpus, but it’s small (40 repos), and “AI-generated by construction” is skewed towards app-builder output. Treat the exact numbers as ballpark.

6. Known limitations

• Beginner code reads as AI. Tutorial-style code by people who are learning shares a lot with generated code. Without the era rule, 2014–2019 hobby projects scored 30–81, and modern beginner code will too.
• Carefully reviewed AI code reads as human. The score is about the low-effort patterns, not about whether an AI was involved at all.
• It’s a sample. About 8 files, up to 8 KB each. A big repo with one sloppy corner may not show it; link a sub-folder with /tree/<branch>/<path> to aim the scan. When fewer than four files are worth judging, the report says to take it with extra salt.
• A small model. clef-flash is fast and cheap, not an expert code reviewer. Its probabilities are hints, not facts.

7. Caching and cost

Reports are stored with the repo, commit SHA, model id and score. Scanning the same commit again reuses the stored report (for a pull request, its latest commit, so a new push gets a fresh scan); scanning a repo within 15 minutes of its last scan reuses that report too. The “Rescan” button skips the cache once the repo has rested for 10 minutes. A typical scan uses about 22,000 input tokens, roughly $0.0008 at $0.038 per million.

8. Daily limits

Every fresh scan (one that actually calls the model) costs real money, so to keep Slopicide free they’re capped per UTC day: 2,000 overall and 30 per person. Cached reports, rate-limited requests and scans that fail before the model is called don’t count. When a cap is reached, fresh scans are paused until midnight UTC: existing reports still open, and a repo whose latest commit was already scanned still gets its report.

9. Privacy

There are no accounts, cookies, analytics or trackers. We only ever read public repositories. We keep the reports (including the short snippets they show) and, with each one, a salted hash of the requester’s IP; the same hash drives the per-minute rate limit and the per-person daily scan count, and those counts are deleted once the day is over. We never store a raw IP address, and source code isn’t kept beyond the snippets on reports.

Scan a repo