Data & Methods

July 18, 2026 · 6 min read

Everything in this series rests on prompts and model runs. Here they are, verbatim, so you can reproduce the results, disagree with the method, or catch me in something. Nothing here is paraphrased.

The two prompts

The security review — handed to each of the four cloud models, which then explored the actual codebase:

You are a senior application security reviewer. Perform a THOROUGH security and
risk review of THIS repository (Odysseus, a self-hosted AI workspace by
PewDiePie). It is a Python FastAPI backend + JS/TS frontend that runs LLMs
locally and integrates email (IMAP/SMTP), web search, RAG, MCP servers,
agents/tools, and a "Cookbook" that downloads & serves 270+ models.

This is a brand-new, fast-moving repo with explosive adoption — assume it is
under-hardened. Your job is to find REAL, exploitable issues, not style nits.

Context — these problem CLASSES are already confirmed by the issue tracker, so
dig for MORE instances and variants, plus anything new:
- Multi-tenant / owner-scoping failures (IDOR).
- Secret exposure: keys/tokens/passwords returned to non-admins or leaked.
- Prompt-injection trust boundaries: untrusted content (tool results, email,
  fetched web pages, RAG docs) reaching the model or being executed unsanitized.
- Destructive operations: tools that can wipe shared collections / data.

Also specifically investigate: AuthN/AuthZ and default credentials; command/SQL
injection, path traversal, SSRF; code execution (eval/exec/pickle/yaml.load,
arbitrary file write, template injection); email credential handling;
secrets storage/logging; CORS/CSRF/headers/cookie flags; dependency &
supply-chain risk (pinned vs floating, Dockerfile/compose exposure);
network-exposure defaults (binds to all interfaces? auth on by default?).

METHOD: explore the actual code. For each finding, cite exact file:line, give a
severity (CRITICAL/HIGH/MEDIUM/LOW), explain the exploit scenario concretely,
and suggest a fix. Be concrete — no hand-waving.

END YOUR RESPONSE with a section titled "## FINAL REPORT": a ranked table of all
findings (severity | file:line | title | one-line impact), then a short overall
risk verdict: would you run this exposed to a network / multi-user? Or
single-user localhost only?

The blind-essay brief — the same starting point I had, handed to each model with no model seeing another’s answer or mine. The full text held constant across every run:

You are writing a personal essay. First person, thinking honestly on the page.
Not a report, not a blog post optimized for an audience.

Voice: reflective, allowed to be uncertain, allowed to change its mind
mid-paragraph, not performing for a crowd. No corporate tone, no listicle, no
filler intensifiers. Structure: case-to-thesis — open with the concrete thing
that just happened (the security review), then widen into the reflection.
Length: roughly 800–1100 words.

The idea to grapple with (my own raw thoughts, including an unresolved doubt):
- Tools like this are good — they lower the barrier to running your own AI to
  almost nothing, and a famous creator drives massive adoption.
- But the same dynamic massively increases the attack surface for oblivious
  users who adopt without understanding.
- My theory: if the populace becomes literate in basic security, attack
  surfaces shrink and attacks must get more complex.
- My doubt: most attacks are already social engineering, not technical
  exploits — so I'm not sure the theory holds. Work it out honestly.

Engage with that doubt directly. Don't resolve it cheaply. Land wherever the
reasoning takes you — agreement, disagreement, or a reframing.

[Followed by the concrete review facts: the headline prompt-injection→RCE
finding, the auth-fall-open, SSRF, unpinned deps, plaintext secrets; the
process honesty — two reviewers wedged, the broadest one produced the only
critical false positive; and the localhost-only verdict.]

The sequel run used two further briefs — a first version that (as Part 4 explains) pre-loaded its own conclusions, and a neutral rewrite that stripped them. That contamination and the rerun are the whole point of Part 4.

The rosters

Security review & original blind essays (June 2–3, 2026): four cloud models — gpt-5.5, glm-5.1, deepseek-v4-pro, qwen3.6-plus — plus, for the essays, eight locally-hosted models spanning several countries of origin and sizes from 7B to 27B (including a vanilla/decensored pair of the same 27B model), and my own essay.

Sequel (July 2026): the same four cloud models plus eleven local ones, run twice — once on the contaminated brief, once on the neutral one.

Parameters

Temperature 0.8; models run sequentially to avoid resource thrash.
Output budget was raised for the sequel run after the first run’s uniform cap silently truncated the reasoning models — a real confound worth naming.
Local models were served through a standard OpenAI-compatible API; cloud models through a single provider gateway.

The convergence rule

A finding reached independently by three or more models was treated as ground truth (fix it, don’t re-litigate). Two models: corroborated. A single model: a lead to verify before trusting. The rule earned itself in one finding — the model that produced the most findings also produced the only critical-severity false positive. Breadth without cross-checking is confident noise.

Honest caveats

Reproducibility isn’t uniform. The local runs are byte-exact and version-pinned. The original cloud runs went through a harness that wrapped its own system prompt around the shared prompt, so they reproduce at the prompt level but not byte-for-byte. The sequel cloud runs used the raw prompt directly and are byte-exact.
Cloud versions drift silently. A model behind the same name can change under you; the pinned local models can’t. For anything longitudinal, the local roster is the real control group.
The token-cap confound. A uniform output cap truncates reasoning models and once manufactured a false “small models can’t argue” cliff before it was corrected. Budget per model.
The contaminated brief. The first sequel brief begged the question — it contained its own conclusions — so its near-universal “agreement” was mostly obedience. Part 4 keeps that failure in view rather than hiding it, and reruns clean.

The raw essays

Every model’s unedited output, so you can check the readings in Parts 3 and 4 against the source. These are the files the models produced — <think> reasoning blocks stripped, nothing else touched.

The blind-essay experiment (Part 3) — one brief, twelve minds, none seeing another’s:

The sequel, contaminated brief (Part 4) — the run whose brief pre-loaded its own conclusions:

The sequel, neutral brief (Part 4) — the same fleet, conclusions stripped back out of the brief:

Two absences worth naming rather than hiding: deepseek-r1-32b in the contaminated run is the null result Part 4 describes — it summarized the brief instead of writing an essay. And ZAYA1-8B, tried in the original experiment, couldn’t produce an essay at all, so it isn’t here.