AI Cleanup Lab
We built a deliberately 'vibe-coded' store, proved it was broken with live exploits and a load test, then rebuilt it hardened — every number measured, before and after.
AI codegen and no-code tools ship stores that demo perfectly and fall over — insecurely — at real scale. To prove the AI Cleanup workflow, we built exactly that: an intentionally vibe-coded store, then attacked it. A static scan found 7 issues (2 critical); 5 of 5 live exploits landed; a load test topped out near 680 req/s with ~650ms tail latency. We re-architected it: 0 findings, 0 exploits, ~33,000 req/s. Same features — different engineering. It's a reproducible lab, not a slide.
Looks shipped. Isn't.
'Vibe-coded' stores — generated by AI or assembled in a no-code tool — look production-ready and aren't. They ship with injectable inputs, secrets in source, missing access control, and architecture that collapses the moment real traffic arrives. Founders find out the hard way: a breach, a leaked card list, or a checkout that times out on launch day. We wanted to show the failure and the fix with evidence, not adjectives — so we built the broken store ourselves and measured everything.
A senior-led path from idea to live.
Build the 'before' honestly
A small store with the exact anti-patterns these tools produce: a search filter built with new Function(), a hardcoded payment key, an unauthenticated admin route, unescaped output, and the whole database re-read from disk on every request.
Prove it's broken
Three deterministic tools: a static scanner (semgrep-style rules), a live exploit prober that actually runs each attack, and a load test. Result: 7 findings, 5/5 exploits landing, ~680 req/s with a ~650ms p99.
Re-architect, don't patch
Same features, rebuilt: parameterized/safe logic, auth with redacted PII, input validation, secrets from env, a rate limiter, and the database loaded once into an in-memory index — no disk on the hot path.
Re-run the exact tests
Same scanner, same exploits, same load test: 0 findings, 0/5 exploits, ~33,000 req/s, ~12ms p99. The whole lab reproduces from one command.
What did the re-architecture actually change?
Measured before and after on the same store, same tests (July 2026). Static findings went from 7 (2 critical) to 0. Live exploits went from 5 of 5 landing to 0 of 5. Throughput rose from ~680 to ~33,000 requests per second, and p99 tail latency dropped from ~650ms to ~12ms — driven by removing a synchronous full-database disk read from every request.
Break it, then rebuild it.
Proving it's broken
- Static scan (SAST) — semgrep-style rules over the source: 7 findings, 2 critical
- Live exploit prober — runs each attack for real, not a checklist: 5 of 5 land
- Load test — 3000 requests at concurrency 60, measuring throughput and tail latency
- Every result written to results.json — the case study's numbers come straight from it
The hardened rebuild
- Code injection → safe substring search (no new Function / eval)
- Hardcoded secret → payment key from env, never returned to the client
- Broken access control → bearer-token auth on admin, card/email data redacted
- Reflected XSS → all user input HTML-escaped on output
- Price manipulation → checkout validates an integer quantity (1–99)
- Scalability → database loaded once into an in-memory index; a rate limiter added
What we shipped, on screen.
Five exploits, live-tested
Each attack runs against both stores — all five land on the before, all five are blocked after.
Same features, ~48× throughput
The before re-reads the whole database from disk per request; the after serves from memory.
The measured scoreboard
Security, exploitability and load — before and after, from a reproducible run.
Where the hard problems were.
Why we measure instead of assert — and exactly how each number was produced.
Evidence, not adjectives
We don't say 'more secure' — a prober runs five real attacks and reports which land. 5/5 before, 0/5 after.
Root cause: new Function()
The search built a filter from user input with new Function() — textbook remote code execution. AI tools generate this pattern constantly.
Architecture beats micro-tuning
Throughput jumped ~48× from one decision: stop re-reading a 733 KB database from disk on every request. Load it once.
Secrets and PII contained
A live payment key was hardcoded and echoed to the client; admin orders leaked emails and card digits. Both closed at the source.
Reproducible in one command
npm run lab boots both stores, runs the scan, exploits and load test, and writes the numbers. Anyone can re-run it.
Honest lab framing
These are localhost, single-machine numbers on a demo store — the multiples are real, and we say exactly how they were produced.
Why we built and fixed it this way.
Every technology earned its place by solving a specific problem better than the alternatives.
SAST scanner (semgrep-style)
Used for: Pattern rules over the source: injection, secrets, access control, unescaped output.
Advantage: Deterministic and re-runnable on every commit — the same class of tool that catches these issues in a real audit.
Live exploit prober
Used for: Sends each real attack and checks whether it succeeds.
Advantage: Proves exploitability instead of asserting it — the difference between 'looks risky' and '5 of 5 land'.
In-memory index (the rebuild)
Used for: The database is loaded once at boot and indexed by id and category.
Advantage: Removes synchronous disk I/O from the request path — the single change behind the ~48× throughput gain.
Env-based secrets + auth + validation
Used for: Payment key from env, bearer-token admin auth, integer-range checkout validation.
Advantage: Closes the injection, secret-exposure, access-control and price-manipulation vectors at their root, not with a filter in front.
What it's built on.
- Node 20+
- Zero dependencies
- Reproducible harness
- Static analysis
- Live exploit probes
- Load test
- Input validation
- AuthN/Z
- Output escaping
- In-memory index
- Rate limiting
- No hot-path disk I/O
The questions buyers ask about this build.
No — and we're explicit about that. It's an R&D lab: MnT built the intentionally broken store to demonstrate the AI Cleanup workflow with evidence rather than a real client's incident. Every number comes from a reproducible run (npm run lab), not an estimate. The vulnerabilities and the fixes are exactly what we find and do on real engagements.
Five root causes an AI codegen tool commonly ships: a search filter built with new Function() (remote code execution), a hardcoded payment key that was also echoed to the client, an admin endpoint with no authentication leaking emails and card digits, search output reflected into HTML unescaped (XSS), and a checkout that accepted a negative quantity for a negative total. A live prober confirms all five before, none after.
The vibe-coded store re-read and re-parsed the entire 733 KB database from disk, synchronously, on every request — serializing Node's event loop. The rebuild loads it once at boot into an in-memory index and never touches disk on the hot path. Same feature set; the gain is architecture, not micro-optimization.
No — they're localhost, single-machine lab numbers on a demo store, and we label them that way. What's meaningful is the delta and its cause: the security results are deterministic (7→0, 5/5→0/5), and the load multiple comes from a specific, explainable architectural fix you can reproduce.
Yes — this is our AI Cleanup service: a security audit (AI-generated code carries measurably more vulnerabilities), a re-architecture for scale, test coverage, and a clean production handover. Book a free architecture workshop and we'll scope your store's real exposure.
Was your store built with AI or no-code?
AI-generated code carries measurably more vulnerabilities. We'll run this exact audit on your store — security, scalability, and a clean production handover. Book a free architecture workshop to scope it.