Case study
Certification exam simulator with an original question bank
Certification prep leaves you a choice between ten official sample questions and leaked exam dumps. This generates an original, domain-weighted bank from the official guides and drills it with spaced repetition.
Problem
I am preparing for five cloud and AI certifications at once: GCP Generative AI Leader, Claude Certified Architect, AWS Cloud Practitioner (CLF-C02), AWS AI Practitioner (AIF-C01) and GCP Associate Cloud Engineer. Each one runs into the same wall. Vendors publish an exam guide with domains and their weights, plus maybe ten sample questions. Everything else on offer is a dump site recycling questions that are under NDA, wrong as often as not, and useless for understanding why an answer is right.
More questions would not fix that. What I wanted was a bank that follows the official blueprint domain by domain, and a way to find out which domain is weakest before the exam finds out for me.
So the tool generates an original bank grounded in the official syllabus, weights it the way the real exam is weighted, and drills it under exam conditions, with spaced repetition on everything I fail.
Architecture
The system splits hard between offline generation and runtime serving. Nothing calls an LLM while a mock exam is running: the bank is a build artifact produced by a pipeline that can be re-run to extend it, and the app that serves it is a plain local web app with no network dependency at all.
Generation reads the official syllabus for a certification, splits a target question count across its domains in proportion to the official weights, and injects the syllabus plus one or two public sample questions as grounding for style, never as content to reproduce. Output is parsed into Pydantic models and deduplicated by content hash, so re-running the script extends the bank instead of duplicating it. The provider is switchable between Gemini, Anthropic, and a local Qwen3 through Ollama.
Quality control is a second, independent pass. A local judge model answers each question without seeing the marked answer, compares its own choice against the stored key, and flags factual errors, ambiguity and undeclared multi-answer questions. It never edits or deletes anything. It writes a verdict, and flagged questions are deactivated rather than removed, so the audit trail survives.
Serving is deliberately boring: FastAPI over the standard library’s sqlite3
with no ORM, six routers, and a React 19 SPA covering the dashboard, practice
mode (immediate feedback, per-option explanations), timed exam mode (proportional
domain sampling, no feedback until the end, result against the real pass
threshold) and an SM-2 review queue. In “local production” mode the same server
also serves the built frontend.
Decisions & trade-offs
- Questions are generated offline and never at runtime. Question load is instant, a mock exam costs nothing in tokens or latency, and the app works with the network unplugged. The cost is that the bank is a build artifact: growing it or fixing a bad domain means re-running a pipeline, not clicking a button.
- Original questions only, with official material as grounding rather than content. Exam guides supply the syllabus and public samples supply the register, and neither is ever reproduced or shown. That took considerably more prompt engineering and validation than scraping a dump would have, in exchange for a bank that is legally and ethically clean.
- A second LLM audits the first one. Generation and validation use different models, and the judge answers independently before it ever sees the key, which is what makes disagreement informative. It adds a pipeline stage over the whole bank, and it deactivated 50 questions (4.7%) that would otherwise have taught me something wrong.
- Raw
sqlite3, no ORM, no auth. One user, one machine, one file. In return I write SQL by hand and have no migration tooling, which is acceptable at this size and would be the first thing to go if this ever became multi-user. - The Gemini free tier is treated as a design constraint, not an obstacle.
Request throttling, exponential backoff honouring the API’s
retryDelay, resumable runs and hash dedup all exist because the daily quota runs out mid-generation. No trade-off worth reporting here: the constraint produced a pipeline that is idempotent and interruptible, which is what it should have been anyway.
Metrics
These are properties of the system as it stands, measured against the live
database. There is no before and after here. The honest number for study
outcomes is that there isn’t one yet: exam_attempts is empty, so nothing in
this table claims the tool improves exam results.
| Metric | Value |
|---|---|
| Questions in the bank | 1,062 across 5 certifications and 23 domains |
| Per certification | 204 to 229 questions |
| Active after review | 1,012 · 50 deactivated by the judge |
| Judge verdicts | 983 ok · 19 minor · 50 flagged (gpt-oss:20b) |
| Generation models used | claude-sonnet-5 (378) · qwen3:30b (274) · claude-opus-4-8 (204) · gemini-2.5-flash (196) |
| Question types | 863 single-answer · 199 multi-answer |
| Difficulty mix | 691 medium · 230 hard · 141 easy |
| English translations | 1,012 (qwen3:30b-a3b), Spanish originals untouched |
| Runtime LLM calls | 0 |
| Recorded exam attempts | 0 (no outcome data yet) |
Lessons learned
Grounding beat model size. The questions that read most like the real exam came from feeding the model the actual syllabus section and one sample for register, not from reaching for a bigger model. A local Qwen3 with good grounding produced 274 usable questions, and the expensive models were not proportionally better.
The judge earned its keep, and it is also the part I trust least. Fifty flagged questions is a 4.7% error rate that would have gone straight into my study time, worth every token. But a single local judge is one opinion. The TFM I built around LLM-as-a-judge validation needed three independent validation layers before its scores meant anything, and this pipeline has none of them. The verdicts are useful as a filter, not as ground truth.
Rate limits produced a better pipeline than a generous quota would have. Being forced to make generation resumable, idempotent and deduplicated turned it into something I can re-run against any certification at any time, which is how it should have been written in the first place.
Nothing here proves it works. The bank is built, reviewed, translated and served, and the review queue is empty because I have not sat a full mock yet. Writing “0 recorded attempts” into a case study is less satisfying than a made-up score improvement, but it is the only version of this section that is true.
Links
- No public repository: this is a single-user tool that runs on
localhost, and parts of the seed material (exam guides, official samples) are not mine to redistribute. - Related: Measuring what prompting techniques actually burn, the LLM-as-a-judge methodology this pipeline borrows from.