Overview
Agent code fails in ways ordinary application code does not: prompt injection, unbounded tool loops, silent data exfiltration, missing human oversight. Catching those is a review problem, and manual review does not scale.
Agent Certifier makes that review graded and repeatable. I built it as a FastAPI and Next.js service that reads a codebase it did not author, runs 60 tests across 6 weighted categories, and returns a score, a certification level, and a report naming what failed and why.
The scoring model
Six categories of ten checks each, weighted toward security and stability:
- Security, 30%: injection, hardcoded secrets, prompt injection, data exfiltration, auth, input validation, dependencies, privilege escalation, information disclosure, LLM code injection.
- Stability, 20%: error handling, infinite loops, unbounded recursion, resource leaks, timeouts, graceful degradation, state management, memory, concurrency, exception specificity.
- Performance, 15%: caching, async I/O, batching, connection pooling, lazy loading, algorithmic complexity, streaming, rate limiting, resource cleanup, token optimization.
- Informatics, 15%: complexity, documentation, modularity, type annotations, response format, logging, config externalization, API contracts, naming, duplication.
- Compliance, 10%: PII handling, retention, audit logging, GDPR, licensing, terms of service, output filtering, consent, data minimization, regulatory awareness.
- Ethics, 10%: bias detection, transparency, human oversight, refusal mechanisms, fairness, explainability, harm prevention, misinformation, manipulation, accountability.
The key decision: never execute the code
Running untrusted agent code in order to grade it would reproduce the exact risks the tool is looking for. So nothing is executed, and every check is pattern and heuristic based.
The cost is some recall: a static check can miss behavior that only shows up at runtime. In return, the grader cannot be turned into an attack surface, which is the right trade for a tool meant to be pointed at code you do not trust.
Engineering
- Backend: FastAPI with async SQLAlchemy over SQLite, with OpenAPI docs.
- Frontend: Next.js 14 with TypeScript and Tailwind, polling live test progress while a run proceeds.
- Output: a certification level from PLATINUM (90+) down through GOLD, SILVER and BRONZE to NOT_CERTIFIED, with a downloadable JSON report and a printable view.
- Reach: ten languages: Python, JavaScript, TypeScript, Go, Java, Ruby, Rust, C++, C# and PHP.
- Deployment: one
docker compose upbrings up both services.