About
Tracecore is an open-source deterministic runtime and evaluation framework for action-oriented AI agents (tool-calling, looping agents that perform real operations like log triage, config fixes, incident recovery, API interactions, and filesystem tasks).
It freezes every variable (agent code, task, seed, budgets, runtime identity) so the exact same run produces bit-for-bit identical outcomes every time. This brings software-grade predictability to agents that are usually probabilistic and flaky.
Key Features
Perfect reproducibility: Run the same episode today or in 6 months: identical traces, artifacts, and results.
Hard budgets & safety guards: Enforce strict limits on steps, tool calls, wall-clock time, filesystem/network access; no infinite loops or surprise credit burns.
Binary validation: Programmatic pass/fail verdicts with structured failure reasons (no subjective LLM scoring).
Immutable artifacts: Schema-validated JSON outputs, hashed for auditability and easy diffs.
CI/CD native: GitHub Action (tracecore-action@v1) gates PRs/merges on deterministic success; add a "Tracecore Verified" badge to your README.
Operations-first focus: Deterministic tasks in suites like filesystem, API, or ops triage where exact correctness beats creative reasoning.
Lightweight & framework-agnostic: Python CLI + FastAPI dashboard; adapters for OpenAI Agents, AutoGen, and more; easy custom agents.
Metrics & observability: Reproducibility rates, budget usage, failure taxonomy, parallel batch runs.
Why it matters Most agent benchmarks rely on vibes, LLM-as-judge, or open-ended tasks. Great for research, but typically chaotic for production. Tracecore treats agents like production software: reproducible, testable, and gateable in CI. It's the missing reliability layer between agent frameworks (LangChain, CrewAI, etc.) and real-world deployment.
Perfect for
SRE/DevOps teams building auto-remediation agents
Anyone shipping agents that touch files, configs, APIs, or production systems
Researchers wanting certifiable, auditable agent performance