Docsloth
Self-hostStart a preview

docsloth.dev

Accuracy scoreboard

Five metrics, one method each: staleness rate, public-surface coverage, verified-claim ratio, time to first docs, and citation validity. Every number on this page comes from a completed run against a frozen fixture, names its denominator, and can be reproduced with the listed command. No result is shown before it is computed.

Current results

Not yet computed. No scoreboard has been computed for this deployment yet. When the run below completes against the frozen fixture, its metrics are published here with their counts; until then this page shows no number and does not estimate one.

Method

  • Staleness rate: claims behind the latest source revision over all public claims, after the regeneration step. A stale claim is one whose evidence row names an older revision than the live source.
  • Public-surface coverage: extracted public symbols, routes, environment variables, CLI flags, schema tables and events that have a reference page, over the surface the extractors parsed. Unsupported constructs are named, never counted as covered.
  • Verified-claim ratio: public claims that resolve to an evidence row in the release’s source revision, over all public claims.
  • Time to first docs: elapsed time from the fixture repository being connected to its first preview bundle being generated, measured by the harness clock.
  • Citation validity: chat-answer citations that resolve to a live evidence row, over all citations in the fixture question set.

Reproduce it

The scoreboard is computed by the cloud evidence script against the frozen fixture. From the repository root:

npx tsx scripts/accuracy-scoreboard-evidence.mjs   # in docsloth-core, produces evidence/accuracy-scoreboard.json
npx tsx apps/api/scripts/accuracy-evidence.ts  # in docsloth-cloud, mirrors the artifact into this app

Limitations

  • The published numbers are a fixture verification of the pipeline, not customer data. Tenant numbers live in each workspace’s accuracy surface.
  • The engine owns claim resolution and metric computation; this page publishes what the run returned and never recomputes a value in the browser.
  • Coverage counts parsed surface only: an extractor that cannot parse a construct reports it explicitly, and that construct is absent from the denominator.
  • Metrics move with the fixture revision; the revision above is part of the result.