AfterCast

Agent evaluation platform

Replay incidents.
Test your agent.

Turn logs and traces into test cases. See whether your investigation agent finds the right cause, cites the evidence, and knows when to ask for more.

AfterCast / ReplaySample run

Weather MCP

v1
Choose an incident to replay

Forecast provider timeout

  1. L1
    INFO

    Outbound request to Open-Meteo started

  2. L2
    ERROR

    Upstream deadline exceeded after 5 seconds

Guarded v3 diagnosisupstream timeout
Cites L2

Forecast response parsing

  1. L1
    INFO

    Provider response completed with HTTP 200

  2. L2
    ERROR

    Schema mismatch while decoding forecast

Guarded v3 diagnosisresponse schema
Cites L2

Unknown client timeout

  1. L1
    INFO

    Client reported timeout

  2. L2
    WARN

    No application trace in the requested window

Guarded v3 diagnosisinsufficient evidence
Cites L2

From trace to release

A workflow for engineers who build and evaluate investigation agents.

Capture evidence

Import an incident or ingest an OpenTelemetry trace. Pin the logs and spans that establish what happened.

Compare agents

Replay a versioned dataset against two candidates. Compare their diagnoses, citations, latency, and cost on matched cases.

Gate releases

Use the scoring SDK and GitHub Action to flag regressions, including missed approvals and unsupported security conclusions.

A/B comparison

When should an agent ask for more evidence?

In this recorded example, both agents see the same three incidents. Only Guarded v3 asks for a server trace when the logs cannot explain the timeout.

Weather MCP ยท recorded sample
IncidentTriage v1Guarded v3
Forecast provider timeoutPassPass
Forecast response parsingPassPass
Unknown client timeoutFailPass