Repo-history example

Serverless AWS dev: one runtime change the repo's own checks missed.

In a measured replay of 12 real Serverless commits from repo history, the same AWS dev checks passed in standard CI every time. DriftFence produced no signal on eight commits, then flagged four runtime-matching changes on the same AWS dev scenario.

With approved runtime-matching behavior fixed in Git, DriftFence would have surfaced this change before merge while the repo's own checks still passed.

See the first divergence Back to results summary

First divergence

AWS dev runtime matching moved from Node 20 to Node 22.

Within this replay window, four commits changed the runtime that Serverless chose for supported local Node execution. The same AWS dev checks still passed in standard CI, and two nearby on-exit scenarios did not trigger DriftFence.

Blocked case

Local Node support updated the function runtime.

Serverless started choosing nodejs22.x instead of the approved nodejs20.x when the local Node runtime was supported in AWS dev mode.

DriftFence report

Scenario

serverless.aws.dev.runtime-match-local-node

Expected

output.runtime = nodejs20.x

Observed

output.runtime = nodejs22.x

Classification

violating

The --on-exit=remove and invalid --on-exit validation scenarios did not trigger DriftFence on the same commits. The flagged change stayed on one specific AWS dev surface.

Across the replay window

Mostly quiet, with one clear runtime change.

This window shows one AWS dev workflow changing while two nearby CLI validation cases did not trigger DriftFence across the same 12 commits.

Earlier commits with no DriftFence signal

Eight commits did not trigger DriftFence

Eight commits produced no DriftFence signal on the AWS dev checks.
The same AWS dev checks still passed in standard CI.
No command failures interrupted the replay.

Blocked case

Four commits changed runtime matching

DriftFence flagged output.runtime moving from nodejs20.x to nodejs22.x.
The same AWS dev checks still passed there in standard CI.
All four flagged commits came from the same AWS dev scenario.

Nearby workflows with no DriftFence signal

Two `on-exit` cases did not trigger DriftFence

--on-exit=remove stayed accepted.
Invalid --on-exit values kept the expected validation error.
The two neighboring CLI checks produced no DriftFence signal throughout.

Method

How it was measured.

This replay fixes one AWS dev baseline, reruns the same checks on each commit, and keeps the runtime-matching workflow separate from the nearby controls that did not trigger DriftFence. Exact commit references are linked below.

Replay size: 12 commits
Fixed workflows: --on-exit=remove, invalid --on-exit validation, and runtime matching for a supported local Node version
The same AWS dev checks rerun on every commit in the window
DriftFence reran on every commit with the same approved baseline
This replay focuses on the runtime-matching workflow that changed in this window

Source material

Method and source files.

The links below show the benchmark notes and pinned replay plan for this Serverless example.

Measurement notes

The benchmark results log records the AWS dev window, including the replay size, outcome mix, and representative first divergence.

Benchmark results

Fixed test plan

The Serverless replay definition fixes the three AWS dev workflows and the replay window used here.

See the product, results, pricing, and trust pages alongside this Serverless example.