Repo-history example

Firebase App Hosting: one release-handling change the repo's own checks missed.

In a measured replay of nine real Firebase CLI commits from repo history, DriftFence produced no signal on the first three App Hosting commits, then started flagging release error handling changes even though the same App Hosting checks still passed. DriftFence kept flagging the same App Hosting scenario on five later commits.

With the approved App Hosting behavior fixed in Git, DriftFence would have surfaced this release change before merge while the repo's own checks still passed.

See the first divergence See the release-it experiment

First divergence

The App Hosting release no longer finished successfully.

The replay had no DriftFence signal through three nearby commits, then Firebase changed App Hosting deploy error handling. The same App Hosting checks still passed in standard CI. DriftFence flagged the behavior change at the operation level.

Representative upstream change

Firebase started throwing after a rollout failed.

The upstream App Hosting release code changed from logging a rollout failure to throwing a Firebase error after the loop. That changed whether the overall release finished successfully.

src/deploy/apphosting/release.ts catch errors during App Hosting deploys

@@ const results = await Promise.allSettled(rollouts);
+let failed = false;
 for (let i = 0; i < results.length; i++) {
   const res = results[i];
   if (res.status === "fulfilled") {
     const backend = await getBackend(projectId, backendIds[i]);
     logLabeledSuccess("apphosting", `Rollout for backend ${backendIds[i]} complete!`);
   } else {
+    failed = true;
     logLabeledWarning("apphosting", `Rollout for backend ${backendIds[i]} failed.`);
-    logLabeledError("apphosting", res.reason);
+    logLabeledError("apphosting", `${res.reason}`);
   }
 }
+if (failed) {
+  throw new FirebaseError(
+    "One or more rollouts failed. Please review the errors above and try again.",
+  );
+}

Relevant standard CI checks

The fixed App Hosting benchmark command still passed.

This replay reran the same App Hosting benchmark command on every commit in the window. On the first flagged commit, that standard CI command still passed with exit code 0.

benchmarks/firebase-tools/plan.template.json replay summary

npx mocha src/deploy/apphosting/benchmark/deploy-source-upload-benchmark.spec.js src/deploy/apphosting/benchmark/prepare-source-deploy-preference-benchmark.spec.js src/deploy/apphosting/benchmark/release-rollout-failure-benchmark.spec.js

replayedCi.verdict: passing
replayedCi.exitCode: 0

DriftFence report

The first reported difference was the completion flag.

DriftFence compared the recorded App Hosting release behavior against the approved contract and flagged the first mismatch at output.completed.

DriftFence report record

protected operation App Hosting release rollout failure

scenario rollout failure does not skip other backends

status VIOLATING

message Mismatch at output.completed.

First divergence

component output

path output.completed

expected output.completed = true

observed output.completed = false

Two other App Hosting scenarios produced no DriftFence signal in the same commit. The flagged change stayed on one specific operation surface.

Across the replay window

DriftFence started flagging the App Hosting change on the fourth commit.

The window shows both sides of the result: nearby commits with no DriftFence signal, and a durable App Hosting change that DriftFence kept flagging after the change.

Earlier commits with no DriftFence signal

Early App Hosting window with no DriftFence signal

Three nearby commits produced no DriftFence signal.
No App Hosting scenario widened or violated.
The same App Hosting checks kept passing in standard CI.

Blocked case

DriftFence started flagging the release handling change.

The same App Hosting checks still passed in standard CI.
DriftFence flagged output.completed
Scenario: App Hosting release rollout failure

Same blocked case later

DriftFence kept flagging the same App Hosting change

Five later commits kept the same DriftFence signal.
The drift stayed on the same App Hosting scenario.
The same App Hosting checks kept passing there too in standard CI.

Method

How it was measured.

This replay fixes one App Hosting baseline, reruns the same App Hosting checks on each commit, and counts the durable signal as one App Hosting change. That avoids overstating the result just because the same change persisted on later commits.

Baseline: one App Hosting commit before the error-handling change
Replay end: the last commit in the nine-commit window
Replay size: 9 commits
Fixed workflows: App Hosting deploy, prepare, and release benchmark-only tests
Verdicts came from replaying the same App Hosting checks and driftfence check --mode enforce on every commit
The replay counts the six later flagged commits as one App Hosting change

Source material

Method and source files.

The links below show the benchmark notes and pinned replay plan for this Firebase example.

Measurement notes

The benchmark results log records the App Hosting window, including the pinned commit range and measured outcomes.

Benchmark results

Fixed test plan

The Firebase replay definition fixes the App Hosting workflows and the pinned replay window used here.

See the product, results, pricing, and trust pages alongside this Firebase example.