P14 — Observability, incident and failure-attribution census

Run: 2026-08-27, Sprint-1 lane S1-L5. Research-only pass; no signups, no execution. Appended to top-companies.jsonl as records P14-C-021 … P14-C-049.

The question this pass was answering

P14's stated open question is "How are failures attributed per capability?" Actionist composes one client application out of heterogeneous capabilities — an intact donor service with its own database, an extracted package, a microfrontend, a connector. When the composed app misbehaves in production the system must attribute the failure to a specific capability, because that attribution is what feeds P15's learning loop (demote the capability) and P14's rollback decision (roll back which component?).

So the research question was sharpened to: which commercial systems can attribute a production failure to a specific component within a composed application, and what exactly do they require to do it? The load-bearing output is not a vendor ranking. It is the requirements intersection — what every serious system demands before it will attribute anything. That intersection is the minimum instrumentation contract, and it is set out in its own section below.

Method

First-party documentation only, fetched live on 2026-08-27, one vendor page at a time, with the mechanism quoted verbatim into each record's claim field. Where a page did not answer the question it was recorded as UNVERIFIED rather than filled from marketing copy or inference — this is why Applitools, Autify, Reflect, Argos, Opsgenie and Akuity are absent rather than present-but-thin, and why FireHydrant (P14-C-049) is recorded as an explicit open gap with the resume path noted.

Two records deliberately carry a weaker claim than the vendor's own framing:

• Datadog Error Tracking (P14-C-047) claims "Which commit probably caused the error" but the page

documents no fingerprinting algorithm and never mentions service or version attribution. Recorded as marketing-adjacent, explicitly not to be ranked alongside Sentry's documented blame algorithm.

• Datadog Synthetics (P14-C-039) asserts you can "automatically trigger a rollback when a critical

test fails" but specifies no exit code, no blocking flag, and no mechanism. Recorded as first-party assertion, weaker than Harness CV or Argo Rollouts where abort semantics are spelled out.

Counts below were re-derived from the file immediately before writing this document.

Count added

29 records, IDs P14-C-021 through P14-C-049. File total is now 49 valid JSONL records (20 pre-existing, preserved and unmodified; 29 appended).

Disposition: 27 census, 2 excluded (Flagsmith P14-C-038, FireHydrant P14-C-049).

Attribution granularity comparison

Granularity is the finest unit the system can name as the culprit. dependency is ranked highest because it is the only granularity that can point outside your own code — which is the case that matters when a capability is an intact third-party donor service.

The distribution itself is a finding: 8 of 29 stop at service, 7 at deploy, and only 4 reach dependency. Most of the commercial market attributes to a deploy unit or a service name. Almost nothing attributes across a boundary into someone else's code, which is precisely Actionist's case.

THE MINIMUM INSTRUMENTATION CONTRACT

This is the primary deliverable. It is derived by intersecting what these vendors actually require before they will attribute anything — not by picking a favourite vendor's model.

The derivation is straightforward once the records are laid side by side. Grafana states the identity set explicitly as five OpenTelemetry resource attributes (P14-C-030). Datadog requires the same three core tags under the name Unified Service Tagging and will not attach a monitor without the service tag (P14-C-021). Elastic states the negative case outright — without instrumentation and traceparent propagation "the connection will not be drawn on the map" (P14-C-046). Harness requires a Monitored Service and a Service Instance Identifier to tell a canary pod from a stable one (P14-C-033). incident.io requires an attribute naming the responsible service or nobody gets paged (P14-C-031). Every one of these is the same requirement wearing a different vendor's vocabulary.

Actionist's HostContract must therefore carry, for every composed capability:

1. A stable capability identifier, emitted on every signal. Not a display name and not a URL —

a stable id that survives redeploy, appears on every span, log line, metric and error, and is the same string the deploy system and the telemetry system both use. This is the single non-negotiable item: service.name (Grafana/OTel), the service tag (Datadog), entity ID (New Relic), the Monitored Service mapping (Harness), and the routing attribute (incident.io) are all this one thing. New Relic supplies the warning about getting it wrong in the other direction: an app reporting as multiple entities "may produce duplicate groups," and their recommended fix — consolidate into one entity — is exactly what a composed app must not do. Capability identity must be deliberately plural where the app is one thing but the capabilities are many.

1. Deploy/release correlation on the same signal: a version and an environment. Datadog's

Automatic Faulty Deployment Detection turns on purely by tagging version; Grafana's service.version exists "to see if a new version has introduced a bug"; Sentry release health needs release plus environment; Bugsnag needs app version plus release stage. Critically this must be per capability, not per app — Sentry release health scores a whole release (P14-C-026) and Rollbar's suspect deploy is "the last deploy prior to the first occurrence" (P14-C-044), a purely temporal heuristic that will systematically blame the wrong capability whenever capabilities deploy independently. Actionist deploys capabilities independently by design, so app-level release correlation is not merely weaker — it is actively misleading.

1. Trace context propagated across every capability boundary. This is the requirement that buys

dependency-level attribution, and it is the one most likely to be skipped because it costs something at every seam. Elastic states the consequence plainly: no traceparent propagation means no edge on the map. Datadog's dependency scoping and Dynatrace's fault tree both rest on the same substrate. Without propagation the ceiling is service; with it, the system can say the fault came from the donor service you called, not your code that called it. Note the failure is silent — an uninstrumented capability does not appear as a gap, it simply does not appear.

1. Ownership metadata resolvable from the identifier. PagerDuty and incident.io both consume an

attribution decision rather than computing one: PagerDuty routes by which integration key the emitter chose, incident.io by an attribute the emitter set. incident.io's contract-enforcement is the model worth copying — marking an attribute required makes it "warn you if your alert sources don't have this attribute configured correctly" and "notify you if we detect alerts that are missing a value." Actionist should enforce the capability identifier the same way, at admission time, rather than discovering at incident time that a capability is anonymous.

1. A per-capability health assertion with machine-readable pass/fail. Attribution answers who;

this answers whether to act. k6 is the cleanest template — thresholds scoped by tag (http_req_duration{type:API}) yielding exit code 0 or non-zero — with the warning that checks alone "do not affect the exit status," so the assertion must be a threshold, not an observation. Argo Rollouts shows the same contract at rollout level via successCondition/failureCondition, and shows the cost of ambiguity: an inconclusive run pauses and "Manual intervention is then needed." A capability that cannot state its own pass/fail cannot be automatically rolled back.

What follows from this for Actionist. Items 1, 2 and 4 are cheap and must be mandatory in the HostContract — they are metadata the host can require and validate at admission. Item 3 is the expensive one and is what separates service-level from dependency-level attribution; it should be a deliberate, costed decision rather than an omission. Item 5 is what makes P14's rollback decision automatable at all, and it should be a per-capability artifact so that a failing capability can be rolled back without rolling back the composed app.

One boundary worth stating: a capability that Actionist cannot instrument caps the whole system's attribution at the call boundary. Vercel (P14-C-045) shows the happy path — when the platform owns the runtime it supplies deploymentId, requestId, invocationId, route and branch for free, no customer instrumentation at all. Dynatrace shows the other route: OneAgent derives topology itself, so the customer supplies agent coverage instead of tags. An intact hosted donor service grants neither. For those, the best achievable is endpoint-level attribution from the caller's side — Vercel's own docs name the shape, where a fault inside a called API "appears only as an Outgoing Request." Actionist should treat instrumentability as a capability admission criterion, because it determines in advance whether that capability can ever be blamed correctly.

Automatic rollback vs alert-only

The most consequential finding here is how few systems close the loop. Nearly the entire observability market stops at notification.

Genuinely automatic, with documented mechanism (2):

• Harness Continuous Verification (P14-C-033) — "During the AI Verify (v1) step, Continuous

Verification automatically triggers a rollback if anomalies are found." Canary classification is mechanical: a pod appearing only after deploy "is identified as a canary pod." Sensitivity is 1σ/2σ/3σ. Caveat: needs a historical ML baseline, so a newly composed capability has none — a cold-start problem that lands directly on Actionist's use case.

• Argo Rollouts (P14-C-035) — "The failed analysis causes the Rollout to abort, setting the canary

weight back to zero"; blue-green post-promotion failure "switches traffic back to the previous stable Replicaset." The only fully declarative, portable contract of the two. Attribution is entirely the customer's job — Argo evaluates an expression and has no model of services or ownership.

Automatic, asserted but mechanism undocumented (1):

• Datadog Synthetics / Continuous Testing (P14-C-039) — claims automatic rollback on critical test

failure; no exit code or blocking flag documented. Verify before relying on it.

Gate only — blocks a pipeline or merge, does not roll back a live deployment (4): Grafana k6 (non-zero exit code), Percy (optional GitHub merge blocker), Chromatic (build status, mechanics unstated), Momentic (CI integration, gating unverified).

Alert only (the rest, 21): Datadog APM and Software Catalog and Error Tracking, Honeycomb, Sentry (all three records), New Relic, Dynatrace, Grafana Cloud App Observability, Elastic APM, Vercel runtime logs, Rollbar, Bugsnag, PagerDuty, incident.io, Rootly, Unleash, Flagsmith, Harness post-deployment rollback (manual, API-triggerable), Harness FME (manual kill).

Two of these deserve flagging because they are commonly assumed to be automatic and are not. Harness post-deployment rollback (P14-C-034) is recorded separately from Harness CV precisely because the two get conflated — every documented path for it is user-initiated. Harness FME/Split (P14-C-036) detects a flag-attributable degradation statistically but leaves the kill to a human: "If you decide to kill a feature flag due to an alert…" An Admin API kill endpoint exists for customer-built automation.

The gap between "can attribute" and "will act" is where Actionist has to build. No surveyed vendor both attributes to a component and rolls that component back automatically. Harness CV and Argo Rollouts act automatically but attribute only to the deploy unit under test; the dependency-level attributors (Dynatrace, Honeycomb, Datadog, Elastic) all stop at alerting.

Unknowns and open gaps

• FireHydrant (P14-C-049) — docs page returned navigational boilerplate only. Resume via

docs.firehydrant.com/llms.txt or by appending .md to the page URL. Title "Service Catalog & Change Events" suggests change-correlation worth chasing.

• Rootly (P14-C-048) — only the docs homepage was retrieved; ownership model unestablished. Do not

rank against PagerDuty or incident.io on mechanism without fetching the catalog pages.

• Applitools — three candidate doc URLs returned 404 and the landing page carried no mechanism

detail (no baseline mechanics, match levels, diff granularity, or CI gating). Not recorded rather than recorded thin.

• Bugsnag stability targets (P14-C-032) — the releases page names the stability center only in

passing. Whether a stability target can gate anything is UNVERIFIED; needs the stability-center docs.

• Datadog Error Tracking (P14-C-047) — fingerprinting algorithm, grouping attributes, and

service/version attribution all absent from the overview page. The commit-attribution claim needs a deeper page before it can be treated as mechanism.

• Datadog Synthetics rollback (P14-C-039) — exit codes and blocking/non-blocking flags need the

datadog-ci package configuration page.

• Split/FME 355-sample significance threshold — appeared in a secondary source, not confirmed on

the primary alerts page. Marked UNVERIFIED in the record.

• Unleash impression-data licence gating — could not be determined from the page examined.
• Flagsmith (P14-C-038) — only the real-time-flags page was examined, so absence of health

monitoring elsewhere in the product is unestablished, not established.

• Not reached this pass: Opsgenie, Split cloud as a distinct entity from Harness FME, Octopus

Deploy, Akuity, GitLab environments+rollback, Buildkite, Spinnaker vendors, Autify, Reflect, Argos, Netlify deploy-linked error attribution, LaunchDarkly per-component signal contract (existing record P14-C-007 not extended — no per-component contract detail found in this pass).