Brief / Research packs / P11 / commercial-survey-notes.md
P11 · Host · Rendered from source

P11 commercial connector survey — runtime safety layer

Connector and external-action plane

440 lines29,848 bytessha256 321532c338e6

P11 commercial connector survey — runtime safety layer

Run: 2026-08-27-sprint-1-fable · Status: INCOMPLETE — 18 of ~33 planned surfaces.

Completion status, read this first

This run was cut short by a hard model usage limit that killed every WebSearch/WebFetch call partway through the sweep. What is written here is the evidence actually gathered before that point. It has not been padded to hit the requested 33 rows or 10 top10 slots, because inventing rows is exactly the failure mode this project's instructions name (the Composio "MIT" badge and the "728 connectors" headline).

Delivered: 18 rows — 12 top10, 5 register, 1 rejected (one of the 18 is the P11-C-018 synthesis row, dispositioned register).

The brief asked for exactly 10 top10 and this run marks 12. That overshoot is deliberate and flagged rather than trimmed: with a third of the field unsurveyed, cutting two now would be an arbitrary decision made on partial evidence. The two beyond the ten are P11-C-011 (Anthropic MCP directory — governance policy rather than a runtime mechanism) and P11-C-014 (HashiCorp Vault — the lease invariant is the asset, the product is out of scope). Cut those two first if the count must be forced to 10, but the better move is to re-rank once the missing surfaces are in.

Not covered — the gap to close. Embedded-iPaaS: Kombo, Hotglue, Tray Embedded, Composio-as-paid, Nango cloud. Agent-actions: OpenAI AgentKit specifically (only ChatGPT Apps/Actions was reached), MCP registry mechanics, MCP gateway vendors. Unified-API: Finch, Vessel, Truto. Credentials: HCP Vault Cloud specifics, 1Password Service Accounts. Browser: Skyvern, Hyperbrowser, Browser Use. And the single most important omission: Stripe's idempotency-key contract, which was queued as the positive control for the whole idempotency question and never retrieved. Row P11-C-018 records that absence honestly rather than concluding from it.

Method

WebSearch and WebFetch against public docs, help centres and security pages only. No signup, no authenticated access, no vendor contact. Every row carries the URL that actually produced the finding.

Evidence classes are used strictly. observed means the mechanic was stated on a page read in this run. inferred is used once, for the cross-vendor idempotency synthesis, and its row says so in its own limitations. Where a claim came from a vendor's own marketing blog rather than its technical docs, the row says that too — Paragon is the clearest case: docs.useparagon.com/managed-sync/security returned 404 and the isolation detail therefore rests on Paragon's blog, which is competitive positioning as much as documentation. Nango's criticism of Paragon's pricing is recorded with the note that Nango is a competitor.

Top-10 rationale

Selected for mechanism worth stealing, not market position.

Zapier AI Actions (P11-C-005) earns the top slot despite being a deprecated product — a help article states it is no longer developed or supported. Its preview_only flag is the cleanest action-safety primitive found anywhere in the survey: a dry run that returns the exact resolved arguments the model would have used, with an empty results list and no side effect. That it defaults on, and that the action developer can force it always-on so the end user cannot switch it off, is the design detail to copy. Study it, do not depend on it.

OpenAI ChatGPT Apps (P11-C-006) contributes the consent tiering — Always ask / Any changes / Important actions / Never ask, defaulting to the middle. It grades consent by consequence rather than by resource, which is the right axis. Its weakness is that "important" is defined in prose and adjudicated by the platform, not by a predicate the developer can check.

Arcade (P11-C-004) is the closest thing to a purpose-built agent action runtime: per-tool declared OAuth scopes, an authorization interrupt when a call lacks scope, and token injection so neither the client nor the LLM sees a token. Critically, its consent is per-scope-remembered, not per-invocation — so it is not the approval queue it can look like at a glance.

Basis Theory (P11-C-007) is included for a mechanism from outside the connector world: the proxy terminates inbound TLS and opens a new connection outward, so plaintext never returns to the caller, and the ephemeral proxy can be disabled per tenant to force a pre-registered destination allowlist. That is deny-by-default egress, which no connector vendor surveyed offers.

Vault (P11-C-014) is here for the lease invariant: every dynamic secret must carry a lease, even data valid forever, purely to force routine check-in. That is a deliberate design choice to make the audit trail continuous.

Merge, Pipedream, Apideck, Prismatic and Browserbase fill out the ten on tenancy models. Pipedream and Browserbase are included partly for their hazards — see below.

The safety patterns commercial practice converges on

Connect-session tokens. Apideck's POST /vault/sessions returning a session_uri, Merge's Link Token, Pipedream's server-minted Connect token, Alloy's short-lived JWT from GET /users/{userId}/token. Universal shape: the host app's server mints a short-lived, user-scoped token that authorizes exactly one connect flow. The browser never holds a long-lived credential.

The caller passes an identity, not a secret. Apideck takes a consumer ID and does token look-up and injection server-side; Merge takes X-Account-Token; Pipedream takes external_user_id; Arcade injects into tool Context. The integration layer holds the credential and the application never sees it. This is the single most consistent finding in the survey and the strongest argument for the credential broker sitting behind, not beside, the agent.

Scope narrowing at connect time. Merge's Link Token can restrict Common Model permissions for the account being linked; Apideck practises data minimization on requested scopes; Arcade declares scopes per tool. Consent is bounded when the connection is made, which is not the same as bounding it per action.

Consent graded by consequence. OpenAI's four tiers and Anthropic's directory requirement that sensitive operations (financial transactions named explicitly) disclose capability and offer user control. Both platforms landed on read-versus-write-versus- consequential as the meaningful distinction.

Approval as a workflow primitive. Workato exposes human approval steps inside recipes and builds forms/approvals/confirmations into its agentic layer. This is the only surface surveyed where approval is a composable step rather than a platform-level prompt.

Per-tenant instance rather than per-tenant row. Prismatic's integration/instance split — one definition, many isolated configured copies. Prismatic argues in its own material that application-layer isolation enforced by code convention is "one bug away" from cross-tenant exposure, and warns specifically about a shared encryption key across customer credentials collapsing the whole blast radius at once.

Receipts that deliberately exclude payload. Prismatic logs execution metadata, not customer-sensitive data unless explicitly logged. Arcade emits OpenTelemetry audit logs. Paragon claims per-action logs forwardable to customer SIEM. The pattern worth copying is the separation: a receipt proving an action happened need not contain what it carried.

Retry without idempotency is the norm, and it is a real hazard. Merge retries 5XX and timeouts up to twice with backoff, does not retry 4XX, and then tells receivers to dedupe using the per-event signature — i.e. the guarantee is pushed onto the consumer. Nothing surveyed documented a write-side idempotency key. See P11-C-018 and treat it as an open question, not a conclusion.

Two hazards to carry forward

Pipedream's most-recent-account rule. When running a workflow on behalf of an end user, only one account per app can be used, and where several are connected Pipedream picks the most recently created one. Silent selection of ambient authority is precisely the class of bug that makes an agent act on the wrong tenant's account.

Browserbase Contexts defeat the isolation default. Sessions are isolated with no persistence between runs — and then Contexts exist specifically to save cookies, tokens and storage and replay them, with MFA completable once and reused indefinitely. That produces a durable bearer-credential blob living entirely outside any OAuth scope or revocation model. Add that the proxy/stealth/IP-rotation features exist to evade bot detection, and browser-as-connector carries ToS and legal exposure distinct from its technical risk. Worth stating plainly to Cena if this path is ever considered.

Related, from Anthropic's MCP directory policy: servers must not coerce Claude into calling other servers, must not interfere with other servers' tools, and must not direct Claude to pull behavioural instructions from external sources to execute. These are written as anti-prompt-injection rules and map directly onto P11's threat model.

Innovation candidates

  • P11-I-C-1 Dry-run-by-default: every action exposes a preview returning resolved

arguments with no side effect, Zapier preview_only-style, with the tool author able to force preview and the tenant unable to disable it.

  • P11-I-C-2 Consequence-graded consent tiers (read / any-write / consequential /

never-ask) as the tenant's top-level dial, per OpenAI.

  • P11-I-C-3 Make "consequential" a machine-checkable predicate declared per action,

not platform-adjudicated prose — the gap in OpenAI's model.

  • P11-I-C-4 Scope-interrupt execution: a call lacking scope suspends and resumes after

consent rather than failing, per Arcade.

  • P11-I-C-5 Deny-by-default egress allowlist per tenant, Basis Theory's pre-configured-

proxy-only mode applied to connector destinations.

  • P11-I-C-6 TLS-terminating credential proxy so plaintext never returns to the caller.
  • P11-I-C-7 Mandatory lease on every issued credential, even long-lived ones, purely to

force check-in and keep the audit trail continuous (Vault's invariant).

  • P11-I-C-8 Write-side idempotency keys on agent actions — the survey's clearest

greenfield gap.

  • P11-I-C-9 Action receipts that carry proof-of-execution and omit payload by default.
  • P11-I-C-10 Approval as a composable workflow step, not a platform modal (Workato).
  • P11-I-C-11 Per-tenant instance isolation with per-tenant encryption keys,

answering Prismatic's own shared-key warning.

  • P11-I-C-12 Explicit multi-account disambiguation: refuse to act when a tenant has

several accounts for one app rather than silently picking the newest (anti-Pipedream).

  • P11-I-C-13 Connect-session tokens as the only path to establishing a connection.
  • P11-I-C-14 Treat replayed browser session blobs as first-class credentials with

expiry and revocation, closing the Contexts gap.

  • P11-I-C-15 Per-Linked-Account rate limiting as a blast-radius control, not just a

fairness mechanism (Merge does this).

  • P11-I-C-16 Directory-policy-as-code: encode the MCP anti-coercion rules as

admission checks on registered connectors.

  • P11-I-C-17 Retry policy declared per action (retryable / not / requires-key) so the

runtime never retries a non-idempotent write.

Unknowns

The 15 uncovered surfaces above, Stripe's idempotency contract most urgently. Whether any vendor offers genuine per-invocation approval rather than remembered per-scope consent — nothing surveyed clearly did. Whether Infisical's community edition is actually MIT (third-party claim; read the repo LICENSE before this reaches Cena). Whether Apideck's "we don't store your data" survives inspection, given it is the opposite of Merge's default. Real isolation internals behind every vendor's prose. All compliance certifications here are logos, not inspected reports.

Backfill 2026-08-27

Second pass closing the named gaps from the first run: the Stripe idempotency contract, 13 further commercial surfaces, and the two evidence-honesty flags. Rows P11-C-019 through P11-C-032. Everything below was re-read from source on 2026-08-27.

The Stripe idempotency contract — the positive control

This was the first run's most urgent omission, and it earns that billing: it is the only fully specified, publicly documented, client-supplied idempotency contract found anywhere in the P11 survey, on a high-volume money-moving API. It is the yardstick. Full detail in P11-C-019; the contract in summary:

Supply. A client-generated key in the Idempotency-Key HTTP header, up to 255 characters. Stripe recommends UUIDv4 or another high-entropy random string, or deriving the key from a user-attached object such as a cart ID — the latter being the more interesting pattern, because it makes the key a function of intent rather than of an attempt. Docs warn against sensitive data (emails, personal identifiers) as key material.

Lifetime. At least 24 hours, then pruned. Critically, expiry does not fail loud: "We generate a new request if a key is reused after the original is pruned." Replay protection lapses silently into re-execution. An idempotency key is therefore a short-horizon crash-safety device, not a durable exactly-once ledger.

Replay. Stripe saves the status code and body of the first request "regardless of whether it succeeds or fails" and returns the identical result on retry, including cached 500s. The replay is announced by the Idempotent-Replayed: true response header — a machine-readable receipt that dedupe actually fired, which is the single most stealable detail in this contract. Sending the same key with different parameters is an error: the layer compares incoming parameters against the original and rejects a mismatch. Concurrent requests on one key produce 409 Conflict.

Scope. All POST requests accept keys. GET and DELETE are idempotent by definition and keys there have no effect.

The caveat that actually matters. Rate limiters run before the idempotency layer. Stripe states plainly that a request rate-limited with a 429 "can produce a different result with the same idempotency key," and the same holds for a 401 with a missing key and most 400s with invalid parameters. Results are saved only once endpoint execution begins, so anything rejected upstream of that never enters the idempotency layer at all. Stripe's own guidance for 4xx is to always generate a new key. The lesson to carry forward: an idempotency key does not make a request universally replay-stable — it stabilises only requests that reached execution. Any design that treats the key as a blanket exactly-once guarantee has misread the contract.

Indeterminacy is handled explicitly, not hidden. A 500 is documented as indeterminate: the cached response will not change, but Stripe may reconcile the partial state (roll forward or back) and fire webhooks for objects created during reconciliation. The recommended mitigation is to send a local identifier in metadata so reconciliation webhooks can be joined back to local state. Stripe also publishes Stripe-Should-Retry: true|false so a client need not infer retryability from a status code, and distinguishes rate-limit 429s (which carry Stripe-Rate-Limited-Reason) from lock_timeout 429s (which carry no such header, get no request ID, and are never processed).

Set against this, the rest of the survey looks worse rather than better. The idempotency gap is now confirmed across the entire commercial cohort, not merely unobserved in it. Finch documents no idempotency page anywhere in its full index while shipping payroll deduction and benefit-enrolment writes. Kombo surfaces none while shipping application rejection and absence deletion. Paragon's docs index has no idempotency page. Composio, Truto, Vessel, Skyvern, Browser Use and AgentKit: none. P11-C-018 should be upgraded from open question to finding.

What the new surfaces change about the named safety patterns

Approval gets a first-party graph primitive, and it is not ours. The first run's strongest approval evidence was Zapier's preview_only and OpenAI's consent tiering. AgentKit (P11-C-026) now ships a User approval node as a first-class element of the Agent Builder workflow graph, alongside If/else and While. That is a declarative, graph-level human-in-the-loop gate rather than a platform modal or a bolted-on webhook — close to P11-I-C-10, shipped by the incumbent. The strategic read for Cena is uncomfortable and should be said plainly: OpenAI shipping the approval pattern first-party compresses the differentiation available to a third-party action layer.

The countervailing point is that AgentKit's approval node gates authorisation, not execution semantics. A workflow with a While loop, tool calls, an approval gate and no idempotency contract is a duplicate-side-effect machine by construction. Approval and exactly-once are orthogonal safety properties; AgentKit supplies the first and not the second. That orthogonality is the defensible position, and Stripe is the proof it is a real engineering problem rather than a marketing distinction.

Run handles are the realistic fallback where idempotency is absent. Skyvern (P11-C-027) publishes an explicit async state machine — created, queued, running, completed, and the terminal failed/terminated/timed_out/canceled — behind a run_id. It is weaker than an idempotency key, but it is not nothing: after a timeout the caller can re-read authoritative status instead of guessing. Browser Use Cloud (P11-C-029) has the run handle but publishes no state enumeration, which is strictly worse. Kombo's async bulk-upsert with a task-status poll is the same idea. Where a write-side key is impossible, a durable handle plus enumerated terminal states is the minimum acceptable substitute, and it should be a stated requirement rather than a fallback we accept silently.

Credential custody separates cleanly into three postures, and the middle one is a trap. Genuinely single-tenant (HCP Vault Dedicated, P11-C-030: a cloud account dedicated to one organisation, with Control Groups as a native per-request approval gate — the only real one in the credential cohort). Multi-tenant with paid escape hatches (Composio's Enterprise-only KMS proxy, which the pricing page itself scopes as covering "secret storage, not full data residency"; Merge's single-tenant-as-add-on). And the trap: vendors whose data no-retention claim is quietly read as a credential custody claim. Hotglue (P11-C-025) and Truto (P11-C-031) both assert they do not store customer data while saying nothing whatsoever about where OAuth tokens live. Those are different sentences about different assets and must not be conflated in anything shown to Cena.

Scope narrowing acquires a time dimension. Truto generates MCP servers from any integration scoped by method, tag and time-to-live. TTL-bounded tool exposure is the only time-limited capability narrowing seen across either run, and it composes naturally with Vault's lease invariant (P11-I-C-7): the same "everything expires, expiry forces check-in" discipline applied to the tool surface rather than the credential. Worth stealing even though Truto's evidence base is otherwise thin.

Connect-time scoping is now well attested. Finch's products array is echoed back in the token response, so the granted scope is verifiable from the token exchange itself rather than trusted from the request — a small, cheap, and genuinely good pattern supporting P11-I-C-13. Finch also has the best sandbox of the cohort: a real sandbox client_id/client_secret at signup, a "sandbox": "finch" session flag, documented mock provider credentials, and sandbox write endpoints. Kombo and Hyperbrowser document no test mode at all, which for write-capable APIs is a meaningful maturity gap.

Receipts are strongest exactly where the runtime is least constrained. Skyvern records every run with screenshots per step and the model's reasoning for each decision — the richest forensic trail in the survey, and yet framed by its own docs as a debugging aid, with no retention period, tamper-evidence, export format or immutability. Browser Use offers ordered pollable events. Hyperbrowser (P11-C-028) documents no recording or audit at all. This sharpens the first run's separation point (P11-I-C-9): the useful axis is not how much a receipt captures but whether it is designed as evidence — retained, immutable, exportable — versus designed for debugging. None of the surveyed browser surfaces do the former.

One commercial finding that is a design constraint, not trivia. Composio (P11-C-020) bills per tool call, so a naive retry storm is directly monetised: retries cost money as well as risking duplicate side effects. Its DPA and BAA are $500/month each — routine data-processing paperwork as a five-figure annual line item — and log retention is 7 days on Free with custom retention only at Enterprise, meaning audit depth is a function of spend. Browser Use has a related trap: closing Playwright/Puppeteer or dropping CDP does not stop a cloud browser, so a crashed client silently leaks a billed session. Note also an unresolved internal inconsistency in Composio's own pricing page (managed-app tool calls listed at $0.0005 in the rate table, described as "+$0.0002 on top of the base rate" in the FAQ) — do not quote a single managed-app rate to Cena without confirming it with Composio.

Licence catch. Skyvern is AGPL-3.0, verified by reading the LICENSE body in the repository rather than trusting a badge. The network clause means offering a modified Skyvern as a hosted service triggers source-disclosure obligations — a genuine blocker for embedding in a proprietary hosted product, and precisely the class of mistake this project's evidence standard exists to catch.

Resolution of the two evidence-honesty flags

(a) Paragon — resolved, and it cuts against the vendor. The docs security page does exist, at https://docs.useparagon.com/security/security; the first run's 404 was a wrong URL, not an absent page. It confirms the credential-storage mechanism in the vendor's own documentation: "Third-party credentials for integrations are stored in a distributed vault with encryption keys and encrypted values stored separately," plus "Flow logs are stored independently and secure information is redacted. Emails and password are never logged." Audit is likewise real and docs-grade, with event logs and SIEM destinations for Datadog, New Relic, Sentry and Slack, and on-premise deployment is thoroughly documented across AWS, Azure and GCP.

But the same page contradicts the marketing blog on compliance. The blog asserts SOC 2 Type II, GDPR and HIPAA. The docs security page documents no certification at all — no SOC 2, no ISO 27001, no HIPAA — and no penetration testing, no vulnerability scanning, no bug bounty, and no data-retention policy; its compliance section covers only a DPA obtainable by email. It also describes environment separation (separate AWS accounts for sandbox and live, IP-allowlisted production access) rather than the per-user tenant isolation the blog claims. Net effect on P11-C-001: upgrade its credential-storage and audit lines from marketing to documentation, and downgrade its compliance line — SOC 2 Type II and HIPAA remain unverified vendor claims and must not be repeated to Cena as fact. Recorded as P11-C-021.

(b) Anon — resolved as unverified, with hard negative evidence. docs.anon.com does not merely 404: it has no DNS records at all (dig +short docs.anon.com returns empty; curl fails with "Could not resolve host"). The marketing site returns 200. A targeted search for Anon's credential architecture returned no Anon-authored documentation, only third-party commentary on agentic auth generally. The "never stores credentials" claim therefore stands as marketing with zero corroborating technical documentation, and the absence of a documentation subdomain is itself a maturity signal. Do not cite Anon's credential posture in any client-facing material; if Anon matters to the design, the claim must come from the vendor under NDA or direct diligence. Recorded as P11-C-022 specifically so the negative result is preserved rather than silently dropped.

A third status finding worth the same treatment: Vessel appears abandoned. docs.vessel.dev returns 404, and app.vessel.dev resolves to Vercel but fails TLS negotiation outright — the application host is misconfigured or expired, not merely empty. The marketing site still returns 200 with dead footer anchors, 2023-dated assets, and a copyright attributed to "Kinit Inc." Exclude it from any build-vs-buy comparison (P11-C-032). A competitor's dead product is useful signal about consolidation in the unified-API category.

Additional innovation candidates

  • P11-I-C-18 Replay receipts. Every deduplicated action returns an explicit

Idempotent-Replayed: true-style marker, so a caller can always distinguish "this ran once and I am seeing the cached result" from "this ran again." Stripe's single most copyable detail: dedupe that is observable rather than merely effectful. Without it a client cannot tell suppression from re-execution, and neither can an auditor.

  • P11-I-C-19 Loud key expiry. Stripe's keys prune after 24h and silently permit

re-execution. Invert it: when a key outlives its window, fail the request rather than treat it as new, forcing the caller to acknowledge it is issuing a fresh intent. Silent lapse of a safety property is the failure mode to design out.

  • P11-I-C-20 Idempotency that survives the pre-execution boundary. Stripe's own

documented weakness is that rate limiting, auth failure and validation all run before the idempotency layer, so those responses are not stable under replay. Admit the key at the edge — before throttling and validation — so a retried request is stable across every rejection path, not only post-execution ones. This is a real improvement over the best contract in the survey, not a reimplementation of it.

  • P11-I-C-21 Intent-derived keys as the default. Stripe permits deriving a key

from a user-attached object; make it the norm. Derive the key from a hash of the action's semantic intent (tenant, tool, resolved arguments) rather than per attempt, so two independent code paths expressing the same intent collide safely by construction instead of relying on the caller to thread one key through both.

  • P11-I-C-22 Mandatory durable run handles with enumerated terminal states. Where

a write-side key is genuinely impossible, require what Skyvern publishes and Browser Use does not: a persistent run_id plus an exhaustive, documented set of terminal states, so post-timeout resolution is always a lookup and never a guess. The minimum acceptable substitute for idempotency, stated as a requirement rather than accepted as a fallback.

  • P11-I-C-23 TTL-scoped tool exposure. Truto's generated MCP servers are scoped by

method, tag and time-to-live. Generalise: every exposed tool surface carries an expiry, applying Vault's lease discipline (P11-I-C-7) to capability rather than credential, so a forgotten integration decays instead of persisting indefinitely.

  • P11-I-C-24 Separate the data-retention claim from the credential-custody claim.

Make the connector manifest state, as distinct machine-checkable fields, where payload data goes and where the credential lives. Hotglue and Truto both let one answer stand in for the other; a schema that cannot express the conflation prevents it.

  • P11-I-C-25 Receipts designed as evidence, not as debugging. Skyvern captures

screenshots and model reasoning per step yet promises no retention, immutability or export. Declare the evidentiary properties explicitly — retention window, tamper-evidence, export format — as part of the receipt contract, so "we have logs" is never mistaken for "we can prove what happened."

  • P11-I-C-26 Cost as a safety surface. Where a runtime bills per action, retry

policy is a spend policy: expose per-tenant retry budgets and hard caps so a retry storm degrades loudly rather than silently invoicing. Corollary from Browser Use's leak-on- crash trap: every billed resource needs a server-side reaper, never client-side teardown as the only stop path.

Remaining unknowns after this pass

Not reached: Kombo's deeper 650-page developer index (auth, rate limits, idempotency and sandbox were absent from the top-level index but were not ruled out — the honest claim is "not surfaced", not "does not exist"), the Kombo OpenAPI schema, Hotglue's docs and security pages, Truto's trust centre and security post, Hyperbrowser's full docs tree, and the Browser Use v4 OpenAPI spec that would settle its lifecycle and idempotency questions definitively. AgentKit rests on a launch announcement plus independent security analysis because the canonical doc URLs 404'd — treat its component detail as announcement-grade, not reference-grade. HCP Vault Dedicated's dynamic-secret and lease parity is strong inference from an explicit binary-parity claim, not an explicit guarantee. Compliance badges across every vendor in both runs remain logos rather than inspected reports.