P15 — Continuous corpus and production learning: research report

Part: P15 · Lane: S1-L5 · Run: 2026-08-27-sprint-1-fable Author: ACTIONIST-S1-L5-RUNTIME · Recorded 2026-08-27 Status: research only · DESIGN_ONLY · UNEXECUTED · NOT_ADMITTED · admitted_blocks 0 Depth contract: OSS met, commercial partially met — exact denominators in §2.

1. Result in one paragraph

P15's design is determined less by which learning technique is best than by three facts about the regime it must operate in: there is currently no production learning data of any kind, the data that will arrive is sparse and expensive (tens of builds, not millions of events), and it is partitioned across contractually-private clients. That combination disqualifies most of the external prior art's machinery and makes cold start the only case that currently exists. The most useful external analogues turned out not to be recommender systems but registry-health scorers, which score software assets on evidence rather than users on preferences. The most important design decision is imported from the local estate and contradicts the entire recommender literature: ranking must be min-gated on the weakest evidence family, never a weighted average — because averaging is precisely the mechanism that would let an asset with unresolved rights outrank one with clean rights.

2. Denominators — exact

2a. The not_applicable determination

There is no coherent commercial market of "production learning loop vendors." The ~100 denominator cannot be built without padding. Three independently checkable strands support this:

1. A retired flagship with no managed successor. Azure AI Personalizer was the one major cloud

vendor's general-purpose "send decisions, send rewards, we update the policy" service. Microsoft closed it to new resources on 20 September 2023 and retired it on 25 August 2026 — two days before this survey. The published migration path is not another managed service but an open-source repository, microsoft/learning-loop. A category whose most prominent managed offering is decommissioned in favour of "run it yourself" is a market failing to sustain, not maturing.

1. The function is always a feature, never a product. Every genuine production-learning loop found

is embedded in something sold as another category: Renovate's merge confidence inside a dependency-update bot, v0's autofixer inside a site builder, Algolia Dynamic Re-Ranking inside search. There is no analyst category, no comparison grid, no competitors positioning against each other on quality of production feedback loop. A ~100 list would be software vendors that happen to contain a loop — a list, not a market.

1. The nearest true analogue is deliberately unbuilt. The closest thing to Actionist's need —

reranking reusable software assets by how they performed in real builds — is served by package-health scoring, and the two most authoritative players (Google's deps.dev, OpenSSF Scorecard) contain no outcome feedback whatsoever; they are static analysis of repository state. The one vendor pooling real production outcomes across projects, Mend's Renovate merge confidence, keeps its algorithm private.

Built instead: 22 records across seven categories weighted toward P15's actual sub-questions — registry_health 6, bandits 4, builder_loop 3, marketplace_ranking 3, ml_eval 2, privacy_aggregation 2, ranking_service 2. Negative results recorded as first-class findings.

This strengthens rather than weakens D-P15-10 (own the mechanism, treat vendors as replaceable): the category's flagship was retired mid-project.

OSS category spread: bayesian_preference 14, privacy_telemetry 12, bandits 11, llm_eval_loop 11, experimentation 10, recommender 10, asset_scoring 10, drift_monitoring 8, edit_learning 6, registry_ranking 5. Evidence classes: observed_behavior 62, first_party_docs 31, secondary 4.

The commercial shortfall is the account-wide quota that killed wave-1 after 5 records. This is the lane's weakest survey and the gap is recorded rather than concealed.

3. The starting position, stated plainly

Zero accepted plans. Zero recorded client edits. Zero incidents. Zero maintenance outcomes. Zero admitted blocks. The local AutoSaaS loop is procedural, not measured — its own maturity section records no machine-readable template registry, no SaaS built by the method, no automated freshness check, no command runner. Ledger entry A37 remains a hypothesis.

P15 is therefore not a system to be tuned into correctness. It must be correct on its first observation and still correct at ten.

4. The regime disqualifies most prior art

Surviving this test: Bayesian and preference-learning methods — Bradley-Terry/Luce (choix), Thompson sampling over posteriors, hierarchical priors (PyMC), Bayesian optimisation (BoTorch/Ax), and bandit libraries validated in small-data settings (MABWiser, Vowpal Wabbit, Open Bandit Pipeline). Failing it: frequentist A/B testing on tens of builds.

5. The scoring shape comes from registries, not recommenders

The strongest documented-methodology analogues are registry-health scorers — OpenSSF Scorecard, ecosyste.ms, Libraries.io SourceRank, Renovate merge-confidence — because they score software assets on evidence, which is structurally the same problem. Renovate's merge-confidence is the closest single precedent: production outcomes of an upgrade feed a recommendation about that upgrade.

The recommender literature contributes cold-start technique (LightFM's hybrid content/collaborative approach). It does not contribute the scoring shape.

6. Min-gating — the deliberate divergence

The Universal Block Framework's anti-averaging rule: evidence_tier may never exceed the minimum tier across the nine families — "A single T1 family pins the whole block at T1 regardless of eight T4s."

Recommenders compute weighted scores by design, so that strength in one dimension compensates for weakness in another. For films that is correct. For a capability being embedded into a client's production system it is catastrophic: an asset with excellent adaptation cost, excellent workflow success and unresolved rights must not outrank a merely-good asset with clean rights.

P15 ranking is min-gated on the weakest evidence family. This is a deliberate divergence from the external census, not an oversight.

One risk, recorded as AP15-7: min-gating could be degenerate if every asset has one weak family. If so, the ranking's job changes from ordering assets to naming the missing evidence — arguably more useful at this stage anyway. E-P15-1 tests this before any live use.

7. Signals come free from P13 and P14

The most efficient finding: designed properly, P13 and P14 already emit P15's inputs. No separate telemetry programme is needed for the primary signals.

Two deserve emphasis. UNDERDETERMINED aggregates are contract defects, not model failures — each names a field the Block Contract is missing, redirecting effort to where the defect actually is. Falsified-belief records are a first-class output — AutoSaaS §8 lists "failed assumptions" among required updates, and nothing in the external census records which belief an outcome falsified. That is a genuinely local idea and a candidate differentiator.

8. Exploration inside a safety envelope

Pure exploitation makes the shelf a ratchet: the first-ranked asset stays ranked because nothing else is tried. But exploration here means trying a less-proven capability on a paying client's project. Acceptable only when: candidates pass every zero-tolerance gate (rights, tenancy, security); never on regulated workloads; bounded declared fraction of builds; human admission decision retained.

Exploration is permitted over quality, never over safety.

9. Catalogue size is not supply

OpenConnector generated 1,445 provider definitions and 15,156 actions locally, and its global connection storage failed the required tenancy model. A ranking signal derived from catalogue breadth would have ranked it top. Related local facts: 8,515 joined UI identities of which only 3,506 are source-bearing; 270 of 1,700 industry-repository pairs complete (1,430 deficit); the ~1.3M index and 850k/80k Mini corpus have no authoritative path (A20/A21 unknown).

Consequence: refresh cadence must be designed against the assets actually in hand, treating the large corpus as an optional feed that may not exist.

10. Negative market signal — verified

Azure AI Personalizer: closed to new resources 20 September 2023, retired 25 August 2026, with Microsoft directing users to the open-source microsoft/learning-loop repository rather than a successor service. The implication is not that contextual bandits are unsound — the OSS census found the technique alive and well maintained — but that this capability must be owned rather than bought. A dependency on a managed learning-loop service is a dependency on a category that has demonstrated it does not sustain.

11. Top 10 OSS projects

VowpalWabbit · BoTorch · MABWiser · Ax · choix · PyMC · zr-obp (Open Bandit Pipeline) · openskill.py · LightFM · GrowthBook. Selected for transferability to sparse asset-reranking with cold starts, not for popularity.

12. The largest commercial risk

Cross-client aggregation may be contractually unavailable (X-P15-2). P15's value proposition assumes learning aggregates across clients. If contracts forbid it, P15 degrades to per-tenant learning, where tens of observations become ones and the sparse-data problem becomes far worse.

This is unknown, not merely unmeasured — and it is resolvable by a conversation rather than code. E-P15-5 (establish with the client owner whether anonymised cross-client aggregation is contractually available) is the cheapest high-value experiment in the entire lane and gates the whole premise.

13. What this report does not establish

• No learning loop exists. Nothing has been ranked or reranked.
• A37 (production learning improves reuse decisions) remains a hypothesis.
• The commercial denominator is 22 of ~100, and the not_applicable determination for a pure

"learning-loop vendor" category was a wave-2 deliverable. Depth contract unmet for that survey.

• No claim about the 1.3M/850k corpus (A20/A21 unknown).
• No claim that clients will permit cross-client aggregation.
• Azure Personalizer retirement status flagged for verification.