From ff3eaff13756f9e353b6a685a793c243b785d87e Mon Sep 17 00:00:00 2001 From: Affaan Mustafa Date: Mon, 18 May 2026 04:30:09 -0400 Subject: [PATCH] docs: refresh billing readback gate evidence --- docs/ECC-2.0-GA-ROADMAP.md | 18 +++++++++------- .../publication-evidence-2026-05-18.md | 21 +++++++++++-------- scripts/operator-readiness-dashboard.js | 6 +++--- .../operator-readiness-dashboard.test.js | 2 +- 4 files changed, 27 insertions(+), 20 deletions(-) diff --git a/docs/ECC-2.0-GA-ROADMAP.md b/docs/ECC-2.0-GA-ROADMAP.md index 7bb01779..21830147 100644 --- a/docs/ECC-2.0-GA-ROADMAP.md +++ b/docs/ECC-2.0-GA-ROADMAP.md @@ -287,8 +287,8 @@ As of 2026-05-18: pass the announcement gate yet. - ECC-Tools commit `95d0bec69dbcf364ed084e983a40d0a94d443d16` adds repeatable aggregate production KV readback with - `npm run billing:kv-readback`: the latest run found 252 - `account-billing:*` records and 252 `billing-state:*` records, but 0 + `npm run billing:kv-readback`: the latest API-authenticated run found 253 + `account-billing:*` records and 253 `billing-state:*` records, but 0 Marketplace-managed Pro `billing-state:*` records, so native-payments copy remains blocked until `--require-ready` and the official internal announcement gate pass. @@ -308,14 +308,18 @@ As of 2026-05-18: billing-state provenance and `--require-ready` plus the official internal announcement gate pass. - ECC-Tools commit `632e059e51b6e1297ba118807c8b5b2adbac74ce` - adds target account billing readback with `npm run billing:kv-readback -- --wrangler --account --require-ready`. + adds target account billing readback with `npm run billing:kv-readback -- --account --require-ready`. The report redacts the account login and raw KV keys, emits only a stable fingerprint plus sanitized readiness booleans, and now requires both `account-billing:` and `billing-state:` before a target Marketplace Pro test account can pass the native-payments announcement - readback gate. CI run `26018941515` succeeded. Native-payments copy remains - blocked until the target Marketplace Pro account has webhook provenance and - this target readback plus the official internal announcement gate pass. + readback gate. CI run `26018941515` succeeded. The 2026-05-18 live recheck + confirmed the ECC vault Cloudflare credential works as bearer API auth for + the ECC Tools account/namespace, while local Wrangler OAuth currently fails + with Cloudflare authentication error `10000`. Native-payments copy remains + blocked by Linear ITO-61 until the target Marketplace Pro account has webhook + provenance and this target readback plus the official internal announcement + gate pass. - Handoff `ecc-supply-chain-audit-20260513-0645.md` under `~/.cluster-swarm/handoffs/` records the May 13 supply-chain sweep: no active lockfile/manifest hit for @@ -683,7 +687,7 @@ is not complete unless the evidence column exists and has been freshly verified. | Claude and Codex plugin publication | Contact/submission path with required artifacts and status | Publication readiness, naming matrix, and May 12 dry-run evidence document plugin validation, clean-checkout Claude tag/install smoke, and Codex marketplace CLI shape | Needs explicit approval for real tag/push and marketplace submission | | Articles, tweets, and announcements | X thread, LinkedIn copy, GitHub release copy, push checklist | Draft launch collateral exists under rc.1 release docs | Needs URL-backed refresh | | AgentShield enterprise iteration | Policy gates, SARIF, packs, provenance, corpus, HTML reports, exception lifecycle audit, baseline drift Action/CLI surfaces, evidence-pack redaction, harness adapter registry, enterprise research roadmap, supply-chain hardened release path, CI-safe baseline fingerprints, corpus accuracy recommendations, remediation workflow phases, env proxy hijack corpus coverage, Mini Shai-Hulud full-campaign package IOCs, CI-provenance evidence packs, plugin-cache runtime-confidence triage, evidence-pack consumer readback, fleet-level evidence-pack routing, fleet review items, checksum-backed policy export, checksum-verified policy promotion, policy promotion review items, package-manager hardening drift detection, npm age-gate guidance correction, workflow action-runtime pin refresh, package-manager hardening Action outputs, policy-promotion Action outputs, ECC-Tools hosted consumption of promotion Action outputs, ECC-Tools operator-visible promotion output values, and ECC-Tools hosted promotion judge audit traces | PRs #53, #55-#64, #67-#69, and #78-#92 landed with test evidence, ECC-Tools #76 consumes the fleet-summary output in hosted security review, #77 surfaces source evidence paths in hosted finding output, and #78 links fleet routes to harness owner review; AgentShield #91 adds `agentshield policy export` bundles for branch-protection review and downstream promotion; AgentShield #92 adds `agentshield policy promote` with digest verification, tamper rejection, explicit pack selection, dry-run review, and JSON output before writing active policy; AgentShield commit `87aec47` adds `reviewItems` for digest evidence, owner review, protected rollout PR handoff, and runtime smoke testing with green local and remote CI; AgentShield commit `28d08c7` adds package-manager hardening drift detection for plaintext registry credentials, lifecycle-script enablement, and weak pnpm/Yarn release-age cooldowns with green local and remote CI; AgentShield commit `659f569` refreshes all workflow action runtime pins to SHA-pinned checkout v6.0.2 and setup-node v6.4.0 with green remote CI and no remaining action-runtime deprecation annotation; AgentShield commit `ee585cd` corrects npm release-age guidance by flagging unsupported npm age keys and keeping enforceable cooldown findings on pnpm/Yarn with green local and remote CI; AgentShield commit `1124535` exposes package-manager hardening status/count outputs and a redacted job-summary section for registry credentials, lifecycle scripts, and release-age gates with green local and remote CI; AgentShield commit `1593925` exposes policy-promotion status/count/digest outputs plus job-summary review items for owner approval, protected rollout, and runtime smoke, and marks runtime smoke verified when the same Action job scans with the promoted policy; ECC-Tools commit `8658951` routes those policy-promotion Action outputs into hosted security review findings and Hosted Promotion Readiness scoring; ECC-Tools commit `16c537f` renders policy-promotion status, pack, review item count, action-required count, and digest in hosted security job comments/check-runs; ECC-Tools commit `05d4e82` renders hosted promotion judge request fingerprints and allowed-citation counts without raw provider output; native PDF export deferred in favor of self-contained HTML plus print-to-PDF until explicit enterprise demand appears; `docs/architecture/agentshield-enterprise-research-roadmap.md` now has baseline drift, evidence-pack bundle, redaction, adapter-registry, supply-chain hardening, hashed baseline fingerprints, corpus accuracy recommendation, remediation workflow, env proxy hijack corpus, Mini Shai-Hulud full-campaign package-table, `ci-context.json` provenance, `plugin-cache` confidence, `evidence-pack inspect` readback, `evidence-pack fleet` routing, fleet `reviewItems`, policy export, policy promotion, policy promotion `reviewItems`, package-manager hardening Action outputs, policy-promotion Action outputs, hosted consumption of promotion Action outputs, operator-visible promotion output values, and hosted promotion judge audit traces landed | Next workflow automation should deepen live operator approval/readback after Marketplace/payment gates | -| ECC Tools next-level app | Billing audit, PR checks, deep analyzer, sync backlog, evaluator/RAG corpus, analysis-depth readiness, hosted execution planning, hosted CI diagnostics, hosted security evidence review, hosted harness compatibility audit, hosted reference-set evaluation, hosted AI routing/cost review, hosted team backlog routing, hosted depth-plan check-run, PR-comment hosted job dispatch, hosted job result history/check-runs, hosted result status command, status-aware depth-plan recommendations, hosted promotion readiness, hosted promotion output scoring, hosted promotion retrieval planning, hosted promotion judge contract, gated hosted promotion judge execution, hosted promotion judge audit trace, payment-announcement readiness, billing announcement preflight, aggregate production billing KV readback, Marketplace webhook provenance, target-account billing readback, AgentShield fleet-summary hosted routing, hosted finding evidence paths, harness-route policy linking, policy-promotion Action-output hosted telemetry, and operator-visible promotion output values | PRs #26-#43 plus #53-#78 landed with test evidence, including AgentShield evidence-pack gap routing, canonical bundle recognition, supply-chain signature gates, PR draft follow-up Linear tracking, evidence-backed/deep-ready repository classification, the `/api/analysis/depth-plan` hosted job plan, `/api/analysis/jobs/ci-diagnostics`, `/api/analysis/jobs/security-evidence-review`, `/api/analysis/jobs/harness-compatibility-audit`, `/api/analysis/jobs/reference-set-evaluation`, `/api/analysis/jobs/ai-routing-cost-review`, `/api/analysis/jobs/team-backlog-routing`, the `ECC Tools / Hosted Depth Plan` check-run, `/ecc-tools analyze --job ...` PR-comment dispatch, non-blocking per-hosted-job result check-runs backed by 30-day result cache records, `/ecc-tools analyze --job status` cache lookup, cache-aware next-job recommendations in the depth-plan check-run, the `ECC Tools / Hosted Promotion Readiness` corpus-backed PR check-run, deterministic hosted-output scoring against cached completed job artifacts/findings, ranked retrieval/model-prompt planning, the fail-closed `hosted-promotion-judge.v1` request contract, opt-in live model-judge execution behind hosted evidence, entitlement, budget, provider, executor, strict JSON, and citation gates, hosted promotion judge request fingerprints plus allowed-citation audit trails, a fail-closed `/api/billing/readiness` `announcementGate` for native GitHub payments claims, `npm run billing:announcement-gate` plus `--preflight` as the non-secret operator verifier, hosted security findings for AgentShield fleet summaries, an `Evidence` column in hosted finding comments/check-runs, hosted harness findings that route AgentShield fleet target paths to harness owners, ECC-Tools commit `8658951` routing AgentShield policy-promotion Action outputs into hosted security review and promotion-readiness scoring, ECC-Tools commit `16c537f` rendering policy-promotion status/pack/count/digest values directly in hosted security job comments/check-runs, ECC-Tools commit `05d4e82` rendering model-judge audit traces without exposing raw provider output, ECC-Tools commit `91a441b` adding the safe billing announcement preflight path, ECC-Tools commit `eb69412` recording the initial production readback state, ECC-Tools commit `95d0bec` adding `npm run billing:kv-readback` with 252 aggregate account-billing and billing-state records but 0 Marketplace Pro billing-state records, ECC-Tools commit `2859678` requiring webhook-derived Marketplace provenance before announcement readiness, ECC-Tools commit `42653f9` adding Wrangler OAuth readback with live aggregate evidence of 253 account-billing records, 253 billing-state records, and 0 ready-like Marketplace Pro states, and ECC-Tools commit `632e059` adding sanitized target-account readback that requires both target key families before `--require-ready` can pass | Next work is create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --wrangler --account --require-ready` and the live announcement gate | +| ECC Tools next-level app | Billing audit, PR checks, deep analyzer, sync backlog, evaluator/RAG corpus, analysis-depth readiness, hosted execution planning, hosted CI diagnostics, hosted security evidence review, hosted harness compatibility audit, hosted reference-set evaluation, hosted AI routing/cost review, hosted team backlog routing, hosted depth-plan check-run, PR-comment hosted job dispatch, hosted job result history/check-runs, hosted result status command, status-aware depth-plan recommendations, hosted promotion readiness, hosted promotion output scoring, hosted promotion retrieval planning, hosted promotion judge contract, gated hosted promotion judge execution, hosted promotion judge audit trace, payment-announcement readiness, billing announcement preflight, aggregate production billing KV readback, Marketplace webhook provenance, target-account billing readback, AgentShield fleet-summary hosted routing, hosted finding evidence paths, harness-route policy linking, policy-promotion Action-output hosted telemetry, and operator-visible promotion output values | PRs #26-#43 plus #53-#78 landed with test evidence, including AgentShield evidence-pack gap routing, canonical bundle recognition, supply-chain signature gates, PR draft follow-up Linear tracking, evidence-backed/deep-ready repository classification, the `/api/analysis/depth-plan` hosted job plan, `/api/analysis/jobs/ci-diagnostics`, `/api/analysis/jobs/security-evidence-review`, `/api/analysis/jobs/harness-compatibility-audit`, `/api/analysis/jobs/reference-set-evaluation`, `/api/analysis/jobs/ai-routing-cost-review`, `/api/analysis/jobs/team-backlog-routing`, the `ECC Tools / Hosted Depth Plan` check-run, `/ecc-tools analyze --job ...` PR-comment dispatch, non-blocking per-hosted-job result check-runs backed by 30-day result cache records, `/ecc-tools analyze --job status` cache lookup, cache-aware next-job recommendations in the depth-plan check-run, the `ECC Tools / Hosted Promotion Readiness` corpus-backed PR check-run, deterministic hosted-output scoring against cached completed job artifacts/findings, ranked retrieval/model-prompt planning, the fail-closed `hosted-promotion-judge.v1` request contract, opt-in live model-judge execution behind hosted evidence, entitlement, budget, provider, executor, strict JSON, and citation gates, hosted promotion judge request fingerprints plus allowed-citation audit trails, a fail-closed `/api/billing/readiness` `announcementGate` for native GitHub payments claims, `npm run billing:announcement-gate` plus `--preflight` as the non-secret operator verifier, hosted security findings for AgentShield fleet summaries, an `Evidence` column in hosted finding comments/check-runs, hosted harness findings that route AgentShield fleet target paths to harness owners, ECC-Tools commit `8658951` routing AgentShield policy-promotion Action outputs into hosted security review and promotion-readiness scoring, ECC-Tools commit `16c537f` rendering policy-promotion status/pack/count/digest values directly in hosted security job comments/check-runs, ECC-Tools commit `05d4e82` rendering model-judge audit traces without exposing raw provider output, ECC-Tools commit `91a441b` adding the safe billing announcement preflight path, ECC-Tools commit `eb69412` recording the initial production readback state, ECC-Tools commit `95d0bec` adding `npm run billing:kv-readback` with 253 aggregate account-billing and billing-state records but 0 Marketplace Pro billing-state records, ECC-Tools commit `2859678` requiring webhook-derived Marketplace provenance before announcement readiness, ECC-Tools commit `42653f9` adding Wrangler OAuth readback with live aggregate evidence of 253 account-billing records, 253 billing-state records, and 0 ready-like Marketplace Pro states, and ECC-Tools commit `632e059` adding sanitized target-account readback that requires both target key families before `--require-ready` can pass; the 2026-05-18 live recheck used the ECC vault Cloudflare bearer token successfully and split out Linear ITO-61 for the target-account blocker because local Wrangler OAuth currently fails with Cloudflare authentication error `10000` | Next work is create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --account --require-ready` with working Cloudflare API auth or repaired Wrangler OAuth, followed by the live announcement gate | | GitGuardian/Dependabot/CodeRabbit-style checks | Non-blocking taxonomy, deterministic follow-up checks, and local supply-chain gates | ECC-Tools risk taxonomy check plus follow-up signals landed, including Skill Quality, Deep Analyzer Evidence, Analyzer Corpus Evidence, RAG/Evaluator Evidence, PR Review/Salvage Evidence, and AgentShield evidence-pack evidence; #1846 added npm registry signature gates; #1848 added the supply-chain incident-response playbook and `pull_request_target` cache-poisoning validator guard; #1851 added the privileged checkout credential-persistence guard; AgentShield #78, JARVIS #13, and ECC-Tools #53 applied the same hardening outside trunk | Current supply-chain gate complete; deeper hosted review features remain future | | Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates plus `docs/architecture/evaluator-rag-prototype.md`, `examples/evaluator-rag-prototype/`, and ECC-Tools PR #40 define read-only stale-salvage, billing-readiness, CI-failure-diagnosis, harness-config-quality, AgentShield policy-exception, skill-quality evidence, deep-analyzer evidence, and RAG/evaluator comparison scenarios with trace, report, playbook, verifier, and predictive-check artifacts; ECC-Tools PRs #68-#72 now turn that corpus into a deterministic PR check-run gate with cached hosted-output scoring, ranked retrieval candidates, a model prompt seed, a fail-closed hosted model-judge request contract, and opt-in live model execution behind strict hosted-evidence gates | Deterministic hosted PR check, cached output scoring, retrieval planning, judge contract, and gated model execution integrated | | Linear roadmap is detailed | Linear project status plus repo mirror | Repo mirror exists; issue creation was retried on 2026-05-12 and remains blocked by the workspace free issue limit; the May 18 sync adds queue-zero/work-items state, #1970/#1971/#1972/#1976 merge evidence, ITO-57 current-head supply-chain refresh comment `0b9931b9-1556-4ebc-a70c-f3635557625d`, ITO-57 defensive-deny scanner recheck reply `6fa15367-d994-4e53-ade3-9462477e1100`, ECC platform progress comment `e32e5b7a-287b-4bf4-9ed7-314389a157e1`, and generated `operator:dashboard` prompt-to-artifact audit for recurring status updates | Needs recurring status updates after each significant merge batch | diff --git a/docs/releases/2.0.0-rc.1/publication-evidence-2026-05-18.md b/docs/releases/2.0.0-rc.1/publication-evidence-2026-05-18.md index ddbfca61..4e2a2403 100644 --- a/docs/releases/2.0.0-rc.1/publication-evidence-2026-05-18.md +++ b/docs/releases/2.0.0-rc.1/publication-evidence-2026-05-18.md @@ -87,15 +87,18 @@ Tracked repositories in the platform audit and work-items sync were: - ECC Tools billing/native-payments copy remains blocked until a Marketplace Pro purchase/webhook path writes ready production `billing-state:*` provenance for the target Marketplace test account, then - `npm run billing:kv-readback -- --wrangler --account --require-ready` - and `npm run billing:announcement-gate -- --account ` return - announcement-ready gates. The latest Wrangler OAuth aggregate readback from - ECC-Tools commit `42653f9` found 253 `account-billing:*` records, 253 - `billing-state:*` records, 0 Marketplace Pro states, 0 ready-like - Marketplace Pro states, and 0 parse failures. ECC-Tools commit `632e059` - adds the follow-up target-account readback mode, redacts the account login - and raw KV key names, and requires both target key families before - `--require-ready` can pass. + `npm run billing:kv-readback -- --account --require-ready` + with working Cloudflare API auth or repaired Wrangler OAuth, followed by + `npm run billing:announcement-gate -- --account `, return + announcement-ready gates. The latest API-authenticated aggregate readback + from the ECC vault Cloudflare credential found 253 `account-billing:*` + records, 253 `billing-state:*` records, 0 Marketplace Pro states, 0 + ready-like Marketplace Pro states, and 0 parse failures; local Wrangler OAuth + currently fails with Cloudflare authentication error `10000`. ECC-Tools + commit `632e059` adds the follow-up target-account readback mode, redacts + the account login and raw KV key names, and requires both target key families + before `--require-ready` can pass. Linear ITO-61 now tracks the exact + target-account acceptance criteria. - Release notes, X, LinkedIn, GitHub release, and longform copy still need final live URLs after release/package/plugin URLs exist. - The local checkout still has unrelated untracked `docs/drafts/`, so a strict diff --git a/scripts/operator-readiness-dashboard.js b/scripts/operator-readiness-dashboard.js index 954b8505..a8b11b37 100644 --- a/scripts/operator-readiness-dashboard.js +++ b/scripts/operator-readiness-dashboard.js @@ -378,12 +378,12 @@ function eccToolsNextLevelEvidence(roadmap) { function eccToolsNextLevelGap(roadmap) { if (roadmap.includes('target account billing readback') || roadmap.includes('632e059')) { - return 'create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --wrangler --account --require-ready` and the live announcement gate'; + return 'create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --account --require-ready` with working Cloudflare API auth or repaired Wrangler OAuth, followed by the live announcement gate'; } if (roadmap.includes('Wrangler OAuth readback') || roadmap.includes('42653f9')) { - return 'create or verify Marketplace-managed Pro billing-state with webhook provenance, then run `billing:kv-readback -- --wrangler --require-ready` and the live announcement gate'; + return 'create or verify Marketplace-managed Pro billing-state with webhook provenance, then run `billing:kv-readback -- --require-ready` with working Cloudflare API auth or repaired Wrangler OAuth, followed by the live announcement gate'; } if (roadmap.includes('Marketplace webhook provenance') @@ -760,7 +760,7 @@ function buildReport(options) { next_work_order: [ 'Regenerate this dashboard from the final release commit before publication evidence is recorded.', 'Repeat ITO-57 Linear/project status sync after the next significant merge batch or advisory-source refresh.', - 'Create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --wrangler --account --require-ready` and the live announcement gate before publishing native-payments copy.', + 'Create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --account --require-ready` with working Cloudflare API auth or repaired Wrangler OAuth, followed by the live announcement gate before publishing native-payments copy.', 'Resume ITO-45, ITO-46, and ITO-56 only after the generated dashboard and final release gates are refreshed.', ], }; diff --git a/tests/scripts/operator-readiness-dashboard.test.js b/tests/scripts/operator-readiness-dashboard.test.js index 098ac762..c59b5cc2 100644 --- a/tests/scripts/operator-readiness-dashboard.test.js +++ b/tests/scripts/operator-readiness-dashboard.test.js @@ -264,7 +264,7 @@ function runTests() { ))); assert.ok(report.requirements.some(item => ( item.id === 'ecc-tools-next-level' - && item.gap === 'create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --wrangler --account --require-ready` and the live announcement gate' + && item.gap === 'create or verify Marketplace-managed Pro target billing-state with webhook provenance, then run `billing:kv-readback -- --account --require-ready` with working Cloudflare API auth or repaired Wrangler OAuth, followed by the live announcement gate' && item.evidence.includes('operator-visible promotion output details') && item.evidence.includes('hosted promotion judge audit traces') && item.evidence.includes('billing announcement preflight')