From 5157ee63f091057ee73828b2a6aba0cc0b68f531 Mon Sep 17 00:00:00 2001 From: Affaan Mustafa Date: Fri, 15 May 2026 17:58:02 -0400 Subject: [PATCH] Generate operator readiness dashboard --- docs/ECC-2.0-GA-ROADMAP.md | 9 +- ...operator-readiness-dashboard-2026-05-15.md | 141 +++++------------- .../2.0.0-rc.1/publication-readiness.md | 2 +- 3 files changed, 49 insertions(+), 103 deletions(-) diff --git a/docs/ECC-2.0-GA-ROADMAP.md b/docs/ECC-2.0-GA-ROADMAP.md index dd043ce5..35cf697d 100644 --- a/docs/ECC-2.0-GA-ROADMAP.md +++ b/docs/ECC-2.0-GA-ROADMAP.md @@ -40,6 +40,11 @@ As of 2026-05-15: - `npm run harness:audit -- --format json` reports 70/70 on current `main`. - `npm run observability:ready` reports 21/21 readiness on current `main`, including the GitHub/Linear/handoff/roadmap progress-sync contract. +- `npm run operator:dashboard -- --allow-untracked docs/drafts/ --write + docs/releases/2.0.0-rc.1/operator-readiness-dashboard-2026-05-15.md` + now regenerates the ITO-44 prompt-to-artifact dashboard from live + `platform:audit` evidence instead of leaving the operator snapshot as a + hand-maintained document. - PR #1846 merged as `797f283036904128bb1b348ae62019eb9f08cf39` and made npm registry signature verification a durable workflow-security gate: workflows that run `npm audit` now need `npm audit signatures`. @@ -505,7 +510,7 @@ is not complete unless the evidence column exists and has been freshly verified. | ECC Tools next-level app | Billing audit, PR checks, deep analyzer, sync backlog, evaluator/RAG corpus, analysis-depth readiness, hosted execution planning, hosted CI diagnostics, hosted security evidence review, hosted harness compatibility audit, hosted reference-set evaluation, hosted AI routing/cost review, hosted team backlog routing, hosted depth-plan check-run, PR-comment hosted job dispatch, hosted job result history/check-runs, hosted result status command, status-aware depth-plan recommendations, hosted promotion readiness, hosted promotion output scoring, hosted promotion retrieval planning, hosted promotion judge contract, gated hosted promotion judge execution, payment-announcement readiness | PRs #26-#43 plus #53-#74 landed with test evidence, including AgentShield evidence-pack gap routing, canonical bundle recognition, supply-chain signature gates, PR draft follow-up Linear tracking, evidence-backed/deep-ready repository classification, the `/api/analysis/depth-plan` hosted job plan, `/api/analysis/jobs/ci-diagnostics`, `/api/analysis/jobs/security-evidence-review`, `/api/analysis/jobs/harness-compatibility-audit`, `/api/analysis/jobs/reference-set-evaluation`, `/api/analysis/jobs/ai-routing-cost-review`, `/api/analysis/jobs/team-backlog-routing`, the `ECC Tools / Hosted Depth Plan` check-run, `/ecc-tools analyze --job ...` PR-comment dispatch, non-blocking per-hosted-job result check-runs backed by 30-day result cache records, `/ecc-tools analyze --job status` cache lookup, cache-aware next-job recommendations in the depth-plan check-run, the `ECC Tools / Hosted Promotion Readiness` corpus-backed PR check-run, deterministic hosted-output scoring against cached completed job artifacts/findings, ranked retrieval/model-prompt planning, the fail-closed `hosted-promotion-judge.v1` request contract, opt-in live model-judge execution behind hosted evidence, entitlement, budget, provider, executor, strict JSON, and citation gates, a fail-closed `/api/billing/readiness` `announcementGate` for native GitHub payments claims, and `npm run billing:announcement-gate` as the non-secret operator verifier | Next work is hosted promotion telemetry, operator review UX, and live Marketplace test-account readback | | GitGuardian/Dependabot/CodeRabbit-style checks | Non-blocking taxonomy, deterministic follow-up checks, and local supply-chain gates | ECC-Tools risk taxonomy check plus follow-up signals landed, including Skill Quality, Deep Analyzer Evidence, Analyzer Corpus Evidence, RAG/Evaluator Evidence, PR Review/Salvage Evidence, and AgentShield evidence-pack evidence; #1846 added npm registry signature gates; #1848 added the supply-chain incident-response playbook and `pull_request_target` cache-poisoning validator guard; #1851 added the privileged checkout credential-persistence guard; AgentShield #78, JARVIS #13, and ECC-Tools #53 applied the same hardening outside trunk | Current supply-chain gate complete; deeper hosted review features remain future | | Harness-agnostic learning system | Audit, adapter matrix, observability, traces, promotion loop | Audit/adapters/observability gates plus `docs/architecture/evaluator-rag-prototype.md`, `examples/evaluator-rag-prototype/`, and ECC-Tools PR #40 define read-only stale-salvage, billing-readiness, CI-failure-diagnosis, harness-config-quality, AgentShield policy-exception, skill-quality evidence, deep-analyzer evidence, and RAG/evaluator comparison scenarios with trace, report, playbook, verifier, and predictive-check artifacts; ECC-Tools PRs #68-#72 now turn that corpus into a deterministic PR check-run gate with cached hosted-output scoring, ranked retrieval candidates, a model prompt seed, a fail-closed hosted model-judge request contract, and opt-in live model execution behind strict hosted-evidence gates | Deterministic hosted PR check, cached output scoring, retrieval planning, judge contract, and gated model execution integrated | -| Linear roadmap is detailed | Linear project status plus repo mirror | Repo mirror exists; issue creation was retried on 2026-05-12 and remains blocked by the workspace free issue limit; this May 15 sync adds ECC #1860, AgentShield #78-#86, JARVIS #13, ECC-Tools #53-#74, resolved queue/discussion counts, and notes that Linear connector status updates after ECC-Tools #68 remain blocked by a connector secret-owner error | Needs recurring status updates after connector recovery | +| Linear roadmap is detailed | Linear project status plus repo mirror | Repo mirror exists; issue creation was retried on 2026-05-12 and remains blocked by the workspace free issue limit; this May 15 sync adds ECC #1860, AgentShield #78-#86, JARVIS #13, ECC-Tools #53-#74, resolved queue/discussion counts, and a generated `operator:dashboard` prompt-to-artifact audit for recurring status updates | Needs recurring status updates after connector recovery | | Flow separation and progress tracking | Flow lanes with owner artifacts and update cadence | This roadmap defines lanes below and `docs/architecture/progress-sync-contract.md` makes GitHub/Linear/handoff/roadmap sync part of the readiness gate | Active | | Realtime Linear sync | Project updates while issue limit is blocked; issues later | ECC-Tools #39 implements opt-in Linear API sync for deferred follow-up backlog items, and ECC-Tools #54 adds copy-ready PR drafts to that backlog when draft PR shells are not opened; `docs/architecture/progress-sync-contract.md` defines the local file-backed realtime boundary while issue capacity is blocked | Needs workspace capacity/config rollout | | Observability for self-use | Local readiness gate, traces, status snapshots, HUD/status contract, risk ledger, progress-sync contract | `npm run observability:ready` reports 21/21 | Complete for local gate | @@ -527,7 +532,7 @@ repo evidence and merge commits. | Evaluation and RAG | Reference-set validation, harness audit, traces, ECC-Tools corpus | Read-only evaluator/RAG prototype plus stale-salvage, billing-readiness, CI-failure-diagnosis, harness-config-quality, AgentShield policy-exception, skill-quality evidence, deep-analyzer evidence, and RAG/evaluator comparison fixtures; ECC-Tools #68 publishes the corpus as a hosted promotion readiness check-run, #69 scores cached hosted job outputs against the same corpus, #70 emits ranked retrieval candidates plus a model prompt seed, #71 adds a fail-closed hosted model-judge request contract, and #72 executes that judge only when explicitly enabled and backed by hosted retrieval citations | Hosted promotion telemetry and operator review UX | | AgentShield enterprise | AgentShield PR evidence and roadmap notes | Remediation workflow depth or corpus expansion follow-up | Next implementation batch | | ECC Tools app | ECC-Tools PR evidence, billing audit, risk taxonomy, evaluator/RAG corpus | ECC-Tools #53 published the supply-chain workflow hardening branch, #54 tracks copy-ready PR drafts in the Linear/project backlog, #55 classifies analysis-depth readiness, #56 exposes the hosted execution plan, #57 executes the first hosted CI diagnostics job, #58 executes the hosted security evidence review job, #59 executes the hosted harness compatibility audit, #60 executes the hosted reference-set evaluation, #61 executes the hosted AI routing/cost review, #62 executes hosted team backlog routing, #63 publishes the hosted depth-plan check-run, #64 dispatches hosted jobs from PR comments, #65 persists hosted result history/check-runs, #66 exposes hosted job status from PR comments, #67 makes depth-plan recommendations cache-aware, #68 publishes hosted promotion readiness from the evaluator/RAG corpus, #69 scores cached hosted job outputs against that corpus, #70 emits ranked retrieval candidates plus a model prompt seed, #71 emits the gated `hosted-promotion-judge.v1` contract without live model calls, #72 adds opt-in live model-judge execution behind hosted-evidence and strict JSON/citation gates, #73 adds a fail-closed native-payments `announcementGate` to billing readiness, and #74 adds `npm run billing:announcement-gate` for operator verification | Live Marketplace test-account readback and hosted promotion telemetry | -| Linear progress | Linear project status updates, `docs/architecture/progress-sync-contract.md`, and this mirror | Status update with queue/evidence/missing gates | Every significant merge batch | +| Linear progress | Linear project status updates, `docs/architecture/progress-sync-contract.md`, generated `operator:dashboard` output, and this mirror | Status update with queue/evidence/missing gates | Every significant merge batch | The project status update should always include: diff --git a/docs/releases/2.0.0-rc.1/operator-readiness-dashboard-2026-05-15.md b/docs/releases/2.0.0-rc.1/operator-readiness-dashboard-2026-05-15.md index 6170c263..1b584101 100644 --- a/docs/releases/2.0.0-rc.1/operator-readiness-dashboard-2026-05-15.md +++ b/docs/releases/2.0.0-rc.1/operator-readiness-dashboard-2026-05-15.md @@ -1,114 +1,55 @@ -# ECC Operator Readiness Dashboard - 2026-05-15 +# ECC Operator Readiness Dashboard -This dashboard is an operator snapshot, not a release approval. Use it to decide -the next ECC 2.0 work batch and to keep Linear, GitHub, and repo evidence in -sync. Before publishing, repeat the checks from the final release commit in a -clean checkout. +This dashboard is generated by `npm run operator:dashboard`. It is an operator snapshot, not release approval. + +Generated: 2026-05-15T21:57:15.388Z +Commit: 3fea237141daf160452b4073bc262fbbb87c3d34 +Status: work remaining ## Current Status | Area | Status | Evidence | | --- | --- | --- | -| PR queue | Current | 0 open PRs across checked repos | -| Issue queue | Current | 0 open issues across checked repos | -| Discussions | Current | 58 main-repo discussions; 0 need maintainer touch; 0 answerable discussions missing accepted answers | -| Local worktree | Current with caveat | `main...origin/main`; unrelated `?? docs/drafts/` ignored | -| Security sweep | Current with follow-up | IOC scan, audits, package-manager hardening, and scheduled watch workflow completed | -| Linear roadmap | Current with follow-up | `ECC Platform Roadmap`, ITO-44 through ITO-59 | -| ECC 2.0 publication | Not complete | Release, npm, plugin, and announcement gates pending | -| AgentShield enterprise depth | In progress | AgentShield #86 merged; live IOC loop still pending | -| ECC Tools next-level app | In progress | Billing announcement gate merged; live readback pending | -| Legacy audit and salvage | Not complete | ITO-55 remains open | - -## Live Command Evidence - -Run these from `everything-claude-code` unless a row says otherwise. - -| Evidence | Command | 2026-05-15 result | -| --- | --- | --- | -| Platform audit | `node scripts/platform-audit.js --json --allow-untracked docs/drafts/` | `ready: true`; open PRs 0/20; open issues 0/20; discussions needing maintainer touch 0; answerable discussions missing accepted answers 0; blocking dirty files 0 | -| Discussion audit | `node scripts/discussion-audit.js --json --repo affaan-m/everything-claude-code` | `ready: true`; 58 discussions sampled; 0 need maintainer touch; 0 answerable discussions missing accepted answers | -| Main repo status | `git status --short --branch` | `## main...origin/main`; `?? docs/drafts/` remains unrelated | -| Main commit | `git rev-parse HEAD` | `f7035b5644ffc857879b71c39353b2141f17c3f0` | -| Main repo PRs/issues | GitHub connector and `gh` readback | 0 open PRs; 0 open issues | -| AgentShield PRs/issues | GitHub connector and `gh` readback | 0 open PRs; 0 open issues | -| ECC Tools PRs/issues | Local `gh pr list` and `gh issue list` | 0 open PRs; 0 open issues | -| Discussion baseline | GraphQL discussion sweep | Main repo #1923 marked answered; no answerable Q&A missing an answer | -| Supply-chain IOC scan | `node scripts/ci/scan-supply-chain-iocs.js --root --home` | Passed; repo/home targeted scan inspected 229 files after clean no-script reinstall | -| IOC unit tests | `node tests/ci/scan-supply-chain-iocs.test.js` | 15/15 passed | -| Dead-man switch persistence sweep | Process, LaunchAgent, and known payload filename sweep for Mini Shai-Hulud markers | No matches | -| Workflow security gate | `node scripts/ci/validate-workflow-security.js` | Passed; 8 workflow files inspected; package-manager test installs disable lifecycle scripts and no Actions cache use remains | -| Supply-chain watch workflow | `.github/workflows/supply-chain-watch.yml` | Scheduled every 6 hours; emits `supply-chain-ioc-report.json` | -| npm signatures and audit | `npm audit signatures && npm audit --audit-level=high` in main | 241 verified signatures, 30 verified attestations, 0 high vulnerabilities | +| PR queue | Current | 0 open PRs across tracked repos | +| Issue queue | Current | 0 open issues across tracked repos | +| Discussions | Current | 0 need maintainer touch; 0 missing accepted answer | +| Local worktree | Current | 0 blocking dirty files; 1 ignored dirty entries | +| Dashboard generation | Current | platform audit ready: true; GitHub skipped: false | +| Publication | Not complete | release, npm, plugin, billing, and announcement gates are tracked below | ## Prompt-To-Artifact Checklist -| Objective requirement | Artifact or evidence | Status | Gap | -| --- | --- | --- | --- | -| Keep PRs under 20 | `scripts/platform-audit.js`; live GitHub readback | Current | Repeat before release | -| Keep issues under 20 | `scripts/platform-audit.js`; live GitHub readback | Current | Repeat before release | -| Respond and manage discussions | `scripts/discussion-audit.js`; #1923 answer marked | Current | Repeat before release | -| ECC 2.0 preview pack ready | `preview-pack-manifest.md`; `publication-readiness.md` | In progress | Final publish evidence still pending | -| Include Hermes specialized skills | `docs/HERMES-SETUP.md`; `skills/hermes-imports/SKILL.md` | In progress | Final preview-pack smoke still pending | -| Name-change and availability path | `naming-and-publication-matrix.md`; ITO-46 | In progress | Final name/package/channel choice not approved | -| Claude plugin publication path | `.claude-plugin/`; `publication-readiness.md`; ITO-46 | In progress | Actual publication still pending | -| Codex plugin publication path | `.codex-plugin/`; repo marketplace evidence; ITO-46 | In progress | Official directory path still pending | -| Release notes and push notifications | `release-notes.md`; `x-thread.md`; `linkedin-post.md`; ITO-47/56 | In progress | Live URLs and publish approval missing | -| AgentShield enterprise iteration | AgentShield PRs #83-#86; ITO-48/49 | In progress | Live IOC update loop and cross-harness depth pending | -| ECC Tools native payments announcement | ECC-Tools #75; ITO-50 | In progress | Live Marketplace test-account readback pending | -| ECC Tools AI-native harness-agnostic roadmap | ITO-51/52/53/54 | In progress | Implementation and hosted deep-analysis proof pending | -| Linear roadmap extremely detailed | Linear `ECC Platform Roadmap`; ITO-44 through ITO-59 | Current | Keep status comments synchronized | -| Legacy work audited, pruned, or attached | `docs/legacy-artifact-inventory.md`; ITO-55 | In progress | Final salvage/prune pass pending | -| Realtime progress tracking with Linear | ITO-54; Linear progress comments | In progress | Productized sync/observability plane pending | -| ECC 2.0 observability | `docs/architecture/observability-readiness.md`; ITO-54 | In progress | Runtime/dashboard implementation pending | +| Objective requirement | Artifact or gate | Status | Evidence | Gap | +| --- | --- | --- | --- | --- | +| Keep public PRs below 20 | scripts/platform-audit.js live GitHub sweep | current | 0 open PRs across 5 tracked repos | repeat before release | +| Keep public issues below 20 | scripts/platform-audit.js live GitHub sweep | current | 0 open issues across 5 tracked repos | repeat before release | +| Respond and manage repository discussions | scripts/platform-audit.js discussion summary | current | 0 need maintainer touch; 0 answerable discussions missing accepted answer | repeat before release | +| Build ITO-44 completion dashboard into a repeatable command | npm run operator:dashboard | complete | operator:dashboard package script exists | keep generated dashboard attached to publication evidence | +| ECC 2.0 preview pack ready | docs/releases/2.0.0-rc.1/preview-pack-manifest.md | in_progress | preview pack manifest is in-tree | final clean-checkout release approval and publish evidence still pending | +| Include Hermes specialized skills safely | docs/HERMES-SETUP.md and skills/hermes-imports/SKILL.md | in_progress | Hermes setup and import skill are present | final preview-pack smoke and release review pending | +| Prepare name-change, Claude plugin, and Codex plugin paths | naming-and-publication-matrix plus publication-readiness | in_progress | naming matrix and plugin readiness gates exist | real tag/push, marketplace submission, and final channel choice remain approval-gated | +| Prepare release notes, articles, tweets, and push notifications | docs/releases/2.0.0-rc.1 social and release-copy files | in_progress | release notes, X thread, and LinkedIn draft are present | URL-backed refresh and publish approval still pending | +| Advance AgentShield enterprise iteration | AgentShield PR evidence plus enterprise roadmap | in_progress | AgentShield enterprise PR evidence is mirrored in the GA roadmap | live IOC update loop and cross-harness depth remain pending | +| Advance ECC Tools native payments and AI-native harness-agnostic app | ECC Tools PR evidence, billing gate, hosted analysis lanes | in_progress | billing announcement gate and hosted analysis lanes are mirrored in the GA roadmap | live Marketplace test-account readback, hosted promotion telemetry, and operator review UX pending | +| Audit, prune, or attach legacy work | docs/stale-pr-salvage-ledger.md and legacy inventory | not_complete | legacy salvage ledger and ITO-55 tracking are present | final translation/manual-review tail remains | +| Keep Linear roadmap detailed and progress tracking synchronized | Linear project mirror plus progress-sync contract | in_progress | repo mirror and progress-sync contract are present | recurring Linear status sync and productized realtime sync remain pending | +| Provide ECC 2.0 observability for self-use | observability readiness gate | complete | observability:ready command and readiness doc exist | runtime/dashboard implementation can continue after release gates | +| Keep Mini Shai-Hulud/TanStack protection loop current | supply-chain watch plus runbook | current | scheduled supply-chain watch and runbook are present | advisory-source refresh automation and Linear status synchronization remain ITO-57 follow-up | -## Linear Operating State +## Top Actions -Project: - - -Active issue state after this pass: - -| Issue | Lane | State | -| --- | --- | --- | -| ITO-44 | Completion audit and readiness dashboard | In Progress | -| ITO-57 | Supply-chain intelligence and local protection loop | In Progress | -| ITO-59 | Discussions and public response queue | Current; Linear status sync pending | - -Still-open lanes: - -- ITO-45: ECC 2.0 preview pack, Hermes skills, packaging, and cross-harness - readiness. -- ITO-46: name availability, Claude plugin, Codex plugin, and package channels. -- ITO-47: release notes, articles, and social copy since last release. -- ITO-48 and ITO-49: AgentShield enterprise iteration and live supply-chain - intelligence. -- ITO-50 through ITO-54: ECC Tools payments, deep analysis, setup - recommendations, queue automation, Linear sync, and observability. -- ITO-55: legacy audit, prune, attach, or salvage. -- ITO-56: final publication gate, release notes, and push notifications. -- ITO-58: ECC Tools GitHub access blocker. - -## Decisions From This Pass - -- The checked GitHub queues are below the explicit target, so the next work - should not spend more time closing nonexistent PRs/issues. -- The discussion queue is current and repeatable through `discussion:audit`. - ITO-59 remains open only for recurring Linear/status synchronization. -- The Mini Shai-Hulud/TanStack protection pass now has a durable scheduled - watch workflow. ITO-57 remains open for advisory-source refresh automation - and Linear status synchronization. -- The release is still blocked by publication, package, plugin, billing, and - announcement gates. Passing `platform:audit` alone is not proof that ECC 2.0 - is publishable. +- `ecc-preview-pack`: final clean-checkout release approval and publish evidence still pending +- `hermes-specialized-skills`: final preview-pack smoke and release review pending +- `naming-and-plugin-publication`: real tag/push, marketplace submission, and final channel choice remain approval-gated +- `release-notes-and-notifications`: URL-backed refresh and publish approval still pending +- `agentshield-enterprise-iteration`: live IOC update loop and cross-harness depth remain pending +- `ecc-tools-next-level`: live Marketplace test-account readback, hosted promotion telemetry, and operator review UX pending +- `legacy-salvage`: final translation/manual-review tail remains +- `linear-roadmap-and-progress`: recurring Linear status sync and productized realtime sync remain pending ## Next Work Order -1. Build the ITO-44 completion dashboard into a repeatable command or generated - markdown artifact. -2. Run `platform:audit` and `discussion:audit` from the final release commit - before recording publication evidence. -3. Continue ITO-57 by adding advisory-source refresh automation and Linear - status synchronization for the scheduled supply-chain watch. -4. Resume release/publication lanes ITO-45, ITO-46, and ITO-56 only after the - readiness dashboard can be refreshed from commands. +1. Regenerate this dashboard from the final release commit before publication evidence is recorded. +2. Continue ITO-57 with advisory-source refresh automation and Linear status synchronization for the scheduled supply-chain watch. +3. Advance ECC Tools live Marketplace test-account readback before publishing native-payments announcement copy. +4. Resume ITO-45, ITO-46, and ITO-56 only after the generated dashboard and final release gates are refreshed. diff --git a/docs/releases/2.0.0-rc.1/publication-readiness.md b/docs/releases/2.0.0-rc.1/publication-readiness.md index 3562e01b..1a05b5ba 100644 --- a/docs/releases/2.0.0-rc.1/publication-readiness.md +++ b/docs/releases/2.0.0-rc.1/publication-readiness.md @@ -75,7 +75,7 @@ Record the exact commit SHA and command output before any publication action: | Queue baseline | `gh pr list` / `gh issue list` across trunk, AgentShield, JARVIS, ECC Tools, and ECC website | Under 20 open PRs and under 20 open issues | `publication-evidence-2026-05-15.md`: platform audit ready, 0 open PRs and 0 open issues across checked repos | | Discussion baseline | `node scripts/discussion-audit.js --json` | No unmanaged active discussion queue and no answerable Q&A missing an accepted answer | `publication-evidence-2026-05-15.md`: 58 trunk discussions, 0 without maintainer touch; other tracked repos disabled or 0 | | Linear roadmap | Linear project and issue readback | Detailed roadmap exists with release, security, AgentShield, ECC Tools, legacy, and observability lanes | `publication-evidence-2026-05-15.md`: project and 16 issue lanes recorded | -| Operator readiness dashboard | `node scripts/platform-audit.js --json --allow-untracked docs/drafts/` plus `node scripts/discussion-audit.js --json` | Current queue state mapped to macro-goal deliverables and incomplete gaps | `operator-readiness-dashboard-2026-05-15.md`: live status, command evidence, Linear state, and next work order | +| Operator readiness dashboard | `npm run operator:dashboard -- --allow-untracked docs/drafts/ --write docs/releases/2.0.0-rc.1/operator-readiness-dashboard-2026-05-15.md` | Current queue state mapped to macro-goal deliverables and incomplete gaps | `operator-readiness-dashboard-2026-05-15.md`: generated live status, prompt-to-artifact checklist, Linear state, and next work order | ## Do Not Publish If