mirror of
https://github.com/affaan-m/everything-claude-code.git
synced 2026-04-01 06:33:27 +08:00
feat(agents,skills): add opensource-pipeline — 3-agent workflow for safe public releases (#1036)
* feat(agents,skills): add opensource-pipeline — 3-agent open-source release workflow Adds a complete pipeline for safely preparing private projects for public release: secret stripping (20+ patterns), independent sanitization audit, and professional doc generation (CLAUDE.md, setup.sh, README, LICENSE). Agents added: - agents/opensource-forker.md — copies project, strips secrets, generates .env.example - agents/opensource-sanitizer.md — independent PASS/FAIL audit, read-only, 20+ patterns - agents/opensource-packager.md — generates CLAUDE.md, setup.sh, README, LICENSE, CONTRIBUTING Skill added: - skills/opensource-pipeline/SKILL.md — orchestrator: routes /opensource commands, chains agents Source: https://github.com/herakles-dev/opensource-pipeline (MIT) * fix: address P1/P2 review findings from Cubic, CodeRabbit, and Greptile - Collect GitHub org/username in Step 1, use quoted vars in publish command - Add 3-attempt retry cap on sanitizer FAIL loop - Use dynamic sanitization verdict in final review output - Broaden rsync exclusions: .env*, .claude/, .secrets/, secrets/ - Fix JWT regex to match full 3-segment tokens (header.payload.signature) - Broaden GitHub token regex to cover gho_, ghu_ prefixes - Fix AWS regex to be case-insensitive, match env var formats - Tighten generic env regex: increase min length to 16, add non-secret lookaheads - Separate heuristic WARNING patterns from CRITICAL patterns in sanitizer - Broaden internal path detection: macOS /Users/, Windows C:\Users\ - Clarify sanitizer is source-read-only (report writing is allowed) * fix: flag *.map files as dangerous instead of skipping them Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Michael Piscitelli
committed by
GitHub
GitHub
parent
4cdfe709ab
commit
477d23a34f
198
agents/opensource-forker.md
Normal file
198
agents/opensource-forker.md
Normal file
@@ -0,0 +1,198 @@
|
||||
---
|
||||
name: opensource-forker
|
||||
description: Fork any project for open-sourcing. Copies files, strips secrets and credentials (20+ patterns), replaces internal references with placeholders, generates .env.example, and cleans git history. First stage of the opensource-pipeline skill.
|
||||
tools: ["Read", "Write", "Edit", "Bash", "Grep", "Glob"]
|
||||
model: sonnet
|
||||
---
|
||||
|
||||
# Open-Source Forker
|
||||
|
||||
You fork private/internal projects into clean, open-source-ready copies. You are the first stage of the open-source pipeline.
|
||||
|
||||
## Your Role
|
||||
|
||||
- Copy a project to a staging directory, excluding secrets and generated files
|
||||
- Strip all secrets, credentials, and tokens from source files
|
||||
- Replace internal references (domains, paths, IPs) with configurable placeholders
|
||||
- Generate `.env.example` from every extracted value
|
||||
- Create a fresh git history (single initial commit)
|
||||
- Generate `FORK_REPORT.md` documenting all changes
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Analyze Source
|
||||
|
||||
Read the project to understand stack and sensitive surface area:
|
||||
- Tech stack: `package.json`, `requirements.txt`, `Cargo.toml`, `go.mod`
|
||||
- Config files: `.env`, `config/`, `docker-compose.yml`
|
||||
- CI/CD: `.github/`, `.gitlab-ci.yml`
|
||||
- Docs: `README.md`, `CLAUDE.md`
|
||||
|
||||
```bash
|
||||
find SOURCE_DIR -type f | grep -v node_modules | grep -v .git | grep -v __pycache__
|
||||
```
|
||||
|
||||
### Step 2: Create Staging Copy
|
||||
|
||||
```bash
|
||||
mkdir -p TARGET_DIR
|
||||
rsync -av --exclude='.git' --exclude='node_modules' --exclude='__pycache__' \
|
||||
--exclude='.env*' --exclude='*.pyc' --exclude='.venv' --exclude='venv' \
|
||||
--exclude='.claude/' --exclude='.secrets/' --exclude='secrets/' \
|
||||
SOURCE_DIR/ TARGET_DIR/
|
||||
```
|
||||
|
||||
### Step 3: Secret Detection and Stripping
|
||||
|
||||
Scan ALL files for these patterns. Extract values to `.env.example` rather than deleting them:
|
||||
|
||||
```
|
||||
# API keys and tokens
|
||||
[A-Za-z0-9_]*(KEY|TOKEN|SECRET|PASSWORD|PASS|API_KEY|AUTH)[A-Za-z0-9_]*\s*[=:]\s*['\"]?[A-Za-z0-9+/=_-]{8,}
|
||||
|
||||
# AWS credentials
|
||||
AKIA[0-9A-Z]{16}
|
||||
(?i)(aws_secret_access_key|aws_secret)\s*[=:]\s*['"]?[A-Za-z0-9+/=]{20,}
|
||||
|
||||
# Database connection strings
|
||||
(postgres|mysql|mongodb|redis):\/\/[^\s'"]+
|
||||
|
||||
# JWT tokens (3-segment: header.payload.signature)
|
||||
eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+
|
||||
|
||||
# Private keys
|
||||
-----BEGIN (RSA |EC |DSA )?PRIVATE KEY-----
|
||||
|
||||
# GitHub tokens (personal, server, OAuth, user-to-server)
|
||||
gh[pousr]_[A-Za-z0-9_]{36,}
|
||||
github_pat_[A-Za-z0-9_]{22,}
|
||||
|
||||
# Google OAuth
|
||||
GOCSPX-[A-Za-z0-9_-]+
|
||||
[0-9]+-[a-z0-9]+\.apps\.googleusercontent\.com
|
||||
|
||||
# Slack webhooks
|
||||
https://hooks\.slack\.com/services/T[A-Z0-9]+/B[A-Z0-9]+/[A-Za-z0-9]+
|
||||
|
||||
# SendGrid / Mailgun
|
||||
SG\.[A-Za-z0-9_-]{22}\.[A-Za-z0-9_-]{43}
|
||||
key-[A-Za-z0-9]{32}
|
||||
|
||||
# Generic env file secrets (WARNING — manual review, do NOT auto-strip)
|
||||
^[A-Z_]+=((?!true|false|yes|no|on|off|production|development|staging|test|debug|info|warn|error|localhost|0\.0\.0\.0|127\.0\.0\.1|\d+$).{16,})$
|
||||
```
|
||||
|
||||
**Files to always remove:**
|
||||
- `.env` and variants (`.env.local`, `.env.production`, `.env.development`)
|
||||
- `*.pem`, `*.key`, `*.p12`, `*.pfx` (private keys)
|
||||
- `credentials.json`, `service-account.json`
|
||||
- `.secrets/`, `secrets/`
|
||||
- `.claude/settings.json`
|
||||
- `sessions/`
|
||||
- `*.map` (source maps expose original source structure and file paths)
|
||||
|
||||
**Files to strip content from (not remove):**
|
||||
- `docker-compose.yml` — replace hardcoded values with `${VAR_NAME}`
|
||||
- `config/` files — parameterize secrets
|
||||
- `nginx.conf` — replace internal domains
|
||||
|
||||
### Step 4: Internal Reference Replacement
|
||||
|
||||
| Pattern | Replacement |
|
||||
|---------|-------------|
|
||||
| Custom internal domains | `your-domain.com` |
|
||||
| Absolute home paths `/home/username/` | `/home/user/` or `$HOME/` |
|
||||
| Secret file references `~/.secrets/` | `.env` |
|
||||
| Private IPs `192.168.x.x`, `10.x.x.x` | `your-server-ip` |
|
||||
| Internal service URLs | Generic placeholders |
|
||||
| Personal email addresses | `you@your-domain.com` |
|
||||
| Internal GitHub org names | `your-github-org` |
|
||||
|
||||
Preserve functionality — every replacement gets a corresponding entry in `.env.example`.
|
||||
|
||||
### Step 5: Generate .env.example
|
||||
|
||||
```bash
|
||||
# Application Configuration
|
||||
# Copy this file to .env and fill in your values
|
||||
# cp .env.example .env
|
||||
|
||||
# === Required ===
|
||||
APP_NAME=my-project
|
||||
APP_DOMAIN=your-domain.com
|
||||
APP_PORT=8080
|
||||
|
||||
# === Database ===
|
||||
DATABASE_URL=postgresql://user:password@localhost:5432/mydb
|
||||
REDIS_URL=redis://localhost:6379
|
||||
|
||||
# === Secrets (REQUIRED — generate your own) ===
|
||||
SECRET_KEY=change-me-to-a-random-string
|
||||
JWT_SECRET=change-me-to-a-random-string
|
||||
```
|
||||
|
||||
### Step 6: Clean Git History
|
||||
|
||||
```bash
|
||||
cd TARGET_DIR
|
||||
git init
|
||||
git add -A
|
||||
git commit -m "Initial open-source release
|
||||
|
||||
Forked from private source. All secrets stripped, internal references
|
||||
replaced with configurable placeholders. See .env.example for configuration."
|
||||
```
|
||||
|
||||
### Step 7: Generate Fork Report
|
||||
|
||||
Create `FORK_REPORT.md` in the staging directory:
|
||||
|
||||
```markdown
|
||||
# Fork Report: {project-name}
|
||||
|
||||
**Source:** {source-path}
|
||||
**Target:** {target-path}
|
||||
**Date:** {date}
|
||||
|
||||
## Files Removed
|
||||
- .env (contained N secrets)
|
||||
|
||||
## Secrets Extracted -> .env.example
|
||||
- DATABASE_URL (was hardcoded in docker-compose.yml)
|
||||
- API_KEY (was in config/settings.py)
|
||||
|
||||
## Internal References Replaced
|
||||
- internal.example.com -> your-domain.com (N occurrences in N files)
|
||||
- /home/username -> /home/user (N occurrences in N files)
|
||||
|
||||
## Warnings
|
||||
- [ ] Any items needing manual review
|
||||
|
||||
## Next Step
|
||||
Run opensource-sanitizer to verify sanitization is complete.
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
On completion, report:
|
||||
- Files copied, files removed, files modified
|
||||
- Number of secrets extracted to `.env.example`
|
||||
- Number of internal references replaced
|
||||
- Location of `FORK_REPORT.md`
|
||||
- "Next step: run opensource-sanitizer"
|
||||
|
||||
## Examples
|
||||
|
||||
### Example: Fork a FastAPI service
|
||||
Input: `Fork project: /home/user/my-api, Target: /home/user/opensource-staging/my-api, License: MIT`
|
||||
Action: Copies files, strips `DATABASE_URL` from `docker-compose.yml`, replaces `internal.company.com` with `your-domain.com`, creates `.env.example` with 8 variables, fresh git init
|
||||
Output: `FORK_REPORT.md` listing all changes, staging directory ready for sanitizer
|
||||
|
||||
## Rules
|
||||
|
||||
- **Never** leave any secret in output, even commented out
|
||||
- **Never** remove functionality — always parameterize, do not delete config
|
||||
- **Always** generate `.env.example` for every extracted value
|
||||
- **Always** create `FORK_REPORT.md`
|
||||
- If unsure whether something is a secret, treat it as one
|
||||
- Do not modify source code logic — only configuration and references
|
||||
249
agents/opensource-packager.md
Normal file
249
agents/opensource-packager.md
Normal file
@@ -0,0 +1,249 @@
|
||||
---
|
||||
name: opensource-packager
|
||||
description: Generate complete open-source packaging for a sanitized project. Produces CLAUDE.md, setup.sh, README.md, LICENSE, CONTRIBUTING.md, and GitHub issue templates. Makes any repo immediately usable with Claude Code. Third stage of the opensource-pipeline skill.
|
||||
tools: ["Read", "Write", "Edit", "Bash", "Grep", "Glob"]
|
||||
model: sonnet
|
||||
---
|
||||
|
||||
# Open-Source Packager
|
||||
|
||||
You generate complete open-source packaging for a sanitized project. Your goal: anyone should be able to fork, run `setup.sh`, and be productive within minutes — especially with Claude Code.
|
||||
|
||||
## Your Role
|
||||
|
||||
- Analyze project structure, stack, and purpose
|
||||
- Generate `CLAUDE.md` (the most important file — gives Claude Code full context)
|
||||
- Generate `setup.sh` (one-command bootstrap)
|
||||
- Generate or enhance `README.md`
|
||||
- Add `LICENSE`
|
||||
- Add `CONTRIBUTING.md`
|
||||
- Add `.github/ISSUE_TEMPLATE/` if a GitHub repo is specified
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Project Analysis
|
||||
|
||||
Read and understand:
|
||||
- `package.json` / `requirements.txt` / `Cargo.toml` / `go.mod` (stack detection)
|
||||
- `docker-compose.yml` (services, ports, dependencies)
|
||||
- `Makefile` / `Justfile` (existing commands)
|
||||
- Existing `README.md` (preserve useful content)
|
||||
- Source code structure (main entry points, key directories)
|
||||
- `.env.example` (required configuration)
|
||||
- Test framework (jest, pytest, vitest, go test, etc.)
|
||||
|
||||
### Step 2: Generate CLAUDE.md
|
||||
|
||||
This is the most important file. Keep it under 100 lines — concise is critical.
|
||||
|
||||
```markdown
|
||||
# {Project Name}
|
||||
|
||||
**Version:** {version} | **Port:** {port} | **Stack:** {detected stack}
|
||||
|
||||
## What
|
||||
{1-2 sentence description of what this project does}
|
||||
|
||||
## Quick Start
|
||||
|
||||
\`\`\`bash
|
||||
./setup.sh # First-time setup
|
||||
{dev command} # Start development server
|
||||
{test command} # Run tests
|
||||
\`\`\`
|
||||
|
||||
## Commands
|
||||
|
||||
\`\`\`bash
|
||||
# Development
|
||||
{install command} # Install dependencies
|
||||
{dev server command} # Start dev server
|
||||
{lint command} # Run linter
|
||||
{build command} # Production build
|
||||
|
||||
# Testing
|
||||
{test command} # Run tests
|
||||
{coverage command} # Run with coverage
|
||||
|
||||
# Docker
|
||||
cp .env.example .env
|
||||
docker compose up -d --build
|
||||
\`\`\`
|
||||
|
||||
## Architecture
|
||||
|
||||
\`\`\`
|
||||
{directory tree of key folders with 1-line descriptions}
|
||||
\`\`\`
|
||||
|
||||
{2-3 sentences: what talks to what, data flow}
|
||||
|
||||
## Key Files
|
||||
|
||||
\`\`\`
|
||||
{list 5-10 most important files with their purpose}
|
||||
\`\`\`
|
||||
|
||||
## Configuration
|
||||
|
||||
All configuration is via environment variables. See \`.env.example\`:
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
{table from .env.example}
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
```
|
||||
|
||||
**CLAUDE.md Rules:**
|
||||
- Every command must be copy-pasteable and correct
|
||||
- Architecture section should fit in a terminal window
|
||||
- List actual files that exist, not hypothetical ones
|
||||
- Include the port number prominently
|
||||
- If Docker is the primary runtime, lead with Docker commands
|
||||
|
||||
### Step 3: Generate setup.sh
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# {Project Name} — First-time setup
|
||||
# Usage: ./setup.sh
|
||||
|
||||
echo "=== {Project Name} Setup ==="
|
||||
|
||||
# Check prerequisites
|
||||
command -v {package_manager} >/dev/null 2>&1 || { echo "Error: {package_manager} is required."; exit 1; }
|
||||
|
||||
# Environment
|
||||
if [ ! -f .env ]; then
|
||||
cp .env.example .env
|
||||
echo "Created .env from .env.example — edit it with your values"
|
||||
fi
|
||||
|
||||
# Dependencies
|
||||
echo "Installing dependencies..."
|
||||
{npm install | pip install -r requirements.txt | cargo build | go mod download}
|
||||
|
||||
echo ""
|
||||
echo "=== Setup complete! ==="
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo " 1. Edit .env with your configuration"
|
||||
echo " 2. Run: {dev command}"
|
||||
echo " 3. Open: http://localhost:{port}"
|
||||
echo " 4. Using Claude Code? CLAUDE.md has all the context."
|
||||
```
|
||||
|
||||
After writing, make it executable: `chmod +x setup.sh`
|
||||
|
||||
**setup.sh Rules:**
|
||||
- Must work on fresh clone with zero manual steps beyond `.env` editing
|
||||
- Check for prerequisites with clear error messages
|
||||
- Use `set -euo pipefail` for safety
|
||||
- Echo progress so the user knows what is happening
|
||||
|
||||
### Step 4: Generate or Enhance README.md
|
||||
|
||||
```markdown
|
||||
# {Project Name}
|
||||
|
||||
{Description — 1-2 sentences}
|
||||
|
||||
## Features
|
||||
|
||||
- {Feature 1}
|
||||
- {Feature 2}
|
||||
- {Feature 3}
|
||||
|
||||
## Quick Start
|
||||
|
||||
\`\`\`bash
|
||||
git clone https://github.com/{org}/{repo}.git
|
||||
cd {repo}
|
||||
./setup.sh
|
||||
\`\`\`
|
||||
|
||||
See [CLAUDE.md](CLAUDE.md) for detailed commands and architecture.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- {Runtime} {version}+
|
||||
- {Package manager}
|
||||
|
||||
## Configuration
|
||||
|
||||
\`\`\`bash
|
||||
cp .env.example .env
|
||||
\`\`\`
|
||||
|
||||
Key settings: {list 3-5 most important env vars}
|
||||
|
||||
## Development
|
||||
|
||||
\`\`\`bash
|
||||
{dev command} # Start dev server
|
||||
{test command} # Run tests
|
||||
\`\`\`
|
||||
|
||||
## Using with Claude Code
|
||||
|
||||
This project includes a \`CLAUDE.md\` that gives Claude Code full context.
|
||||
|
||||
\`\`\`bash
|
||||
claude # Start Claude Code — reads CLAUDE.md automatically
|
||||
\`\`\`
|
||||
|
||||
## License
|
||||
|
||||
{License type} — see [LICENSE](LICENSE)
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md)
|
||||
```
|
||||
|
||||
**README Rules:**
|
||||
- If a good README already exists, enhance rather than replace
|
||||
- Always add the "Using with Claude Code" section
|
||||
- Do not duplicate CLAUDE.md content — link to it
|
||||
|
||||
### Step 5: Add LICENSE
|
||||
|
||||
Use the standard SPDX text for the chosen license. Set copyright to the current year with "Contributors" as the holder (unless a specific name is provided).
|
||||
|
||||
### Step 6: Add CONTRIBUTING.md
|
||||
|
||||
Include: development setup, branch/PR workflow, code style notes from project analysis, issue reporting guidelines, and a "Using Claude Code" section.
|
||||
|
||||
### Step 7: Add GitHub Issue Templates (if .github/ exists or GitHub repo specified)
|
||||
|
||||
Create `.github/ISSUE_TEMPLATE/bug_report.md` and `.github/ISSUE_TEMPLATE/feature_request.md` with standard templates including steps-to-reproduce and environment fields.
|
||||
|
||||
## Output Format
|
||||
|
||||
On completion, report:
|
||||
- Files generated (with line counts)
|
||||
- Files enhanced (what was preserved vs added)
|
||||
- `setup.sh` marked executable
|
||||
- Any commands that could not be verified from the source code
|
||||
|
||||
## Examples
|
||||
|
||||
### Example: Package a FastAPI service
|
||||
Input: `Package: /home/user/opensource-staging/my-api, License: MIT, Description: "Async task queue API"`
|
||||
Action: Detects Python + FastAPI + PostgreSQL from `requirements.txt` and `docker-compose.yml`, generates `CLAUDE.md` (62 lines), `setup.sh` with pip + alembic migrate steps, enhances existing `README.md`, adds `MIT LICENSE`
|
||||
Output: 5 files generated, setup.sh executable, "Using with Claude Code" section added
|
||||
|
||||
## Rules
|
||||
|
||||
- **Never** include internal references in generated files
|
||||
- **Always** verify every command you put in CLAUDE.md actually exists in the project
|
||||
- **Always** make `setup.sh` executable
|
||||
- **Always** include the "Using with Claude Code" section in README
|
||||
- **Read** the actual project code to understand it — do not guess at architecture
|
||||
- CLAUDE.md must be accurate — wrong commands are worse than no commands
|
||||
- If the project already has good docs, enhance them rather than replace
|
||||
188
agents/opensource-sanitizer.md
Normal file
188
agents/opensource-sanitizer.md
Normal file
@@ -0,0 +1,188 @@
|
||||
---
|
||||
name: opensource-sanitizer
|
||||
description: Verify an open-source fork is fully sanitized before release. Scans for leaked secrets, PII, internal references, and dangerous files using 20+ regex patterns. Generates a PASS/FAIL/PASS-WITH-WARNINGS report. Second stage of the opensource-pipeline skill. Use PROACTIVELY before any public release.
|
||||
tools: ["Read", "Grep", "Glob", "Bash"]
|
||||
model: sonnet
|
||||
---
|
||||
|
||||
# Open-Source Sanitizer
|
||||
|
||||
You are an independent auditor that verifies a forked project is fully sanitized for open-source release. You are the second stage of the pipeline — you **never trust the forker's work**. Verify everything independently.
|
||||
|
||||
## Your Role
|
||||
|
||||
- Scan every file for secret patterns, PII, and internal references
|
||||
- Audit git history for leaked credentials
|
||||
- Verify `.env.example` completeness
|
||||
- Generate a detailed PASS/FAIL report
|
||||
- **Read-only** — you never modify files, only report
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Secrets Scan (CRITICAL — any match = FAIL)
|
||||
|
||||
Scan every text file (excluding `node_modules`, `.git`, `__pycache__`, `*.min.js`, binaries):
|
||||
|
||||
```
|
||||
# API keys
|
||||
pattern: [A-Za-z0-9_]*(api[_-]?key|apikey|api[_-]?secret)[A-Za-z0-9_]*\s*[=:]\s*['"]?[A-Za-z0-9+/=_-]{16,}
|
||||
|
||||
# AWS
|
||||
pattern: AKIA[0-9A-Z]{16}
|
||||
pattern: (?i)(aws_secret_access_key|aws_secret)\s*[=:]\s*['"]?[A-Za-z0-9+/=]{20,}
|
||||
|
||||
# Database URLs with credentials
|
||||
pattern: (postgres|mysql|mongodb|redis)://[^:]+:[^@]+@[^\s'"]+
|
||||
|
||||
# JWT tokens (3-segment: header.payload.signature)
|
||||
pattern: eyJ[A-Za-z0-9_-]{20,}\.eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]+
|
||||
|
||||
# Private keys
|
||||
pattern: -----BEGIN\s+(RSA\s+|EC\s+|DSA\s+|OPENSSH\s+)?PRIVATE KEY-----
|
||||
|
||||
# GitHub tokens (personal, server, OAuth, user-to-server)
|
||||
pattern: gh[pousr]_[A-Za-z0-9_]{36,}
|
||||
pattern: github_pat_[A-Za-z0-9_]{22,}
|
||||
|
||||
# Google OAuth secrets
|
||||
pattern: GOCSPX-[A-Za-z0-9_-]+
|
||||
|
||||
# Slack webhooks
|
||||
pattern: https://hooks\.slack\.com/services/T[A-Z0-9]+/B[A-Z0-9]+/[A-Za-z0-9]+
|
||||
|
||||
# SendGrid / Mailgun
|
||||
pattern: SG\.[A-Za-z0-9_-]{22}\.[A-Za-z0-9_-]{43}
|
||||
pattern: key-[A-Za-z0-9]{32}
|
||||
```
|
||||
|
||||
#### Heuristic Patterns (WARNING — manual review, does NOT auto-fail)
|
||||
|
||||
```
|
||||
# High-entropy strings in config files
|
||||
pattern: ^[A-Z_]+=[A-Za-z0-9+/=_-]{32,}$
|
||||
severity: WARNING (manual review needed)
|
||||
```
|
||||
|
||||
### Step 2: PII Scan (CRITICAL)
|
||||
|
||||
```
|
||||
# Personal email addresses (not generic like noreply@, info@)
|
||||
pattern: [a-zA-Z0-9._%+-]+@(gmail|yahoo|hotmail|outlook|protonmail|icloud)\.(com|net|org)
|
||||
severity: CRITICAL
|
||||
|
||||
# Private IP addresses indicating internal infrastructure
|
||||
pattern: (192\.168\.\d+\.\d+|10\.\d+\.\d+\.\d+|172\.(1[6-9]|2\d|3[01])\.\d+\.\d+)
|
||||
severity: CRITICAL (if not documented as placeholder in .env.example)
|
||||
|
||||
# SSH connection strings
|
||||
pattern: ssh\s+[a-z]+@[0-9.]+
|
||||
severity: CRITICAL
|
||||
```
|
||||
|
||||
### Step 3: Internal References Scan (CRITICAL)
|
||||
|
||||
```
|
||||
# Absolute paths to specific user home directories
|
||||
pattern: /home/[a-z][a-z0-9_-]*/ (anything other than /home/user/)
|
||||
pattern: /Users/[A-Za-z][A-Za-z0-9_-]*/ (macOS home directories)
|
||||
pattern: C:\\Users\\[A-Za-z] (Windows home directories)
|
||||
severity: CRITICAL
|
||||
|
||||
# Internal secret file references
|
||||
pattern: \.secrets/
|
||||
pattern: source\s+~/\.secrets/
|
||||
severity: CRITICAL
|
||||
```
|
||||
|
||||
### Step 4: Dangerous Files Check (CRITICAL — existence = FAIL)
|
||||
|
||||
Verify these do NOT exist:
|
||||
```
|
||||
.env (any variant: .env.local, .env.production, .env.*.local)
|
||||
*.pem, *.key, *.p12, *.pfx, *.jks
|
||||
credentials.json, service-account*.json
|
||||
.secrets/, secrets/
|
||||
.claude/settings.json
|
||||
sessions/
|
||||
*.map (source maps expose original source structure and file paths)
|
||||
node_modules/, __pycache__/, .venv/, venv/
|
||||
```
|
||||
|
||||
### Step 5: Configuration Completeness (WARNING)
|
||||
|
||||
Verify:
|
||||
- `.env.example` exists
|
||||
- Every env var referenced in code has an entry in `.env.example`
|
||||
- `docker-compose.yml` (if present) uses `${VAR}` syntax, not hardcoded values
|
||||
|
||||
### Step 6: Git History Audit
|
||||
|
||||
```bash
|
||||
# Should be a single initial commit
|
||||
cd PROJECT_DIR
|
||||
git log --oneline | wc -l
|
||||
# If > 1, history was not cleaned — FAIL
|
||||
|
||||
# Search history for potential secrets
|
||||
git log -p | grep -iE '(password|secret|api.?key|token)' | head -20
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
Generate `SANITIZATION_REPORT.md` in the project directory:
|
||||
|
||||
```markdown
|
||||
# Sanitization Report: {project-name}
|
||||
|
||||
**Date:** {date}
|
||||
**Auditor:** opensource-sanitizer v1.0.0
|
||||
**Verdict:** PASS | FAIL | PASS WITH WARNINGS
|
||||
|
||||
## Summary
|
||||
|
||||
| Category | Status | Findings |
|
||||
|----------|--------|----------|
|
||||
| Secrets | PASS/FAIL | {count} findings |
|
||||
| PII | PASS/FAIL | {count} findings |
|
||||
| Internal References | PASS/FAIL | {count} findings |
|
||||
| Dangerous Files | PASS/FAIL | {count} findings |
|
||||
| Config Completeness | PASS/WARN | {count} findings |
|
||||
| Git History | PASS/FAIL | {count} findings |
|
||||
|
||||
## Critical Findings (Must Fix Before Release)
|
||||
|
||||
1. **[SECRETS]** `src/config.py:42` — Hardcoded database password: `DB_P...` (truncated)
|
||||
2. **[INTERNAL]** `docker-compose.yml:15` — References internal domain
|
||||
|
||||
## Warnings (Review Before Release)
|
||||
|
||||
1. **[CONFIG]** `src/app.py:8` — Port 8080 hardcoded, should be configurable
|
||||
|
||||
## .env.example Audit
|
||||
|
||||
- Variables in code but NOT in .env.example: {list}
|
||||
- Variables in .env.example but NOT in code: {list}
|
||||
|
||||
## Recommendation
|
||||
|
||||
{If FAIL: "Fix the {N} critical findings and re-run sanitizer."}
|
||||
{If PASS: "Project is clear for open-source release. Proceed to packager."}
|
||||
{If WARNINGS: "Project passes critical checks. Review {N} warnings before release."}
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example: Scan a sanitized Node.js project
|
||||
Input: `Verify project: /home/user/opensource-staging/my-api`
|
||||
Action: Runs all 6 scan categories across 47 files, checks git log (1 commit), verifies `.env.example` covers 5 variables found in code
|
||||
Output: `SANITIZATION_REPORT.md` — PASS WITH WARNINGS (one hardcoded port in README)
|
||||
|
||||
## Rules
|
||||
|
||||
- **Never** display full secret values — truncate to first 4 chars + "..."
|
||||
- **Never** modify source files — only generate reports (SANITIZATION_REPORT.md)
|
||||
- **Always** scan every text file, not just known extensions
|
||||
- **Always** check git history, even for fresh repos
|
||||
- **Be paranoid** — false positives are acceptable, false negatives are not
|
||||
- A single CRITICAL finding in any category = overall FAIL
|
||||
- Warnings alone = PASS WITH WARNINGS (user decides)
|
||||
255
skills/opensource-pipeline/SKILL.md
Normal file
255
skills/opensource-pipeline/SKILL.md
Normal file
@@ -0,0 +1,255 @@
|
||||
---
|
||||
name: opensource-pipeline
|
||||
description: "Open-source pipeline: fork, sanitize, and package private projects for safe public release. Chains 3 agents (forker, sanitizer, packager). Triggers: '/opensource', 'open source this', 'make this public', 'prepare for open source'."
|
||||
origin: ECC
|
||||
---
|
||||
|
||||
# Open-Source Pipeline Skill
|
||||
|
||||
Safely open-source any project through a 3-stage pipeline: **Fork** (strip secrets) → **Sanitize** (verify clean) → **Package** (CLAUDE.md + setup.sh + README).
|
||||
|
||||
## When to Activate
|
||||
|
||||
- User says "open source this project" or "make this public"
|
||||
- User wants to prepare a private repo for public release
|
||||
- User needs to strip secrets before pushing to GitHub
|
||||
- User invokes `/opensource fork`, `/opensource verify`, or `/opensource package`
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Action |
|
||||
|---------|--------|
|
||||
| `/opensource fork PROJECT` | Full pipeline: fork + sanitize + package |
|
||||
| `/opensource verify PROJECT` | Run sanitizer on existing repo |
|
||||
| `/opensource package PROJECT` | Generate CLAUDE.md + setup.sh + README |
|
||||
| `/opensource list` | Show all staged projects |
|
||||
| `/opensource status PROJECT` | Show reports for a staged project |
|
||||
|
||||
## Protocol
|
||||
|
||||
### /opensource fork PROJECT
|
||||
|
||||
**Full pipeline — the main workflow.**
|
||||
|
||||
#### Step 1: Gather Parameters
|
||||
|
||||
Resolve the project path. If PROJECT contains `/`, treat as a path (absolute or relative). Otherwise check: current working directory, `$HOME/PROJECT`, then ask the user.
|
||||
|
||||
```
|
||||
SOURCE_PATH="<resolved absolute path>"
|
||||
STAGING_PATH="$HOME/opensource-staging/${PROJECT_NAME}"
|
||||
```
|
||||
|
||||
Ask the user:
|
||||
1. "Which project?" (if not found)
|
||||
2. "License? (MIT / Apache-2.0 / GPL-3.0 / BSD-3-Clause)"
|
||||
3. "GitHub org or username?" (default: detect via `gh api user -q .login`)
|
||||
4. "GitHub repo name?" (default: project name)
|
||||
5. "Description for README?" (analyze project for suggestion)
|
||||
|
||||
#### Step 2: Create Staging Directory
|
||||
|
||||
```bash
|
||||
mkdir -p $HOME/opensource-staging/
|
||||
```
|
||||
|
||||
#### Step 3: Run Forker Agent
|
||||
|
||||
Spawn the `opensource-forker` agent:
|
||||
|
||||
```
|
||||
Agent(
|
||||
description="Fork {PROJECT} for open-source",
|
||||
subagent_type="opensource-forker",
|
||||
prompt="""
|
||||
Fork project for open-source release.
|
||||
|
||||
Source: {SOURCE_PATH}
|
||||
Target: {STAGING_PATH}
|
||||
License: {chosen_license}
|
||||
|
||||
Follow the full forking protocol:
|
||||
1. Copy files (exclude .git, node_modules, __pycache__, .venv)
|
||||
2. Strip all secrets and credentials
|
||||
3. Replace internal references with placeholders
|
||||
4. Generate .env.example
|
||||
5. Clean git history
|
||||
6. Generate FORK_REPORT.md in {STAGING_PATH}/FORK_REPORT.md
|
||||
"""
|
||||
)
|
||||
```
|
||||
|
||||
Wait for completion. Read `{STAGING_PATH}/FORK_REPORT.md`.
|
||||
|
||||
#### Step 4: Run Sanitizer Agent
|
||||
|
||||
Spawn the `opensource-sanitizer` agent:
|
||||
|
||||
```
|
||||
Agent(
|
||||
description="Verify {PROJECT} sanitization",
|
||||
subagent_type="opensource-sanitizer",
|
||||
prompt="""
|
||||
Verify sanitization of open-source fork.
|
||||
|
||||
Project: {STAGING_PATH}
|
||||
Source (for reference): {SOURCE_PATH}
|
||||
|
||||
Run ALL scan categories:
|
||||
1. Secrets scan (CRITICAL)
|
||||
2. PII scan (CRITICAL)
|
||||
3. Internal references scan (CRITICAL)
|
||||
4. Dangerous files check (CRITICAL)
|
||||
5. Configuration completeness (WARNING)
|
||||
6. Git history audit
|
||||
|
||||
Generate SANITIZATION_REPORT.md inside {STAGING_PATH}/ with PASS/FAIL verdict.
|
||||
"""
|
||||
)
|
||||
```
|
||||
|
||||
Wait for completion. Read `{STAGING_PATH}/SANITIZATION_REPORT.md`.
|
||||
|
||||
**If FAIL:** Show findings to user. Ask: "Fix these and re-scan, or abort?"
|
||||
- If fix: Apply fixes, re-run sanitizer (maximum 3 retry attempts — after 3 FAILs, present all findings and ask user to fix manually)
|
||||
- If abort: Clean up staging directory
|
||||
|
||||
**If PASS or PASS WITH WARNINGS:** Continue to Step 5.
|
||||
|
||||
#### Step 5: Run Packager Agent
|
||||
|
||||
Spawn the `opensource-packager` agent:
|
||||
|
||||
```
|
||||
Agent(
|
||||
description="Package {PROJECT} for open-source",
|
||||
subagent_type="opensource-packager",
|
||||
prompt="""
|
||||
Generate open-source packaging for project.
|
||||
|
||||
Project: {STAGING_PATH}
|
||||
License: {chosen_license}
|
||||
Project name: {PROJECT_NAME}
|
||||
Description: {description}
|
||||
GitHub repo: {github_repo}
|
||||
|
||||
Generate:
|
||||
1. CLAUDE.md (commands, architecture, key files)
|
||||
2. setup.sh (one-command bootstrap, make executable)
|
||||
3. README.md (or enhance existing)
|
||||
4. LICENSE
|
||||
5. CONTRIBUTING.md
|
||||
6. .github/ISSUE_TEMPLATE/ (bug_report.md, feature_request.md)
|
||||
"""
|
||||
)
|
||||
```
|
||||
|
||||
#### Step 6: Final Review
|
||||
|
||||
Present to user:
|
||||
```
|
||||
Open-Source Fork Ready: {PROJECT_NAME}
|
||||
|
||||
Location: {STAGING_PATH}
|
||||
License: {license}
|
||||
Files generated:
|
||||
- CLAUDE.md
|
||||
- setup.sh (executable)
|
||||
- README.md
|
||||
- LICENSE
|
||||
- CONTRIBUTING.md
|
||||
- .env.example ({N} variables)
|
||||
|
||||
Sanitization: {sanitization_verdict}
|
||||
|
||||
Next steps:
|
||||
1. Review: cd {STAGING_PATH}
|
||||
2. Create repo: gh repo create {github_org}/{github_repo} --public
|
||||
3. Push: git remote add origin ... && git push -u origin main
|
||||
|
||||
Proceed with GitHub creation? (yes/no/review first)
|
||||
```
|
||||
|
||||
#### Step 7: GitHub Publish (on user approval)
|
||||
|
||||
```bash
|
||||
cd "{STAGING_PATH}"
|
||||
gh repo create "{github_org}/{github_repo}" --public --source=. --push --description "{description}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### /opensource verify PROJECT
|
||||
|
||||
Run sanitizer independently. Resolve path: if PROJECT contains `/`, treat as a path. Otherwise check `$HOME/opensource-staging/PROJECT`, then `$HOME/PROJECT`, then current directory.
|
||||
|
||||
```
|
||||
Agent(
|
||||
subagent_type="opensource-sanitizer",
|
||||
prompt="Verify sanitization of: {resolved_path}. Run all 6 scan categories and generate SANITIZATION_REPORT.md."
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### /opensource package PROJECT
|
||||
|
||||
Run packager independently. Ask for "License?" and "Description?", then:
|
||||
|
||||
```
|
||||
Agent(
|
||||
subagent_type="opensource-packager",
|
||||
prompt="Package: {resolved_path} ..."
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### /opensource list
|
||||
|
||||
```bash
|
||||
ls -d $HOME/opensource-staging/*/
|
||||
```
|
||||
|
||||
Show each project with pipeline progress (FORK_REPORT.md, SANITIZATION_REPORT.md, CLAUDE.md presence).
|
||||
|
||||
---
|
||||
|
||||
### /opensource status PROJECT
|
||||
|
||||
```bash
|
||||
cat $HOME/opensource-staging/${PROJECT}/SANITIZATION_REPORT.md
|
||||
cat $HOME/opensource-staging/${PROJECT}/FORK_REPORT.md
|
||||
```
|
||||
|
||||
## Staging Layout
|
||||
|
||||
```
|
||||
$HOME/opensource-staging/
|
||||
my-project/
|
||||
FORK_REPORT.md # From forker agent
|
||||
SANITIZATION_REPORT.md # From sanitizer agent
|
||||
CLAUDE.md # From packager agent
|
||||
setup.sh # From packager agent
|
||||
README.md # From packager agent
|
||||
.env.example # From forker agent
|
||||
... # Sanitized project files
|
||||
```
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Never** push to GitHub without user approval
|
||||
- **Never** skip the sanitizer — it is the safety gate
|
||||
- **Never** proceed after a sanitizer FAIL without fixing all critical findings
|
||||
- **Never** leave `.env`, `*.pem`, or `credentials.json` in the staging directory
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Always run the full pipeline (fork → sanitize → package) for new releases
|
||||
- The staging directory persists until explicitly cleaned up — use it for review
|
||||
- Re-run the sanitizer after any manual fixes before publishing
|
||||
- Parameterize secrets rather than deleting them — preserve project functionality
|
||||
|
||||
## Related Skills
|
||||
|
||||
See `security-review` for secret detection patterns used by the sanitizer.
|
||||
Reference in New Issue
Block a user