feat: add ai-regression-testing skill (#433)

Patterns for catching regressions introduced by AI coding agents. Covers sandbox/production parity testing, API response shape verification, and integration with bug-check workflows. Based on real-world experience where AI (Claude Code) introduced the same bug 4 times because the same model wrote and reviewed the code — only automated tests caught it. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-30 13:43:26 +08:00 · 2026-03-17 05:35:31 +09:00
parent 113119dc6f
commit c2f2f9517c
1 changed files with 385 additions and 0 deletions
--- a/skills/ai-regression-testing/SKILL.md
+++ b/skills/ai-regression-testing/SKILL.md
@@ -0,0 +1,385 @@
 ---
 name: ai-regression-testing
 description: Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code.
 origin: ECC
 ---
 # AI Regression Testing
 Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it — creating systematic blind spots that only automated tests can catch.
 ## When to Activate
 - AI agent (Claude Code, Cursor, Codex) has modified API routes or backend logic
 - A bug was found and fixed — need to prevent re-introduction
 - Project has a sandbox/mock mode that can be leveraged for DB-free testing
 - Running `/bug-check` or similar review commands after code changes
 - Multiple code paths exist (sandbox vs production, feature flags, etc.)
 ## The Core Problem
 When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern:
 ```
 AI writes fix → AI reviews fix → AI says "looks correct" → Bug still exists
 ```
 **Real-world example** (observed in production):
 ```
 Fix 1: Added notification_settings to API response
  → Forgot to add it to the SELECT query
  → AI reviewed and missed it (same blind spot)
 Fix 2: Added it to SELECT query
  → TypeScript build error (column not in generated types)
  → AI reviewed Fix 1 but didn't catch the SELECT issue
 Fix 3: Changed to SELECT *
  → Fixed production path, forgot sandbox path
  → AI reviewed and missed it AGAIN (4th occurrence)
 Fix 4: Test caught it instantly on first run ✅
 ```
 The pattern: **sandbox/production path inconsistency** is the #1 AI-introduced regression.
 ## Sandbox-Mode API Testing
 Most projects with AI-friendly architecture have a sandbox/mock mode. This is the key to fast, DB-free API testing.
 ### Setup (Vitest + Next.js App Router)
 ```typescript
 // vitest.config.ts
 import { defineConfig } from "vitest/config";
 import path from "path";
 export default defineConfig({
  test: {
    environment: "node",
    globals: true,
    include: ["__tests__/**/*.test.ts"],
    setupFiles: ["__tests__/setup.ts"],
  },
  resolve: {
    alias: {
      "@": path.resolve(__dirname, "."),
    },
  },
 });
 ```
 ```typescript
 // __tests__/setup.ts
 // Force sandbox mode — no database needed
 process.env.SANDBOX_MODE = "true";
 process.env.NEXT_PUBLIC_SUPABASE_URL = "";
 process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY = "";
 ```
 ### Test Helper for Next.js API Routes
 ```typescript
 // __tests__/helpers.ts
 import { NextRequest } from "next/server";
 export function createTestRequest(
  url: string,
  options?: {
    method?: string;
    body?: Record<string, unknown>;
    headers?: Record<string, string>;
    sandboxUserId?: string;
  },
 ): NextRequest {
  const { method = "GET", body, headers = {}, sandboxUserId } = options || {};
  const fullUrl = url.startsWith("http") ? url : `http://localhost:3000${url}`;
  const reqHeaders: Record<string, string> = { ...headers };
  if (sandboxUserId) {
    reqHeaders["x-sandbox-user-id"] = sandboxUserId;
  }
  const init: { method: string; headers: Record<string, string>; body?: string } = {
    method,
    headers: reqHeaders,
  };
  if (body) {
    init.body = JSON.stringify(body);
    reqHeaders["content-type"] = "application/json";
  }
  return new NextRequest(fullUrl, init);
 }
 export async function parseResponse(response: Response) {
  const json = await response.json();
  return { status: response.status, json };
 }
 ```
 ### Writing Regression Tests
 The key principle: **write tests for bugs that were found, not for code that works**.
 ```typescript
 // __tests__/api/user/profile.test.ts
 import { describe, it, expect } from "vitest";
 import { createTestRequest, parseResponse } from "../../helpers";
 import { GET, PATCH } from "@/app/api/user/profile/route";
 // Define the contract — what fields MUST be in the response
 const REQUIRED_FIELDS = [
  "id",
  "email",
  "full_name",
  "phone",
  "role",
  "created_at",
  "avatar_url",
  "notification_settings",  // ← Added after bug found it missing
 ];
 describe("GET /api/user/profile", () => {
  it("returns all required fields", async () => {
    const req = createTestRequest("/api/user/profile");
    const res = await GET(req);
    const { status, json } = await parseResponse(res);
    expect(status).toBe(200);
    for (const field of REQUIRED_FIELDS) {
      expect(json.data).toHaveProperty(field);
    }
  });
  // Regression test — this exact bug was introduced by AI 4 times
  it("notification_settings is not undefined (BUG-R1 regression)", async () => {
    const req = createTestRequest("/api/user/profile");
    const res = await GET(req);
    const { json } = await parseResponse(res);
    expect("notification_settings" in json.data).toBe(true);
    const ns = json.data.notification_settings;
    expect(ns === null || typeof ns === "object").toBe(true);
  });
 });
 ```
 ### Testing Sandbox/Production Parity
 The most common AI regression: fixing production path but forgetting sandbox path (or vice versa).
 ```typescript
 // Test that sandbox responses match the expected contract
 describe("GET /api/user/messages (conversation list)", () => {
  it("includes partner_name in sandbox mode", async () => {
    const req = createTestRequest("/api/user/messages", {
      sandboxUserId: "user-001",
    });
    const res = await GET(req);
    const { json } = await parseResponse(res);
    // This caught a bug where partner_name was added
    // to production path but not sandbox path
    if (json.data.length > 0) {
      for (const conv of json.data) {
        expect("partner_name" in conv).toBe(true);
      }
    }
  });
 });
 ```
 ## Integrating Tests into Bug-Check Workflow
 ### Custom Command Definition
 ```markdown
 <!-- .claude/commands/bug-check.md -->
 # Bug Check
 ## Step 1: Automated Tests (mandatory, cannot skip)
 Run these commands FIRST before any code review:
    npm run test       # Vitest test suite
    npm run build      # TypeScript type check + build
 - If tests fail → report as highest priority bug
 - If build fails → report type errors as highest priority
 - Only proceed to Step 2 if both pass
 ## Step 2: Code Review (AI review)
 1. Sandbox / production path consistency
 2. API response shape matches frontend expectations
 3. SELECT clause completeness
 4. Error handling with rollback
 5. Optimistic update race conditions
 ## Step 3: For each bug fixed, propose a regression test
 ```
 ### The Workflow
 ```
 User: "バグチェックして" (or "/bug-check")
  │
  ├─ Step 1: npm run test
  │   ├─ FAIL → Bug found mechanically (no AI judgment needed)
  │   └─ PASS → Continue
  │
  ├─ Step 2: npm run build
  │   ├─ FAIL → Type error found mechanically
  │   └─ PASS → Continue
  │
  ├─ Step 3: AI code review (with known blind spots in mind)
  │   └─ Findings reported
  │
  └─ Step 4: For each fix, write a regression test
      └─ Next bug-check catches if fix breaks
 ```
 ## Common AI Regression Patterns
 ### Pattern 1: Sandbox/Production Path Mismatch
 **Frequency**: Most common (observed in 3 out of 4 regressions)
 ```typescript
 // ❌ AI adds field to production path only
 if (isSandboxMode()) {
  return { data: { id, email, name } };  // Missing new field
 }
 // Production path
 return { data: { id, email, name, notification_settings } };
 // ✅ Both paths must return the same shape
 if (isSandboxMode()) {
  return { data: { id, email, name, notification_settings: null } };
 }
 return { data: { id, email, name, notification_settings } };
 ```
 **Test to catch it**:
 ```typescript
 it("sandbox and production return same fields", async () => {
  // In test env, sandbox mode is forced ON
  const res = await GET(createTestRequest("/api/user/profile"));
  const { json } = await parseResponse(res);
  for (const field of REQUIRED_FIELDS) {
    expect(json.data).toHaveProperty(field);
  }
 });
 ```
 ### Pattern 2: SELECT Clause Omission
 **Frequency**: Common with Supabase/Prisma when adding new columns
 ```typescript
 // ❌ New column added to response but not to SELECT
 const { data } = await supabase
  .from("users")
  .select("id, email, name")  // notification_settings not here
  .single();
 return { data: { ...data, notification_settings: data.notification_settings } };
 // → notification_settings is always undefined
 // ✅ Use SELECT * or explicitly include new columns
 const { data } = await supabase
  .from("users")
  .select("*")
  .single();
 ```
 ### Pattern 3: Error State Leakage
 **Frequency**: Moderate — when adding error handling to existing components
 ```typescript
 // ❌ Error state set but old data not cleared
 catch (err) {
  setError("Failed to load");
  // reservations still shows data from previous tab!
 }
 // ✅ Clear related state on error
 catch (err) {
  setReservations([]);  // Clear stale data
  setError("Failed to load");
 }
 ```
 ### Pattern 4: Optimistic Update Without Proper Rollback
 ```typescript
 // ❌ No rollback on failure
 const handleRemove = async (id: string) => {
  setItems(prev => prev.filter(i => i.id !== id));
  await fetch(`/api/items/${id}`, { method: "DELETE" });
  // If API fails, item is gone from UI but still in DB
 };
 // ✅ Capture previous state and rollback on failure
 const handleRemove = async (id: string) => {
  const prevItems = [...items];
  setItems(prev => prev.filter(i => i.id !== id));
  try {
    const res = await fetch(`/api/items/${id}`, { method: "DELETE" });
    if (!res.ok) throw new Error("API error");
  } catch {
    setItems(prevItems);  // Rollback
    alert("削除に失敗しました");
  }
 };
 ```
 ## Strategy: Test Where Bugs Were Found
 Don't aim for 100% coverage. Instead:
 ```
 Bug found in /api/user/profile     → Write test for profile API
 Bug found in /api/user/messages    → Write test for messages API
 Bug found in /api/user/favorites   → Write test for favorites API
 No bug in /api/user/notifications  → Don't write test (yet)
 ```
 **Why this works with AI development:**
 1. AI tends to make the **same category of mistake** repeatedly
 2. Bugs cluster in complex areas (auth, multi-path logic, state management)
 3. Once tested, that exact regression **cannot happen again**
 4. Test count grows organically with bug fixes — no wasted effort
 ## Quick Reference
 | AI Regression Pattern | Test Strategy | Priority |
 |---|---|---|
 | Sandbox/production mismatch | Assert same response shape in sandbox mode | 🔴 High |
 | SELECT clause omission | Assert all required fields in response | 🔴 High |
 | Error state leakage | Assert state cleanup on error | 🟡 Medium |
 | Missing rollback | Assert state restored on API failure | 🟡 Medium |
 | Type cast masking null | Assert field is not undefined | 🟡 Medium |
 ## DO / DON'T
 **DO:**
 - Write tests immediately after finding a bug (before fixing it if possible)
 - Test the API response shape, not the implementation
 - Run tests as the first step of every bug-check
 - Keep tests fast (< 1 second total with sandbox mode)
 - Name tests after the bug they prevent (e.g., "BUG-R1 regression")
 **DON'T:**
 - Write tests for code that has never had a bug
 - Trust AI self-review as a substitute for automated tests
 - Skip sandbox path testing because "it's just mock data"
 - Write integration tests when unit tests suffice
 - Aim for coverage percentage — aim for regression prevention