Files
lolympus-agent/agent-improvements-plan.md
T
2026-05-14 14:28:59 -07:00

887 lines
42 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# lolympus-agent — Research-Driven Improvements (Codename: goofy-plum)
> **Audience**: Codex(또는 다른 AI 에이전트)가 self-contained 하게 작업할 수 있도록 작성됨. 기존 `plan.md` (Phase 0–5)와 `evaluation-lively-island.md` (Phase 6–9)가 이미 production 인프라(shadow eval / judge / drift / promote / tournament)를 완성한 상태를 전제로 한다.
>
> **충돌 시 우선순위**: 본 문서 > `evaluation-lively-island.md` > `plan.md`. 단 Phase 0–9의 인터페이스/스키마/env 컨벤션은 그대로 따른다.
>
> **참고 논문 (papers/ 내)**
> - **Paper A** — arXiv 2602.12430v3 (Xu & Yan, 2026.02) *Agent Skills for LLMs*. SKILL.md 명세, progressive disclosure(L1/L2/L3), Tool Search Tool, Programmatic Tool Calling, Skill Trust & Lifecycle Governance(G1–G4 gates · T1–T4 tiers).
> - **Paper B** — arXiv 2502.09054v1 (Zellinger et al., 2025.02) *Cost-Saving LLM Cascades with Early Abstention*. cheap model이 cascade 초반에 abstain → cost −13%, error −5%, abstention +4.1%.
> - **Paper C** — arXiv 2605.06350v1 (Bouchard, 2026.05) *Decision-Theoretic Characterization of LLM Cascades*. k>2 cascade는 잘 튜닝된 2-model pairwise envelope 대비 우월하지 않음. **Pre-generation diagnostic router**가 4/5 benchmarks에서 cascade 우위.
---
## 1. Context — 왜 이 작업
### 문제 (현 production agent의 격차)
1. **비용 효율의 구조적 한계** — `OnboardingOrchestrator.run()`은 `derived.tier`만 보고 tier0/1/2 LLM 호출. classify confidence가 낮아도 동일 비용. Paper B/C 기준 cheap-model generation 자체를 피할 수 있는 *pre-generation* 결정 지점이 부재.
2. **Tool 사용의 견고성/확장성 한계** — `tool-loop.ts`는 `maxHops=2` 고정, retry/error 분류 없음. tool 1개 실패 시 throw로 전체 흐름 정지. 신규 skill 도입 시 도구 prompt bloat 폭증 구조.
3. **Skill 추상화 부재** — `PromptBuilder.buildSystemPrompt()`가 persona/policy/state/examples/context를 매 요청 1회 직렬화(~1800 tokens). 신규 skill 추가가 단기 결정됨에 따라 SKILL.md progressive disclosure 도입 필수.
4. **자동 propose의 안전망 부재** — `eval/loop/propose.ts`는 LLM이 생성한 prompt variant를 직접 file에 기록. injection / unintended behavior 차단 게이트 없음. Paper A: community skill 26.1%에 취약점.
### 의도한 결과 (3개월 horizon)
- 평균 응답 비용 **−15~20%**, p50 latency **−10~15%**
- LLM judge overall **regression 0** (또는 +0.3 이상)
- 신규 skill 추가 시간을 monolithic 대비 **5x 단축**할 수 있는 SKILL.md 인프라
- 자동 propose 변형이 G1–G4 gate를 거쳐 prod에 **안전 승급**되는 폐쇄 루프
### 비목표
- 코드 실행 sandbox 형태의 Programmatic Tool Calling (DAG 형태로 한정)
- 외부 embedding 서비스 의존 (로컬 Qwen-embed 만 사용; 미정인 동안 v3 router 보류)
- 단일 회사 외부 trust tier T4(vendor-certified) 자동화 — 인간 검토 유지
---
## 2. 사용자 결정 (반영됨)
| 항목 | 결정 |
|---|---|
| Phase α 우선 가치 | **균형** — Early Abstention + Tool Error Recovery + Programmatic 시범 |
| Abstention 정책 | **Hybrid (대화 위치 기반)** — 초반 turn은 짧은 확인 질문, 반복/혼란 시그널은 `[NO_RESPONSE]` |
| Skill 확장 | **단기 내 추가 예정** → SKILL.md(#1) Phase β로, Governance(#7) Phase γ 앞부분으로 |
---
## 3. Relationship to Existing Plans (변경 영향)
### 그대로 사용 (변경 없음)
- `src/eval/{capture,anonymize,runner,types,shadow-bus,shadow-worker,aggregator,archive,drift,promote}.ts`
- `src/eval/judges/{rule-judge,llm-judge}.ts`
- `src/eval/loop/{tournament,elo}.ts`
- `src/eval/review/{queue,label-archive,calibration}.ts`
- `src/eval/golden/*.jsonl` (스키마 호환 유지)
- Redis/Garnet 키, NATS subjects, env 컨벤션
- 익명화(`anonymize.ts`) — 신규 user-memory 쓰기에 강제 적용
### 확장 / 변경
- `src/skills/onboarding/runtime/{orchestrator,tool-loop,critic,trace}.ts` — 행동 분기 추가, 시그니처 일부 확장
- `src/skills/onboarding/state/{derive,onboarding-state,blackboard}.ts` — `routingDecision`, `abstentionScore`, long-term key 추가
- `src/skills/onboarding/prompts/{builder,layers,examples}.ts` — SKILL.md mode 분기, retriever 호출
- `src/llm/routing.ts` — `selectModelsForProvider` · `selectProviderOrder` 시그니처에 `routingDecision` 옵션 추가
- `src/eval/judges/process-judge.ts` — PRM forward score 통합
- `src/eval/loop/propose.ts` — T1 자동 진입 + gate 호출
- `src/config.ts` — 신규 env 항목 (Section 4 참조)
### 신규 파일
| 경로 | 역할 | Phase |
|---|---|---|
| `src/skills/onboarding/runtime/abstention.ts` | Hybrid early abstention 결정 | α |
| `src/skills/onboarding/runtime/tool-plan-runner.ts` | DAG 기반 programmatic tool calling | α |
| `src/skills/onboarding/runtime/router.ts` | Pre-gen diagnostic router | β |
| `src/skills/onboarding/runtime/runtime-flag-manifest.ts` | tournament 가능한 runtime flag variant | α (시작) / β (확장) |
| `src/skills/onboarding/SKILL.md` | L1 metadata + L2 body | β |
| `src/skills/onboarding/resources/*.md` | L3 resources (nickname-format, heart-rule, discord-format, age-policy) | β |
| `src/skills/skill-loader.ts` | SkillRegistry — 모든 SKILL.md 인덱스 | β |
| `src/skills/onboarding/state/user-memory.ts` | UserMemoryStore (long-term) | β |
| `src/skills/onboarding/prompts/example-store.ts` | ExampleRetriever | β |
| `src/eval/judges/prm.ts` | Process Reward Model (v1 Ridge) | β |
| `src/eval/judges/attribution.ts` | outcome → trace step root cause | β |
| `src/eval/governance/gates.ts` | G1–G4 gates | γ |
| `src/eval/governance/trust.ts` | T1–T4 promotion/demotion | γ |
| `src/tools/tool-search.ts` | search_tools meta tool | γ |
### 신규 의존
- 없음. 모든 신규 기능은 기존 의존(Garnet, MongoDB driver, Zod, LlmClient, Pino, OpenTelemetry)로 구현 가능.
- (v3 router·example-store v2 시점에) 로컬 Qwen-embed가 필요 — 그 시점 별도 결정 (현 단계 보류).
---
## 4. 신규 환경 변수 / 설정
`src/config.ts`에 추가. 모든 신기능은 default OFF로 시작 → shadow eval 후 ON.
```
# Phase α
AGENT_ABSTENTION_ENABLED=false
AGENT_ABSTENTION_THRESHOLD=0.35 # ξ — [NO_RESPONSE] 임계
AGENT_ABSTENTION_CLARIFY_THRESHOLD=0.45 # φ_clarify — clarify 질문 임계
AGENT_TOOL_RETRY_MAX=1
AGENT_TOOL_LOOP_MAX_HOPS=3 # 기존 2 → 3 (precondition retry 여유)
AGENT_PROGRAMMATIC_TOOL_CALLING=false # variant manifest 노출도 가능
# Phase β
AGENT_SKILL_MD_MODE=false # SKILL.md L1/L2/L3 disclosure
AGENT_ROUTER_MODE=static # static | learned-v1 | learned-v2
AGENT_ROUTER_WEIGHTS_PATH=
AGENT_USER_MEMORY_ENABLED=false
AGENT_USER_MEMORY_TTL_DAYS=30
EVAL_PRM_ENABLED=false
EVAL_PRM_WEIGHTS_PATH=
# Phase γ
AGENT_TRUST_MIN_TIER=T2 # 미달 variant는 control fallback
AGENT_CRITIC_MAX_ITERATIONS=2
AGENT_TOOL_SEARCH_ENABLED=false
```
---
## 5. Phase α (Week 1-2) — 즉시 효과·낮은 위험
### TASK-α-2-01 · Hybrid Early Abstention
**파일**: `src/skills/onboarding/runtime/abstention.ts` (신규), `src/skills/onboarding/runtime/orchestrator.ts`, `src/skills/onboarding/state/onboarding-state.ts`, `src/config.ts`, `src/skills/onboarding/onboarding-templates.ts`
**시그니처**:
```ts
// src/skills/onboarding/runtime/abstention.ts
export type AbstentionAction =
| { kind: "proceed" }
| { kind: "clarify"; templateKey: "abstain-clarify-default" | string }
| { kind: "no-response" };
export interface AbstentionDecision {
action: AbstentionAction;
reason:
| "low-confidence-early-turn"
| "low-confidence-repeated"
| "off-topic-strong"
| "flagged-confused"
| null;
score: number; // 0..1
features: {
classifyConfidence: number;
intentKind: Intent["kind"];
turnIndex: number;
sameIntentStreak: number;
flaggedIssues: number;
};
}
export function decideEarlyAbstention(input: {
classified: ClassifiedIntent;
derived: DerivedState;
blackboard: Blackboard | null;
request: AgentRequest;
thresholds: {
abstain: number; // AGENT_ABSTENTION_THRESHOLD
clarify: number; // AGENT_ABSTENTION_CLARIFY_THRESHOLD
};
}): AbstentionDecision;
```
**룰 (Hybrid)**:
1. `intent.kind === "off-topic-to-other"` && `confidence > 0.9` → `no-response`, reason=`off-topic-strong`.
2. `flaggedIssues.length > 0` || `sameIntentStreak >= 3` (blackboard.lastIntent 비교) → `no-response` if confidence < abstain, reason=`flagged-confused` or `low-confidence-repeated`.
3. `turnIndex <= 1` (greetedAt 미존재) && `confidence < clarify` && intent ∈ {unknown, chitchat} → `clarify` (templateKey=`abstain-clarify-default` = "조금 더 구체적으로 알려주시면 도와드릴게요!"), reason=`low-confidence-early-turn`.
4. 그 외 → `proceed`.
**orchestrator 통합** (`runtime/orchestrator.ts:run()` 진입부 직후):
```ts
if (getAbstentionEnabled()) {
const decision = decideEarlyAbstention({
classified, derived, blackboard, request,
thresholds: { abstain: getAbstentionThreshold(), clarify: getAbstentionClarifyThreshold() },
});
derived.abstentionScore = decision.score;
traceCollector?.startStep("classify").finish({
confidence: classified.confidence,
intent: classified.intent.kind,
source: classified.source,
abstention: decision, // process-trace에 기록
});
if (decision.action.kind === "no-response") {
return { text: "[NO_RESPONSE]", validation: { status: "pass", reasons: ["early-abstention"] } };
}
if (decision.action.kind === "clarify") {
return { text: getOnboardingTemplate(decision.action.templateKey), validation: { status: "pass", reasons: ["early-abstention-clarify"] } };
}
}
```
**템플릿 추가** (`onboarding-templates.ts`):
```ts
"abstain-clarify-default": "조금 더 구체적으로 알려주시면 도와드릴게요!",
```
**Trace 확장** (`runtime/trace.ts`):
- `ProcessTraceStep<classify>.output`에 `abstention?: AbstentionDecision` 필드 추가 (옵셔널, 기존 호환).
**Eval 연동**:
- `src/eval/aggregator.ts`: slice에 `abstentionReason`, `abstentionAction` 차원 추가.
- `src/eval/drift.ts`: abstention rate (action≠proceed 비율)를 metric으로 등록.
- 통과 기준: 7일 shadow에서 cost −10% 이상, false-abstention < 2%, quality regression < 0.05.
**Acceptance**:
- [ ] `pnpm test src/skills/onboarding/runtime/abstention.test.ts` (신규 unit test 5+ case: off-topic / early-low-conf / repeated / proceed / boundary)
- [ ] env OFF에서 기존 behavior 100% 일치 (snapshot 비교)
- [ ] env ON 24h shadow에서 abstention rate 0.05~0.25 사이 (밴드 밖이면 임계 재튜닝)
---
### TASK-α-8a-01 · Structured Tool Error Recovery
**파일**: `src/skills/onboarding/runtime/tool-loop.ts`, `src/tools/tool.ts`, `src/skills/onboarding/runtime/trace.ts`, `src/config.ts`
**타입 확장** (`src/tools/tool.ts`):
```ts
export type ToolErrorKind = "args-invalid" | "transient" | "precondition-failed" | "permanent";
export interface ToolExecutionFailure {
kind: ToolErrorKind;
toolName: string;
message: string;
retryable: boolean;
details?: unknown;
}
```
**tool-loop.ts 변경 요지** (현 `:113-137` 영역):
- zod parse 실패 → `args-invalid` → 1 hop 자기교정 (assistant tool_call → user "이전 args가 schema 어긋남" 안내). 이때 hop budget을 consume 하되 별도 카운터 `selfCorrectionHops` 추적해 무한 루프 차단(max 1).
- fetch/timeout 등 network error 감지 (`AbortError`/`FetchError`/timeout) → `transient` → 200ms backoff → 1회 retry (`AGENT_TOOL_RETRY_MAX`). hop budget 사용 안 함.
- tool 내부 throw → `permanent` → `{ ok: false, errorKind: "permanent", message }` 를 tool 결과로 message에 push, hop 계속.
- precondition 검증 실패 (예: session 없음) → tool이 `result.ok=false`로 정상 반환하도록 builtins 수정 → tool-loop에서 `precondition-failed`로 분류, hop budget consume 안 함.
**maxHops**: 기본 `getAgentToolLoopMaxHops()` (default 3). precondition-failed로 인한 추가 hop은 +1 까지만 허용.
**Trace 확장**:
```ts
toolCalls?: Array<ProcessTraceStep<{
name: string;
args: unknown;
ok: boolean;
resultSummary: string;
errorKind?: ToolErrorKind; // 신규
retried?: boolean; // 신규
}>>;
```
**재사용**:
- `parseToolCalls` (`src/tools/parser.ts`) — 변경 없음
- 기존 `traceStep?.finish({ ok: false, ... })` 경로
**Acceptance**:
- [ ] `src/skills/onboarding/runtime/tool-loop.test.ts` (신규): mock tool throw / network timeout / zod-invalid / precondition-failed 각 케이스
- [ ] tool-heavy slice(nickname-rejected, discord-verify-question) shadow eval에서 success rate +3% 이상
- [ ] p95 latency 변화 < +200ms
---
### TASK-α-4a-01 · Programmatic Tool Calling 시범 (선언적 DAG)
**파일**: `src/skills/onboarding/runtime/tool-plan-runner.ts` (신규), `src/skills/onboarding/runtime/planner-schema.ts`, `src/skills/onboarding/runtime/orchestrator.ts`
**스키마 확장** (`planner-schema.ts`):
```ts
export const toolPlanStepSchema = z.object({
id: z.string().min(1),
tool: z.string().min(1),
argsFromContext: z.record(z.string(), z.string()).optional(), // "$request.nickname"
argsLiteral: z.record(z.string(), z.unknown()).optional(),
dependsOn: z.array(z.string()).optional(),
on: z.object({ error: z.enum(["abort", "fallback-llm"]).optional() }).optional(),
});
export const toolPlanSchema = z.object({
steps: z.array(toolPlanStepSchema).max(4), // 안전 한계
composeWith: z.enum(["all-then-llm", "first-then-llm"]),
});
export const plannerOutputSchema = z.object({
approach: z.string(),
mustInclude: z.array(z.string()).optional(),
mustAvoid: z.array(z.string()).optional(),
suggestedTools: z.array(z.string()).optional(),
targetLengthChars: z.number().optional(),
toolPlan: toolPlanSchema.optional(), // NEW
});
```
**runner** (`tool-plan-runner.ts`):
```ts
export interface ToolPlanResult {
results: Record<string /* step id */, unknown>;
failures: ToolExecutionFailure[];
durationMs: number;
}
export async function runToolPlan(
plan: ToolPlan,
tools: readonly Tool<unknown, unknown>[],
ctx: ToolContext,
request: AgentRequest,
): Promise<ToolPlanResult>;
```
- DAG 정렬 (`dependsOn`) 후 `Promise.all`로 동일 레벨 동시 실행 (단 동일 NATS RPC 큐 부담 고려해 `Promise.all` 최대 2 병렬).
- `$request.nickname`, `$request.userId`, `$session.openQuestions[0]` 등 path 표현식 resolve.
- 실패 시 `on.error === "fallback-llm"`이면 tool-loop으로 fallback (free-form), `abort`이면 즉시 종료.
**orchestrator 통합** (`run()` planner 출력 처리 부분):
- planner output에 `toolPlan` 있고 `getProgrammaticToolCalling()` ON이면 `runToolPlan` 먼저 → 결과를 message[]에 tool messages로 push → responder 1회 호출.
- 없으면 기존 `runWithTools` 경로.
**variant 노출**:
- `src/eval/proposals/planner-with-toolplan.json` 신규 — planner system layer에 `"가능하면 toolPlan 필드를 채워주세요"` append.
- Tournament로 control vs toolplan 비교. 통과 기준 ELO +20, latency p50 −150ms 이상.
**Acceptance**:
- [ ] `src/skills/onboarding/runtime/tool-plan-runner.test.ts` (신규): 정상 DAG / 의존성 누락 / step 실패 fallback 케이스
- [ ] process-judge 확장: `toolPlan.steps.length` vs `actualToolCalls.length` consistency check (`src/eval/judges/process-judge.ts`)
---
### TASK-α-FLAG-01 · RuntimeFlagManifest 인프라
**파일**: `src/skills/onboarding/runtime/runtime-flag-manifest.ts` (신규), `src/skills/onboarding/runtime/orchestrator.ts`, `src/eval/loop/tournament.ts`
**타입**:
```ts
export interface RuntimeFlagManifest {
variantId: string; // e.g. "abstention-on", "router-v1"
description: string;
flags: Partial<{
abstentionEnabled: boolean;
abstentionThreshold: number;
abstentionClarifyThreshold: number;
programmaticToolCalling: boolean;
toolLoopMaxHops: number;
routerMode: "static" | "learned-v1" | "learned-v2";
userMemoryEnabled: boolean;
criticMaxIterations: number;
}>;
}
export function loadRuntimeFlagManifest(variantId: string): RuntimeFlagManifest | null;
export function resolveRuntimeFlags(
base: ResolvedRuntimeFlags,
variant?: RuntimeFlagManifest,
): ResolvedRuntimeFlags;
```
**효과**: `PromptVariantManifest`(prompt만)와 별도로 행동 변화 variant를 tournament로 비교 가능. `tournament.ts`에서 `loadVariantBundle(variantId)`가 prompt+runtime 둘 다 로드하도록 확장.
**Acceptance**:
- [ ] `runtime-flag-manifest.test.ts`: variant 로드 / merge / 잘못된 키 거부
- [ ] tournament가 동일 골든셋에 대해 control / abstention-on / toolplan-on 3종 ELO 산출 가능
---
### Phase α Exit 게이트
전부 만족 시 Phase β 진입.
| 메트릭 | 측정 위치 | 목표 |
|---|---|---|
| Cost (shadow 7일 평균) | `eval/aggregator.ts` slice별 latencySumMs * tier price | −10% vs control |
| LLM judge overall | `aggregator.llmOverallAvg` | ≥ baseline − 0.05 |
| Abstention false-positive | `review/queue.ts` 라벨 vs decision | < 2% |
| Tool failure → soft-template | `validator.ts` reject 사례 비율 | −30% |
| Regression alerts | `drift.ts` significant count | 0 |
---
## 6. Phase β (Week 3-5) — Skill 확장 대비 + 평가 정밀화
### TASK-β-1-01 · SKILL.md L1/L2/L3 Progressive Disclosure
**핵심 가정**: 단기 내 추가 skill(discord-help / community-rules / identity-room 중 1+) 도입 결정. SKILL.md 인프라가 없으면 prompt 폭증.
**신규 파일**:
- `src/skills/onboarding/SKILL.md` — frontmatter + body
- `src/skills/onboarding/resources/{nickname-format,heart-rule,discord-format,age-policy}.md`
- `src/skills/skill-loader.ts` — SkillRegistry
**SKILL.md frontmatter 스키마**:
```yaml
---
name: onboarding
description: 카카오톡 오픈채팅 새내기 등록을 도와주는 친절한 선배 역할.
version: 1
intents: [greeting, acknowledgement, status-question, help-how, help-where, nickname-question, photo-question, discord-question, discord-verify-question, age-question]
triggers: [user-message, new-member, nickname-invalid, photo-failed, progress-update, timeout-warning, completed]
tools: [validate-nickname, check-onboarding-progress, lookup-community-rule, get-room-context, verify-discord-nickname]
resources:
nickname-format: ./resources/nickname-format.md
heart-rule: ./resources/heart-rule.md
discord-format: ./resources/discord-format.md
age-policy: ./resources/age-policy.md
trustTier: T3
---
```
**SkillRegistry**:
```ts
// src/skills/skill-loader.ts
export interface SkillManifest {
name: string;
description: string;
version: number;
intents: string[];
triggers: string[];
tools: string[];
resources: Record<string, string>; // key → relative path
trustTier: TrustTier;
bodyPath: string; // SKILL.md absolute path
}
export class SkillRegistry {
static load(rootDir: string): SkillRegistry; // 모든 SKILL.md frontmatter 메모리 로드
list(): SkillManifest[];
get(name: string): SkillManifest | null;
loadBody(name: string): Promise<string>; // L2 lazy load
loadResource(name: string, key: string): Promise<string>; // L3 lazy load
}
```
**PromptBuilder 재설계** (`prompts/builder.ts`):
- env `AGENT_SKILL_MD_MODE=true`일 때:
1. system 1차 메시지: 모든 skill manifest의 `description` 만 ~30 tokens/skill 직렬화 (L1).
2. 매칭된 skill의 body(`loadBody`)를 system 2차 메시지로 (L2).
3. planner output에 `requestResources: ["nickname-format"]` 있으면 responder 호출 직전에 해당 resource MD를 system 3차 메시지로 inject (L3).
- 기존 layer 함수(`personaLayer`, `policyLayer`, ...)는 그대로 두되 `policyLayer`만 intent-conditional layer로 분리되어 L2 단계에서 inject. (점진 마이그레이션)
**planner-schema 확장**:
```ts
requestResources: z.array(z.string()).optional(),
```
**마이그레이션**:
- Step 1: SKILL.md를 mirror로 생성. 기존 prompt 결과와 token 차이 측정.
- Step 2: feature flag ON 후 24h shadow eval. quality regression < 0.05 확인.
- Step 3: L3 resource를 nickname-related intent부터 적용 (가장 적은 intent → 점진).
**Acceptance**:
- [ ] SkillRegistry unit test: 잘못된 frontmatter / 빈 resources / 중복 name
- [ ] snapshot test: system prompt token count baseline ↔ SKILL.md mode (≤ −40%)
- [ ] 30일 long-soak shadow: regression < 0.05
- [ ] dry-run: 새 skill 추가가 SKILL.md 1개 + registry 1줄로 동작
---
### TASK-β-5-01 · Process Reward Model (PRM) + Bidirectional Attribution
**신규 파일**:
- `src/eval/judges/prm.ts` — feature extractor + Ridge regression (사용 라이브러리: 직접 구현, dependencies 최소)
- `src/eval/judges/attribution.ts` — outcome → trace step root cause
**Feature vector** (Section 5 #5 PRM 참조):
```ts
export interface PrmFeatureVector {
classifyConfidence: number; classifySource: 0|1;
planApproach: string; // categorical hash
toolCallCount: number; toolErrorCount: number;
criticDecision: 0|1; validateStatus: 0|1|2; tier: 0|1|2;
abstentionScore: number; // Phase α에서 추가됨
}
```
**Forward model (v1)**: Ridge regression on label-archive humanScore vs features. weights는 `EVAL_PRM_WEIGHTS_PATH`(JSON)에 저장.
**Backward attribution**:
- 각 entry에 대해 trace step별 counterfactual delta (step output 평균값으로 swap 시 PRM score 변화) → `attributedScoreLoss`.
- 결과 collection: `eval_process_attribution` (MongoDB).
**process-judge 통합** (`eval/judges/process-judge.ts`):
```ts
// 기존 ProcessJudgeResult에 추가
prmScore?: number; // PRM forward
prmFeatures?: PrmFeatureVector;
rootCauseStep?: ProcessTraceStepName;
rootCauseScoreLoss?: number;
```
**Background job 추가** (`workers/background.ts`):
- 새 tick: `EVAL_ATTRIBUTION_TICK_MS` (default 6h). 최근 archive entries에 대해 attribution 계산 후 collection 적재.
- 라벨 수가 N(env `EVAL_PRM_MIN_LABELS`, default 500) 미만이면 forward 모델은 "explanation only" 모드(score 출력 X, root cause만).
**propose 입력 확장** (`eval/loop/propose.ts`):
- `buildProposalInputFromArchive`에 `rootCauses: RootCauseReport[]` 필드 추가. LLM이 root cause 분포를 보고 더 타겟팅된 prompt diff 생성.
**Acceptance**:
- [ ] `prm.test.ts`: feature vector 안정성, weight 로드/저장
- [ ] `attribution.test.ts`: synthetic trace에서 root cause step 식별 정확도 ≥ 0.8
- [ ] tournament: PRM-augmented variant가 ELO +30 차이를 식별 가능 (synthetic label)
---
### TASK-β-3-01 · Pre-Generation Diagnostic Router (v1 룰)
**신규 파일**: `src/skills/onboarding/runtime/router.ts`
**Decision space**:
```ts
export type RoutingDecision =
| { kind: "abstain"; reason: string } // = TASK-α-2 abstention
| { kind: "template-only"; templateKey: string }
| { kind: "tier0-only" }
| { kind: "tier1-direct"; skipPlanner: boolean }
| { kind: "tier1-with-tools" }
| { kind: "tier2-full" };
export interface DiagnosticRouter {
decide(input: RouterInput): RoutingDecision;
}
```
**v1 rules** (`router.ts` 초안):
- `intent.kind === "off-topic-to-other"` && conf > 0.9 → `abstain`
- `intent.kind === "greeting"` && conf > 0.95 && no prior greeting → `template-only`
- `intent.kind === "acknowledgement"` && conf > 0.9 → `template-only`
- `phase.kind === "nickname-rejected"` → `tier1-with-tools`
- `phase.kind === "completed"` && intent ≠ status-question → `template-only`
- `phase.kind ∈ {fresh, nickname-pending, photo-pending, both-pending}` && intent ∈ {nickname-question, photo-question, help-how} → `tier1-with-tools`
- 그 외 → `tier0-only`
**derive.ts 통합**: `deriveState()` 끝에서 `router.decide(...)` 호출, 결과를 `DerivedState.routingDecision`에 저장.
**llm/routing.ts 확장**:
- `selectModelsForProvider(provider, options, llamaFallbackModel)` 시그니처에 `routingDecision?: RoutingDecision` 추가.
- llama 분기(`routing.ts:24-25`)는 `routingDecision.kind`가 `tier1-with-tools` 이상이면 fallback 모델 대신 explicit fallback 사용; 현 단일 모델 환경에서는 effective change 0 (multi-llama 환경 대비 hook 추가).
- `selectProviderOrder(endpoints, options)`도 `routingDecision`을 받아 cliproxy/openrouter/llama 우선순위를 동적 조정 (예: `template-only` decision이면 llama only).
**Acceptance**:
- [ ] `router.test.ts`: 30+ 케이스로 v1 룰 커버
- [ ] Shadow A/B 24h: cost −8%, quality flat (LLM judge overall delta < 0.05)
- [ ] `routingDecision`을 `ShadowEvalEnvelope`에 기록 (`shadow-bus.ts`) — aggregator slice 추가
---
### TASK-β-3-02 · Router v2 학습
**의존**: TASK-β-5-01 (PRM features). 데이터 ≥ 5000 archive entries.
**파일**: `src/skills/onboarding/runtime/router.ts` 확장, `scripts/train-router.ts` (신규 script)
**모델**: lightweight logistic regression (multi-class: 6 decision kinds). 입력은 PRM `extractFeatures` + 사용자 메시지 길이/언어 식별.
**학습 데이터 라벨링**:
- archive에서 actual derived.tier · validation status · llmJudge.overall 기반 ground truth tier 추정:
- llmJudge.overall ≥ 4 + tier=0 → `tier0-only` 충분
- llmJudge.overall < 3 + tier=0 → `tier1-direct` 필요했음 (under-routed)
- llmJudge.overall < 3 + tier=1 → `tier2-full` 필요했음
- 5% exploration noise (`epsilon=0.05`) 도입 — 무작위로 1 tier ↑
**Acceptance**:
- [ ] `train-router.ts` 가 weights JSON 생성 (`AGENT_ROUTER_WEIGHTS_PATH`)
- [ ] Tournament: `router-v1` vs `router-v2` 비교 — v2 cost −5% 추가 절감
---
### TASK-β-6-01 · Persistent User Memory + Adaptive Few-shot
**신규 파일**:
- `src/skills/onboarding/state/user-memory.ts`
- `src/skills/onboarding/prompts/example-store.ts`
**Garnet 키**: `lolympus-agent:user:profile:{anonHash(userId)}` (TTL `AGENT_USER_MEMORY_TTL_DAYS` × 86400).
**UserMemoryStore**:
```ts
export interface UserProfile {
userId: string; // anonHash
firstSeenAt: string; lastSeenAt: string;
totalTurns: number;
preferredStyle: "concise" | "warm" | "neutral";
pastIntents: { kind: Intent["kind"]; count: number }[];
flaggedIssues: string[];
completedOnboardingAt?: string;
}
export class UserMemoryStore {
constructor(private garnet: GarnetClient, private salt: string) {}
async load(userId: string): Promise<UserProfile | null>;
async observeTurn(userId: string, derived: DerivedState, response: AgentResponse): Promise<UserProfile>;
}
```
**PII 강제**:
- 모든 string write에 `anonymizeText` (`src/eval/anonymize.ts`) 적용. unit test로 nickname/이메일/번호 미저장 검증.
- userId 자체는 `anonymizeIdentifier("user", userId)` 해시 사용.
**ExampleRetriever** (`prompts/example-store.ts`):
- v1: intent + phase로 필터링한 golden examples 중 user.preferredStyle과 매칭되는 최대 3개 반환.
- v2 (embedding): 보류 — 로컬 Qwen-embed 도입 결정 후.
**examplesLayer 시그니처 확장** (`prompts/examples.ts`):
```ts
export function examplesLayer(intent: Intent, retriever: ExampleRetriever, query: ExampleQuery): string;
```
**variant 노출**: `runtime-flag-manifest` `userMemoryEnabled: true`.
**Acceptance**:
- [ ] `user-memory.test.ts`: PII redaction 강제, observeTurn idempotent
- [ ] 같은 user(anon) 재방문 시 LLM judge friendliness 평균 +0.2 (shadow eval)
---
### Phase β Exit 게이트
| 메트릭 | 목표 |
|---|---|
| system prompt token | baseline 대비 −40% (SKILL.md mode ON) |
| 누적 cost (β 종료) | α 대비 추가 −8% |
| PRM attribution AUC | ≥ 0.75 (라벨 1000+) |
| 신규 skill dry-run | SKILL.md 1개 + registry 1줄로 동작 확인 |
---
## 7. Phase γ (Week 6+) — Governance + Reflection
### TASK-γ-7-01 · Skill Trust & Lifecycle Governance (G1–G4 · T1–T4)
**신규 파일**: `src/eval/governance/gates.ts`, `src/eval/governance/trust.ts`
**Gates 시그니처**:
```ts
export interface GateResult { passed: boolean; reason?: string; evidence?: unknown; }
export function runG1Static(manifest: PromptVariantManifest | RuntimeFlagManifest): GateResult;
// URL allowlist (validator.ts:8-11 재사용), layer 길이 제한, persona override 차단, 위험 키워드
export async function runG2Semantic(manifest, control, llmClient: LlmClient): Promise<GateResult>;
// LLM이 변형의 핵심 행동 변화 의도 분류 — "변경이 의도된 범위 내인가?"
export async function runG3Sandbox(manifest, evalRunner, goldenSetPath, n: number = 200): Promise<GateResult>;
// shadow run on N=200 cases, validator reject rate / llm overall 변화 측정
export function runG4Permission(manifest, toolRegistry: ToolRegistry, skillRegistry: SkillRegistry): GateResult;
// required tools/resources가 registry에 등록되어 있는지
```
**Trust Tiers** (`trust.ts`):
```ts
export type TrustTier = "T1-unvetted" | "T2-community" | "T3-org-vetted" | "T4-certified";
// Promotion 규칙
// T1 → T2: G1 + G4 통과 (자동)
// T2 → T3: G3 sandbox + tournament 100+ matches + ELO > control + 30 + drift OK 7일 (자동)
// T3 → T4: human review label-archive 평균 ≥ 4.0 + reviewer 2명 이상 동의 (semi-auto)
//
// Demotion: drift significant 또는 G3 sandbox regression → 즉시 T1로 강등 + alert
export async function evaluatePromotion(variantId: string, services: GovernanceServices): Promise<TrustTier>;
export async function applyDemotion(variantId: string, reason: string): Promise<void>;
```
**propose.ts 통합** (`eval/loop/propose.ts`):
- 새 proposal은 항상 T1로 저장. `writeVariantManifest` 호출 직후 `runG1Static` + `runG4Permission` 실행 → 통과 시 T2로 자동 승급.
**런타임 강제** (`src/skills/onboarding/skill.ts`):
- `getPromptVariantId()` 로드 시 `trust.getActiveTier(variantId)` 확인. `AGENT_TRUST_MIN_TIER` (default T2) 미달이면 control fallback.
**Background job**:
- `workers/background.ts`에 `EVAL_PROMOTION_TICK_MS` (default 30m). 모든 T1/T2 variant 대상 promotion 평가 실행.
**Acceptance**:
- [ ] `gates.test.ts`: 의도적 prompt injection variant가 G1에서 차단
- [ ] T1→T2 자동 승급, T2→T3 tournament 게이트 통과 시 자동 승급 시뮬레이션
- [ ] Drift significant 발생 시 즉시 강등 통합 테스트
---
### TASK-γ-4b-01 · Tool Search Tool
**신규 파일**: `src/tools/tool-search.ts`. `src/tools/registry.ts`에 `searchByDescription(query, k)` 추가.
```ts
export const toolSearchTool: Tool<{ query: string; topK?: number }, { tools: Array<{ name: string; description: string }> }> = {
name: "search_tools",
description: "도구 카탈로그에서 가장 적합한 도구 후보를 검색합니다.",
argsSchema: z.object({ query: z.string().min(2), topK: z.number().int().min(1).max(5).default(3) }),
resultSchema: z.object({ tools: z.array(z.object({ name: z.string(), description: z.string() })) }),
async execute(args, ctx) { /* registry.searchByDescription */ },
};
```
**registry 변경** (`tools/registry.ts`):
- `searchByDescription(query, k)`: 단순 keyword overlap → BM25 (lite, 의존 추가 없이 inline 구현).
- 도구 등록 시 description 인덱싱.
**orchestrator 통합** (`runtime/orchestrator.ts`):
- skill manifest의 `tools` 배열이 N 초과(env `AGENT_TOOL_SEARCH_THRESHOLD`, default 10)일 때만 `search_tools`만 노출 + planner가 `requestTools: [...]` 통해 expand.
**Acceptance**:
- [ ] 20 tool mock registry에서 nickname 관련 query → validate-nickname top-1 회수
- [ ] 도구 5개 미만 환경에서는 effective change 0 (기존 path)
---
### TASK-γ-8b-01 · Multi-step Self-Critic (Reflection)
**파일**: `src/skills/onboarding/runtime/critic.ts`, `src/skills/onboarding/runtime/trace.ts`
**시그니처 확장**:
```ts
export interface CritiqueResultV2 {
changed: boolean;
text: string;
iterations: number;
abstained: boolean;
rubricScores: { accuracy: number; scope: number; tone: number };
prmScoreBefore?: number;
prmScoreAfter?: number;
}
class OnboardingCritic {
async critiqueIterative(
draft: string, derived: DerivedState, messages: ChatMessage[],
options: { maxIters: number; deadlineMs: number; prm?: PrmModel },
): Promise<CritiqueResultV2>;
}
```
**알고리즘**:
1. PRM forward score → `prmScoreBefore`. score < `criticRubricFloor` && tier == 2 → 반복.
2. rubric 기반 prompt로 LLM 호출 → 점수 + 개선안.
3. `revise()` 1회. 다시 PRM. `prmScoreAfter - prmScoreBefore < δ`면 abort.
4. `iterations <= maxIters` (default 2). 결과 trace에 기록.
5. PRM이 `abstainFloor` 미만이면 `[NO_RESPONSE]` 반환 (Paper B 통합).
**Tier 적용**: tier 2만. tier 0/1은 기존 1-shot critic 유지.
**deadline 강제**: 매 iteration 시작 시 `Date.now() < deadlineMs` 체크. 초과면 즉시 반환.
**Acceptance**:
- [ ] tier 2 slice에서 LLM judge accuracy +0.3
- [ ] critic 평균 iterations 2 이하, p99 latency +400ms 이하
---
### Phase γ Exit 게이트
| 메트릭 | 목표 |
|---|---|
| 자동 propose-deploy 주기 | weekly 1+ variant T2 자동 승급 |
| Tier2 PRM rubric score | +0.3 이상 |
| 누적 cost (γ 종료) | baseline 대비 −15~20% |
---
## 8. 평가 시스템 통합 (변경별)
| 변경 | shadow-bus | aggregator slice | tournament | drift | promote |
|---|---|---|---|---|---|
| #2 Abstention | `decision.reason`/`action` | abstention rate | — | abstention drift | abstention-mismatch |
| #8a Tool Recovery | `errorKind` in trace | tool failure rate | — | tool error drift | tool-failure |
| #4a Programmatic | `toolPlan` | tool success rate | plan-mode vs free | tool round-trip | tool-failure |
| #1 SKILL.md | `disclosureMode` | system token usage | L1L2L3 vs flat | — | regression |
| #5 PRM | trace 그대로 | PRM score column | PRM-aug vs raw | PRM score drift | rule-prm-disagree |
| #3 Router | `routingDecision` | tier mix | router-on vs off | tier-mix drift | wrong-router |
| #6 Memory | `userProfileLoadedAt` | 재방문 user slice | with-mem vs without | memory hit | low-retention |
| #7 Governance | `variantTrustTier` | tier별 quality | 승급 게이트 | demotion alert | T1 강등 |
| #8b Self-Critic | `iterations`, `rubricScores` | tier2 cost | iter-1 vs iter-2 | critic loop drift | critic-loop-failure |
---
## 9. 검증 — End-to-End
### 개발 중
```bash
# 1. unit + integration test
pnpm test
# 2. golden set 회귀
pnpm run eval:run --input src/eval/golden/onboarding.jsonl
# ruleJudge.score, llmJudge.overall regression < 0.05 확인
# 3. tournament (variant 등록 후)
pnpm run eval:tournament --variants control,<new-variant>
# ELO 차이 측정
# 4. drift (5분 tick)
pnpm run eval:report --slice tier=0
```
### 운영 검증 (각 Phase 종료 후 1주)
- Aggregator dashboard: slice별 quality/cost/latency 추세
- Drift 알람 false-positive 0 (변경 의도와 일치)
- Promote candidates에 의미 있는 root cause 포함
### Rollback
- 모든 신규 행동은 env flag 또는 `RuntimeFlagManifest` 뒤에 배치 → 즉시 OFF
- shadow eval 단계에서 production 영향 0 (counterfactual 기록만)
---
## 10. 핵심 수정 파일 — 진입점 (Codex가 첫 PR에서 손댈 파일)
**Phase α 첫 PR (TASK-α-2-01 + TASK-α-FLAG-01)**:
1. `src/skills/onboarding/runtime/abstention.ts` (신규)
2. `src/skills/onboarding/runtime/orchestrator.ts` — abstention gate 진입
3. `src/skills/onboarding/state/onboarding-state.ts` — `DerivedState.abstentionScore?` 추가
4. `src/skills/onboarding/onboarding-templates.ts` — clarify 템플릿
5. `src/skills/onboarding/runtime/trace.ts` — classify step에 `abstention?` 필드
6. `src/skills/onboarding/runtime/runtime-flag-manifest.ts` (신규)
7. `src/config.ts` — `AGENT_ABSTENTION_*` env 추가
8. `src/skills/onboarding/runtime/abstention.test.ts` (신규)
**Phase α 두 번째 PR (TASK-α-8a-01)**:
1. `src/tools/tool.ts` — `ToolErrorKind`, `ToolExecutionFailure`
2. `src/skills/onboarding/runtime/tool-loop.ts` — retry/error 분류
3. `src/skills/onboarding/runtime/trace.ts` — toolCalls output에 `errorKind?` 추가
4. `src/skills/onboarding/runtime/tool-loop.test.ts` (신규)
**Phase α 세 번째 PR (TASK-α-4a-01)**:
1. `src/skills/onboarding/runtime/planner-schema.ts` — `toolPlan?` 필드
2. `src/skills/onboarding/runtime/tool-plan-runner.ts` (신규)
3. `src/skills/onboarding/runtime/orchestrator.ts` — toolPlan 우선 분기
4. `src/eval/proposals/planner-with-toolplan.json` (신규 variant)
5. `src/eval/judges/process-judge.ts` — toolPlan consistency check
6. `src/skills/onboarding/runtime/tool-plan-runner.test.ts` (신규)
**Phase β 핵심**:
- `src/skills/skill-loader.ts`, `src/skills/onboarding/SKILL.md`, `src/skills/onboarding/resources/*.md`
- `src/skills/onboarding/prompts/builder.ts` — SKILL.md mode
- `src/eval/judges/prm.ts`, `src/eval/judges/attribution.ts`, `src/eval/judges/process-judge.ts`
- `src/skills/onboarding/runtime/router.ts`, `src/skills/onboarding/state/derive.ts`, `src/llm/routing.ts`
- `src/skills/onboarding/state/user-memory.ts`, `src/skills/onboarding/prompts/example-store.ts`
**Phase γ**:
- `src/eval/governance/{gates,trust}.ts`
- `src/eval/loop/propose.ts` (T1 entry)
- `src/tools/{tool-search.ts,registry.ts}`
- `src/skills/onboarding/runtime/critic.ts`
---
## 11. 위험 & 미해결 질문
### 위험 (mitigation 동반)
1. **#2 abstention false-negative** → ξ=0.35 보수, in-conversation abstain 차단(turnIndex ≥ 2일 때 clarify 안 함), 24h shadow 우선.
2. **#3 router 학습 데이터 편향** (현 정책 재현) → epsilon=0.05 exploration noise, 라벨 보강.
3. **#5 PRM 초기 라벨 부족** → `EVAL_PRM_MIN_LABELS=500` 미만 시 explanation-only 모드(forward score 미노출).
4. **#6 PII 누출** → `UserMemoryStore.update`에 anonymize wrapper 강제, audit job 매일 1회.
5. **#1 SKILL.md 행동 변화** → 30일 long-soak, regression 시 `AGENT_SKILL_MD_MODE=false` 즉시 rollback.
6. **#7 propose 속도 저하** → T1→T2 자동(G1+G4만 요구)으로 가속 trade-off.
7. **#8b critic loop 폭주** → `AGENT_CRITIC_MAX_ITERATIONS=2` 강제, deadlineMs 체크.
### 미해결 질문 (구현 시작 전 사용자 확인 필요)
- [ ] **현재 label-archive 누적 라벨 수**? PRM v1 위해 ≥ 500. 부족하면 review queue 가속을 Phase α에 추가.
- [ ] **재방문 user rate** (chatroom 별 30일 동일 user turn 분포)? 메모리 ROI 산정.
- [ ] **로컬 embedding infra** 추가 가능? (router v3, example-store v2 위해 Qwen-embed 64d). 외부 사용 가능?
- [ ] **다음 skill 후보 명세** — discord-help · community-rules · identity-room 중 어떤 것이 먼저? SKILL.md L3 resources 설계 영향.
---
## 12. 작업 체크리스트 (codex가 PR 단위로 진행)
### Phase α (Week 1-2)
- [x] PR-A1: TASK-α-2-01 (Hybrid Early Abstention) + TASK-α-FLAG-01 (RuntimeFlagManifest)
- [x] PR-A2: TASK-α-8a-01 (Structured Tool Error Recovery)
- [x] PR-A3: TASK-α-4a-01 (Programmatic Tool Calling 시범)
- [ ] Phase α Exit 게이트 검증 (Section 5)
### Phase β (Week 3-5)
- [x] PR-B1: TASK-β-1-01 (SKILL.md L1/L2/L3) — *skill 추가 결정에 따라 최우선*
- [x] PR-B2: TASK-β-5-01 (PRM + Attribution)
- [x] PR-B3: TASK-β-3-01 (Router v1 룰)
- [x] PR-B4: TASK-β-6-01 (User Memory + Adaptive Few-shot)
- [x] PR-B5: TASK-β-3-02 (Router v2 학습) — *PRM 라벨 충분 시*
- [ ] Phase β Exit 게이트 검증
### Phase γ (Week 6+)
- [x] PR-C1: TASK-γ-7-01 (Trust & Governance)
- [x] PR-C2: TASK-γ-4b-01 (Tool Search Tool) — *skill/tool 증가 시점*
- [x] PR-C3: TASK-γ-8b-01 (Multi-step Self-Critic)
- [ ] Phase γ Exit 게이트 검증
### Verification Run Log
- [x] 2026-05-13T06:35Z local static gates: `npm run typecheck`, `npm run lint`, `npm test` 통과 (43 files / 168 tests).
- [x] 2026-05-13T06:02Z golden eval without LLM judge: passRate=1, ruleAverageScore=1, averageLatencyMs=1.4ms. Report: `eval-reports/2026-05-13T06-02-46-431Z.json`.
- [x] 2026-05-13T06:04Z all-new-flags smoke eval without LLM judge: passRate=1, ruleAverageScore=1, averageLatencyMs=1.8ms. Flags: abstention, programmatic tool calling, SKILL.md mode, user memory, tool search, critic iterations=2, static router. Report: `eval-reports/2026-05-13T06-04-40-217Z.json`.
- [x] 2026-05-13T06:33Z fixed OpenRouter `/api/v1` endpoint normalization to `/api/v1/chat/completions`; added `llm-client.test.ts`. `npm run typecheck`, `npm run lint`, targeted LLM client tests 통과.
- [ ] 2026-05-13T06:03Z full golden eval with LLM judge: blocked/degraded by LLM endpoint health. cliproxy timed out, OpenRouter returned HTML instead of JSON, llama timed out intermittently. Report: `eval-reports/2026-05-13T06-03-54-205Z.json` (not accepted as a quality gate).
- [ ] 2026-05-13T06:32Z full golden eval retry after URL fix: OpenRouter URL path is now correct, but gate is still blocked/degraded by provider health/model availability. cliproxy returned unknown-provider for the judge model, OpenRouter free models returned upstream 429 rate-limit, and many generation calls produced empty fallback responses. Report: `eval-reports/2026-05-13T06-32-45-317Z.json` (not accepted as a quality gate).
- [ ] Aggregator/report/drift/tournament operational gates: blocked locally because `MONGODB_CONNECTION_STRING` is not available in current eval environment.
---
## 13. 비고
- 사용자가 highlight한 `src/llm/routing.ts:24-25` (`return [llamaFallbackModel]`) — 현 코드는 `provider === "llama"`일 때 단일 fallback. TASK-β-3-01에서 `selectModelsForProvider` 시그니처에 `routingDecision`을 inject하면 llama provider도 router의 `requiredTier`에 따라 모델 선택 가능 (현재는 단일 모델이라 effective change 0, multi-llama 환경 대비 hook).
- Phase α만으로 measurable 효과: cost −10%·quality flat·latency p50 −10%.
- 모든 PR은 `evaluation-lively-island.md`의 평가 인프라(shadow eval, tournament, drift)를 활용해 검증 가능.
- Plan 원본(우선순위 분석·상세 트레이드오프)은 `~/.claude/plans/papers-goofy-plum.md`에 보관. 본 문서는 codex actionable view.