Files
lolympus-agent/agent-improvements-plan.md
T
2026-05-14 14:28:59 -07:00

42 KiB
Raw Blame History

lolympus-agent — Research-Driven Improvements (Codename: goofy-plum)

Audience: Codex(또는 다른 AI 에이전트)가 self-contained 하게 작업할 수 있도록 작성됨. 기존 plan.md (Phase 0–5)와 evaluation-lively-island.md (Phase 6–9)가 이미 production 인프라(shadow eval / judge / drift / promote / tournament)를 완성한 상태를 전제로 한다.

충돌 시 우선순위: 본 문서 > evaluation-lively-island.md > plan.md. 단 Phase 0–9의 인터페이스/스키마/env 컨벤션은 그대로 따른다.

참고 논문 (papers/ 내)

  • Paper A — arXiv 2602.12430v3 (Xu & Yan, 2026.02) Agent Skills for LLMs. SKILL.md 명세, progressive disclosure(L1/L2/L3), Tool Search Tool, Programmatic Tool Calling, Skill Trust & Lifecycle Governance(G1–G4 gates · T1–T4 tiers).
  • Paper B — arXiv 2502.09054v1 (Zellinger et al., 2025.02) Cost-Saving LLM Cascades with Early Abstention. cheap model이 cascade 초반에 abstain → cost −13%, error −5%, abstention +4.1%.
  • Paper C — arXiv 2605.06350v1 (Bouchard, 2026.05) Decision-Theoretic Characterization of LLM Cascades. k>2 cascade는 잘 튜닝된 2-model pairwise envelope 대비 우월하지 않음. Pre-generation diagnostic router가 4/5 benchmarks에서 cascade 우위.

1. Context — 왜 이 작업

문제 (현 production agent의 격차)

  1. 비용 효율의 구조적 한계 — OnboardingOrchestrator.run()은 derived.tier만 보고 tier0/1/2 LLM 호출. classify confidence가 낮아도 동일 비용. Paper B/C 기준 cheap-model generation 자체를 피할 수 있는 pre-generation 결정 지점이 부재.
  2. Tool 사용의 견고성/확장성 한계 — tool-loop.ts는 maxHops=2 고정, retry/error 분류 없음. tool 1개 실패 시 throw로 전체 흐름 정지. 신규 skill 도입 시 도구 prompt bloat 폭증 구조.
  3. Skill 추상화 부재 — PromptBuilder.buildSystemPrompt()가 persona/policy/state/examples/context를 매 요청 1회 직렬화(~1800 tokens). 신규 skill 추가가 단기 결정됨에 따라 SKILL.md progressive disclosure 도입 필수.
  4. 자동 propose의 안전망 부재 — eval/loop/propose.ts는 LLM이 생성한 prompt variant를 직접 file에 기록. injection / unintended behavior 차단 게이트 없음. Paper A: community skill 26.1%에 취약점.

의도한 결과 (3개월 horizon)

  • 평균 응답 비용 −15~20%, p50 latency −10~15%
  • LLM judge overall regression 0 (또는 +0.3 이상)
  • 신규 skill 추가 시간을 monolithic 대비 5x 단축할 수 있는 SKILL.md 인프라
  • 자동 propose 변형이 G1–G4 gate를 거쳐 prod에 안전 승급되는 폐쇄 루프

비목표

  • 코드 실행 sandbox 형태의 Programmatic Tool Calling (DAG 형태로 한정)
  • 외부 embedding 서비스 의존 (로컬 Qwen-embed 만 사용; 미정인 동안 v3 router 보류)
  • 단일 회사 외부 trust tier T4(vendor-certified) 자동화 — 인간 검토 유지

2. 사용자 결정 (반영됨)

항목 결정
Phase α 우선 가치 균형 — Early Abstention + Tool Error Recovery + Programmatic 시범
Abstention 정책 Hybrid (대화 위치 기반) — 초반 turn은 짧은 확인 질문, 반복/혼란 시그널은 [NO_RESPONSE]
Skill 확장 단기 내 추가 예정 → SKILL.md(#1) Phase β로, Governance(#7) Phase γ 앞부분으로

3. Relationship to Existing Plans (변경 영향)

그대로 사용 (변경 없음)

  • src/eval/{capture,anonymize,runner,types,shadow-bus,shadow-worker,aggregator,archive,drift,promote}.ts
  • src/eval/judges/{rule-judge,llm-judge}.ts
  • src/eval/loop/{tournament,elo}.ts
  • src/eval/review/{queue,label-archive,calibration}.ts
  • src/eval/golden/*.jsonl (스키마 호환 유지)
  • Redis/Garnet 키, NATS subjects, env 컨벤션
  • 익명화(anonymize.ts) — 신규 user-memory 쓰기에 강제 적용

확장 / 변경

  • src/skills/onboarding/runtime/{orchestrator,tool-loop,critic,trace}.ts — 행동 분기 추가, 시그니처 일부 확장
  • src/skills/onboarding/state/{derive,onboarding-state,blackboard}.ts — routingDecision, abstentionScore, long-term key 추가
  • src/skills/onboarding/prompts/{builder,layers,examples}.ts — SKILL.md mode 분기, retriever 호출
  • src/llm/routing.ts — selectModelsForProvider · selectProviderOrder 시그니처에 routingDecision 옵션 추가
  • src/eval/judges/process-judge.ts — PRM forward score 통합
  • src/eval/loop/propose.ts — T1 자동 진입 + gate 호출
  • src/config.ts — 신규 env 항목 (Section 4 참조)

신규 파일

경로 역할 Phase
src/skills/onboarding/runtime/abstention.ts Hybrid early abstention 결정 α
src/skills/onboarding/runtime/tool-plan-runner.ts DAG 기반 programmatic tool calling α
src/skills/onboarding/runtime/router.ts Pre-gen diagnostic router β
src/skills/onboarding/runtime/runtime-flag-manifest.ts tournament 가능한 runtime flag variant α (시작) / β (확장)
src/skills/onboarding/SKILL.md L1 metadata + L2 body β
src/skills/onboarding/resources/*.md L3 resources (nickname-format, heart-rule, discord-format, age-policy) β
src/skills/skill-loader.ts SkillRegistry — 모든 SKILL.md 인덱스 β
src/skills/onboarding/state/user-memory.ts UserMemoryStore (long-term) β
src/skills/onboarding/prompts/example-store.ts ExampleRetriever β
src/eval/judges/prm.ts Process Reward Model (v1 Ridge) β
src/eval/judges/attribution.ts outcome → trace step root cause β
src/eval/governance/gates.ts G1–G4 gates γ
src/eval/governance/trust.ts T1–T4 promotion/demotion γ
src/tools/tool-search.ts search_tools meta tool γ

신규 의존

  • 없음. 모든 신규 기능은 기존 의존(Garnet, MongoDB driver, Zod, LlmClient, Pino, OpenTelemetry)로 구현 가능.
  • (v3 router·example-store v2 시점에) 로컬 Qwen-embed가 필요 — 그 시점 별도 결정 (현 단계 보류).

4. 신규 환경 변수 / 설정

src/config.ts에 추가. 모든 신기능은 default OFF로 시작 → shadow eval 후 ON.

# Phase α
AGENT_ABSTENTION_ENABLED=false
AGENT_ABSTENTION_THRESHOLD=0.35         # ξ — [NO_RESPONSE] 임계
AGENT_ABSTENTION_CLARIFY_THRESHOLD=0.45 # φ_clarify — clarify 질문 임계
AGENT_TOOL_RETRY_MAX=1
AGENT_TOOL_LOOP_MAX_HOPS=3              # 기존 2 → 3 (precondition retry 여유)
AGENT_PROGRAMMATIC_TOOL_CALLING=false   # variant manifest 노출도 가능

# Phase β
AGENT_SKILL_MD_MODE=false               # SKILL.md L1/L2/L3 disclosure
AGENT_ROUTER_MODE=static                # static | learned-v1 | learned-v2
AGENT_ROUTER_WEIGHTS_PATH=
AGENT_USER_MEMORY_ENABLED=false
AGENT_USER_MEMORY_TTL_DAYS=30
EVAL_PRM_ENABLED=false
EVAL_PRM_WEIGHTS_PATH=

# Phase γ
AGENT_TRUST_MIN_TIER=T2                 # 미달 variant는 control fallback
AGENT_CRITIC_MAX_ITERATIONS=2
AGENT_TOOL_SEARCH_ENABLED=false

5. Phase α (Week 1-2) — 즉시 효과·낮은 위험

TASK-α-2-01 · Hybrid Early Abstention

파일: src/skills/onboarding/runtime/abstention.ts (신규), src/skills/onboarding/runtime/orchestrator.ts, src/skills/onboarding/state/onboarding-state.ts, src/config.ts, src/skills/onboarding/onboarding-templates.ts

시그니처:

// src/skills/onboarding/runtime/abstention.ts
export type AbstentionAction =
  | { kind: "proceed" }
  | { kind: "clarify"; templateKey: "abstain-clarify-default" | string }
  | { kind: "no-response" };

export interface AbstentionDecision {
  action: AbstentionAction;
  reason:
    | "low-confidence-early-turn"
    | "low-confidence-repeated"
    | "off-topic-strong"
    | "flagged-confused"
    | null;
  score: number; // 0..1
  features: {
    classifyConfidence: number;
    intentKind: Intent["kind"];
    turnIndex: number;
    sameIntentStreak: number;
    flaggedIssues: number;
  };
}

export function decideEarlyAbstention(input: {
  classified: ClassifiedIntent;
  derived: DerivedState;
  blackboard: Blackboard | null;
  request: AgentRequest;
  thresholds: {
    abstain: number;     // AGENT_ABSTENTION_THRESHOLD
    clarify: number;     // AGENT_ABSTENTION_CLARIFY_THRESHOLD
  };
}): AbstentionDecision;

룰 (Hybrid):

  1. intent.kind === "off-topic-to-other" && confidence > 0.9 → no-response, reason=off-topic-strong.
  2. flaggedIssues.length > 0 || sameIntentStreak >= 3 (blackboard.lastIntent 비교) → no-response if confidence < abstain, reason=flagged-confused or low-confidence-repeated.
  3. turnIndex <= 1 (greetedAt 미존재) && confidence < clarify && intent ∈ {unknown, chitchat} → clarify (templateKey=abstain-clarify-default = "조금 더 구체적으로 알려주시면 도와드릴게요!"), reason=low-confidence-early-turn.
  4. 그 외 → proceed.

orchestrator 통합 (runtime/orchestrator.ts:run() 진입부 직후):

if (getAbstentionEnabled()) {
  const decision = decideEarlyAbstention({
    classified, derived, blackboard, request,
    thresholds: { abstain: getAbstentionThreshold(), clarify: getAbstentionClarifyThreshold() },
  });
  derived.abstentionScore = decision.score;
  traceCollector?.startStep("classify").finish({
    confidence: classified.confidence,
    intent: classified.intent.kind,
    source: classified.source,
    abstention: decision,            // process-trace에 기록
  });
  if (decision.action.kind === "no-response") {
    return { text: "[NO_RESPONSE]", validation: { status: "pass", reasons: ["early-abstention"] } };
  }
  if (decision.action.kind === "clarify") {
    return { text: getOnboardingTemplate(decision.action.templateKey), validation: { status: "pass", reasons: ["early-abstention-clarify"] } };
  }
}

템플릿 추가 (onboarding-templates.ts):

"abstain-clarify-default": "조금 더 구체적으로 알려주시면 도와드릴게요!",

Trace 확장 (runtime/trace.ts):

  • ProcessTraceStep<classify>.output에 abstention?: AbstentionDecision 필드 추가 (옵셔널, 기존 호환).

Eval 연동:

  • src/eval/aggregator.ts: slice에 abstentionReason, abstentionAction 차원 추가.
  • src/eval/drift.ts: abstention rate (action≠proceed 비율)를 metric으로 등록.
  • 통과 기준: 7일 shadow에서 cost −10% 이상, false-abstention < 2%, quality regression < 0.05.

Acceptance:

  • pnpm test src/skills/onboarding/runtime/abstention.test.ts (신규 unit test 5+ case: off-topic / early-low-conf / repeated / proceed / boundary)
  • env OFF에서 기존 behavior 100% 일치 (snapshot 비교)
  • env ON 24h shadow에서 abstention rate 0.05~0.25 사이 (밴드 밖이면 임계 재튜닝)

TASK-α-8a-01 · Structured Tool Error Recovery

파일: src/skills/onboarding/runtime/tool-loop.ts, src/tools/tool.ts, src/skills/onboarding/runtime/trace.ts, src/config.ts

타입 확장 (src/tools/tool.ts):

export type ToolErrorKind = "args-invalid" | "transient" | "precondition-failed" | "permanent";
export interface ToolExecutionFailure {
  kind: ToolErrorKind;
  toolName: string;
  message: string;
  retryable: boolean;
  details?: unknown;
}

tool-loop.ts 변경 요지 (현 :113-137 영역):

  • zod parse 실패 → args-invalid → 1 hop 자기교정 (assistant tool_call → user "이전 args가 schema 어긋남" 안내). 이때 hop budget을 consume 하되 별도 카운터 selfCorrectionHops 추적해 무한 루프 차단(max 1).
  • fetch/timeout 등 network error 감지 (AbortError/FetchError/timeout) → transient → 200ms backoff → 1회 retry (AGENT_TOOL_RETRY_MAX). hop budget 사용 안 함.
  • tool 내부 throw → permanent → { ok: false, errorKind: "permanent", message } 를 tool 결과로 message에 push, hop 계속.
  • precondition 검증 실패 (예: session 없음) → tool이 result.ok=false로 정상 반환하도록 builtins 수정 → tool-loop에서 precondition-failed로 분류, hop budget consume 안 함.

maxHops: 기본 getAgentToolLoopMaxHops() (default 3). precondition-failed로 인한 추가 hop은 +1 까지만 허용.

Trace 확장:

toolCalls?: Array<ProcessTraceStep<{
  name: string;
  args: unknown;
  ok: boolean;
  resultSummary: string;
  errorKind?: ToolErrorKind;   // 신규
  retried?: boolean;            // 신규
}>>;

재사용:

  • parseToolCalls (src/tools/parser.ts) — 변경 없음
  • 기존 traceStep?.finish({ ok: false, ... }) 경로

Acceptance:

  • src/skills/onboarding/runtime/tool-loop.test.ts (신규): mock tool throw / network timeout / zod-invalid / precondition-failed 각 케이스
  • tool-heavy slice(nickname-rejected, discord-verify-question) shadow eval에서 success rate +3% 이상
  • p95 latency 변화 < +200ms

TASK-α-4a-01 · Programmatic Tool Calling 시범 (선언적 DAG)

파일: src/skills/onboarding/runtime/tool-plan-runner.ts (신규), src/skills/onboarding/runtime/planner-schema.ts, src/skills/onboarding/runtime/orchestrator.ts

스키마 확장 (planner-schema.ts):

export const toolPlanStepSchema = z.object({
  id: z.string().min(1),
  tool: z.string().min(1),
  argsFromContext: z.record(z.string(), z.string()).optional(),  // "$request.nickname"
  argsLiteral: z.record(z.string(), z.unknown()).optional(),
  dependsOn: z.array(z.string()).optional(),
  on: z.object({ error: z.enum(["abort", "fallback-llm"]).optional() }).optional(),
});
export const toolPlanSchema = z.object({
  steps: z.array(toolPlanStepSchema).max(4),    // 안전 한계
  composeWith: z.enum(["all-then-llm", "first-then-llm"]),
});
export const plannerOutputSchema = z.object({
  approach: z.string(),
  mustInclude: z.array(z.string()).optional(),
  mustAvoid: z.array(z.string()).optional(),
  suggestedTools: z.array(z.string()).optional(),
  targetLengthChars: z.number().optional(),
  toolPlan: toolPlanSchema.optional(),          // NEW
});

runner (tool-plan-runner.ts):

export interface ToolPlanResult {
  results: Record<string /* step id */, unknown>;
  failures: ToolExecutionFailure[];
  durationMs: number;
}
export async function runToolPlan(
  plan: ToolPlan,
  tools: readonly Tool<unknown, unknown>[],
  ctx: ToolContext,
  request: AgentRequest,
): Promise<ToolPlanResult>;
  • DAG 정렬 (dependsOn) 후 Promise.all로 동일 레벨 동시 실행 (단 동일 NATS RPC 큐 부담 고려해 Promise.all 최대 2 병렬).
  • $request.nickname, $request.userId, $session.openQuestions[0] 등 path 표현식 resolve.
  • 실패 시 on.error === "fallback-llm"이면 tool-loop으로 fallback (free-form), abort이면 즉시 종료.

orchestrator 통합 (run() planner 출력 처리 부분):

  • planner output에 toolPlan 있고 getProgrammaticToolCalling() ON이면 runToolPlan 먼저 → 결과를 message[]에 tool messages로 push → responder 1회 호출.
  • 없으면 기존 runWithTools 경로.

variant 노출:

  • src/eval/proposals/planner-with-toolplan.json 신규 — planner system layer에 "가능하면 toolPlan 필드를 채워주세요" append.
  • Tournament로 control vs toolplan 비교. 통과 기준 ELO +20, latency p50 −150ms 이상.

Acceptance:

  • src/skills/onboarding/runtime/tool-plan-runner.test.ts (신규): 정상 DAG / 의존성 누락 / step 실패 fallback 케이스
  • process-judge 확장: toolPlan.steps.length vs actualToolCalls.length consistency check (src/eval/judges/process-judge.ts)

TASK-α-FLAG-01 · RuntimeFlagManifest 인프라

파일: src/skills/onboarding/runtime/runtime-flag-manifest.ts (신규), src/skills/onboarding/runtime/orchestrator.ts, src/eval/loop/tournament.ts

타입:

export interface RuntimeFlagManifest {
  variantId: string;       // e.g. "abstention-on", "router-v1"
  description: string;
  flags: Partial<{
    abstentionEnabled: boolean;
    abstentionThreshold: number;
    abstentionClarifyThreshold: number;
    programmaticToolCalling: boolean;
    toolLoopMaxHops: number;
    routerMode: "static" | "learned-v1" | "learned-v2";
    userMemoryEnabled: boolean;
    criticMaxIterations: number;
  }>;
}
export function loadRuntimeFlagManifest(variantId: string): RuntimeFlagManifest | null;
export function resolveRuntimeFlags(
  base: ResolvedRuntimeFlags,
  variant?: RuntimeFlagManifest,
): ResolvedRuntimeFlags;

효과: PromptVariantManifest(prompt만)와 별도로 행동 변화 variant를 tournament로 비교 가능. tournament.ts에서 loadVariantBundle(variantId)가 prompt+runtime 둘 다 로드하도록 확장.

Acceptance:

  • runtime-flag-manifest.test.ts: variant 로드 / merge / 잘못된 키 거부
  • tournament가 동일 골든셋에 대해 control / abstention-on / toolplan-on 3종 ELO 산출 가능

Phase α Exit 게이트

전부 만족 시 Phase β 진입.

메트릭 측정 위치 목표
Cost (shadow 7일 평균) eval/aggregator.ts slice별 latencySumMs * tier price −10% vs control
LLM judge overall aggregator.llmOverallAvg ≥ baseline − 0.05
Abstention false-positive review/queue.ts 라벨 vs decision < 2%
Tool failure → soft-template validator.ts reject 사례 비율 −30%
Regression alerts drift.ts significant count 0

6. Phase β (Week 3-5) — Skill 확장 대비 + 평가 정밀화

TASK-β-1-01 · SKILL.md L1/L2/L3 Progressive Disclosure

핵심 가정: 단기 내 추가 skill(discord-help / community-rules / identity-room 중 1+) 도입 결정. SKILL.md 인프라가 없으면 prompt 폭증.

신규 파일:

  • src/skills/onboarding/SKILL.md — frontmatter + body
  • src/skills/onboarding/resources/{nickname-format,heart-rule,discord-format,age-policy}.md
  • src/skills/skill-loader.ts — SkillRegistry

SKILL.md frontmatter 스키마:

---
name: onboarding
description: 카카오톡 오픈채팅 새내기 등록을 도와주는 친절한 선배 역할.
version: 1
intents: [greeting, acknowledgement, status-question, help-how, help-where, nickname-question, photo-question, discord-question, discord-verify-question, age-question]
triggers: [user-message, new-member, nickname-invalid, photo-failed, progress-update, timeout-warning, completed]
tools: [validate-nickname, check-onboarding-progress, lookup-community-rule, get-room-context, verify-discord-nickname]
resources:
  nickname-format: ./resources/nickname-format.md
  heart-rule: ./resources/heart-rule.md
  discord-format: ./resources/discord-format.md
  age-policy: ./resources/age-policy.md
trustTier: T3
---

SkillRegistry:

// src/skills/skill-loader.ts
export interface SkillManifest {
  name: string;
  description: string;
  version: number;
  intents: string[];
  triggers: string[];
  tools: string[];
  resources: Record<string, string>; // key → relative path
  trustTier: TrustTier;
  bodyPath: string;  // SKILL.md absolute path
}
export class SkillRegistry {
  static load(rootDir: string): SkillRegistry;          // 모든 SKILL.md frontmatter 메모리 로드
  list(): SkillManifest[];
  get(name: string): SkillManifest | null;
  loadBody(name: string): Promise<string>;              // L2 lazy load
  loadResource(name: string, key: string): Promise<string>; // L3 lazy load
}

PromptBuilder 재설계 (prompts/builder.ts):

  • env AGENT_SKILL_MD_MODE=true일 때:
    1. system 1차 메시지: 모든 skill manifest의 description 만 ~30 tokens/skill 직렬화 (L1).
    2. 매칭된 skill의 body(loadBody)를 system 2차 메시지로 (L2).
    3. planner output에 requestResources: ["nickname-format"] 있으면 responder 호출 직전에 해당 resource MD를 system 3차 메시지로 inject (L3).
  • 기존 layer 함수(personaLayer, policyLayer, ...)는 그대로 두되 policyLayer만 intent-conditional layer로 분리되어 L2 단계에서 inject. (점진 마이그레이션)

planner-schema 확장:

requestResources: z.array(z.string()).optional(),

마이그레이션:

  • Step 1: SKILL.md를 mirror로 생성. 기존 prompt 결과와 token 차이 측정.
  • Step 2: feature flag ON 후 24h shadow eval. quality regression < 0.05 확인.
  • Step 3: L3 resource를 nickname-related intent부터 적용 (가장 적은 intent → 점진).

Acceptance:

  • SkillRegistry unit test: 잘못된 frontmatter / 빈 resources / 중복 name
  • snapshot test: system prompt token count baseline ↔ SKILL.md mode (≤ −40%)
  • 30일 long-soak shadow: regression < 0.05
  • dry-run: 새 skill 추가가 SKILL.md 1개 + registry 1줄로 동작

TASK-β-5-01 · Process Reward Model (PRM) + Bidirectional Attribution

신규 파일:

  • src/eval/judges/prm.ts — feature extractor + Ridge regression (사용 라이브러리: 직접 구현, dependencies 최소)
  • src/eval/judges/attribution.ts — outcome → trace step root cause

Feature vector (Section 5 #5 PRM 참조):

export interface PrmFeatureVector {
  classifyConfidence: number; classifySource: 0|1;
  planApproach: string;           // categorical hash
  toolCallCount: number; toolErrorCount: number;
  criticDecision: 0|1; validateStatus: 0|1|2; tier: 0|1|2;
  abstentionScore: number;        // Phase α에서 추가됨
}

Forward model (v1): Ridge regression on label-archive humanScore vs features. weights는 EVAL_PRM_WEIGHTS_PATH(JSON)에 저장.

Backward attribution:

  • 각 entry에 대해 trace step별 counterfactual delta (step output 평균값으로 swap 시 PRM score 변화) → attributedScoreLoss.
  • 결과 collection: eval_process_attribution (MongoDB).

process-judge 통합 (eval/judges/process-judge.ts):

// 기존 ProcessJudgeResult에 추가
prmScore?: number;            // PRM forward
prmFeatures?: PrmFeatureVector;
rootCauseStep?: ProcessTraceStepName;
rootCauseScoreLoss?: number;

Background job 추가 (workers/background.ts):

  • 새 tick: EVAL_ATTRIBUTION_TICK_MS (default 6h). 최근 archive entries에 대해 attribution 계산 후 collection 적재.
  • 라벨 수가 N(env EVAL_PRM_MIN_LABELS, default 500) 미만이면 forward 모델은 "explanation only" 모드(score 출력 X, root cause만).

propose 입력 확장 (eval/loop/propose.ts):

  • buildProposalInputFromArchive에 rootCauses: RootCauseReport[] 필드 추가. LLM이 root cause 분포를 보고 더 타겟팅된 prompt diff 생성.

Acceptance:

  • prm.test.ts: feature vector 안정성, weight 로드/저장
  • attribution.test.ts: synthetic trace에서 root cause step 식별 정확도 ≥ 0.8
  • tournament: PRM-augmented variant가 ELO +30 차이를 식별 가능 (synthetic label)

TASK-β-3-01 · Pre-Generation Diagnostic Router (v1 룰)

신규 파일: src/skills/onboarding/runtime/router.ts

Decision space:

export type RoutingDecision =
  | { kind: "abstain"; reason: string }              // = TASK-α-2 abstention
  | { kind: "template-only"; templateKey: string }
  | { kind: "tier0-only" }
  | { kind: "tier1-direct"; skipPlanner: boolean }
  | { kind: "tier1-with-tools" }
  | { kind: "tier2-full" };

export interface DiagnosticRouter {
  decide(input: RouterInput): RoutingDecision;
}

v1 rules (router.ts 초안):

  • intent.kind === "off-topic-to-other" && conf > 0.9 → abstain
  • intent.kind === "greeting" && conf > 0.95 && no prior greeting → template-only
  • intent.kind === "acknowledgement" && conf > 0.9 → template-only
  • phase.kind === "nickname-rejected" → tier1-with-tools
  • phase.kind === "completed" && intent ≠ status-question → template-only
  • phase.kind ∈ {fresh, nickname-pending, photo-pending, both-pending} && intent ∈ {nickname-question, photo-question, help-how} → tier1-with-tools
  • 그 외 → tier0-only

derive.ts 통합: deriveState() 끝에서 router.decide(...) 호출, 결과를 DerivedState.routingDecision에 저장.

llm/routing.ts 확장:

  • selectModelsForProvider(provider, options, llamaFallbackModel) 시그니처에 routingDecision?: RoutingDecision 추가.
  • llama 분기(routing.ts:24-25)는 routingDecision.kind가 tier1-with-tools 이상이면 fallback 모델 대신 explicit fallback 사용; 현 단일 모델 환경에서는 effective change 0 (multi-llama 환경 대비 hook 추가).
  • selectProviderOrder(endpoints, options)도 routingDecision을 받아 cliproxy/openrouter/llama 우선순위를 동적 조정 (예: template-only decision이면 llama only).

Acceptance:

  • router.test.ts: 30+ 케이스로 v1 룰 커버
  • Shadow A/B 24h: cost −8%, quality flat (LLM judge overall delta < 0.05)
  • routingDecision을 ShadowEvalEnvelope에 기록 (shadow-bus.ts) — aggregator slice 추가

TASK-β-3-02 · Router v2 학습

의존: TASK-β-5-01 (PRM features). 데이터 ≥ 5000 archive entries.

파일: src/skills/onboarding/runtime/router.ts 확장, scripts/train-router.ts (신규 script)

모델: lightweight logistic regression (multi-class: 6 decision kinds). 입력은 PRM extractFeatures + 사용자 메시지 길이/언어 식별.

학습 데이터 라벨링:

  • archive에서 actual derived.tier · validation status · llmJudge.overall 기반 ground truth tier 추정:
    • llmJudge.overall ≥ 4 + tier=0 → tier0-only 충분
    • llmJudge.overall < 3 + tier=0 → tier1-direct 필요했음 (under-routed)
    • llmJudge.overall < 3 + tier=1 → tier2-full 필요했음
  • 5% exploration noise (epsilon=0.05) 도입 — 무작위로 1 tier ↑

Acceptance:

  • train-router.ts 가 weights JSON 생성 (AGENT_ROUTER_WEIGHTS_PATH)
  • Tournament: router-v1 vs router-v2 비교 — v2 cost −5% 추가 절감

TASK-β-6-01 · Persistent User Memory + Adaptive Few-shot

신규 파일:

  • src/skills/onboarding/state/user-memory.ts
  • src/skills/onboarding/prompts/example-store.ts

Garnet 키: lolympus-agent:user:profile:{anonHash(userId)} (TTL AGENT_USER_MEMORY_TTL_DAYS × 86400).

UserMemoryStore:

export interface UserProfile {
  userId: string;             // anonHash
  firstSeenAt: string; lastSeenAt: string;
  totalTurns: number;
  preferredStyle: "concise" | "warm" | "neutral";
  pastIntents: { kind: Intent["kind"]; count: number }[];
  flaggedIssues: string[];
  completedOnboardingAt?: string;
}
export class UserMemoryStore {
  constructor(private garnet: GarnetClient, private salt: string) {}
  async load(userId: string): Promise<UserProfile | null>;
  async observeTurn(userId: string, derived: DerivedState, response: AgentResponse): Promise<UserProfile>;
}

PII 강제:

  • 모든 string write에 anonymizeText (src/eval/anonymize.ts) 적용. unit test로 nickname/이메일/번호 미저장 검증.
  • userId 자체는 anonymizeIdentifier("user", userId) 해시 사용.

ExampleRetriever (prompts/example-store.ts):

  • v1: intent + phase로 필터링한 golden examples 중 user.preferredStyle과 매칭되는 최대 3개 반환.
  • v2 (embedding): 보류 — 로컬 Qwen-embed 도입 결정 후.

examplesLayer 시그니처 확장 (prompts/examples.ts):

export function examplesLayer(intent: Intent, retriever: ExampleRetriever, query: ExampleQuery): string;

variant 노출: runtime-flag-manifest userMemoryEnabled: true.

Acceptance:

  • user-memory.test.ts: PII redaction 강제, observeTurn idempotent
  • 같은 user(anon) 재방문 시 LLM judge friendliness 평균 +0.2 (shadow eval)

Phase β Exit 게이트

메트릭 목표
system prompt token baseline 대비 −40% (SKILL.md mode ON)
누적 cost (β 종료) α 대비 추가 −8%
PRM attribution AUC ≥ 0.75 (라벨 1000+)
신규 skill dry-run SKILL.md 1개 + registry 1줄로 동작 확인

7. Phase γ (Week 6+) — Governance + Reflection

TASK-γ-7-01 · Skill Trust & Lifecycle Governance (G1–G4 · T1–T4)

신규 파일: src/eval/governance/gates.ts, src/eval/governance/trust.ts

Gates 시그니처:

export interface GateResult { passed: boolean; reason?: string; evidence?: unknown; }

export function runG1Static(manifest: PromptVariantManifest | RuntimeFlagManifest): GateResult;
// URL allowlist (validator.ts:8-11 재사용), layer 길이 제한, persona override 차단, 위험 키워드

export async function runG2Semantic(manifest, control, llmClient: LlmClient): Promise<GateResult>;
// LLM이 변형의 핵심 행동 변화 의도 분류 — "변경이 의도된 범위 내인가?"

export async function runG3Sandbox(manifest, evalRunner, goldenSetPath, n: number = 200): Promise<GateResult>;
// shadow run on N=200 cases, validator reject rate / llm overall 변화 측정

export function runG4Permission(manifest, toolRegistry: ToolRegistry, skillRegistry: SkillRegistry): GateResult;
// required tools/resources가 registry에 등록되어 있는지

Trust Tiers (trust.ts):

export type TrustTier = "T1-unvetted" | "T2-community" | "T3-org-vetted" | "T4-certified";

// Promotion 규칙
// T1 → T2: G1 + G4 통과 (자동)
// T2 → T3: G3 sandbox + tournament 100+ matches + ELO > control + 30 + drift OK 7일 (자동)
// T3 → T4: human review label-archive 평균 ≥ 4.0 + reviewer 2명 이상 동의 (semi-auto)
//
// Demotion: drift significant 또는 G3 sandbox regression → 즉시 T1로 강등 + alert

export async function evaluatePromotion(variantId: string, services: GovernanceServices): Promise<TrustTier>;
export async function applyDemotion(variantId: string, reason: string): Promise<void>;

propose.ts 통합 (eval/loop/propose.ts):

  • 새 proposal은 항상 T1로 저장. writeVariantManifest 호출 직후 runG1Static + runG4Permission 실행 → 통과 시 T2로 자동 승급.

런타임 강제 (src/skills/onboarding/skill.ts):

  • getPromptVariantId() 로드 시 trust.getActiveTier(variantId) 확인. AGENT_TRUST_MIN_TIER (default T2) 미달이면 control fallback.

Background job:

  • workers/background.ts에 EVAL_PROMOTION_TICK_MS (default 30m). 모든 T1/T2 variant 대상 promotion 평가 실행.

Acceptance:

  • gates.test.ts: 의도적 prompt injection variant가 G1에서 차단
  • T1→T2 자동 승급, T2→T3 tournament 게이트 통과 시 자동 승급 시뮬레이션
  • Drift significant 발생 시 즉시 강등 통합 테스트

TASK-γ-4b-01 · Tool Search Tool

신규 파일: src/tools/tool-search.ts. src/tools/registry.ts에 searchByDescription(query, k) 추가.

export const toolSearchTool: Tool<{ query: string; topK?: number }, { tools: Array<{ name: string; description: string }> }> = {
  name: "search_tools",
  description: "도구 카탈로그에서 가장 적합한 도구 후보를 검색합니다.",
  argsSchema: z.object({ query: z.string().min(2), topK: z.number().int().min(1).max(5).default(3) }),
  resultSchema: z.object({ tools: z.array(z.object({ name: z.string(), description: z.string() })) }),
  async execute(args, ctx) { /* registry.searchByDescription */ },
};

registry 변경 (tools/registry.ts):

  • searchByDescription(query, k): 단순 keyword overlap → BM25 (lite, 의존 추가 없이 inline 구현).
  • 도구 등록 시 description 인덱싱.

orchestrator 통합 (runtime/orchestrator.ts):

  • skill manifest의 tools 배열이 N 초과(env AGENT_TOOL_SEARCH_THRESHOLD, default 10)일 때만 search_tools만 노출 + planner가 requestTools: [...] 통해 expand.

Acceptance:

  • 20 tool mock registry에서 nickname 관련 query → validate-nickname top-1 회수
  • 도구 5개 미만 환경에서는 effective change 0 (기존 path)

TASK-γ-8b-01 · Multi-step Self-Critic (Reflection)

파일: src/skills/onboarding/runtime/critic.ts, src/skills/onboarding/runtime/trace.ts

시그니처 확장:

export interface CritiqueResultV2 {
  changed: boolean;
  text: string;
  iterations: number;
  abstained: boolean;
  rubricScores: { accuracy: number; scope: number; tone: number };
  prmScoreBefore?: number;
  prmScoreAfter?: number;
}

class OnboardingCritic {
  async critiqueIterative(
    draft: string, derived: DerivedState, messages: ChatMessage[],
    options: { maxIters: number; deadlineMs: number; prm?: PrmModel },
  ): Promise<CritiqueResultV2>;
}

알고리즘:

  1. PRM forward score → prmScoreBefore. score < criticRubricFloor && tier == 2 → 반복.
  2. rubric 기반 prompt로 LLM 호출 → 점수 + 개선안.
  3. revise() 1회. 다시 PRM. prmScoreAfter - prmScoreBefore < δ면 abort.
  4. iterations <= maxIters (default 2). 결과 trace에 기록.
  5. PRM이 abstainFloor 미만이면 [NO_RESPONSE] 반환 (Paper B 통합).

Tier 적용: tier 2만. tier 0/1은 기존 1-shot critic 유지.

deadline 강제: 매 iteration 시작 시 Date.now() < deadlineMs 체크. 초과면 즉시 반환.

Acceptance:

  • tier 2 slice에서 LLM judge accuracy +0.3
  • critic 평균 iterations 2 이하, p99 latency +400ms 이하

Phase γ Exit 게이트

메트릭 목표
자동 propose-deploy 주기 weekly 1+ variant T2 자동 승급
Tier2 PRM rubric score +0.3 이상
누적 cost (γ 종료) baseline 대비 −15~20%

8. 평가 시스템 통합 (변경별)

변경 shadow-bus aggregator slice tournament drift promote
#2 Abstention decision.reason/action abstention rate — abstention drift abstention-mismatch
#8a Tool Recovery errorKind in trace tool failure rate — tool error drift tool-failure
#4a Programmatic toolPlan tool success rate plan-mode vs free tool round-trip tool-failure
#1 SKILL.md disclosureMode system token usage L1L2L3 vs flat — regression
#5 PRM trace 그대로 PRM score column PRM-aug vs raw PRM score drift rule-prm-disagree
#3 Router routingDecision tier mix router-on vs off tier-mix drift wrong-router
#6 Memory userProfileLoadedAt 재방문 user slice with-mem vs without memory hit low-retention
#7 Governance variantTrustTier tier별 quality 승급 게이트 demotion alert T1 강등
#8b Self-Critic iterations, rubricScores tier2 cost iter-1 vs iter-2 critic loop drift critic-loop-failure

9. 검증 — End-to-End

개발 중

# 1. unit + integration test
pnpm test

# 2. golden set 회귀
pnpm run eval:run --input src/eval/golden/onboarding.jsonl
# ruleJudge.score, llmJudge.overall regression < 0.05 확인

# 3. tournament (variant 등록 후)
pnpm run eval:tournament --variants control,<new-variant>
# ELO 차이 측정

# 4. drift (5분 tick)
pnpm run eval:report --slice tier=0

운영 검증 (각 Phase 종료 후 1주)

  • Aggregator dashboard: slice별 quality/cost/latency 추세
  • Drift 알람 false-positive 0 (변경 의도와 일치)
  • Promote candidates에 의미 있는 root cause 포함

Rollback

  • 모든 신규 행동은 env flag 또는 RuntimeFlagManifest 뒤에 배치 → 즉시 OFF
  • shadow eval 단계에서 production 영향 0 (counterfactual 기록만)

10. 핵심 수정 파일 — 진입점 (Codex가 첫 PR에서 손댈 파일)

Phase α 첫 PR (TASK-α-2-01 + TASK-α-FLAG-01):

  1. src/skills/onboarding/runtime/abstention.ts (신규)
  2. src/skills/onboarding/runtime/orchestrator.ts — abstention gate 진입
  3. src/skills/onboarding/state/onboarding-state.ts — DerivedState.abstentionScore? 추가
  4. src/skills/onboarding/onboarding-templates.ts — clarify 템플릿
  5. src/skills/onboarding/runtime/trace.ts — classify step에 abstention? 필드
  6. src/skills/onboarding/runtime/runtime-flag-manifest.ts (신규)
  7. src/config.ts — AGENT_ABSTENTION_* env 추가
  8. src/skills/onboarding/runtime/abstention.test.ts (신규)

Phase α 두 번째 PR (TASK-α-8a-01):

  1. src/tools/tool.ts — ToolErrorKind, ToolExecutionFailure
  2. src/skills/onboarding/runtime/tool-loop.ts — retry/error 분류
  3. src/skills/onboarding/runtime/trace.ts — toolCalls output에 errorKind? 추가
  4. src/skills/onboarding/runtime/tool-loop.test.ts (신규)

Phase α 세 번째 PR (TASK-α-4a-01):

  1. src/skills/onboarding/runtime/planner-schema.ts — toolPlan? 필드
  2. src/skills/onboarding/runtime/tool-plan-runner.ts (신규)
  3. src/skills/onboarding/runtime/orchestrator.ts — toolPlan 우선 분기
  4. src/eval/proposals/planner-with-toolplan.json (신규 variant)
  5. src/eval/judges/process-judge.ts — toolPlan consistency check
  6. src/skills/onboarding/runtime/tool-plan-runner.test.ts (신규)

Phase β 핵심:

  • src/skills/skill-loader.ts, src/skills/onboarding/SKILL.md, src/skills/onboarding/resources/*.md
  • src/skills/onboarding/prompts/builder.ts — SKILL.md mode
  • src/eval/judges/prm.ts, src/eval/judges/attribution.ts, src/eval/judges/process-judge.ts
  • src/skills/onboarding/runtime/router.ts, src/skills/onboarding/state/derive.ts, src/llm/routing.ts
  • src/skills/onboarding/state/user-memory.ts, src/skills/onboarding/prompts/example-store.ts

Phase γ:

  • src/eval/governance/{gates,trust}.ts
  • src/eval/loop/propose.ts (T1 entry)
  • src/tools/{tool-search.ts,registry.ts}
  • src/skills/onboarding/runtime/critic.ts

11. 위험 & 미해결 질문

위험 (mitigation 동반)

  1. #2 abstention false-negative → ξ=0.35 보수, in-conversation abstain 차단(turnIndex ≥ 2일 때 clarify 안 함), 24h shadow 우선.
  2. #3 router 학습 데이터 편향 (현 정책 재현) → epsilon=0.05 exploration noise, 라벨 보강.
  3. #5 PRM 초기 라벨 부족 → EVAL_PRM_MIN_LABELS=500 미만 시 explanation-only 모드(forward score 미노출).
  4. #6 PII 누출 → UserMemoryStore.update에 anonymize wrapper 강제, audit job 매일 1회.
  5. #1 SKILL.md 행동 변화 → 30일 long-soak, regression 시 AGENT_SKILL_MD_MODE=false 즉시 rollback.
  6. #7 propose 속도 저하 → T1→T2 자동(G1+G4만 요구)으로 가속 trade-off.
  7. #8b critic loop 폭주 → AGENT_CRITIC_MAX_ITERATIONS=2 강제, deadlineMs 체크.

미해결 질문 (구현 시작 전 사용자 확인 필요)

  • 현재 label-archive 누적 라벨 수? PRM v1 위해 ≥ 500. 부족하면 review queue 가속을 Phase α에 추가.
  • 재방문 user rate (chatroom 별 30일 동일 user turn 분포)? 메모리 ROI 산정.
  • 로컬 embedding infra 추가 가능? (router v3, example-store v2 위해 Qwen-embed 64d). 외부 사용 가능?
  • 다음 skill 후보 명세 — discord-help · community-rules · identity-room 중 어떤 것이 먼저? SKILL.md L3 resources 설계 영향.

12. 작업 체크리스트 (codex가 PR 단위로 진행)

Phase α (Week 1-2)

  • PR-A1: TASK-α-2-01 (Hybrid Early Abstention) + TASK-α-FLAG-01 (RuntimeFlagManifest)
  • PR-A2: TASK-α-8a-01 (Structured Tool Error Recovery)
  • PR-A3: TASK-α-4a-01 (Programmatic Tool Calling 시범)
  • Phase α Exit 게이트 검증 (Section 5)

Phase β (Week 3-5)

  • PR-B1: TASK-β-1-01 (SKILL.md L1/L2/L3) — skill 추가 결정에 따라 최우선
  • PR-B2: TASK-β-5-01 (PRM + Attribution)
  • PR-B3: TASK-β-3-01 (Router v1 룰)
  • PR-B4: TASK-β-6-01 (User Memory + Adaptive Few-shot)
  • PR-B5: TASK-β-3-02 (Router v2 학습) — PRM 라벨 충분 시
  • Phase β Exit 게이트 검증

Phase γ (Week 6+)

  • PR-C1: TASK-γ-7-01 (Trust & Governance)
  • PR-C2: TASK-γ-4b-01 (Tool Search Tool) — skill/tool 증가 시점
  • PR-C3: TASK-γ-8b-01 (Multi-step Self-Critic)
  • Phase γ Exit 게이트 검증

Verification Run Log

  • 2026-05-13T06:35Z local static gates: npm run typecheck, npm run lint, npm test 통과 (43 files / 168 tests).
  • 2026-05-13T06:02Z golden eval without LLM judge: passRate=1, ruleAverageScore=1, averageLatencyMs=1.4ms. Report: eval-reports/2026-05-13T06-02-46-431Z.json.
  • 2026-05-13T06:04Z all-new-flags smoke eval without LLM judge: passRate=1, ruleAverageScore=1, averageLatencyMs=1.8ms. Flags: abstention, programmatic tool calling, SKILL.md mode, user memory, tool search, critic iterations=2, static router. Report: eval-reports/2026-05-13T06-04-40-217Z.json.
  • 2026-05-13T06:33Z fixed OpenRouter /api/v1 endpoint normalization to /api/v1/chat/completions; added llm-client.test.ts. npm run typecheck, npm run lint, targeted LLM client tests 통과.
  • 2026-05-13T06:03Z full golden eval with LLM judge: blocked/degraded by LLM endpoint health. cliproxy timed out, OpenRouter returned HTML instead of JSON, llama timed out intermittently. Report: eval-reports/2026-05-13T06-03-54-205Z.json (not accepted as a quality gate).
  • 2026-05-13T06:32Z full golden eval retry after URL fix: OpenRouter URL path is now correct, but gate is still blocked/degraded by provider health/model availability. cliproxy returned unknown-provider for the judge model, OpenRouter free models returned upstream 429 rate-limit, and many generation calls produced empty fallback responses. Report: eval-reports/2026-05-13T06-32-45-317Z.json (not accepted as a quality gate).
  • Aggregator/report/drift/tournament operational gates: blocked locally because MONGODB_CONNECTION_STRING is not available in current eval environment.

13. 비고

  • 사용자가 highlight한 src/llm/routing.ts:24-25 (return [llamaFallbackModel]) — 현 코드는 provider === "llama"일 때 단일 fallback. TASK-β-3-01에서 selectModelsForProvider 시그니처에 routingDecision을 inject하면 llama provider도 router의 requiredTier에 따라 모델 선택 가능 (현재는 단일 모델이라 effective change 0, multi-llama 환경 대비 hook).
  • Phase α만으로 measurable 효과: cost −10%·quality flat·latency p50 −10%.
  • 모든 PR은 evaluation-lively-island.md의 평가 인프라(shadow eval, tournament, drift)를 활용해 검증 가능.
  • Plan 원본(우선순위 분석·상세 트레이드오프)은 ~/.claude/plans/papers-goofy-plum.md에 보관. 본 문서는 codex actionable view.