# Lolympus Agent — 시스템 아키텍처 및 운영 흐름 > Iter1~12 (2026-05-13 ~ 2026-05-19) 완료 시점 기준의 통합 architecture + operations 문서입니다. > 신규 기여자/운영자/AI agent의 단일 진입점. --- ## 0. 한눈에 보기 **Lolympus Agent**는 카카오톡 오픈채팅 운영을 자동화하는 **NATS 메시지 드리븐 멀티 에이전트 시스템**이다. - **도메인**: 카카오톡 오픈채팅 가입 안내(onboarding) + 커뮤니티 규칙 모더레이션(moderation) - **외부 진입**: NATS subject `lolympus-agent.request` - **응답 형태**: NATS reply (단일 publish/요청) - **핵심 자산**: - Skill SSOT 디렉토리 (`src/skills/{onboarding,moderation,supervisor}/`) - 통합 dispatch loop (`src/agent/dispatch.ts`, production + eval 공용) - 3-layer memory + Agentic RAG (`src/memory/`) - Bounded Self-Escalation Graph (handoff + transition matrix + budget + supervisor fallback) - Eval 파이프라인 (capture → run → propose → tournament → promote) - 자체 metrics + Phoenix LLM tracing - Slack webhook + operations CLI ### 권장 시작 순서 1. **§2 아키텍처 다이어그램** — 전체 그림 2. **§3 메시지 흐름** — 한 요청이 어떻게 흐르는지 3. **§4~9** — 각 컴포넌트 깊이 있게 4. **§16 cheat sheet** — 자주 쓰는 명령 --- ## 1. 기술 스택 요약 | 영역 | 선택 | 위치 | |---|---|---| | 언어 | TypeScript (Node 22+) | `tsconfig.json` | | 메시지 | NATS | `src/common/nats.ts` | | 캐시/세션 | Garnet (Redis 호환) | `src/common/garnet.ts` | | 영속 저장 | MongoDB (mongoose) | `src/room-context/`, `src/memory/mongo-*.ts` | | LLM provider | OpenRouter / OpenAI / cliproxy / llama (multi-provider) | `src/llm/llm-client.ts`, `src/llm/routing.ts` | | Embedding | OpenRouter free 우선 → OpenAI fallback | `src/memory/embedding-client.ts` | | Vector store | MongoDB collection + in-app cosine (Vector Search 마이그레이션 path 예약) | `src/memory/mongo-long-term-memory.ts`, `src/memory/cosine.ts` | | BM25 | in-app | `src/memory/bm25.ts` | | 관측 | OTel SDK + Phoenix OTLP + 자체 metrics 모듈 | `src/observability/` | | 알림 | NATS publish + Slack incoming webhook | `src/observability/alert.ts`, `slack-notifier.ts` | | Secret | Infisical → IMDS → env | `src/common/secret-manager.ts` | | Test | Vitest | `*.test.ts` | | Plan/Eval 데이터 | MongoDB collections + Garnet streams | 각 모듈 | 새 의존성 도입 시 §15 환경변수 + KNOWN_SECRET_KEYS 갱신 확인. --- ## 2. 아키텍처 다이어그램 ``` [NATS subject "lolympus-agent.request"] │ ▼ ┌─────────────────────────────────┐ │ Agent.handleRequest │ src/agent/agent.ts (~250 lines) │ - runWithContext (AsyncLocal)│ │ - withSpan (OTel) │ │ - budget 생성, NATS reply │ └────────────┬────────────────────┘ │ dispatch(input, deps) ▼ ┌─────────────────────────────────┐ │ dispatch loop │ src/agent/dispatch.ts (~580 lines) │ ├─ selectSkillForRequest │ priority + canHandle 필터 │ ├─ runSkillOnce │ skill.handle() 호출 │ ├─ normalizeControlTokens │ [HANDOFF:...] 파싱 │ ├─ applyRuntimeVerifier │ retry/escalate │ ├─ applyEscalationTrigger │ confidence/risk → handoff │ ├─ prepareHandoff │ TransitionMatrix + Budget 검증 │ └─ recurse (max depth 4) │ 같은 dispatch 재진입 └────────────┬────────────────────┘ │ ┌───────┴───────┐ ▼ ▼ [Onboarding] [Moderation] [Supervisor] ├ intents ├ intents ├ capability-prompt (LLM tier0) ├ prompt-data ├ prompt-data ├ routing-rules (regex fast-path) ├ tools (5) ├ tools (1) └ review-queue append + alert ├ runtime/ ├ skill.ts └ skill.ts ┌─────────────────────────────────┐ │ MemoryService (Iter10-11) │ src/memory/ │ ├─ WorkingMemory (조립) │ 매 turn │ ├─ SessionMemory (12h Garnet)│ blackboard + conv + summary │ └─ LongTermMemory (30d+) │ │ ├─ UserProfile (Garnet) │ │ ├─ Knowledge (Mongo+RAG) │ embedding + cosine + BM25 + reranker │ └─ EscalationHistory (Mongo)│ user-level review queue rollup └─────────────────────────────────┘ ┌─────────────────────────────────┐ │ ToolRegistry │ src/tools/ │ ├─ Tool.requires (sandbox) │ │ ├─ scoped-context │ least-privileged ctx │ └─ audit stream │ lolympus-agent:tool-calls └─────────────────────────────────┘ ┌─────────────────────────────────┐ │ Observability │ src/observability/ │ ├─ Pino logger (skill child) │ │ ├─ OTel tracing → Phoenix │ gen_ai.* + memory.* + handoff.* │ ├─ Self-built metrics │ /metrics endpoint (9464) │ └─ AlertService │ NATS + Slack webhook └─────────────────────────────────┘ ┌─────────────────────────────────┐ │ Eval Pipeline │ src/eval/ │ ├─ Capture (shadow-bus) │ prod traffic 샘플링 │ ├─ Run (runner) │ golden cases 실행 │ ├─ Judges (rule/llm/process/ │ │ │ handoff/memory) │ │ ├─ Propose (variant 생성) │ │ ├─ Tournament (A/B) │ │ └─ Promote (T4→T2) │ └─────────────────────────────────┘ ``` --- ## 3. 메시지 수신 → 응답 흐름 ### 3.1 진입점 — `Agent.handleRequest` `src/agent/agent.ts`. 흐름: 1. **NATS raw 수신**: `natsClient.subscribeRaw(getRequestSubject(), this.handleRequest)` (start time) 2. **JSON 파싱 + 검증**: `isValidRequest(parsed)` 통과 안 하면 즉시 empty response 3. **`runWithContext`** 호출 — `src/observability/context.ts`. AsyncLocalStorage로 `RequestContext` 주입: - `requestId`, `chatroomId`, `userId`, `skill`, `trigger` - `logger` (skill name child binding) - `startedAt`, `deadlineAt` - `budget`, `handoffDepth`, `handoffChain` (Iter4) 4. **`withSpan('agent.handle')`** OTel 외부 span 시작 5. **`dispatch(input, deps)`** 호출 (실제 비즈니스 로직은 dispatch.ts에 위임 — Iter6 추출) 6. **단일 NATS reply**: `natsClient.respond(msg, response)` (line 144 등) — dispatch 종료 후 정확히 1회 ### 3.2 dispatch loop — `src/agent/dispatch.ts` 핵심 재귀 함수 `dispatch(input, deps)`: ``` 인자: input: { budget, chain, depth, history, request, skill } deps: { selectSkillForRequest, skills, transitionMatrix, shouldApplyRuntimeVerifier, getSkillJudgeConfig, onHandoff?, onDispatchEnd?, evalPlan? } 순서: 1) runSkillOnce(input) → response # skill.handle() 2) normalizeControlTokens(response) # [HANDOFF:..] → metadata 3) applyRuntimeVerifier(input, response) # pass/retry/escalate (Iter6) 4) applyEscalationTrigger(...) # confidence/risk → handoff 5) withHandoffMetadata(response, chain, depth) # handoffChain 노출 (Iter5) 6) readHandoffRequest(response) # metadata.handoffRequest └─ null → 최종 응답 반환 7) prepareHandoff(...) # next skill + packet 합성 ├─ resolveHandoffTarget (TransitionMatrix 검증) ├─ budget.assertBeforeHandoff (max-hops/loop/cost) ├─ budget.recordHandoff(target) └─ buildHandoffPacket (PII 차단, evidence_refs) 8) dispatch(next) 재귀 ``` ### 3.3 dispatch 종료 후 - `metadata.handoffChain: string[]` 자동 부착 (eval에서 chain 검증 가능) - `metadata.handoffDepth: number` - `metadata.handoffRequest`는 마지막 handoff 정보 (없으면 미부착) - `metadata.routedBy`: supervisor의 경우 `"regex-fast-path" | "llm" | "rules-fallback"` (Iter12) ### 3.4 단일 publish 보장 NATS publish 지점은 `handleRequest` 끝 한 곳뿐. 4 hop 재귀라도 사용자는 응답 한 번만 받음. --- ## 4. Skill 라우팅 ### 4.1 Skill 등록 (bootstrap) `src/main.ts`: ``` 1. SkillRegistry.load("src/skills") - 각 폴더의 SKILL.md frontmatter 파싱 → SkillManifest 배열 2. factories map = { onboarding, moderation, supervisor } 3. 등록 루프: agent.registerSkill(factory({ ...deps })) - deps: garnetClient, llmClient, roomContextService, shadowEvalBus, natsClient, memoryService, escalationHistoryService, ... 4. TransitionMatrix.fromRegistry(skillRegistry) → Agent에 주입 ``` ### 4.2 SKILL.md frontmatter (capability card) 각 `src/skills//SKILL.md`: ```yaml name: onboarding description: ... version: 1 priority: 70 # canHandle 공동 당첨 시 desc 정렬 intents: [greeting, ..., age-question] triggers: [user-message, new-member, ...] tools: [validate-nickname, ...] resources: nickname-format: ./resources/nickname-format.md ... trustTier: T3 # T1(가장 신뢰) ~ T4(가장 보수적) domain: onboarding # Iter4 capability card canHandle: [...] cannotHandle: [...] handoffTargets: [moderation, supervisor] riskTier: low | medium | high outputContract: format: text maxChars: 600 ``` ### 4.3 라우팅 결정 — `selectSkillForRequest` `src/agent/agent.ts:28-37`: ``` 1. skills.filter(s => s.canHandle(request)) - 각 Skill의 canHandle: 보통 request.skill === self.name (정확 매칭) 2. priority desc 정렬 3. [0] 반환 ``` - onboarding: `request.skill === "onboarding"` (priority 70) - moderation: `request.skill === "moderation"` (priority 50) - supervisor: `request.skill === "supervisor"` OR fallback (priority 10) ### 4.4 TransitionMatrix (handoff 그래프) `src/skills/transition-matrix.ts`. 각 capability card의 `handoffTargets`를 모아 그래프 구성. `supervisor`는 모든 항목의 fallback 자동 포함. ``` isAllowed(from, to) # true/false targetsOf(from) # 허용 target 집합 validateGraph() # 빌드타임 일관성 ``` Dispatch loop에서 handoff 시 `transitionMatrix.isAllowed()` 검사. 거부 시 강제 supervisor로 escalation. ### 4.5 검증 — `npm run lint:skills` `src/skills/validate-manifests.ts`: - intents 배열 ⊆ 코드 `IntentKind` (manifest-binding.ts에서 derive) - tools 배열 == ToolRegistry tool names - resources 키들 실제 파일 존재 - priority 중복 0 - handoffTargets 항목이 실제 skill name 또는 `supervisor` - cannotHandle ∩ canHandle = ∅ 매 PR 시 CI에서 `lint:skills` 통과 필수. --- ## 5. Memory 계층 자료의 강조점 "context engineering > prompt engineering" + MemoryAgentBench 4축 (accurate retrieval / test-time learning / long-range understanding / selective forgetting) 모두 구현. ### 5.1 3-layer 인터페이스 (Iter10) `src/memory/memory-types.ts`: ```ts WorkingMemory // 매 turn 조립. dispatch loop가 사용 SessionMemory // 현 대화 세션. Garnet 12h TTL LongTermMemory // 영속. Garnet (profile) + MongoDB (knowledge) ``` ### 5.2 SessionMemory (Garnet, 12h~24h) | 데이터 | Garnet key | TTL | 소스 | |---|---|---|---| | Blackboard (skill-local) | `lolympus-agent:bb:{skill}:{chatroom}:{user}` | 12h | `src/skills/onboarding/state/blackboard.ts` | | Conversation history | `lolympus-agent:conv:{chatroom}:{user}` | 12h | `src/conversation/conversation-store.ts`, max 20 메시지 | | Conversation summary | `lolympus-agent:summary:{chatroom}:{user}` | 24h | `src/conversation/summarizer.ts`, 5+ 메시지 trigger | ### 5.3 LongTermMemory (Garnet + MongoDB, 30d+) | 데이터 | 저장소 | TTL | |---|---|---| | UserProfile (preferredStyle, pastIntents, flaggedIssues, longTermSummary) | Garnet `lolympus-agent:user:profile:{user}` | 30d (env: `AGENT_USER_MEMORY_TTL_DAYS`) | | KnowledgeDocuments (RAG) | MongoDB collection `knowledge_documents` | 영속 (eviction policy로 archive) | | UserEscalationHistory | MongoDB collection `user_escalation_history` | 영속 | `UserProfile` 필드 (`src/skills/onboarding/state/user-memory.ts`): ```ts userId: string // anonymizeText() 처리 firstSeenAt, lastSeenAt: string completedOnboardingAt?: string totalTurns: number pastIntents: Array<{kind, count}> // 최근 10개 preferredStyle: "warm" | "concise" | "neutral" // 자동 추론 flaggedIssues: string[] // 최대 10 longTermSummary?: { text, updatedAt, turnSnapshot } // Iter11 ``` ### 5.4 WorkingMemory 조립 매 요청 시 `message-builder.ts`에서 호출: ```ts const wm = await memoryService.assembleWorkingMemory({ skill: "onboarding", chatroomId, userId, includeBlackboard: true, includeProfile: true, retrieval: shouldRetrieve(derived) ? { query: ..., options: { skill, category, topK: 2 } } : undefined, }); // wm: { conversationHistory, summary, blackboard, userProfile, retrievedKnowledge } ``` `shouldRetrieve(derived)` (Iter10 Phase D): help-how, help-where, nickname-question, discord-question, age-question 5개 intent만 RAG. greeting/acknowledgement는 skip(비용 절감). ### 5.5 LLM prompt 주입 블록 `message-builder.ts` + `prompt-data.ts`: ``` [reference] ← retrieval된 knowledge chunk [user_context] ← test-time learning snippets (자주 묻는 intent 등) [long_term_memory] ← UserProfile.longTermSummary (Iter11) [user_escalation_history] ← supervisor capability-prompt에만 (internal only) ``` --- ## 6. Agentic RAG 자료의 "modular > monolithic" + "retrieval은 pre-filter → rerank → compression" 권고 적용. ### 6.1 Embedding (`src/memory/embedding-client.ts`) 추상화 인터페이스 + 2개 구현 (Iter10): ```ts interface EmbeddingClient { embed(text): Promise embedBatch(texts): Promise model: string dimensions: number } createEmbeddingClient() // env: EMBEDDING_PROVIDER ("openrouter" | "openai") // OpenRouter 우선 → 미지원 시 OpenAI fallback // env: EMBEDDING_MODEL (예: "text-embedding-3-small") // env: EMBEDDING_API_KEY 또는 OPENROUTER_API_KEY ``` ### 6.2 Ingest — `npm run ingest:knowledge` `scripts/ingest-knowledge.ts` → `src/memory/knowledge-ingest.ts`: 1. `src/skills/{onboarding,moderation}/resources/*.md` 5개 파일 로드 2. `## ` heading 단위로 split 3. 각 chunk를 `{ skill, category, text, metadata: { sourceFile, sectionTitle } }`로 build 4. `embeddingClient.embed(text)` → vector 5. MongoDB `knowledge_documents` collection upsert (`metadata.sourceFile + sectionTitle` 기준 idempotent) 6. `embeddingModel` 필드 자동 기록 (마이그레이션 식별용) 총 20~50 chunks 예상. 향후 1000+ 시 MongoDB Vector Search 마이그레이션. ### 6.3 Retrieval pipeline (Iter10 → Iter11) `src/memory/mongo-long-term-memory.ts:search()`: ``` 1. embeddingClient.embed(query) → query vector 2. MongoDB find with { skill?, category?, metadata.archived: { $ne: true } } → 후보 N개 (env: AGENT_RAG_CANDIDATE_POOL_SIZE, 기본 50) 3. (옵션) Hybrid (Iter11 Phase A) - cosine score (in-app, src/memory/cosine.ts) - BM25 score (in-app, src/memory/bm25.ts) - 가중 평균 (각 0.5) 4. top-K 선정 (기본 10) 5. (옵션) Reranker (Iter11 Phase A) - LlmReranker (tier 0, maxTokens 200, jsonMode) - 실패 시 원래 순서 (graceful) 6. top-N 반환 (기본 5, env: AGENT_RAG_RERANKER_TOP_K) ``` 각 단계에서 Phoenix span attribute: - `memory.retrieval.candidate_count` - `memory.retrieval.top_score` - `memory.retrieval.duration_ms` - `memory.reranker.candidates_count` - `memory.reranker.changed_order` (boolean) ### 6.4 Selective forgetting (Iter11 Phase C) `src/memory/forgetting-policy.ts` + `src/memory/contradiction-detector.ts`: 3가지 규칙: - **Relevance**: 30일간 reference 0회 + `referenceCount30d < 3` → archive - **Freshness**: `updatedAt` 90일 이상 + 사용 빈도 낮음 → archive - **Contradiction**: 같은 `sourceFile + sectionTitle` 그룹 내 텍스트 30%+ 차이 또는 명시적 키워드 → 기존 archive, 새 doc primary (staging 필수) `scripts/memory-eviction-cron.ts` (`npm run memory:evict --dry-run`): - archived는 `metadata.archived = true` flag만 (delete 아님). 복구 가능 - search() 자동으로 archived 제외 ### 6.5 사용 site **Moderation `lookup-rule` tool** (Iter10 Phase C): ``` args: { topic: string, topK?: number } → memoryService.longTerm.search(topic, { skill: "moderation", category: "rule" }) → tool result에 chunk text 포함 → LLM이 retrieval된 reference로 답변 ``` **Onboarding `message-builder.ts`** (Iter10 Phase D): - 5개 intent에서만 retrieval - `[reference]` block에 chunk 주입 - LLM이 ground truth 기반 답변 --- ## 7. Tool 실행 ### 7.1 Tool 인터페이스 (Iter12 Phase B) `src/tools/tool.ts`: ```ts interface Tool { name: string description: string argsSchema: ZodSchema resultSchema: ZodSchema requires?: readonly ToolPermission[] // Iter12: 권한 declaration execute(args: A, ctx: ToolContext): Promise> } type ToolPermission = | "garnet" | "nats" | "room-context" | "user-session" | "tool-registry" | "memory-search" | "memory-profile" ``` ### 7.2 Scoped Context Injection (Iter12 Phase B) `src/tools/scoped-context.ts`: ``` buildScopedContext(tool.requires ?? [], fullContext) → scopedContext ``` - 권한 없는 필드 접근 시: - `AGENT_TOOL_SANDBOX_ENABLED=true`: throw - `AGENT_TOOL_SANDBOX_ENABLED=false` (log-only): warn 로그만 ### 7.3 등록된 tool | Tool | Skill | requires | |---|---|---| | validate-nickname | onboarding | `[]` (pure) | | check-onboarding-progress | onboarding | `["garnet", "user-session"]` | | lookup-community-rule | onboarding | `["memory-search"]` | | get-room-context | onboarding | `["room-context"]` | | verify-discord-nickname | onboarding | `["garnet", "nats", "user-session"]` | | lookup-rule | moderation | `["memory-search"]` | ### 7.4 Tool Audit (Iter12 Phase B) `src/tools/audit.ts` + Garnet stream `lolympus-agent:tool-calls`: 매 tool 호출 후 envelope append: ```ts { type: "tool-call", toolName, chatroomId, userId, argsHash: string, // PII 마스킹된 hash (원본 args 미저장) resultSummary, ok, errorKind?, durationMs, skill, traceId, calledAt } ``` - maxLen 100_000 자동 trim - Operations CLI: `npm run tool-audit:list`, `npm run tool-audit:show ` ### 7.5 외부 통합 (lolympus 도메인) - **Discord**: `verify-discord-nickname` tool이 NATS subject `lolympus.discord.lookup-members`로 외부 microservice에 위임 (HTTP 직접 호출 X) - **Kakao**: session에 nickname 정보만 (외부 API 호출 없음, 클라이언트 제공) --- ## 8. Handoff + Supervisor (Bounded Self-Escalation Graph) 자료의 "Bounded Self-Escalation Graph" 8가지 규칙 모두 적용 (Iter4-6). ### 8.1 HandoffRequest schema (Iter4) `src/agent/types.ts`: ```ts interface HandoffRequest { targetAgent: string reason: HandoffReason // 8가지 enum taskSpec: string evidenceRefs: string[] // toolCall id 또는 messageId confidence: number // 0..1 risk: "low" | "medium" | "high" expectedOutput: string userVisibleMessage?: string } type HandoffReason = | "out-of-scope" | "missing-tool" | "low-confidence" | "validation-failed" | "same-agent-revisit" | "max-hop" | "high-risk-action" | "policy-risk" ``` ### 8.2 [HANDOFF:..] 토큰 파이프라인 (Iter4 Phase D) 기존 `[NO_RESPONSE]` 패턴을 확장. 5단계: 1. **Prompt** — `prompt-data.ts` persona에 사용법 명시 (capability card 기반 자동 합성) 2. **Abstention** — `runtime/abstention.ts`의 `AbstentionAction`에 `{kind:"handoff", targetSkill, reason}` variant 3. **Validator** — `runtime/validator.ts:142+`에서 `[HANDOFF:..]` token pass 4. **Text generator** — `runtime/text-generator.ts:79+`에서 `{status:"handoff", request}` 반환 5. **Skill.handle** — `makeEmptyResponse + metadata.handoffRequest` 반환 Parser: `src/skills/control-token-parser.ts` (regex `/\[HANDOFF:([a-z0-9_-]+):([a-z-]+)\]/giu`). ### 8.3 HandoffPacket (Iter4 Phase E) `src/agent/handoff-packet.ts`. PII 차단 minimal context: ```ts { summary: string // src/conversation/summarizer.ts 결과 userGoal: string // 240자 제한, PII 제거 currentState: Record // phase/responseMode/tier만 (userMessage 원문 미포함) handoffReason: HandoffReason evidence: HandoffEvidence[] // toolCalls[].id constraints: string[] expectedOutputSchema: string sourceChain: string[] // 거쳐간 skill 경로 } ``` 다음 dispatch hop에서 `request.context.handoffPacket`으로 전달. 수신 skill은 `readHandoffPacket(request)` 호출. ### 8.4 BudgetTracker (Iter4 Phase F + Iter12 Phase A) `src/agent/budget.ts`: | 한도 | 기본 | env | |---|---|---| | MAX_HANDOFFS | 4 | `AGENT_MAX_HANDOFFS` | | MAX_SAME_AGENT_VISITS | 2 | `AGENT_MAX_SAME_AGENT_VISITS` | | MAX_TOTAL_TOOL_CALLS | 20 | `AGENT_MAX_TOOL_CALLS` | | MAX_COST_USD | 0.5 | `AGENT_MAX_COST_USD` | `assertBeforeHandoff(target)` verdict: `"ok" | "max-hops" | "same-agent-loop" | "tool-budget" | "cost-budget"`. **Iter12 핵심**: `recordLlmCost(costUsd)` — LlmClient 호출 끝에서 자동 hook (`context.budget?.recordLlmCost(cost)`). snapshot에 `totalLlmCostUsd`, `totalToolCostUsd` 분리 노출. ### 8.5 Supervisor (Iter4 Phase G + Iter6 + Iter12 Phase A) `src/skills/supervisor/skill.ts`. **LLM + 룰 기반 hybrid**: ``` 1. readHandoffPacket(request) 2. routeByRegex(packet) → { target, confidence } 3. confidence ≥ 0.85 (env: SUPERVISOR_FAST_PATH_CONFIDENCE) → LLM skip. metadata.routedBy = "regex-fast-path" ← Iter12 G 패턴 4. 그 외 → routeByLlm(packet): - capability-prompt에 모든 skill capability card 자동 합성 - + escalation history (Iter11) [user_escalation_history] block - tier 0, maxTokens 150, jsonMode - { target, confidence, reason } JSON 응답 5. LLM 실패 → 정규식 fallback. metadata.routedBy = "rules-fallback" 6. appendSupervisorReview() → review queue + audit 7. AlertService.sendAlert (NATS + Slack, Iter7) 8. 응답 텍스트: 안정성 위해 template 유지 (Iter6 결정) ``` ### 8.6 Runtime Evaluator (Iter4 Phase H) `SkillJudgeConfig.runtimeVerifier?: (input) => "pass" | "retry" | "escalate"`: - pass → 진행 - retry → 1회 재호출 (`trigger="retry"`, `MAX_RUNTIME_RETRIES=1`) - escalate → supervisor 강제 handoff (`reason="validation-failed"`) Onboarding 기본 verifier: `validateOnboardingResponse`. Moderation: PII/외부 링크 차단 regex. --- ## 9. Cost-aware Tier Selection (Iter12 Phase A) 자료 G 패턴 + OpenInference cost 활용. ### 9.1 Tier 구조 | Tier | 용도 | 기본 모델 | maxTokens | env | |---|---|---|---|---| | 0 | 분류, 짧은 응답, supervisor LLM, reranker | gpt-5.3-codex-spark | 384 | `TIER0_MODEL`, `TIER0_MAX_TOKENS` | | 1 | planner / responder | gpt-5.4 / gpt-5.3 | 512 | `TIER1_PLANNER_MODEL`, `TIER1_RESPONDER_MODEL` | | 2 | critic | gpt-5.4 | 512 | `TIER2_CRITIC_MODEL` | ### 9.2 Router 결정 입력 (Iter12 확장) `src/skills/onboarding/runtime/router.ts:decidePreGenerationRouting()`: ```ts input: { classified: { intent, confidence }, derived, classifyConfidence: number, // Iter12 intentRiskTier?: "low"|"medium"|"high", // Iter12 (capability card) userEscalationCount?: number, // Iter12 (escalation history) recentLatencyP95Ms?: number // Iter12 (선택) } ``` 룰: - confidence ≥ 0.9 + low-risk → tier 0 (cost 최소) - confidence < 0.5 또는 high-risk + escalation > 3 → tier 1 (quality 우선) - p95 latency SLA 위반 추세 → tier 0 (degradation 보호) ### 9.3 LLM cost 추적 (Iter9 + Iter12) `src/llm/llm-client.ts`: ```ts const costUsd = estimateChatCost(model, prompt_tokens, completion_tokens); metrics.tokenCostUsd.observe(costUsd, { model, skill: getContext()?.skill }); getContext()?.budget?.recordLlmCost(costUsd); // Iter12 // Phoenix span: gen_ai.usage.cost_usd = costUsd ``` `estimateChatCost` (`src/llm/cost.ts`): 모델별 input/output rate 테이블. ### 9.4 환경 변수 (cost-aware) | env | 기본 | 효과 | |---|---|---| | `AGENT_COST_AWARE_TIER_ENABLED` | true | confidence 기반 tier 결정 | | `AGENT_BUDGET_LLM_COST_ENABLED` | true | LlmClient → Budget 자동 hook | | `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | regex 명확 시 LLM skip | | `SUPERVISOR_FAST_PATH_CONFIDENCE` | 0.85 | 임계값 | --- ## 10. Evaluation 파이프라인 자료의 "평가 가능한 실행 환경 없이 데모는 의미 없음" 권고. Iter3에서 skill-aware로 전환됨. ### 10.1 Golden set 구조 (Iter2 + 후속) ``` src/eval/golden/ ├── onboarding/ │ ├── cases.jsonl # 활성 golden, ~11개 │ ├── candidate.jsonl # 자동 수집 후보 (promote 후보) │ └── capture.jsonl # 실시간 prod 캡처 buffer ├── moderation/ │ └── cases.jsonl # 5개 (Iter2) ├── handoff/ │ └── cases.jsonl # 6개 (Iter5) └── memory/ └── cases.jsonl # 6개 (Iter11, 4축 × 1~2개) ``` EvalCase 구조 (`src/eval/types.ts`): ```ts { id, name, notes, input: { request, conversationHistory, roomContext, session, blackboard, summary }, expected: { intent?, mustIncludeAny?, mustNotInclude?, responseMode?, processExpected?: { expectedTools, expectedApproach, minSteps, toolMode }, expectedHandoff?: { shouldHandoff, expectedReason, expectedTarget }, // Iter5 expectedFinalSkill?: string, expectedSourceChain?: { mode: "exact"|"prefix"|"contains", chain }, expectedMemory?: MemoryJudgeExpected // Iter11 } } ``` ### 10.2 Runner 흐름 (`src/eval/runner.ts`) ``` 1. resolveEvalSetPath(setName) "onboarding/cases" → src/eval/golden/onboarding/cases.jsonl "golden" → legacy alias for "onboarding/cases" 2. parseEvalCaseLine() → EvalCase[] 3. for each case: a. SKILL_RUNNER_FACTORIES[skill]({deps}) → Skill instance b. handoff/cases set이면 runEvalDispatch (mini-dispatcher, Iter5) 그 외엔 skill.handle() 직접 호출 c. judges 실행: - rule-judge (mustIncludeAny, mustNotInclude, intent) - llm-judge (skill별 평가 축, SkillJudgeConfig) - process-judge (tool trace 검증) - handoff-judge (chain, reason, target) — handoff set만 - memory-judge (retrieval/long-range/test-time/forgetting) — memory set만 d. summary 집계 4. baseline (eval-reports/baseline.{skill}.json) 비교 5. eval-reports/.json 저장 ``` ### 10.3 Judges (`src/eval/judges/`) | Judge | 책임 | Iter | |---|---|---| | `rule-judge.ts` | mustIncludeAny / mustNotInclude / intent 매칭 | Iter3 일반화 | | `llm-judge.ts` | skill별 평가 축. `SkillJudgeConfig.axes` | Iter3 | | `process-judge.ts` | tool trace, skill별 tool registry | Iter3 | | `handoff-judge.ts` | dispatch chain 검증 | Iter5 | | `memory-judge.ts` | retrieval 정확도, long-range 회상 등 | Iter11 | | `attribution.ts` | slice 기반 root cause | (이전) | `SkillJudgeConfig` (`src/eval/judges/skill-config.ts`): ```ts { onboarding: { axes: ["friendliness", "accuracy", "concision", "scope"], toolRegistryFactory: createOnboardingToolRegistry, validator: validateOnboardingResponse, promptTemplate, tournamentInstruction, runtimeVerifier?: (input) => "pass"|"retry"|"escalate" }, moderation: { axes: ["appropriateness", "accuracy", "helpfulness"], toolRegistryFactory: createModerationToolRegistry, ... } } ``` ### 10.4 자동 개선 루프 ``` [1] Capture (shadow-eval-bus) — prod 트래픽 sample → shadow envelope ↓ [2] Run — golden cases 실행, judges 적용, eval-reports/.json ↓ [3] Propose (eval:propose CLI) — 저점수 slice → LLM이 prompt variant 생성 → src/eval/proposals/.json ↓ [4] Tournament (eval:tournament) — control vs variant A/B, ELO 갱신 ↓ [5] Governance gates (G1~G4) — trust tier 검증 ↓ [6] Promote (eval:promote) — golden 후보 → 활성 cases ``` ### 10.5 변형 적용 (PROMPT_VARIANT_ID) `src/skills/onboarding/prompts/builder.ts`: - `loadPromptVariantManifest(variantId)` → variant JSON 로드 - `applyLayerDiff(layers, diff)` — `persona/policy/state/context/examples/tool-policy` 6 layer 중 일부 replace/append/remove - env `PROMPT_VARIANT_ID=` → 프로덕션에서 활성 ### 10.6 Calibration / Drift / Governance - `src/eval/aggregator.ts` — SliceMetrics (intent/phase/tier 차원) - `src/eval/drift.ts` — baseline 대비 점수 하락 감지 - `src/eval/governance/trust.ts` — T1~T4 enum - `src/eval/governance/gates.ts` — variant 승격 4 gate (G1~G4) - `src/eval/router-shadow-report.ts` (Iter6) — learned vs static routing divergence --- ## 11. Memory 평가 (Iter11) `src/eval/golden/memory/cases.jsonl` 6 cases: | # | 이름 | 검증 | |---|---|---| | 1 | retrieval-accurate | "광고 도배" 질문 → moderation lookup-rule retrieval 정확 chunk | | 2 | long-range-understanding | 5턴 전 "닉네임 LolName_KR" → 회상 | | 3 | test-time-learning | flaggedIssues 3회 누적 → prophylactic guidance | | 4 | selective-forgetting | 신규 doc 30% 차이 + "정정" 키워드 → 기존 archive | | 5 | escalation-history | 사용자 3회 광고 시도 → supervisor가 moderation 우선 라우팅 | | 6 | memory-contradiction | "닉네임 X로 변경" → profile 반영, 다음 응답이 새 닉네임 인지 | `MemoryJudgeResult` (`src/eval/judges/memory-judge.ts`): - `expectedRetrievedDocIds`, `expectedRecalledFact`, `expectedProphylacticHint`, `expectedArchivedDocPattern`, `expectedSupervisorTarget`, `expectedContradictionResolved` 명령: `npm run eval -- --set=memory/cases`. --- ## 12. Operations — 운영 루프 ### 12.1 Review Queue 두 종류 | Queue | Garnet stream | 목적 | 진입 | |---|---|---|---| | Eval Review | `lolympus-agent:eval:review:queue` | 학습용 (저점수/희귀 slice/rule-llm 불일치) | runner가 자동 enqueue | | Supervisor Handoff | `lolympus-agent:review-queue` | 운영용 (escalation case 사람 검토) | supervisor.skill.ts | ### 12.2 Operations CLI (Iter5 + Iter12) `src/operations/cli.ts`: ```bash # Supervisor handoff queue npm run review:list -- --limit=20 --since=24h npm run review:show -- [--history] # --history는 Iter6 audit log npm run review:resolve -- --action=manual-reply --note="..." npm run review:stats -- --since=7d # Tool audit (Iter12) npm run tool-audit:list -- --skill=onboarding --since=1h --limit=50 npm run tool-audit:show -- ``` ### 12.3 Resolution audit log (Iter6) 이중 저장: - **Hash** `lolympus-agent:review-resolved:` — latest state (덮어쓰기, 7d TTL) - **Stream** `lolympus-agent:review-resolutions` — append-only (운영자 변경 이력) `getResolutionHistory(garnet, id)` — 사용자별 resolution 변경 이력 조회 (Iter11 escalation history 입력). ### 12.4 Alert flow (Iter7) `src/observability/alert.ts:AlertService.sendAlert(payload)`: ``` 1. Garnet dedupe key 검사 (5분 throttle) key: lolympus-agent:alert:dedupe:: 2. NATS publish → AGENT_ALERT_SUBJECT 3. Slack webhook POST (env: SLACK_ALERT_WEBHOOK_URL) - fire-and-forget, AbortSignal.timeout 3초 - block kit message (severity emoji + 필드) 4. logger.warn (구조화 로그) ``` AlertPayload: ```ts { kind: "archive-stalled"|"endpoint-down"|"eval-drift"|"eval-score-drop" |"latency"|"llm-error-rate"|"supervisor-handoff", metric, severity, skill?, threshold, value, window, timestamp, ... } ``` ### 12.5 Slack message 포맷 (Iter7) `src/observability/slack-notifier.ts:formatSlackMessage()`: - severity emoji (`:information_source:`/`:warning:`/`:rotating_light:`) - Section + fields (metric / value vs threshold / window / skill) - suggestedAction (있다면) --- ## 13. Observability ### 13.1 Logger (`src/observability/logger.ts`) - Pino + AsyncLocalStorage - `getLogger()` → context의 child logger 자동 (skill, requestId, chatroomId, userId) - redact: `authorization`, `env.`, `secrets.`, `request.` (KNOWN_SECRET_KEYS 확장됨, Iter12) ### 13.2 Tracing (Iter9) `src/observability/tracing.ts`: - OTel NodeSDK + auto-instrumentation - Exporter: env `PHOENIX_OTLP_ENDPOINT` 설정 시 OTLP HTTP, 미설정 시 ConsoleSpanExporter (또는 OFF, env: `OTEL_EXPORTER`) - Sampling: `OTEL_SAMPLING_RATIO` (prod 0.1, dev 1.0) `src/observability/spans.ts` helper: - `withSkillSpan(skill, attrs, fn)` — `skill..handle` - `withSpan(name, attrs, fn)` — generic + auto-error - `withClassifierSpan`, `withToolSpan`, `withLlmSpan` ### 13.3 LLM Span 속성 (Iter9 OpenInference) `src/llm/llm-client.ts`: ``` gen_ai.request.model gen_ai.request.temperature gen_ai.request.max_tokens gen_ai.system (provider name) gen_ai.response.id gen_ai.response.finish_reasons[] gen_ai.usage.input_tokens gen_ai.usage.output_tokens gen_ai.usage.cost_usd ← Iter9, Iter12에서 routing 활용 llm.token_count.prompt ← Phoenix 친화 alias llm.token_count.completion gen_ai.prompt.0.role/content ← optional (PHOENIX_LOG_PROMPTS) gen_ai.completion.0.content ← optional + PII 마스킹 ``` Memory 관련 (Iter10-11): - `memory.retrieval.candidate_count` - `memory.retrieval.top_score` - `memory.retrieval.duration_ms` - `memory.reranker.changed_order` Handoff 관련: - `agent.handoff_depth` - `agent.handoff_chain` - `supervisor.routed_by` ### 13.4 Metrics (Iter4 Phase I) 자체 구현 (`src/observability/metrics.ts`). `/metrics` HTTP endpoint (`metrics-server.ts`, port 기본 9464). 10 counter/histogram: | Metric | 타입 | Labels | |---|---|---| | `agent_handoff_total` | counter | from, to, reason | | `agent_handoff_loop_total` | counter | from | | `agent_handoff_count` | histogram (0..4) | — | | `agent_human_override_total` | counter | kind | | `agent_missed_escalation_total` | counter | — | | `agent_routing_accuracy_total` | counter | result | | `agent_unnecessary_escalation_total` | counter | — | | `agent_validator_fail_total` | counter | skill, stage | | `agent_token_cost_usd` | histogram | model, skill | | `agent_time_to_resolution_ms` | histogram | skill | `/healthz` endpoint: 200 "ok" (Iter7). ### 13.5 Alert threshold (skill별, Iter7) `src/config.ts`: ``` getAlertThresholdP95LatencyMs(skill?: string) getAlertThresholdEvalScoreDropPct(skill?: string) getAlertThresholdLlmErrorRate(skill?: string) getAlertThresholdDriftPct(skill?: string) ``` env 패턴: `ALERT_THRESHOLD__` 우선 → `ALERT_THRESHOLD_` fallback. 예: `ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING=5000` 또는 글로벌 `ALERT_THRESHOLD_P95_LATENCY_MS=3000`. --- ## 14. 보안 ### 14.1 Secret Manager (`src/common/secret-manager.ts`) 계층: 1. **Infisical** (외부 secret manager, `INFISICAL_SITE_URL`) 2. **AWS IMDS** (EC2 메타데이터 fallback) 3. **환경변수** (마지막 fallback) `KNOWN_SECRET_KEYS` (`src/common/known-secret-keys.ts`, Iter12 확장): ``` INFISICAL_*, NATS_*, GARNET_*, MONGODB_*, SLACK_ALERT_WEBHOOK_URL, OPENROUTER_API_KEY, AGENT_LLM_API_KEY, OPENAI_API_KEY, EMBEDDING_API_KEY, PHOENIX_API_KEY ``` `getSecret(key)`: KNOWN_SECRET_KEYS 검증. `getRawSecret(key)`: 임의 키 (운영 도구용). ### 14.2 NATS Subject ACL (Iter12 Phase C) `src/common/nats-skill-acl.ts:DEFAULT_NATS_ACL`: ```ts [ { skill: "onboarding", allowedPublishSubjects: ["lolympus.discord.lookup-members"], allowedSubscribeSubjects: [] }, { skill: "moderation", allowedPublishSubjects: [], ... }, { skill: "supervisor", allowedPublishSubjects: ["lolympus.agent.alert"], ... } ] ``` `src/common/nats.ts`의 publish wrapper: - `getNatsAclMode()`: - `"off"`: 검사 skip - `"log-only"` (기본): warn 로그, 정상 publish - `"enforce"`: throw + alert 운영 흐름: log-only 1개월 → false positive 없으면 enforce 전환 (Iter13+ task). ### 14.3 PII 차단 - **HandoffPacket**: `userMessage` 원문 포함 X. `userGoal`은 240자 + anonymize - **Memory user-profile**: `anonymizeText()` 처리 - **Tool audit**: `argsHash`만 저장, 원본 args X - **Phoenix prompt content**: `PHOENIX_LOG_PROMPTS=false` 기본 - **escalation history**: supervisor system prompt 내부만, 응답 텍스트에 노출 금지 --- ## 15. 환경 변수 + 배포 ### 15.1 인프라 | env | 용도 | |---|---| | `NATS_ENDPOINT`, `NATS_USER`, `NATS_PASS` | NATS 연결 | | `GARNET_ENDPOINT`, `GARNET_USERID`, `GARNET_PASSWORD` | Garnet 연결 | | `MONGODB_CONNECTION_STRING`, `MONGODB_PROJECT_ID` | MongoDB (optional) | | `INFISICAL_CLIENT_ID`, `INFISICAL_CLIENT_SECRET` | Secret manager | ### 15.2 모델 | env | 기본 | 용도 | |---|---|---| | `TIER0_MODEL` | gpt-5.3-codex-spark | 분류, supervisor LLM, reranker | | `TIER1_PLANNER_MODEL`, `TIER1_RESPONDER_MODEL` | gpt-5.4, gpt-5.3 | 메인 응답 | | `TIER2_CRITIC_MODEL` | gpt-5.4 | critic | | `TIER0_MAX_TOKENS` | 384 | | | `EMBEDDING_PROVIDER` | openrouter | OpenRouter→OpenAI fallback | | `EMBEDDING_MODEL` | text-embedding-3-small | | | `EMBEDDING_API_KEY` | — | (또는 OPENROUTER/OPENAI 키 재사용) | ### 15.3 기능 flag (Iter별) | env | 기본 | Iter | |---|---|---| | `AGENT_ROUTER_MODE` | static | Iter3 | | `AGENT_RAG_ENABLED` | true | Iter10 | | `AGENT_RAG_HYBRID_ENABLED` | true | Iter11 | | `AGENT_RAG_RERANKER_ENABLED` | true | Iter11 | | `AGENT_RAG_CANDIDATE_POOL_SIZE` | 50 | Iter11 | | `AGENT_RAG_RERANKER_TOP_K` | 5 | Iter11 | | `AGENT_LONG_TERM_SUMMARY_ENABLED` | true | Iter11 | | `AGENT_MEMORY_EVICTION_ENABLED` | false (cron만) | Iter11 | | `AGENT_COST_AWARE_TIER_ENABLED` | true | Iter12 | | `AGENT_BUDGET_LLM_COST_ENABLED` | true | Iter12 | | `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | Iter12 | | `AGENT_TOOL_SANDBOX_ENABLED` | false (1주 검증 후 true) | Iter12 | | `AGENT_NATS_ACL_MODE` | log-only | Iter12 | | `AGENT_SUPERVISOR_LLM_ENABLED` | true | Iter6 | | `AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED` | false (template 우선) | Iter8 | ### 15.4 한도 (Budget) | env | 기본 | |---|---| | `AGENT_MAX_HANDOFFS` | 4 | | `AGENT_MAX_SAME_AGENT_VISITS` | 2 | | `AGENT_MAX_TOOL_CALLS` | 20 | | `AGENT_MAX_COST_USD` | 0.5 | ### 15.5 임계값 (Alert) | env (skill suffix optional) | 기본 | |---|---| | `ALERT_THRESHOLD_P95_LATENCY_MS` | 3000 | | `ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT` | 10 | | `ALERT_THRESHOLD_LLM_ERROR_RATE` | 0.1 | | `ALERT_THRESHOLD_DRIFT_PCT` | 15 | | `ALERT_RATE_LIMIT_SEC` | 300 | ### 15.6 Observability | env | 기본 | |---|---| | `PHOENIX_OTLP_ENDPOINT` | unset (console fallback) | | `PHOENIX_LOG_PROMPTS` | false | | `SLACK_ALERT_WEBHOOK_URL` | unset (Slack 비활성) | | `SLACK_WEBHOOK_TIMEOUT_MS` | 3000 | | `METRICS_PORT` | 9464 | | `OTEL_SAMPLING_RATIO` | 0.1 (prod) | ### 15.7 Bootstrap (`src/main.ts`) 순서: ``` 1. initLogger() 2. initTracing() (Phoenix 또는 console) 3. startMetricsServer() (port 9464) 4. SecretManager.load (Infisical + IMDS) 5. NatsClient.connect 6. GarnetClient.connect 7. initMongoDB (optional) 8. SkillRegistry.load("src/skills") 9. TransitionMatrix.fromRegistry 10. ConversationStore + ShadowEvalBus 11. memoryService, escalationHistoryService, alertService (Slack notifier) 12. factories map { onboarding, moderation, supervisor } 13. for manifest in registry: agent.registerSkill(factory(deps)) 14. agent.start() (NATS subscribe) 15. shutdown hooks ``` --- ## 16. 도메인 데이터 (Lolympus 카카오톡 오픈채팅) ### 16.1 Onboarding (`src/skills/onboarding/`) **Capability card**: - priority: 70, trustTier: T3, riskTier: low - domain: onboarding - handoffTargets: [moderation, supervisor] **Intent** (`intents.ts`, 12개): - greeting, acknowledgement - status-question - help-how, help-where - nickname-question, photo-question - discord-question, discord-verify-question - age-question - chitchat, off-topic-to-other, unknown **Tools** (5개): validate-nickname, check-onboarding-progress, lookup-community-rule, get-room-context, verify-discord-nickname **Resources** (4개 .md): nickname-format, heart-rule, discord-format, age-policy **핵심 로직**: - `OnboardingIntentClassifier` (3-stage: trigger → regex → LLM) - `deriveState(session, trigger, intent)` → phase + responseMode + tier - `runtime/`: orchestrator, router, validator, critic, tool-plan-runner, message-builder, text-generator, abstention 등 ~10개 모듈 ### 16.2 Moderation (`src/skills/moderation/`) **Capability card**: - priority: 50, trustTier: T4, riskTier: medium - domain: moderation - handoffTargets: [onboarding, supervisor] **Intent** (5개): greeting, rule-question, violation-report, appeal, off-topic **Tools** (1개): lookup-rule (Iter10에서 RAG 기반으로 전환) **Resources** (1개 .md): community-rules (욕설, 광고, 도배, 신고 절차, 제재, 이의제기, 미성년자, 개인정보 등 섹션) ### 16.3 Supervisor (`src/skills/supervisor/`) **Capability card**: - priority: 10 (최저, fallback 전용), riskTier: medium - domain: meta - canHandle: ["*"] - handoffTargets: [onboarding, moderation] **핵심**: - `capability-prompt.ts` — 모든 skill의 capability card를 system prompt로 자동 합성 - `routing-rules.ts` — regex 기반 키워드 매칭 (Iter12 fast-path) - `review-queue.ts` — appendSupervisorReview + getResolutionHistory + 도메인 escalation history 연동 - 응답 텍스트: 안정성 위해 template ("운영자에게 전달했어요") ### 16.4 User Escalation History (Iter11 — lolympus 차별점) `user_escalation_history` collection (MongoDB): ```ts { userId, chatroomId, escalations: [{ queuedAt, reason, target, resolution: { resolvedAt, action, note }, packetSummary, sourceChain }], totalEscalations, lastEscalationAt, topReasons: [{ reason, count }] } ``` Supervisor가 매 escalation 시 history 조회 → capability prompt의 `[user_escalation_history]` 블록 주입: > "이 사용자는 광고 시도 3회 (out-of-scope) → 다음 위반 시 즉시 차단 권장" > "닉네임 검증 5회 실패 → 친절한 step-by-step 우선" --- ## 17. 명령어 cheat sheet ### 17.1 개발 ```bash npm run typecheck # TypeScript 검증 npm run test # Vitest 전체 npm run test -- --reporter=verbose # 상세 npm run lint # ESLint npm run lint:skills # SKILL.md ↔ 코드 일관성 npm run build # tsc + tsc-alias npm run dev # 로컬 부트스트랩 ``` ### 17.2 Evaluation ```bash # Run npm run eval -- --set=onboarding/cases npm run eval -- --set=moderation/cases npm run eval:handoff # = --set=handoff/cases npm run eval -- --set=memory/cases # Iter11 # Baseline / Report npm run eval:baseline npm run eval:report # 자동 개선 loop npm run eval:capture npm run eval:propose -- --slice="tier=0,intent=help-how,phase=initial,skill=onboarding" npm run eval:tournament -- --skill=onboarding npm run eval:promote npm run eval:calibrate -- --check npm run eval:router-shadow-report -- --since=7d # Iter6 ``` ### 17.3 Operations (Iter5 + Iter12) ```bash # Supervisor review queue npm run review:list npm run review:show -- npm run review:show -- --history # Iter6 resolution audit npm run review:resolve -- --action=manual-reply --note="..." npm run review:stats # Tool audit npm run tool-audit:list npm run tool-audit:show -- ``` ### 17.4 Memory (Iter10-11) ```bash npm run ingest:knowledge # resources/*.md → MongoDB embedding npm run memory:evict -- --dry-run --since=30d npm run memory:evict -- --since=30d # 실제 archive ``` ### 17.5 디버깅 ```bash PHOENIX_OTLP_ENDPOINT=http://localhost:6006/v1/traces npm run dev curl http://localhost:9464/metrics | grep agent_ curl http://localhost:9464/healthz # Specific log level LOG_LEVEL=debug npm run dev # RAG/sandbox 비활성으로 회귀 확인 AGENT_RAG_ENABLED=false npm run eval -- --set=onboarding/cases AGENT_TOOL_SANDBOX_ENABLED=false npm test ``` --- ## 18. 참고 자료 ### 18.1 Iter별 plan 문서 `docs/`에 dev 브랜치 보존 (main에는 미포함): | 문서 | 내용 | |---|---| | `refactor-skill-ssot.md` | Iter1 plan | | `refactor-skill-ssot-iter2~8.md` | Iter2~8 각 plan | | `optimization-roadmap.md` | Iter8 closing 시 trigger 정의 | | `refactor-skill-ssot-iter10~12.md` | Iter10~12 plan | | `optimization-roadmap-memory-ops.md` | Iter9~12 통합 로드맵 | | `service-evaluation-operations.md` | 운영자 매뉴얼 | | `최신기술동향.md`, `멀티에이전트동향.md` | 자료 reference | ### 18.2 외부 자료 - **MemoryAgentBench** — accurate retrieval, test-time learning, long-range understanding, selective forgetting - **Bounded Self-Escalation Graph** — Microsoft orchestration guide - **OpenInference semantic conventions** — `gen_ai.*`, `llm.*` span attributes - **MCP / A2A** — 향후 채택 후보 (Iter13+ trigger 기반) ### 18.3 다음 단계 trigger (Iter13+) `docs/optimization-roadmap.md`의 trigger 발동 시 개별 plan으로 시작: | 항목 | Trigger | |---|---| | NATS ACL enforce 전환 | log-only 1개월 안정 + false positive 0 | | Tool sandbox enforce 전환 | log-only 1주 안정 | | MongoDB Vector Search 마이그레이션 | preview → GA | | Resolution hash sunset | Iter6 dual-write 2개월 안정 | | Cross-room context aggregation | cross-room 패턴 발견 | | Parallel fan-out | handoff 평균 hop ≥ 2.0 | | Web UI for review queue | review 진입 ≥ N/day 1개월 | | Workflow auto-discovery (AFlow) | propose/tournament 안정화 | | MCP/A2A 채택 | 외부 ecosystem 연결 필요 | 각 항목은 evidence(metric/운영 사례) 기반으로 plan 작성. --- ## 19. 변경 이력 (commit 별) main 브랜치의 주요 milestones: | sha | 메시지 | Iter | |---|---|---| | `bc0f8e0` | Add secret coverage and NATS subject ACL | 12 Phase C | | `d011a28` | Add tool permission sandbox and audit stream | 12 Phase B | | `21df14c` | Add cost-aware routing and LLM budget tracking | 12 Phase A | | `211de18` | Add hybrid memory retrieval and escalation history | 11 | | `1a86bde` | Add memory layers and RAG retrieval | 10 | | `1d0be56` | Add Phoenix OTLP tracing and LLM span attributes | 9 | | (이전) | Close multi-agent plan and add router shadow reporting | 8 | | (이전) | Add Slack alerts and skill-aware monitoring | 7 | | (이전) | Unify dispatch and add supervisor routing signals | 6 | | (이전) | Close supervisor review loop and add handoff eval | 5 | | (이전) | Make eval pipeline skill-aware and validate SKILL manifests | 3 | 각 commit의 detail은 `git log -p` 또는 dev 브랜치의 `docs/refactor-skill-ssot-iterN.md` 참고. --- ## 끝 이 문서가 outdated되면 last-verified commit을 footer에 명시: - last-verified: `bc0f8e0` (2026-05-19) 업데이트 책임: 신규 큰 변경(새 skill 추가, 새 memory layer, 새 metric 등) PR 시 함께 갱신.