Files

1330 lines
48 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Lolympus Agent — 시스템 아키텍처 및 운영 흐름
> Iter1~12 (2026-05-13 ~ 2026-05-19) 완료 시점 기준의 통합 architecture + operations 문서입니다.
> 신규 기여자/운영자/AI agent의 단일 진입점.
---
## 0. 한눈에 보기
**Lolympus Agent**는 카카오톡 오픈채팅 운영을 자동화하는 **NATS 메시지 드리븐 멀티 에이전트 시스템**이다.
- **도메인**: 카카오톡 오픈채팅 가입 안내(onboarding) + 커뮤니티 규칙 모더레이션(moderation)
- **외부 진입**: NATS subject `lolympus-agent.request`
- **응답 형태**: NATS reply (단일 publish/요청)
- **핵심 자산**:
- Skill SSOT 디렉토리 (`src/skills/{onboarding,moderation,supervisor}/`)
- 통합 dispatch loop (`src/agent/dispatch.ts`, production + eval 공용)
- 3-layer memory + Agentic RAG (`src/memory/`)
- Bounded Self-Escalation Graph (handoff + transition matrix + budget + supervisor fallback)
- Eval 파이프라인 (capture → run → propose → tournament → promote)
- 자체 metrics + Phoenix LLM tracing
- Slack webhook + operations CLI
### 권장 시작 순서
1. **§2 아키텍처 다이어그램** — 전체 그림
2. **§3 메시지 흐름** — 한 요청이 어떻게 흐르는지
3. **§4~9** — 각 컴포넌트 깊이 있게
4. **§16 cheat sheet** — 자주 쓰는 명령
---
## 1. 기술 스택 요약
| 영역 | 선택 | 위치 |
|---|---|---|
| 언어 | TypeScript (Node 22+) | `tsconfig.json` |
| 메시지 | NATS | `src/common/nats.ts` |
| 캐시/세션 | Garnet (Redis 호환) | `src/common/garnet.ts` |
| 영속 저장 | MongoDB (mongoose) | `src/room-context/`, `src/memory/mongo-*.ts` |
| LLM provider | OpenRouter / OpenAI / cliproxy / llama (multi-provider) | `src/llm/llm-client.ts`, `src/llm/routing.ts` |
| Embedding | OpenRouter free 우선 → OpenAI fallback | `src/memory/embedding-client.ts` |
| Vector store | MongoDB collection + in-app cosine (Vector Search 마이그레이션 path 예약) | `src/memory/mongo-long-term-memory.ts`, `src/memory/cosine.ts` |
| BM25 | in-app | `src/memory/bm25.ts` |
| 관측 | OTel SDK + Phoenix OTLP + 자체 metrics 모듈 | `src/observability/` |
| 알림 | NATS publish + Slack incoming webhook | `src/observability/alert.ts`, `slack-notifier.ts` |
| Secret | Infisical → IMDS → env | `src/common/secret-manager.ts` |
| Test | Vitest | `*.test.ts` |
| Plan/Eval 데이터 | MongoDB collections + Garnet streams | 각 모듈 |
새 의존성 도입 시 §15 환경변수 + KNOWN_SECRET_KEYS 갱신 확인.
---
## 2. 아키텍처 다이어그램
```
[NATS subject "lolympus-agent.request"]
│
▼
┌─────────────────────────────────┐
│ Agent.handleRequest │ src/agent/agent.ts (~250 lines)
│ - runWithContext (AsyncLocal)│
│ - withSpan (OTel) │
│ - budget 생성, NATS reply │
└────────────┬────────────────────┘
│ dispatch(input, deps)
▼
┌─────────────────────────────────┐
│ dispatch loop │ src/agent/dispatch.ts (~580 lines)
│ ├─ selectSkillForRequest │ priority + canHandle 필터
│ ├─ runSkillOnce │ skill.handle() 호출
│ ├─ normalizeControlTokens │ [HANDOFF:...] 파싱
│ ├─ applyRuntimeVerifier │ retry/escalate
│ ├─ applyEscalationTrigger │ confidence/risk → handoff
│ ├─ prepareHandoff │ TransitionMatrix + Budget 검증
│ └─ recurse (max depth 4) │ 같은 dispatch 재진입
└────────────┬────────────────────┘
│
┌───────┴───────┐
▼ ▼
[Onboarding] [Moderation] [Supervisor]
├ intents ├ intents ├ capability-prompt (LLM tier0)
├ prompt-data ├ prompt-data ├ routing-rules (regex fast-path)
├ tools (5) ├ tools (1) └ review-queue append + alert
├ runtime/ ├ skill.ts
└ skill.ts
┌─────────────────────────────────┐
│ MemoryService (Iter10-11) │ src/memory/
│ ├─ WorkingMemory (조립) │ 매 turn
│ ├─ SessionMemory (12h Garnet)│ blackboard + conv + summary
│ └─ LongTermMemory (30d+) │
│ ├─ UserProfile (Garnet) │
│ ├─ Knowledge (Mongo+RAG) │ embedding + cosine + BM25 + reranker
│ └─ EscalationHistory (Mongo)│ user-level review queue rollup
└─────────────────────────────────┘
┌─────────────────────────────────┐
│ ToolRegistry │ src/tools/
│ ├─ Tool.requires (sandbox) │
│ ├─ scoped-context │ least-privileged ctx
│ └─ audit stream │ lolympus-agent:tool-calls
└─────────────────────────────────┘
┌─────────────────────────────────┐
│ Observability │ src/observability/
│ ├─ Pino logger (skill child) │
│ ├─ OTel tracing → Phoenix │ gen_ai.* + memory.* + handoff.*
│ ├─ Self-built metrics │ /metrics endpoint (9464)
│ └─ AlertService │ NATS + Slack webhook
└─────────────────────────────────┘
┌─────────────────────────────────┐
│ Eval Pipeline │ src/eval/
│ ├─ Capture (shadow-bus) │ prod traffic 샘플링
│ ├─ Run (runner) │ golden cases 실행
│ ├─ Judges (rule/llm/process/ │
│ │ handoff/memory) │
│ ├─ Propose (variant 생성) │
│ ├─ Tournament (A/B) │
│ └─ Promote (T4→T2) │
└─────────────────────────────────┘
```
---
## 3. 메시지 수신 → 응답 흐름
### 3.1 진입점 — `Agent.handleRequest`
`src/agent/agent.ts`. 흐름:
1. **NATS raw 수신**: `natsClient.subscribeRaw(getRequestSubject(), this.handleRequest)` (start time)
2. **JSON 파싱 + 검증**: `isValidRequest(parsed)` 통과 안 하면 즉시 empty response
3. **`runWithContext`** 호출 — `src/observability/context.ts`. AsyncLocalStorage로 `RequestContext` 주입:
- `requestId`, `chatroomId`, `userId`, `skill`, `trigger`
- `logger` (skill name child binding)
- `startedAt`, `deadlineAt`
- `budget`, `handoffDepth`, `handoffChain` (Iter4)
4. **`withSpan('agent.handle')`** OTel 외부 span 시작
5. **`dispatch(input, deps)`** 호출 (실제 비즈니스 로직은 dispatch.ts에 위임 — Iter6 추출)
6. **단일 NATS reply**: `natsClient.respond(msg, response)` (line 144 등) — dispatch 종료 후 정확히 1회
### 3.2 dispatch loop — `src/agent/dispatch.ts`
핵심 재귀 함수 `dispatch(input, deps)`:
```
인자:
input: { budget, chain, depth, history, request, skill }
deps: { selectSkillForRequest, skills, transitionMatrix,
shouldApplyRuntimeVerifier, getSkillJudgeConfig,
onHandoff?, onDispatchEnd?, evalPlan? }
순서:
1) runSkillOnce(input) → response # skill.handle()
2) normalizeControlTokens(response) # [HANDOFF:..] → metadata
3) applyRuntimeVerifier(input, response) # pass/retry/escalate (Iter6)
4) applyEscalationTrigger(...) # confidence/risk → handoff
5) withHandoffMetadata(response, chain, depth) # handoffChain 노출 (Iter5)
6) readHandoffRequest(response) # metadata.handoffRequest
└─ null → 최종 응답 반환
7) prepareHandoff(...) # next skill + packet 합성
├─ resolveHandoffTarget (TransitionMatrix 검증)
├─ budget.assertBeforeHandoff (max-hops/loop/cost)
├─ budget.recordHandoff(target)
└─ buildHandoffPacket (PII 차단, evidence_refs)
8) dispatch(next) 재귀
```
### 3.3 dispatch 종료 후
- `metadata.handoffChain: string[]` 자동 부착 (eval에서 chain 검증 가능)
- `metadata.handoffDepth: number`
- `metadata.handoffRequest`는 마지막 handoff 정보 (없으면 미부착)
- `metadata.routedBy`: supervisor의 경우 `"regex-fast-path" | "llm" | "rules-fallback"` (Iter12)
### 3.4 단일 publish 보장
NATS publish 지점은 `handleRequest` 끝 한 곳뿐. 4 hop 재귀라도 사용자는 응답 한 번만 받음.
---
## 4. Skill 라우팅
### 4.1 Skill 등록 (bootstrap)
`src/main.ts`:
```
1. SkillRegistry.load("src/skills")
- 각 폴더의 SKILL.md frontmatter 파싱 → SkillManifest 배열
2. factories map = { onboarding, moderation, supervisor }
3. 등록 루프: agent.registerSkill(factory({ ...deps }))
- deps: garnetClient, llmClient, roomContextService, shadowEvalBus,
natsClient, memoryService, escalationHistoryService, ...
4. TransitionMatrix.fromRegistry(skillRegistry) → Agent에 주입
```
### 4.2 SKILL.md frontmatter (capability card)
각 `src/skills/<name>/SKILL.md`:
```yaml
name: onboarding
description: ...
version: 1
priority: 70 # canHandle 공동 당첨 시 desc 정렬
intents: [greeting, ..., age-question]
triggers: [user-message, new-member, ...]
tools: [validate-nickname, ...]
resources:
nickname-format: ./resources/nickname-format.md
...
trustTier: T3 # T1(가장 신뢰) ~ T4(가장 보수적)
domain: onboarding # Iter4 capability card
canHandle: [...]
cannotHandle: [...]
handoffTargets: [moderation, supervisor]
riskTier: low | medium | high
outputContract:
format: text
maxChars: 600
```
### 4.3 라우팅 결정 — `selectSkillForRequest`
`src/agent/agent.ts:28-37`:
```
1. skills.filter(s => s.canHandle(request))
- 각 Skill의 canHandle: 보통 request.skill === self.name (정확 매칭)
2. priority desc 정렬
3. [0] 반환
```
- onboarding: `request.skill === "onboarding"` (priority 70)
- moderation: `request.skill === "moderation"` (priority 50)
- supervisor: `request.skill === "supervisor"` OR fallback (priority 10)
### 4.4 TransitionMatrix (handoff 그래프)
`src/skills/transition-matrix.ts`. 각 capability card의 `handoffTargets`를 모아 그래프 구성. `supervisor`는 모든 항목의 fallback 자동 포함.
```
isAllowed(from, to) # true/false
targetsOf(from) # 허용 target 집합
validateGraph() # 빌드타임 일관성
```
Dispatch loop에서 handoff 시 `transitionMatrix.isAllowed()` 검사. 거부 시 강제 supervisor로 escalation.
### 4.5 검증 — `npm run lint:skills`
`src/skills/validate-manifests.ts`:
- intents 배열 ⊆ 코드 `IntentKind` (manifest-binding.ts에서 derive)
- tools 배열 == ToolRegistry tool names
- resources 키들 실제 파일 존재
- priority 중복 0
- handoffTargets 항목이 실제 skill name 또는 `supervisor`
- cannotHandle ∩ canHandle = ∅
매 PR 시 CI에서 `lint:skills` 통과 필수.
---
## 5. Memory 계층
자료의 강조점 "context engineering > prompt engineering" + MemoryAgentBench 4축 (accurate retrieval / test-time learning / long-range understanding / selective forgetting) 모두 구현.
### 5.1 3-layer 인터페이스 (Iter10)
`src/memory/memory-types.ts`:
```ts
WorkingMemory // 매 turn 조립. dispatch loop가 사용
SessionMemory // 현 대화 세션. Garnet 12h TTL
LongTermMemory // 영속. Garnet (profile) + MongoDB (knowledge)
```
### 5.2 SessionMemory (Garnet, 12h~24h)
| 데이터 | Garnet key | TTL | 소스 |
|---|---|---|---|
| Blackboard (skill-local) | `lolympus-agent:bb:{skill}:{chatroom}:{user}` | 12h | `src/skills/onboarding/state/blackboard.ts` |
| Conversation history | `lolympus-agent:conv:{chatroom}:{user}` | 12h | `src/conversation/conversation-store.ts`, max 20 메시지 |
| Conversation summary | `lolympus-agent:summary:{chatroom}:{user}` | 24h | `src/conversation/summarizer.ts`, 5+ 메시지 trigger |
### 5.3 LongTermMemory (Garnet + MongoDB, 30d+)
| 데이터 | 저장소 | TTL |
|---|---|---|
| UserProfile (preferredStyle, pastIntents, flaggedIssues, longTermSummary) | Garnet `lolympus-agent:user:profile:{user}` | 30d (env: `AGENT_USER_MEMORY_TTL_DAYS`) |
| KnowledgeDocuments (RAG) | MongoDB collection `knowledge_documents` | 영속 (eviction policy로 archive) |
| UserEscalationHistory | MongoDB collection `user_escalation_history` | 영속 |
`UserProfile` 필드 (`src/skills/onboarding/state/user-memory.ts`):
```ts
userId: string // anonymizeText() 처리
firstSeenAt, lastSeenAt: string
completedOnboardingAt?: string
totalTurns: number
pastIntents: Array<{kind, count}> // 최근 10개
preferredStyle: "warm" | "concise" | "neutral" // 자동 추론
flaggedIssues: string[] // 최대 10
longTermSummary?: { text, updatedAt, turnSnapshot } // Iter11
```
### 5.4 WorkingMemory 조립
매 요청 시 `message-builder.ts`에서 호출:
```ts
const wm = await memoryService.assembleWorkingMemory({
skill: "onboarding",
chatroomId, userId,
includeBlackboard: true,
includeProfile: true,
retrieval: shouldRetrieve(derived)
? { query: ..., options: { skill, category, topK: 2 } }
: undefined,
});
// wm: { conversationHistory, summary, blackboard, userProfile, retrievedKnowledge }
```
`shouldRetrieve(derived)` (Iter10 Phase D): help-how, help-where, nickname-question, discord-question, age-question 5개 intent만 RAG. greeting/acknowledgement는 skip(비용 절감).
### 5.5 LLM prompt 주입 블록
`message-builder.ts` + `prompt-data.ts`:
```
[reference] ← retrieval된 knowledge chunk
[user_context] ← test-time learning snippets (자주 묻는 intent 등)
[long_term_memory] ← UserProfile.longTermSummary (Iter11)
[user_escalation_history] ← supervisor capability-prompt에만 (internal only)
```
---
## 6. Agentic RAG
자료의 "modular > monolithic" + "retrieval은 pre-filter → rerank → compression" 권고 적용.
### 6.1 Embedding (`src/memory/embedding-client.ts`)
추상화 인터페이스 + 2개 구현 (Iter10):
```ts
interface EmbeddingClient {
embed(text): Promise<number[]>
embedBatch(texts): Promise<number[][]>
model: string
dimensions: number
}
createEmbeddingClient()
// env: EMBEDDING_PROVIDER ("openrouter" | "openai")
// OpenRouter 우선 → 미지원 시 OpenAI fallback
// env: EMBEDDING_MODEL (예: "text-embedding-3-small")
// env: EMBEDDING_API_KEY 또는 OPENROUTER_API_KEY
```
### 6.2 Ingest — `npm run ingest:knowledge`
`scripts/ingest-knowledge.ts` → `src/memory/knowledge-ingest.ts`:
1. `src/skills/{onboarding,moderation}/resources/*.md` 5개 파일 로드
2. `## ` heading 단위로 split
3. 각 chunk를 `{ skill, category, text, metadata: { sourceFile, sectionTitle } }`로 build
4. `embeddingClient.embed(text)` → vector
5. MongoDB `knowledge_documents` collection upsert (`metadata.sourceFile + sectionTitle` 기준 idempotent)
6. `embeddingModel` 필드 자동 기록 (마이그레이션 식별용)
총 20~50 chunks 예상. 향후 1000+ 시 MongoDB Vector Search 마이그레이션.
### 6.3 Retrieval pipeline (Iter10 → Iter11)
`src/memory/mongo-long-term-memory.ts:search()`:
```
1. embeddingClient.embed(query) → query vector
2. MongoDB find with { skill?, category?, metadata.archived: { $ne: true } }
→ 후보 N개 (env: AGENT_RAG_CANDIDATE_POOL_SIZE, 기본 50)
3. (옵션) Hybrid (Iter11 Phase A)
- cosine score (in-app, src/memory/cosine.ts)
- BM25 score (in-app, src/memory/bm25.ts)
- 가중 평균 (각 0.5)
4. top-K 선정 (기본 10)
5. (옵션) Reranker (Iter11 Phase A)
- LlmReranker (tier 0, maxTokens 200, jsonMode)
- 실패 시 원래 순서 (graceful)
6. top-N 반환 (기본 5, env: AGENT_RAG_RERANKER_TOP_K)
```
각 단계에서 Phoenix span attribute:
- `memory.retrieval.candidate_count`
- `memory.retrieval.top_score`
- `memory.retrieval.duration_ms`
- `memory.reranker.candidates_count`
- `memory.reranker.changed_order` (boolean)
### 6.4 Selective forgetting (Iter11 Phase C)
`src/memory/forgetting-policy.ts` + `src/memory/contradiction-detector.ts`:
3가지 규칙:
- **Relevance**: 30일간 reference 0회 + `referenceCount30d < 3` → archive
- **Freshness**: `updatedAt` 90일 이상 + 사용 빈도 낮음 → archive
- **Contradiction**: 같은 `sourceFile + sectionTitle` 그룹 내 텍스트 30%+ 차이 또는 명시적 키워드 → 기존 archive, 새 doc primary (staging 필수)
`scripts/memory-eviction-cron.ts` (`npm run memory:evict --dry-run`):
- archived는 `metadata.archived = true` flag만 (delete 아님). 복구 가능
- search() 자동으로 archived 제외
### 6.5 사용 site
**Moderation `lookup-rule` tool** (Iter10 Phase C):
```
args: { topic: string, topK?: number }
→ memoryService.longTerm.search(topic, { skill: "moderation", category: "rule" })
→ tool result에 chunk text 포함
→ LLM이 retrieval된 reference로 답변
```
**Onboarding `message-builder.ts`** (Iter10 Phase D):
- 5개 intent에서만 retrieval
- `[reference]` block에 chunk 주입
- LLM이 ground truth 기반 답변
---
## 7. Tool 실행
### 7.1 Tool 인터페이스 (Iter12 Phase B)
`src/tools/tool.ts`:
```ts
interface Tool<A, R> {
name: string
description: string
argsSchema: ZodSchema<A>
resultSchema: ZodSchema<R>
requires?: readonly ToolPermission[] // Iter12: 권한 declaration
execute(args: A, ctx: ToolContext): Promise<ToolResult<R>>
}
type ToolPermission =
| "garnet" | "nats" | "room-context"
| "user-session" | "tool-registry"
| "memory-search" | "memory-profile"
```
### 7.2 Scoped Context Injection (Iter12 Phase B)
`src/tools/scoped-context.ts`:
```
buildScopedContext(tool.requires ?? [], fullContext) → scopedContext
```
- 권한 없는 필드 접근 시:
- `AGENT_TOOL_SANDBOX_ENABLED=true`: throw
- `AGENT_TOOL_SANDBOX_ENABLED=false` (log-only): warn 로그만
### 7.3 등록된 tool
| Tool | Skill | requires |
|---|---|---|
| validate-nickname | onboarding | `[]` (pure) |
| check-onboarding-progress | onboarding | `["garnet", "user-session"]` |
| lookup-community-rule | onboarding | `["memory-search"]` |
| get-room-context | onboarding | `["room-context"]` |
| verify-discord-nickname | onboarding | `["garnet", "nats", "user-session"]` |
| lookup-rule | moderation | `["memory-search"]` |
### 7.4 Tool Audit (Iter12 Phase B)
`src/tools/audit.ts` + Garnet stream `lolympus-agent:tool-calls`:
매 tool 호출 후 envelope append:
```ts
{
type: "tool-call",
toolName, chatroomId, userId,
argsHash: string, // PII 마스킹된 hash (원본 args 미저장)
resultSummary, ok, errorKind?,
durationMs, skill, traceId, calledAt
}
```
- maxLen 100_000 자동 trim
- Operations CLI: `npm run tool-audit:list`, `npm run tool-audit:show <id>`
### 7.5 외부 통합 (lolympus 도메인)
- **Discord**: `verify-discord-nickname` tool이 NATS subject `lolympus.discord.lookup-members`로 외부 microservice에 위임 (HTTP 직접 호출 X)
- **Kakao**: session에 nickname 정보만 (외부 API 호출 없음, 클라이언트 제공)
---
## 8. Handoff + Supervisor (Bounded Self-Escalation Graph)
자료의 "Bounded Self-Escalation Graph" 8가지 규칙 모두 적용 (Iter4-6).
### 8.1 HandoffRequest schema (Iter4)
`src/agent/types.ts`:
```ts
interface HandoffRequest {
targetAgent: string
reason: HandoffReason // 8가지 enum
taskSpec: string
evidenceRefs: string[] // toolCall id 또는 messageId
confidence: number // 0..1
risk: "low" | "medium" | "high"
expectedOutput: string
userVisibleMessage?: string
}
type HandoffReason =
| "out-of-scope" | "missing-tool" | "low-confidence"
| "validation-failed" | "same-agent-revisit"
| "max-hop" | "high-risk-action" | "policy-risk"
```
### 8.2 [HANDOFF:..] 토큰 파이프라인 (Iter4 Phase D)
기존 `[NO_RESPONSE]` 패턴을 확장. 5단계:
1. **Prompt** — `prompt-data.ts` persona에 사용법 명시 (capability card 기반 자동 합성)
2. **Abstention** — `runtime/abstention.ts`의 `AbstentionAction`에 `{kind:"handoff", targetSkill, reason}` variant
3. **Validator** — `runtime/validator.ts:142+`에서 `[HANDOFF:..]` token pass
4. **Text generator** — `runtime/text-generator.ts:79+`에서 `{status:"handoff", request}` 반환
5. **Skill.handle** — `makeEmptyResponse + metadata.handoffRequest` 반환
Parser: `src/skills/control-token-parser.ts` (regex `/\[HANDOFF:([a-z0-9_-]+):([a-z-]+)\]/giu`).
### 8.3 HandoffPacket (Iter4 Phase E)
`src/agent/handoff-packet.ts`. PII 차단 minimal context:
```ts
{
summary: string // src/conversation/summarizer.ts 결과
userGoal: string // 240자 제한, PII 제거
currentState: Record // phase/responseMode/tier만 (userMessage 원문 미포함)
handoffReason: HandoffReason
evidence: HandoffEvidence[] // toolCalls[].id
constraints: string[]
expectedOutputSchema: string
sourceChain: string[] // 거쳐간 skill 경로
}
```
다음 dispatch hop에서 `request.context.handoffPacket`으로 전달. 수신 skill은 `readHandoffPacket(request)` 호출.
### 8.4 BudgetTracker (Iter4 Phase F + Iter12 Phase A)
`src/agent/budget.ts`:
| 한도 | 기본 | env |
|---|---|---|
| MAX_HANDOFFS | 4 | `AGENT_MAX_HANDOFFS` |
| MAX_SAME_AGENT_VISITS | 2 | `AGENT_MAX_SAME_AGENT_VISITS` |
| MAX_TOTAL_TOOL_CALLS | 20 | `AGENT_MAX_TOOL_CALLS` |
| MAX_COST_USD | 0.5 | `AGENT_MAX_COST_USD` |
`assertBeforeHandoff(target)` verdict: `"ok" | "max-hops" | "same-agent-loop" | "tool-budget" | "cost-budget"`.
**Iter12 핵심**: `recordLlmCost(costUsd)` — LlmClient 호출 끝에서 자동 hook (`context.budget?.recordLlmCost(cost)`). snapshot에 `totalLlmCostUsd`, `totalToolCostUsd` 분리 노출.
### 8.5 Supervisor (Iter4 Phase G + Iter6 + Iter12 Phase A)
`src/skills/supervisor/skill.ts`. **LLM + 룰 기반 hybrid**:
```
1. readHandoffPacket(request)
2. routeByRegex(packet) → { target, confidence }
3. confidence ≥ 0.85 (env: SUPERVISOR_FAST_PATH_CONFIDENCE)
→ LLM skip. metadata.routedBy = "regex-fast-path" ← Iter12 G 패턴
4. 그 외 → routeByLlm(packet):
- capability-prompt에 모든 skill capability card 자동 합성
- + escalation history (Iter11) [user_escalation_history] block
- tier 0, maxTokens 150, jsonMode
- { target, confidence, reason } JSON 응답
5. LLM 실패 → 정규식 fallback. metadata.routedBy = "rules-fallback"
6. appendSupervisorReview() → review queue + audit
7. AlertService.sendAlert (NATS + Slack, Iter7)
8. 응답 텍스트: 안정성 위해 template 유지 (Iter6 결정)
```
### 8.6 Runtime Evaluator (Iter4 Phase H)
`SkillJudgeConfig.runtimeVerifier?: (input) => "pass" | "retry" | "escalate"`:
- pass → 진행
- retry → 1회 재호출 (`trigger="retry"`, `MAX_RUNTIME_RETRIES=1`)
- escalate → supervisor 강제 handoff (`reason="validation-failed"`)
Onboarding 기본 verifier: `validateOnboardingResponse`. Moderation: PII/외부 링크 차단 regex.
---
## 9. Cost-aware Tier Selection (Iter12 Phase A)
자료 G 패턴 + OpenInference cost 활용.
### 9.1 Tier 구조
| Tier | 용도 | 기본 모델 | maxTokens | env |
|---|---|---|---|---|
| 0 | 분류, 짧은 응답, supervisor LLM, reranker | gpt-5.3-codex-spark | 384 | `TIER0_MODEL`, `TIER0_MAX_TOKENS` |
| 1 | planner / responder | gpt-5.4 / gpt-5.3 | 512 | `TIER1_PLANNER_MODEL`, `TIER1_RESPONDER_MODEL` |
| 2 | critic | gpt-5.4 | 512 | `TIER2_CRITIC_MODEL` |
### 9.2 Router 결정 입력 (Iter12 확장)
`src/skills/onboarding/runtime/router.ts:decidePreGenerationRouting()`:
```ts
input: {
classified: { intent, confidence },
derived,
classifyConfidence: number, // Iter12
intentRiskTier?: "low"|"medium"|"high", // Iter12 (capability card)
userEscalationCount?: number, // Iter12 (escalation history)
recentLatencyP95Ms?: number // Iter12 (선택)
}
```
룰:
- confidence ≥ 0.9 + low-risk → tier 0 (cost 최소)
- confidence < 0.5 또는 high-risk + escalation > 3 → tier 1 (quality 우선)
- p95 latency SLA 위반 추세 → tier 0 (degradation 보호)
### 9.3 LLM cost 추적 (Iter9 + Iter12)
`src/llm/llm-client.ts`:
```ts
const costUsd = estimateChatCost(model, prompt_tokens, completion_tokens);
metrics.tokenCostUsd.observe(costUsd, { model, skill: getContext()?.skill });
getContext()?.budget?.recordLlmCost(costUsd); // Iter12
// Phoenix span: gen_ai.usage.cost_usd = costUsd
```
`estimateChatCost` (`src/llm/cost.ts`): 모델별 input/output rate 테이블.
### 9.4 환경 변수 (cost-aware)
| env | 기본 | 효과 |
|---|---|---|
| `AGENT_COST_AWARE_TIER_ENABLED` | true | confidence 기반 tier 결정 |
| `AGENT_BUDGET_LLM_COST_ENABLED` | true | LlmClient → Budget 자동 hook |
| `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | regex 명확 시 LLM skip |
| `SUPERVISOR_FAST_PATH_CONFIDENCE` | 0.85 | 임계값 |
---
## 10. Evaluation 파이프라인
자료의 "평가 가능한 실행 환경 없이 데모는 의미 없음" 권고. Iter3에서 skill-aware로 전환됨.
### 10.1 Golden set 구조 (Iter2 + 후속)
```
src/eval/golden/
├── onboarding/
│ ├── cases.jsonl # 활성 golden, ~11개
│ ├── candidate.jsonl # 자동 수집 후보 (promote 후보)
│ └── capture.jsonl # 실시간 prod 캡처 buffer
├── moderation/
│ └── cases.jsonl # 5개 (Iter2)
├── handoff/
│ └── cases.jsonl # 6개 (Iter5)
└── memory/
└── cases.jsonl # 6개 (Iter11, 4축 × 1~2개)
```
EvalCase 구조 (`src/eval/types.ts`):
```ts
{
id, name, notes,
input: { request, conversationHistory, roomContext, session, blackboard, summary },
expected: {
intent?, mustIncludeAny?, mustNotInclude?, responseMode?,
processExpected?: { expectedTools, expectedApproach, minSteps, toolMode },
expectedHandoff?: { shouldHandoff, expectedReason, expectedTarget }, // Iter5
expectedFinalSkill?: string,
expectedSourceChain?: { mode: "exact"|"prefix"|"contains", chain },
expectedMemory?: MemoryJudgeExpected // Iter11
}
}
```
### 10.2 Runner 흐름 (`src/eval/runner.ts`)
```
1. resolveEvalSetPath(setName)
"onboarding/cases" → src/eval/golden/onboarding/cases.jsonl
"golden" → legacy alias for "onboarding/cases"
2. parseEvalCaseLine() → EvalCase[]
3. for each case:
a. SKILL_RUNNER_FACTORIES[skill]({deps}) → Skill instance
b. handoff/cases set이면 runEvalDispatch (mini-dispatcher, Iter5)
그 외엔 skill.handle() 직접 호출
c. judges 실행:
- rule-judge (mustIncludeAny, mustNotInclude, intent)
- llm-judge (skill별 평가 축, SkillJudgeConfig)
- process-judge (tool trace 검증)
- handoff-judge (chain, reason, target) — handoff set만
- memory-judge (retrieval/long-range/test-time/forgetting) — memory set만
d. summary 집계
4. baseline (eval-reports/baseline.{skill}.json) 비교
5. eval-reports/<ts>.json 저장
```
### 10.3 Judges (`src/eval/judges/`)
| Judge | 책임 | Iter |
|---|---|---|
| `rule-judge.ts` | mustIncludeAny / mustNotInclude / intent 매칭 | Iter3 일반화 |
| `llm-judge.ts` | skill별 평가 축. `SkillJudgeConfig.axes` | Iter3 |
| `process-judge.ts` | tool trace, skill별 tool registry | Iter3 |
| `handoff-judge.ts` | dispatch chain 검증 | Iter5 |
| `memory-judge.ts` | retrieval 정확도, long-range 회상 등 | Iter11 |
| `attribution.ts` | slice 기반 root cause | (이전) |
`SkillJudgeConfig` (`src/eval/judges/skill-config.ts`):
```ts
{
onboarding: {
axes: ["friendliness", "accuracy", "concision", "scope"],
toolRegistryFactory: createOnboardingToolRegistry,
validator: validateOnboardingResponse,
promptTemplate, tournamentInstruction,
runtimeVerifier?: (input) => "pass"|"retry"|"escalate"
},
moderation: {
axes: ["appropriateness", "accuracy", "helpfulness"],
toolRegistryFactory: createModerationToolRegistry,
...
}
}
```
### 10.4 자동 개선 루프
```
[1] Capture (shadow-eval-bus) — prod 트래픽 sample → shadow envelope
↓
[2] Run — golden cases 실행, judges 적용, eval-reports/<ts>.json
↓
[3] Propose (eval:propose CLI) — 저점수 slice → LLM이 prompt variant 생성
→ src/eval/proposals/<variantId>.json
↓
[4] Tournament (eval:tournament) — control vs variant A/B, ELO 갱신
↓
[5] Governance gates (G1~G4) — trust tier 검증
↓
[6] Promote (eval:promote) — golden 후보 → 활성 cases
```
### 10.5 변형 적용 (PROMPT_VARIANT_ID)
`src/skills/onboarding/prompts/builder.ts`:
- `loadPromptVariantManifest(variantId)` → variant JSON 로드
- `applyLayerDiff(layers, diff)` — `persona/policy/state/context/examples/tool-policy` 6 layer 중 일부 replace/append/remove
- env `PROMPT_VARIANT_ID=<id>` → 프로덕션에서 활성
### 10.6 Calibration / Drift / Governance
- `src/eval/aggregator.ts` — SliceMetrics (intent/phase/tier 차원)
- `src/eval/drift.ts` — baseline 대비 점수 하락 감지
- `src/eval/governance/trust.ts` — T1~T4 enum
- `src/eval/governance/gates.ts` — variant 승격 4 gate (G1~G4)
- `src/eval/router-shadow-report.ts` (Iter6) — learned vs static routing divergence
---
## 11. Memory 평가 (Iter11)
`src/eval/golden/memory/cases.jsonl` 6 cases:
| # | 이름 | 검증 |
|---|---|---|
| 1 | retrieval-accurate | "광고 도배" 질문 → moderation lookup-rule retrieval 정확 chunk |
| 2 | long-range-understanding | 5턴 전 "닉네임 LolName_KR" → 회상 |
| 3 | test-time-learning | flaggedIssues 3회 누적 → prophylactic guidance |
| 4 | selective-forgetting | 신규 doc 30% 차이 + "정정" 키워드 → 기존 archive |
| 5 | escalation-history | 사용자 3회 광고 시도 → supervisor가 moderation 우선 라우팅 |
| 6 | memory-contradiction | "닉네임 X로 변경" → profile 반영, 다음 응답이 새 닉네임 인지 |
`MemoryJudgeResult` (`src/eval/judges/memory-judge.ts`):
- `expectedRetrievedDocIds`, `expectedRecalledFact`, `expectedProphylacticHint`, `expectedArchivedDocPattern`, `expectedSupervisorTarget`, `expectedContradictionResolved`
명령: `npm run eval -- --set=memory/cases`.
---
## 12. Operations — 운영 루프
### 12.1 Review Queue 두 종류
| Queue | Garnet stream | 목적 | 진입 |
|---|---|---|---|
| Eval Review | `lolympus-agent:eval:review:queue` | 학습용 (저점수/희귀 slice/rule-llm 불일치) | runner가 자동 enqueue |
| Supervisor Handoff | `lolympus-agent:review-queue` | 운영용 (escalation case 사람 검토) | supervisor.skill.ts |
### 12.2 Operations CLI (Iter5 + Iter12)
`src/operations/cli.ts`:
```bash
# Supervisor handoff queue
npm run review:list -- --limit=20 --since=24h
npm run review:show -- <id> [--history] # --history는 Iter6 audit log
npm run review:resolve -- <id> --action=manual-reply --note="..."
npm run review:stats -- --since=7d
# Tool audit (Iter12)
npm run tool-audit:list -- --skill=onboarding --since=1h --limit=50
npm run tool-audit:show -- <stream-id>
```
### 12.3 Resolution audit log (Iter6)
이중 저장:
- **Hash** `lolympus-agent:review-resolved:<id>` — latest state (덮어쓰기, 7d TTL)
- **Stream** `lolympus-agent:review-resolutions` — append-only (운영자 변경 이력)
`getResolutionHistory(garnet, id)` — 사용자별 resolution 변경 이력 조회 (Iter11 escalation history 입력).
### 12.4 Alert flow (Iter7)
`src/observability/alert.ts:AlertService.sendAlert(payload)`:
```
1. Garnet dedupe key 검사 (5분 throttle)
key: lolympus-agent:alert:dedupe:<kind>:<skill>
2. NATS publish → AGENT_ALERT_SUBJECT
3. Slack webhook POST (env: SLACK_ALERT_WEBHOOK_URL)
- fire-and-forget, AbortSignal.timeout 3초
- block kit message (severity emoji + 필드)
4. logger.warn (구조화 로그)
```
AlertPayload:
```ts
{
kind: "archive-stalled"|"endpoint-down"|"eval-drift"|"eval-score-drop"
|"latency"|"llm-error-rate"|"supervisor-handoff",
metric, severity, skill?, threshold, value, window, timestamp, ...
}
```
### 12.5 Slack message 포맷 (Iter7)
`src/observability/slack-notifier.ts:formatSlackMessage()`:
- severity emoji (`:information_source:`/`:warning:`/`:rotating_light:`)
- Section + fields (metric / value vs threshold / window / skill)
- suggestedAction (있다면)
---
## 13. Observability
### 13.1 Logger (`src/observability/logger.ts`)
- Pino + AsyncLocalStorage
- `getLogger()` → context의 child logger 자동 (skill, requestId, chatroomId, userId)
- redact: `authorization`, `env.<secret>`, `secrets.<secret>`, `request.<secret>` (KNOWN_SECRET_KEYS 확장됨, Iter12)
### 13.2 Tracing (Iter9)
`src/observability/tracing.ts`:
- OTel NodeSDK + auto-instrumentation
- Exporter: env `PHOENIX_OTLP_ENDPOINT` 설정 시 OTLP HTTP, 미설정 시 ConsoleSpanExporter (또는 OFF, env: `OTEL_EXPORTER`)
- Sampling: `OTEL_SAMPLING_RATIO` (prod 0.1, dev 1.0)
`src/observability/spans.ts` helper:
- `withSkillSpan(skill, attrs, fn)` — `skill.<name>.handle`
- `withSpan(name, attrs, fn)` — generic + auto-error
- `withClassifierSpan`, `withToolSpan`, `withLlmSpan`
### 13.3 LLM Span 속성 (Iter9 OpenInference)
`src/llm/llm-client.ts`:
```
gen_ai.request.model
gen_ai.request.temperature
gen_ai.request.max_tokens
gen_ai.system (provider name)
gen_ai.response.id
gen_ai.response.finish_reasons[]
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.cost_usd ← Iter9, Iter12에서 routing 활용
llm.token_count.prompt ← Phoenix 친화 alias
llm.token_count.completion
gen_ai.prompt.0.role/content ← optional (PHOENIX_LOG_PROMPTS)
gen_ai.completion.0.content ← optional + PII 마스킹
```
Memory 관련 (Iter10-11):
- `memory.retrieval.candidate_count`
- `memory.retrieval.top_score`
- `memory.retrieval.duration_ms`
- `memory.reranker.changed_order`
Handoff 관련:
- `agent.handoff_depth`
- `agent.handoff_chain`
- `supervisor.routed_by`
### 13.4 Metrics (Iter4 Phase I)
자체 구현 (`src/observability/metrics.ts`). `/metrics` HTTP endpoint (`metrics-server.ts`, port 기본 9464). 10 counter/histogram:
| Metric | 타입 | Labels |
|---|---|---|
| `agent_handoff_total` | counter | from, to, reason |
| `agent_handoff_loop_total` | counter | from |
| `agent_handoff_count` | histogram (0..4) | — |
| `agent_human_override_total` | counter | kind |
| `agent_missed_escalation_total` | counter | — |
| `agent_routing_accuracy_total` | counter | result |
| `agent_unnecessary_escalation_total` | counter | — |
| `agent_validator_fail_total` | counter | skill, stage |
| `agent_token_cost_usd` | histogram | model, skill |
| `agent_time_to_resolution_ms` | histogram | skill |
`/healthz` endpoint: 200 "ok" (Iter7).
### 13.5 Alert threshold (skill별, Iter7)
`src/config.ts`:
```
getAlertThresholdP95LatencyMs(skill?: string)
getAlertThresholdEvalScoreDropPct(skill?: string)
getAlertThresholdLlmErrorRate(skill?: string)
getAlertThresholdDriftPct(skill?: string)
```
env 패턴: `ALERT_THRESHOLD_<METRIC>_<SKILL_UPPER>` 우선 → `ALERT_THRESHOLD_<METRIC>` fallback.
예: `ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING=5000` 또는 글로벌 `ALERT_THRESHOLD_P95_LATENCY_MS=3000`.
---
## 14. 보안
### 14.1 Secret Manager (`src/common/secret-manager.ts`)
계층:
1. **Infisical** (외부 secret manager, `INFISICAL_SITE_URL`)
2. **AWS IMDS** (EC2 메타데이터 fallback)
3. **환경변수** (마지막 fallback)
`KNOWN_SECRET_KEYS` (`src/common/known-secret-keys.ts`, Iter12 확장):
```
INFISICAL_*, NATS_*, GARNET_*, MONGODB_*, SLACK_ALERT_WEBHOOK_URL,
OPENROUTER_API_KEY, AGENT_LLM_API_KEY, OPENAI_API_KEY,
EMBEDDING_API_KEY, PHOENIX_API_KEY
```
`getSecret(key)`: KNOWN_SECRET_KEYS 검증. `getRawSecret(key)`: 임의 키 (운영 도구용).
### 14.2 NATS Subject ACL (Iter12 Phase C)
`src/common/nats-skill-acl.ts:DEFAULT_NATS_ACL`:
```ts
[
{
skill: "onboarding",
allowedPublishSubjects: ["lolympus.discord.lookup-members"],
allowedSubscribeSubjects: []
},
{ skill: "moderation", allowedPublishSubjects: [], ... },
{ skill: "supervisor", allowedPublishSubjects: ["lolympus.agent.alert"], ... }
]
```
`src/common/nats.ts`의 publish wrapper:
- `getNatsAclMode()`:
- `"off"`: 검사 skip
- `"log-only"` (기본): warn 로그, 정상 publish
- `"enforce"`: throw + alert
운영 흐름: log-only 1개월 → false positive 없으면 enforce 전환 (Iter13+ task).
### 14.3 PII 차단
- **HandoffPacket**: `userMessage` 원문 포함 X. `userGoal`은 240자 + anonymize
- **Memory user-profile**: `anonymizeText()` 처리
- **Tool audit**: `argsHash`만 저장, 원본 args X
- **Phoenix prompt content**: `PHOENIX_LOG_PROMPTS=false` 기본
- **escalation history**: supervisor system prompt 내부만, 응답 텍스트에 노출 금지
---
## 15. 환경 변수 + 배포
### 15.1 인프라
| env | 용도 |
|---|---|
| `NATS_ENDPOINT`, `NATS_USER`, `NATS_PASS` | NATS 연결 |
| `GARNET_ENDPOINT`, `GARNET_USERID`, `GARNET_PASSWORD` | Garnet 연결 |
| `MONGODB_CONNECTION_STRING`, `MONGODB_PROJECT_ID` | MongoDB (optional) |
| `INFISICAL_CLIENT_ID`, `INFISICAL_CLIENT_SECRET` | Secret manager |
### 15.2 모델
| env | 기본 | 용도 |
|---|---|---|
| `TIER0_MODEL` | gpt-5.3-codex-spark | 분류, supervisor LLM, reranker |
| `TIER1_PLANNER_MODEL`, `TIER1_RESPONDER_MODEL` | gpt-5.4, gpt-5.3 | 메인 응답 |
| `TIER2_CRITIC_MODEL` | gpt-5.4 | critic |
| `TIER0_MAX_TOKENS` | 384 | |
| `EMBEDDING_PROVIDER` | openrouter | OpenRouter→OpenAI fallback |
| `EMBEDDING_MODEL` | text-embedding-3-small | |
| `EMBEDDING_API_KEY` | — | (또는 OPENROUTER/OPENAI 키 재사용) |
### 15.3 기능 flag (Iter별)
| env | 기본 | Iter |
|---|---|---|
| `AGENT_ROUTER_MODE` | static | Iter3 |
| `AGENT_RAG_ENABLED` | true | Iter10 |
| `AGENT_RAG_HYBRID_ENABLED` | true | Iter11 |
| `AGENT_RAG_RERANKER_ENABLED` | true | Iter11 |
| `AGENT_RAG_CANDIDATE_POOL_SIZE` | 50 | Iter11 |
| `AGENT_RAG_RERANKER_TOP_K` | 5 | Iter11 |
| `AGENT_LONG_TERM_SUMMARY_ENABLED` | true | Iter11 |
| `AGENT_MEMORY_EVICTION_ENABLED` | false (cron만) | Iter11 |
| `AGENT_COST_AWARE_TIER_ENABLED` | true | Iter12 |
| `AGENT_BUDGET_LLM_COST_ENABLED` | true | Iter12 |
| `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | Iter12 |
| `AGENT_TOOL_SANDBOX_ENABLED` | false (1주 검증 후 true) | Iter12 |
| `AGENT_NATS_ACL_MODE` | log-only | Iter12 |
| `AGENT_SUPERVISOR_LLM_ENABLED` | true | Iter6 |
| `AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED` | false (template 우선) | Iter8 |
### 15.4 한도 (Budget)
| env | 기본 |
|---|---|
| `AGENT_MAX_HANDOFFS` | 4 |
| `AGENT_MAX_SAME_AGENT_VISITS` | 2 |
| `AGENT_MAX_TOOL_CALLS` | 20 |
| `AGENT_MAX_COST_USD` | 0.5 |
### 15.5 임계값 (Alert)
| env (skill suffix optional) | 기본 |
|---|---|
| `ALERT_THRESHOLD_P95_LATENCY_MS` | 3000 |
| `ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT` | 10 |
| `ALERT_THRESHOLD_LLM_ERROR_RATE` | 0.1 |
| `ALERT_THRESHOLD_DRIFT_PCT` | 15 |
| `ALERT_RATE_LIMIT_SEC` | 300 |
### 15.6 Observability
| env | 기본 |
|---|---|
| `PHOENIX_OTLP_ENDPOINT` | unset (console fallback) |
| `PHOENIX_LOG_PROMPTS` | false |
| `SLACK_ALERT_WEBHOOK_URL` | unset (Slack 비활성) |
| `SLACK_WEBHOOK_TIMEOUT_MS` | 3000 |
| `METRICS_PORT` | 9464 |
| `OTEL_SAMPLING_RATIO` | 0.1 (prod) |
### 15.7 Bootstrap (`src/main.ts`)
순서:
```
1. initLogger()
2. initTracing() (Phoenix 또는 console)
3. startMetricsServer() (port 9464)
4. SecretManager.load (Infisical + IMDS)
5. NatsClient.connect
6. GarnetClient.connect
7. initMongoDB (optional)
8. SkillRegistry.load("src/skills")
9. TransitionMatrix.fromRegistry
10. ConversationStore + ShadowEvalBus
11. memoryService, escalationHistoryService, alertService (Slack notifier)
12. factories map { onboarding, moderation, supervisor }
13. for manifest in registry: agent.registerSkill(factory(deps))
14. agent.start() (NATS subscribe)
15. shutdown hooks
```
---
## 16. 도메인 데이터 (Lolympus 카카오톡 오픈채팅)
### 16.1 Onboarding (`src/skills/onboarding/`)
**Capability card**:
- priority: 70, trustTier: T3, riskTier: low
- domain: onboarding
- handoffTargets: [moderation, supervisor]
**Intent** (`intents.ts`, 12개):
- greeting, acknowledgement
- status-question
- help-how, help-where
- nickname-question, photo-question
- discord-question, discord-verify-question
- age-question
- chitchat, off-topic-to-other, unknown
**Tools** (5개): validate-nickname, check-onboarding-progress, lookup-community-rule, get-room-context, verify-discord-nickname
**Resources** (4개 .md): nickname-format, heart-rule, discord-format, age-policy
**핵심 로직**:
- `OnboardingIntentClassifier` (3-stage: trigger → regex → LLM)
- `deriveState(session, trigger, intent)` → phase + responseMode + tier
- `runtime/`: orchestrator, router, validator, critic, tool-plan-runner, message-builder, text-generator, abstention 등 ~10개 모듈
### 16.2 Moderation (`src/skills/moderation/`)
**Capability card**:
- priority: 50, trustTier: T4, riskTier: medium
- domain: moderation
- handoffTargets: [onboarding, supervisor]
**Intent** (5개): greeting, rule-question, violation-report, appeal, off-topic
**Tools** (1개): lookup-rule (Iter10에서 RAG 기반으로 전환)
**Resources** (1개 .md): community-rules (욕설, 광고, 도배, 신고 절차, 제재, 이의제기, 미성년자, 개인정보 등 섹션)
### 16.3 Supervisor (`src/skills/supervisor/`)
**Capability card**:
- priority: 10 (최저, fallback 전용), riskTier: medium
- domain: meta
- canHandle: ["*"]
- handoffTargets: [onboarding, moderation]
**핵심**:
- `capability-prompt.ts` — 모든 skill의 capability card를 system prompt로 자동 합성
- `routing-rules.ts` — regex 기반 키워드 매칭 (Iter12 fast-path)
- `review-queue.ts` — appendSupervisorReview + getResolutionHistory + 도메인 escalation history 연동
- 응답 텍스트: 안정성 위해 template ("운영자에게 전달했어요")
### 16.4 User Escalation History (Iter11 — lolympus 차별점)
`user_escalation_history` collection (MongoDB):
```ts
{
userId, chatroomId,
escalations: [{
queuedAt, reason, target,
resolution: { resolvedAt, action, note },
packetSummary, sourceChain
}],
totalEscalations, lastEscalationAt,
topReasons: [{ reason, count }]
}
```
Supervisor가 매 escalation 시 history 조회 → capability prompt의 `[user_escalation_history]` 블록 주입:
> "이 사용자는 광고 시도 3회 (out-of-scope) → 다음 위반 시 즉시 차단 권장"
> "닉네임 검증 5회 실패 → 친절한 step-by-step 우선"
---
## 17. 명령어 cheat sheet
### 17.1 개발
```bash
npm run typecheck # TypeScript 검증
npm run test # Vitest 전체
npm run test -- --reporter=verbose # 상세
npm run lint # ESLint
npm run lint:skills # SKILL.md ↔ 코드 일관성
npm run build # tsc + tsc-alias
npm run dev # 로컬 부트스트랩
```
### 17.2 Evaluation
```bash
# Run
npm run eval -- --set=onboarding/cases
npm run eval -- --set=moderation/cases
npm run eval:handoff # = --set=handoff/cases
npm run eval -- --set=memory/cases # Iter11
# Baseline / Report
npm run eval:baseline
npm run eval:report
# 자동 개선 loop
npm run eval:capture
npm run eval:propose -- --slice="tier=0,intent=help-how,phase=initial,skill=onboarding"
npm run eval:tournament -- --skill=onboarding
npm run eval:promote
npm run eval:calibrate -- --check
npm run eval:router-shadow-report -- --since=7d # Iter6
```
### 17.3 Operations (Iter5 + Iter12)
```bash
# Supervisor review queue
npm run review:list
npm run review:show -- <id>
npm run review:show -- <id> --history # Iter6 resolution audit
npm run review:resolve -- <id> --action=manual-reply --note="..."
npm run review:stats
# Tool audit
npm run tool-audit:list
npm run tool-audit:show -- <id>
```
### 17.4 Memory (Iter10-11)
```bash
npm run ingest:knowledge # resources/*.md → MongoDB embedding
npm run memory:evict -- --dry-run --since=30d
npm run memory:evict -- --since=30d # 실제 archive
```
### 17.5 디버깅
```bash
PHOENIX_OTLP_ENDPOINT=http://localhost:6006/v1/traces npm run dev
curl http://localhost:9464/metrics | grep agent_
curl http://localhost:9464/healthz
# Specific log level
LOG_LEVEL=debug npm run dev
# RAG/sandbox 비활성으로 회귀 확인
AGENT_RAG_ENABLED=false npm run eval -- --set=onboarding/cases
AGENT_TOOL_SANDBOX_ENABLED=false npm test
```
---
## 18. 참고 자료
### 18.1 Iter별 plan 문서
`docs/`에 dev 브랜치 보존 (main에는 미포함):
| 문서 | 내용 |
|---|---|
| `refactor-skill-ssot.md` | Iter1 plan |
| `refactor-skill-ssot-iter2~8.md` | Iter2~8 각 plan |
| `optimization-roadmap.md` | Iter8 closing 시 trigger 정의 |
| `refactor-skill-ssot-iter10~12.md` | Iter10~12 plan |
| `optimization-roadmap-memory-ops.md` | Iter9~12 통합 로드맵 |
| `service-evaluation-operations.md` | 운영자 매뉴얼 |
| `최신기술동향.md`, `멀티에이전트동향.md` | 자료 reference |
### 18.2 외부 자료
- **MemoryAgentBench** — accurate retrieval, test-time learning, long-range understanding, selective forgetting
- **Bounded Self-Escalation Graph** — Microsoft orchestration guide
- **OpenInference semantic conventions** — `gen_ai.*`, `llm.*` span attributes
- **MCP / A2A** — 향후 채택 후보 (Iter13+ trigger 기반)
### 18.3 다음 단계 trigger (Iter13+)
`docs/optimization-roadmap.md`의 trigger 발동 시 개별 plan으로 시작:
| 항목 | Trigger |
|---|---|
| NATS ACL enforce 전환 | log-only 1개월 안정 + false positive 0 |
| Tool sandbox enforce 전환 | log-only 1주 안정 |
| MongoDB Vector Search 마이그레이션 | preview → GA |
| Resolution hash sunset | Iter6 dual-write 2개월 안정 |
| Cross-room context aggregation | cross-room 패턴 발견 |
| Parallel fan-out | handoff 평균 hop ≥ 2.0 |
| Web UI for review queue | review 진입 ≥ N/day 1개월 |
| Workflow auto-discovery (AFlow) | propose/tournament 안정화 |
| MCP/A2A 채택 | 외부 ecosystem 연결 필요 |
각 항목은 evidence(metric/운영 사례) 기반으로 plan 작성.
---
## 19. 변경 이력 (commit 별)
main 브랜치의 주요 milestones:
| sha | 메시지 | Iter |
|---|---|---|
| `bc0f8e0` | Add secret coverage and NATS subject ACL | 12 Phase C |
| `d011a28` | Add tool permission sandbox and audit stream | 12 Phase B |
| `21df14c` | Add cost-aware routing and LLM budget tracking | 12 Phase A |
| `211de18` | Add hybrid memory retrieval and escalation history | 11 |
| `1a86bde` | Add memory layers and RAG retrieval | 10 |
| `1d0be56` | Add Phoenix OTLP tracing and LLM span attributes | 9 |
| (이전) | Close multi-agent plan and add router shadow reporting | 8 |
| (이전) | Add Slack alerts and skill-aware monitoring | 7 |
| (이전) | Unify dispatch and add supervisor routing signals | 6 |
| (이전) | Close supervisor review loop and add handoff eval | 5 |
| (이전) | Make eval pipeline skill-aware and validate SKILL manifests | 3 |
각 commit의 detail은 `git log <sha> -p` 또는 dev 브랜치의 `docs/refactor-skill-ssot-iterN.md` 참고.
---
## 끝
이 문서가 outdated되면 last-verified commit을 footer에 명시:
- last-verified: `bc0f8e0` (2026-05-19)
업데이트 책임: 신규 큰 변경(새 skill 추가, 새 memory layer, 새 metric 등) PR 시 함께 갱신.