1330 lines
48 KiB
Markdown
1330 lines
48 KiB
Markdown
# Lolympus Agent — 시스템 아키텍처 및 운영 흐름
|
||
|
||
> Iter1~12 (2026-05-13 ~ 2026-05-19) 완료 시점 기준의 통합 architecture + operations 문서입니다.
|
||
> 신규 기여자/운영자/AI agent의 단일 진입점.
|
||
|
||
---
|
||
|
||
## 0. 한눈에 보기
|
||
|
||
**Lolympus Agent**는 카카오톡 오픈채팅 운영을 자동화하는 **NATS 메시지 드리븐 멀티 에이전트 시스템**이다.
|
||
|
||
- **도메인**: 카카오톡 오픈채팅 가입 안내(onboarding) + 커뮤니티 규칙 모더레이션(moderation)
|
||
- **외부 진입**: NATS subject `lolympus-agent.request`
|
||
- **응답 형태**: NATS reply (단일 publish/요청)
|
||
- **핵심 자산**:
|
||
- Skill SSOT 디렉토리 (`src/skills/{onboarding,moderation,supervisor}/`)
|
||
- 통합 dispatch loop (`src/agent/dispatch.ts`, production + eval 공용)
|
||
- 3-layer memory + Agentic RAG (`src/memory/`)
|
||
- Bounded Self-Escalation Graph (handoff + transition matrix + budget + supervisor fallback)
|
||
- Eval 파이프라인 (capture → run → propose → tournament → promote)
|
||
- 자체 metrics + Phoenix LLM tracing
|
||
- Slack webhook + operations CLI
|
||
|
||
### 권장 시작 순서
|
||
1. **§2 아키텍처 다이어그램** — 전체 그림
|
||
2. **§3 메시지 흐름** — 한 요청이 어떻게 흐르는지
|
||
3. **§4~9** — 각 컴포넌트 깊이 있게
|
||
4. **§16 cheat sheet** — 자주 쓰는 명령
|
||
|
||
---
|
||
|
||
## 1. 기술 스택 요약
|
||
|
||
| 영역 | 선택 | 위치 |
|
||
|---|---|---|
|
||
| 언어 | TypeScript (Node 22+) | `tsconfig.json` |
|
||
| 메시지 | NATS | `src/common/nats.ts` |
|
||
| 캐시/세션 | Garnet (Redis 호환) | `src/common/garnet.ts` |
|
||
| 영속 저장 | MongoDB (mongoose) | `src/room-context/`, `src/memory/mongo-*.ts` |
|
||
| LLM provider | OpenRouter / OpenAI / cliproxy / llama (multi-provider) | `src/llm/llm-client.ts`, `src/llm/routing.ts` |
|
||
| Embedding | OpenRouter free 우선 → OpenAI fallback | `src/memory/embedding-client.ts` |
|
||
| Vector store | MongoDB collection + in-app cosine (Vector Search 마이그레이션 path 예약) | `src/memory/mongo-long-term-memory.ts`, `src/memory/cosine.ts` |
|
||
| BM25 | in-app | `src/memory/bm25.ts` |
|
||
| 관측 | OTel SDK + Phoenix OTLP + 자체 metrics 모듈 | `src/observability/` |
|
||
| 알림 | NATS publish + Slack incoming webhook | `src/observability/alert.ts`, `slack-notifier.ts` |
|
||
| Secret | Infisical → IMDS → env | `src/common/secret-manager.ts` |
|
||
| Test | Vitest | `*.test.ts` |
|
||
| Plan/Eval 데이터 | MongoDB collections + Garnet streams | 각 모듈 |
|
||
|
||
새 의존성 도입 시 §15 환경변수 + KNOWN_SECRET_KEYS 갱신 확인.
|
||
|
||
---
|
||
|
||
## 2. 아키텍처 다이어그램
|
||
|
||
```
|
||
[NATS subject "lolympus-agent.request"]
|
||
│
|
||
▼
|
||
┌─────────────────────────────────┐
|
||
│ Agent.handleRequest │ src/agent/agent.ts (~250 lines)
|
||
│ - runWithContext (AsyncLocal)│
|
||
│ - withSpan (OTel) │
|
||
│ - budget 생성, NATS reply │
|
||
└────────────┬────────────────────┘
|
||
│ dispatch(input, deps)
|
||
▼
|
||
┌─────────────────────────────────┐
|
||
│ dispatch loop │ src/agent/dispatch.ts (~580 lines)
|
||
│ ├─ selectSkillForRequest │ priority + canHandle 필터
|
||
│ ├─ runSkillOnce │ skill.handle() 호출
|
||
│ ├─ normalizeControlTokens │ [HANDOFF:...] 파싱
|
||
│ ├─ applyRuntimeVerifier │ retry/escalate
|
||
│ ├─ applyEscalationTrigger │ confidence/risk → handoff
|
||
│ ├─ prepareHandoff │ TransitionMatrix + Budget 검증
|
||
│ └─ recurse (max depth 4) │ 같은 dispatch 재진입
|
||
└────────────┬────────────────────┘
|
||
│
|
||
┌───────┴───────┐
|
||
▼ ▼
|
||
[Onboarding] [Moderation] [Supervisor]
|
||
├ intents ├ intents ├ capability-prompt (LLM tier0)
|
||
├ prompt-data ├ prompt-data ├ routing-rules (regex fast-path)
|
||
├ tools (5) ├ tools (1) └ review-queue append + alert
|
||
├ runtime/ ├ skill.ts
|
||
└ skill.ts
|
||
|
||
┌─────────────────────────────────┐
|
||
│ MemoryService (Iter10-11) │ src/memory/
|
||
│ ├─ WorkingMemory (조립) │ 매 turn
|
||
│ ├─ SessionMemory (12h Garnet)│ blackboard + conv + summary
|
||
│ └─ LongTermMemory (30d+) │
|
||
│ ├─ UserProfile (Garnet) │
|
||
│ ├─ Knowledge (Mongo+RAG) │ embedding + cosine + BM25 + reranker
|
||
│ └─ EscalationHistory (Mongo)│ user-level review queue rollup
|
||
└─────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────┐
|
||
│ ToolRegistry │ src/tools/
|
||
│ ├─ Tool.requires (sandbox) │
|
||
│ ├─ scoped-context │ least-privileged ctx
|
||
│ └─ audit stream │ lolympus-agent:tool-calls
|
||
└─────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────┐
|
||
│ Observability │ src/observability/
|
||
│ ├─ Pino logger (skill child) │
|
||
│ ├─ OTel tracing → Phoenix │ gen_ai.* + memory.* + handoff.*
|
||
│ ├─ Self-built metrics │ /metrics endpoint (9464)
|
||
│ └─ AlertService │ NATS + Slack webhook
|
||
└─────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────┐
|
||
│ Eval Pipeline │ src/eval/
|
||
│ ├─ Capture (shadow-bus) │ prod traffic 샘플링
|
||
│ ├─ Run (runner) │ golden cases 실행
|
||
│ ├─ Judges (rule/llm/process/ │
|
||
│ │ handoff/memory) │
|
||
│ ├─ Propose (variant 생성) │
|
||
│ ├─ Tournament (A/B) │
|
||
│ └─ Promote (T4→T2) │
|
||
└─────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 3. 메시지 수신 → 응답 흐름
|
||
|
||
### 3.1 진입점 — `Agent.handleRequest`
|
||
|
||
`src/agent/agent.ts`. 흐름:
|
||
|
||
1. **NATS raw 수신**: `natsClient.subscribeRaw(getRequestSubject(), this.handleRequest)` (start time)
|
||
2. **JSON 파싱 + 검증**: `isValidRequest(parsed)` 통과 안 하면 즉시 empty response
|
||
3. **`runWithContext`** 호출 — `src/observability/context.ts`. AsyncLocalStorage로 `RequestContext` 주입:
|
||
- `requestId`, `chatroomId`, `userId`, `skill`, `trigger`
|
||
- `logger` (skill name child binding)
|
||
- `startedAt`, `deadlineAt`
|
||
- `budget`, `handoffDepth`, `handoffChain` (Iter4)
|
||
4. **`withSpan('agent.handle')`** OTel 외부 span 시작
|
||
5. **`dispatch(input, deps)`** 호출 (실제 비즈니스 로직은 dispatch.ts에 위임 — Iter6 추출)
|
||
6. **단일 NATS reply**: `natsClient.respond(msg, response)` (line 144 등) — dispatch 종료 후 정확히 1회
|
||
|
||
### 3.2 dispatch loop — `src/agent/dispatch.ts`
|
||
|
||
핵심 재귀 함수 `dispatch(input, deps)`:
|
||
|
||
```
|
||
인자:
|
||
input: { budget, chain, depth, history, request, skill }
|
||
deps: { selectSkillForRequest, skills, transitionMatrix,
|
||
shouldApplyRuntimeVerifier, getSkillJudgeConfig,
|
||
onHandoff?, onDispatchEnd?, evalPlan? }
|
||
|
||
순서:
|
||
1) runSkillOnce(input) → response # skill.handle()
|
||
2) normalizeControlTokens(response) # [HANDOFF:..] → metadata
|
||
3) applyRuntimeVerifier(input, response) # pass/retry/escalate (Iter6)
|
||
4) applyEscalationTrigger(...) # confidence/risk → handoff
|
||
5) withHandoffMetadata(response, chain, depth) # handoffChain 노출 (Iter5)
|
||
6) readHandoffRequest(response) # metadata.handoffRequest
|
||
└─ null → 최종 응답 반환
|
||
7) prepareHandoff(...) # next skill + packet 합성
|
||
├─ resolveHandoffTarget (TransitionMatrix 검증)
|
||
├─ budget.assertBeforeHandoff (max-hops/loop/cost)
|
||
├─ budget.recordHandoff(target)
|
||
└─ buildHandoffPacket (PII 차단, evidence_refs)
|
||
8) dispatch(next) 재귀
|
||
```
|
||
|
||
### 3.3 dispatch 종료 후
|
||
|
||
- `metadata.handoffChain: string[]` 자동 부착 (eval에서 chain 검증 가능)
|
||
- `metadata.handoffDepth: number`
|
||
- `metadata.handoffRequest`는 마지막 handoff 정보 (없으면 미부착)
|
||
- `metadata.routedBy`: supervisor의 경우 `"regex-fast-path" | "llm" | "rules-fallback"` (Iter12)
|
||
|
||
### 3.4 단일 publish 보장
|
||
|
||
NATS publish 지점은 `handleRequest` 끝 한 곳뿐. 4 hop 재귀라도 사용자는 응답 한 번만 받음.
|
||
|
||
---
|
||
|
||
## 4. Skill 라우팅
|
||
|
||
### 4.1 Skill 등록 (bootstrap)
|
||
|
||
`src/main.ts`:
|
||
|
||
```
|
||
1. SkillRegistry.load("src/skills")
|
||
- 각 폴더의 SKILL.md frontmatter 파싱 → SkillManifest 배열
|
||
2. factories map = { onboarding, moderation, supervisor }
|
||
3. 등록 루프: agent.registerSkill(factory({ ...deps }))
|
||
- deps: garnetClient, llmClient, roomContextService, shadowEvalBus,
|
||
natsClient, memoryService, escalationHistoryService, ...
|
||
4. TransitionMatrix.fromRegistry(skillRegistry) → Agent에 주입
|
||
```
|
||
|
||
### 4.2 SKILL.md frontmatter (capability card)
|
||
|
||
각 `src/skills/<name>/SKILL.md`:
|
||
|
||
```yaml
|
||
name: onboarding
|
||
description: ...
|
||
version: 1
|
||
priority: 70 # canHandle 공동 당첨 시 desc 정렬
|
||
intents: [greeting, ..., age-question]
|
||
triggers: [user-message, new-member, ...]
|
||
tools: [validate-nickname, ...]
|
||
resources:
|
||
nickname-format: ./resources/nickname-format.md
|
||
...
|
||
trustTier: T3 # T1(가장 신뢰) ~ T4(가장 보수적)
|
||
domain: onboarding # Iter4 capability card
|
||
canHandle: [...]
|
||
cannotHandle: [...]
|
||
handoffTargets: [moderation, supervisor]
|
||
riskTier: low | medium | high
|
||
outputContract:
|
||
format: text
|
||
maxChars: 600
|
||
```
|
||
|
||
### 4.3 라우팅 결정 — `selectSkillForRequest`
|
||
|
||
`src/agent/agent.ts:28-37`:
|
||
|
||
```
|
||
1. skills.filter(s => s.canHandle(request))
|
||
- 각 Skill의 canHandle: 보통 request.skill === self.name (정확 매칭)
|
||
2. priority desc 정렬
|
||
3. [0] 반환
|
||
```
|
||
|
||
- onboarding: `request.skill === "onboarding"` (priority 70)
|
||
- moderation: `request.skill === "moderation"` (priority 50)
|
||
- supervisor: `request.skill === "supervisor"` OR fallback (priority 10)
|
||
|
||
### 4.4 TransitionMatrix (handoff 그래프)
|
||
|
||
`src/skills/transition-matrix.ts`. 각 capability card의 `handoffTargets`를 모아 그래프 구성. `supervisor`는 모든 항목의 fallback 자동 포함.
|
||
|
||
```
|
||
isAllowed(from, to) # true/false
|
||
targetsOf(from) # 허용 target 집합
|
||
validateGraph() # 빌드타임 일관성
|
||
```
|
||
|
||
Dispatch loop에서 handoff 시 `transitionMatrix.isAllowed()` 검사. 거부 시 강제 supervisor로 escalation.
|
||
|
||
### 4.5 검증 — `npm run lint:skills`
|
||
|
||
`src/skills/validate-manifests.ts`:
|
||
|
||
- intents 배열 ⊆ 코드 `IntentKind` (manifest-binding.ts에서 derive)
|
||
- tools 배열 == ToolRegistry tool names
|
||
- resources 키들 실제 파일 존재
|
||
- priority 중복 0
|
||
- handoffTargets 항목이 실제 skill name 또는 `supervisor`
|
||
- cannotHandle ∩ canHandle = ∅
|
||
|
||
매 PR 시 CI에서 `lint:skills` 통과 필수.
|
||
|
||
---
|
||
|
||
## 5. Memory 계층
|
||
|
||
자료의 강조점 "context engineering > prompt engineering" + MemoryAgentBench 4축 (accurate retrieval / test-time learning / long-range understanding / selective forgetting) 모두 구현.
|
||
|
||
### 5.1 3-layer 인터페이스 (Iter10)
|
||
|
||
`src/memory/memory-types.ts`:
|
||
|
||
```ts
|
||
WorkingMemory // 매 turn 조립. dispatch loop가 사용
|
||
SessionMemory // 현 대화 세션. Garnet 12h TTL
|
||
LongTermMemory // 영속. Garnet (profile) + MongoDB (knowledge)
|
||
```
|
||
|
||
### 5.2 SessionMemory (Garnet, 12h~24h)
|
||
|
||
| 데이터 | Garnet key | TTL | 소스 |
|
||
|---|---|---|---|
|
||
| Blackboard (skill-local) | `lolympus-agent:bb:{skill}:{chatroom}:{user}` | 12h | `src/skills/onboarding/state/blackboard.ts` |
|
||
| Conversation history | `lolympus-agent:conv:{chatroom}:{user}` | 12h | `src/conversation/conversation-store.ts`, max 20 메시지 |
|
||
| Conversation summary | `lolympus-agent:summary:{chatroom}:{user}` | 24h | `src/conversation/summarizer.ts`, 5+ 메시지 trigger |
|
||
|
||
### 5.3 LongTermMemory (Garnet + MongoDB, 30d+)
|
||
|
||
| 데이터 | 저장소 | TTL |
|
||
|---|---|---|
|
||
| UserProfile (preferredStyle, pastIntents, flaggedIssues, longTermSummary) | Garnet `lolympus-agent:user:profile:{user}` | 30d (env: `AGENT_USER_MEMORY_TTL_DAYS`) |
|
||
| KnowledgeDocuments (RAG) | MongoDB collection `knowledge_documents` | 영속 (eviction policy로 archive) |
|
||
| UserEscalationHistory | MongoDB collection `user_escalation_history` | 영속 |
|
||
|
||
`UserProfile` 필드 (`src/skills/onboarding/state/user-memory.ts`):
|
||
|
||
```ts
|
||
userId: string // anonymizeText() 처리
|
||
firstSeenAt, lastSeenAt: string
|
||
completedOnboardingAt?: string
|
||
totalTurns: number
|
||
pastIntents: Array<{kind, count}> // 최근 10개
|
||
preferredStyle: "warm" | "concise" | "neutral" // 자동 추론
|
||
flaggedIssues: string[] // 최대 10
|
||
longTermSummary?: { text, updatedAt, turnSnapshot } // Iter11
|
||
```
|
||
|
||
### 5.4 WorkingMemory 조립
|
||
|
||
매 요청 시 `message-builder.ts`에서 호출:
|
||
|
||
```ts
|
||
const wm = await memoryService.assembleWorkingMemory({
|
||
skill: "onboarding",
|
||
chatroomId, userId,
|
||
includeBlackboard: true,
|
||
includeProfile: true,
|
||
retrieval: shouldRetrieve(derived)
|
||
? { query: ..., options: { skill, category, topK: 2 } }
|
||
: undefined,
|
||
});
|
||
// wm: { conversationHistory, summary, blackboard, userProfile, retrievedKnowledge }
|
||
```
|
||
|
||
`shouldRetrieve(derived)` (Iter10 Phase D): help-how, help-where, nickname-question, discord-question, age-question 5개 intent만 RAG. greeting/acknowledgement는 skip(비용 절감).
|
||
|
||
### 5.5 LLM prompt 주입 블록
|
||
|
||
`message-builder.ts` + `prompt-data.ts`:
|
||
|
||
```
|
||
[reference] ← retrieval된 knowledge chunk
|
||
[user_context] ← test-time learning snippets (자주 묻는 intent 등)
|
||
[long_term_memory] ← UserProfile.longTermSummary (Iter11)
|
||
[user_escalation_history] ← supervisor capability-prompt에만 (internal only)
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Agentic RAG
|
||
|
||
자료의 "modular > monolithic" + "retrieval은 pre-filter → rerank → compression" 권고 적용.
|
||
|
||
### 6.1 Embedding (`src/memory/embedding-client.ts`)
|
||
|
||
추상화 인터페이스 + 2개 구현 (Iter10):
|
||
|
||
```ts
|
||
interface EmbeddingClient {
|
||
embed(text): Promise<number[]>
|
||
embedBatch(texts): Promise<number[][]>
|
||
model: string
|
||
dimensions: number
|
||
}
|
||
|
||
createEmbeddingClient()
|
||
// env: EMBEDDING_PROVIDER ("openrouter" | "openai")
|
||
// OpenRouter 우선 → 미지원 시 OpenAI fallback
|
||
// env: EMBEDDING_MODEL (예: "text-embedding-3-small")
|
||
// env: EMBEDDING_API_KEY 또는 OPENROUTER_API_KEY
|
||
```
|
||
|
||
### 6.2 Ingest — `npm run ingest:knowledge`
|
||
|
||
`scripts/ingest-knowledge.ts` → `src/memory/knowledge-ingest.ts`:
|
||
|
||
1. `src/skills/{onboarding,moderation}/resources/*.md` 5개 파일 로드
|
||
2. `## ` heading 단위로 split
|
||
3. 각 chunk를 `{ skill, category, text, metadata: { sourceFile, sectionTitle } }`로 build
|
||
4. `embeddingClient.embed(text)` → vector
|
||
5. MongoDB `knowledge_documents` collection upsert (`metadata.sourceFile + sectionTitle` 기준 idempotent)
|
||
6. `embeddingModel` 필드 자동 기록 (마이그레이션 식별용)
|
||
|
||
총 20~50 chunks 예상. 향후 1000+ 시 MongoDB Vector Search 마이그레이션.
|
||
|
||
### 6.3 Retrieval pipeline (Iter10 → Iter11)
|
||
|
||
`src/memory/mongo-long-term-memory.ts:search()`:
|
||
|
||
```
|
||
1. embeddingClient.embed(query) → query vector
|
||
2. MongoDB find with { skill?, category?, metadata.archived: { $ne: true } }
|
||
→ 후보 N개 (env: AGENT_RAG_CANDIDATE_POOL_SIZE, 기본 50)
|
||
3. (옵션) Hybrid (Iter11 Phase A)
|
||
- cosine score (in-app, src/memory/cosine.ts)
|
||
- BM25 score (in-app, src/memory/bm25.ts)
|
||
- 가중 평균 (각 0.5)
|
||
4. top-K 선정 (기본 10)
|
||
5. (옵션) Reranker (Iter11 Phase A)
|
||
- LlmReranker (tier 0, maxTokens 200, jsonMode)
|
||
- 실패 시 원래 순서 (graceful)
|
||
6. top-N 반환 (기본 5, env: AGENT_RAG_RERANKER_TOP_K)
|
||
```
|
||
|
||
각 단계에서 Phoenix span attribute:
|
||
- `memory.retrieval.candidate_count`
|
||
- `memory.retrieval.top_score`
|
||
- `memory.retrieval.duration_ms`
|
||
- `memory.reranker.candidates_count`
|
||
- `memory.reranker.changed_order` (boolean)
|
||
|
||
### 6.4 Selective forgetting (Iter11 Phase C)
|
||
|
||
`src/memory/forgetting-policy.ts` + `src/memory/contradiction-detector.ts`:
|
||
|
||
3가지 규칙:
|
||
- **Relevance**: 30일간 reference 0회 + `referenceCount30d < 3` → archive
|
||
- **Freshness**: `updatedAt` 90일 이상 + 사용 빈도 낮음 → archive
|
||
- **Contradiction**: 같은 `sourceFile + sectionTitle` 그룹 내 텍스트 30%+ 차이 또는 명시적 키워드 → 기존 archive, 새 doc primary (staging 필수)
|
||
|
||
`scripts/memory-eviction-cron.ts` (`npm run memory:evict --dry-run`):
|
||
- archived는 `metadata.archived = true` flag만 (delete 아님). 복구 가능
|
||
- search() 자동으로 archived 제외
|
||
|
||
### 6.5 사용 site
|
||
|
||
**Moderation `lookup-rule` tool** (Iter10 Phase C):
|
||
```
|
||
args: { topic: string, topK?: number }
|
||
→ memoryService.longTerm.search(topic, { skill: "moderation", category: "rule" })
|
||
→ tool result에 chunk text 포함
|
||
→ LLM이 retrieval된 reference로 답변
|
||
```
|
||
|
||
**Onboarding `message-builder.ts`** (Iter10 Phase D):
|
||
- 5개 intent에서만 retrieval
|
||
- `[reference]` block에 chunk 주입
|
||
- LLM이 ground truth 기반 답변
|
||
|
||
---
|
||
|
||
## 7. Tool 실행
|
||
|
||
### 7.1 Tool 인터페이스 (Iter12 Phase B)
|
||
|
||
`src/tools/tool.ts`:
|
||
|
||
```ts
|
||
interface Tool<A, R> {
|
||
name: string
|
||
description: string
|
||
argsSchema: ZodSchema<A>
|
||
resultSchema: ZodSchema<R>
|
||
requires?: readonly ToolPermission[] // Iter12: 권한 declaration
|
||
execute(args: A, ctx: ToolContext): Promise<ToolResult<R>>
|
||
}
|
||
|
||
type ToolPermission =
|
||
| "garnet" | "nats" | "room-context"
|
||
| "user-session" | "tool-registry"
|
||
| "memory-search" | "memory-profile"
|
||
```
|
||
|
||
### 7.2 Scoped Context Injection (Iter12 Phase B)
|
||
|
||
`src/tools/scoped-context.ts`:
|
||
|
||
```
|
||
buildScopedContext(tool.requires ?? [], fullContext) → scopedContext
|
||
```
|
||
|
||
- 권한 없는 필드 접근 시:
|
||
- `AGENT_TOOL_SANDBOX_ENABLED=true`: throw
|
||
- `AGENT_TOOL_SANDBOX_ENABLED=false` (log-only): warn 로그만
|
||
|
||
### 7.3 등록된 tool
|
||
|
||
| Tool | Skill | requires |
|
||
|---|---|---|
|
||
| validate-nickname | onboarding | `[]` (pure) |
|
||
| check-onboarding-progress | onboarding | `["garnet", "user-session"]` |
|
||
| lookup-community-rule | onboarding | `["memory-search"]` |
|
||
| get-room-context | onboarding | `["room-context"]` |
|
||
| verify-discord-nickname | onboarding | `["garnet", "nats", "user-session"]` |
|
||
| lookup-rule | moderation | `["memory-search"]` |
|
||
|
||
### 7.4 Tool Audit (Iter12 Phase B)
|
||
|
||
`src/tools/audit.ts` + Garnet stream `lolympus-agent:tool-calls`:
|
||
|
||
매 tool 호출 후 envelope append:
|
||
```ts
|
||
{
|
||
type: "tool-call",
|
||
toolName, chatroomId, userId,
|
||
argsHash: string, // PII 마스킹된 hash (원본 args 미저장)
|
||
resultSummary, ok, errorKind?,
|
||
durationMs, skill, traceId, calledAt
|
||
}
|
||
```
|
||
|
||
- maxLen 100_000 자동 trim
|
||
- Operations CLI: `npm run tool-audit:list`, `npm run tool-audit:show <id>`
|
||
|
||
### 7.5 외부 통합 (lolympus 도메인)
|
||
|
||
- **Discord**: `verify-discord-nickname` tool이 NATS subject `lolympus.discord.lookup-members`로 외부 microservice에 위임 (HTTP 직접 호출 X)
|
||
- **Kakao**: session에 nickname 정보만 (외부 API 호출 없음, 클라이언트 제공)
|
||
|
||
---
|
||
|
||
## 8. Handoff + Supervisor (Bounded Self-Escalation Graph)
|
||
|
||
자료의 "Bounded Self-Escalation Graph" 8가지 규칙 모두 적용 (Iter4-6).
|
||
|
||
### 8.1 HandoffRequest schema (Iter4)
|
||
|
||
`src/agent/types.ts`:
|
||
|
||
```ts
|
||
interface HandoffRequest {
|
||
targetAgent: string
|
||
reason: HandoffReason // 8가지 enum
|
||
taskSpec: string
|
||
evidenceRefs: string[] // toolCall id 또는 messageId
|
||
confidence: number // 0..1
|
||
risk: "low" | "medium" | "high"
|
||
expectedOutput: string
|
||
userVisibleMessage?: string
|
||
}
|
||
|
||
type HandoffReason =
|
||
| "out-of-scope" | "missing-tool" | "low-confidence"
|
||
| "validation-failed" | "same-agent-revisit"
|
||
| "max-hop" | "high-risk-action" | "policy-risk"
|
||
```
|
||
|
||
### 8.2 [HANDOFF:..] 토큰 파이프라인 (Iter4 Phase D)
|
||
|
||
기존 `[NO_RESPONSE]` 패턴을 확장. 5단계:
|
||
|
||
1. **Prompt** — `prompt-data.ts` persona에 사용법 명시 (capability card 기반 자동 합성)
|
||
2. **Abstention** — `runtime/abstention.ts`의 `AbstentionAction`에 `{kind:"handoff", targetSkill, reason}` variant
|
||
3. **Validator** — `runtime/validator.ts:142+`에서 `[HANDOFF:..]` token pass
|
||
4. **Text generator** — `runtime/text-generator.ts:79+`에서 `{status:"handoff", request}` 반환
|
||
5. **Skill.handle** — `makeEmptyResponse + metadata.handoffRequest` 반환
|
||
|
||
Parser: `src/skills/control-token-parser.ts` (regex `/\[HANDOFF:([a-z0-9_-]+):([a-z-]+)\]/giu`).
|
||
|
||
### 8.3 HandoffPacket (Iter4 Phase E)
|
||
|
||
`src/agent/handoff-packet.ts`. PII 차단 minimal context:
|
||
|
||
```ts
|
||
{
|
||
summary: string // src/conversation/summarizer.ts 결과
|
||
userGoal: string // 240자 제한, PII 제거
|
||
currentState: Record // phase/responseMode/tier만 (userMessage 원문 미포함)
|
||
handoffReason: HandoffReason
|
||
evidence: HandoffEvidence[] // toolCalls[].id
|
||
constraints: string[]
|
||
expectedOutputSchema: string
|
||
sourceChain: string[] // 거쳐간 skill 경로
|
||
}
|
||
```
|
||
|
||
다음 dispatch hop에서 `request.context.handoffPacket`으로 전달. 수신 skill은 `readHandoffPacket(request)` 호출.
|
||
|
||
### 8.4 BudgetTracker (Iter4 Phase F + Iter12 Phase A)
|
||
|
||
`src/agent/budget.ts`:
|
||
|
||
| 한도 | 기본 | env |
|
||
|---|---|---|
|
||
| MAX_HANDOFFS | 4 | `AGENT_MAX_HANDOFFS` |
|
||
| MAX_SAME_AGENT_VISITS | 2 | `AGENT_MAX_SAME_AGENT_VISITS` |
|
||
| MAX_TOTAL_TOOL_CALLS | 20 | `AGENT_MAX_TOOL_CALLS` |
|
||
| MAX_COST_USD | 0.5 | `AGENT_MAX_COST_USD` |
|
||
|
||
`assertBeforeHandoff(target)` verdict: `"ok" | "max-hops" | "same-agent-loop" | "tool-budget" | "cost-budget"`.
|
||
|
||
**Iter12 핵심**: `recordLlmCost(costUsd)` — LlmClient 호출 끝에서 자동 hook (`context.budget?.recordLlmCost(cost)`). snapshot에 `totalLlmCostUsd`, `totalToolCostUsd` 분리 노출.
|
||
|
||
### 8.5 Supervisor (Iter4 Phase G + Iter6 + Iter12 Phase A)
|
||
|
||
`src/skills/supervisor/skill.ts`. **LLM + 룰 기반 hybrid**:
|
||
|
||
```
|
||
1. readHandoffPacket(request)
|
||
2. routeByRegex(packet) → { target, confidence }
|
||
3. confidence ≥ 0.85 (env: SUPERVISOR_FAST_PATH_CONFIDENCE)
|
||
→ LLM skip. metadata.routedBy = "regex-fast-path" ← Iter12 G 패턴
|
||
4. 그 외 → routeByLlm(packet):
|
||
- capability-prompt에 모든 skill capability card 자동 합성
|
||
- + escalation history (Iter11) [user_escalation_history] block
|
||
- tier 0, maxTokens 150, jsonMode
|
||
- { target, confidence, reason } JSON 응답
|
||
5. LLM 실패 → 정규식 fallback. metadata.routedBy = "rules-fallback"
|
||
6. appendSupervisorReview() → review queue + audit
|
||
7. AlertService.sendAlert (NATS + Slack, Iter7)
|
||
8. 응답 텍스트: 안정성 위해 template 유지 (Iter6 결정)
|
||
```
|
||
|
||
### 8.6 Runtime Evaluator (Iter4 Phase H)
|
||
|
||
`SkillJudgeConfig.runtimeVerifier?: (input) => "pass" | "retry" | "escalate"`:
|
||
|
||
- pass → 진행
|
||
- retry → 1회 재호출 (`trigger="retry"`, `MAX_RUNTIME_RETRIES=1`)
|
||
- escalate → supervisor 강제 handoff (`reason="validation-failed"`)
|
||
|
||
Onboarding 기본 verifier: `validateOnboardingResponse`. Moderation: PII/외부 링크 차단 regex.
|
||
|
||
---
|
||
|
||
## 9. Cost-aware Tier Selection (Iter12 Phase A)
|
||
|
||
자료 G 패턴 + OpenInference cost 활용.
|
||
|
||
### 9.1 Tier 구조
|
||
|
||
| Tier | 용도 | 기본 모델 | maxTokens | env |
|
||
|---|---|---|---|---|
|
||
| 0 | 분류, 짧은 응답, supervisor LLM, reranker | gpt-5.3-codex-spark | 384 | `TIER0_MODEL`, `TIER0_MAX_TOKENS` |
|
||
| 1 | planner / responder | gpt-5.4 / gpt-5.3 | 512 | `TIER1_PLANNER_MODEL`, `TIER1_RESPONDER_MODEL` |
|
||
| 2 | critic | gpt-5.4 | 512 | `TIER2_CRITIC_MODEL` |
|
||
|
||
### 9.2 Router 결정 입력 (Iter12 확장)
|
||
|
||
`src/skills/onboarding/runtime/router.ts:decidePreGenerationRouting()`:
|
||
|
||
```ts
|
||
input: {
|
||
classified: { intent, confidence },
|
||
derived,
|
||
classifyConfidence: number, // Iter12
|
||
intentRiskTier?: "low"|"medium"|"high", // Iter12 (capability card)
|
||
userEscalationCount?: number, // Iter12 (escalation history)
|
||
recentLatencyP95Ms?: number // Iter12 (선택)
|
||
}
|
||
```
|
||
|
||
룰:
|
||
- confidence ≥ 0.9 + low-risk → tier 0 (cost 최소)
|
||
- confidence < 0.5 또는 high-risk + escalation > 3 → tier 1 (quality 우선)
|
||
- p95 latency SLA 위반 추세 → tier 0 (degradation 보호)
|
||
|
||
### 9.3 LLM cost 추적 (Iter9 + Iter12)
|
||
|
||
`src/llm/llm-client.ts`:
|
||
|
||
```ts
|
||
const costUsd = estimateChatCost(model, prompt_tokens, completion_tokens);
|
||
metrics.tokenCostUsd.observe(costUsd, { model, skill: getContext()?.skill });
|
||
getContext()?.budget?.recordLlmCost(costUsd); // Iter12
|
||
// Phoenix span: gen_ai.usage.cost_usd = costUsd
|
||
```
|
||
|
||
`estimateChatCost` (`src/llm/cost.ts`): 모델별 input/output rate 테이블.
|
||
|
||
### 9.4 환경 변수 (cost-aware)
|
||
|
||
| env | 기본 | 효과 |
|
||
|---|---|---|
|
||
| `AGENT_COST_AWARE_TIER_ENABLED` | true | confidence 기반 tier 결정 |
|
||
| `AGENT_BUDGET_LLM_COST_ENABLED` | true | LlmClient → Budget 자동 hook |
|
||
| `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | regex 명확 시 LLM skip |
|
||
| `SUPERVISOR_FAST_PATH_CONFIDENCE` | 0.85 | 임계값 |
|
||
|
||
---
|
||
|
||
## 10. Evaluation 파이프라인
|
||
|
||
자료의 "평가 가능한 실행 환경 없이 데모는 의미 없음" 권고. Iter3에서 skill-aware로 전환됨.
|
||
|
||
### 10.1 Golden set 구조 (Iter2 + 후속)
|
||
|
||
```
|
||
src/eval/golden/
|
||
├── onboarding/
|
||
│ ├── cases.jsonl # 활성 golden, ~11개
|
||
│ ├── candidate.jsonl # 자동 수집 후보 (promote 후보)
|
||
│ └── capture.jsonl # 실시간 prod 캡처 buffer
|
||
├── moderation/
|
||
│ └── cases.jsonl # 5개 (Iter2)
|
||
├── handoff/
|
||
│ └── cases.jsonl # 6개 (Iter5)
|
||
└── memory/
|
||
└── cases.jsonl # 6개 (Iter11, 4축 × 1~2개)
|
||
```
|
||
|
||
EvalCase 구조 (`src/eval/types.ts`):
|
||
|
||
```ts
|
||
{
|
||
id, name, notes,
|
||
input: { request, conversationHistory, roomContext, session, blackboard, summary },
|
||
expected: {
|
||
intent?, mustIncludeAny?, mustNotInclude?, responseMode?,
|
||
processExpected?: { expectedTools, expectedApproach, minSteps, toolMode },
|
||
expectedHandoff?: { shouldHandoff, expectedReason, expectedTarget }, // Iter5
|
||
expectedFinalSkill?: string,
|
||
expectedSourceChain?: { mode: "exact"|"prefix"|"contains", chain },
|
||
expectedMemory?: MemoryJudgeExpected // Iter11
|
||
}
|
||
}
|
||
```
|
||
|
||
### 10.2 Runner 흐름 (`src/eval/runner.ts`)
|
||
|
||
```
|
||
1. resolveEvalSetPath(setName)
|
||
"onboarding/cases" → src/eval/golden/onboarding/cases.jsonl
|
||
"golden" → legacy alias for "onboarding/cases"
|
||
2. parseEvalCaseLine() → EvalCase[]
|
||
3. for each case:
|
||
a. SKILL_RUNNER_FACTORIES[skill]({deps}) → Skill instance
|
||
b. handoff/cases set이면 runEvalDispatch (mini-dispatcher, Iter5)
|
||
그 외엔 skill.handle() 직접 호출
|
||
c. judges 실행:
|
||
- rule-judge (mustIncludeAny, mustNotInclude, intent)
|
||
- llm-judge (skill별 평가 축, SkillJudgeConfig)
|
||
- process-judge (tool trace 검증)
|
||
- handoff-judge (chain, reason, target) — handoff set만
|
||
- memory-judge (retrieval/long-range/test-time/forgetting) — memory set만
|
||
d. summary 집계
|
||
4. baseline (eval-reports/baseline.{skill}.json) 비교
|
||
5. eval-reports/<ts>.json 저장
|
||
```
|
||
|
||
### 10.3 Judges (`src/eval/judges/`)
|
||
|
||
| Judge | 책임 | Iter |
|
||
|---|---|---|
|
||
| `rule-judge.ts` | mustIncludeAny / mustNotInclude / intent 매칭 | Iter3 일반화 |
|
||
| `llm-judge.ts` | skill별 평가 축. `SkillJudgeConfig.axes` | Iter3 |
|
||
| `process-judge.ts` | tool trace, skill별 tool registry | Iter3 |
|
||
| `handoff-judge.ts` | dispatch chain 검증 | Iter5 |
|
||
| `memory-judge.ts` | retrieval 정확도, long-range 회상 등 | Iter11 |
|
||
| `attribution.ts` | slice 기반 root cause | (이전) |
|
||
|
||
`SkillJudgeConfig` (`src/eval/judges/skill-config.ts`):
|
||
|
||
```ts
|
||
{
|
||
onboarding: {
|
||
axes: ["friendliness", "accuracy", "concision", "scope"],
|
||
toolRegistryFactory: createOnboardingToolRegistry,
|
||
validator: validateOnboardingResponse,
|
||
promptTemplate, tournamentInstruction,
|
||
runtimeVerifier?: (input) => "pass"|"retry"|"escalate"
|
||
},
|
||
moderation: {
|
||
axes: ["appropriateness", "accuracy", "helpfulness"],
|
||
toolRegistryFactory: createModerationToolRegistry,
|
||
...
|
||
}
|
||
}
|
||
```
|
||
|
||
### 10.4 자동 개선 루프
|
||
|
||
```
|
||
[1] Capture (shadow-eval-bus) — prod 트래픽 sample → shadow envelope
|
||
↓
|
||
[2] Run — golden cases 실행, judges 적용, eval-reports/<ts>.json
|
||
↓
|
||
[3] Propose (eval:propose CLI) — 저점수 slice → LLM이 prompt variant 생성
|
||
→ src/eval/proposals/<variantId>.json
|
||
↓
|
||
[4] Tournament (eval:tournament) — control vs variant A/B, ELO 갱신
|
||
↓
|
||
[5] Governance gates (G1~G4) — trust tier 검증
|
||
↓
|
||
[6] Promote (eval:promote) — golden 후보 → 활성 cases
|
||
```
|
||
|
||
### 10.5 변형 적용 (PROMPT_VARIANT_ID)
|
||
|
||
`src/skills/onboarding/prompts/builder.ts`:
|
||
|
||
- `loadPromptVariantManifest(variantId)` → variant JSON 로드
|
||
- `applyLayerDiff(layers, diff)` — `persona/policy/state/context/examples/tool-policy` 6 layer 중 일부 replace/append/remove
|
||
- env `PROMPT_VARIANT_ID=<id>` → 프로덕션에서 활성
|
||
|
||
### 10.6 Calibration / Drift / Governance
|
||
|
||
- `src/eval/aggregator.ts` — SliceMetrics (intent/phase/tier 차원)
|
||
- `src/eval/drift.ts` — baseline 대비 점수 하락 감지
|
||
- `src/eval/governance/trust.ts` — T1~T4 enum
|
||
- `src/eval/governance/gates.ts` — variant 승격 4 gate (G1~G4)
|
||
- `src/eval/router-shadow-report.ts` (Iter6) — learned vs static routing divergence
|
||
|
||
---
|
||
|
||
## 11. Memory 평가 (Iter11)
|
||
|
||
`src/eval/golden/memory/cases.jsonl` 6 cases:
|
||
|
||
| # | 이름 | 검증 |
|
||
|---|---|---|
|
||
| 1 | retrieval-accurate | "광고 도배" 질문 → moderation lookup-rule retrieval 정확 chunk |
|
||
| 2 | long-range-understanding | 5턴 전 "닉네임 LolName_KR" → 회상 |
|
||
| 3 | test-time-learning | flaggedIssues 3회 누적 → prophylactic guidance |
|
||
| 4 | selective-forgetting | 신규 doc 30% 차이 + "정정" 키워드 → 기존 archive |
|
||
| 5 | escalation-history | 사용자 3회 광고 시도 → supervisor가 moderation 우선 라우팅 |
|
||
| 6 | memory-contradiction | "닉네임 X로 변경" → profile 반영, 다음 응답이 새 닉네임 인지 |
|
||
|
||
`MemoryJudgeResult` (`src/eval/judges/memory-judge.ts`):
|
||
- `expectedRetrievedDocIds`, `expectedRecalledFact`, `expectedProphylacticHint`, `expectedArchivedDocPattern`, `expectedSupervisorTarget`, `expectedContradictionResolved`
|
||
|
||
명령: `npm run eval -- --set=memory/cases`.
|
||
|
||
---
|
||
|
||
## 12. Operations — 운영 루프
|
||
|
||
### 12.1 Review Queue 두 종류
|
||
|
||
| Queue | Garnet stream | 목적 | 진입 |
|
||
|---|---|---|---|
|
||
| Eval Review | `lolympus-agent:eval:review:queue` | 학습용 (저점수/희귀 slice/rule-llm 불일치) | runner가 자동 enqueue |
|
||
| Supervisor Handoff | `lolympus-agent:review-queue` | 운영용 (escalation case 사람 검토) | supervisor.skill.ts |
|
||
|
||
### 12.2 Operations CLI (Iter5 + Iter12)
|
||
|
||
`src/operations/cli.ts`:
|
||
|
||
```bash
|
||
# Supervisor handoff queue
|
||
npm run review:list -- --limit=20 --since=24h
|
||
npm run review:show -- <id> [--history] # --history는 Iter6 audit log
|
||
npm run review:resolve -- <id> --action=manual-reply --note="..."
|
||
npm run review:stats -- --since=7d
|
||
|
||
# Tool audit (Iter12)
|
||
npm run tool-audit:list -- --skill=onboarding --since=1h --limit=50
|
||
npm run tool-audit:show -- <stream-id>
|
||
```
|
||
|
||
### 12.3 Resolution audit log (Iter6)
|
||
|
||
이중 저장:
|
||
- **Hash** `lolympus-agent:review-resolved:<id>` — latest state (덮어쓰기, 7d TTL)
|
||
- **Stream** `lolympus-agent:review-resolutions` — append-only (운영자 변경 이력)
|
||
|
||
`getResolutionHistory(garnet, id)` — 사용자별 resolution 변경 이력 조회 (Iter11 escalation history 입력).
|
||
|
||
### 12.4 Alert flow (Iter7)
|
||
|
||
`src/observability/alert.ts:AlertService.sendAlert(payload)`:
|
||
|
||
```
|
||
1. Garnet dedupe key 검사 (5분 throttle)
|
||
key: lolympus-agent:alert:dedupe:<kind>:<skill>
|
||
2. NATS publish → AGENT_ALERT_SUBJECT
|
||
3. Slack webhook POST (env: SLACK_ALERT_WEBHOOK_URL)
|
||
- fire-and-forget, AbortSignal.timeout 3초
|
||
- block kit message (severity emoji + 필드)
|
||
4. logger.warn (구조화 로그)
|
||
```
|
||
|
||
AlertPayload:
|
||
```ts
|
||
{
|
||
kind: "archive-stalled"|"endpoint-down"|"eval-drift"|"eval-score-drop"
|
||
|"latency"|"llm-error-rate"|"supervisor-handoff",
|
||
metric, severity, skill?, threshold, value, window, timestamp, ...
|
||
}
|
||
```
|
||
|
||
### 12.5 Slack message 포맷 (Iter7)
|
||
|
||
`src/observability/slack-notifier.ts:formatSlackMessage()`:
|
||
- severity emoji (`:information_source:`/`:warning:`/`:rotating_light:`)
|
||
- Section + fields (metric / value vs threshold / window / skill)
|
||
- suggestedAction (있다면)
|
||
|
||
---
|
||
|
||
## 13. Observability
|
||
|
||
### 13.1 Logger (`src/observability/logger.ts`)
|
||
|
||
- Pino + AsyncLocalStorage
|
||
- `getLogger()` → context의 child logger 자동 (skill, requestId, chatroomId, userId)
|
||
- redact: `authorization`, `env.<secret>`, `secrets.<secret>`, `request.<secret>` (KNOWN_SECRET_KEYS 확장됨, Iter12)
|
||
|
||
### 13.2 Tracing (Iter9)
|
||
|
||
`src/observability/tracing.ts`:
|
||
- OTel NodeSDK + auto-instrumentation
|
||
- Exporter: env `PHOENIX_OTLP_ENDPOINT` 설정 시 OTLP HTTP, 미설정 시 ConsoleSpanExporter (또는 OFF, env: `OTEL_EXPORTER`)
|
||
- Sampling: `OTEL_SAMPLING_RATIO` (prod 0.1, dev 1.0)
|
||
|
||
`src/observability/spans.ts` helper:
|
||
- `withSkillSpan(skill, attrs, fn)` — `skill.<name>.handle`
|
||
- `withSpan(name, attrs, fn)` — generic + auto-error
|
||
- `withClassifierSpan`, `withToolSpan`, `withLlmSpan`
|
||
|
||
### 13.3 LLM Span 속성 (Iter9 OpenInference)
|
||
|
||
`src/llm/llm-client.ts`:
|
||
```
|
||
gen_ai.request.model
|
||
gen_ai.request.temperature
|
||
gen_ai.request.max_tokens
|
||
gen_ai.system (provider name)
|
||
gen_ai.response.id
|
||
gen_ai.response.finish_reasons[]
|
||
gen_ai.usage.input_tokens
|
||
gen_ai.usage.output_tokens
|
||
gen_ai.usage.cost_usd ← Iter9, Iter12에서 routing 활용
|
||
llm.token_count.prompt ← Phoenix 친화 alias
|
||
llm.token_count.completion
|
||
gen_ai.prompt.0.role/content ← optional (PHOENIX_LOG_PROMPTS)
|
||
gen_ai.completion.0.content ← optional + PII 마스킹
|
||
```
|
||
|
||
Memory 관련 (Iter10-11):
|
||
- `memory.retrieval.candidate_count`
|
||
- `memory.retrieval.top_score`
|
||
- `memory.retrieval.duration_ms`
|
||
- `memory.reranker.changed_order`
|
||
|
||
Handoff 관련:
|
||
- `agent.handoff_depth`
|
||
- `agent.handoff_chain`
|
||
- `supervisor.routed_by`
|
||
|
||
### 13.4 Metrics (Iter4 Phase I)
|
||
|
||
자체 구현 (`src/observability/metrics.ts`). `/metrics` HTTP endpoint (`metrics-server.ts`, port 기본 9464). 10 counter/histogram:
|
||
|
||
| Metric | 타입 | Labels |
|
||
|---|---|---|
|
||
| `agent_handoff_total` | counter | from, to, reason |
|
||
| `agent_handoff_loop_total` | counter | from |
|
||
| `agent_handoff_count` | histogram (0..4) | — |
|
||
| `agent_human_override_total` | counter | kind |
|
||
| `agent_missed_escalation_total` | counter | — |
|
||
| `agent_routing_accuracy_total` | counter | result |
|
||
| `agent_unnecessary_escalation_total` | counter | — |
|
||
| `agent_validator_fail_total` | counter | skill, stage |
|
||
| `agent_token_cost_usd` | histogram | model, skill |
|
||
| `agent_time_to_resolution_ms` | histogram | skill |
|
||
|
||
`/healthz` endpoint: 200 "ok" (Iter7).
|
||
|
||
### 13.5 Alert threshold (skill별, Iter7)
|
||
|
||
`src/config.ts`:
|
||
```
|
||
getAlertThresholdP95LatencyMs(skill?: string)
|
||
getAlertThresholdEvalScoreDropPct(skill?: string)
|
||
getAlertThresholdLlmErrorRate(skill?: string)
|
||
getAlertThresholdDriftPct(skill?: string)
|
||
```
|
||
|
||
env 패턴: `ALERT_THRESHOLD_<METRIC>_<SKILL_UPPER>` 우선 → `ALERT_THRESHOLD_<METRIC>` fallback.
|
||
|
||
예: `ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING=5000` 또는 글로벌 `ALERT_THRESHOLD_P95_LATENCY_MS=3000`.
|
||
|
||
---
|
||
|
||
## 14. 보안
|
||
|
||
### 14.1 Secret Manager (`src/common/secret-manager.ts`)
|
||
|
||
계층:
|
||
1. **Infisical** (외부 secret manager, `INFISICAL_SITE_URL`)
|
||
2. **AWS IMDS** (EC2 메타데이터 fallback)
|
||
3. **환경변수** (마지막 fallback)
|
||
|
||
`KNOWN_SECRET_KEYS` (`src/common/known-secret-keys.ts`, Iter12 확장):
|
||
```
|
||
INFISICAL_*, NATS_*, GARNET_*, MONGODB_*, SLACK_ALERT_WEBHOOK_URL,
|
||
OPENROUTER_API_KEY, AGENT_LLM_API_KEY, OPENAI_API_KEY,
|
||
EMBEDDING_API_KEY, PHOENIX_API_KEY
|
||
```
|
||
|
||
`getSecret(key)`: KNOWN_SECRET_KEYS 검증. `getRawSecret(key)`: 임의 키 (운영 도구용).
|
||
|
||
### 14.2 NATS Subject ACL (Iter12 Phase C)
|
||
|
||
`src/common/nats-skill-acl.ts:DEFAULT_NATS_ACL`:
|
||
```ts
|
||
[
|
||
{
|
||
skill: "onboarding",
|
||
allowedPublishSubjects: ["lolympus.discord.lookup-members"],
|
||
allowedSubscribeSubjects: []
|
||
},
|
||
{ skill: "moderation", allowedPublishSubjects: [], ... },
|
||
{ skill: "supervisor", allowedPublishSubjects: ["lolympus.agent.alert"], ... }
|
||
]
|
||
```
|
||
|
||
`src/common/nats.ts`의 publish wrapper:
|
||
- `getNatsAclMode()`:
|
||
- `"off"`: 검사 skip
|
||
- `"log-only"` (기본): warn 로그, 정상 publish
|
||
- `"enforce"`: throw + alert
|
||
|
||
운영 흐름: log-only 1개월 → false positive 없으면 enforce 전환 (Iter13+ task).
|
||
|
||
### 14.3 PII 차단
|
||
|
||
- **HandoffPacket**: `userMessage` 원문 포함 X. `userGoal`은 240자 + anonymize
|
||
- **Memory user-profile**: `anonymizeText()` 처리
|
||
- **Tool audit**: `argsHash`만 저장, 원본 args X
|
||
- **Phoenix prompt content**: `PHOENIX_LOG_PROMPTS=false` 기본
|
||
- **escalation history**: supervisor system prompt 내부만, 응답 텍스트에 노출 금지
|
||
|
||
---
|
||
|
||
## 15. 환경 변수 + 배포
|
||
|
||
### 15.1 인프라
|
||
|
||
| env | 용도 |
|
||
|---|---|
|
||
| `NATS_ENDPOINT`, `NATS_USER`, `NATS_PASS` | NATS 연결 |
|
||
| `GARNET_ENDPOINT`, `GARNET_USERID`, `GARNET_PASSWORD` | Garnet 연결 |
|
||
| `MONGODB_CONNECTION_STRING`, `MONGODB_PROJECT_ID` | MongoDB (optional) |
|
||
| `INFISICAL_CLIENT_ID`, `INFISICAL_CLIENT_SECRET` | Secret manager |
|
||
|
||
### 15.2 모델
|
||
|
||
| env | 기본 | 용도 |
|
||
|---|---|---|
|
||
| `TIER0_MODEL` | gpt-5.3-codex-spark | 분류, supervisor LLM, reranker |
|
||
| `TIER1_PLANNER_MODEL`, `TIER1_RESPONDER_MODEL` | gpt-5.4, gpt-5.3 | 메인 응답 |
|
||
| `TIER2_CRITIC_MODEL` | gpt-5.4 | critic |
|
||
| `TIER0_MAX_TOKENS` | 384 | |
|
||
| `EMBEDDING_PROVIDER` | openrouter | OpenRouter→OpenAI fallback |
|
||
| `EMBEDDING_MODEL` | text-embedding-3-small | |
|
||
| `EMBEDDING_API_KEY` | — | (또는 OPENROUTER/OPENAI 키 재사용) |
|
||
|
||
### 15.3 기능 flag (Iter별)
|
||
|
||
| env | 기본 | Iter |
|
||
|---|---|---|
|
||
| `AGENT_ROUTER_MODE` | static | Iter3 |
|
||
| `AGENT_RAG_ENABLED` | true | Iter10 |
|
||
| `AGENT_RAG_HYBRID_ENABLED` | true | Iter11 |
|
||
| `AGENT_RAG_RERANKER_ENABLED` | true | Iter11 |
|
||
| `AGENT_RAG_CANDIDATE_POOL_SIZE` | 50 | Iter11 |
|
||
| `AGENT_RAG_RERANKER_TOP_K` | 5 | Iter11 |
|
||
| `AGENT_LONG_TERM_SUMMARY_ENABLED` | true | Iter11 |
|
||
| `AGENT_MEMORY_EVICTION_ENABLED` | false (cron만) | Iter11 |
|
||
| `AGENT_COST_AWARE_TIER_ENABLED` | true | Iter12 |
|
||
| `AGENT_BUDGET_LLM_COST_ENABLED` | true | Iter12 |
|
||
| `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | Iter12 |
|
||
| `AGENT_TOOL_SANDBOX_ENABLED` | false (1주 검증 후 true) | Iter12 |
|
||
| `AGENT_NATS_ACL_MODE` | log-only | Iter12 |
|
||
| `AGENT_SUPERVISOR_LLM_ENABLED` | true | Iter6 |
|
||
| `AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED` | false (template 우선) | Iter8 |
|
||
|
||
### 15.4 한도 (Budget)
|
||
|
||
| env | 기본 |
|
||
|---|---|
|
||
| `AGENT_MAX_HANDOFFS` | 4 |
|
||
| `AGENT_MAX_SAME_AGENT_VISITS` | 2 |
|
||
| `AGENT_MAX_TOOL_CALLS` | 20 |
|
||
| `AGENT_MAX_COST_USD` | 0.5 |
|
||
|
||
### 15.5 임계값 (Alert)
|
||
|
||
| env (skill suffix optional) | 기본 |
|
||
|---|---|
|
||
| `ALERT_THRESHOLD_P95_LATENCY_MS` | 3000 |
|
||
| `ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT` | 10 |
|
||
| `ALERT_THRESHOLD_LLM_ERROR_RATE` | 0.1 |
|
||
| `ALERT_THRESHOLD_DRIFT_PCT` | 15 |
|
||
| `ALERT_RATE_LIMIT_SEC` | 300 |
|
||
|
||
### 15.6 Observability
|
||
|
||
| env | 기본 |
|
||
|---|---|
|
||
| `PHOENIX_OTLP_ENDPOINT` | unset (console fallback) |
|
||
| `PHOENIX_LOG_PROMPTS` | false |
|
||
| `SLACK_ALERT_WEBHOOK_URL` | unset (Slack 비활성) |
|
||
| `SLACK_WEBHOOK_TIMEOUT_MS` | 3000 |
|
||
| `METRICS_PORT` | 9464 |
|
||
| `OTEL_SAMPLING_RATIO` | 0.1 (prod) |
|
||
|
||
### 15.7 Bootstrap (`src/main.ts`)
|
||
|
||
순서:
|
||
```
|
||
1. initLogger()
|
||
2. initTracing() (Phoenix 또는 console)
|
||
3. startMetricsServer() (port 9464)
|
||
4. SecretManager.load (Infisical + IMDS)
|
||
5. NatsClient.connect
|
||
6. GarnetClient.connect
|
||
7. initMongoDB (optional)
|
||
8. SkillRegistry.load("src/skills")
|
||
9. TransitionMatrix.fromRegistry
|
||
10. ConversationStore + ShadowEvalBus
|
||
11. memoryService, escalationHistoryService, alertService (Slack notifier)
|
||
12. factories map { onboarding, moderation, supervisor }
|
||
13. for manifest in registry: agent.registerSkill(factory(deps))
|
||
14. agent.start() (NATS subscribe)
|
||
15. shutdown hooks
|
||
```
|
||
|
||
---
|
||
|
||
## 16. 도메인 데이터 (Lolympus 카카오톡 오픈채팅)
|
||
|
||
### 16.1 Onboarding (`src/skills/onboarding/`)
|
||
|
||
**Capability card**:
|
||
- priority: 70, trustTier: T3, riskTier: low
|
||
- domain: onboarding
|
||
- handoffTargets: [moderation, supervisor]
|
||
|
||
**Intent** (`intents.ts`, 12개):
|
||
- greeting, acknowledgement
|
||
- status-question
|
||
- help-how, help-where
|
||
- nickname-question, photo-question
|
||
- discord-question, discord-verify-question
|
||
- age-question
|
||
- chitchat, off-topic-to-other, unknown
|
||
|
||
**Tools** (5개): validate-nickname, check-onboarding-progress, lookup-community-rule, get-room-context, verify-discord-nickname
|
||
|
||
**Resources** (4개 .md): nickname-format, heart-rule, discord-format, age-policy
|
||
|
||
**핵심 로직**:
|
||
- `OnboardingIntentClassifier` (3-stage: trigger → regex → LLM)
|
||
- `deriveState(session, trigger, intent)` → phase + responseMode + tier
|
||
- `runtime/`: orchestrator, router, validator, critic, tool-plan-runner, message-builder, text-generator, abstention 등 ~10개 모듈
|
||
|
||
### 16.2 Moderation (`src/skills/moderation/`)
|
||
|
||
**Capability card**:
|
||
- priority: 50, trustTier: T4, riskTier: medium
|
||
- domain: moderation
|
||
- handoffTargets: [onboarding, supervisor]
|
||
|
||
**Intent** (5개): greeting, rule-question, violation-report, appeal, off-topic
|
||
|
||
**Tools** (1개): lookup-rule (Iter10에서 RAG 기반으로 전환)
|
||
|
||
**Resources** (1개 .md): community-rules (욕설, 광고, 도배, 신고 절차, 제재, 이의제기, 미성년자, 개인정보 등 섹션)
|
||
|
||
### 16.3 Supervisor (`src/skills/supervisor/`)
|
||
|
||
**Capability card**:
|
||
- priority: 10 (최저, fallback 전용), riskTier: medium
|
||
- domain: meta
|
||
- canHandle: ["*"]
|
||
- handoffTargets: [onboarding, moderation]
|
||
|
||
**핵심**:
|
||
- `capability-prompt.ts` — 모든 skill의 capability card를 system prompt로 자동 합성
|
||
- `routing-rules.ts` — regex 기반 키워드 매칭 (Iter12 fast-path)
|
||
- `review-queue.ts` — appendSupervisorReview + getResolutionHistory + 도메인 escalation history 연동
|
||
- 응답 텍스트: 안정성 위해 template ("운영자에게 전달했어요")
|
||
|
||
### 16.4 User Escalation History (Iter11 — lolympus 차별점)
|
||
|
||
`user_escalation_history` collection (MongoDB):
|
||
|
||
```ts
|
||
{
|
||
userId, chatroomId,
|
||
escalations: [{
|
||
queuedAt, reason, target,
|
||
resolution: { resolvedAt, action, note },
|
||
packetSummary, sourceChain
|
||
}],
|
||
totalEscalations, lastEscalationAt,
|
||
topReasons: [{ reason, count }]
|
||
}
|
||
```
|
||
|
||
Supervisor가 매 escalation 시 history 조회 → capability prompt의 `[user_escalation_history]` 블록 주입:
|
||
> "이 사용자는 광고 시도 3회 (out-of-scope) → 다음 위반 시 즉시 차단 권장"
|
||
> "닉네임 검증 5회 실패 → 친절한 step-by-step 우선"
|
||
|
||
---
|
||
|
||
## 17. 명령어 cheat sheet
|
||
|
||
### 17.1 개발
|
||
|
||
```bash
|
||
npm run typecheck # TypeScript 검증
|
||
npm run test # Vitest 전체
|
||
npm run test -- --reporter=verbose # 상세
|
||
npm run lint # ESLint
|
||
npm run lint:skills # SKILL.md ↔ 코드 일관성
|
||
npm run build # tsc + tsc-alias
|
||
npm run dev # 로컬 부트스트랩
|
||
```
|
||
|
||
### 17.2 Evaluation
|
||
|
||
```bash
|
||
# Run
|
||
npm run eval -- --set=onboarding/cases
|
||
npm run eval -- --set=moderation/cases
|
||
npm run eval:handoff # = --set=handoff/cases
|
||
npm run eval -- --set=memory/cases # Iter11
|
||
|
||
# Baseline / Report
|
||
npm run eval:baseline
|
||
npm run eval:report
|
||
|
||
# 자동 개선 loop
|
||
npm run eval:capture
|
||
npm run eval:propose -- --slice="tier=0,intent=help-how,phase=initial,skill=onboarding"
|
||
npm run eval:tournament -- --skill=onboarding
|
||
npm run eval:promote
|
||
npm run eval:calibrate -- --check
|
||
npm run eval:router-shadow-report -- --since=7d # Iter6
|
||
```
|
||
|
||
### 17.3 Operations (Iter5 + Iter12)
|
||
|
||
```bash
|
||
# Supervisor review queue
|
||
npm run review:list
|
||
npm run review:show -- <id>
|
||
npm run review:show -- <id> --history # Iter6 resolution audit
|
||
npm run review:resolve -- <id> --action=manual-reply --note="..."
|
||
npm run review:stats
|
||
|
||
# Tool audit
|
||
npm run tool-audit:list
|
||
npm run tool-audit:show -- <id>
|
||
```
|
||
|
||
### 17.4 Memory (Iter10-11)
|
||
|
||
```bash
|
||
npm run ingest:knowledge # resources/*.md → MongoDB embedding
|
||
npm run memory:evict -- --dry-run --since=30d
|
||
npm run memory:evict -- --since=30d # 실제 archive
|
||
```
|
||
|
||
### 17.5 디버깅
|
||
|
||
```bash
|
||
PHOENIX_OTLP_ENDPOINT=http://localhost:6006/v1/traces npm run dev
|
||
curl http://localhost:9464/metrics | grep agent_
|
||
curl http://localhost:9464/healthz
|
||
|
||
# Specific log level
|
||
LOG_LEVEL=debug npm run dev
|
||
|
||
# RAG/sandbox 비활성으로 회귀 확인
|
||
AGENT_RAG_ENABLED=false npm run eval -- --set=onboarding/cases
|
||
AGENT_TOOL_SANDBOX_ENABLED=false npm test
|
||
```
|
||
|
||
---
|
||
|
||
## 18. 참고 자료
|
||
|
||
### 18.1 Iter별 plan 문서
|
||
|
||
`docs/`에 dev 브랜치 보존 (main에는 미포함):
|
||
|
||
| 문서 | 내용 |
|
||
|---|---|
|
||
| `refactor-skill-ssot.md` | Iter1 plan |
|
||
| `refactor-skill-ssot-iter2~8.md` | Iter2~8 각 plan |
|
||
| `optimization-roadmap.md` | Iter8 closing 시 trigger 정의 |
|
||
| `refactor-skill-ssot-iter10~12.md` | Iter10~12 plan |
|
||
| `optimization-roadmap-memory-ops.md` | Iter9~12 통합 로드맵 |
|
||
| `service-evaluation-operations.md` | 운영자 매뉴얼 |
|
||
| `최신기술동향.md`, `멀티에이전트동향.md` | 자료 reference |
|
||
|
||
### 18.2 외부 자료
|
||
|
||
- **MemoryAgentBench** — accurate retrieval, test-time learning, long-range understanding, selective forgetting
|
||
- **Bounded Self-Escalation Graph** — Microsoft orchestration guide
|
||
- **OpenInference semantic conventions** — `gen_ai.*`, `llm.*` span attributes
|
||
- **MCP / A2A** — 향후 채택 후보 (Iter13+ trigger 기반)
|
||
|
||
### 18.3 다음 단계 trigger (Iter13+)
|
||
|
||
`docs/optimization-roadmap.md`의 trigger 발동 시 개별 plan으로 시작:
|
||
|
||
| 항목 | Trigger |
|
||
|---|---|
|
||
| NATS ACL enforce 전환 | log-only 1개월 안정 + false positive 0 |
|
||
| Tool sandbox enforce 전환 | log-only 1주 안정 |
|
||
| MongoDB Vector Search 마이그레이션 | preview → GA |
|
||
| Resolution hash sunset | Iter6 dual-write 2개월 안정 |
|
||
| Cross-room context aggregation | cross-room 패턴 발견 |
|
||
| Parallel fan-out | handoff 평균 hop ≥ 2.0 |
|
||
| Web UI for review queue | review 진입 ≥ N/day 1개월 |
|
||
| Workflow auto-discovery (AFlow) | propose/tournament 안정화 |
|
||
| MCP/A2A 채택 | 외부 ecosystem 연결 필요 |
|
||
|
||
각 항목은 evidence(metric/운영 사례) 기반으로 plan 작성.
|
||
|
||
---
|
||
|
||
## 19. 변경 이력 (commit 별)
|
||
|
||
main 브랜치의 주요 milestones:
|
||
|
||
| sha | 메시지 | Iter |
|
||
|---|---|---|
|
||
| `bc0f8e0` | Add secret coverage and NATS subject ACL | 12 Phase C |
|
||
| `d011a28` | Add tool permission sandbox and audit stream | 12 Phase B |
|
||
| `21df14c` | Add cost-aware routing and LLM budget tracking | 12 Phase A |
|
||
| `211de18` | Add hybrid memory retrieval and escalation history | 11 |
|
||
| `1a86bde` | Add memory layers and RAG retrieval | 10 |
|
||
| `1d0be56` | Add Phoenix OTLP tracing and LLM span attributes | 9 |
|
||
| (이전) | Close multi-agent plan and add router shadow reporting | 8 |
|
||
| (이전) | Add Slack alerts and skill-aware monitoring | 7 |
|
||
| (이전) | Unify dispatch and add supervisor routing signals | 6 |
|
||
| (이전) | Close supervisor review loop and add handoff eval | 5 |
|
||
| (이전) | Make eval pipeline skill-aware and validate SKILL manifests | 3 |
|
||
|
||
각 commit의 detail은 `git log <sha> -p` 또는 dev 브랜치의 `docs/refactor-skill-ssot-iterN.md` 참고.
|
||
|
||
---
|
||
|
||
## 끝
|
||
|
||
이 문서가 outdated되면 last-verified commit을 footer에 명시:
|
||
- last-verified: `bc0f8e0` (2026-05-19)
|
||
|
||
업데이트 책임: 신규 큰 변경(새 skill 추가, 새 memory layer, 새 metric 등) PR 시 함께 갱신.
|