48 KiB
Lolympus Agent — 시스템 아키텍처 및 운영 흐름
Iter1~12 (2026-05-13 ~ 2026-05-19) 완료 시점 기준의 통합 architecture + operations 문서입니다. 신규 기여자/운영자/AI agent의 단일 진입점.
0. 한눈에 보기
Lolympus Agent는 카카오톡 오픈채팅 운영을 자동화하는 NATS 메시지 드리븐 멀티 에이전트 시스템이다.
- 도메인: 카카오톡 오픈채팅 가입 안내(onboarding) + 커뮤니티 규칙 모더레이션(moderation)
- 외부 진입: NATS subject
lolympus-agent.request - 응답 형태: NATS reply (단일 publish/요청)
- 핵심 자산:
- Skill SSOT 디렉토리 (
src/skills/{onboarding,moderation,supervisor}/) - 통합 dispatch loop (
src/agent/dispatch.ts, production + eval 공용) - 3-layer memory + Agentic RAG (
src/memory/) - Bounded Self-Escalation Graph (handoff + transition matrix + budget + supervisor fallback)
- Eval 파이프라인 (capture → run → propose → tournament → promote)
- 자체 metrics + Phoenix LLM tracing
- Slack webhook + operations CLI
- Skill SSOT 디렉토리 (
권장 시작 순서
- §2 아키텍처 다이어그램 — 전체 그림
- §3 메시지 흐름 — 한 요청이 어떻게 흐르는지
- §4~9 — 각 컴포넌트 깊이 있게
- §16 cheat sheet — 자주 쓰는 명령
1. 기술 스택 요약
| 영역 | 선택 | 위치 |
|---|---|---|
| 언어 | TypeScript (Node 22+) | tsconfig.json |
| 메시지 | NATS | src/common/nats.ts |
| 캐시/세션 | Garnet (Redis 호환) | src/common/garnet.ts |
| 영속 저장 | MongoDB (mongoose) | src/room-context/, src/memory/mongo-*.ts |
| LLM provider | OpenRouter / OpenAI / cliproxy / llama (multi-provider) | src/llm/llm-client.ts, src/llm/routing.ts |
| Embedding | OpenRouter free 우선 → OpenAI fallback | src/memory/embedding-client.ts |
| Vector store | MongoDB collection + in-app cosine (Vector Search 마이그레이션 path 예약) | src/memory/mongo-long-term-memory.ts, src/memory/cosine.ts |
| BM25 | in-app | src/memory/bm25.ts |
| 관측 | OTel SDK + Phoenix OTLP + 자체 metrics 모듈 | src/observability/ |
| 알림 | NATS publish + Slack incoming webhook | src/observability/alert.ts, slack-notifier.ts |
| Secret | Infisical → IMDS → env | src/common/secret-manager.ts |
| Test | Vitest | *.test.ts |
| Plan/Eval 데이터 | MongoDB collections + Garnet streams | 각 모듈 |
새 의존성 도입 시 §15 환경변수 + KNOWN_SECRET_KEYS 갱신 확인.
2. 아키텍처 다이어그램
[NATS subject "lolympus-agent.request"]
│
▼
┌─────────────────────────────────┐
│ Agent.handleRequest │ src/agent/agent.ts (~250 lines)
│ - runWithContext (AsyncLocal)│
│ - withSpan (OTel) │
│ - budget 생성, NATS reply │
└────────────┬────────────────────┘
│ dispatch(input, deps)
▼
┌─────────────────────────────────┐
│ dispatch loop │ src/agent/dispatch.ts (~580 lines)
│ ├─ selectSkillForRequest │ priority + canHandle 필터
│ ├─ runSkillOnce │ skill.handle() 호출
│ ├─ normalizeControlTokens │ [HANDOFF:...] 파싱
│ ├─ applyRuntimeVerifier │ retry/escalate
│ ├─ applyEscalationTrigger │ confidence/risk → handoff
│ ├─ prepareHandoff │ TransitionMatrix + Budget 검증
│ └─ recurse (max depth 4) │ 같은 dispatch 재진입
└────────────┬────────────────────┘
│
┌───────┴───────┐
▼ ▼
[Onboarding] [Moderation] [Supervisor]
├ intents ├ intents ├ capability-prompt (LLM tier0)
├ prompt-data ├ prompt-data ├ routing-rules (regex fast-path)
├ tools (5) ├ tools (1) └ review-queue append + alert
├ runtime/ ├ skill.ts
└ skill.ts
┌─────────────────────────────────┐
│ MemoryService (Iter10-11) │ src/memory/
│ ├─ WorkingMemory (조립) │ 매 turn
│ ├─ SessionMemory (12h Garnet)│ blackboard + conv + summary
│ └─ LongTermMemory (30d+) │
│ ├─ UserProfile (Garnet) │
│ ├─ Knowledge (Mongo+RAG) │ embedding + cosine + BM25 + reranker
│ └─ EscalationHistory (Mongo)│ user-level review queue rollup
└─────────────────────────────────┘
┌─────────────────────────────────┐
│ ToolRegistry │ src/tools/
│ ├─ Tool.requires (sandbox) │
│ ├─ scoped-context │ least-privileged ctx
│ └─ audit stream │ lolympus-agent:tool-calls
└─────────────────────────────────┘
┌─────────────────────────────────┐
│ Observability │ src/observability/
│ ├─ Pino logger (skill child) │
│ ├─ OTel tracing → Phoenix │ gen_ai.* + memory.* + handoff.*
│ ├─ Self-built metrics │ /metrics endpoint (9464)
│ └─ AlertService │ NATS + Slack webhook
└─────────────────────────────────┘
┌─────────────────────────────────┐
│ Eval Pipeline │ src/eval/
│ ├─ Capture (shadow-bus) │ prod traffic 샘플링
│ ├─ Run (runner) │ golden cases 실행
│ ├─ Judges (rule/llm/process/ │
│ │ handoff/memory) │
│ ├─ Propose (variant 생성) │
│ ├─ Tournament (A/B) │
│ └─ Promote (T4→T2) │
└─────────────────────────────────┘
3. 메시지 수신 → 응답 흐름
3.1 진입점 — Agent.handleRequest
src/agent/agent.ts. 흐름:
- NATS raw 수신:
natsClient.subscribeRaw(getRequestSubject(), this.handleRequest)(start time) - JSON 파싱 + 검증:
isValidRequest(parsed)통과 안 하면 즉시 empty response runWithContext호출 —src/observability/context.ts. AsyncLocalStorage로RequestContext주입:requestId,chatroomId,userId,skill,triggerlogger(skill name child binding)startedAt,deadlineAtbudget,handoffDepth,handoffChain(Iter4)
withSpan('agent.handle')OTel 외부 span 시작dispatch(input, deps)호출 (실제 비즈니스 로직은 dispatch.ts에 위임 — Iter6 추출)- 단일 NATS reply:
natsClient.respond(msg, response)(line 144 등) — dispatch 종료 후 정확히 1회
3.2 dispatch loop — src/agent/dispatch.ts
핵심 재귀 함수 dispatch(input, deps):
인자:
input: { budget, chain, depth, history, request, skill }
deps: { selectSkillForRequest, skills, transitionMatrix,
shouldApplyRuntimeVerifier, getSkillJudgeConfig,
onHandoff?, onDispatchEnd?, evalPlan? }
순서:
1) runSkillOnce(input) → response # skill.handle()
2) normalizeControlTokens(response) # [HANDOFF:..] → metadata
3) applyRuntimeVerifier(input, response) # pass/retry/escalate (Iter6)
4) applyEscalationTrigger(...) # confidence/risk → handoff
5) withHandoffMetadata(response, chain, depth) # handoffChain 노출 (Iter5)
6) readHandoffRequest(response) # metadata.handoffRequest
└─ null → 최종 응답 반환
7) prepareHandoff(...) # next skill + packet 합성
├─ resolveHandoffTarget (TransitionMatrix 검증)
├─ budget.assertBeforeHandoff (max-hops/loop/cost)
├─ budget.recordHandoff(target)
└─ buildHandoffPacket (PII 차단, evidence_refs)
8) dispatch(next) 재귀
3.3 dispatch 종료 후
metadata.handoffChain: string[]자동 부착 (eval에서 chain 검증 가능)metadata.handoffDepth: numbermetadata.handoffRequest는 마지막 handoff 정보 (없으면 미부착)metadata.routedBy: supervisor의 경우"regex-fast-path" | "llm" | "rules-fallback"(Iter12)
3.4 단일 publish 보장
NATS publish 지점은 handleRequest 끝 한 곳뿐. 4 hop 재귀라도 사용자는 응답 한 번만 받음.
4. Skill 라우팅
4.1 Skill 등록 (bootstrap)
src/main.ts:
1. SkillRegistry.load("src/skills")
- 각 폴더의 SKILL.md frontmatter 파싱 → SkillManifest 배열
2. factories map = { onboarding, moderation, supervisor }
3. 등록 루프: agent.registerSkill(factory({ ...deps }))
- deps: garnetClient, llmClient, roomContextService, shadowEvalBus,
natsClient, memoryService, escalationHistoryService, ...
4. TransitionMatrix.fromRegistry(skillRegistry) → Agent에 주입
4.2 SKILL.md frontmatter (capability card)
각 src/skills/<name>/SKILL.md:
name: onboarding
description: ...
version: 1
priority: 70 # canHandle 공동 당첨 시 desc 정렬
intents: [greeting, ..., age-question]
triggers: [user-message, new-member, ...]
tools: [validate-nickname, ...]
resources:
nickname-format: ./resources/nickname-format.md
...
trustTier: T3 # T1(가장 신뢰) ~ T4(가장 보수적)
domain: onboarding # Iter4 capability card
canHandle: [...]
cannotHandle: [...]
handoffTargets: [moderation, supervisor]
riskTier: low | medium | high
outputContract:
format: text
maxChars: 600
4.3 라우팅 결정 — selectSkillForRequest
src/agent/agent.ts:28-37:
1. skills.filter(s => s.canHandle(request))
- 각 Skill의 canHandle: 보통 request.skill === self.name (정확 매칭)
2. priority desc 정렬
3. [0] 반환
- onboarding:
request.skill === "onboarding"(priority 70) - moderation:
request.skill === "moderation"(priority 50) - supervisor:
request.skill === "supervisor"OR fallback (priority 10)
4.4 TransitionMatrix (handoff 그래프)
src/skills/transition-matrix.ts. 각 capability card의 handoffTargets를 모아 그래프 구성. supervisor는 모든 항목의 fallback 자동 포함.
isAllowed(from, to) # true/false
targetsOf(from) # 허용 target 집합
validateGraph() # 빌드타임 일관성
Dispatch loop에서 handoff 시 transitionMatrix.isAllowed() 검사. 거부 시 강제 supervisor로 escalation.
4.5 검증 — npm run lint:skills
src/skills/validate-manifests.ts:
- intents 배열 ⊆ 코드
IntentKind(manifest-binding.ts에서 derive) - tools 배열 == ToolRegistry tool names
- resources 키들 실제 파일 존재
- priority 중복 0
- handoffTargets 항목이 실제 skill name 또는
supervisor - cannotHandle ∩ canHandle = ∅
매 PR 시 CI에서 lint:skills 통과 필수.
5. Memory 계층
자료의 강조점 "context engineering > prompt engineering" + MemoryAgentBench 4축 (accurate retrieval / test-time learning / long-range understanding / selective forgetting) 모두 구현.
5.1 3-layer 인터페이스 (Iter10)
src/memory/memory-types.ts:
WorkingMemory // 매 turn 조립. dispatch loop가 사용
SessionMemory // 현 대화 세션. Garnet 12h TTL
LongTermMemory // 영속. Garnet (profile) + MongoDB (knowledge)
5.2 SessionMemory (Garnet, 12h~24h)
| 데이터 | Garnet key | TTL | 소스 |
|---|---|---|---|
| Blackboard (skill-local) | lolympus-agent:bb:{skill}:{chatroom}:{user} |
12h | src/skills/onboarding/state/blackboard.ts |
| Conversation history | lolympus-agent:conv:{chatroom}:{user} |
12h | src/conversation/conversation-store.ts, max 20 메시지 |
| Conversation summary | lolympus-agent:summary:{chatroom}:{user} |
24h | src/conversation/summarizer.ts, 5+ 메시지 trigger |
5.3 LongTermMemory (Garnet + MongoDB, 30d+)
| 데이터 | 저장소 | TTL |
|---|---|---|
| UserProfile (preferredStyle, pastIntents, flaggedIssues, longTermSummary) | Garnet lolympus-agent:user:profile:{user} |
30d (env: AGENT_USER_MEMORY_TTL_DAYS) |
| KnowledgeDocuments (RAG) | MongoDB collection knowledge_documents |
영속 (eviction policy로 archive) |
| UserEscalationHistory | MongoDB collection user_escalation_history |
영속 |
UserProfile 필드 (src/skills/onboarding/state/user-memory.ts):
userId: string // anonymizeText() 처리
firstSeenAt, lastSeenAt: string
completedOnboardingAt?: string
totalTurns: number
pastIntents: Array<{kind, count}> // 최근 10개
preferredStyle: "warm" | "concise" | "neutral" // 자동 추론
flaggedIssues: string[] // 최대 10
longTermSummary?: { text, updatedAt, turnSnapshot } // Iter11
5.4 WorkingMemory 조립
매 요청 시 message-builder.ts에서 호출:
const wm = await memoryService.assembleWorkingMemory({
skill: "onboarding",
chatroomId, userId,
includeBlackboard: true,
includeProfile: true,
retrieval: shouldRetrieve(derived)
? { query: ..., options: { skill, category, topK: 2 } }
: undefined,
});
// wm: { conversationHistory, summary, blackboard, userProfile, retrievedKnowledge }
shouldRetrieve(derived) (Iter10 Phase D): help-how, help-where, nickname-question, discord-question, age-question 5개 intent만 RAG. greeting/acknowledgement는 skip(비용 절감).
5.5 LLM prompt 주입 블록
message-builder.ts + prompt-data.ts:
[reference] ← retrieval된 knowledge chunk
[user_context] ← test-time learning snippets (자주 묻는 intent 등)
[long_term_memory] ← UserProfile.longTermSummary (Iter11)
[user_escalation_history] ← supervisor capability-prompt에만 (internal only)
6. Agentic RAG
자료의 "modular > monolithic" + "retrieval은 pre-filter → rerank → compression" 권고 적용.
6.1 Embedding (src/memory/embedding-client.ts)
추상화 인터페이스 + 2개 구현 (Iter10):
interface EmbeddingClient {
embed(text): Promise<number[]>
embedBatch(texts): Promise<number[][]>
model: string
dimensions: number
}
createEmbeddingClient()
// env: EMBEDDING_PROVIDER ("openrouter" | "openai")
// OpenRouter 우선 → 미지원 시 OpenAI fallback
// env: EMBEDDING_MODEL (예: "text-embedding-3-small")
// env: EMBEDDING_API_KEY 또는 OPENROUTER_API_KEY
6.2 Ingest — npm run ingest:knowledge
scripts/ingest-knowledge.ts → src/memory/knowledge-ingest.ts:
src/skills/{onboarding,moderation}/resources/*.md5개 파일 로드##heading 단위로 split- 각 chunk를
{ skill, category, text, metadata: { sourceFile, sectionTitle } }로 build embeddingClient.embed(text)→ vector- MongoDB
knowledge_documentscollection upsert (metadata.sourceFile + sectionTitle기준 idempotent) embeddingModel필드 자동 기록 (마이그레이션 식별용)
총 20~50 chunks 예상. 향후 1000+ 시 MongoDB Vector Search 마이그레이션.
6.3 Retrieval pipeline (Iter10 → Iter11)
src/memory/mongo-long-term-memory.ts:search():
1. embeddingClient.embed(query) → query vector
2. MongoDB find with { skill?, category?, metadata.archived: { $ne: true } }
→ 후보 N개 (env: AGENT_RAG_CANDIDATE_POOL_SIZE, 기본 50)
3. (옵션) Hybrid (Iter11 Phase A)
- cosine score (in-app, src/memory/cosine.ts)
- BM25 score (in-app, src/memory/bm25.ts)
- 가중 평균 (각 0.5)
4. top-K 선정 (기본 10)
5. (옵션) Reranker (Iter11 Phase A)
- LlmReranker (tier 0, maxTokens 200, jsonMode)
- 실패 시 원래 순서 (graceful)
6. top-N 반환 (기본 5, env: AGENT_RAG_RERANKER_TOP_K)
각 단계에서 Phoenix span attribute:
memory.retrieval.candidate_countmemory.retrieval.top_scorememory.retrieval.duration_msmemory.reranker.candidates_countmemory.reranker.changed_order(boolean)
6.4 Selective forgetting (Iter11 Phase C)
src/memory/forgetting-policy.ts + src/memory/contradiction-detector.ts:
3가지 규칙:
- Relevance: 30일간 reference 0회 +
referenceCount30d < 3→ archive - Freshness:
updatedAt90일 이상 + 사용 빈도 낮음 → archive - Contradiction: 같은
sourceFile + sectionTitle그룹 내 텍스트 30%+ 차이 또는 명시적 키워드 → 기존 archive, 새 doc primary (staging 필수)
scripts/memory-eviction-cron.ts (npm run memory:evict --dry-run):
- archived는
metadata.archived = trueflag만 (delete 아님). 복구 가능 - search() 자동으로 archived 제외
6.5 사용 site
Moderation lookup-rule tool (Iter10 Phase C):
args: { topic: string, topK?: number }
→ memoryService.longTerm.search(topic, { skill: "moderation", category: "rule" })
→ tool result에 chunk text 포함
→ LLM이 retrieval된 reference로 답변
Onboarding message-builder.ts (Iter10 Phase D):
- 5개 intent에서만 retrieval
[reference]block에 chunk 주입- LLM이 ground truth 기반 답변
7. Tool 실행
7.1 Tool 인터페이스 (Iter12 Phase B)
src/tools/tool.ts:
interface Tool<A, R> {
name: string
description: string
argsSchema: ZodSchema<A>
resultSchema: ZodSchema<R>
requires?: readonly ToolPermission[] // Iter12: 권한 declaration
execute(args: A, ctx: ToolContext): Promise<ToolResult<R>>
}
type ToolPermission =
| "garnet" | "nats" | "room-context"
| "user-session" | "tool-registry"
| "memory-search" | "memory-profile"
7.2 Scoped Context Injection (Iter12 Phase B)
src/tools/scoped-context.ts:
buildScopedContext(tool.requires ?? [], fullContext) → scopedContext
- 권한 없는 필드 접근 시:
AGENT_TOOL_SANDBOX_ENABLED=true: throwAGENT_TOOL_SANDBOX_ENABLED=false(log-only): warn 로그만
7.3 등록된 tool
| Tool | Skill | requires |
|---|---|---|
| validate-nickname | onboarding | [] (pure) |
| check-onboarding-progress | onboarding | ["garnet", "user-session"] |
| lookup-community-rule | onboarding | ["memory-search"] |
| get-room-context | onboarding | ["room-context"] |
| verify-discord-nickname | onboarding | ["garnet", "nats", "user-session"] |
| lookup-rule | moderation | ["memory-search"] |
7.4 Tool Audit (Iter12 Phase B)
src/tools/audit.ts + Garnet stream lolympus-agent:tool-calls:
매 tool 호출 후 envelope append:
{
type: "tool-call",
toolName, chatroomId, userId,
argsHash: string, // PII 마스킹된 hash (원본 args 미저장)
resultSummary, ok, errorKind?,
durationMs, skill, traceId, calledAt
}
- maxLen 100_000 자동 trim
- Operations CLI:
npm run tool-audit:list,npm run tool-audit:show <id>
7.5 외부 통합 (lolympus 도메인)
- Discord:
verify-discord-nicknametool이 NATS subjectlolympus.discord.lookup-members로 외부 microservice에 위임 (HTTP 직접 호출 X) - Kakao: session에 nickname 정보만 (외부 API 호출 없음, 클라이언트 제공)
8. Handoff + Supervisor (Bounded Self-Escalation Graph)
자료의 "Bounded Self-Escalation Graph" 8가지 규칙 모두 적용 (Iter4-6).
8.1 HandoffRequest schema (Iter4)
src/agent/types.ts:
interface HandoffRequest {
targetAgent: string
reason: HandoffReason // 8가지 enum
taskSpec: string
evidenceRefs: string[] // toolCall id 또는 messageId
confidence: number // 0..1
risk: "low" | "medium" | "high"
expectedOutput: string
userVisibleMessage?: string
}
type HandoffReason =
| "out-of-scope" | "missing-tool" | "low-confidence"
| "validation-failed" | "same-agent-revisit"
| "max-hop" | "high-risk-action" | "policy-risk"
8.2 [HANDOFF:..] 토큰 파이프라인 (Iter4 Phase D)
기존 [NO_RESPONSE] 패턴을 확장. 5단계:
- Prompt —
prompt-data.tspersona에 사용법 명시 (capability card 기반 자동 합성) - Abstention —
runtime/abstention.ts의AbstentionAction에{kind:"handoff", targetSkill, reason}variant - Validator —
runtime/validator.ts:142+에서[HANDOFF:..]token pass - Text generator —
runtime/text-generator.ts:79+에서{status:"handoff", request}반환 - Skill.handle —
makeEmptyResponse + metadata.handoffRequest반환
Parser: src/skills/control-token-parser.ts (regex /\[HANDOFF:([a-z0-9_-]+):([a-z-]+)\]/giu).
8.3 HandoffPacket (Iter4 Phase E)
src/agent/handoff-packet.ts. PII 차단 minimal context:
{
summary: string // src/conversation/summarizer.ts 결과
userGoal: string // 240자 제한, PII 제거
currentState: Record // phase/responseMode/tier만 (userMessage 원문 미포함)
handoffReason: HandoffReason
evidence: HandoffEvidence[] // toolCalls[].id
constraints: string[]
expectedOutputSchema: string
sourceChain: string[] // 거쳐간 skill 경로
}
다음 dispatch hop에서 request.context.handoffPacket으로 전달. 수신 skill은 readHandoffPacket(request) 호출.
8.4 BudgetTracker (Iter4 Phase F + Iter12 Phase A)
src/agent/budget.ts:
| 한도 | 기본 | env |
|---|---|---|
| MAX_HANDOFFS | 4 | AGENT_MAX_HANDOFFS |
| MAX_SAME_AGENT_VISITS | 2 | AGENT_MAX_SAME_AGENT_VISITS |
| MAX_TOTAL_TOOL_CALLS | 20 | AGENT_MAX_TOOL_CALLS |
| MAX_COST_USD | 0.5 | AGENT_MAX_COST_USD |
assertBeforeHandoff(target) verdict: "ok" | "max-hops" | "same-agent-loop" | "tool-budget" | "cost-budget".
Iter12 핵심: recordLlmCost(costUsd) — LlmClient 호출 끝에서 자동 hook (context.budget?.recordLlmCost(cost)). snapshot에 totalLlmCostUsd, totalToolCostUsd 분리 노출.
8.5 Supervisor (Iter4 Phase G + Iter6 + Iter12 Phase A)
src/skills/supervisor/skill.ts. LLM + 룰 기반 hybrid:
1. readHandoffPacket(request)
2. routeByRegex(packet) → { target, confidence }
3. confidence ≥ 0.85 (env: SUPERVISOR_FAST_PATH_CONFIDENCE)
→ LLM skip. metadata.routedBy = "regex-fast-path" ← Iter12 G 패턴
4. 그 외 → routeByLlm(packet):
- capability-prompt에 모든 skill capability card 자동 합성
- + escalation history (Iter11) [user_escalation_history] block
- tier 0, maxTokens 150, jsonMode
- { target, confidence, reason } JSON 응답
5. LLM 실패 → 정규식 fallback. metadata.routedBy = "rules-fallback"
6. appendSupervisorReview() → review queue + audit
7. AlertService.sendAlert (NATS + Slack, Iter7)
8. 응답 텍스트: 안정성 위해 template 유지 (Iter6 결정)
8.6 Runtime Evaluator (Iter4 Phase H)
SkillJudgeConfig.runtimeVerifier?: (input) => "pass" | "retry" | "escalate":
- pass → 진행
- retry → 1회 재호출 (
trigger="retry",MAX_RUNTIME_RETRIES=1) - escalate → supervisor 강제 handoff (
reason="validation-failed")
Onboarding 기본 verifier: validateOnboardingResponse. Moderation: PII/외부 링크 차단 regex.
9. Cost-aware Tier Selection (Iter12 Phase A)
자료 G 패턴 + OpenInference cost 활용.
9.1 Tier 구조
| Tier | 용도 | 기본 모델 | maxTokens | env |
|---|---|---|---|---|
| 0 | 분류, 짧은 응답, supervisor LLM, reranker | gpt-5.3-codex-spark | 384 | TIER0_MODEL, TIER0_MAX_TOKENS |
| 1 | planner / responder | gpt-5.4 / gpt-5.3 | 512 | TIER1_PLANNER_MODEL, TIER1_RESPONDER_MODEL |
| 2 | critic | gpt-5.4 | 512 | TIER2_CRITIC_MODEL |
9.2 Router 결정 입력 (Iter12 확장)
src/skills/onboarding/runtime/router.ts:decidePreGenerationRouting():
input: {
classified: { intent, confidence },
derived,
classifyConfidence: number, // Iter12
intentRiskTier?: "low"|"medium"|"high", // Iter12 (capability card)
userEscalationCount?: number, // Iter12 (escalation history)
recentLatencyP95Ms?: number // Iter12 (선택)
}
룰:
- confidence ≥ 0.9 + low-risk → tier 0 (cost 최소)
- confidence < 0.5 또는 high-risk + escalation > 3 → tier 1 (quality 우선)
- p95 latency SLA 위반 추세 → tier 0 (degradation 보호)
9.3 LLM cost 추적 (Iter9 + Iter12)
src/llm/llm-client.ts:
const costUsd = estimateChatCost(model, prompt_tokens, completion_tokens);
metrics.tokenCostUsd.observe(costUsd, { model, skill: getContext()?.skill });
getContext()?.budget?.recordLlmCost(costUsd); // Iter12
// Phoenix span: gen_ai.usage.cost_usd = costUsd
estimateChatCost (src/llm/cost.ts): 모델별 input/output rate 테이블.
9.4 환경 변수 (cost-aware)
| env | 기본 | 효과 |
|---|---|---|
AGENT_COST_AWARE_TIER_ENABLED |
true | confidence 기반 tier 결정 |
AGENT_BUDGET_LLM_COST_ENABLED |
true | LlmClient → Budget 자동 hook |
AGENT_SUPERVISOR_FAST_PATH_ENABLED |
true | regex 명확 시 LLM skip |
SUPERVISOR_FAST_PATH_CONFIDENCE |
0.85 | 임계값 |
10. Evaluation 파이프라인
자료의 "평가 가능한 실행 환경 없이 데모는 의미 없음" 권고. Iter3에서 skill-aware로 전환됨.
10.1 Golden set 구조 (Iter2 + 후속)
src/eval/golden/
├── onboarding/
│ ├── cases.jsonl # 활성 golden, ~11개
│ ├── candidate.jsonl # 자동 수집 후보 (promote 후보)
│ └── capture.jsonl # 실시간 prod 캡처 buffer
├── moderation/
│ └── cases.jsonl # 5개 (Iter2)
├── handoff/
│ └── cases.jsonl # 6개 (Iter5)
└── memory/
└── cases.jsonl # 6개 (Iter11, 4축 × 1~2개)
EvalCase 구조 (src/eval/types.ts):
{
id, name, notes,
input: { request, conversationHistory, roomContext, session, blackboard, summary },
expected: {
intent?, mustIncludeAny?, mustNotInclude?, responseMode?,
processExpected?: { expectedTools, expectedApproach, minSteps, toolMode },
expectedHandoff?: { shouldHandoff, expectedReason, expectedTarget }, // Iter5
expectedFinalSkill?: string,
expectedSourceChain?: { mode: "exact"|"prefix"|"contains", chain },
expectedMemory?: MemoryJudgeExpected // Iter11
}
}
10.2 Runner 흐름 (src/eval/runner.ts)
1. resolveEvalSetPath(setName)
"onboarding/cases" → src/eval/golden/onboarding/cases.jsonl
"golden" → legacy alias for "onboarding/cases"
2. parseEvalCaseLine() → EvalCase[]
3. for each case:
a. SKILL_RUNNER_FACTORIES[skill]({deps}) → Skill instance
b. handoff/cases set이면 runEvalDispatch (mini-dispatcher, Iter5)
그 외엔 skill.handle() 직접 호출
c. judges 실행:
- rule-judge (mustIncludeAny, mustNotInclude, intent)
- llm-judge (skill별 평가 축, SkillJudgeConfig)
- process-judge (tool trace 검증)
- handoff-judge (chain, reason, target) — handoff set만
- memory-judge (retrieval/long-range/test-time/forgetting) — memory set만
d. summary 집계
4. baseline (eval-reports/baseline.{skill}.json) 비교
5. eval-reports/<ts>.json 저장
10.3 Judges (src/eval/judges/)
| Judge | 책임 | Iter |
|---|---|---|
rule-judge.ts |
mustIncludeAny / mustNotInclude / intent 매칭 | Iter3 일반화 |
llm-judge.ts |
skill별 평가 축. SkillJudgeConfig.axes |
Iter3 |
process-judge.ts |
tool trace, skill별 tool registry | Iter3 |
handoff-judge.ts |
dispatch chain 검증 | Iter5 |
memory-judge.ts |
retrieval 정확도, long-range 회상 등 | Iter11 |
attribution.ts |
slice 기반 root cause | (이전) |
SkillJudgeConfig (src/eval/judges/skill-config.ts):
{
onboarding: {
axes: ["friendliness", "accuracy", "concision", "scope"],
toolRegistryFactory: createOnboardingToolRegistry,
validator: validateOnboardingResponse,
promptTemplate, tournamentInstruction,
runtimeVerifier?: (input) => "pass"|"retry"|"escalate"
},
moderation: {
axes: ["appropriateness", "accuracy", "helpfulness"],
toolRegistryFactory: createModerationToolRegistry,
...
}
}
10.4 자동 개선 루프
[1] Capture (shadow-eval-bus) — prod 트래픽 sample → shadow envelope
↓
[2] Run — golden cases 실행, judges 적용, eval-reports/<ts>.json
↓
[3] Propose (eval:propose CLI) — 저점수 slice → LLM이 prompt variant 생성
→ src/eval/proposals/<variantId>.json
↓
[4] Tournament (eval:tournament) — control vs variant A/B, ELO 갱신
↓
[5] Governance gates (G1~G4) — trust tier 검증
↓
[6] Promote (eval:promote) — golden 후보 → 활성 cases
10.5 변형 적용 (PROMPT_VARIANT_ID)
src/skills/onboarding/prompts/builder.ts:
loadPromptVariantManifest(variantId)→ variant JSON 로드applyLayerDiff(layers, diff)—persona/policy/state/context/examples/tool-policy6 layer 중 일부 replace/append/remove- env
PROMPT_VARIANT_ID=<id>→ 프로덕션에서 활성
10.6 Calibration / Drift / Governance
src/eval/aggregator.ts— SliceMetrics (intent/phase/tier 차원)src/eval/drift.ts— baseline 대비 점수 하락 감지src/eval/governance/trust.ts— T1~T4 enumsrc/eval/governance/gates.ts— variant 승격 4 gate (G1~G4)src/eval/router-shadow-report.ts(Iter6) — learned vs static routing divergence
11. Memory 평가 (Iter11)
src/eval/golden/memory/cases.jsonl 6 cases:
| # | 이름 | 검증 |
|---|---|---|
| 1 | retrieval-accurate | "광고 도배" 질문 → moderation lookup-rule retrieval 정확 chunk |
| 2 | long-range-understanding | 5턴 전 "닉네임 LolName_KR" → 회상 |
| 3 | test-time-learning | flaggedIssues 3회 누적 → prophylactic guidance |
| 4 | selective-forgetting | 신규 doc 30% 차이 + "정정" 키워드 → 기존 archive |
| 5 | escalation-history | 사용자 3회 광고 시도 → supervisor가 moderation 우선 라우팅 |
| 6 | memory-contradiction | "닉네임 X로 변경" → profile 반영, 다음 응답이 새 닉네임 인지 |
MemoryJudgeResult (src/eval/judges/memory-judge.ts):
expectedRetrievedDocIds,expectedRecalledFact,expectedProphylacticHint,expectedArchivedDocPattern,expectedSupervisorTarget,expectedContradictionResolved
명령: npm run eval -- --set=memory/cases.
12. Operations — 운영 루프
12.1 Review Queue 두 종류
| Queue | Garnet stream | 목적 | 진입 |
|---|---|---|---|
| Eval Review | lolympus-agent:eval:review:queue |
학습용 (저점수/희귀 slice/rule-llm 불일치) | runner가 자동 enqueue |
| Supervisor Handoff | lolympus-agent:review-queue |
운영용 (escalation case 사람 검토) | supervisor.skill.ts |
12.2 Operations CLI (Iter5 + Iter12)
src/operations/cli.ts:
# Supervisor handoff queue
npm run review:list -- --limit=20 --since=24h
npm run review:show -- <id> [--history] # --history는 Iter6 audit log
npm run review:resolve -- <id> --action=manual-reply --note="..."
npm run review:stats -- --since=7d
# Tool audit (Iter12)
npm run tool-audit:list -- --skill=onboarding --since=1h --limit=50
npm run tool-audit:show -- <stream-id>
12.3 Resolution audit log (Iter6)
이중 저장:
- Hash
lolympus-agent:review-resolved:<id>— latest state (덮어쓰기, 7d TTL) - Stream
lolympus-agent:review-resolutions— append-only (운영자 변경 이력)
getResolutionHistory(garnet, id) — 사용자별 resolution 변경 이력 조회 (Iter11 escalation history 입력).
12.4 Alert flow (Iter7)
src/observability/alert.ts:AlertService.sendAlert(payload):
1. Garnet dedupe key 검사 (5분 throttle)
key: lolympus-agent:alert:dedupe:<kind>:<skill>
2. NATS publish → AGENT_ALERT_SUBJECT
3. Slack webhook POST (env: SLACK_ALERT_WEBHOOK_URL)
- fire-and-forget, AbortSignal.timeout 3초
- block kit message (severity emoji + 필드)
4. logger.warn (구조화 로그)
AlertPayload:
{
kind: "archive-stalled"|"endpoint-down"|"eval-drift"|"eval-score-drop"
|"latency"|"llm-error-rate"|"supervisor-handoff",
metric, severity, skill?, threshold, value, window, timestamp, ...
}
12.5 Slack message 포맷 (Iter7)
src/observability/slack-notifier.ts:formatSlackMessage():
- severity emoji (
:information_source:/:warning:/:rotating_light:) - Section + fields (metric / value vs threshold / window / skill)
- suggestedAction (있다면)
13. Observability
13.1 Logger (src/observability/logger.ts)
- Pino + AsyncLocalStorage
getLogger()→ context의 child logger 자동 (skill, requestId, chatroomId, userId)- redact:
authorization,env.<secret>,secrets.<secret>,request.<secret>(KNOWN_SECRET_KEYS 확장됨, Iter12)
13.2 Tracing (Iter9)
src/observability/tracing.ts:
- OTel NodeSDK + auto-instrumentation
- Exporter: env
PHOENIX_OTLP_ENDPOINT설정 시 OTLP HTTP, 미설정 시 ConsoleSpanExporter (또는 OFF, env:OTEL_EXPORTER) - Sampling:
OTEL_SAMPLING_RATIO(prod 0.1, dev 1.0)
src/observability/spans.ts helper:
withSkillSpan(skill, attrs, fn)—skill.<name>.handlewithSpan(name, attrs, fn)— generic + auto-errorwithClassifierSpan,withToolSpan,withLlmSpan
13.3 LLM Span 속성 (Iter9 OpenInference)
src/llm/llm-client.ts:
gen_ai.request.model
gen_ai.request.temperature
gen_ai.request.max_tokens
gen_ai.system (provider name)
gen_ai.response.id
gen_ai.response.finish_reasons[]
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.cost_usd ← Iter9, Iter12에서 routing 활용
llm.token_count.prompt ← Phoenix 친화 alias
llm.token_count.completion
gen_ai.prompt.0.role/content ← optional (PHOENIX_LOG_PROMPTS)
gen_ai.completion.0.content ← optional + PII 마스킹
Memory 관련 (Iter10-11):
memory.retrieval.candidate_countmemory.retrieval.top_scorememory.retrieval.duration_msmemory.reranker.changed_order
Handoff 관련:
agent.handoff_depthagent.handoff_chainsupervisor.routed_by
13.4 Metrics (Iter4 Phase I)
자체 구현 (src/observability/metrics.ts). /metrics HTTP endpoint (metrics-server.ts, port 기본 9464). 10 counter/histogram:
| Metric | 타입 | Labels |
|---|---|---|
agent_handoff_total |
counter | from, to, reason |
agent_handoff_loop_total |
counter | from |
agent_handoff_count |
histogram (0..4) | — |
agent_human_override_total |
counter | kind |
agent_missed_escalation_total |
counter | — |
agent_routing_accuracy_total |
counter | result |
agent_unnecessary_escalation_total |
counter | — |
agent_validator_fail_total |
counter | skill, stage |
agent_token_cost_usd |
histogram | model, skill |
agent_time_to_resolution_ms |
histogram | skill |
/healthz endpoint: 200 "ok" (Iter7).
13.5 Alert threshold (skill별, Iter7)
src/config.ts:
getAlertThresholdP95LatencyMs(skill?: string)
getAlertThresholdEvalScoreDropPct(skill?: string)
getAlertThresholdLlmErrorRate(skill?: string)
getAlertThresholdDriftPct(skill?: string)
env 패턴: ALERT_THRESHOLD_<METRIC>_<SKILL_UPPER> 우선 → ALERT_THRESHOLD_<METRIC> fallback.
예: ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING=5000 또는 글로벌 ALERT_THRESHOLD_P95_LATENCY_MS=3000.
14. 보안
14.1 Secret Manager (src/common/secret-manager.ts)
계층:
- Infisical (외부 secret manager,
INFISICAL_SITE_URL) - AWS IMDS (EC2 메타데이터 fallback)
- 환경변수 (마지막 fallback)
KNOWN_SECRET_KEYS (src/common/known-secret-keys.ts, Iter12 확장):
INFISICAL_*, NATS_*, GARNET_*, MONGODB_*, SLACK_ALERT_WEBHOOK_URL,
OPENROUTER_API_KEY, AGENT_LLM_API_KEY, OPENAI_API_KEY,
EMBEDDING_API_KEY, PHOENIX_API_KEY
getSecret(key): KNOWN_SECRET_KEYS 검증. getRawSecret(key): 임의 키 (운영 도구용).
14.2 NATS Subject ACL (Iter12 Phase C)
src/common/nats-skill-acl.ts:DEFAULT_NATS_ACL:
[
{
skill: "onboarding",
allowedPublishSubjects: ["lolympus.discord.lookup-members"],
allowedSubscribeSubjects: []
},
{ skill: "moderation", allowedPublishSubjects: [], ... },
{ skill: "supervisor", allowedPublishSubjects: ["lolympus.agent.alert"], ... }
]
src/common/nats.ts의 publish wrapper:
getNatsAclMode():"off": 검사 skip"log-only"(기본): warn 로그, 정상 publish"enforce": throw + alert
운영 흐름: log-only 1개월 → false positive 없으면 enforce 전환 (Iter13+ task).
14.3 PII 차단
- HandoffPacket:
userMessage원문 포함 X.userGoal은 240자 + anonymize - Memory user-profile:
anonymizeText()처리 - Tool audit:
argsHash만 저장, 원본 args X - Phoenix prompt content:
PHOENIX_LOG_PROMPTS=false기본 - escalation history: supervisor system prompt 내부만, 응답 텍스트에 노출 금지
15. 환경 변수 + 배포
15.1 인프라
| env | 용도 |
|---|---|
NATS_ENDPOINT, NATS_USER, NATS_PASS |
NATS 연결 |
GARNET_ENDPOINT, GARNET_USERID, GARNET_PASSWORD |
Garnet 연결 |
MONGODB_CONNECTION_STRING, MONGODB_PROJECT_ID |
MongoDB (optional) |
INFISICAL_CLIENT_ID, INFISICAL_CLIENT_SECRET |
Secret manager |
15.2 모델
| env | 기본 | 용도 |
|---|---|---|
TIER0_MODEL |
gpt-5.3-codex-spark | 분류, supervisor LLM, reranker |
TIER1_PLANNER_MODEL, TIER1_RESPONDER_MODEL |
gpt-5.4, gpt-5.3 | 메인 응답 |
TIER2_CRITIC_MODEL |
gpt-5.4 | critic |
TIER0_MAX_TOKENS |
384 | |
EMBEDDING_PROVIDER |
openrouter | OpenRouter→OpenAI fallback |
EMBEDDING_MODEL |
text-embedding-3-small | |
EMBEDDING_API_KEY |
— | (또는 OPENROUTER/OPENAI 키 재사용) |
15.3 기능 flag (Iter별)
| env | 기본 | Iter |
|---|---|---|
AGENT_ROUTER_MODE |
static | Iter3 |
AGENT_RAG_ENABLED |
true | Iter10 |
AGENT_RAG_HYBRID_ENABLED |
true | Iter11 |
AGENT_RAG_RERANKER_ENABLED |
true | Iter11 |
AGENT_RAG_CANDIDATE_POOL_SIZE |
50 | Iter11 |
AGENT_RAG_RERANKER_TOP_K |
5 | Iter11 |
AGENT_LONG_TERM_SUMMARY_ENABLED |
true | Iter11 |
AGENT_MEMORY_EVICTION_ENABLED |
false (cron만) | Iter11 |
AGENT_COST_AWARE_TIER_ENABLED |
true | Iter12 |
AGENT_BUDGET_LLM_COST_ENABLED |
true | Iter12 |
AGENT_SUPERVISOR_FAST_PATH_ENABLED |
true | Iter12 |
AGENT_TOOL_SANDBOX_ENABLED |
false (1주 검증 후 true) | Iter12 |
AGENT_NATS_ACL_MODE |
log-only | Iter12 |
AGENT_SUPERVISOR_LLM_ENABLED |
true | Iter6 |
AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED |
false (template 우선) | Iter8 |
15.4 한도 (Budget)
| env | 기본 |
|---|---|
AGENT_MAX_HANDOFFS |
4 |
AGENT_MAX_SAME_AGENT_VISITS |
2 |
AGENT_MAX_TOOL_CALLS |
20 |
AGENT_MAX_COST_USD |
0.5 |
15.5 임계값 (Alert)
| env (skill suffix optional) | 기본 |
|---|---|
ALERT_THRESHOLD_P95_LATENCY_MS |
3000 |
ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT |
10 |
ALERT_THRESHOLD_LLM_ERROR_RATE |
0.1 |
ALERT_THRESHOLD_DRIFT_PCT |
15 |
ALERT_RATE_LIMIT_SEC |
300 |
15.6 Observability
| env | 기본 |
|---|---|
PHOENIX_OTLP_ENDPOINT |
unset (console fallback) |
PHOENIX_LOG_PROMPTS |
false |
SLACK_ALERT_WEBHOOK_URL |
unset (Slack 비활성) |
SLACK_WEBHOOK_TIMEOUT_MS |
3000 |
METRICS_PORT |
9464 |
OTEL_SAMPLING_RATIO |
0.1 (prod) |
15.7 Bootstrap (src/main.ts)
순서:
1. initLogger()
2. initTracing() (Phoenix 또는 console)
3. startMetricsServer() (port 9464)
4. SecretManager.load (Infisical + IMDS)
5. NatsClient.connect
6. GarnetClient.connect
7. initMongoDB (optional)
8. SkillRegistry.load("src/skills")
9. TransitionMatrix.fromRegistry
10. ConversationStore + ShadowEvalBus
11. memoryService, escalationHistoryService, alertService (Slack notifier)
12. factories map { onboarding, moderation, supervisor }
13. for manifest in registry: agent.registerSkill(factory(deps))
14. agent.start() (NATS subscribe)
15. shutdown hooks
16. 도메인 데이터 (Lolympus 카카오톡 오픈채팅)
16.1 Onboarding (src/skills/onboarding/)
Capability card:
- priority: 70, trustTier: T3, riskTier: low
- domain: onboarding
- handoffTargets: [moderation, supervisor]
Intent (intents.ts, 12개):
- greeting, acknowledgement
- status-question
- help-how, help-where
- nickname-question, photo-question
- discord-question, discord-verify-question
- age-question
- chitchat, off-topic-to-other, unknown
Tools (5개): validate-nickname, check-onboarding-progress, lookup-community-rule, get-room-context, verify-discord-nickname
Resources (4개 .md): nickname-format, heart-rule, discord-format, age-policy
핵심 로직:
OnboardingIntentClassifier(3-stage: trigger → regex → LLM)deriveState(session, trigger, intent)→ phase + responseMode + tierruntime/: orchestrator, router, validator, critic, tool-plan-runner, message-builder, text-generator, abstention 등 ~10개 모듈
16.2 Moderation (src/skills/moderation/)
Capability card:
- priority: 50, trustTier: T4, riskTier: medium
- domain: moderation
- handoffTargets: [onboarding, supervisor]
Intent (5개): greeting, rule-question, violation-report, appeal, off-topic
Tools (1개): lookup-rule (Iter10에서 RAG 기반으로 전환)
Resources (1개 .md): community-rules (욕설, 광고, 도배, 신고 절차, 제재, 이의제기, 미성년자, 개인정보 등 섹션)
16.3 Supervisor (src/skills/supervisor/)
Capability card:
- priority: 10 (최저, fallback 전용), riskTier: medium
- domain: meta
- canHandle: ["*"]
- handoffTargets: [onboarding, moderation]
핵심:
capability-prompt.ts— 모든 skill의 capability card를 system prompt로 자동 합성routing-rules.ts— regex 기반 키워드 매칭 (Iter12 fast-path)review-queue.ts— appendSupervisorReview + getResolutionHistory + 도메인 escalation history 연동- 응답 텍스트: 안정성 위해 template ("운영자에게 전달했어요")
16.4 User Escalation History (Iter11 — lolympus 차별점)
user_escalation_history collection (MongoDB):
{
userId, chatroomId,
escalations: [{
queuedAt, reason, target,
resolution: { resolvedAt, action, note },
packetSummary, sourceChain
}],
totalEscalations, lastEscalationAt,
topReasons: [{ reason, count }]
}
Supervisor가 매 escalation 시 history 조회 → capability prompt의 [user_escalation_history] 블록 주입:
"이 사용자는 광고 시도 3회 (out-of-scope) → 다음 위반 시 즉시 차단 권장" "닉네임 검증 5회 실패 → 친절한 step-by-step 우선"
17. 명령어 cheat sheet
17.1 개발
npm run typecheck # TypeScript 검증
npm run test # Vitest 전체
npm run test -- --reporter=verbose # 상세
npm run lint # ESLint
npm run lint:skills # SKILL.md ↔ 코드 일관성
npm run build # tsc + tsc-alias
npm run dev # 로컬 부트스트랩
17.2 Evaluation
# Run
npm run eval -- --set=onboarding/cases
npm run eval -- --set=moderation/cases
npm run eval:handoff # = --set=handoff/cases
npm run eval -- --set=memory/cases # Iter11
# Baseline / Report
npm run eval:baseline
npm run eval:report
# 자동 개선 loop
npm run eval:capture
npm run eval:propose -- --slice="tier=0,intent=help-how,phase=initial,skill=onboarding"
npm run eval:tournament -- --skill=onboarding
npm run eval:promote
npm run eval:calibrate -- --check
npm run eval:router-shadow-report -- --since=7d # Iter6
17.3 Operations (Iter5 + Iter12)
# Supervisor review queue
npm run review:list
npm run review:show -- <id>
npm run review:show -- <id> --history # Iter6 resolution audit
npm run review:resolve -- <id> --action=manual-reply --note="..."
npm run review:stats
# Tool audit
npm run tool-audit:list
npm run tool-audit:show -- <id>
17.4 Memory (Iter10-11)
npm run ingest:knowledge # resources/*.md → MongoDB embedding
npm run memory:evict -- --dry-run --since=30d
npm run memory:evict -- --since=30d # 실제 archive
17.5 디버깅
PHOENIX_OTLP_ENDPOINT=http://localhost:6006/v1/traces npm run dev
curl http://localhost:9464/metrics | grep agent_
curl http://localhost:9464/healthz
# Specific log level
LOG_LEVEL=debug npm run dev
# RAG/sandbox 비활성으로 회귀 확인
AGENT_RAG_ENABLED=false npm run eval -- --set=onboarding/cases
AGENT_TOOL_SANDBOX_ENABLED=false npm test
18. 참고 자료
18.1 Iter별 plan 문서
docs/에 dev 브랜치 보존 (main에는 미포함):
| 문서 | 내용 |
|---|---|
refactor-skill-ssot.md |
Iter1 plan |
refactor-skill-ssot-iter2~8.md |
Iter2~8 각 plan |
optimization-roadmap.md |
Iter8 closing 시 trigger 정의 |
refactor-skill-ssot-iter10~12.md |
Iter10~12 plan |
optimization-roadmap-memory-ops.md |
Iter9~12 통합 로드맵 |
service-evaluation-operations.md |
운영자 매뉴얼 |
최신기술동향.md, 멀티에이전트동향.md |
자료 reference |
18.2 외부 자료
- MemoryAgentBench — accurate retrieval, test-time learning, long-range understanding, selective forgetting
- Bounded Self-Escalation Graph — Microsoft orchestration guide
- OpenInference semantic conventions —
gen_ai.*,llm.*span attributes - MCP / A2A — 향후 채택 후보 (Iter13+ trigger 기반)
18.3 다음 단계 trigger (Iter13+)
docs/optimization-roadmap.md의 trigger 발동 시 개별 plan으로 시작:
| 항목 | Trigger |
|---|---|
| NATS ACL enforce 전환 | log-only 1개월 안정 + false positive 0 |
| Tool sandbox enforce 전환 | log-only 1주 안정 |
| MongoDB Vector Search 마이그레이션 | preview → GA |
| Resolution hash sunset | Iter6 dual-write 2개월 안정 |
| Cross-room context aggregation | cross-room 패턴 발견 |
| Parallel fan-out | handoff 평균 hop ≥ 2.0 |
| Web UI for review queue | review 진입 ≥ N/day 1개월 |
| Workflow auto-discovery (AFlow) | propose/tournament 안정화 |
| MCP/A2A 채택 | 외부 ecosystem 연결 필요 |
각 항목은 evidence(metric/운영 사례) 기반으로 plan 작성.
19. 변경 이력 (commit 별)
main 브랜치의 주요 milestones:
| sha | 메시지 | Iter |
|---|---|---|
bc0f8e0 |
Add secret coverage and NATS subject ACL | 12 Phase C |
d011a28 |
Add tool permission sandbox and audit stream | 12 Phase B |
21df14c |
Add cost-aware routing and LLM budget tracking | 12 Phase A |
211de18 |
Add hybrid memory retrieval and escalation history | 11 |
1a86bde |
Add memory layers and RAG retrieval | 10 |
1d0be56 |
Add Phoenix OTLP tracing and LLM span attributes | 9 |
| (이전) | Close multi-agent plan and add router shadow reporting | 8 |
| (이전) | Add Slack alerts and skill-aware monitoring | 7 |
| (이전) | Unify dispatch and add supervisor routing signals | 6 |
| (이전) | Close supervisor review loop and add handoff eval | 5 |
| (이전) | Make eval pipeline skill-aware and validate SKILL manifests | 3 |
각 commit의 detail은 git log <sha> -p 또는 dev 브랜치의 docs/refactor-skill-ssot-iterN.md 참고.
끝
이 문서가 outdated되면 last-verified commit을 footer에 명시:
- last-verified:
bc0f8e0(2026-05-19)
업데이트 책임: 신규 큰 변경(새 skill 추가, 새 memory layer, 새 metric 등) PR 시 함께 갱신.