Files

48 KiB
Raw Permalink Blame History

Lolympus Agent — 시스템 아키텍처 및 운영 흐름

Iter1~12 (2026-05-13 ~ 2026-05-19) 완료 시점 기준의 통합 architecture + operations 문서입니다. 신규 기여자/운영자/AI agent의 단일 진입점.


0. 한눈에 보기

Lolympus Agent는 카카오톡 오픈채팅 운영을 자동화하는 NATS 메시지 드리븐 멀티 에이전트 시스템이다.

  • 도메인: 카카오톡 오픈채팅 가입 안내(onboarding) + 커뮤니티 규칙 모더레이션(moderation)
  • 외부 진입: NATS subject lolympus-agent.request
  • 응답 형태: NATS reply (단일 publish/요청)
  • 핵심 자산:
    • Skill SSOT 디렉토리 (src/skills/{onboarding,moderation,supervisor}/)
    • 통합 dispatch loop (src/agent/dispatch.ts, production + eval 공용)
    • 3-layer memory + Agentic RAG (src/memory/)
    • Bounded Self-Escalation Graph (handoff + transition matrix + budget + supervisor fallback)
    • Eval 파이프라인 (capture → run → propose → tournament → promote)
    • 자체 metrics + Phoenix LLM tracing
    • Slack webhook + operations CLI

권장 시작 순서

  1. §2 아키텍처 다이어그램 — 전체 그림
  2. §3 메시지 흐름 — 한 요청이 어떻게 흐르는지
  3. §4~9 — 각 컴포넌트 깊이 있게
  4. §16 cheat sheet — 자주 쓰는 명령

1. 기술 스택 요약

영역 선택 위치
언어 TypeScript (Node 22+) tsconfig.json
메시지 NATS src/common/nats.ts
캐시/세션 Garnet (Redis 호환) src/common/garnet.ts
영속 저장 MongoDB (mongoose) src/room-context/, src/memory/mongo-*.ts
LLM provider OpenRouter / OpenAI / cliproxy / llama (multi-provider) src/llm/llm-client.ts, src/llm/routing.ts
Embedding OpenRouter free 우선 → OpenAI fallback src/memory/embedding-client.ts
Vector store MongoDB collection + in-app cosine (Vector Search 마이그레이션 path 예약) src/memory/mongo-long-term-memory.ts, src/memory/cosine.ts
BM25 in-app src/memory/bm25.ts
관측 OTel SDK + Phoenix OTLP + 자체 metrics 모듈 src/observability/
알림 NATS publish + Slack incoming webhook src/observability/alert.ts, slack-notifier.ts
Secret Infisical → IMDS → env src/common/secret-manager.ts
Test Vitest *.test.ts
Plan/Eval 데이터 MongoDB collections + Garnet streams 각 모듈

새 의존성 도입 시 §15 환경변수 + KNOWN_SECRET_KEYS 갱신 확인.


2. 아키텍처 다이어그램

[NATS subject "lolympus-agent.request"]
                │
                ▼
   ┌─────────────────────────────────┐
   │     Agent.handleRequest         │  src/agent/agent.ts (~250 lines)
   │     - runWithContext (AsyncLocal)│
   │     - withSpan (OTel)           │
   │     - budget 생성, NATS reply   │
   └────────────┬────────────────────┘
                │ dispatch(input, deps)
                ▼
   ┌─────────────────────────────────┐
   │     dispatch loop                │  src/agent/dispatch.ts (~580 lines)
   │     ├─ selectSkillForRequest     │  priority + canHandle 필터
   │     ├─ runSkillOnce              │  skill.handle() 호출
   │     ├─ normalizeControlTokens    │  [HANDOFF:...] 파싱
   │     ├─ applyRuntimeVerifier      │  retry/escalate
   │     ├─ applyEscalationTrigger    │  confidence/risk → handoff
   │     ├─ prepareHandoff            │  TransitionMatrix + Budget 검증
   │     └─ recurse (max depth 4)     │  같은 dispatch 재진입
   └────────────┬────────────────────┘
                │
        ┌───────┴───────┐
        ▼               ▼
   [Onboarding]    [Moderation]    [Supervisor]
   ├ intents       ├ intents       ├ capability-prompt (LLM tier0)
   ├ prompt-data   ├ prompt-data   ├ routing-rules (regex fast-path)
   ├ tools (5)     ├ tools (1)     └ review-queue append + alert
   ├ runtime/      ├ skill.ts
   └ skill.ts

   ┌─────────────────────────────────┐
   │     MemoryService (Iter10-11)   │  src/memory/
   │     ├─ WorkingMemory (조립)      │  매 turn
   │     ├─ SessionMemory (12h Garnet)│  blackboard + conv + summary
   │     └─ LongTermMemory (30d+)    │
   │        ├─ UserProfile (Garnet)  │
   │        ├─ Knowledge (Mongo+RAG) │  embedding + cosine + BM25 + reranker
   │        └─ EscalationHistory (Mongo)│  user-level review queue rollup
   └─────────────────────────────────┘

   ┌─────────────────────────────────┐
   │     ToolRegistry                │  src/tools/
   │     ├─ Tool.requires (sandbox)  │
   │     ├─ scoped-context           │  least-privileged ctx
   │     └─ audit stream             │  lolympus-agent:tool-calls
   └─────────────────────────────────┘

   ┌─────────────────────────────────┐
   │     Observability                │  src/observability/
   │     ├─ Pino logger (skill child) │
   │     ├─ OTel tracing → Phoenix    │  gen_ai.* + memory.* + handoff.*
   │     ├─ Self-built metrics        │  /metrics endpoint (9464)
   │     └─ AlertService              │  NATS + Slack webhook
   └─────────────────────────────────┘

   ┌─────────────────────────────────┐
   │     Eval Pipeline                │  src/eval/
   │     ├─ Capture (shadow-bus)      │  prod traffic 샘플링
   │     ├─ Run (runner)              │  golden cases 실행
   │     ├─ Judges (rule/llm/process/ │
   │     │  handoff/memory)           │
   │     ├─ Propose (variant 생성)    │
   │     ├─ Tournament (A/B)          │
   │     └─ Promote (T4→T2)           │
   └─────────────────────────────────┘

3. 메시지 수신 → 응답 흐름

3.1 진입점 — Agent.handleRequest

src/agent/agent.ts. 흐름:

  1. NATS raw 수신: natsClient.subscribeRaw(getRequestSubject(), this.handleRequest) (start time)
  2. JSON 파싱 + 검증: isValidRequest(parsed) 통과 안 하면 즉시 empty response
  3. runWithContext 호출 — src/observability/context.ts. AsyncLocalStorage로 RequestContext 주입:
    • requestId, chatroomId, userId, skill, trigger
    • logger (skill name child binding)
    • startedAt, deadlineAt
    • budget, handoffDepth, handoffChain (Iter4)
  4. withSpan('agent.handle') OTel 외부 span 시작
  5. dispatch(input, deps) 호출 (실제 비즈니스 로직은 dispatch.ts에 위임 — Iter6 추출)
  6. 단일 NATS reply: natsClient.respond(msg, response) (line 144 등) — dispatch 종료 후 정확히 1회

3.2 dispatch loop — src/agent/dispatch.ts

핵심 재귀 함수 dispatch(input, deps):

인자:
  input: { budget, chain, depth, history, request, skill }
  deps:  { selectSkillForRequest, skills, transitionMatrix,
           shouldApplyRuntimeVerifier, getSkillJudgeConfig,
           onHandoff?, onDispatchEnd?, evalPlan? }

순서:
  1) runSkillOnce(input) → response                   # skill.handle()
  2) normalizeControlTokens(response)                 # [HANDOFF:..] → metadata
  3) applyRuntimeVerifier(input, response)            # pass/retry/escalate (Iter6)
  4) applyEscalationTrigger(...)                      # confidence/risk → handoff
  5) withHandoffMetadata(response, chain, depth)      # handoffChain 노출 (Iter5)
  6) readHandoffRequest(response)                     # metadata.handoffRequest
     └─ null → 최종 응답 반환
  7) prepareHandoff(...)                              # next skill + packet 합성
     ├─ resolveHandoffTarget (TransitionMatrix 검증)
     ├─ budget.assertBeforeHandoff (max-hops/loop/cost)
     ├─ budget.recordHandoff(target)
     └─ buildHandoffPacket (PII 차단, evidence_refs)
  8) dispatch(next) 재귀

3.3 dispatch 종료 후

  • metadata.handoffChain: string[] 자동 부착 (eval에서 chain 검증 가능)
  • metadata.handoffDepth: number
  • metadata.handoffRequest는 마지막 handoff 정보 (없으면 미부착)
  • metadata.routedBy: supervisor의 경우 "regex-fast-path" | "llm" | "rules-fallback" (Iter12)

3.4 단일 publish 보장

NATS publish 지점은 handleRequest 끝 한 곳뿐. 4 hop 재귀라도 사용자는 응답 한 번만 받음.


4. Skill 라우팅

4.1 Skill 등록 (bootstrap)

src/main.ts:

1. SkillRegistry.load("src/skills")
   - 각 폴더의 SKILL.md frontmatter 파싱 → SkillManifest 배열
2. factories map = { onboarding, moderation, supervisor }
3. 등록 루프: agent.registerSkill(factory({ ...deps }))
   - deps: garnetClient, llmClient, roomContextService, shadowEvalBus,
           natsClient, memoryService, escalationHistoryService, ...
4. TransitionMatrix.fromRegistry(skillRegistry) → Agent에 주입

4.2 SKILL.md frontmatter (capability card)

각 src/skills/<name>/SKILL.md:

name: onboarding
description: ...
version: 1
priority: 70           # canHandle 공동 당첨 시 desc 정렬
intents: [greeting, ..., age-question]
triggers: [user-message, new-member, ...]
tools: [validate-nickname, ...]
resources:
  nickname-format: ./resources/nickname-format.md
  ...
trustTier: T3          # T1(가장 신뢰) ~ T4(가장 보수적)
domain: onboarding              # Iter4 capability card
canHandle: [...]
cannotHandle: [...]
handoffTargets: [moderation, supervisor]
riskTier: low | medium | high
outputContract:
  format: text
  maxChars: 600

4.3 라우팅 결정 — selectSkillForRequest

src/agent/agent.ts:28-37:

1. skills.filter(s => s.canHandle(request))
   - 각 Skill의 canHandle: 보통 request.skill === self.name (정확 매칭)
2. priority desc 정렬
3. [0] 반환
  • onboarding: request.skill === "onboarding" (priority 70)
  • moderation: request.skill === "moderation" (priority 50)
  • supervisor: request.skill === "supervisor" OR fallback (priority 10)

4.4 TransitionMatrix (handoff 그래프)

src/skills/transition-matrix.ts. 각 capability card의 handoffTargets를 모아 그래프 구성. supervisor는 모든 항목의 fallback 자동 포함.

isAllowed(from, to)    # true/false
targetsOf(from)        # 허용 target 집합
validateGraph()        # 빌드타임 일관성

Dispatch loop에서 handoff 시 transitionMatrix.isAllowed() 검사. 거부 시 강제 supervisor로 escalation.

4.5 검증 — npm run lint:skills

src/skills/validate-manifests.ts:

  • intents 배열 ⊆ 코드 IntentKind (manifest-binding.ts에서 derive)
  • tools 배열 == ToolRegistry tool names
  • resources 키들 실제 파일 존재
  • priority 중복 0
  • handoffTargets 항목이 실제 skill name 또는 supervisor
  • cannotHandle ∩ canHandle = ∅

매 PR 시 CI에서 lint:skills 통과 필수.


5. Memory 계층

자료의 강조점 "context engineering > prompt engineering" + MemoryAgentBench 4축 (accurate retrieval / test-time learning / long-range understanding / selective forgetting) 모두 구현.

5.1 3-layer 인터페이스 (Iter10)

src/memory/memory-types.ts:

WorkingMemory   // 매 turn 조립. dispatch loop가 사용
SessionMemory   // 현 대화 세션. Garnet 12h TTL
LongTermMemory  // 영속. Garnet (profile) + MongoDB (knowledge)

5.2 SessionMemory (Garnet, 12h~24h)

데이터 Garnet key TTL 소스
Blackboard (skill-local) lolympus-agent:bb:{skill}:{chatroom}:{user} 12h src/skills/onboarding/state/blackboard.ts
Conversation history lolympus-agent:conv:{chatroom}:{user} 12h src/conversation/conversation-store.ts, max 20 메시지
Conversation summary lolympus-agent:summary:{chatroom}:{user} 24h src/conversation/summarizer.ts, 5+ 메시지 trigger

5.3 LongTermMemory (Garnet + MongoDB, 30d+)

데이터 저장소 TTL
UserProfile (preferredStyle, pastIntents, flaggedIssues, longTermSummary) Garnet lolympus-agent:user:profile:{user} 30d (env: AGENT_USER_MEMORY_TTL_DAYS)
KnowledgeDocuments (RAG) MongoDB collection knowledge_documents 영속 (eviction policy로 archive)
UserEscalationHistory MongoDB collection user_escalation_history 영속

UserProfile 필드 (src/skills/onboarding/state/user-memory.ts):

userId: string                                // anonymizeText() 처리
firstSeenAt, lastSeenAt: string
completedOnboardingAt?: string
totalTurns: number
pastIntents: Array<{kind, count}>            // 최근 10개
preferredStyle: "warm" | "concise" | "neutral"  // 자동 추론
flaggedIssues: string[]                       // 최대 10
longTermSummary?: { text, updatedAt, turnSnapshot }  // Iter11

5.4 WorkingMemory 조립

매 요청 시 message-builder.ts에서 호출:

const wm = await memoryService.assembleWorkingMemory({
  skill: "onboarding",
  chatroomId, userId,
  includeBlackboard: true,
  includeProfile: true,
  retrieval: shouldRetrieve(derived)
    ? { query: ..., options: { skill, category, topK: 2 } }
    : undefined,
});
// wm: { conversationHistory, summary, blackboard, userProfile, retrievedKnowledge }

shouldRetrieve(derived) (Iter10 Phase D): help-how, help-where, nickname-question, discord-question, age-question 5개 intent만 RAG. greeting/acknowledgement는 skip(비용 절감).

5.5 LLM prompt 주입 블록

message-builder.ts + prompt-data.ts:

[reference]                ← retrieval된 knowledge chunk
[user_context]             ← test-time learning snippets (자주 묻는 intent 등)
[long_term_memory]         ← UserProfile.longTermSummary (Iter11)
[user_escalation_history]  ← supervisor capability-prompt에만 (internal only)

6. Agentic RAG

자료의 "modular > monolithic" + "retrieval은 pre-filter → rerank → compression" 권고 적용.

6.1 Embedding (src/memory/embedding-client.ts)

추상화 인터페이스 + 2개 구현 (Iter10):

interface EmbeddingClient {
  embed(text): Promise<number[]>
  embedBatch(texts): Promise<number[][]>
  model: string
  dimensions: number
}

createEmbeddingClient()
  // env: EMBEDDING_PROVIDER ("openrouter" | "openai")
  // OpenRouter 우선 → 미지원 시 OpenAI fallback
  // env: EMBEDDING_MODEL (예: "text-embedding-3-small")
  // env: EMBEDDING_API_KEY 또는 OPENROUTER_API_KEY

6.2 Ingest — npm run ingest:knowledge

scripts/ingest-knowledge.ts → src/memory/knowledge-ingest.ts:

  1. src/skills/{onboarding,moderation}/resources/*.md 5개 파일 로드
  2. ## heading 단위로 split
  3. 각 chunk를 { skill, category, text, metadata: { sourceFile, sectionTitle } }로 build
  4. embeddingClient.embed(text) → vector
  5. MongoDB knowledge_documents collection upsert (metadata.sourceFile + sectionTitle 기준 idempotent)
  6. embeddingModel 필드 자동 기록 (마이그레이션 식별용)

총 20~50 chunks 예상. 향후 1000+ 시 MongoDB Vector Search 마이그레이션.

6.3 Retrieval pipeline (Iter10 → Iter11)

src/memory/mongo-long-term-memory.ts:search():

1. embeddingClient.embed(query) → query vector
2. MongoDB find with { skill?, category?, metadata.archived: { $ne: true } }
   → 후보 N개 (env: AGENT_RAG_CANDIDATE_POOL_SIZE, 기본 50)
3. (옵션) Hybrid (Iter11 Phase A)
   - cosine score (in-app, src/memory/cosine.ts)
   - BM25 score (in-app, src/memory/bm25.ts)
   - 가중 평균 (각 0.5)
4. top-K 선정 (기본 10)
5. (옵션) Reranker (Iter11 Phase A)
   - LlmReranker (tier 0, maxTokens 200, jsonMode)
   - 실패 시 원래 순서 (graceful)
6. top-N 반환 (기본 5, env: AGENT_RAG_RERANKER_TOP_K)

각 단계에서 Phoenix span attribute:

  • memory.retrieval.candidate_count
  • memory.retrieval.top_score
  • memory.retrieval.duration_ms
  • memory.reranker.candidates_count
  • memory.reranker.changed_order (boolean)

6.4 Selective forgetting (Iter11 Phase C)

src/memory/forgetting-policy.ts + src/memory/contradiction-detector.ts:

3가지 규칙:

  • Relevance: 30일간 reference 0회 + referenceCount30d < 3 → archive
  • Freshness: updatedAt 90일 이상 + 사용 빈도 낮음 → archive
  • Contradiction: 같은 sourceFile + sectionTitle 그룹 내 텍스트 30%+ 차이 또는 명시적 키워드 → 기존 archive, 새 doc primary (staging 필수)

scripts/memory-eviction-cron.ts (npm run memory:evict --dry-run):

  • archived는 metadata.archived = true flag만 (delete 아님). 복구 가능
  • search() 자동으로 archived 제외

6.5 사용 site

Moderation lookup-rule tool (Iter10 Phase C):

args: { topic: string, topK?: number }
→ memoryService.longTerm.search(topic, { skill: "moderation", category: "rule" })
→ tool result에 chunk text 포함
→ LLM이 retrieval된 reference로 답변

Onboarding message-builder.ts (Iter10 Phase D):

  • 5개 intent에서만 retrieval
  • [reference] block에 chunk 주입
  • LLM이 ground truth 기반 답변

7. Tool 실행

7.1 Tool 인터페이스 (Iter12 Phase B)

src/tools/tool.ts:

interface Tool<A, R> {
  name: string
  description: string
  argsSchema: ZodSchema<A>
  resultSchema: ZodSchema<R>
  requires?: readonly ToolPermission[]   // Iter12: 권한 declaration
  execute(args: A, ctx: ToolContext): Promise<ToolResult<R>>
}

type ToolPermission =
  | "garnet" | "nats" | "room-context"
  | "user-session" | "tool-registry"
  | "memory-search" | "memory-profile"

7.2 Scoped Context Injection (Iter12 Phase B)

src/tools/scoped-context.ts:

buildScopedContext(tool.requires ?? [], fullContext) → scopedContext
  • 권한 없는 필드 접근 시:
    • AGENT_TOOL_SANDBOX_ENABLED=true: throw
    • AGENT_TOOL_SANDBOX_ENABLED=false (log-only): warn 로그만

7.3 등록된 tool

Tool Skill requires
validate-nickname onboarding [] (pure)
check-onboarding-progress onboarding ["garnet", "user-session"]
lookup-community-rule onboarding ["memory-search"]
get-room-context onboarding ["room-context"]
verify-discord-nickname onboarding ["garnet", "nats", "user-session"]
lookup-rule moderation ["memory-search"]

7.4 Tool Audit (Iter12 Phase B)

src/tools/audit.ts + Garnet stream lolympus-agent:tool-calls:

매 tool 호출 후 envelope append:

{
  type: "tool-call",
  toolName, chatroomId, userId,
  argsHash: string,         // PII 마스킹된 hash (원본 args 미저장)
  resultSummary, ok, errorKind?,
  durationMs, skill, traceId, calledAt
}
  • maxLen 100_000 자동 trim
  • Operations CLI: npm run tool-audit:list, npm run tool-audit:show <id>

7.5 외부 통합 (lolympus 도메인)

  • Discord: verify-discord-nickname tool이 NATS subject lolympus.discord.lookup-members로 외부 microservice에 위임 (HTTP 직접 호출 X)
  • Kakao: session에 nickname 정보만 (외부 API 호출 없음, 클라이언트 제공)

8. Handoff + Supervisor (Bounded Self-Escalation Graph)

자료의 "Bounded Self-Escalation Graph" 8가지 규칙 모두 적용 (Iter4-6).

8.1 HandoffRequest schema (Iter4)

src/agent/types.ts:

interface HandoffRequest {
  targetAgent: string
  reason: HandoffReason         // 8가지 enum
  taskSpec: string
  evidenceRefs: string[]        // toolCall id 또는 messageId
  confidence: number            // 0..1
  risk: "low" | "medium" | "high"
  expectedOutput: string
  userVisibleMessage?: string
}

type HandoffReason =
  | "out-of-scope" | "missing-tool" | "low-confidence"
  | "validation-failed" | "same-agent-revisit"
  | "max-hop" | "high-risk-action" | "policy-risk"

8.2 [HANDOFF:..] 토큰 파이프라인 (Iter4 Phase D)

기존 [NO_RESPONSE] 패턴을 확장. 5단계:

  1. Prompt — prompt-data.ts persona에 사용법 명시 (capability card 기반 자동 합성)
  2. Abstention — runtime/abstention.ts의 AbstentionAction에 {kind:"handoff", targetSkill, reason} variant
  3. Validator — runtime/validator.ts:142+에서 [HANDOFF:..] token pass
  4. Text generator — runtime/text-generator.ts:79+에서 {status:"handoff", request} 반환
  5. Skill.handle — makeEmptyResponse + metadata.handoffRequest 반환

Parser: src/skills/control-token-parser.ts (regex /\[HANDOFF:([a-z0-9_-]+):([a-z-]+)\]/giu).

8.3 HandoffPacket (Iter4 Phase E)

src/agent/handoff-packet.ts. PII 차단 minimal context:

{
  summary: string                  // src/conversation/summarizer.ts 결과
  userGoal: string                 // 240자 제한, PII 제거
  currentState: Record             // phase/responseMode/tier만 (userMessage 원문 미포함)
  handoffReason: HandoffReason
  evidence: HandoffEvidence[]      // toolCalls[].id
  constraints: string[]
  expectedOutputSchema: string
  sourceChain: string[]            // 거쳐간 skill 경로
}

다음 dispatch hop에서 request.context.handoffPacket으로 전달. 수신 skill은 readHandoffPacket(request) 호출.

8.4 BudgetTracker (Iter4 Phase F + Iter12 Phase A)

src/agent/budget.ts:

한도 기본 env
MAX_HANDOFFS 4 AGENT_MAX_HANDOFFS
MAX_SAME_AGENT_VISITS 2 AGENT_MAX_SAME_AGENT_VISITS
MAX_TOTAL_TOOL_CALLS 20 AGENT_MAX_TOOL_CALLS
MAX_COST_USD 0.5 AGENT_MAX_COST_USD

assertBeforeHandoff(target) verdict: "ok" | "max-hops" | "same-agent-loop" | "tool-budget" | "cost-budget".

Iter12 핵심: recordLlmCost(costUsd) — LlmClient 호출 끝에서 자동 hook (context.budget?.recordLlmCost(cost)). snapshot에 totalLlmCostUsd, totalToolCostUsd 분리 노출.

8.5 Supervisor (Iter4 Phase G + Iter6 + Iter12 Phase A)

src/skills/supervisor/skill.ts. LLM + 룰 기반 hybrid:

1. readHandoffPacket(request)
2. routeByRegex(packet) → { target, confidence }
3. confidence ≥ 0.85 (env: SUPERVISOR_FAST_PATH_CONFIDENCE)
   → LLM skip. metadata.routedBy = "regex-fast-path"   ← Iter12 G 패턴
4. 그 외 → routeByLlm(packet):
   - capability-prompt에 모든 skill capability card 자동 합성
   - + escalation history (Iter11) [user_escalation_history] block
   - tier 0, maxTokens 150, jsonMode
   - { target, confidence, reason } JSON 응답
5. LLM 실패 → 정규식 fallback. metadata.routedBy = "rules-fallback"
6. appendSupervisorReview() → review queue + audit
7. AlertService.sendAlert (NATS + Slack, Iter7)
8. 응답 텍스트: 안정성 위해 template 유지 (Iter6 결정)

8.6 Runtime Evaluator (Iter4 Phase H)

SkillJudgeConfig.runtimeVerifier?: (input) => "pass" | "retry" | "escalate":

  • pass → 진행
  • retry → 1회 재호출 (trigger="retry", MAX_RUNTIME_RETRIES=1)
  • escalate → supervisor 강제 handoff (reason="validation-failed")

Onboarding 기본 verifier: validateOnboardingResponse. Moderation: PII/외부 링크 차단 regex.


9. Cost-aware Tier Selection (Iter12 Phase A)

자료 G 패턴 + OpenInference cost 활용.

9.1 Tier 구조

Tier 용도 기본 모델 maxTokens env
0 분류, 짧은 응답, supervisor LLM, reranker gpt-5.3-codex-spark 384 TIER0_MODEL, TIER0_MAX_TOKENS
1 planner / responder gpt-5.4 / gpt-5.3 512 TIER1_PLANNER_MODEL, TIER1_RESPONDER_MODEL
2 critic gpt-5.4 512 TIER2_CRITIC_MODEL

9.2 Router 결정 입력 (Iter12 확장)

src/skills/onboarding/runtime/router.ts:decidePreGenerationRouting():

input: {
  classified: { intent, confidence },
  derived,
  classifyConfidence: number,              // Iter12
  intentRiskTier?: "low"|"medium"|"high",  // Iter12 (capability card)
  userEscalationCount?: number,            // Iter12 (escalation history)
  recentLatencyP95Ms?: number              // Iter12 (선택)
}

룰:

  • confidence ≥ 0.9 + low-risk → tier 0 (cost 최소)
  • confidence < 0.5 또는 high-risk + escalation > 3 → tier 1 (quality 우선)
  • p95 latency SLA 위반 추세 → tier 0 (degradation 보호)

9.3 LLM cost 추적 (Iter9 + Iter12)

src/llm/llm-client.ts:

const costUsd = estimateChatCost(model, prompt_tokens, completion_tokens);
metrics.tokenCostUsd.observe(costUsd, { model, skill: getContext()?.skill });
getContext()?.budget?.recordLlmCost(costUsd);   // Iter12
// Phoenix span: gen_ai.usage.cost_usd = costUsd

estimateChatCost (src/llm/cost.ts): 모델별 input/output rate 테이블.

9.4 환경 변수 (cost-aware)

env 기본 효과
AGENT_COST_AWARE_TIER_ENABLED true confidence 기반 tier 결정
AGENT_BUDGET_LLM_COST_ENABLED true LlmClient → Budget 자동 hook
AGENT_SUPERVISOR_FAST_PATH_ENABLED true regex 명확 시 LLM skip
SUPERVISOR_FAST_PATH_CONFIDENCE 0.85 임계값

10. Evaluation 파이프라인

자료의 "평가 가능한 실행 환경 없이 데모는 의미 없음" 권고. Iter3에서 skill-aware로 전환됨.

10.1 Golden set 구조 (Iter2 + 후속)

src/eval/golden/
├── onboarding/
│   ├── cases.jsonl          # 활성 golden, ~11개
│   ├── candidate.jsonl      # 자동 수집 후보 (promote 후보)
│   └── capture.jsonl        # 실시간 prod 캡처 buffer
├── moderation/
│   └── cases.jsonl          # 5개 (Iter2)
├── handoff/
│   └── cases.jsonl          # 6개 (Iter5)
└── memory/
    └── cases.jsonl          # 6개 (Iter11, 4축 × 1~2개)

EvalCase 구조 (src/eval/types.ts):

{
  id, name, notes,
  input: { request, conversationHistory, roomContext, session, blackboard, summary },
  expected: {
    intent?, mustIncludeAny?, mustNotInclude?, responseMode?,
    processExpected?: { expectedTools, expectedApproach, minSteps, toolMode },
    expectedHandoff?: { shouldHandoff, expectedReason, expectedTarget },  // Iter5
    expectedFinalSkill?: string,
    expectedSourceChain?: { mode: "exact"|"prefix"|"contains", chain },
    expectedMemory?: MemoryJudgeExpected   // Iter11
  }
}

10.2 Runner 흐름 (src/eval/runner.ts)

1. resolveEvalSetPath(setName)
   "onboarding/cases" → src/eval/golden/onboarding/cases.jsonl
   "golden" → legacy alias for "onboarding/cases"
2. parseEvalCaseLine() → EvalCase[]
3. for each case:
   a. SKILL_RUNNER_FACTORIES[skill]({deps}) → Skill instance
   b. handoff/cases set이면 runEvalDispatch (mini-dispatcher, Iter5)
      그 외엔 skill.handle() 직접 호출
   c. judges 실행:
      - rule-judge (mustIncludeAny, mustNotInclude, intent)
      - llm-judge (skill별 평가 축, SkillJudgeConfig)
      - process-judge (tool trace 검증)
      - handoff-judge (chain, reason, target) — handoff set만
      - memory-judge (retrieval/long-range/test-time/forgetting) — memory set만
   d. summary 집계
4. baseline (eval-reports/baseline.{skill}.json) 비교
5. eval-reports/<ts>.json 저장

10.3 Judges (src/eval/judges/)

Judge 책임 Iter
rule-judge.ts mustIncludeAny / mustNotInclude / intent 매칭 Iter3 일반화
llm-judge.ts skill별 평가 축. SkillJudgeConfig.axes Iter3
process-judge.ts tool trace, skill별 tool registry Iter3
handoff-judge.ts dispatch chain 검증 Iter5
memory-judge.ts retrieval 정확도, long-range 회상 등 Iter11
attribution.ts slice 기반 root cause (이전)

SkillJudgeConfig (src/eval/judges/skill-config.ts):

{
  onboarding: {
    axes: ["friendliness", "accuracy", "concision", "scope"],
    toolRegistryFactory: createOnboardingToolRegistry,
    validator: validateOnboardingResponse,
    promptTemplate, tournamentInstruction,
    runtimeVerifier?: (input) => "pass"|"retry"|"escalate"
  },
  moderation: {
    axes: ["appropriateness", "accuracy", "helpfulness"],
    toolRegistryFactory: createModerationToolRegistry,
    ...
  }
}

10.4 자동 개선 루프

[1] Capture (shadow-eval-bus) — prod 트래픽 sample → shadow envelope
                                    ↓
[2] Run — golden cases 실행, judges 적용, eval-reports/<ts>.json
                                    ↓
[3] Propose (eval:propose CLI) — 저점수 slice → LLM이 prompt variant 생성
                                  → src/eval/proposals/<variantId>.json
                                    ↓
[4] Tournament (eval:tournament) — control vs variant A/B, ELO 갱신
                                    ↓
[5] Governance gates (G1~G4) — trust tier 검증
                                    ↓
[6] Promote (eval:promote) — golden 후보 → 활성 cases

10.5 변형 적용 (PROMPT_VARIANT_ID)

src/skills/onboarding/prompts/builder.ts:

  • loadPromptVariantManifest(variantId) → variant JSON 로드
  • applyLayerDiff(layers, diff) — persona/policy/state/context/examples/tool-policy 6 layer 중 일부 replace/append/remove
  • env PROMPT_VARIANT_ID=<id> → 프로덕션에서 활성

10.6 Calibration / Drift / Governance

  • src/eval/aggregator.ts — SliceMetrics (intent/phase/tier 차원)
  • src/eval/drift.ts — baseline 대비 점수 하락 감지
  • src/eval/governance/trust.ts — T1~T4 enum
  • src/eval/governance/gates.ts — variant 승격 4 gate (G1~G4)
  • src/eval/router-shadow-report.ts (Iter6) — learned vs static routing divergence

11. Memory 평가 (Iter11)

src/eval/golden/memory/cases.jsonl 6 cases:

# 이름 검증
1 retrieval-accurate "광고 도배" 질문 → moderation lookup-rule retrieval 정확 chunk
2 long-range-understanding 5턴 전 "닉네임 LolName_KR" → 회상
3 test-time-learning flaggedIssues 3회 누적 → prophylactic guidance
4 selective-forgetting 신규 doc 30% 차이 + "정정" 키워드 → 기존 archive
5 escalation-history 사용자 3회 광고 시도 → supervisor가 moderation 우선 라우팅
6 memory-contradiction "닉네임 X로 변경" → profile 반영, 다음 응답이 새 닉네임 인지

MemoryJudgeResult (src/eval/judges/memory-judge.ts):

  • expectedRetrievedDocIds, expectedRecalledFact, expectedProphylacticHint, expectedArchivedDocPattern, expectedSupervisorTarget, expectedContradictionResolved

명령: npm run eval -- --set=memory/cases.


12. Operations — 운영 루프

12.1 Review Queue 두 종류

Queue Garnet stream 목적 진입
Eval Review lolympus-agent:eval:review:queue 학습용 (저점수/희귀 slice/rule-llm 불일치) runner가 자동 enqueue
Supervisor Handoff lolympus-agent:review-queue 운영용 (escalation case 사람 검토) supervisor.skill.ts

12.2 Operations CLI (Iter5 + Iter12)

src/operations/cli.ts:

# Supervisor handoff queue
npm run review:list -- --limit=20 --since=24h
npm run review:show -- <id> [--history]      # --history는 Iter6 audit log
npm run review:resolve -- <id> --action=manual-reply --note="..."
npm run review:stats -- --since=7d

# Tool audit (Iter12)
npm run tool-audit:list -- --skill=onboarding --since=1h --limit=50
npm run tool-audit:show -- <stream-id>

12.3 Resolution audit log (Iter6)

이중 저장:

  • Hash lolympus-agent:review-resolved:<id> — latest state (덮어쓰기, 7d TTL)
  • Stream lolympus-agent:review-resolutions — append-only (운영자 변경 이력)

getResolutionHistory(garnet, id) — 사용자별 resolution 변경 이력 조회 (Iter11 escalation history 입력).

12.4 Alert flow (Iter7)

src/observability/alert.ts:AlertService.sendAlert(payload):

1. Garnet dedupe key 검사 (5분 throttle)
   key: lolympus-agent:alert:dedupe:<kind>:<skill>
2. NATS publish → AGENT_ALERT_SUBJECT
3. Slack webhook POST (env: SLACK_ALERT_WEBHOOK_URL)
   - fire-and-forget, AbortSignal.timeout 3초
   - block kit message (severity emoji + 필드)
4. logger.warn (구조화 로그)

AlertPayload:

{
  kind: "archive-stalled"|"endpoint-down"|"eval-drift"|"eval-score-drop"
       |"latency"|"llm-error-rate"|"supervisor-handoff",
  metric, severity, skill?, threshold, value, window, timestamp, ...
}

12.5 Slack message 포맷 (Iter7)

src/observability/slack-notifier.ts:formatSlackMessage():

  • severity emoji (:information_source:/:warning:/:rotating_light:)
  • Section + fields (metric / value vs threshold / window / skill)
  • suggestedAction (있다면)

13. Observability

13.1 Logger (src/observability/logger.ts)

  • Pino + AsyncLocalStorage
  • getLogger() → context의 child logger 자동 (skill, requestId, chatroomId, userId)
  • redact: authorization, env.<secret>, secrets.<secret>, request.<secret> (KNOWN_SECRET_KEYS 확장됨, Iter12)

13.2 Tracing (Iter9)

src/observability/tracing.ts:

  • OTel NodeSDK + auto-instrumentation
  • Exporter: env PHOENIX_OTLP_ENDPOINT 설정 시 OTLP HTTP, 미설정 시 ConsoleSpanExporter (또는 OFF, env: OTEL_EXPORTER)
  • Sampling: OTEL_SAMPLING_RATIO (prod 0.1, dev 1.0)

src/observability/spans.ts helper:

  • withSkillSpan(skill, attrs, fn) — skill.<name>.handle
  • withSpan(name, attrs, fn) — generic + auto-error
  • withClassifierSpan, withToolSpan, withLlmSpan

13.3 LLM Span 속성 (Iter9 OpenInference)

src/llm/llm-client.ts:

gen_ai.request.model
gen_ai.request.temperature
gen_ai.request.max_tokens
gen_ai.system                        (provider name)
gen_ai.response.id
gen_ai.response.finish_reasons[]
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.cost_usd                ← Iter9, Iter12에서 routing 활용
llm.token_count.prompt               ← Phoenix 친화 alias
llm.token_count.completion
gen_ai.prompt.0.role/content         ← optional (PHOENIX_LOG_PROMPTS)
gen_ai.completion.0.content          ← optional + PII 마스킹

Memory 관련 (Iter10-11):

  • memory.retrieval.candidate_count
  • memory.retrieval.top_score
  • memory.retrieval.duration_ms
  • memory.reranker.changed_order

Handoff 관련:

  • agent.handoff_depth
  • agent.handoff_chain
  • supervisor.routed_by

13.4 Metrics (Iter4 Phase I)

자체 구현 (src/observability/metrics.ts). /metrics HTTP endpoint (metrics-server.ts, port 기본 9464). 10 counter/histogram:

Metric 타입 Labels
agent_handoff_total counter from, to, reason
agent_handoff_loop_total counter from
agent_handoff_count histogram (0..4) —
agent_human_override_total counter kind
agent_missed_escalation_total counter —
agent_routing_accuracy_total counter result
agent_unnecessary_escalation_total counter —
agent_validator_fail_total counter skill, stage
agent_token_cost_usd histogram model, skill
agent_time_to_resolution_ms histogram skill

/healthz endpoint: 200 "ok" (Iter7).

13.5 Alert threshold (skill별, Iter7)

src/config.ts:

getAlertThresholdP95LatencyMs(skill?: string)
getAlertThresholdEvalScoreDropPct(skill?: string)
getAlertThresholdLlmErrorRate(skill?: string)
getAlertThresholdDriftPct(skill?: string)

env 패턴: ALERT_THRESHOLD_<METRIC>_<SKILL_UPPER> 우선 → ALERT_THRESHOLD_<METRIC> fallback.

예: ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING=5000 또는 글로벌 ALERT_THRESHOLD_P95_LATENCY_MS=3000.


14. 보안

14.1 Secret Manager (src/common/secret-manager.ts)

계층:

  1. Infisical (외부 secret manager, INFISICAL_SITE_URL)
  2. AWS IMDS (EC2 메타데이터 fallback)
  3. 환경변수 (마지막 fallback)

KNOWN_SECRET_KEYS (src/common/known-secret-keys.ts, Iter12 확장):

INFISICAL_*, NATS_*, GARNET_*, MONGODB_*, SLACK_ALERT_WEBHOOK_URL,
OPENROUTER_API_KEY, AGENT_LLM_API_KEY, OPENAI_API_KEY,
EMBEDDING_API_KEY, PHOENIX_API_KEY

getSecret(key): KNOWN_SECRET_KEYS 검증. getRawSecret(key): 임의 키 (운영 도구용).

14.2 NATS Subject ACL (Iter12 Phase C)

src/common/nats-skill-acl.ts:DEFAULT_NATS_ACL:

[
  {
    skill: "onboarding",
    allowedPublishSubjects: ["lolympus.discord.lookup-members"],
    allowedSubscribeSubjects: []
  },
  { skill: "moderation", allowedPublishSubjects: [], ... },
  { skill: "supervisor", allowedPublishSubjects: ["lolympus.agent.alert"], ... }
]

src/common/nats.ts의 publish wrapper:

  • getNatsAclMode():
    • "off": 검사 skip
    • "log-only" (기본): warn 로그, 정상 publish
    • "enforce": throw + alert

운영 흐름: log-only 1개월 → false positive 없으면 enforce 전환 (Iter13+ task).

14.3 PII 차단

  • HandoffPacket: userMessage 원문 포함 X. userGoal은 240자 + anonymize
  • Memory user-profile: anonymizeText() 처리
  • Tool audit: argsHash만 저장, 원본 args X
  • Phoenix prompt content: PHOENIX_LOG_PROMPTS=false 기본
  • escalation history: supervisor system prompt 내부만, 응답 텍스트에 노출 금지

15. 환경 변수 + 배포

15.1 인프라

env 용도
NATS_ENDPOINT, NATS_USER, NATS_PASS NATS 연결
GARNET_ENDPOINT, GARNET_USERID, GARNET_PASSWORD Garnet 연결
MONGODB_CONNECTION_STRING, MONGODB_PROJECT_ID MongoDB (optional)
INFISICAL_CLIENT_ID, INFISICAL_CLIENT_SECRET Secret manager

15.2 모델

env 기본 용도
TIER0_MODEL gpt-5.3-codex-spark 분류, supervisor LLM, reranker
TIER1_PLANNER_MODEL, TIER1_RESPONDER_MODEL gpt-5.4, gpt-5.3 메인 응답
TIER2_CRITIC_MODEL gpt-5.4 critic
TIER0_MAX_TOKENS 384
EMBEDDING_PROVIDER openrouter OpenRouter→OpenAI fallback
EMBEDDING_MODEL text-embedding-3-small
EMBEDDING_API_KEY — (또는 OPENROUTER/OPENAI 키 재사용)

15.3 기능 flag (Iter별)

env 기본 Iter
AGENT_ROUTER_MODE static Iter3
AGENT_RAG_ENABLED true Iter10
AGENT_RAG_HYBRID_ENABLED true Iter11
AGENT_RAG_RERANKER_ENABLED true Iter11
AGENT_RAG_CANDIDATE_POOL_SIZE 50 Iter11
AGENT_RAG_RERANKER_TOP_K 5 Iter11
AGENT_LONG_TERM_SUMMARY_ENABLED true Iter11
AGENT_MEMORY_EVICTION_ENABLED false (cron만) Iter11
AGENT_COST_AWARE_TIER_ENABLED true Iter12
AGENT_BUDGET_LLM_COST_ENABLED true Iter12
AGENT_SUPERVISOR_FAST_PATH_ENABLED true Iter12
AGENT_TOOL_SANDBOX_ENABLED false (1주 검증 후 true) Iter12
AGENT_NATS_ACL_MODE log-only Iter12
AGENT_SUPERVISOR_LLM_ENABLED true Iter6
AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED false (template 우선) Iter8

15.4 한도 (Budget)

env 기본
AGENT_MAX_HANDOFFS 4
AGENT_MAX_SAME_AGENT_VISITS 2
AGENT_MAX_TOOL_CALLS 20
AGENT_MAX_COST_USD 0.5

15.5 임계값 (Alert)

env (skill suffix optional) 기본
ALERT_THRESHOLD_P95_LATENCY_MS 3000
ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT 10
ALERT_THRESHOLD_LLM_ERROR_RATE 0.1
ALERT_THRESHOLD_DRIFT_PCT 15
ALERT_RATE_LIMIT_SEC 300

15.6 Observability

env 기본
PHOENIX_OTLP_ENDPOINT unset (console fallback)
PHOENIX_LOG_PROMPTS false
SLACK_ALERT_WEBHOOK_URL unset (Slack 비활성)
SLACK_WEBHOOK_TIMEOUT_MS 3000
METRICS_PORT 9464
OTEL_SAMPLING_RATIO 0.1 (prod)

15.7 Bootstrap (src/main.ts)

순서:

1. initLogger()
2. initTracing() (Phoenix 또는 console)
3. startMetricsServer() (port 9464)
4. SecretManager.load (Infisical + IMDS)
5. NatsClient.connect
6. GarnetClient.connect
7. initMongoDB (optional)
8. SkillRegistry.load("src/skills")
9. TransitionMatrix.fromRegistry
10. ConversationStore + ShadowEvalBus
11. memoryService, escalationHistoryService, alertService (Slack notifier)
12. factories map { onboarding, moderation, supervisor }
13. for manifest in registry: agent.registerSkill(factory(deps))
14. agent.start() (NATS subscribe)
15. shutdown hooks

16. 도메인 데이터 (Lolympus 카카오톡 오픈채팅)

16.1 Onboarding (src/skills/onboarding/)

Capability card:

  • priority: 70, trustTier: T3, riskTier: low
  • domain: onboarding
  • handoffTargets: [moderation, supervisor]

Intent (intents.ts, 12개):

  • greeting, acknowledgement
  • status-question
  • help-how, help-where
  • nickname-question, photo-question
  • discord-question, discord-verify-question
  • age-question
  • chitchat, off-topic-to-other, unknown

Tools (5개): validate-nickname, check-onboarding-progress, lookup-community-rule, get-room-context, verify-discord-nickname

Resources (4개 .md): nickname-format, heart-rule, discord-format, age-policy

핵심 로직:

  • OnboardingIntentClassifier (3-stage: trigger → regex → LLM)
  • deriveState(session, trigger, intent) → phase + responseMode + tier
  • runtime/: orchestrator, router, validator, critic, tool-plan-runner, message-builder, text-generator, abstention 등 ~10개 모듈

16.2 Moderation (src/skills/moderation/)

Capability card:

  • priority: 50, trustTier: T4, riskTier: medium
  • domain: moderation
  • handoffTargets: [onboarding, supervisor]

Intent (5개): greeting, rule-question, violation-report, appeal, off-topic

Tools (1개): lookup-rule (Iter10에서 RAG 기반으로 전환)

Resources (1개 .md): community-rules (욕설, 광고, 도배, 신고 절차, 제재, 이의제기, 미성년자, 개인정보 등 섹션)

16.3 Supervisor (src/skills/supervisor/)

Capability card:

  • priority: 10 (최저, fallback 전용), riskTier: medium
  • domain: meta
  • canHandle: ["*"]
  • handoffTargets: [onboarding, moderation]

핵심:

  • capability-prompt.ts — 모든 skill의 capability card를 system prompt로 자동 합성
  • routing-rules.ts — regex 기반 키워드 매칭 (Iter12 fast-path)
  • review-queue.ts — appendSupervisorReview + getResolutionHistory + 도메인 escalation history 연동
  • 응답 텍스트: 안정성 위해 template ("운영자에게 전달했어요")

16.4 User Escalation History (Iter11 — lolympus 차별점)

user_escalation_history collection (MongoDB):

{
  userId, chatroomId,
  escalations: [{
    queuedAt, reason, target,
    resolution: { resolvedAt, action, note },
    packetSummary, sourceChain
  }],
  totalEscalations, lastEscalationAt,
  topReasons: [{ reason, count }]
}

Supervisor가 매 escalation 시 history 조회 → capability prompt의 [user_escalation_history] 블록 주입:

"이 사용자는 광고 시도 3회 (out-of-scope) → 다음 위반 시 즉시 차단 권장" "닉네임 검증 5회 실패 → 친절한 step-by-step 우선"


17. 명령어 cheat sheet

17.1 개발

npm run typecheck                           # TypeScript 검증
npm run test                                # Vitest 전체
npm run test -- --reporter=verbose         # 상세
npm run lint                                # ESLint
npm run lint:skills                         # SKILL.md ↔ 코드 일관성
npm run build                               # tsc + tsc-alias
npm run dev                                 # 로컬 부트스트랩

17.2 Evaluation

# Run
npm run eval -- --set=onboarding/cases
npm run eval -- --set=moderation/cases
npm run eval:handoff                        # = --set=handoff/cases
npm run eval -- --set=memory/cases          # Iter11

# Baseline / Report
npm run eval:baseline
npm run eval:report

# 자동 개선 loop
npm run eval:capture
npm run eval:propose -- --slice="tier=0,intent=help-how,phase=initial,skill=onboarding"
npm run eval:tournament -- --skill=onboarding
npm run eval:promote
npm run eval:calibrate -- --check
npm run eval:router-shadow-report -- --since=7d     # Iter6

17.3 Operations (Iter5 + Iter12)

# Supervisor review queue
npm run review:list
npm run review:show -- <id>
npm run review:show -- <id> --history       # Iter6 resolution audit
npm run review:resolve -- <id> --action=manual-reply --note="..."
npm run review:stats

# Tool audit
npm run tool-audit:list
npm run tool-audit:show -- <id>

17.4 Memory (Iter10-11)

npm run ingest:knowledge                    # resources/*.md → MongoDB embedding
npm run memory:evict -- --dry-run --since=30d
npm run memory:evict -- --since=30d         # 실제 archive

17.5 디버깅

PHOENIX_OTLP_ENDPOINT=http://localhost:6006/v1/traces npm run dev
curl http://localhost:9464/metrics | grep agent_
curl http://localhost:9464/healthz

# Specific log level
LOG_LEVEL=debug npm run dev

# RAG/sandbox 비활성으로 회귀 확인
AGENT_RAG_ENABLED=false npm run eval -- --set=onboarding/cases
AGENT_TOOL_SANDBOX_ENABLED=false npm test

18. 참고 자료

18.1 Iter별 plan 문서

docs/에 dev 브랜치 보존 (main에는 미포함):

문서 내용
refactor-skill-ssot.md Iter1 plan
refactor-skill-ssot-iter2~8.md Iter2~8 각 plan
optimization-roadmap.md Iter8 closing 시 trigger 정의
refactor-skill-ssot-iter10~12.md Iter10~12 plan
optimization-roadmap-memory-ops.md Iter9~12 통합 로드맵
service-evaluation-operations.md 운영자 매뉴얼
최신기술동향.md, 멀티에이전트동향.md 자료 reference

18.2 외부 자료

  • MemoryAgentBench — accurate retrieval, test-time learning, long-range understanding, selective forgetting
  • Bounded Self-Escalation Graph — Microsoft orchestration guide
  • OpenInference semantic conventions — gen_ai.*, llm.* span attributes
  • MCP / A2A — 향후 채택 후보 (Iter13+ trigger 기반)

18.3 다음 단계 trigger (Iter13+)

docs/optimization-roadmap.md의 trigger 발동 시 개별 plan으로 시작:

항목 Trigger
NATS ACL enforce 전환 log-only 1개월 안정 + false positive 0
Tool sandbox enforce 전환 log-only 1주 안정
MongoDB Vector Search 마이그레이션 preview → GA
Resolution hash sunset Iter6 dual-write 2개월 안정
Cross-room context aggregation cross-room 패턴 발견
Parallel fan-out handoff 평균 hop ≥ 2.0
Web UI for review queue review 진입 ≥ N/day 1개월
Workflow auto-discovery (AFlow) propose/tournament 안정화
MCP/A2A 채택 외부 ecosystem 연결 필요

각 항목은 evidence(metric/운영 사례) 기반으로 plan 작성.


19. 변경 이력 (commit 별)

main 브랜치의 주요 milestones:

sha 메시지 Iter
bc0f8e0 Add secret coverage and NATS subject ACL 12 Phase C
d011a28 Add tool permission sandbox and audit stream 12 Phase B
21df14c Add cost-aware routing and LLM budget tracking 12 Phase A
211de18 Add hybrid memory retrieval and escalation history 11
1a86bde Add memory layers and RAG retrieval 10
1d0be56 Add Phoenix OTLP tracing and LLM span attributes 9
(이전) Close multi-agent plan and add router shadow reporting 8
(이전) Add Slack alerts and skill-aware monitoring 7
(이전) Unify dispatch and add supervisor routing signals 6
(이전) Close supervisor review loop and add handoff eval 5
(이전) Make eval pipeline skill-aware and validate SKILL manifests 3

각 commit의 detail은 git log <sha> -p 또는 dev 브랜치의 docs/refactor-skill-ssot-iterN.md 참고.


끝

이 문서가 outdated되면 last-verified commit을 footer에 명시:

  • last-verified: bc0f8e0 (2026-05-19)

업데이트 책임: 신규 큰 변경(새 skill 추가, 새 memory layer, 새 metric 등) PR 시 함께 갱신.