This commit is contained in:
dahyeon committed 2026-05-12 08:16:31 -07:00
1 parent 1453433b62
commit da76af3826
65 files changed
+11089 -143

No files matched your search

+44 -1
View File
@@ -306,6 +306,9 @@ src/skills/
| `CLASSIFIER_MAX_TOKENS` | 16 | intent JSON만 받으므로 매우 작게 |
| `LLM_JUDGE_MODEL` | `gpt-5.3-codex-spark` | Phase 5 shadow eval judge |
| `LLM_JUDGE_MAX_TOKENS` | 128 | 4축 점수 + 한줄 이유 |
| `EVAL_PROCESS_LLM_JUDGE_ENABLED` | `false` | Phase 7 shadow-worker process judge LLM 보조 평가 활성화 |
| `EVAL_PROCESS_LLM_JUDGE_SAMPLING_RATE` | 0.1 | Phase 7 process LLM judge sampling rate |
| `EVAL_PROCESS_LLM_JUDGE_MAX_TOKENS` | 256 | Phase 7 process LLM judge 출력 상한 |
| `SAFETY_MARGIN_MS` | 500 | NATS 5s timeout에서 차감 (§7) |
| `TIER0_ESTIMATED_CALL_MS` | 600 | deadline 계산용 |
| `TIER1_ESTIMATED_CALL_MS` | 1200 | |
@@ -324,11 +327,35 @@ src/skills/
| `ALERT_THRESHOLD_P95_LATENCY_MS` | 3000 | 초과 시 KakaoTalk alert |
| `ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT` | 10 | baseline 대비 하락률 |
| `ALERT_THRESHOLD_LLM_ERROR_RATE` | 0.1 | 1분 윈도우 에러율 |
| `ALERT_THRESHOLD_DRIFT_PCT` | 15 | Phase 6 24h vs 7d slice 점수 하락률 |
| `ALERT_RATE_LIMIT_SEC` | 300 | 동일 종류 알림 재발송 최소 간격 |
| `BLACKBOARD_TTL_SEC` | 43200 | 12h |
| `CONVERSATION_SUMMARY_TTL_SEC` | 86400 | 24h |
| `EVAL_STREAM_MAXLEN` | 10000 | Redis Stream MAXLEN ~ |
| `EVAL_RESULTS_TTL_SEC` | 2592000 | 30일 |
| `EVAL_AGG_TICK_MS` | 300000 | Phase 6 slice aggregator tick |
| `EVAL_AGG_TTL_SEC` | 604800 | Phase 6 slice Hash TTL |
| `EVAL_AGG_LATENCY_SAMPLE_CAP` | 200 | Phase 6 p50/p95 계산용 latency sample cap |
| `EVAL_ARCHIVE_TICK_MS` | 3600000 | Phase 6 Redis eval stream → Mongo archive tick |
| `EVAL_ARCHIVE_TTL_DAYS` | 30 | Phase 6 `eval_archive.evaluatedAt` TTL index |
| `EVAL_ARCHIVE_BATCH_SIZE` | 500 | Phase 6 archive tick당 stream entry 처리 수 |
| `EVAL_DRIFT_TICK_MS` | 86400000 | Phase 6 drift detection tick |
| `EVAL_PROMOTE_TICK_MS` | 86400000 | Phase 6 golden candidate promotion tick |
| `EVAL_PROMOTE_LOW_SCORE` | 2 | Phase 6 low-score candidate 임계 |
| `EVAL_PROMOTE_RARE_THRESHOLD` | 5 | Phase 6 rare-slice candidate 임계 |
| `EVAL_PROMOTE_PER_CATEGORY_LIMIT` | 10 | Phase 6 promotion 카테고리별 최대 후보 수 |
| `EVAL_REVIEW_TICK_MS` | 300000 | Phase 8 review-routing tick |
| `EVAL_REVIEW_QUEUE_MAXLEN` | 500 | Phase 8 review queue Stream MAXLEN |
| `EVAL_REVIEW_LABEL_MAXLEN` | 5000 | Phase 8 labeled Stream MAXLEN |
| `EVAL_REVIEW_PER_CATEGORY_LIMIT` | 5 | Phase 8 tick당 카테고리별 enqueue 한도 |
| `EVAL_REVIEW_DEDUPE_TTL_SEC` | 604800 | Phase 8 review enqueue dedupe TTL |
| `EVAL_CALIBRATION_MIN_CORR` | 0.5 | Phase 8 사람/LLM calibration 최소 상관 |
| `REVIEWER_ID` | git user.email fallback | Phase 8 review CLI reviewedBy 식별자 |
| `LLM_PROPOSE_MODEL` | `LLM_JUDGE_MODEL` | Phase 9 propose 단계 모델 |
| `LLM_PROPOSE_MAX_TOKENS` | 1024 | Phase 9 proposal JSON 출력 상한 |
| `EVAL_TOURNAMENT_MAX_MATCHES` | 50 | Phase 9 tournament 최대 match 수 |
| `EVAL_TOURNAMENT_MIN_CASES` | 10 | Phase 9 propose/tournament 최소 archive case 수 |
| `PROMPT_VARIANT_ID` | — | Phase 9 운영 prompt variant manifest 적용 |
| `BACKGROUND_STREAM_MAXLEN` | 5000 | |
---
@@ -354,7 +381,13 @@ src/skills/
**AlertPayload 예시**:
```ts
interface AlertPayload {
kind: "latency" | "eval-score-drop" | "llm-error-rate" | "endpoint-down";
kind:
| "archive-stalled"
| "endpoint-down"
| "eval-drift"
| "eval-score-drop"
| "latency"
| "llm-error-rate";
severity: "info" | "warn" | "critical";
metric: string;
value: number;
@@ -420,10 +453,20 @@ span/log 일관성 위해 커스텀 attribute는 아래 네이밍만 사용. das
| `lolympus-agent:background:dlq` | **Stream** | MAXLEN ~ 1000 (리터럴, 작은 값이라 env 미등록) | Phase 5 dead-letter. retry 실패 시 XADD. 수동 조사용 |
| `lolympus-agent:eval:results` | **Stream** | MAXLEN ~ `$EVAL_STREAM_MAXLEN` (기본 10000) | Phase 5 shadow eval 결과 |
| `lolympus-agent:eval:baseline` | String (JSON) | no TTL | Phase 5 최근 안정 baseline 점수 |
| `lolympus-agent:eval:agg:{tier}:{intent}:{phase}` | Hash | `$EVAL_AGG_TTL_SEC` | Phase 6 slice별 count/score/validator/latency metric |
| `lolympus-agent:eval:agg:cursor` | String | no TTL | Phase 6 마지막 aggregated eval stream id |
| `lolympus-agent:eval:archive:cursor` | String | no TTL | Phase 6 마지막 Mongo archived eval stream id |
| `lolympus-agent:eval:review:queue` | Stream | MAXLEN ~ `$EVAL_REVIEW_QUEUE_MAXLEN` | Phase 8 사람 검토 대기 큐. Consumer group: `reviewers` |
| `lolympus-agent:eval:review:labeled` | Stream | MAXLEN ~ `$EVAL_REVIEW_LABEL_MAXLEN` | Phase 8 사람 라벨 적립 Stream |
| `lolympus-agent:eval:review:dedupe:{evalStreamId}` | String | `$EVAL_REVIEW_DEDUPE_TTL_SEC` | Phase 8 동일 eval result 중복 검토 라우팅 방지 |
| `lolympus-agent:eval:review:cursor` | String | no TTL | Phase 8 마지막 review-routed eval stream id |
| `lolympus-agent:eval:review:label-archive:cursor` | String | no TTL | Phase 8 마지막 Mongo archived review label stream id |
| `lolympus-agent:metrics:tier:{YYYYMMDDHH}` | Hash | 7일 | tier 0/1/2 분포 카운터 (시작/최종 tier, escalation 카운트 포함) |
| `lolympus-agent:metrics:escalation:{YYYYMMDDHH}` | Hash | 7일 | tier escalation from→to 카운트 |
| `lolympus-agent:alert:dedupe:{kind}` | String | `ALERT_RATE_LIMIT_SEC` | 동일 알림 rate limit |
| `lolympus-agent:alert:dedupe:endpoint:{endpointHash}` | String | `ALERT_RATE_LIMIT_SEC` | endpoint-down 알림 per-endpoint rate limit |
| `lolympus-agent:alert:dedupe:eval-drift:{tier}:{intent}:{phase}` | String | `ALERT_RATE_LIMIT_SEC` | Phase 6 drift alert per-slice rate limit |
| `lolympus-agent:alert:dedupe:archive-stalled` | String | `ALERT_RATE_LIMIT_SEC` | Phase 6 archive failure alert rate limit |
---