diff --git a/docs/service-evaluation-operations.md b/docs/service-evaluation-operations.md new file mode 100644 index 0000000..88d91e0 --- /dev/null +++ b/docs/service-evaluation-operations.md @@ -0,0 +1,593 @@ +# lolympus-agent Service and Evaluation Operations + +이 문서는 `lolympus-agent`의 운영 흐름과 evaluation 파이프라인을 한 곳에서 보기 위한 runbook이다. 실시간 응답 경로, background worker, shadow evaluation, human review, calibration, closed-loop prompt experiment까지 포함한다. + +## 1. 프로세스 구성 + +운영 프로세스는 크게 두 개다. + +| 프로세스 | 엔트리포인트 | 역할 | +|---|---|---| +| agent | `src/main.ts` | NATS request/reply를 받아 실시간 온보딩 응답 생성 | +| worker | `src/workers/background.ts` | Redis Stream background job 처리, shadow eval, archive, drift, review routing | + +PM2 기준으로는 `ecosystem.config.cjs`에서 agent와 worker를 함께 띄운다. + +```bash +npm run pm2:dev +npm run pm2:test +npm run pm2:prod +``` + +개발 환경 단독 실행은 기존 스크립트를 사용한다. + +```bash +npm run start:dev +``` + +## 2. Bootstrap 흐름 + +`src/main.ts`의 기동 순서는 다음과 같다. + +1. `loadEnvironmentFromNodeEnv()`로 `env-vars/{NODE_ENV}.env` 로드 +2. logger/tracing 초기화 +3. NATS 연결 +4. Garnet/Redis 연결 +5. LLM endpoint와 provider key 구성 +6. MongoDB 연결, 가능하면 room context 활성화 +7. `Agent` 생성 +8. `OnboardingSkill` 등록 +9. `lolympus-agent.request` 구독 시작 + +중요한 점은 NATS가 가장 먼저 외부 TLS 연결을 시도한다는 것이다. 따라서 bootstrap fatal error가 `certificate has expired`이고 `✓ NATS connected` 로그가 없었다면 1순위 원인은 `NATS_ENDPOINT` 인증서 만료다. + +## 3. 실시간 응답 흐름 + +본체는 NATS request/reply로 agent에 요청한다. + +```text +lolympus-agent.request + -> Agent.handleRequest + -> OnboardingSkill.handle + -> AgentResponse +``` + +`OnboardingSkill.handle()` 내부 흐름: + +1. Garnet에서 온보딩 세션 로드 +2. user message와 trigger 기반 intent classify +3. session, trigger, intent로 `DerivedState` 생성 +4. `shouldRespond=false`면 empty response 반환 +5. `responseMode=template`이면 template fallback 응답 +6. LLM 경로면 prompt build 후 orchestrator 실행 +7. tier에 따라 planner/responder/critic/tool-loop 사용 +8. validator로 응답 검증 +9. 최종 응답 반환 +10. blackboard 업데이트 +11. shadow eval event를 background stream에 비동기 발행 + +실시간 사용자 응답은 9번에서 끝난다. Evaluation은 11번 이후 worker가 비동기로 처리한다. + +## 4. Prompt와 Tool 경로 + +Prompt는 `src/skills/onboarding/prompts/builder.ts`가 만든다. + +기본 layer: + +- persona +- policy +- state +- examples +- context + +`PROMPT_VARIANT_ID`가 설정되면 `src/eval/proposals/{variantId}.json` manifest를 읽어 prompt layer diff를 적용한다. 운영 반영은 env로 variant를 선택하는 방식이며, proposal 생성과 tournament는 별도 CLI에서 수행한다. + +Tool loop는 `responseMode=llm-with-tools`일 때 작동한다. Built-in tool은 `src/tools/builtins/*`에 있다. + +## 5. Background Worker 흐름 + +Worker는 `lolympus-agent:background` Redis Stream을 consumer group `bg-worker`로 읽는다. + +| Job type | 처리 | +|---|---| +| `shadow-eval` | 실시간 응답을 rule/LLM/process 기준으로 평가 | +| `summarize` | 오래된 대화 일부를 요약해 summary slot에 저장 | + +Worker는 pending job을 복구하고, delivery count가 한도를 넘으면 DLQ로 이동한다. + +| Stream | 용도 | +|---|---| +| `lolympus-agent:background` | background job queue | +| `lolympus-agent:background:dlq` | 반복 실패 job 저장 | + +## 6. Shadow Evaluation 흐름 + +실시간 응답이 만들어지면 `ShadowEvalBus`가 background stream에 `shadow-eval` job을 발행한다. + +Worker는 이 job을 받아 다음을 수행한다. + +1. validator 재실행 +2. LLM judge 실행 +3. process trace가 있으면 process judge 실행 +4. latency, tier, intent, phase, judge 결과 저장 + +결과는 Redis Stream에 저장된다. + +```text +lolympus-agent:eval:results +``` + +이 Stream이 이후 evaluation 운영의 원천 데이터다. + +## 7. 주기 Evaluation 작업 + +Worker는 일정 간격으로 eval result를 가공한다. + +| 작업 | 기본 주기 | 결과 | +|---|---:|---| +| aggregation | `EVAL_AGG_TICK_MS` 5분 | slice별 Redis Hash metric | +| archive | `EVAL_ARCHIVE_TICK_MS` 1시간 | MongoDB `eval_archive` | +| drift detection | `EVAL_DRIFT_TICK_MS` 24시간 | `eval-drift` alert | +| promotion | `EVAL_PROMOTE_TICK_MS` 24시간 | golden candidate jsonl | +| review routing | `EVAL_REVIEW_TICK_MS` 5분 | human review queue | +| review label archive | archive tick과 함께 | MongoDB `eval_review_labels` | + +Slice 정의: + +```text +tier × intent × phase +``` + +예: + +```text +0:photo-question:photo-pending +1:discord-question:step1-completed +``` + +## 8. Redis Key Schema + +주요 Redis/Garnet key: + +| Key | Type | 용도 | +|---|---|---| +| `lolympus-agent:background` | Stream | worker job queue | +| `lolympus-agent:background:dlq` | Stream | worker dead-letter queue | +| `lolympus-agent:eval:results` | Stream | shadow eval 결과 | +| `lolympus-agent:eval:baseline` | String | 최근 안정 baseline report | +| `lolympus-agent:eval:agg:{tier}:{intent}:{phase}` | Hash | slice별 count/score/latency/process metric | +| `lolympus-agent:eval:agg:cursor` | String | aggregation cursor | +| `lolympus-agent:eval:archive:cursor` | String | eval archive cursor | +| `lolympus-agent:eval:review:queue` | Stream | human review 대기 큐 | +| `lolympus-agent:eval:review:labeled` | Stream | human review label | +| `lolympus-agent:eval:review:cursor` | String | review routing cursor | +| `lolympus-agent:eval:review:dedupe:{evalStreamId}` | String | 동일 eval result 중복 라우팅 방지 | +| `lolympus-agent:eval:review:label-archive:cursor` | String | review label archive cursor | +| `lolympus-agent:alert:dedupe:{kind}` | String | alert rate limit | + +## 9. MongoDB Collections + +| Collection | 생성/사용 모듈 | 용도 | +|---|---|---| +| `eval_archive` | `src/eval/archive.ts` | shadow eval 결과 장기 보관 | +| `eval_review_labels` | `src/eval/review/label-archive.ts` | 사람 라벨 장기 보관 | +| `eval_tournaments` | `src/eval/loop/tournament.ts` | prompt variant tournament 결과 | + +`eval_archive`는 `evaluatedAt` 기준 TTL index를 가진다. 기본 보관 기간은 `EVAL_ARCHIVE_TTL_DAYS=30`이다. + +## 10. Human Review 흐름 + +Review routing은 eval result 중 사람이 봐야 할 케이스를 큐에 넣는다. + +조건: + +- LLM judge overall이 낮음 +- validator와 LLM judge가 불일치 +- validator soft-warn +- rare slice + +큐: + +```text +lolympus-agent:eval:review:queue +``` + +라벨링: + +```bash +npm run eval:review -- --limit=20 +``` + +특정 reason만 보기: + +```bash +npm run eval:review -- --reason=low-score --limit=10 +``` + +stale pending item을 다시 큐에 넣기: + +```bash +npm run eval:review -- --reset-pending +``` + +세션 라벨을 JSONL로 저장: + +```bash +npm run eval:review -- --limit=20 --export-labeled=eval-reports/review-labels.jsonl +``` + +라벨 결과는 `lolympus-agent:eval:review:labeled` Stream에 쌓이고 archive tick에서 `eval_review_labels`로 영속화된다. + +## 11. Calibration 흐름 + +Calibration은 사람 점수와 LLM judge 점수의 상관을 계산한다. + +```bash +npm run eval:calibrate +``` + +CI나 수동 gate로 쓰려면: + +```bash +npm run eval:calibrate -- --check +``` + +`pairs < 10`이면 표본이 부족하므로 경고만 출력하고 실패 처리하지 않는다. 충분한 라벨이 쌓이면 `EVAL_CALIBRATION_MIN_CORR` 기준으로 LLM judge 신뢰도를 감시한다. + +## 12. Report와 Promotion + +Slice별 약한 영역 확인: + +```bash +npm run eval:report -- --worst=20 +``` + +최근 7일 대신 다른 window: + +```bash +npm run eval:report -- --worst=20 --since=24h +npm run eval:report -- --top=20 --since=30d +``` + +Golden candidate 확인: + +```bash +npm run eval:promote -- --dry-run +``` + +Candidate 파일에 추가: + +```bash +npm run eval:promote +``` + +기본 출력: + +```text +src/eval/golden/onboarding/candidate.jsonl +``` + +Promotion 우선순위: + +1. human-labeled fail +2. low-score +3. rule/LLM disagree +4. rare-slice + +candidate는 바로 golden으로 승격되는 것이 아니라 사람이 직접 검토해 `onboarding/cases.jsonl`로 옮긴다. + +## 13. Closed Loop 흐름 + +Phase 9의 closed loop는 prompt/tool 개선안을 만들고 tournament로 비교하는 흐름이다. + +### 13.1 Proposal 생성 + +약한 slice를 지정한다. + +```bash +npm run eval:propose -- --slice=tier=1,intent=photo-question,phase=photo-pending +``` + +archive case 수가 부족해도 강제로 실행하려면: + +```bash +npm run eval:propose -- --slice=tier=1,intent=photo-question,phase=photo-pending --force +``` + +결과는 git-tracked proposal manifest로 저장된다. + +```text +src/eval/proposals/{variantId}.json +``` + +### 13.2 Tournament 실행 + +Control과 variant를 비교한다. + +```bash +npm run eval:tournament -- --variants=control, +``` + +특정 archive case만 지정: + +```bash +npm run eval:tournament -- --variants=control, --case-ids=1-0,2-0 +``` + +결과 조회: + +```bash +npm run eval:tournament-results +npm run eval:tournament-results -- --variant= +``` + +Tournament는 A/B 위치 편향을 줄이기 위해 variant 순서를 바꾼 match도 실행하고 ELO를 갱신한다. 결과는 MongoDB `eval_tournaments`에 저장된다. + +### 13.3 운영 반영 + +운영 prompt에 variant를 반영하려면 env를 설정한다. + +```env +PROMPT_VARIANT_ID= +``` + +반영 전에는 반드시 다음을 확인한다. + +```bash +npm run typecheck +npm run lint +npm run build +npm test +npm run eval -- --skip-llm-judge --no-publish-report +``` + +Variant 적용은 자동 머지가 아니다. 사람이 tournament 결과와 sample match를 보고 결정한다. + +## 14. Eval CLI 요약 + +| 명령 | 용도 | +|---|---| +| `npm run eval` | golden set 실행 | +| `npm run eval:capture` | MongoDB events에서 golden scaffold 추출 | +| `npm run eval:baseline` | 안정 baseline 저장 | +| `npm run eval:report` | archive slice report 출력 | +| `npm run eval:promote` | golden candidate 추출 | +| `npm run eval:review` | human review CLI | +| `npm run eval:calibrate` | 사람/LLM judge calibration | +| `npm run eval:propose` | prompt/tool improvement proposal 생성 | +| `npm run eval:tournament` | prompt variant A/B tournament 실행 | +| `npm run eval:tournament-results` | tournament 결과 조회 | + +## 15. 주요 Env + +| Env | 기본값 | 용도 | +|---|---:|---| +| `EVAL_STREAM_MAXLEN` | 10000 | eval results Stream maxlen | +| `EVAL_AGG_TICK_MS` | 300000 | aggregation tick | +| `EVAL_AGG_TTL_SEC` | 604800 | slice metric TTL | +| `EVAL_ARCHIVE_TICK_MS` | 3600000 | archive tick | +| `EVAL_ARCHIVE_TTL_DAYS` | 30 | archive TTL days | +| `EVAL_DRIFT_TICK_MS` | 86400000 | drift tick | +| `EVAL_PROMOTE_TICK_MS` | 86400000 | promotion tick | +| `EVAL_REVIEW_TICK_MS` | 300000 | review routing tick | +| `EVAL_REVIEW_QUEUE_MAXLEN` | 500 | review queue maxlen | +| `EVAL_REVIEW_LABEL_MAXLEN` | 5000 | review label Stream maxlen | +| `EVAL_REVIEW_DEDUPE_TTL_SEC` | 604800 | review dedupe TTL | +| `EVAL_CALIBRATION_MIN_CORR` | 0.5 | calibration gate | +| `REVIEWER_ID` | git user.email | human review 식별자 | +| `LLM_PROPOSE_MODEL` | `LLM_JUDGE_MODEL` | proposal model | +| `LLM_PROPOSE_MAX_TOKENS` | 1024 | proposal max tokens | +| `EVAL_TOURNAMENT_MAX_MATCHES` | 50 | tournament match cap | +| `EVAL_TOURNAMENT_MIN_CASES` | 10 | proposal 최소 archive cases | +| `PROMPT_VARIANT_ID` | unset | 운영 prompt variant 선택 | +| `METRICS_PORT` | 9464 | Prometheus `/metrics` HTTP endpoint port | +| `SLACK_ALERT_WEBHOOK_URL` | unset | alert Slack incoming webhook URL | +| `SLACK_WEBHOOK_TIMEOUT_MS` | 3000 | Slack webhook POST timeout | +| `ALERT_THRESHOLD_P95_LATENCY_MS` | 3000 | 글로벌 p95 latency alert 기준 | +| `ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT` | 10 | 글로벌 eval score drop alert 기준 | +| `ALERT_THRESHOLD_LLM_ERROR_RATE` | 0.1 | 글로벌 LLM judge error rate alert 기준 | +| `ALERT_THRESHOLD_DRIFT_PCT` | 15 | 글로벌 drift alert 기준 | + +## 16. Alert와 Monitoring + +Worker는 다음 종류의 alert를 보낼 수 있다. + +| Alert kind | 의미 | +|---|---| +| `latency` | 최근 shadow eval latency p95 초과 | +| `llm-error-rate` | LLM judge 실패율 상승 | +| `eval-score-drop` | baseline 대비 eval score 하락 | +| `eval-drift` | 24h vs 7d slice score drift | +| `endpoint-down` | LLM circuit breaker open | +| `archive-stalled` | MongoDB archive 연속 실패 | + +Alert는 `lolympus-agent.alert` NATS subject로 발행되고, dedupe key로 rate limit된다. +`SLACK_ALERT_WEBHOOK_URL`이 설정되어 있으면 같은 alert가 Slack incoming webhook으로도 전송된다. +Slack 전송은 NATS publish 이후 비동기로 실행되므로 Slack 장애가 agent 응답 경로를 막지 않는다. + +### 16.1 Skill별 threshold + +Alert threshold는 글로벌 env를 기본으로 쓰고, skill별 env가 있으면 우선 적용한다. +skill 이름은 대문자 env suffix로 정규화된다. + +```env +ALERT_THRESHOLD_P95_LATENCY_MS=3000 +ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING=5000 +ALERT_THRESHOLD_P95_LATENCY_MS_MODERATION=2000 + +ALERT_THRESHOLD_LLM_ERROR_RATE=0.1 +ALERT_THRESHOLD_LLM_ERROR_RATE_ONBOARDING=0.2 + +ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT=10 +ALERT_THRESHOLD_DRIFT_PCT=15 +``` + +같은 alert kind라도 `skill`이 다르면 dedupe key가 분리된다. +예를 들어 onboarding latency alert와 moderation latency alert는 서로 throttle하지 않는다. +skill이 없는 cross-system alert는 기존처럼 글로벌 dedupe를 사용한다. + +### 16.2 Prometheus scrape + +Agent process는 `METRICS_PORT`에서 HTTP endpoint를 연다. + +| Endpoint | 응답 | +|---|---| +| `/healthz` | `ok` | +| `/metrics` | Prometheus text exposition | + +Prometheus scrape 예시: + +```yaml +scrape_configs: + - job_name: lolympus-agent + static_configs: + - targets: + - 127.0.0.1:9464 +``` + +권장 dashboard panel: + +| Metric | 권장 panel | +|---|---| +| `agent_handoff_total` | skill/from/to/reason별 handoff rate | +| `agent_handoff_loop_total` | skill별 loop 발생 수 | +| `agent_human_override_total` | supervisor/manual review 유입 | +| `agent_missed_escalation_total` | skill별 missed escalation | +| `agent_routing_accuracy_total` | eval routing accuracy | +| `agent_unnecessary_escalation_total` | skill별 unnecessary escalation | +| `agent_validator_fail_total` | skill/stage별 validator fail | +| `agent_handoff_count` | skill별 request당 handoff histogram | +| `agent_time_to_resolution_ms` | skill별 처리 시간 histogram | +| `agent_token_cost_usd` | skill/model별 token cost histogram | + +PromQL 예시: + +```promql +sum by (skill) (rate(agent_handoff_total[5m])) +histogram_quantile(0.95, sum by (skill, le) (rate(agent_time_to_resolution_ms_bucket[5m]))) +sum by (skill, stage) (rate(agent_validator_fail_total[5m])) +``` + +### 16.3 Slack webhook 설정 + +Slack incoming webhook URL을 env로 설정한다. + +```env +SLACK_ALERT_WEBHOOK_URL=https://hooks.slack.com/services/... +SLACK_WEBHOOK_TIMEOUT_MS=3000 +``` + +Webhook URL은 secret으로 취급한다. 로그 redaction 대상에 포함되어 있으며, Slack payload에는 metric, value, threshold, window, skill, suggested action만 포함하고 사용자 메시지 본문은 포함하지 않는다. + +## 17. 장애 확인 Runbook + +### 17.1 Bootstrap에서 `certificate has expired` + +증상: + +```text +[Bootstrap] Fatal startup error +message: certificate has expired +code: CERT_HAS_EXPIRED +``` + +`src/main.ts` 기준 NATS 연결이 첫 외부 TLS 연결이다. `✓ NATS connected` 로그가 없으면 `NATS_ENDPOINT` 인증서가 1순위 원인이다. + +확인: + +```bash +node --trace-warnings --import tsx src/main.ts +``` + +인증서 날짜 확인: + +```bash +openssl s_client -connect : -servername /dev/null \ + | openssl x509 -noout -dates -subject +``` + +`✓ NATS connected` 이후 같은 오류가 나면 다음 후보는 Garnet 또는 MongoDB다. + +### 17.2 LLM endpoint warning + +증상: + +```text +[LLM] Endpoint unavailable via circuit breaker +[LLM] Request error +[LLM] All endpoints failed +``` + +확인 순서: + +1. `CLI_PROXY_API_URLS` +2. `OPENROUTER_API_KEY`, `OPENROUTER_API_URL` +3. `AGENT_LLM_API_URLS` +4. `LLAMA_SERVER_API_URL` +5. provider별 API key + +`npm run eval -- --skip-llm-judge`는 LLM judge를 생략하지만 실제 응답 생성 LLM은 여전히 호출될 수 있다. endpoint가 모두 실패하면 template/fallback 경로로 통과할 수 있다. + +### 17.3 MongoDB 미설정 + +Worker가 다음과 비슷한 로그를 남긴다. + +```text +MongoDB connection string missing; eval archive/drift/promote disabled +``` + +영향: + +- 실시간 응답은 가능 +- shadow eval result Stream은 쌓임 +- archive, drift, promote, calibration, tournament는 제한됨 + +필요 env: + +```env +MONGODB_CONNECTION_STRING= +MONGODB_PROJECT_ID= +``` + +### 17.4 Review queue가 밀림 + +확인: + +```bash +npm run eval:review -- --limit=5 +``` + +stale pending item 정리: + +```bash +npm run eval:review -- --reset-pending +``` + +같은 케이스가 반복 라우팅되면 dedupe TTL과 cursor를 확인한다. + +```text +lolympus-agent:eval:review:cursor +lolympus-agent:eval:review:dedupe:{evalStreamId} +``` + +### 17.5 Archive stalled + +`archive-stalled` alert가 뜨면 MongoDB 연결과 index 생성 권한을 확인한다. + +관련 cursor: + +```text +lolympus-agent:eval:archive:cursor +lolympus-agent:eval:review:label-archive:cursor +``` + +cursor가 진행되지 않으면 worker가 다음 tick에서 재시도한다. + +## 18. 운영 원칙 + +- 실시간 응답 경로는 가능한 짧게 유지한다. +- Evaluation은 Redis Stream을 통해 비동기로 처리한다. +- Golden set 승격은 사람이 직접 결정한다. +- Prompt variant 운영 반영은 tournament 결과와 eval smoke를 본 뒤 결정한다. +- 자동 rollback이나 자동 merge는 하지 않는다. +- 모든 timestamp는 UTC ISO-8601을 사용한다. diff --git a/src/agent/agent.ts b/src/agent/agent.ts index d258d8a..1651bc1 100644 --- a/src/agent/agent.ts +++ b/src/agent/agent.ts @@ -174,11 +174,15 @@ export class Agent { metrics.handoffTotal.inc({ from: event.from, reason: event.reason, + skill: event.from, to: event.to, }); }, onHandoffLoop: (event) => { - metrics.handoffLoopTotal.inc({ from: event.from }); + metrics.handoffLoopTotal.inc({ + from: event.from, + skill: event.from, + }); }, onValidatorFail: (event) => { metrics.validatorFailTotal.inc({ @@ -206,8 +210,13 @@ export class Agent { } await this.natsClient.respond(msg, response); - metrics.handoffCount.observe(Math.max(0, budget.snapshot().handoffs)); - metrics.timeToResolutionMs.observe(Date.now() - startedAt); + metrics.handoffCount.observe( + Math.max(0, budget.snapshot().handoffs), + { skill: skill.name }, + ); + metrics.timeToResolutionMs.observe(Date.now() - startedAt, { + skill: skill.name, + }); rootSpan.setAttribute( "agent.source", response.text.trim() ? response.source : "empty", diff --git a/src/common/known-secret-keys.ts b/src/common/known-secret-keys.ts index ed97491..48d6f4a 100644 --- a/src/common/known-secret-keys.ts +++ b/src/common/known-secret-keys.ts @@ -9,6 +9,7 @@ export const KNOWN_SECRET_KEYS = [ "GARNET_PASSWORD", "MONGODB_CONNECTION_STRING", "MONGODB_PROJECT_ID", + "SLACK_ALERT_WEBHOOK_URL", ] as const; export type KnownSecretKey = (typeof KNOWN_SECRET_KEYS)[number]; diff --git a/src/common/nats-subjects.ts b/src/common/nats-subjects.ts index 281902c..4e2b82f 100644 --- a/src/common/nats-subjects.ts +++ b/src/common/nats-subjects.ts @@ -17,6 +17,7 @@ export interface AlertPayload { | "supervisor-handoff"; metric: string; severity: "info" | "warn" | "critical"; + skill?: string | undefined; suggestedAction?: string | undefined; threshold: number; timestamp: string; diff --git a/src/config.test.ts b/src/config.test.ts new file mode 100644 index 0000000..a620b79 --- /dev/null +++ b/src/config.test.ts @@ -0,0 +1,62 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +import { + getAlertThresholdDriftPct, + getAlertThresholdEvalScoreDropPct, + getAlertThresholdLlmErrorRate, + getAlertThresholdP95LatencyMs, + getSlackAlertWebhookUrl, + getSlackWebhookTimeoutMs, +} from "./config.js"; + +const ENV_KEYS = [ + "ALERT_THRESHOLD_DRIFT_PCT", + "ALERT_THRESHOLD_DRIFT_PCT_ONBOARDING", + "ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT", + "ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT_MODERATION", + "ALERT_THRESHOLD_LLM_ERROR_RATE", + "ALERT_THRESHOLD_LLM_ERROR_RATE_ONBOARDING", + "ALERT_THRESHOLD_P95_LATENCY_MS", + "ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING", + "SLACK_ALERT_WEBHOOK_URL", + "SLACK_WEBHOOK_TIMEOUT_MS", +] as const; + +describe("config", () => { + afterEach(() => { + for (const key of ENV_KEYS) { + vi.stubEnv(key, undefined); + } + vi.unstubAllEnvs(); + }); + + it("prefers skill-specific alert thresholds", () => { + vi.stubEnv("ALERT_THRESHOLD_P95_LATENCY_MS", "3000"); + vi.stubEnv("ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING", "5000"); + vi.stubEnv("ALERT_THRESHOLD_LLM_ERROR_RATE", "0.1"); + vi.stubEnv("ALERT_THRESHOLD_LLM_ERROR_RATE_ONBOARDING", "0.25"); + + expect(getAlertThresholdP95LatencyMs("onboarding")).toBe(5000); + expect(getAlertThresholdP95LatencyMs("moderation")).toBe(3000); + expect(getAlertThresholdLlmErrorRate("onboarding")).toBe(0.25); + }); + + it("normalizes skill names and falls back to global values", () => { + vi.stubEnv("ALERT_THRESHOLD_DRIFT_PCT", "15"); + vi.stubEnv("ALERT_THRESHOLD_DRIFT_PCT_HIGH_RISK", "7"); + vi.stubEnv("ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT", "10"); + vi.stubEnv("ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT_MODERATION", "6"); + + expect(getAlertThresholdDriftPct("high-risk")).toBe(7); + expect(getAlertThresholdDriftPct("onboarding")).toBe(15); + expect(getAlertThresholdEvalScoreDropPct("moderation")).toBe(6); + }); + + it("reads Slack webhook configuration", () => { + vi.stubEnv("SLACK_ALERT_WEBHOOK_URL", " https://hooks.slack.test/x "); + vi.stubEnv("SLACK_WEBHOOK_TIMEOUT_MS", "1234"); + + expect(getSlackAlertWebhookUrl()).toBe("https://hooks.slack.test/x"); + expect(getSlackWebhookTimeoutMs()).toBe(1234); + }); +}); diff --git a/src/config.ts b/src/config.ts index d462410..1c1bdf1 100644 --- a/src/config.ts +++ b/src/config.ts @@ -29,6 +29,47 @@ function getRatioNumber( : fallback; } +function normalizeSkillEnvSuffix(skill: string | undefined): string | undefined { + const suffix = skill?.trim().toUpperCase().replace(/[^A-Z0-9]/gu, "_"); + return suffix || undefined; +} + +function getPositiveNumberForSkill( + envKey: string, + skill: string | undefined, + fallback: number, +): number { + const skillSuffix = normalizeSkillEnvSuffix(skill); + if (skillSuffix) { + const raw = process.env[`${envKey}_${skillSuffix}`]?.trim(); + if (raw) { + const parsed = Number(raw); + if (Number.isFinite(parsed) && parsed > 0) { + return parsed; + } + } + } + return getPositiveNumber(envKey, fallback); +} + +function getRatioNumberForSkill( + envKey: string, + skill: string | undefined, + fallback: number, +): number { + const skillSuffix = normalizeSkillEnvSuffix(skill); + if (skillSuffix) { + const raw = process.env[`${envKey}_${skillSuffix}`]?.trim(); + if (raw) { + const parsed = Number(raw); + if (Number.isFinite(parsed) && parsed >= 0 && parsed <= 1) { + return parsed; + } + } + } + return getRatioNumber(envKey, fallback); +} + function getBoolean(envKey: string, fallback: boolean): boolean { const raw = process.env[envKey]?.trim().toLowerCase(); if (!raw) return fallback; @@ -539,22 +580,42 @@ export function getShadowEvalSamplingRate(tier: 0 | 1 | 2): number { } } -export function getAlertThresholdP95LatencyMs(): number { - return getPositiveNumber("ALERT_THRESHOLD_P95_LATENCY_MS", 3_000); +export function getAlertThresholdP95LatencyMs(skill?: string): number { + return getPositiveNumberForSkill( + "ALERT_THRESHOLD_P95_LATENCY_MS", + skill, + 3_000, + ); } -export function getAlertThresholdEvalScoreDropPct(): number { - return getPositiveNumber("ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT", 10); +export function getAlertThresholdEvalScoreDropPct(skill?: string): number { + return getPositiveNumberForSkill( + "ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT", + skill, + 10, + ); } -export function getAlertThresholdLlmErrorRate(): number { - return getRatioNumber("ALERT_THRESHOLD_LLM_ERROR_RATE", 0.1); +export function getAlertThresholdLlmErrorRate(skill?: string): number { + return getRatioNumberForSkill( + "ALERT_THRESHOLD_LLM_ERROR_RATE", + skill, + 0.1, + ); } -export function getAlertThresholdDriftPct(): number { - return getPositiveNumber("ALERT_THRESHOLD_DRIFT_PCT", 15); +export function getAlertThresholdDriftPct(skill?: string): number { + return getPositiveNumberForSkill("ALERT_THRESHOLD_DRIFT_PCT", skill, 15); } export function getAlertRateLimitSec(): number { return getPositiveNumber("ALERT_RATE_LIMIT_SEC", 300); } + +export function getSlackAlertWebhookUrl(): string | undefined { + return process.env["SLACK_ALERT_WEBHOOK_URL"]?.trim() || undefined; +} + +export function getSlackWebhookTimeoutMs(): number { + return getPositiveNumber("SLACK_WEBHOOK_TIMEOUT_MS", 3_000); +} diff --git a/src/eval/shadow-worker.test.ts b/src/eval/shadow-worker.test.ts index e3b3afe..7c16859 100644 --- a/src/eval/shadow-worker.test.ts +++ b/src/eval/shadow-worker.test.ts @@ -1,9 +1,11 @@ -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; +import type { AlertPayload } from "@common/nats-subjects"; import { calculateEvalScoreDropPct, calculateLlmErrorRate, calculatePercentile, + ShadowWorker, } from "./shadow-worker.js"; describe("calculatePercentile", () => { @@ -40,14 +42,73 @@ describe("calculateLlmErrorRate", () => { llmOverall: null, recordedAt: 1, responseLatencyMs: 100, + skill: "onboarding", }, { llmJudgeErrored: false, llmOverall: 4.2, recordedAt: 2, responseLatencyMs: 200, + skill: "onboarding", }, ]), ).toBe(0.5); }); }); + +describe("ShadowWorker alerts", () => { + it("emits latency alerts with the observation skill", async () => { + const sendAlert = vi.fn().mockResolvedValue(true); + const worker = Object.assign(Object.create(ShadowWorker.prototype), { + alertSender: { sendAlert }, + recentEvalObservations: [ + { + llmJudgeErrored: false, + llmOverall: 4, + recordedAt: Date.now(), + responseLatencyMs: 4500, + skill: "onboarding", + }, + { + llmJudgeErrored: false, + llmOverall: 4, + recordedAt: Date.now(), + responseLatencyMs: 4600, + skill: "onboarding", + }, + { + llmJudgeErrored: false, + llmOverall: 4, + recordedAt: Date.now(), + responseLatencyMs: 4700, + skill: "onboarding", + }, + { + llmJudgeErrored: false, + llmOverall: 4, + recordedAt: Date.now(), + responseLatencyMs: 4800, + skill: "onboarding", + }, + { + llmJudgeErrored: false, + llmOverall: 4, + recordedAt: Date.now(), + responseLatencyMs: 4900, + skill: "onboarding", + }, + ], + }) as { + checkLatencyAlert: (now: number) => Promise; + }; + + await worker.checkLatencyAlert(Date.now()); + + expect(sendAlert).toHaveBeenCalledWith( + expect.objectContaining>({ + kind: "latency", + skill: "onboarding", + }), + ); + }); +}); diff --git a/src/eval/shadow-worker.ts b/src/eval/shadow-worker.ts index 470dfd2..cbd7a41 100644 --- a/src/eval/shadow-worker.ts +++ b/src/eval/shadow-worker.ts @@ -112,6 +112,7 @@ interface EvalObservation { llmOverall: number | null; recordedAt: number; responseLatencyMs: number | null; + skill: string; } function delay(ms: number): Promise { @@ -331,6 +332,18 @@ export function calculateLlmErrorRate( return errorCount / observations.length; } +function groupObservationsBySkill( + observations: EvalObservation[], +): Map { + const groups = new Map(); + for (const observation of observations) { + const values = groups.get(observation.skill) ?? []; + values.push(observation); + groups.set(observation.skill, values); + } + return groups; +} + export class ShadowWorker { private archiveFailureCount = 0; private lastAlertCheckAt = 0; @@ -586,6 +599,7 @@ export class ShadowWorker { llmOverall, recordedAt: Date.now(), responseLatencyMs: job.latencyMs ?? null, + skill: job.skill, }); const resultFields: Record = { @@ -938,17 +952,22 @@ export class ShadowWorker { return; } - const threshold = getAlertThresholdDriftPct(); const reports = await computeDriftReports(this.mongoConnection, { nowMs: now }); - const significantReports = reports.filter((report) => report.significant); + const significantReports = reports.filter((report) => { + const threshold = getAlertThresholdDriftPct(report.slice.skill); + return report.recent24hCount >= 30 && report.dropPct >= threshold; + }); for (const report of significantReports) { const slice = sliceKeyToString(report.slice); + const skill = report.slice.skill ?? "onboarding"; + const threshold = getAlertThresholdDriftPct(skill); await this.alertSender.sendAlert( { kind: "eval-drift", metric: "shadow_eval.drift.llm_overall_drop_pct", severity: report.dropPct >= threshold * 1.5 ? "critical" : "warn", + skill, suggestedAction: `${slice} slice의 최근 실패 케이스와 prompt/tool 변경 이력을 확인해 주세요.`, threshold, timestamp: new Date(now).toISOString(), @@ -1059,29 +1078,40 @@ export class ShadowWorker { } private async checkLatencyAlert(now: number): Promise { - const latencySamples = this.recentEvalObservations.flatMap((observation) => - observation.responseLatencyMs === null ? [] : [observation.responseLatencyMs], - ); - if (latencySamples.length < MIN_LATENCY_OBSERVATIONS || !this.alertSender) { + if (!this.alertSender) { return; } - const p95LatencyMs = calculatePercentile(latencySamples, 0.95); - const threshold = getAlertThresholdP95LatencyMs(); - if (p95LatencyMs === null || p95LatencyMs <= threshold) { - return; - } + for (const [skill, observations] of groupObservationsBySkill( + this.recentEvalObservations, + )) { + const latencySamples = observations.flatMap((observation) => + observation.responseLatencyMs === null + ? [] + : [observation.responseLatencyMs], + ); + if (latencySamples.length < MIN_LATENCY_OBSERVATIONS) { + continue; + } - await this.alertSender.sendAlert({ - kind: "latency", - metric: "shadow_eval.p95_latency_ms", - severity: p95LatencyMs >= threshold * 1.5 ? "critical" : "warn", - suggestedAction: "최근 LLM endpoint 지연과 tool loop 호출 수를 확인해 주세요.", - threshold, - timestamp: new Date(now).toISOString(), - value: p95LatencyMs, - window: ALERT_WINDOW_LABEL, - }); + const p95LatencyMs = calculatePercentile(latencySamples, 0.95); + const threshold = getAlertThresholdP95LatencyMs(skill); + if (p95LatencyMs === null || p95LatencyMs <= threshold) { + continue; + } + + await this.alertSender.sendAlert({ + kind: "latency", + metric: "shadow_eval.p95_latency_ms", + severity: p95LatencyMs >= threshold * 1.5 ? "critical" : "warn", + skill, + suggestedAction: "최근 LLM endpoint 지연과 tool loop 호출 수를 확인해 주세요.", + threshold, + timestamp: new Date(now).toISOString(), + value: p95LatencyMs, + window: ALERT_WINDOW_LABEL, + }); + } } private async checkEvalScoreDropAlert(now: number): Promise { @@ -1089,59 +1119,70 @@ export class ShadowWorker { return; } - const scoreSamples = this.recentEvalObservations.flatMap((observation) => - observation.llmOverall === null ? [] : [observation.llmOverall], - ); - if (scoreSamples.length < MIN_SCORE_OBSERVATIONS) { - return; - } - - const currentAverage = - scoreSamples.reduce((sum, score) => sum + score, 0) / scoreSamples.length; const baselineAverage = await this.loadBaselineOverallScore(); - const dropPct = calculateEvalScoreDropPct(currentAverage, baselineAverage); - const threshold = getAlertThresholdEvalScoreDropPct(); + for (const [skill, observations] of groupObservationsBySkill( + this.recentEvalObservations, + )) { + const scoreSamples = observations.flatMap((observation) => + observation.llmOverall === null ? [] : [observation.llmOverall], + ); + if (scoreSamples.length < MIN_SCORE_OBSERVATIONS) { + continue; + } - if (dropPct === null || dropPct <= threshold) { - return; + const currentAverage = + scoreSamples.reduce((sum, score) => sum + score, 0) / scoreSamples.length; + const dropPct = calculateEvalScoreDropPct(currentAverage, baselineAverage); + const threshold = getAlertThresholdEvalScoreDropPct(skill); + + if (dropPct === null || dropPct <= threshold) { + continue; + } + + await this.alertSender.sendAlert({ + kind: "eval-score-drop", + metric: "shadow_eval.llm_overall_drop_pct", + severity: dropPct >= threshold * 1.5 ? "critical" : "warn", + skill, + suggestedAction: "최근 prompt 변경, tool 결과, fallback 비율을 함께 점검해 주세요.", + threshold, + timestamp: new Date(now).toISOString(), + value: Number(dropPct.toFixed(3)), + window: ALERT_WINDOW_LABEL, + }); } - - await this.alertSender.sendAlert({ - kind: "eval-score-drop", - metric: "shadow_eval.llm_overall_drop_pct", - severity: dropPct >= threshold * 1.5 ? "critical" : "warn", - suggestedAction: "최근 prompt 변경, tool 결과, fallback 비율을 함께 점검해 주세요.", - threshold, - timestamp: new Date(now).toISOString(), - value: Number(dropPct.toFixed(3)), - window: ALERT_WINDOW_LABEL, - }); } private async checkLlmErrorRateAlert(now: number): Promise { - if ( - !this.alertSender || - this.recentEvalObservations.length < MIN_ERROR_RATE_OBSERVATIONS - ) { + if (!this.alertSender) { return; } - const errorRate = calculateLlmErrorRate(this.recentEvalObservations); - const threshold = getAlertThresholdLlmErrorRate(); - if (errorRate === null || errorRate <= threshold) { - return; - } + for (const [skill, observations] of groupObservationsBySkill( + this.recentEvalObservations, + )) { + if (observations.length < MIN_ERROR_RATE_OBSERVATIONS) { + continue; + } - await this.alertSender.sendAlert({ - kind: "llm-error-rate", - metric: "shadow_eval.llm_error_rate", - severity: errorRate >= threshold * 1.5 ? "critical" : "warn", - suggestedAction: "judge endpoint 응답 상태와 circuit breaker open 여부를 확인해 주세요.", - threshold, - timestamp: new Date(now).toISOString(), - value: Number(errorRate.toFixed(4)), - window: ALERT_WINDOW_LABEL, - }); + const errorRate = calculateLlmErrorRate(observations); + const threshold = getAlertThresholdLlmErrorRate(skill); + if (errorRate === null || errorRate <= threshold) { + continue; + } + + await this.alertSender.sendAlert({ + kind: "llm-error-rate", + metric: "shadow_eval.llm_error_rate", + severity: errorRate >= threshold * 1.5 ? "critical" : "warn", + skill, + suggestedAction: "judge endpoint 응답 상태와 circuit breaker open 여부를 확인해 주세요.", + threshold, + timestamp: new Date(now).toISOString(), + value: Number(errorRate.toFixed(4)), + window: ALERT_WINDOW_LABEL, + }); + } } private async checkOpenCircuitAlerts(now: number): Promise { diff --git a/src/llm/llm-client.ts b/src/llm/llm-client.ts index 9e10700..aa582ac 100644 --- a/src/llm/llm-client.ts +++ b/src/llm/llm-client.ts @@ -7,7 +7,7 @@ import type { LlmEndpointConfig, ProviderName, } from "./types.js"; -import { getRequestLogger } from "@obs/context"; +import { getContext, getRequestLogger } from "@obs/context"; import { withLlmSpan } from "@obs/spans"; import { getLlmHealthTimeoutMs, @@ -219,6 +219,10 @@ export class LlmClient { payload.usage?.prompt_tokens, payload.usage?.completion_tokens, ), + { + model: request.model, + skill: getContext()?.skill, + }, ); span.setAttributes({ diff --git a/src/observability/alert.test.ts b/src/observability/alert.test.ts index 520f395..042368e 100644 --- a/src/observability/alert.test.ts +++ b/src/observability/alert.test.ts @@ -8,6 +8,7 @@ const payload: AlertPayload = { kind: "latency", metric: "shadow_eval.p95_latency_ms", severity: "warn", + skill: "onboarding", threshold: 3000, timestamp: "2026-04-17T00:00:00.000Z", value: 4100, @@ -23,7 +24,11 @@ describe("AlertService", () => { const result = await service.sendAlert(payload); expect(result).toBe(true); - expect(setStringIfAbsent).toHaveBeenCalledTimes(1); + expect(setStringIfAbsent).toHaveBeenCalledWith( + "lolympus-agent:alert:dedupe:latency:onboarding", + payload.timestamp, + expect.any(Number), + ); expect(publish).toHaveBeenCalledWith(AGENT_ALERT_SUBJECT, payload); }); @@ -37,4 +42,35 @@ describe("AlertService", () => { expect(result).toBe(false); expect(publish).not.toHaveBeenCalled(); }); + + it("keeps dedupe global when payload has no skill", async () => { + const setStringIfAbsent = vi.fn().mockResolvedValue(true); + const publish = vi.fn().mockResolvedValue(undefined); + const service = new AlertService({ setStringIfAbsent }, { publish }); + + await service.sendAlert({ ...payload, skill: undefined }); + + expect(setStringIfAbsent).toHaveBeenCalledWith( + "lolympus-agent:alert:dedupe:latency", + payload.timestamp, + expect.any(Number), + ); + }); + + it("notifies Slack without changing publish result", async () => { + const setStringIfAbsent = vi.fn().mockResolvedValue(true); + const publish = vi.fn().mockResolvedValue(undefined); + const notify = vi.fn().mockResolvedValue(false); + const service = new AlertService( + { setStringIfAbsent }, + { publish }, + { notify }, + ); + + const result = await service.sendAlert(payload); + + expect(result).toBe(true); + expect(publish).toHaveBeenCalledWith(AGENT_ALERT_SUBJECT, payload); + expect(notify).toHaveBeenCalledWith(payload); + }); }); diff --git a/src/observability/alert.ts b/src/observability/alert.ts index 967233d..161fc78 100644 --- a/src/observability/alert.ts +++ b/src/observability/alert.ts @@ -3,6 +3,7 @@ import { AGENT_ALERT_SUBJECT, type AlertPayload } from "@common/nats-subjects"; import type { NatsClient } from "@common/nats"; import { getAlertRateLimitSec } from "../config.js"; import { getLogger } from "./logger.js"; +import type { SlackNotifier } from "./slack-notifier.js"; export interface AlertSender { sendAlert( @@ -13,8 +14,11 @@ export interface AlertSender { ): Promise; } -function getAlertDedupeKey(kind: AlertPayload["kind"]): string { - return `lolympus-agent:alert:dedupe:${kind}`; +function getAlertDedupeKey( + kind: AlertPayload["kind"], + skill?: string, +): string { + return `lolympus-agent:alert:dedupe:${kind}${skill ? `:${skill}` : ""}`; } export class AlertService implements AlertSender { @@ -23,6 +27,7 @@ export class AlertService implements AlertSender { constructor( private readonly garnetClient: Pick, private readonly natsClient: Pick, + private readonly slackNotifier?: SlackNotifier | undefined, ) {} async sendAlert( @@ -31,7 +36,8 @@ export class AlertService implements AlertSender { dedupeKey?: string | undefined; }, ): Promise { - const dedupeKey = options?.dedupeKey ?? getAlertDedupeKey(payload.kind); + const dedupeKey = + options?.dedupeKey ?? getAlertDedupeKey(payload.kind, payload.skill); const claimed = await this.garnetClient.setStringIfAbsent( dedupeKey, payload.timestamp, @@ -43,6 +49,7 @@ export class AlertService implements AlertSender { { dedupeKey, kind: payload.kind, + skill: payload.skill, }, "[Alert] Suppressed by rate limit", ); @@ -55,12 +62,43 @@ export class AlertService implements AlertSender { kind: payload.kind, metric: payload.metric, severity: payload.severity, + skill: payload.skill, threshold: payload.threshold, value: payload.value, window: payload.window, }, "[Alert] Published", ); + this.notifySlack(payload); return true; } + + private notifySlack(payload: AlertPayload): void { + if (!this.slackNotifier) { + return; + } + + void this.slackNotifier.notify(payload) + .then((ok) => { + if (!ok) { + this.logger.warn( + { + kind: payload.kind, + skill: payload.skill, + }, + "[Alert] Slack notify failed", + ); + } + }) + .catch((error) => { + this.logger.warn( + { + err: error, + kind: payload.kind, + skill: payload.skill, + }, + "[Alert] Slack notify failed", + ); + }); + } } diff --git a/src/observability/metrics-server.test.ts b/src/observability/metrics-server.test.ts new file mode 100644 index 0000000..344533e --- /dev/null +++ b/src/observability/metrics-server.test.ts @@ -0,0 +1,25 @@ +import { describe, expect, it, vi } from "vitest"; + +import { handleMetricsRequest } from "./metrics-server.js"; + +describe("metrics server", () => { + it("serves metrics and health checks", () => { + const writeHead = vi.fn(); + const end = vi.fn(); + + handleMetricsRequest({ url: "/healthz" }, { end, writeHead }); + expect(writeHead).toHaveBeenCalledWith(200, { "Content-Type": "text/plain" }); + expect(end).toHaveBeenCalledWith("ok"); + + writeHead.mockClear(); + end.mockClear(); + + handleMetricsRequest({ url: "/metrics" }, { end, writeHead }); + expect(writeHead).toHaveBeenCalledWith(200, { + "Content-Type": "text/plain; version=0.0.4", + }); + expect(String(end.mock.calls[0]?.[0])).toContain( + "# HELP agent_handoff_total", + ); + }); +}); diff --git a/src/observability/metrics-server.ts b/src/observability/metrics-server.ts index 36e9383..cf4ff72 100644 --- a/src/observability/metrics-server.ts +++ b/src/observability/metrics-server.ts @@ -1,4 +1,9 @@ -import { createServer, type Server } from "node:http"; +import { + createServer, + type IncomingMessage, + type Server, + type ServerResponse, +} from "node:http"; import { getMetricsPort } from "../config.js"; import { getLogger } from "./logger.js"; @@ -6,23 +11,34 @@ import { renderMetrics } from "./metrics.js"; let server: Server | null = null; +export function handleMetricsRequest( + request: Pick, + response: Pick, +): void { + if (request.url === "/healthz") { + response.writeHead(200, { "Content-Type": "text/plain" }); + response.end("ok"); + return; + } + + if (request.url !== "/metrics") { + response.writeHead(404); + response.end("not found"); + return; + } + + response.writeHead(200, { + "Content-Type": "text/plain; version=0.0.4", + }); + response.end(renderMetrics()); +} + export function startMetricsServer(port = getMetricsPort()): Server { if (server) { return server; } - server = createServer((request, response) => { - if (request.url !== "/metrics") { - response.writeHead(404); - response.end("not found"); - return; - } - - response.writeHead(200, { - "Content-Type": "text/plain; version=0.0.4", - }); - response.end(renderMetrics()); - }); + server = createServer(handleMetricsRequest); server.listen(port, () => { getLogger().info({ port }, "✓ Metrics server listening"); diff --git a/src/observability/metrics.test.ts b/src/observability/metrics.test.ts index 0c03cef..efc6930 100644 --- a/src/observability/metrics.test.ts +++ b/src/observability/metrics.test.ts @@ -26,15 +26,18 @@ describe("metrics", () => { metrics.handoffTotal.inc({ from: "onboarding", reason: "out-of-scope", + skill: "onboarding", to: "moderation", }); - metrics.handoffCount.observe(2); + metrics.handoffCount.observe(2, { skill: "onboarding" }); const body = renderMetrics(); expect(body).toContain( - 'agent_handoff_total{from="onboarding",reason="out-of-scope",to="moderation"} 1', + 'agent_handoff_total{from="onboarding",reason="out-of-scope",skill="onboarding",to="moderation"} 1', + ); + expect(body).toContain( + 'agent_handoff_count_bucket{skill="onboarding",le="2"} 1', ); - expect(body).toContain('agent_handoff_count_bucket{le="2"} 1'); }); }); diff --git a/src/observability/metrics.ts b/src/observability/metrics.ts index 60e57de..0dbaf0c 100644 --- a/src/observability/metrics.ts +++ b/src/observability/metrics.ts @@ -52,7 +52,7 @@ class CounterMetric { } class HistogramMetric { - private readonly values: number[] = []; + private readonly samples = new Map(); constructor( readonly name: string, @@ -60,10 +60,15 @@ class HistogramMetric { private readonly buckets: number[], ) {} - observe(value: number): void { - if (Number.isFinite(value)) { - this.values.push(value); + observe(value: number, labels: Labels = {}): void { + if (!Number.isFinite(value)) { + return; } + + const key = labelKey(labels); + const sample = this.samples.get(key) ?? { labels, values: [] }; + sample.values.push(value); + this.samples.set(key, sample); } render(): string[] { @@ -71,15 +76,31 @@ class HistogramMetric { `# HELP ${this.name} ${this.help}`, `# TYPE ${this.name} histogram`, ]; - for (const bucket of this.buckets) { - const count = this.values.filter((value) => value <= bucket).length; - lines.push(`${this.name}_bucket{le="${bucket}"} ${count}`); + + for (const sample of this.samples.values()) { + for (const bucket of this.buckets) { + const count = sample.values.filter((value) => value <= bucket).length; + lines.push( + `${this.name}_bucket${formatLabels({ + ...sample.labels, + le: bucket, + })} ${count}`, + ); + } + lines.push( + `${this.name}_bucket${formatLabels({ + ...sample.labels, + le: "+Inf", + })} ${sample.values.length}`, + ); + lines.push( + `${this.name}_sum${formatLabels(sample.labels)} ${sample.values.reduce( + (sum, value) => sum + value, + 0, + )}`, + ); + lines.push(`${this.name}_count${formatLabels(sample.labels)} ${sample.values.length}`); } - lines.push(`${this.name}_bucket{le="+Inf"} ${this.values.length}`); - lines.push( - `${this.name}_sum ${this.values.reduce((sum, value) => sum + value, 0)}`, - ); - lines.push(`${this.name}_count ${this.values.length}`); return lines; } } diff --git a/src/observability/slack-notifier.test.ts b/src/observability/slack-notifier.test.ts new file mode 100644 index 0000000..0256418 --- /dev/null +++ b/src/observability/slack-notifier.test.ts @@ -0,0 +1,89 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +import type { AlertPayload } from "@common/nats-subjects"; +import { + formatSlackMessage, + SlackWebhookNotifier, +} from "./slack-notifier.js"; + +const payload: AlertPayload = { + kind: "latency", + metric: "shadow_eval.p95_latency_ms", + severity: "critical", + skill: "onboarding", + suggestedAction: "Check endpoint latency.", + threshold: 3000, + timestamp: "2026-04-17T00:00:00.000Z", + value: 4500, + window: "5m", +}; + +describe("formatSlackMessage", () => { + it("formats severity, metric, skill, and action", () => { + const message = formatSlackMessage(payload); + + expect(message.text).toBe( + ":rotating_light: [latency] shadow_eval.p95_latency_ms", + ); + expect(JSON.stringify(message.blocks)).toContain("*Skill*\\nonboarding"); + expect(JSON.stringify(message.blocks)).toContain("Check endpoint latency."); + }); +}); + +describe("SlackWebhookNotifier", () => { + afterEach(() => { + vi.unstubAllEnvs(); + }); + + it("posts the formatted Slack message", async () => { + const fetchImpl = vi.fn().mockResolvedValue(new Response("ok")); + const notifier = new SlackWebhookNotifier( + "https://hooks.slack.test/services/test", + fetchImpl, + ); + + await expect(notifier.notify(payload)).resolves.toBe(true); + + const call = fetchImpl.mock.calls[0]; + expect(call?.[0]).toBe("https://hooks.slack.test/services/test"); + expect(call?.[1]?.body).toBe(JSON.stringify(formatSlackMessage(payload))); + expect(call?.[1]?.headers).toEqual({ "Content-Type": "application/json" }); + expect(call?.[1]?.method).toBe("POST"); + expect(call?.[1]?.signal).toBeInstanceOf(AbortSignal); + }); + + it("returns false for non-2xx responses", async () => { + const fetchImpl = vi + .fn() + .mockResolvedValue(new Response("rate limited", { status: 429 })); + const notifier = new SlackWebhookNotifier( + "https://hooks.slack.test/services/test", + fetchImpl, + ); + + await expect(notifier.notify(payload)).resolves.toBe(false); + }); + + it("returns false when fetch rejects", async () => { + const fetchImpl = vi.fn().mockRejectedValue(new Error("timeout")); + const notifier = new SlackWebhookNotifier( + "https://hooks.slack.test/services/test", + fetchImpl, + ); + + await expect(notifier.notify(payload)).resolves.toBe(false); + }); + + it("uses the configured timeout signal", async () => { + vi.stubEnv("SLACK_WEBHOOK_TIMEOUT_MS", "1234"); + const fetchImpl = vi.fn().mockResolvedValue(new Response("ok")); + const notifier = new SlackWebhookNotifier( + "https://hooks.slack.test/services/test", + fetchImpl, + ); + + await notifier.notify(payload); + + expect(fetchImpl.mock.calls[0]?.[1]?.signal).toBeInstanceOf(AbortSignal); + }); +}); diff --git a/src/observability/slack-notifier.ts b/src/observability/slack-notifier.ts new file mode 100644 index 0000000..3f25ebd --- /dev/null +++ b/src/observability/slack-notifier.ts @@ -0,0 +1,102 @@ +import type { AlertPayload } from "@common/nats-subjects"; +import { + getSlackAlertWebhookUrl, + getSlackWebhookTimeoutMs, +} from "../config.js"; +import { getLogger } from "./logger.js"; + +export interface SlackTextObject { + text: string; + type: "mrkdwn" | "plain_text"; +} + +export interface SlackBlock { + fields?: SlackTextObject[] | undefined; + text?: SlackTextObject | undefined; + type: "section"; +} + +export interface SlackMessage { + blocks: SlackBlock[]; + text: string; +} + +export interface SlackNotifier { + notify(payload: AlertPayload): Promise; +} + +const SEVERITY_EMOJI: Record = { + critical: ":rotating_light:", + info: ":information_source:", + warn: ":warning:", +}; + +function mrkdwn(text: string): SlackTextObject { + return { text, type: "mrkdwn" }; +} + +export function formatSlackMessage(payload: AlertPayload): SlackMessage { + const blocks: SlackBlock[] = [ + { + text: mrkdwn( + `*${payload.severity.toUpperCase()}* - \`${payload.kind}\``, + ), + type: "section", + }, + { + fields: [ + mrkdwn(`*Metric*\n${payload.metric}`), + mrkdwn(`*Value / Threshold*\n${payload.value} / ${payload.threshold}`), + mrkdwn(`*Window*\n${payload.window}`), + mrkdwn(`*Skill*\n${payload.skill ?? "global"}`), + ], + type: "section", + }, + ]; + + if (payload.suggestedAction?.trim()) { + blocks.push({ + text: mrkdwn(`*Suggested action*\n${payload.suggestedAction}`), + type: "section", + }); + } + + return { + blocks, + text: `${SEVERITY_EMOJI[payload.severity]} [${payload.kind}] ${payload.metric}`, + }; +} + +export class SlackWebhookNotifier implements SlackNotifier { + constructor( + private readonly webhookUrl: string, + private readonly fetchImpl: typeof fetch = fetch, + ) {} + + async notify(payload: AlertPayload): Promise { + try { + const response = await this.fetchImpl(this.webhookUrl, { + body: JSON.stringify(formatSlackMessage(payload)), + headers: { "Content-Type": "application/json" }, + method: "POST", + signal: AbortSignal.timeout(getSlackWebhookTimeoutMs()), + }); + return response.ok; + } catch (error) { + getLogger().warn( + { + err: error, + kind: payload.kind, + skill: payload.skill, + }, + "[Slack] Notification failed", + ); + return false; + } + } +} + +export function createSlackNotifierFromEnv(): SlackNotifier | undefined { + const webhookUrl = getSlackAlertWebhookUrl(); + return webhookUrl ? new SlackWebhookNotifier(webhookUrl) : undefined; +} diff --git a/src/skills/supervisor/skill.ts b/src/skills/supervisor/skill.ts index 203dce0..c8f1e84 100644 --- a/src/skills/supervisor/skill.ts +++ b/src/skills/supervisor/skill.ts @@ -10,6 +10,7 @@ import type { NatsClient } from "@common/nats"; import type { LlmClient } from "@llm/llm-client"; import { AlertService } from "@obs/alert"; import { metrics } from "@obs/metrics"; +import { createSlackNotifierFromEnv } from "@obs/slack-notifier"; import type { Skill } from "@skills/skill"; import type { SkillCapabilityCard, SkillRegistry } from "@skills/skill-loader"; import { getLogger } from "@obs/logger"; @@ -52,7 +53,11 @@ export class SupervisorSkill implements Skill { private readonly llmClient?: Pick | undefined, private readonly skillRegistry?: SkillRegistry | undefined, ) { - this.alertService = new AlertService(garnetClient, natsClient); + this.alertService = new AlertService( + garnetClient, + natsClient, + createSlackNotifierFromEnv(), + ); } canHandle(request: AgentRequest): boolean { @@ -83,13 +88,14 @@ export class SupervisorSkill implements Skill { target: target ?? "human-review", userId: request.userId, }); - metrics.humanOverrideTotal.inc({ kind: reason }); + metrics.humanOverrideTotal.inc({ kind: reason, skill: this.name }); await this.alertService.sendAlert( { kind: "supervisor-handoff", metric: "agent.supervisor_handoff", severity: "warn", + skill: this.name, suggestedAction: "Review the supervisor handoff queue.", threshold: 0, timestamp: new Date().toISOString(), diff --git a/src/workers/background.ts b/src/workers/background.ts index e0ecacb..a215a4c 100644 --- a/src/workers/background.ts +++ b/src/workers/background.ts @@ -21,6 +21,9 @@ const { getRawSecret } = await import("../common/secret-manager.js"); const { ConversationStore } = await import("../conversation/index.js"); const { ShadowWorker } = await import("../eval/shadow-worker.js"); const { AlertService } = await import("../observability/alert.js"); +const { createSlackNotifierFromEnv } = await import( + "../observability/slack-notifier.js" +); const { initMongoDB, closeMongoDB } = await import("../common/mongodb.js"); function firstNonEmpty( @@ -175,7 +178,11 @@ async function bootstrap(): Promise { providerApiKeys, }); const conversationStore = new ConversationStore(garnetClient); - const alertService = new AlertService(garnetClient, natsClient); + const alertService = new AlertService( + garnetClient, + natsClient, + createSlackNotifierFromEnv(), + ); const mongoConnectionString = firstNonEmpty( getRawSecret("MONGODB_CONNECTION_STRING"), process.env["MONGODB_CONNECTION_STRING"],