Compare commits

..
3 Commits
Author SHA1 Message Date
dahyeon 040e372ead Add env example and ignore local env files 2026-05-14 14:45:50 -07:00
dahyeon 221e9391e0 Add skill governance and runtime routing 2026-05-14 14:28:59 -07:00
dahyeon a900016be4 Add evaluation operations and reporting 2026-05-12 08:16:31 -07:00
211 changed files with 3357 additions and 18533 deletions

No files matched your search

-44
View File
@@ -1,44 +0,0 @@
# CLAUDE.md
`lolympus-agent`는 NATS 메시지 드리븐 멀티 에이전트 시스템입니다. 자세한 구조는 [docs/multi-agent-architecture.md](docs/multi-agent-architecture.md)를 먼저 읽으세요.
## 작업 시 자주 쓰는 명령
| 명령 | 용도 |
| ------------------------------------------------- | ------------------------------------------ |
| `npm run typecheck` | TypeScript 타입 검증 |
| `npm run lint` | ESLint와 SKILL.md 일관성 검증 |
| `npm run lint:skills` | SKILL.md와 코드 manifest 검증 |
| `npm test` | 전체 테스트 |
| `npm run eval -- --set=onboarding/cases` | onboarding golden 평가 |
| `npm run eval:moderation -- --skip-llm-judge` | moderation golden smoke |
| `npm run eval:handoff -- --skip-llm-judge` | handoff 경로 회귀 |
| `npm run eval:router-shadow-report -- --since=7d` | static router와 learned router shadow 비교 |
| `npm run review:list -- --limit=5` | supervisor review queue 확인 |
## 핵심 디렉토리
- `src/skills/<name>/` - skill별 단일 정의 위치. `SKILL.md`, intent, prompt data, tools, runtime을 함께 둔다.
- `src/agent/dispatch.ts` - production과 eval이 공유하는 handoff dispatch loop.
- `src/eval/` - golden, shadow archive, proposal, tournament, router training/reporting.
- `src/operations/` - supervisor review queue 운영 CLI.
- `src/observability/` - logger, tracing, metrics, alert, Slack notifier.
- `docs/service-evaluation-operations.md` - 운영 runbook.
## 변경 시 주의
- SKILL.md를 바꾸면 `npm run lint:skills`를 실행한다.
- prompt, routing, handoff를 바꾸면 onboarding, moderation, handoff eval smoke를 함께 확인한다.
- 새 skill은 `src/skills/<name>/`에 정의하고 factory/registry에 등록한 뒤 golden case를 추가한다.
- plan 문서(`docs/refactor-skill-ssot*.md`)는 작업 지시와 기록이다. 공개 main에 병합할 때는 별도 요청이 없는 한 제외한다.
## 운영
- Review queue 처리: [docs/service-evaluation-operations.md](docs/service-evaluation-operations.md)
- Architecture: [docs/multi-agent-architecture.md](docs/multi-agent-architecture.md)
- 고도화 시작 조건: [docs/optimization-roadmap.md](docs/optimization-roadmap.md)
- Resolution hash sunset: [docs/resolution-hash-sunset-procedure.md](docs/resolution-hash-sunset-procedure.md)
## 상태
운영 가능한 멀티 에이전트 기반은 Iter8 (2026-05-18)에 완료되었습니다. 이후 고도화는 [docs/optimization-roadmap.md](docs/optimization-roadmap.md)의 trigger 조건에 따라 별도 track으로 진행합니다.
+544
View File
@@ -0,0 +1,544 @@
# Machine Env Service
IMDSv2-compatible local metadata daemon for homelab machines. Each machine runs one daemon, and local processes or containers can fetch machine-scoped metadata and allowlisted Infisical secrets through HTTP without holding Infisical credentials.
## API Reference
Use `BASE_URL=http://169.254.169.254` on a host with the link-local listener. For local development use `BASE_URL=http://127.0.0.1:8675`. For Docker Desktop on macOS use `BASE_URL=http://host.docker.internal:8675`.
Issue an IMDSv2-style token:
```bash
BASE_URL=http://169.254.169.254
TOKEN=$(
curl -fsS -X PUT \
-H "X-aws-ec2-metadata-token-ttl-seconds: 60" \
"$BASE_URL/latest/api/token"
)
```
Token TTL must be between `1` and `21600` seconds. All metadata, dynamic data, user-data, and secret endpoints require:
```text
X-aws-ec2-metadata-token: <token>
```
| Method | Path | Description |
| ------ | -------------------------------------------- | ---------------------------------------------------------------------------------- |
| `PUT` | `/latest/api/token` | Issue an opaque metadata token. Requires `X-aws-ec2-metadata-token-ttl-seconds`. |
| `GET` | `/latest/meta-data/` | List top-level metadata keys. |
| `GET` | `/latest/meta-data/environment` | Infisical environment configured for this machine. |
| `GET` | `/latest/meta-data/hostname` | Machine hostname. |
| `GET` | `/latest/meta-data/local-hostname` | Local hostname. |
| `GET` | `/latest/meta-data/local-ipv4` | Primary local IPv4. |
| `GET` | `/latest/meta-data/mac` | Primary MAC address. |
| `GET` | `/latest/meta-data/machine-id` | Machine ID from config or OS. |
| `GET` | `/latest/meta-data/role` | Optional machine role. Returns `404` if unset. |
| `GET` | `/latest/meta-data/tags/` | List tag namespaces. Currently `instance/`. |
| `GET` | `/latest/meta-data/tags/<KEY>` | Read a machine tag by key. |
| `GET` | `/latest/meta-data/tags/instance/` | List machine tag keys. |
| `GET` | `/latest/meta-data/tags/instance/<KEY>` | Read a machine tag by key. |
| `GET` | `/latest/meta-data/secrets/` | List allowlisted secret keys currently present in cache. |
| `GET` | `/latest/meta-data/secrets/<KEY>` | Read one allowlisted secret value as plain text. |
| `GET` | `/latest/meta-data/secrets-bundle` | Read all cached allowlisted secrets as JSON. Requires `ALLOW_SECRETS_BUNDLE=true`. |
| `GET` | `/latest/dynamic/` | List dynamic-data categories. |
| `GET` | `/latest/dynamic/instance-identity/` | List instance identity fields. |
| `GET` | `/latest/dynamic/instance-identity/document` | Machine identity document as JSON. |
| `GET` | `/latest/user-data` | Configured user-data as plain text. Returns `404` if unset. |
### Curl Examples
List top-level metadata:
```bash
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/meta-data/"
```
Read basic machine metadata:
```bash
curl -fsS -H "X-aws-ec2-metadata-token: $TOKEN" "$BASE_URL/latest/meta-data/hostname"
curl -fsS -H "X-aws-ec2-metadata-token: $TOKEN" "$BASE_URL/latest/meta-data/local-ipv4"
curl -fsS -H "X-aws-ec2-metadata-token: $TOKEN" "$BASE_URL/latest/meta-data/mac"
curl -fsS -H "X-aws-ec2-metadata-token: $TOKEN" "$BASE_URL/latest/meta-data/machine-id"
curl -fsS -H "X-aws-ec2-metadata-token: $TOKEN" "$BASE_URL/latest/meta-data/environment"
```
Read role:
```bash
curl -i \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/meta-data/role"
```
List and read tags:
```bash
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/meta-data/tags/instance/"
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/meta-data/tags/instance/site"
```
Read the identity document:
```bash
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/dynamic/instance-identity/document"
```
Read user-data:
```bash
curl -i \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/user-data"
```
List and read secrets:
```bash
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/meta-data/secrets/"
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/meta-data/secrets/FOO"
```
Read the optional secret bundle:
```bash
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
"$BASE_URL/latest/meta-data/secrets-bundle"
```
Expected failure cases:
```bash
# Missing token: 401
curl -i "$BASE_URL/latest/meta-data/"
# SSRF guard: 403
curl -i \
-H "X-Forwarded-For: 1.2.3.4" \
"$BASE_URL/latest/meta-data/"
# Token endpoint with wrong method: 405
curl -i "$BASE_URL/latest/api/token"
```
## Configuration
The daemon reads its own dotenv-style config file.
Default paths:
```text
Linux: /etc/machine-env-service/env
macOS: /usr/local/etc/machine-env-service/env
```
Override with:
```bash
MACHINE_ENV_CONFIG=/path/to/env
```
Minimum config:
```env
INFISICAL_ACCESS_TOKEN=
INFISICAL_PROJECT_ID=
INFISICAL_ENVIRONMENT=prod
KNOWN_SECRET_KEYS=FOO,BAR
METADATA_PORT=8675
REFRESH_INTERVAL_SEC=60
```
Optional machine metadata:
```env
MACHINE_ID=
MACHINE_HOSTNAME=
MACHINE_LOCAL_HOSTNAME=
MACHINE_LOCAL_IPV4=
MACHINE_MAC=
MACHINE_ROLE=homelab-node
MACHINE_TAGS=site=home,tier=prod
USER_DATA=
USER_DATA_FILE=
```
If machine fields are empty, the daemon derives hostname, IPv4, MAC, and machine-id from the OS where possible.
Secret allowlist variables are checked in this order:
```text
KNOWN_SECRET_KEYS
EXPOSED_SECRET_KEYS
SECRET_ALLOWLIST
SECRET_WHITELIST
```
Values can be comma-separated or whitespace-separated.
## Local Development
Build:
```bash
npm ci
npm run build
```
Run locally without binding `169.254.169.254:80`:
```bash
NODE_ENV=dev LINK_LOCAL_LISTENER_ENABLED=false npm start
```
Request a token:
```bash
TOKEN=$(
curl -fsS -X PUT \
-H "X-aws-ec2-metadata-token-ttl-seconds: 60" \
http://127.0.0.1:8675/latest/api/token
)
```
Fetch metadata:
```bash
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
http://127.0.0.1:8675/latest/meta-data/hostname
```
Fetch a secret:
```bash
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
http://127.0.0.1:8675/latest/meta-data/secrets/FOO
```
Or use:
```bash
METADATA_BASE_URL=http://127.0.0.1:8675 ./examples/fetch-secret.sh FOO
```
## Ubuntu Install
The included systemd units assume the app lives at `/opt/machine-env-service`.
Install files:
```bash
sudo mkdir -p /opt/machine-env-service
sudo chown -R "$USER:$USER" /opt/machine-env-service
rsync -a \
--exclude .git \
--exclude node_modules \
--exclude dist \
--exclude env-vars \
./ /opt/machine-env-service/
cd /opt/machine-env-service
npm ci
npm run build
```
Create service user:
```bash
sudo useradd --system --home /nonexistent --shell /usr/sbin/nologin machine-env || true
```
Create config:
```bash
sudo install -d -o machine-env -g machine-env -m 0700 /etc/machine-env-service
sudo cp deploy/systemd/machine-env-service.env.example /etc/machine-env-service/env
sudo nano /etc/machine-env-service/env
sudo chown machine-env:machine-env /etc/machine-env-service/env
sudo chmod 0600 /etc/machine-env-service/env
```
Install and start systemd units:
```bash
sudo cp deploy/systemd/machine-env-setup-link-local.service /etc/systemd/system/
sudo cp deploy/systemd/machine-env-service.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now machine-env-setup-link-local.service
sudo systemctl enable --now machine-env-service.service
```
Check logs:
```bash
systemctl status machine-env-service.service
journalctl -u machine-env-service.service -f
```
Test the link-local endpoint:
```bash
TOKEN=$(
curl -fsS -X PUT \
-H "X-aws-ec2-metadata-token-ttl-seconds: 60" \
http://169.254.169.254/latest/api/token
)
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/
```
## macOS Install
The included launchd plists assume:
```text
App path: /opt/machine-env-service
Node path: /usr/local/bin/node
Config path: /usr/local/etc/machine-env-service/env
```
Check your Node path first:
```bash
command -v node
```
If it is not `/usr/local/bin/node`, either update `deploy/launchd/us.dahyeon.machine-env-service.plist` before installing or create a system-visible symlink:
```bash
sudo mkdir -p /usr/local/bin
sudo ln -sf "$(command -v node)" /usr/local/bin/node
```
When Node comes from `nvm`, the symlink is the quickest local setup. A Homebrew Node installed at a stable path is usually better for a long-lived daemon.
Install files:
```bash
sudo mkdir -p /opt/machine-env-service
sudo chown -R "$(id -un):admin" /opt/machine-env-service
rsync -a \
--exclude .git \
--exclude node_modules \
--exclude dist \
--exclude env-vars \
./ /opt/machine-env-service/
cd /opt/machine-env-service
npm ci
npm run build
```
Create config:
```bash
sudo install -d -m 0700 /usr/local/etc/machine-env-service
sudo cp deploy/launchd/machine-env-service.env.example /usr/local/etc/machine-env-service/env
sudo nano /usr/local/etc/machine-env-service/env
sudo chown root:wheel /usr/local/etc/machine-env-service/env
sudo chmod 0600 /usr/local/etc/machine-env-service/env
```
Install and start launchd daemons:
```bash
sudo cp deploy/launchd/us.dahyeon.machine-env-setup-link-local.plist /Library/LaunchDaemons/
sudo cp deploy/launchd/us.dahyeon.machine-env-service.plist /Library/LaunchDaemons/
sudo chown root:wheel /Library/LaunchDaemons/us.dahyeon.machine-env-*.plist
sudo chmod 0644 /Library/LaunchDaemons/us.dahyeon.machine-env-*.plist
sudo launchctl bootstrap system /Library/LaunchDaemons/us.dahyeon.machine-env-setup-link-local.plist
sudo launchctl bootstrap system /Library/LaunchDaemons/us.dahyeon.machine-env-service.plist
```
Check logs:
```bash
sudo launchctl print system/us.dahyeon.machine-env-service
tail -f /var/log/machine-env-service.log
tail -f /var/log/machine-env-service.err.log
```
Test:
```bash
TOKEN=$(
curl -fsS -X PUT \
-H "X-aws-ec2-metadata-token-ttl-seconds: 60" \
http://169.254.169.254/latest/api/token
)
curl -fsS \
-H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/hostname
```
Docker Desktop on macOS usually cannot route container traffic to host `169.254.169.254`. Use the LAN listener instead:
```text
http://host.docker.internal:8675/latest/meta-data/...
```
## Update
Restarting the daemon invalidates issued IMDSv2 tokens. Clients should request a new token after an update.
If `/opt/machine-env-service` is a git checkout:
```bash
cd /opt/machine-env-service
git pull
npm ci
npm run build
```
If you deploy from another checkout:
```bash
rsync -a --delete \
--exclude .git \
--exclude node_modules \
--exclude dist \
--exclude env-vars \
./ /opt/machine-env-service/
cd /opt/machine-env-service
npm ci
npm run build
```
Do not overwrite the machine config file:
```text
Linux: /etc/machine-env-service/env
macOS: /usr/local/etc/machine-env-service/env
```
Restart after code-only updates:
```bash
sudo systemctl restart machine-env-service.service
```
```bash
sudo launchctl kickstart -k system/us.dahyeon.machine-env-service
```
If systemd unit files changed:
```bash
sudo cp deploy/systemd/machine-env-setup-link-local.service /etc/systemd/system/
sudo cp deploy/systemd/machine-env-service.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl restart machine-env-setup-link-local.service
sudo systemctl restart machine-env-service.service
```
If launchd plist files changed:
```bash
sudo cp deploy/launchd/us.dahyeon.machine-env-setup-link-local.plist /Library/LaunchDaemons/
sudo cp deploy/launchd/us.dahyeon.machine-env-service.plist /Library/LaunchDaemons/
sudo chown root:wheel /Library/LaunchDaemons/us.dahyeon.machine-env-*.plist
sudo chmod 0644 /Library/LaunchDaemons/us.dahyeon.machine-env-*.plist
sudo launchctl bootout system /Library/LaunchDaemons/us.dahyeon.machine-env-service.plist 2>/dev/null || true
sudo launchctl bootout system /Library/LaunchDaemons/us.dahyeon.machine-env-setup-link-local.plist 2>/dev/null || true
sudo launchctl bootstrap system /Library/LaunchDaemons/us.dahyeon.machine-env-setup-link-local.plist
sudo launchctl bootstrap system /Library/LaunchDaemons/us.dahyeon.machine-env-service.plist
```
Check after updating:
```bash
systemctl status machine-env-service.service
journalctl -u machine-env-service.service -n 100
```
```bash
sudo launchctl print system/us.dahyeon.machine-env-service
tail -n 100 /var/log/machine-env-service.err.log
```
## Operations
Restart:
```bash
sudo systemctl restart machine-env-service.service
```
```bash
sudo launchctl kickstart -k system/us.dahyeon.machine-env-service
```
Stop:
```bash
sudo systemctl stop machine-env-service.service
```
```bash
sudo launchctl bootout system /Library/LaunchDaemons/us.dahyeon.machine-env-service.plist
```
## Troubleshooting
If `curl http://169.254.169.254/latest/api/token` fails with `Couldn't connect to server`, check the link-local alias and daemon separately:
```bash
ifconfig lo0 | grep 169.254.169.254
sudo launchctl print system/us.dahyeon.machine-env-service
```
If launchd shows `last exit code = 78: EX_CONFIG` and the plist points to `/usr/local/bin/node`, verify that Node exists there:
```bash
ls -l /usr/local/bin/node
command -v node
```
Fix a missing `/usr/local/bin/node` with:
```bash
sudo mkdir -p /usr/local/bin
sudo ln -sf "$(command -v node)" /usr/local/bin/node
sudo launchctl kickstart -k system/us.dahyeon.machine-env-service
```
Then retry token issuance.
## Security Notes
- Keep the config file mode at `0600`.
- Do not commit `env-vars/*.env`, production config files, or access tokens.
- Only allowlist secrets that should be readable by local machine workloads.
- Block `METADATA_PORT` from untrusted networks.
- Requests with `X-Forwarded-For` are rejected.
- Metadata requests require IMDSv2-style tokens; token TTL is `1..21600` seconds.
-375
View File
@@ -1,375 +0,0 @@
# lolympus-agent
`lolympus-agent`는 카카오톡 기반으로 운영되는 롤림푸스 커뮤니티의 운영 자동화를 위한 NATS 메시지 드리븐 멀티 에이전트 시스템입니다. 신규 유저 온보딩과 커뮤니티 규칙을 묻는 메시지에 대한 응답을 지원합니다.
English version: [English](#english)
## 개요
카카오톡 운영방에는 닉네임 형식, 하트 인증, 디스코드 프로필, 커뮤니티 규칙처럼 반복해서 들어오는 질문이 많습니다. 이 서비스는 그런 메시지를 받아 현재 상태를 확인하고, 필요한 경우 짧은 안내 메시지를 생성합니다.
응답하기 애매하거나 자동화하기 어려운 요청은 supervisor review queue로 넘겨 운영자가 확인할 수 있게 합니다.
## 목적
반복적인 온보딩 안내와 규칙 질의 응답을 자동화하되, 운영자가 직접 답하는 톤과 안전성을 최대한 유지하는 것이 목표입니다.
단순한 케이스는 template이나 가벼운 LLM 응답으로 처리하고, 더 복잡한 케이스는 tool 호출, handoff, review queue를 통해 다룹니다.
## 기능
**1. 온보딩 응답**
- 신규 멤버, 닉네임, 사진 인증, 진행 상태, timeout, 완료 이벤트를 처리합니다.
- 사용자 메시지에 맞춰 짧은 안내를 생성합니다.
- LLM이 필요 없는 케이스는 deterministic template로 빠르게 응답합니다.
**2. 닉네임과 진행 상태 확인**
- 커뮤니티 닉네임 형식을 검사합니다.
- Redis/Garnet에 저장된 온보딩 세션 진행 상태를 확인합니다.
- MongoDB가 설정되어 있으면 room context를 함께 참고합니다.
**3. 멀티 에이전트 라우팅**
- onboarding, moderation, supervisor skill을 분리해 다룹니다.
- `src/agent/dispatch.ts`의 공통 dispatch loop가 skill 간 handoff를 처리합니다.
- transition matrix와 budget guardrail로 잘못된 handoff나 loop를 막습니다.
**4. Tool-aware LLM runtime**
- planner, responder, critic, tool-loop 실행 모드를 지원합니다.
- local, proxy, OpenRouter 호환 LLM provider를 사용할 수 있습니다.
- circuit breaker, health check, response validation을 포함합니다.
**5. Evaluation pipeline**
- golden case와 shadow evaluation case를 수집합니다.
- rule, LLM, process, handoff judge를 실행합니다.
- review queue, calibration, drift check, promotion, tournament를 통해 prompt와 routing 개선 근거를 만듭니다.
**6. Background worker와 운영 알림**
- shadow eval과 archive 작업을 실시간 응답 경로 밖에서 처리합니다.
- eval 결과를 MongoDB에 저장하고, 필요한 경우 NATS/Slack alert를 보냅니다.
- supervisor review queue로 운영자가 확인해야 하는 케이스를 모읍니다.
## 사용 기술
- TypeScript
- Node.js
- NATS
- Redis / Garnet
- MongoDB / Mongoose
- Infisical
- OpenTelemetry
- Pino
- Zod
- Vitest
- ESLint
- Prettier
## 실행 방법
1. 의존성을 설치합니다.
```bash
npm install
```
2. 환경 변수를 준비합니다.
```bash
cp env-vars/env.example env-vars/dev.env
```
`env-vars/env.example`이 없는 환경이라면 아래 [환경 변수](#환경-변수) 섹션을 기준으로 `env-vars/dev.env`를 직접 만듭니다.
3. 타입과 테스트를 확인합니다.
```bash
npm run typecheck
npm test
```
4. 개발 환경으로 실행합니다.
```bash
npm run start:dev
```
PM2로 agent와 worker를 함께 실행하려면 다음 명령을 사용합니다.
```bash
npm run pm2:dev
```
## 환경 변수
Infisical 연동을 위한 환경 변수가 필요합니다.
```env
INFISICAL_CLIENT_SECRET=your_client_secret
INFISICAL_CLIENT_ID=your_client_id
INFISICAL_PROJECT_ID=your_project_id
NODE_ENV=dev|test|prod
```
### 환경 변수 로드 우선순위
환경 변수는 다음 순서로 로드되며, 먼저 로드된 값이 우선합니다.
1. **LocalMachine 전역 환경 변수**
- Linux/macOS: `~/source/global-env-vars.txt`
- Windows: `%USERPROFILE%\source\global-env-vars.txt`
- 모든 프로젝트에서 공통으로 사용하는 전역 설정입니다.
2. **프로젝트별 환경 변수 파일**
- `./env-vars/{NODE_ENV}.env`
- 프로젝트 특화 설정입니다. 예: `dev.env`, `test.env`, `prod.env`
3. **쉘 환경 변수**
- 현재 쉘 세션에서만 쓰는 임시 설정입니다.
동일한 키가 여러 곳에 정의되어 있으면 우선순위가 높은 값이 사용됩니다. 앱 내부 dotenv 로더는 `override: false`로 동작하므로, 이미 설정된 값은 `env-vars/{NODE_ENV}.env`가 덮어쓰지 않습니다.
### IMDS
Infisical client id/secret은 환경 변수로 직접 넣을 수도 있고, IMDS 호환 local metadata service에서 읽을 수도 있습니다.
기본 후보는 개발 환경에서 `http://127.0.0.1:8675`, 운영 환경에서 `http://169.254.169.254`입니다. 필요하면 `METADATA_BASE_URL`, `IMDS_BASE_URL`, `MACHINE_ENV_BASE_URL`로 바꿀 수 있고, `IMDS_DISABLED=true`로 비활성화할 수 있습니다.
### Infisical Secrets
다음 시크릿이 Infisical에 설정되어 있어야 합니다.
| Secret | 설명 |
| --------------------------- | ------------------------- |
| `NATS_ENDPOINT` | NATS 서버 엔드포인트 |
| `NATS_USER` | NATS 사용자명 |
| `NATS_PASS` | NATS 비밀번호 |
| `GARNET_ENDPOINT` | Garnet/Redis 엔드포인트 |
| `GARNET_USERID` | Garnet/Redis 사용자명 |
| `GARNET_PASSWORD` | Garnet/Redis 비밀번호 |
| `MONGODB_CONNECTION_STRING` | MongoDB 연결 문자열 |
| `MONGODB_PROJECT_ID` | MongoDB project id |
| `SLACK_ALERT_WEBHOOK_URL` | Slack alert webhook, 선택 |
MinIO를 함께 사용하는 배포라면 아래 값도 Infisical에 둡니다.
| Secret | 설명 |
| ------------------ | --------------------- |
| `MINIO_ENDPOINT` | MinIO 서버 엔드포인트 |
| `MINIO_ACCESS_KEY` | MinIO 액세스 키 |
| `MINIO_SECRET_KEY` | MinIO 시크릿 키 |
## 자주 쓰는 명령
```bash
npm run build
npm run typecheck
npm run lint
npm test
npm run eval -- --skip-llm-judge
npm run eval:moderation -- --skip-llm-judge
npm run eval:handoff -- --skip-llm-judge
npm run eval:report -- --worst=20 --since=7d
npm run eval:router-shadow-report -- --since=7d
npm run review:list -- --limit=10
```
운영 환경에서는 실시간 agent(`src/main.ts`)와 background worker(`src/workers/background.ts`)를 함께 실행합니다.
## 문서
- [docs/multi-agent-architecture.md](docs/multi-agent-architecture.md): 전체 구조
- [docs/service-evaluation-operations.md](docs/service-evaluation-operations.md): 운영 runbook
- [docs/optimization-roadmap.md](docs/optimization-roadmap.md): 고도화 시작 조건
- [docs/resolution-hash-sunset-procedure.md](docs/resolution-hash-sunset-procedure.md): supervisor resolution hash 정리 절차
---
## English
# lolympus-agent
`lolympus-agent` is a NATS-driven multi-agent system for automating operations in the KakaoTalk-based Lolympus community. It supports new-user onboarding and replies to messages that ask about community rules.
## Overview
KakaoTalk operation rooms get many repeated questions about nickname format, heart verification, Discord profiles, and community rules. This service receives those messages, checks the current state, and produces short guidance when it can.
Unclear or risky requests are sent to the supervisor review queue so an operator can check them.
## Purpose
The goal is to automate repetitive onboarding and rule-answering work while keeping the tone and safety close to a human operator.
Simple cases use templates or lightweight LLM responses. More complex cases go through tool calls, handoff, and review queues.
## Functionalities
**1. Onboarding responses**
- Handles new member, nickname, photo, progress, timeout, and completion events.
- Generates short replies for user messages.
- Falls back to deterministic templates when an LLM response is not needed.
**2. Nickname and progress checks**
- Validates the community nickname format.
- Checks onboarding session progress from Redis/Garnet.
- Looks up room context when MongoDB is configured.
**3. Multi-agent routing**
- Separates onboarding, moderation, and supervisor skills.
- Handles skill-to-skill handoff through the shared dispatch loop in `src/agent/dispatch.ts`.
- Uses a transition matrix and budget guardrails to block invalid handoffs and loops.
**4. Tool-aware LLM runtime**
- Supports planner, responder, critic, and tool-loop execution modes.
- Can use local, proxy, or OpenRouter-compatible LLM providers.
- Includes circuit breaking, health checks, and response validation.
**5. Evaluation pipeline**
- Captures golden and shadow evaluation cases.
- Runs rule, LLM, process, and handoff judges.
- Supports review queues, calibration, drift checks, promotion, and tournaments to guide prompt and routing improvements.
**6. Background worker and alerts**
- Processes shadow eval and archive jobs outside the real-time response path.
- Stores eval results in MongoDB and sends NATS/Slack alerts when needed.
- Collects cases that need operator review in the supervisor review queue.
## Used technologies
- TypeScript
- Node.js
- NATS
- Redis / Garnet
- MongoDB / Mongoose
- Infisical
- OpenTelemetry
- Pino
- Zod
- Vitest
- ESLint
- Prettier
## How to run
1. Install dependencies.
```bash
npm install
```
2. Prepare environment variables.
```bash
cp env-vars/env.example env-vars/dev.env
```
If `env-vars/env.example` is not available, create `env-vars/dev.env` using the [Environment variables](#environment-variables) section below.
3. Run checks.
```bash
npm run typecheck
npm test
```
4. Start the dev service.
```bash
npm run start:dev
```
To run the agent and worker together with PM2:
```bash
npm run pm2:dev
```
## Environment variables
Infisical integration needs these variables.
```env
INFISICAL_CLIENT_SECRET=your_client_secret
INFISICAL_CLIENT_ID=your_client_id
INFISICAL_PROJECT_ID=your_project_id
NODE_ENV=dev|test|prod
```
### Loading priority
Environment variables are loaded in this order. Earlier values win.
1. **LocalMachine global environment variables**
- Linux/macOS: `~/source/global-env-vars.txt`
- Windows: `%USERPROFILE%\source\global-env-vars.txt`
- Shared settings used across local projects.
2. **Project environment file**
- `./env-vars/{NODE_ENV}.env`
- Project-specific values such as `dev.env`, `test.env`, and `prod.env`.
3. **Shell environment variables**
- Temporary values for the current shell session.
If the same key is defined in several places, the higher-priority value is used. The internal dotenv loader uses `override: false`, so `env-vars/{NODE_ENV}.env` does not overwrite values that already exist.
### IMDS
Infisical client id/secret can be provided directly as environment variables or fetched from an IMDS-compatible local metadata service.
The default candidates are `http://127.0.0.1:8675` in development and `http://169.254.169.254` in production. Override them with `METADATA_BASE_URL`, `IMDS_BASE_URL`, or `MACHINE_ENV_BASE_URL`; disable IMDS with `IMDS_DISABLED=true`.
### Infisical Secrets
These secrets should exist in Infisical.
| Secret | Description |
| --------------------------- | ---------------------------- |
| `NATS_ENDPOINT` | NATS server endpoint |
| `NATS_USER` | NATS username |
| `NATS_PASS` | NATS password |
| `GARNET_ENDPOINT` | Garnet/Redis endpoint |
| `GARNET_USERID` | Garnet/Redis username |
| `GARNET_PASSWORD` | Garnet/Redis password |
| `MONGODB_CONNECTION_STRING` | MongoDB connection string |
| `MONGODB_PROJECT_ID` | MongoDB project id |
| `SLACK_ALERT_WEBHOOK_URL` | Optional Slack alert webhook |
Deployments that also use MinIO should keep these in Infisical as well.
| Secret | Description |
| ------------------ | --------------------- |
| `MINIO_ENDPOINT` | MinIO server endpoint |
| `MINIO_ACCESS_KEY` | MinIO access key |
| `MINIO_SECRET_KEY` | MinIO secret key |
## Useful commands
```bash
npm run build
npm run typecheck
npm run lint
npm test
npm run eval -- --skip-llm-judge
npm run eval:moderation -- --skip-llm-judge
npm run eval:handoff -- --skip-llm-judge
npm run eval:report -- --worst=20 --since=7d
npm run eval:router-shadow-report -- --since=7d
npm run review:list -- --limit=10
```
In production, run both the real-time agent (`src/main.ts`) and the background worker (`src/workers/background.ts`).
## Documentation
- [docs/multi-agent-architecture.md](docs/multi-agent-architecture.md): full architecture
- [docs/service-evaluation-operations.md](docs/service-evaluation-operations.md): operations runbook
- [docs/optimization-roadmap.md](docs/optimization-roadmap.md): optimization triggers
- [docs/resolution-hash-sunset-procedure.md](docs/resolution-hash-sunset-procedure.md): supervisor resolution hash cleanup procedure
-178
View File
@@ -1,178 +0,0 @@
# lolympus-agent Multi-Agent Architecture
Last verified: Iter8 implementation branch, 2026-05-18.
## 1. 개요
`lolympus-agent`는 Lolympus 커뮤니티 운영을 돕는 NATS 메시지 드리븐 멀티 skill agent다. 실시간 요청은 `Agent.handleRequest()`로 들어오고, skill registry가 요청 skill을 찾아 `Skill.handle()`을 실행한다. 품질 관리는 shadow evaluation, MongoDB archive, human review, proposal/tournament loop로 이어진다.
운영 목표는 세 가지다.
- 실시간 응답 경로는 짧고 예측 가능하게 유지한다.
- skill, prompt, intent, resource, tool 정의는 각 `src/skills/<name>/` 폴더에 모은다.
- production과 eval은 같은 dispatch, handoff, budget 규칙을 사용한다.
## 2. 아키텍처 다이어그램
### 실시간 메시지 흐름
```text
NATS lolympus-agent.request
-> src/main.ts
-> Agent.handleRequest()
-> src/agent/dispatch.ts
-> SkillRegistry.get(request.skill)
-> Skill.handle(request, history)
-> AgentResponse
```
### Handoff 흐름
```text
Skill response
-> control token/parser or metadata.handoffRequest
-> transition-matrix validation
-> BudgetTracker
-> buildHandoffPacket()
-> dispatch target skill
-> supervisor fallback when blocked
```
`src/agent/dispatch.ts`는 production과 eval에서 모두 쓰인다. handoff chain, depth, last request는 metadata와 logger context에 남는다.
### Eval 흐름
```text
real-time response
-> ShadowEvalBus
-> lolympus-agent:background
-> ShadowWorker
-> lolympus-agent:eval:results
-> aggregation/archive/drift/review
-> propose/tournament/promote
```
### Operation 흐름
```text
supervisor handoff
-> lolympus-agent:review-queue
-> review:list/show/resolve/stats
-> resolution stream + compatibility hash
-> Slack/NATS alert
```
## 3. Skill 모델
모든 skill은 `src/skills/<name>/` 아래에 둔다. 필수 진입점은 다음과 같다.
| 파일 | 역할 |
| --------------------------------- | --------------------------------------- |
| `SKILL.md` | 사람이 읽는 capability card와 운영 설명 |
| `skill.ts` | `Skill` 구현체 |
| `intents.ts` | skill이 해석하는 intent 종류 |
| `prompt-data.ts` 또는 `prompts/*` | prompt layer, 예시, resource 연결 |
| `tools.ts` 또는 `resources/*` | skill 전용 tool/resource 정의 |
`src/skills/skill-loader.ts`가 `SKILL.md`를 읽어 `SkillManifest`로 만들고, `src/skills/validate-manifests.ts`가 문서와 코드의 일관성을 검사한다. `handoffTargets`는 `src/skills/transition-matrix.ts`에서 검증되며 self-loop와 unknown target을 막는다.
현재 핵심 skill은 `onboarding`, `moderation`, `supervisor`다. `supervisor`는 모든 skill의 fallback target이며, specialist가 처리하지 못한 요청을 review queue로 넘긴다.
## 4. Dispatch 모델
Dispatch는 `src/agent/dispatch.ts`가 담당한다.
- 응답 text 안의 control token은 parser가 제거하고 metadata로 구조화한다.
- skill이 직접 `metadata.handoffRequest`를 넣어도 같은 path로 처리된다.
- `HandoffPacket`은 원 요청, source response, handoff reason, evidence, source chain을 보존한다.
- `BudgetTracker`는 최대 handoff 수, 동일 skill 재방문, tool call, 비용 한도를 강제한다.
- transition matrix가 허용하지 않는 target은 supervisor로 reroute된다.
- budget이 막히면 `same-agent-revisit`, `max-hop`, `high-risk-action` 같은 reason으로 supervisor fallback이 만들어진다.
이 구조 덕분에 eval handoff golden도 실제 production dispatch와 같은 규칙을 검증한다.
## 5. Eval 모델
`src/eval/`은 네 계층으로 나뉜다.
| 계층 | 주요 파일 | 설명 |
| ------ | ------------------------------------ | ----------------------------------- |
| Golden | `src/eval/golden/*/cases.jsonl` | 사람이 관리하는 고정 회귀 케이스 |
| Runner | `runner.ts`, `handoff-dispatch.ts` | skill-aware eval 실행 |
| Judges | `judges/*` | rule, LLM, process, handoff 판단 |
| Loop | `archive.ts`, `promote.ts`, `loop/*` | archive 기반 개선 후보와 tournament |
Router 학습은 `src/eval/router-training.ts`가 MongoDB archive에서 training sample을 만들고, `src/skills/onboarding/runtime/router.ts`가 learned model을 저장한다. Iter8에서 추가된 `src/eval/router-shadow-report.ts`는 supervisor outcome과 shadow eval을 조인해 static router와 learned router의 divergence를 보고한다.
## 6. Observability
관측성은 세 갈래다.
- Logger context: requestId, skill, handoff chain, depth를 포함한다.
- Metrics: `src/observability/metrics.ts`가 Prometheus text exposition을 제공한다.
- Alerts: `src/observability/alert.ts`가 NATS와 Slack으로 운영 이벤트를 발행한다.
대표 metric:
| Metric | 용도 |
| ----------------------------- | ----------------------------- |
| `agent_handoff_total` | skill 간 handoff rate |
| `agent_handoff_loop_total` | loop guardrail 발생 |
| `agent_human_override_total` | supervisor/manual review 진입 |
| `agent_validator_fail_total` | validator 실패 |
| `agent_time_to_resolution_ms` | skill별 처리 시간 |
| `agent_token_cost_usd` | model/skill 비용 추정 |
`METRICS_PORT`의 `/metrics`와 `/healthz`를 scrape할 수 있다. Slack alert는 `SLACK_ALERT_WEBHOOK_URL`이 있을 때만 전송된다.
## 7. 데이터 저장소
### Garnet/Redis
| Key | Type | 용도 |
| ----------------------------------- | ------ | ------------------------------ |
| `lolympus-agent:background` | Stream | shadow eval과 summary job |
| `lolympus-agent:eval:results` | Stream | shadow eval 결과 |
| `lolympus-agent:eval:agg:*` | Hash | slice별 aggregate |
| `lolympus-agent:eval:review:queue` | Stream | eval human review |
| `lolympus-agent:review-queue` | Stream | supervisor review |
| `lolympus-agent:review-resolutions` | Stream | supervisor resolution audit |
| `lolympus-agent:review-resolved:*` | Hash | legacy compatibility read path |
| `lolympus-agent:alert:dedupe:*` | String | alert dedupe/rate limit |
### MongoDB
| Collection | 용도 |
| -------------------- | ------------------------------ |
| `eval_archive` | shadow eval 장기 archive |
| `eval_review_labels` | 사람 review label archive |
| `eval_tournaments` | prompt variant tournament 결과 |
Conversation, blackboard, summary state는 Garnet key와 TTL로 관리한다.
## 8. 확장 가이드
새 skill을 추가할 때는 다음 순서로 진행한다.
1. `src/skills/<name>/` 폴더를 만들고 `SKILL.md`, `skill.ts`, 필요한 `intents.ts`, `prompt-data.ts`, `tools.ts`를 둔다.
2. `SKILL.md`에 `name`, `domain`, `riskTier`, `canHandle`, `cannotHandle`, `handoffTargets`, `outputContract`, `triggers`를 작성한다.
3. factory/registry에 skill을 등록하고 transition matrix가 target을 인식하게 한다.
4. 필요한 resource는 skill 폴더의 `resources/`에 두고 prompt builder가 읽게 한다.
5. 최소 golden case를 `src/eval/golden/<name>/cases.jsonl`에 추가한다.
6. `npm run lint:skills`, `npm test`, 해당 skill eval smoke를 실행한다.
7. handoff가 필요한 skill이면 `src/eval/golden/handoff/cases.jsonl`에 교차 skill 케이스를 추가한다.
## 9. References
| 문서 | 의미 |
| ------------------------------------------------------------ | ------------------------------------------------------- |
| [refactor-skill-ssot.md](refactor-skill-ssot.md) | Iter1, skill/prompt/intent SSOT 시작 |
| [refactor-skill-ssot-iter2.md](refactor-skill-ssot-iter2.md) | Iter2, tool/eval path skill-aware화 |
| [refactor-skill-ssot-iter3.md](refactor-skill-ssot-iter3.md) | Iter3, eval pipeline과 SKILL.md 검증 강화 |
| [refactor-skill-ssot-iter4.md](refactor-skill-ssot-iter4.md) | Iter4, 멀티 에이전트 운영 조건 정의 |
| [refactor-skill-ssot-iter5.md](refactor-skill-ssot-iter5.md) | Iter5, bounded graph와 structured handoff |
| [refactor-skill-ssot-iter6.md](refactor-skill-ssot-iter6.md) | Iter6, supervisor review loop와 router training signals |
| [refactor-skill-ssot-iter7.md](refactor-skill-ssot-iter7.md) | Iter7, observability와 Slack alert |
| [refactor-skill-ssot-iter8.md](refactor-skill-ssot-iter8.md) | Iter8, closing iteration |
운영 runbook은 [service-evaluation-operations.md](service-evaluation-operations.md), 고도화 trigger는 [optimization-roadmap.md](optimization-roadmap.md), resolution hash sunset은 [resolution-hash-sunset-procedure.md](resolution-hash-sunset-procedure.md)를 참고한다.
-66
View File
@@ -1,66 +0,0 @@
# Optimization Roadmap
Last verified: Iter8 implementation branch, 2026-05-18.
운영 가능한 멀티 에이전트 기반은 Iter8에서 닫았다. 이 문서는 다음 고도화를 추측으로 시작하지 않기 위한 trigger 표다. Trigger는 권장 시작점이며, 최종 착수는 사람이 실제 비용, 운영 부담, 사용자 영향도를 보고 결정한다.
## Trigger Summary
| 항목 | Trigger 조건 | 측정 지표 | Plan 작성 시작 시점 |
| ------------------------- | ------------------------------------------------------- | ----------------------------------------- | ------------------------------- |
| Cost-aware routing | 월 LLM 비용이 운영 예산을 초과 | `agent_token_cost_usd` 합계 | 비용 초과가 2개월 연속 발생 |
| Parallel fan-out | handoff 평균 hop이 2.0 이상 또는 timeout 빈발 | `agent_handoff_count` p50/p95 | hop >= 2.0이 1주 지속 |
| Cross-room context | 방 사이의 동일 사용자/패턴 이슈가 반복 | 신규 metric 필요 | sensor 먼저 추가 후 데이터 확인 |
| Resolution hash sunset | dual-write 안정 2개월 + stream/hash divergence 0 | sunset 점검 script 또는 spot check | 조건 충족 후 sunset plan 실행 |
| Review queue Web UI | 일일 review queue 진입이 10건 이상 또는 운영자 2명 이상 | `agent_human_override_total` daily | 1개월 지속 |
| Runtime skill discovery | 신규 skill 배포가 주 1회 이상 | 배포 기록, `lint:skills` 변경 빈도 | 빈도 조건 충족 |
| Multi-skill orchestration | handoff chain 평균이 3 hop 이상 | `agent_handoff_count` p90, handoff golden | 평균 3 hop 이상 반복 |
| Supervisor 자동 학습 | router shadow agreement rate가 80% 미만 | `eval:router-shadow-report` | 2개 이상 window에서 80% 미만 |
## Cost-aware routing
왜 지금 안 하는가: 현재는 비용보다 correctness와 운영 가능성이 더 큰 리스크다. tier별 routing과 learned router가 이미 비용을 일부 통제한다.
시작 시 검토할 것: model별 token cost 산정, skill별 비용 dashboard, router decision의 false negative 비용, fallback 품질이다. 관련 코드는 `src/observability/metrics.ts`, `src/skills/onboarding/runtime/router.ts`, `src/eval/router-training.ts`다.
## Parallel fan-out
왜 지금 안 하는가: handoff graph가 아직 작고, 병렬 실행은 비용과 race condition을 늘린다.
시작 시 검토할 것: fan-out 대상 skill의 idempotency, response merge policy, budget accounting, timeout policy다. 관련 코드는 `src/agent/dispatch.ts`, `src/agent/budget.ts`, `src/eval/golden/handoff/cases.jsonl`다.
## Cross-room context
왜 지금 안 하는가: room 간 context가 실제로 필요한 빈도와 개인정보 리스크가 아직 수치화되지 않았다.
시작 시 검토할 것: 사용자 단위 aggregation metric, retention/TTL, opt-in 정책, PII redaction이다. 먼저 sensor와 metric을 추가하고 한 달 이상 관찰한다.
## Resolution hash sunset
왜 지금 안 하는가: Iter6에서 stream audit와 hash compatibility dual-write를 막 붙였다. 바로 제거하면 운영 rollback 경로가 좁아진다.
시작 시 검토할 것: stream과 hash의 일치율, `review:show`와 `review:stats` read path, TTL 이후 복구 가능성이다. 절차는 [resolution-hash-sunset-procedure.md](resolution-hash-sunset-procedure.md)를 따른다.
## Review queue Web UI
왜 지금 안 하는가: CLI로 충분한 규모인지 확인되지 않았다. Web UI는 인증, audit, 권한, 배포 표면을 늘린다.
시작 시 검토할 것: 일일 queue 양, 운영자 수, 자주 쓰는 resolution action, note template, audit export 필요성이다. CLI는 `src/operations/cli.ts`가 기준이다.
## Runtime skill discovery
왜 지금 안 하는가: 현재 skill 수가 적고 정적 registry가 변경 영향 파악에 유리하다.
시작 시 검토할 것: skill package format, hot reload, SKILL.md validation timing, transition matrix reload, 배포 rollback 방식이다.
## Multi-skill orchestration
왜 지금 안 하는가: 지금의 handoff는 sequential dispatch로 충분하다. orchestration graph를 일반화하면 디버깅 난도가 크게 올라간다.
시작 시 검토할 것: graph DSL, response merge, partial failure, supervisor override, eval judge 확장이다.
## Supervisor 자동 학습
왜 지금 안 하는가: supervisor routing과 userMessage는 이제 막 안정 가드를 갖췄다. 자동 학습은 충분한 resolution label과 shadow divergence evidence가 필요하다.
시작 시 검토할 것: `npm run eval:router-shadow-report -- --since=7d`, supervisor resolution action 분포, target별 false route, human note taxonomy다.
+575
View File
@@ -0,0 +1,575 @@
# Lolympus Agent 포트폴리오 정리본
> Confluence에 그대로 옮겨도 읽히도록 작성한 상세 설명 자료입니다.
> Mermaid 다이어그램은 Confluence Mermaid 매크로나 Mermaid 지원 플러그인에 붙여 넣으면 됩니다.
## 1. 프로젝트 한 줄 소개
Lolympus Agent는 게임 커뮤니티의 신규 멤버 온보딩을 자동화하는 TypeScript 기반 LLM Agent입니다.
카카오톡 오픈채팅 본체 서비스가 NATS request/reply로 요청을 보내면, Agent가 Redis/Garnet에 저장된 온보딩 세션, 최근 대화, MongoDB 채팅방 맥락, LLM 응답, 도구 호출 결과를 조합해 신규 멤버에게 필요한 안내만 짧고 정확하게 반환합니다.
핵심은 단순 챗봇이 아니라, 커뮤니티 규칙과 유저 상태를 기반으로 LLM을 제어하는 운영형 온보딩 에이전트라는 점입니다.
## 2. 포트폴리오 요약
| 항목 | 내용 |
| --- | --- |
| 프로젝트명 | Lolympus Agent |
| 목적 | 게임 커뮤니티 신규 멤버 등록 절차 자동 안내 |
| 주요 기능 | 닉네임 형식 안내, 하트 인증 안내, 진행상황 안내, 마감 임박 안내, 디스코드 닉네임 확인, 잡담/타인 멘션 응답 스킵 |
| 핵심 기술 | TypeScript, Node.js, NATS, Redis/Garnet, MongoDB/Mongoose, OpenTelemetry, Pino, Zod, Vitest, LLM provider routing |
| 아키텍처 키워드 | Event-driven, request/reply agent, state-driven orchestration, tool calling, shadow evaluation, circuit breaker, background worker |
| 운영 포인트 | 요청 단위 tracing/logging, LLM endpoint circuit breaker, shadow eval sampling, alert subject 발행, Redis Stream 기반 worker/DLQ |
## 3. 문제 정의
커뮤니티 신규 멤버 등록은 반복적인 안내가 많지만, 실제 응답은 단순 FAQ보다 복잡합니다.
1. 유저마다 남은 절차가 다릅니다.
예를 들어 닉네임은 완료했지만 하트 인증은 안 했을 수 있고, 1단계 이후 디스코드 프로필 단계가 남아 있을 수도 있습니다.
2. 질문 의도가 짧고 모호합니다.
"이제 뭐 남았죠?", "어디서 해요?", "닉변 됐나요?" 같은 메시지는 세션 상태와 최근 대화 맥락을 함께 봐야 정확히 답할 수 있습니다.
3. LLM만 사용하면 규칙 위반 가능성이 있습니다.
디스코드 안내를 너무 일찍 하거나, 잘못된 닉네임 예시를 들거나, 유저가 묻지 않은 절차를 길게 붙이는 문제가 생길 수 있습니다.
4. 운영 환경에서는 응답 시간과 장애 대응도 중요합니다.
LLM endpoint 장애, 네트워크 지연, 도구 호출 실패가 있어도 서비스 전체가 멈추지 않아야 합니다.
이 프로젝트는 위 문제를 해결하기 위해 "결정론적 상태 판단 + LLM 응답 생성 + validator + fallback template + shadow eval" 구조로 설계했습니다.
## 4. 전체 아키텍처
```mermaid
flowchart LR
Main["카카오톡 본체 서비스"] -->|"NATS request/reply<br/>lolympus-agent.request"| Agent["Agent<br/>src/agent/agent.ts"]
Agent -->|"대화 조회/저장"| Conv[("Garnet/Redis<br/>conversation history")]
Agent --> Skill["OnboardingSkill<br/>src/skills/onboarding/skill.ts"]
Skill -->|"온보딩 세션 조회"| Session[("Garnet/Redis<br/>lolympus:onboarding:*")]
Skill -->|"최근 방 대화 조회"| Mongo[("MongoDB<br/>events collection")]
Skill --> Classifier["Intent Classifier<br/>regex first, LLM fallback"]
Skill --> Derived["State Derivation<br/>phase, intent, tier"]
Skill --> Prompt["Prompt Builder<br/>persona/policy/state/examples/context"]
Skill --> Orchestrator["Orchestrator<br/>tier 0/1/2"]
Orchestrator --> LLM["LlmClient<br/>provider routing + circuit breaker"]
LLM --> Provider1["CLI Proxy"]
LLM --> Provider2["OpenRouter"]
LLM --> Provider3["Local Llama"]
Orchestrator --> Tools["Tool Loop<br/>Zod schema tools"]
Tools -->|"Discord lookup"| Main
Tools -->|"rule/session/context lookup"| Session
Skill --> Validator["Validator<br/>state consistency, URL whitelist, length"]
Skill -->|"shadow eval job"| BgStream[("Redis Stream<br/>lolympus-agent:background")]
BgStream --> Worker["Background Worker<br/>src/workers/background.ts"]
Worker --> Eval[("Eval results stream")]
Worker --> Alert["NATS alert<br/>lolympus-agent.alert"]
Agent -->|"AgentResponse"| Main
```
## 5. 요청 처리 시퀀스
```mermaid
sequenceDiagram
autonumber
participant Main as 카카오톡 본체
participant NATS as NATS
participant Agent as Agent
participant Conv as ConversationStore
participant Skill as OnboardingSkill
participant Redis as Garnet/Redis
participant Mongo as MongoDB RoomContext
participant LLM as LlmClient
participant Eval as ShadowEvalBus
Main->>NATS: request(lolympus-agent.request, AgentRequest)
NATS->>Agent: raw message 전달
Agent->>Agent: request schema 검증
Agent->>Agent: AsyncLocalStorage request context 생성
Agent->>Conv: getHistory(chatroomId, userId)
Conv->>Redis: lolympus-agent:conv:* 조회
Redis-->>Conv: 최근 대화
Conv-->>Agent: history
opt userMessage가 있는 경우
Agent->>Conv: appendMessage(user)
Conv->>Redis: conversation 저장
end
Agent->>Skill: handle(request, history)
Skill->>Redis: 온보딩 세션 조회
Redis-->>Skill: OnboardingSession
Skill->>Skill: classify(trigger, userMessage)
Skill->>Skill: deriveState(session, trigger, intent)
alt shouldRespond = false
Skill-->>Agent: empty response
else template mode
Skill->>Skill: template fallback 생성
Skill-->>Agent: template response
else LLM mode
par context load
Skill->>Redis: blackboard 조회
Skill->>Redis: summary 조회
Skill->>Mongo: recent room messages 조회
end
Skill->>Skill: PromptBuilder.build()
Skill->>LLM: chatCompletion(messages, tier/model/deadline)
LLM-->>Skill: draft response
Skill->>Skill: validateOnboardingResponse()
alt validator reject or LLM failure
Skill->>Skill: template fallback
end
Skill-->>Agent: final response
end
opt 응답 text가 있는 경우
Skill->>Redis: blackboard update
Skill->>Eval: shadow eval publish
Eval->>Redis: XADD lolympus-agent:background
Agent->>Conv: appendMessage(assistant)
end
Agent->>NATS: respond(AgentResponse)
NATS-->>Main: response
```
## 6. 디스코드 닉네임 확인 시퀀스
디스코드 닉네임 확인은 LLM이 직접 추측하지 않도록 tool calling으로 분리했습니다.
LLM은 "조회가 필요한 상황"을 판단하고, 실제 확인은 `verify-discord-nickname` 도구가 NATS로 본체 서비스에 요청합니다.
```mermaid
sequenceDiagram
autonumber
participant User as 유저
participant Agent as OnboardingSkill
participant Orchestrator as Orchestrator
participant LLM as LLM
participant ToolLoop as ToolLoop
participant Tool as verify-discord-nickname
participant Main as 본체 Discord Lookup
participant Redis as Garnet/Redis
User->>Agent: "디코 닉변 됐나요?"
Agent->>Agent: intent = discord-verify-question
Agent->>Agent: responseMode = llm-with-tools, tier = 1
Agent->>Orchestrator: run(derived, messages, tools)
Orchestrator->>LLM: planner JSON 요청
LLM-->>Orchestrator: suggestedTools = [verify-discord-nickname]
Orchestrator->>ToolLoop: responder with native/Hermes tools
ToolLoop->>LLM: tool 사용 가능한 메시지 전달
LLM-->>ToolLoop: tool_call verify-discord-nickname
ToolLoop->>Tool: execute()
Tool->>Tool: 세션 닉네임 형식 검사
Tool->>Main: NATS request(lolympus.discord.lookup-members)
Main-->>Tool: 후보 Discord member 목록
Tool->>Tool: Kakao nickname 기반 Discord nickname rule 검증
opt Discord memberId 확인됨
Tool->>Redis: session.discordMemberId 저장
end
Tool-->>ToolLoop: 구조화된 결과 반환
ToolLoop->>LLM: tool result 전달
LLM-->>ToolLoop: 최종 한국어 응답
ToolLoop-->>Orchestrator: final response
Orchestrator-->>Agent: response
```
## 7. Shadow Eval 및 Alert 시퀀스
운영 트래픽 전체를 동기 평가하면 응답 지연이 커지므로, 실제 응답과 평가를 분리했습니다.
Agent는 샘플링된 요청만 Redis Stream에 넣고, Background Worker가 비동기로 validator와 LLM judge를 실행합니다.
```mermaid
sequenceDiagram
autonumber
participant Skill as OnboardingSkill
participant Stream as Redis Stream
participant Worker as ShadowWorker
participant Rule as Rule Judge
participant LLMJudge as LLM Judge
participant Results as Eval Results Stream
participant Alert as AlertService
participant NATS as NATS
Skill->>Stream: XADD type=shadow-eval
Worker->>Stream: XREADGROUP bg-worker
Worker->>Rule: validator 기반 rule judge
Worker->>LLMJudge: friendliness/accuracy/concision/scope 평가
Worker->>Results: XADD shadow-eval-result
Worker->>Worker: 최근 5분 latency/score/error 집계
alt p95 latency, score drop, LLM error rate, circuit open 임계값 초과
Worker->>Alert: sendAlert()
Alert->>Stream: dedupe key SET NX
Alert->>NATS: publish lolympus-agent.alert
end
Worker->>Stream: XACK
```
## 8. 상태 모델
온보딩 응답은 "지금 유저가 어느 단계인지"와 "무엇을 물었는지"를 분리해 판단합니다.
```mermaid
stateDiagram-v2
[*] --> fresh
fresh --> both_pending: 세션 있음, 닉네임/사진 미완료
both_pending --> nickname_pending: 사진 완료
both_pending --> photo_pending: 닉네임 완료
nickname_pending --> step1_completed: 닉네임 완료
photo_pending --> step1_completed: 사진 완료
step1_completed --> completed: 확장 추적 항목 모두 완료
step1_completed --> deadline_near: 마감 임박
both_pending --> nickname_rejected: nickname-invalid trigger
both_pending --> photo_rejected: photo-failed trigger
nickname_pending --> nickname_rejected: nickname-invalid trigger
photo_pending --> photo_rejected: photo-failed trigger
fresh --> timed_out: status = timed_out
completed --> [*]
timed_out --> [*]
```
주요 intent는 다음과 같습니다.
| Intent | 의미 | 처리 방향 |
| --- | --- | --- |
| `greeting` | 인사 | 짧게 응답 |
| `acknowledgement` | 감사/확인 | 짧게 응답 |
| `status-question` | 남은 절차 질문 | 완료된 절차 제외, 남은 것만 안내 |
| `help-how` | 방법 질문 | 해당 단계 방법만 안내 |
| `help-where` | 위치 질문 | 위치/경로만 안내 |
| `nickname-question` | 카카오톡 닉네임 질문 | 형식/오류 원인 안내 |
| `photo-question` | 하트 인증 질문 | 하트 인증만 안내 |
| `discord-question` | 디스코드 일반 질문 | 1단계 완료 여부에 따라 제한 |
| `discord-verify-question` | 디스코드 닉네임 확인 | tool calling으로 실제 조회 |
| `chitchat`, `off-topic-to-other` | 잡담/타인 대상 메시지 | `[NO_RESPONSE]` 또는 빈 응답 |
## 9. 핵심 구현 포인트
### 9.1 NATS 기반 Agent 레이어
관련 파일: `src/main.ts`, `src/agent/agent.ts`, `src/agent/types.ts`
- `src/main.ts`에서 환경변수 로딩, logger/tracing 초기화, NATS/Garnet/MongoDB 연결, LLM client와 skill 등록을 수행합니다.
- Agent는 `lolympus-agent.request` subject를 구독하고 request/reply 패턴으로 응답합니다.
- 모든 요청은 `AsyncLocalStorage` 기반 request context로 감싸져 requestId, chatroomId, userId, trigger가 로그와 span에 자동 포함됩니다.
- 요청 전후로 conversation history를 Redis에 저장해 multi-turn 응답 품질을 유지합니다.
### 9.2 Regex-first intent classifier
관련 파일: `src/skills/onboarding/state/classifier.ts`, `src/skills/onboarding/message-filter.ts`
Classifier는 비용과 latency를 줄이기 위해 regex를 먼저 사용합니다.
- trigger가 명확한 이벤트는 바로 intent로 매핑합니다. 예: `nickname-invalid` -> `nickname-question`
- 인사, 감사, 진행상황 질문, 위치 질문, 디스코드 확인 질문 등 자주 나오는 표현은 regex로 처리합니다.
- regex confidence가 낮은 경우에만 LLM JSON classifier로 fallback합니다.
이 구조는 운영상 유리합니다. 흔한 질문은 LLM 호출 없이 빠르게 처리하고, 애매한 질문만 LLM에 맡길 수 있습니다.
### 9.3 Derived State로 응답 모드 결정
관련 파일: `src/skills/onboarding/state/derive.ts`, `src/skills/onboarding/state/transitions.ts`
`deriveState()`는 세션, trigger, intent를 조합해 다음 값을 결정합니다.
- `phase`: 현재 온보딩 단계
- `shouldRespond`: 응답할지 스킵할지
- `responseMode`: `skip`, `template`, `llm`, `llm-with-tools`
- `tier`: LLM orchestration 단계
- `suggestedMaxTokens`: 상황별 token budget
- `allowShortResponse`: 짧은 응답 허용 여부
예를 들어 닉네임 오류 상태는 `llm-with-tools`, tier 1로 승격됩니다. 반대로 잡담이나 타인 멘션은 응답하지 않습니다.
### 9.4 Layered Prompt Builder
관련 파일: `src/skills/onboarding/prompts/builder.ts`, `src/skills/onboarding/prompts/layers.ts`
Prompt는 하나의 거대한 system prompt가 아니라 여러 layer를 조립합니다.
1. `personaLayer`: 커뮤니티 선배 역할과 핵심 규칙
2. `policyLayer`: intent별 답변 정책
3. `stateLayer`: 현재 세션 상태와 남은 절차
4. `examplesLayer`: intent별 few-shot 예시
5. `contextLayer`: 최근 대화, 요약, 방 분위기, blackboard 메모
이 방식의 장점은 변경 범위를 줄일 수 있다는 점입니다.
예를 들어 디스코드 응답 정책을 수정할 때 전체 prompt를 건드리지 않고 `policyLayer`와 관련 예시만 조정할 수 있습니다.
### 9.5 Tier 기반 LLM Orchestration
관련 파일: `src/skills/onboarding/runtime/orchestrator.ts`, `src/skills/onboarding/runtime/critic.ts`
모든 요청을 동일하게 처리하지 않고 난이도에 따라 tier를 나눕니다.
| Tier | 흐름 | 사용 상황 |
| --- | --- | --- |
| Tier 0 | Direct responder | 인사, 상태 질문, 일반 안내 |
| Tier 1 | Planner -> Responder | 닉네임 오류, 디스코드 확인, 마감 임박 |
| Tier 2 | Planner -> Responder -> Critic/Rewrite | 더 높은 정확도가 필요한 복잡 상황 |
Orchestrator는 request deadline을 확인해 남은 시간이 부족하면 고비용 tier를 포기하고 더 빠른 경로로 내려갑니다.
즉, 품질과 응답 시간을 함께 관리합니다.
### 9.6 LLM Provider Routing과 Circuit Breaker
관련 파일: `src/llm/llm-client.ts`, `src/llm/routing.ts`, `src/llm/circuit.ts`
LLM client는 provider 우선순위를 적용합니다.
1. CLI Proxy
2. OpenRouter
3. Local Llama
각 endpoint는 health check와 circuit breaker 상태를 거칩니다.
연속 실패가 발생하면 해당 endpoint를 cooldown 상태로 두고 다른 provider로 fallback합니다.
### 9.7 Tool Calling 호환 계층
관련 파일: `src/tools/*`, `src/skills/onboarding/runtime/tool-loop.ts`
Tool interface는 Zod schema를 포함합니다.
```ts
interface Tool<A, R> {
name: string;
description: string;
argsSchema: ZodType<A>;
resultSchema: ZodType<R>;
execute(args: A, ctx: ToolContext): Promise<R>;
}
```
지원 도구:
| Tool | 역할 |
| --- | --- |
| `validate-nickname` | 카카오톡 닉네임 형식 검사 |
| `check-onboarding-progress` | 현재 남은 절차 구조화 |
| `lookup-community-rule` | 나이 제한, 티어 코드, 하트 규칙 등 상수 조회 |
| `get-room-context` | 최근 채팅방 메시지 조회 |
| `verify-discord-nickname` | NATS로 본체에 Discord member lookup 요청 |
Parser는 OpenAI native `tool_calls`와 Hermes `<tool_call>` 포맷을 모두 처리합니다.
덕분에 provider가 바뀌어도 Agent runtime은 같은 tool interface를 사용할 수 있습니다.
### 9.8 LLM 응답 Validator
관련 파일: `src/skills/onboarding/runtime/validator.ts`
LLM 응답은 그대로 내보내지 않고 후처리와 검증을 거칩니다.
- `<think>` 제거
- 허용되지 않은 URL 제거
- 마크다운 제거
- 최대 길이 제한
- 빈 응답과 너무 짧은 응답 reject
- 디스코드 안내가 1단계 완료 전 과하게 나가는지 검사
- 잘못된 티어 코드가 포함된 닉네임 예시 reject
- intent별 기대 키워드 soft warning
- 반복 phrase soft warning
reject된 응답은 재시도하거나 template fallback으로 내려갑니다.
### 9.9 Memory와 Context
관련 파일: `src/conversation/conversation-store.ts`, `src/conversation/summarizer.ts`, `src/skills/onboarding/state/blackboard.ts`, `src/room-context/room-context-service.ts`
대화 품질을 위해 세 가지 context를 사용합니다.
| Context | 저장 위치 | 목적 |
| --- | --- | --- |
| Conversation history | `lolympus-agent:conv:{chatroomId}:{userId}` | 최근 user/assistant turn 유지 |
| Conversation summary | `lolympus-agent:summary:{chatroomId}:{userId}` | 긴 대화 압축 |
| Blackboard | `lolympus-agent:bb:{chatroomId}:{userId}` | 최근 intent, 열린 질문, 닉네임 시도, tone memory 저장 |
| Room context | MongoDB `events` collection | 최근 방 분위기와 타인 대상 메시지 판단 |
5턴 이상 누적되면 background stream에 summarize job을 넣고 worker가 비동기로 요약합니다.
### 9.10 Eval, Observability, Alert
관련 파일: `src/eval/*`, `src/observability/*`, `src/workers/background.ts`
운영 품질 관리를 위해 다음 구조를 갖고 있습니다.
- Pino logger: 환경별 log level, secret redaction
- OpenTelemetry spans: agent, skill, classifier, LLM, tool 단위 span
- ShadowEvalBus: 실제 응답을 샘플링해 Redis Stream에 기록
- ShadowWorker: rule judge, LLM judge, summary job 처리
- AlertService: latency, eval score drop, LLM error rate, open circuit 발생 시 NATS alert 발행
- DLQ: background job이 반복 실패하면 `lolympus-agent:background:dlq`로 이동
## 10. 주요 데이터 계약
### 10.1 Agent Request
```ts
interface AgentRequest {
requestId: string;
skill: string;
chatroomId: string;
userId: string;
nickname: string;
trigger: string;
userMessage?: string;
context?: Record<string, unknown>;
}
```
### 10.2 Agent Response
```ts
interface AgentResponse {
requestId: string;
text: string;
confidence: number;
source: "llm" | "template";
metadata?: Record<string, unknown>;
}
```
### 10.3 Redis/Garnet Key 설계
| Key | 목적 |
| --- | --- |
| `lolympus:onboarding:{chatroomId}:{userId}` | 본체가 작성하는 온보딩 세션. Agent는 주로 read |
| `lolympus-agent:conv:{chatroomId}:{userId}` | 최근 대화 |
| `lolympus-agent:summary:{chatroomId}:{userId}` | 대화 요약 |
| `lolympus-agent:bb:{chatroomId}:{userId}` | 온보딩 blackboard |
| `lolympus-agent:background` | shadow eval 및 summarize job stream |
| `lolympus-agent:background:dlq` | worker 실패 job DLQ |
| `lolympus-agent:eval:results` | shadow eval 결과 |
| `lolympus-agent:circuit:*` | LLM endpoint circuit breaker 상태 |
| `lolympus-agent:alert:dedupe:*` | alert rate limit/dedupe |
### 10.4 NATS Subject
| Subject | 방향 | 목적 |
| --- | --- | --- |
| `lolympus-agent.request` | 본체 -> Agent | 온보딩 응답 요청 |
| `lolympus.discord.lookup-members` | Agent -> 본체 | Discord member 조회 |
| `lolympus-agent.alert` | Agent worker -> 본체 | 운영 alert |
| `lolympus-agent.eval.report` | Eval CLI -> 본체/관찰자 | eval report 발행 |
## 11. 현재 리포지토리 기준 품질 지표
현재 저장된 리포지토리 스냅샷 기준입니다. 운영 KPI라기보다는 회귀 방지용 품질 자료로 보는 것이 안전합니다.
| 항목 | 현재 값 |
| --- | --- |
| 테스트 파일 | `src/**/*.test.ts` 기준 16개 |
| 주요 테스트 영역 | LLM routing, alert, eval worker, judges, shadow bus, parser, classifier, state derive, prompt builder, validator, tool loop, orchestrator, built-in tools |
| Golden eval case | 3개 |
| 저장된 baseline pass rate | 1.0 |
| 저장된 baseline rule average score | 1.0 |
| 저장된 baseline average latency | 3ms |
| 저장된 baseline LLM judge case | 0개 |
포트폴리오에는 다음처럼 쓰는 것이 좋습니다.
> 현재는 rule-based golden eval을 smoke benchmark로 운영하고 있으며, shadow eval과 LLM judge를 통해 운영 트래픽 기반 품질 평가로 확장 가능한 구조를 구현했습니다.
## 12. 포트폴리오에서 강조할 수 있는 기술적 성과
### 12.1 LLM을 제품 로직 안에 안전하게 넣은 구조
LLM에게 모든 판단을 맡기지 않고, `Intent -> DerivedState -> Prompt -> Orchestrator -> Validator -> Fallback` 파이프라인으로 감쌌습니다.
덕분에 커뮤니티 규칙, 유저 진행 상태, 응답 범위 제한을 코드 레벨에서 제어할 수 있습니다.
### 12.2 비용과 latency를 고려한 hybrid classifier
자주 나오는 짧은 메시지는 regex로 처리하고, 애매한 메시지만 LLM classifier로 넘깁니다.
이 방식은 LLM 호출 비용을 줄이고 p50 latency를 낮추는 데 유리합니다.
### 12.3 Provider 장애를 고려한 LLM client
Provider priority, health check, circuit breaker, fallback model을 한 레이어에 모았습니다.
외부 LLM endpoint가 불안정해도 서비스는 다른 endpoint 또는 template fallback으로 계속 응답할 수 있습니다.
### 12.4 Tool calling으로 사실 확인 분리
디스코드 닉네임 확인처럼 실제 조회가 필요한 기능은 LLM 추측이 아니라 tool execution으로 처리합니다.
또한 OpenAI tool call과 Hermes tool call을 모두 파싱해 provider 교체 가능성을 높였습니다.
### 12.5 운영 평가 루프
실제 응답을 shadow eval stream에 샘플링하고, background worker가 rule judge/LLM judge를 비동기로 수행합니다.
평가 결과가 나빠지거나 latency가 올라가면 alert subject로 알릴 수 있어, 단순 구현을 넘어 운영 가능한 LLM 시스템 형태를 갖췄습니다.
## 13. Confluence 페이지 구성 추천
Confluence에는 아래 순서로 넣으면 읽기 좋습니다.
1. 프로젝트 한 줄 소개
2. 문제 정의
3. 전체 아키텍처 다이어그램
4. 요청 처리 시퀀스 다이어그램
5. 핵심 구현 포인트
6. 운영 안정성 및 평가 구조
7. 데이터 계약과 저장소 설계
8. 성과/품질 지표
9. 배운 점과 개선 계획
## 14. 면접/리뷰에서 말하기 좋은 포인트
### Q. 왜 LLM만 쓰지 않고 상태 기반 파이프라인을 만들었나요?
온보딩 안내는 "친절한 문장"보다 "현재 단계에 맞는 정확한 안내"가 더 중요했습니다.
LLM은 표현 생성에 강하지만, 디스코드 안내 가능 여부나 닉네임 형식 같은 규칙 판단은 코드가 더 안정적입니다. 그래서 상태 판단과 validator는 코드가 담당하고, LLM은 최종 문장 생성과 애매한 의도 해석에 집중하게 했습니다.
### Q. 왜 tier를 나눴나요?
모든 요청에 planner, tool, critic을 붙이면 품질은 올라갈 수 있지만 latency와 비용이 커집니다.
인사나 단순 진행상황 질문은 direct response로 빠르게 처리하고, 닉네임 오류나 디스코드 확인처럼 정확도가 중요한 경우만 tier 1 이상으로 승격했습니다.
### Q. 환각은 어떻게 줄였나요?
세 가지 방식으로 줄였습니다.
1. prompt에 현재 세션 상태와 남은 절차를 명시합니다.
2. 실제 조회가 필요한 것은 tool calling으로 분리합니다.
3. 최종 응답을 validator로 검사하고, reject되면 재시도 또는 template fallback을 사용합니다.
### Q. 운영 중 품질 저하는 어떻게 감지하나요?
실제 응답을 Redis Stream에 shadow eval job으로 기록하고, worker가 rule judge와 LLM judge를 비동기로 실행합니다.
최근 5분 기준 p95 latency, LLM judge 점수 하락, judge error rate, LLM circuit open 상태를 감지해 NATS alert를 발행합니다.
### Q. 이 프로젝트에서 가장 어려웠던 점은 무엇인가요?
LLM 응답을 자연스럽게 만들면서도, 커뮤니티 규칙과 진행 상태를 벗어나지 않게 제어하는 점이 가장 어려웠습니다.
이를 해결하기 위해 prompt engineering만 하는 대신, 상태 모델, 도구, validator, fallback, eval을 하나의 파이프라인으로 묶었습니다.
## 15. 개선 계획
현재 구조에서 다음을 추가하면 포트폴리오 완성도가 더 올라갑니다.
| 개선 항목 | 기대 효과 |
| --- | --- |
| Golden set 30-50개로 확장 | 회귀 테스트 신뢰도 향상 |
| 운영 dashboard 추가 | latency, fallback rate, judge score 추이를 시각화 |
| OTLP exporter 연동 | Jaeger/Grafana Tempo 등으로 trace 분석 |
| prompt/version metadata 저장 | 응답 품질 저하 원인 추적 |
| eval case 자동 익명화 강화 | 실제 운영 데이터 기반 평가셋 구축 |
| 실패 응답 샘플 리뷰 페이지 | 운영자가 직접 prompt/tool 개선 포인트 확인 |
## 16. 포트폴리오용 짧은 소개 문장
> TypeScript와 NATS 기반으로 게임 커뮤니티 신규 멤버 온보딩을 자동화하는 LLM Agent를 설계/구현했습니다. Redis/Garnet 세션 상태, MongoDB 채팅 맥락, regex-first intent classifier, tier 기반 LLM orchestration, Zod schema tool calling, validator/fallback, shadow eval worker를 결합해 단순 챗봇이 아니라 운영 가능한 온보딩 자동화 시스템으로 구성했습니다.
## 17. 이력서용 bullet 예시
- NATS request/reply 기반 LLM Agent를 구현해 카카오톡 커뮤니티 신규 멤버 온보딩 안내를 자동화했습니다.
- Regex-first intent classifier와 상태 기반 `DerivedState` 모델을 설계해 응답 여부, LLM 사용 여부, tool calling 필요 여부를 결정했습니다.
- Tier 0/1/2 orchestration을 도입해 단순 질문은 빠르게, 검증이 필요한 질문은 planner/tool/critic 경로로 처리하도록 구성했습니다.
- Zod schema 기반 tool registry와 OpenAI/Hermes tool call parser를 구현해 provider에 덜 종속적인 tool execution 구조를 만들었습니다.
- LLM 응답 validator, URL whitelist, state consistency check, template fallback으로 커뮤니티 규칙 위반 가능성을 낮췄습니다.
- Redis Stream 기반 shadow eval worker와 alert pipeline을 구현해 운영 트래픽 기반 품질 평가 및 장애 감지 구조를 마련했습니다.
-101
View File
@@ -1,101 +0,0 @@
# Supervisor Resolution Hash Sunset Procedure
Last verified: Iter8 implementation branch, 2026-05-18.
Supervisor resolution은 현재 두 곳에 기록된다.
- Audit stream: `lolympus-agent:review-resolutions`
- Compatibility hash: `lolympus-agent:review-resolved:{reviewId}`
Hash는 기존 read path와 rollback을 위한 compatibility layer다. 실제 sunset은 최소 1-2개월 운영 후 아래 조건을 만족할 때만 진행한다.
## 선결 조건
- Dual-write 운영 기간이 2개월 이상이다.
- Stream과 hash의 resolution 데이터 일치율이 100%다.
- `review:list`, `review:show`, `review:resolve`, `review:stats`가 stream 기반 확인으로 대체 가능한 상태다.
- 최근 2주간 supervisor resolution 관련 incident가 없다.
- Sunset 실행 전 main branch의 전체 회귀가 통과한다.
권장 점검:
```bash
npm run review:stats
npm run review:list -- --include-resolved --limit=20
npm run typecheck
npm test
```
필요하면 별도 consistency script를 만든다.
```text
scripts/verify-dual-write-consistency.ts
```
이 script는 `lolympus-agent:review-resolutions`의 최신 resolution과 `lolympus-agent:review-resolved:{reviewId}` hash를 비교해 divergence를 출력해야 한다.
## Sunset 단계
### Step 1. Hash write 중단
`src/skills/supervisor/review-queue.ts`의 `resolveSupervisorReview()`에서 `setHash()`와 `expireKey()` 호출을 제거한다. Stream append는 유지한다.
검증:
```bash
npm run typecheck
npm test -- src/skills/supervisor/review-queue.test.ts src/operations/cli.test.ts
```
Rollback: 해당 commit revert. 기존 hash TTL 동안 read fallback은 유지된다.
### Step 2. Stream-only write로 1개월 운영
새 resolution은 stream에만 쌓이고, 기존 hash는 TTL까지 남는다. 이 기간 동안 `review:list/show/stats`가 정상 동작하는지 확인한다.
관찰 항목:
- unresolved item이 비정상적으로 늘지 않는가
- resolution history가 누락되지 않는가
- 운영자가 `review:show`에서 이전 resolution을 확인할 수 있는가
Rollback: `setHash()`와 `expireKey()`를 복구한다.
### Step 3. Hash read fallback 제거
`getSupervisorReview()`와 `listSupervisorReview()`에서 hash lookup을 제거하고, resolution stream join으로만 상태를 계산한다.
검증:
```bash
npm run review:list -- --include-resolved --limit=5
npm run review:stats
npm test -- src/skills/supervisor/review-queue.test.ts src/operations/cli.test.ts
```
Rollback: hash lookup 코드를 복구한다. Step 1 이후 새 데이터는 hash에 없으므로 rollback 전에 stream 기반 backfill이 필요한지 판단한다.
### Step 4. 기존 hash 정리
TTL이 짧으면 자연 만료를 기다린다. 운영상 즉시 정리가 필요하면 cleanup script를 별도 PR로 만든다.
주의: cleanup script는 반드시 dry-run을 기본값으로 둔다.
### Step 5. 상수와 문서 정리
`SUPERVISOR_RESOLVED_HASH_PREFIX` 상수와 관련 env/문서 언급을 제거한다. `docs/service-evaluation-operations.md`의 Redis key schema도 갱신한다.
검증:
```bash
rg "review-resolved|SUPERVISOR_RESOLVED_HASH_PREFIX" src docs
npm run lint
npm test
```
## Rollback 원칙
- Step 1-2는 단순 revert로 rollback 가능하다.
- Step 3 이후 rollback은 stream-to-hash backfill 필요 여부를 먼저 판단한다.
- Step 4 cleanup 이후에는 hash 데이터 복구를 기대하지 않는다.
- 모든 단계는 작은 PR로 나누고, 각 PR은 독립 revert 가능해야 한다.
+82 -320
View File
@@ -6,10 +6,10 @@
운영 프로세스는 크게 두 개다. 운영 프로세스는 크게 두 개다.
| 프로세스 | 엔트리포인트 | 역할 | | 프로세스 | 엔트리포인트 | 역할 |
| -------- | --------------------------- | ----------------------------------------------------------------------------- | |---|---|---|
| agent | `src/main.ts` | NATS request/reply를 받아 실시간 온보딩 응답 생성 | | agent | `src/main.ts` | NATS request/reply를 받아 실시간 온보딩 응답 생성 |
| worker | `src/workers/background.ts` | Redis Stream background job 처리, shadow eval, archive, drift, review routing | | worker | `src/workers/background.ts` | Redis Stream background job 처리, shadow eval, archive, drift, review routing |
PM2 기준으로는 `ecosystem.config.cjs`에서 agent와 worker를 함께 띄운다. PM2 기준으로는 `ecosystem.config.cjs`에서 agent와 worker를 함께 띄운다.
@@ -88,17 +88,17 @@ Tool loop는 `responseMode=llm-with-tools`일 때 작동한다. Built-in tool은
Worker는 `lolympus-agent:background` Redis Stream을 consumer group `bg-worker`로 읽는다. Worker는 `lolympus-agent:background` Redis Stream을 consumer group `bg-worker`로 읽는다.
| Job type | 처리 | | Job type | 처리 |
| ------------- | --------------------------------------------- | |---|---|
| `shadow-eval` | 실시간 응답을 rule/LLM/process 기준으로 평가 | | `shadow-eval` | 실시간 응답을 rule/LLM/process 기준으로 평가 |
| `summarize` | 오래된 대화 일부를 요약해 summary slot에 저장 | | `summarize` | 오래된 대화 일부를 요약해 summary slot에 저장 |
Worker는 pending job을 복구하고, delivery count가 한도를 넘으면 DLQ로 이동한다. Worker는 pending job을 복구하고, delivery count가 한도를 넘으면 DLQ로 이동한다.
| Stream | 용도 | | Stream | 용도 |
| ------------------------------- | -------------------- | |---|---|
| `lolympus-agent:background` | background job queue | | `lolympus-agent:background` | background job queue |
| `lolympus-agent:background:dlq` | 반복 실패 job 저장 | | `lolympus-agent:background:dlq` | 반복 실패 job 저장 |
## 6. Shadow Evaluation 흐름 ## 6. Shadow Evaluation 흐름
@@ -123,14 +123,14 @@ lolympus-agent:eval:results
Worker는 일정 간격으로 eval result를 가공한다. Worker는 일정 간격으로 eval result를 가공한다.
| 작업 | 기본 주기 | 결과 | | 작업 | 기본 주기 | 결과 |
| -------------------- | ----------------------------: | ---------------------------- | |---|---:|---|
| aggregation | `EVAL_AGG_TICK_MS` 5분 | slice별 Redis Hash metric | | aggregation | `EVAL_AGG_TICK_MS` 5분 | slice별 Redis Hash metric |
| archive | `EVAL_ARCHIVE_TICK_MS` 1시간 | MongoDB `eval_archive` | | archive | `EVAL_ARCHIVE_TICK_MS` 1시간 | MongoDB `eval_archive` |
| drift detection | `EVAL_DRIFT_TICK_MS` 24시간 | `eval-drift` alert | | drift detection | `EVAL_DRIFT_TICK_MS` 24시간 | `eval-drift` alert |
| promotion | `EVAL_PROMOTE_TICK_MS` 24시간 | golden candidate jsonl | | promotion | `EVAL_PROMOTE_TICK_MS` 24시간 | golden candidate jsonl |
| review routing | `EVAL_REVIEW_TICK_MS` 5분 | human review queue | | review routing | `EVAL_REVIEW_TICK_MS` 5분 | human review queue |
| review label archive | archive tick과 함께 | MongoDB `eval_review_labels` | | review label archive | archive tick과 함께 | MongoDB `eval_review_labels` |
Slice 정의: Slice 정의:
@@ -149,29 +149,29 @@ tier × intent × phase
주요 Redis/Garnet key: 주요 Redis/Garnet key:
| Key | Type | 용도 | | Key | Type | 용도 |
| -------------------------------------------------- | ------ | ------------------------------------------ | |---|---|---|
| `lolympus-agent:background` | Stream | worker job queue | | `lolympus-agent:background` | Stream | worker job queue |
| `lolympus-agent:background:dlq` | Stream | worker dead-letter queue | | `lolympus-agent:background:dlq` | Stream | worker dead-letter queue |
| `lolympus-agent:eval:results` | Stream | shadow eval 결과 | | `lolympus-agent:eval:results` | Stream | shadow eval 결과 |
| `lolympus-agent:eval:baseline` | String | 최근 안정 baseline report | | `lolympus-agent:eval:baseline` | String | 최근 안정 baseline report |
| `lolympus-agent:eval:agg:{tier}:{intent}:{phase}` | Hash | slice별 count/score/latency/process metric | | `lolympus-agent:eval:agg:{tier}:{intent}:{phase}` | Hash | slice별 count/score/latency/process metric |
| `lolympus-agent:eval:agg:cursor` | String | aggregation cursor | | `lolympus-agent:eval:agg:cursor` | String | aggregation cursor |
| `lolympus-agent:eval:archive:cursor` | String | eval archive cursor | | `lolympus-agent:eval:archive:cursor` | String | eval archive cursor |
| `lolympus-agent:eval:review:queue` | Stream | human review 대기 큐 | | `lolympus-agent:eval:review:queue` | Stream | human review 대기 큐 |
| `lolympus-agent:eval:review:labeled` | Stream | human review label | | `lolympus-agent:eval:review:labeled` | Stream | human review label |
| `lolympus-agent:eval:review:cursor` | String | review routing cursor | | `lolympus-agent:eval:review:cursor` | String | review routing cursor |
| `lolympus-agent:eval:review:dedupe:{evalStreamId}` | String | 동일 eval result 중복 라우팅 방지 | | `lolympus-agent:eval:review:dedupe:{evalStreamId}` | String | 동일 eval result 중복 라우팅 방지 |
| `lolympus-agent:eval:review:label-archive:cursor` | String | review label archive cursor | | `lolympus-agent:eval:review:label-archive:cursor` | String | review label archive cursor |
| `lolympus-agent:alert:dedupe:{kind}` | String | alert rate limit | | `lolympus-agent:alert:dedupe:{kind}` | String | alert rate limit |
## 9. MongoDB Collections ## 9. MongoDB Collections
| Collection | 생성/사용 모듈 | 용도 | | Collection | 생성/사용 모듈 | 용도 |
| -------------------- | ---------------------------------- | ------------------------------ | |---|---|---|
| `eval_archive` | `src/eval/archive.ts` | shadow eval 결과 장기 보관 | | `eval_archive` | `src/eval/archive.ts` | shadow eval 결과 장기 보관 |
| `eval_review_labels` | `src/eval/review/label-archive.ts` | 사람 라벨 장기 보관 | | `eval_review_labels` | `src/eval/review/label-archive.ts` | 사람 라벨 장기 보관 |
| `eval_tournaments` | `src/eval/loop/tournament.ts` | prompt variant tournament 결과 | | `eval_tournaments` | `src/eval/loop/tournament.ts` | prompt variant tournament 결과 |
`eval_archive`는 `evaluatedAt` 기준 TTL index를 가진다. 기본 보관 기간은 `EVAL_ARCHIVE_TTL_DAYS=30`이다. `eval_archive`는 `evaluatedAt` 기준 TTL index를 가진다. 기본 보관 기간은 `EVAL_ARCHIVE_TTL_DAYS=30`이다.
@@ -218,57 +218,6 @@ npm run eval:review -- --limit=20 --export-labeled=eval-reports/review-labels.js
라벨 결과는 `lolympus-agent:eval:review:labeled` Stream에 쌓이고 archive tick에서 `eval_review_labels`로 영속화된다. 라벨 결과는 `lolympus-agent:eval:review:labeled` Stream에 쌓이고 archive tick에서 `eval_review_labels`로 영속화된다.
### 10.1 Supervisor Review Queue
Supervisor review queue는 skill handoff가 budget, confidence, policy, validation guardrail에 걸렸을 때 운영자가 확인하는 큐다.
큐:
```text
lolympus-agent:review-queue
```
Resolution audit:
```text
lolympus-agent:review-resolutions
```
기본 workflow:
1. Slack 또는 NATS에서 `supervisor-handoff` alert를 확인한다.
2. 최근 queue를 본다.
3. 개별 review를 열어 handoff packet, source chain, reason을 확인한다.
4. 운영자 답변, reroute, dismiss 중 하나로 resolution을 기록한다.
5. `review:stats`로 pending이 줄었는지 확인한다.
```bash
npm run review:list -- --limit=10
npm run review:show -- <reviewId> --history
npm run review:resolve -- <reviewId> --action=manual-reply --note="운영자 확인 완료"
npm run review:stats
```
특정 reason만 볼 때:
```bash
npm run review:list -- --reason=low-confidence --limit=10
```
Resolution history는 stream에 쌓이며, compatibility hash는 sunset 전까지 read fallback으로 유지된다. Sunset 조건과 절차는 [resolution-hash-sunset-procedure.md](resolution-hash-sunset-procedure.md)를 따른다.
### 10.2 Tool Audit
Tool call은 성공과 실패 모두 `lolympus-agent:tool-calls` Stream에 남긴다. 원본 args는 저장하지 않고, 민감 키를 제거한 뒤 hash만 기록한다.
```bash
npm run tool-audit:list -- --limit=20
npm run tool-audit:list -- --skill=onboarding --since=24h
npm run tool-audit:show -- <callId>
```
Stream은 `AGENT_TOOL_AUDIT_MAX_LEN` 기준으로 trim된다. 기본값은 `100000`이다.
## 11. Calibration 흐름 ## 11. Calibration 흐름
Calibration은 사람 점수와 LLM judge 점수의 상관을 계산한다. Calibration은 사람 점수와 LLM judge 점수의 상관을 계산한다.
@@ -315,7 +264,7 @@ npm run eval:promote
기본 출력: 기본 출력:
```text ```text
src/eval/golden/onboarding/candidate.jsonl src/eval/golden/onboarding.candidate.jsonl
``` ```
Promotion 우선순위: Promotion 우선순위:
@@ -325,7 +274,7 @@ Promotion 우선순위:
3. rule/LLM disagree 3. rule/LLM disagree
4. rare-slice 4. rare-slice
candidate는 바로 golden으로 승격되는 것이 아니라 사람이 직접 검토해 `onboarding/cases.jsonl`로 옮긴다. candidate는 바로 golden으로 승격되는 것이 아니라 사람이 직접 검토해 `onboarding.jsonl`로 옮긴다.
## 13. Closed Loop 흐름 ## 13. Closed Loop 흐름
@@ -396,243 +345,56 @@ Variant 적용은 자동 머지가 아니다. 사람이 tournament 결과와 sam
## 14. Eval CLI 요약 ## 14. Eval CLI 요약
| 명령 | 용도 | | 명령 | 용도 |
| ----------------------------------- | ---------------------------------------------- | |---|---|
| `npm run eval` | golden set 실행 | | `npm run eval` | golden set 실행 |
| `npm run eval:capture` | MongoDB events에서 golden scaffold 추출 | | `npm run eval:capture` | MongoDB events에서 golden scaffold 추출 |
| `npm run eval:baseline` | 안정 baseline 저장 | | `npm run eval:baseline` | 안정 baseline 저장 |
| `npm run eval:report` | archive slice report 출력 | | `npm run eval:report` | archive slice report 출력 |
| `npm run eval:promote` | golden candidate 추출 | | `npm run eval:promote` | golden candidate 추출 |
| `npm run eval:review` | human review CLI | | `npm run eval:review` | human review CLI |
| `npm run eval:calibrate` | 사람/LLM judge calibration | | `npm run eval:calibrate` | 사람/LLM judge calibration |
| `npm run eval:propose` | prompt/tool improvement proposal 생성 | | `npm run eval:propose` | prompt/tool improvement proposal 생성 |
| `npm run eval:tournament` | prompt variant A/B tournament 실행 | | `npm run eval:tournament` | prompt variant A/B tournament 실행 |
| `npm run eval:tournament-results` | tournament 결과 조회 | | `npm run eval:tournament-results` | tournament 결과 조회 |
| `npm run eval:train-router` | archive 기반 learned router weights 생성 |
| `npm run eval:router-shadow-report` | static router와 learned router divergence 보고 |
### 14.1 자주 쓰는 명령 Cheat Sheet
평가:
```bash
npm run eval -- --set=onboarding/cases --skip-llm-judge
npm run eval:moderation -- --skip-llm-judge
npm run eval:handoff -- --skip-llm-judge
npm run eval:report -- --worst=20 --since=7d
npm run eval:propose -- --slice=tier=1,intent=status-question,phase=deadline-near
npm run eval:tournament -- --variants=control,<variantId> --max-matches=50
```
학습과 shadow 비교:
```bash
npm run eval:train-router -- --output=.local/router-weights.json --force
npm run eval:router-shadow-report -- --since=7d --weights=.local/router-weights.json
```
운영:
```bash
npm run review:list -- --limit=10
npm run review:show -- <reviewId> --history
npm run review:resolve -- <reviewId> --action=dismiss --note="중복"
npm run review:stats
```
검증:
```bash
npm run typecheck
npm run lint
npm run lint:skills
npm test
```
## 15. 주요 Env ## 15. 주요 Env
| Env | 기본값 | 용도 | | Env | 기본값 | 용도 |
| --------------------------------------- | ----------------: | ---------------------------------------- | |---|---:|---|
| `EVAL_STREAM_MAXLEN` | 10000 | eval results Stream maxlen | | `EVAL_STREAM_MAXLEN` | 10000 | eval results Stream maxlen |
| `EVAL_AGG_TICK_MS` | 300000 | aggregation tick | | `EVAL_AGG_TICK_MS` | 300000 | aggregation tick |
| `EVAL_AGG_TTL_SEC` | 604800 | slice metric TTL | | `EVAL_AGG_TTL_SEC` | 604800 | slice metric TTL |
| `EVAL_ARCHIVE_TICK_MS` | 3600000 | archive tick | | `EVAL_ARCHIVE_TICK_MS` | 3600000 | archive tick |
| `EVAL_ARCHIVE_TTL_DAYS` | 30 | archive TTL days | | `EVAL_ARCHIVE_TTL_DAYS` | 30 | archive TTL days |
| `EVAL_DRIFT_TICK_MS` | 86400000 | drift tick | | `EVAL_DRIFT_TICK_MS` | 86400000 | drift tick |
| `EVAL_PROMOTE_TICK_MS` | 86400000 | promotion tick | | `EVAL_PROMOTE_TICK_MS` | 86400000 | promotion tick |
| `EVAL_REVIEW_TICK_MS` | 300000 | review routing tick | | `EVAL_REVIEW_TICK_MS` | 300000 | review routing tick |
| `EVAL_REVIEW_QUEUE_MAXLEN` | 500 | review queue maxlen | | `EVAL_REVIEW_QUEUE_MAXLEN` | 500 | review queue maxlen |
| `EVAL_REVIEW_LABEL_MAXLEN` | 5000 | review label Stream maxlen | | `EVAL_REVIEW_LABEL_MAXLEN` | 5000 | review label Stream maxlen |
| `EVAL_REVIEW_DEDUPE_TTL_SEC` | 604800 | review dedupe TTL | | `EVAL_REVIEW_DEDUPE_TTL_SEC` | 604800 | review dedupe TTL |
| `EVAL_CALIBRATION_MIN_CORR` | 0.5 | calibration gate | | `EVAL_CALIBRATION_MIN_CORR` | 0.5 | calibration gate |
| `REVIEWER_ID` | git user.email | human review 식별자 | | `REVIEWER_ID` | git user.email | human review 식별자 |
| `LLM_PROPOSE_MODEL` | `LLM_JUDGE_MODEL` | proposal model | | `LLM_PROPOSE_MODEL` | `LLM_JUDGE_MODEL` | proposal model |
| `LLM_PROPOSE_MAX_TOKENS` | 1024 | proposal max tokens | | `LLM_PROPOSE_MAX_TOKENS` | 1024 | proposal max tokens |
| `EVAL_TOURNAMENT_MAX_MATCHES` | 50 | tournament match cap | | `EVAL_TOURNAMENT_MAX_MATCHES` | 50 | tournament match cap |
| `EVAL_TOURNAMENT_MIN_CASES` | 10 | proposal 최소 archive cases | | `EVAL_TOURNAMENT_MIN_CASES` | 10 | proposal 최소 archive cases |
| `PROMPT_VARIANT_ID` | unset | 운영 prompt variant 선택 | | `PROMPT_VARIANT_ID` | unset | 운영 prompt variant 선택 |
| `AGENT_ROUTER_WEIGHTS_PATH` | unset | learned router weights path |
| `AGENT_BUDGET_LLM_COST_ENABLED` | true | LLM token cost를 request budget에 누적 |
| `AGENT_COST_AWARE_TIER_ENABLED` | true | confidence/cost-aware tier selection |
| `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | supervisor regex fast-path 사용 |
| `SUPERVISOR_FAST_PATH_CONFIDENCE` | 0.85 | fast-path 최소 규칙 confidence |
| `AGENT_TOOL_SANDBOX_ENABLED` | false | tool context 최소 권한 강제 여부 |
| `AGENT_TOOL_AUDIT_MAX_LEN` | 100000 | tool audit Stream maxlen |
| `AGENT_NATS_ACL_MODE` | log-only | NATS subject ACL 모드 |
| `AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED` | true | supervisor LLM userMessage 사용 여부 |
| `PHOENIX_OTLP_ENDPOINT` | unset | Phoenix OTLP trace endpoint |
| `PHOENIX_LOG_PROMPTS` | false | LLM prompt/completion content trace 기록 |
| `METRICS_PORT` | 9464 | Prometheus `/metrics` HTTP endpoint port |
| `SLACK_ALERT_WEBHOOK_URL` | unset | alert Slack incoming webhook URL |
| `SLACK_WEBHOOK_TIMEOUT_MS` | 3000 | Slack webhook POST timeout |
| `ALERT_THRESHOLD_P95_LATENCY_MS` | 3000 | 글로벌 p95 latency alert 기준 |
| `ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT` | 10 | 글로벌 eval score drop alert 기준 |
| `ALERT_THRESHOLD_LLM_ERROR_RATE` | 0.1 | 글로벌 LLM judge error rate alert 기준 |
| `ALERT_THRESHOLD_DRIFT_PCT` | 15 | 글로벌 drift alert 기준 |
## 16. Alert와 Monitoring ## 16. Alert와 Monitoring
Worker는 다음 종류의 alert를 보낼 수 있다. Worker는 다음 종류의 alert를 보낼 수 있다.
| Alert kind | 의미 | | Alert kind | 의미 |
| -------------------- | --------------------------------- | |---|---|
| `latency` | 최근 shadow eval latency p95 초과 | | `latency` | 최근 shadow eval latency p95 초과 |
| `llm-error-rate` | LLM judge 실패율 상승 | | `llm-error-rate` | LLM judge 실패율 상승 |
| `eval-score-drop` | baseline 대비 eval score 하락 | | `eval-score-drop` | baseline 대비 eval score 하락 |
| `eval-drift` | 24h vs 7d slice score drift | | `eval-drift` | 24h vs 7d slice score drift |
| `endpoint-down` | LLM circuit breaker open | | `endpoint-down` | LLM circuit breaker open |
| `archive-stalled` | MongoDB archive 연속 실패 | | `archive-stalled` | MongoDB archive 연속 실패 |
| `supervisor-handoff` | supervisor review queue 진입 |
Alert는 `lolympus-agent.alert` NATS subject로 발행되고, dedupe key로 rate limit된다. Alert는 `lolympus-agent.alert` NATS subject로 발행되고, dedupe key로 rate limit된다.
`SLACK_ALERT_WEBHOOK_URL`이 설정되어 있으면 같은 alert가 Slack incoming webhook으로도 전송된다.
Slack 전송은 NATS publish 이후 비동기로 실행되므로 Slack 장애가 agent 응답 경로를 막지 않는다.
Alert 처리 workflow:
1. Slack 메시지의 `kind`, `skill`, `metric`, `value`, `threshold`, `window`를 확인한다.
2. `supervisor-handoff`면 `review:list`와 `review:show`로 packet을 확인한다.
3. `latency`면 `/metrics`의 p95 panel과 최근 deploy를 확인한다.
4. `llm-error-rate`나 `endpoint-down`이면 LLM endpoint env와 circuit breaker 로그를 확인한다.
5. `eval-score-drop` 또는 `eval-drift`면 `eval:report`, archive sample, golden smoke를 확인한다.
6. 조치 후 dedupe window가 끝나도 반복되는지 본다.
### 16.1 Skill별 threshold
Alert threshold는 글로벌 env를 기본으로 쓰고, skill별 env가 있으면 우선 적용한다.
skill 이름은 대문자 env suffix로 정규화된다.
```env
ALERT_THRESHOLD_P95_LATENCY_MS=3000
ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING=5000
ALERT_THRESHOLD_P95_LATENCY_MS_MODERATION=2000
ALERT_THRESHOLD_LLM_ERROR_RATE=0.1
ALERT_THRESHOLD_LLM_ERROR_RATE_ONBOARDING=0.2
ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT=10
ALERT_THRESHOLD_DRIFT_PCT=15
```
같은 alert kind라도 `skill`이 다르면 dedupe key가 분리된다.
예를 들어 onboarding latency alert와 moderation latency alert는 서로 throttle하지 않는다.
skill이 없는 cross-system alert는 기존처럼 글로벌 dedupe를 사용한다.
### 16.2 Secret, Tool Sandbox, NATS ACL
Secret은 Infisical을 우선으로 로드한다. Infisical credential은 로컬 env 또는 IMDS에서 읽고, 새 secret env를 추가할 때는 `src/common/known-secret-keys.ts`에 함께 등록해야 로그 redaction 대상에 들어간다. LLM, embedding, Phoenix API key도 redaction 대상이다.
Tool은 `requires` 선언으로 필요한 context만 받는다. `AGENT_TOOL_SANDBOX_ENABLED=false`가 기본이라 운영 초기는 호환성을 유지한다. 1주 정도 audit을 본 뒤 false positive가 없으면 true로 전환한다.
NATS subject ACL은 `src/common/nats-skill-acl.ts`의 `DEFAULT_NATS_ACL`에서 관리한다. 기본 `AGENT_NATS_ACL_MODE=log-only`는 위반을 warn으로만 남기고 publish/request는 그대로 진행한다. 1개월 운영 후 false positive가 없으면 `enforce`로 전환한다.
### 16.3 Phoenix trace
LLM 호출, skill 실행, tool 실행 span은 OpenTelemetry trace로 남긴다.
`PHOENIX_OTLP_ENDPOINT`가 없으면 기존 설정을 따른다. 개발 환경 기본값은 trace exporter 비활성화이고, `OTEL_EXPORTER=stdout`이면 콘솔 exporter를 쓴다.
Phoenix를 로컬에서 볼 때:
```bash
docker run --rm -p 6006:6006 arizephoenix/phoenix:latest
```
Agent/worker env:
```env
PHOENIX_OTLP_ENDPOINT=http://localhost:6006/v1/traces
OTEL_SAMPLING_RATIO=1
PHOENIX_LOG_PROMPTS=false
```
`PHOENIX_OTLP_ENDPOINT`가 설정되면 `OTEL_EXPORTER=none`이어도 Phoenix OTLP exporter를 우선 사용한다.
Phoenix UI에서는 `service.name=lolympus-agent`, `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.cost_usd`, `llm.token_count.prompt`, `llm.token_count.completion` 속성으로 LLM span을 필터링한다.
Prompt와 completion 본문은 기본적으로 기록하지 않는다.
디버깅 때문에 필요할 때만 `PHOENIX_LOG_PROMPTS=true`를 켠다. 이 경우 email, 전화번호, token/API key 형태의 값은 span 기록 전에 masking된다.
### 16.4 Prometheus scrape
Agent process는 `METRICS_PORT`에서 HTTP endpoint를 연다.
| Endpoint | 응답 |
| ---------- | -------------------------- |
| `/healthz` | `ok` |
| `/metrics` | Prometheus text exposition |
Prometheus scrape 예시:
```yaml
scrape_configs:
- job_name: lolympus-agent
static_configs:
- targets:
- 127.0.0.1:9464
```
권장 dashboard panel:
| Metric | 권장 panel |
| ------------------------------------ | ----------------------------------- |
| `agent_handoff_total` | skill/from/to/reason별 handoff rate |
| `agent_handoff_loop_total` | skill별 loop 발생 수 |
| `agent_human_override_total` | supervisor/manual review 유입 |
| `agent_missed_escalation_total` | skill별 missed escalation |
| `agent_routing_accuracy_total` | eval routing accuracy |
| `agent_unnecessary_escalation_total` | skill별 unnecessary escalation |
| `agent_validator_fail_total` | skill/stage별 validator fail |
| `agent_handoff_count` | skill별 request당 handoff histogram |
| `agent_time_to_resolution_ms` | skill별 처리 시간 histogram |
| `agent_token_cost_usd` | skill/model별 token cost histogram |
PromQL 예시:
```promql
sum by (skill) (rate(agent_handoff_total[5m]))
histogram_quantile(0.95, sum by (skill, le) (rate(agent_time_to_resolution_ms_bucket[5m])))
sum by (skill, stage) (rate(agent_validator_fail_total[5m]))
```
운영 SLI 권장 시작점:
| SLI | 권장 시작값 |
| -------------------------- | -------------------: |
| request p95 latency | 3초 이하 |
| supervisor escalation rate | skill별 일 10건 이하 |
| LLM judge error rate | 10% 이하 |
| eval drift | 15% 이하 |
| archive stalled | 3회 연속 실패 0건 |
### 16.5 Slack webhook 설정
Slack incoming webhook URL을 env로 설정한다.
```env
SLACK_ALERT_WEBHOOK_URL=https://hooks.slack.com/services/...
SLACK_WEBHOOK_TIMEOUT_MS=3000
```
Webhook URL은 secret으로 취급한다. 로그 redaction 대상에 포함되어 있으며, Slack payload에는 metric, value, threshold, window, skill, suggested action만 포함하고 사용자 메시지 본문은 포함하지 않는다.
## 17. 장애 확인 Runbook ## 17. 장애 확인 Runbook
File diff suppressed because it is too large. Load diff
+11 -27
View File
@@ -16,35 +16,19 @@ if ($isMacOS -or $isLinux) {
$env:ENV_FILE = "$PSScriptRoot\env-vars\$env:NODE_ENV.env" $env:ENV_FILE = "$PSScriptRoot\env-vars\$env:NODE_ENV.env"
} }
$GLOBAL_ENV_FILE = Join-Path $HOME "source/global-env-vars.txt" # 환경 변수 파일 확인
if (-Not (Test-Path $env:ENV_FILE)) {
function Import-EnvFileIfPresent { Write-Error "The file '$env:ENV_FILE' does not exist."
param( exit 1
[string]$Path,
[switch]$Required
)
if (-Not (Test-Path $Path)) {
if ($Required) {
Write-Error "The file '$Path' does not exist."
exit 1
}
return
}
Get-Content $Path | ForEach-Object {
if ($_ -match '^\s*([^#]+?)\s*=\s*(.*)$') {
$key, $value = $matches[1].Trim(), $matches[2].Trim().Trim("`"")
if (-not [System.Environment]::GetEnvironmentVariable($key)) {
[System.Environment]::SetEnvironmentVariable($key, $value)
}
}
}
} }
# 환경 변수 파일 확인 # 'dahyeon-local.env' 파일에서 환경 변수 설정
Import-EnvFileIfPresent -Path $GLOBAL_ENV_FILE Get-Content $env:ENV_FILE | ForEach-Object {
Import-EnvFileIfPresent -Path $env:ENV_FILE -Required if ($_ -match '^\s*([^#]+?)\s*=\s*(.*)$') {
$key, $value = $matches[1].Trim(), $matches[2].Trim().Trim("`"")
[System.Environment]::SetEnvironmentVariable($key, $value)
}
}
Get-ChildItem Env: | Where-Object { Get-ChildItem Env: | Where-Object {
$_.Name -eq "NODE_ENV" -or $_.Name -eq "NODE_ENV" -or
-1
View File
@@ -12,7 +12,6 @@
"@infisical/sdk": "^5.0.0", "@infisical/sdk": "^5.0.0",
"@opentelemetry/api": "^1.9.0", "@opentelemetry/api": "^1.9.0",
"@opentelemetry/auto-instrumentations-node": "^0.66.0", "@opentelemetry/auto-instrumentations-node": "^0.66.0",
"@opentelemetry/exporter-trace-otlp-http": "^0.207.0",
"@opentelemetry/sdk-node": "^0.207.0", "@opentelemetry/sdk-node": "^0.207.0",
"@opentelemetry/sdk-trace-base": "^2.1.0", "@opentelemetry/sdk-trace-base": "^2.1.0",
"@opentelemetry/semantic-conventions": "^1.37.0", "@opentelemetry/semantic-conventions": "^1.37.0",
+2 -17
View File
@@ -9,13 +9,10 @@
"test": "vitest run", "test": "vitest run",
"test:watch": "vitest", "test:watch": "vitest",
"test:coverage": "vitest run --coverage", "test:coverage": "vitest run --coverage",
"lint": "eslint . && npm run lint:skills", "lint": "eslint .",
"lint:skills": "node --import tsx src/skills/validate-manifests.ts",
"lint:fix": "eslint . --fix", "lint:fix": "eslint . --fix",
"format": "prettier --write .", "format": "prettier --write .",
"eval": "node --import tsx src/eval/cli.ts run --set=onboarding/cases", "eval": "node --import tsx src/eval/cli.ts run --set=golden",
"eval:moderation": "node --import tsx src/eval/cli.ts run --set=moderation/cases",
"eval:handoff": "node --import tsx src/eval/cli.ts run --set=handoff/cases",
"eval:capture": "node --import tsx src/eval/cli.ts capture", "eval:capture": "node --import tsx src/eval/cli.ts capture",
"eval:baseline": "node --import tsx src/eval/cli.ts baseline", "eval:baseline": "node --import tsx src/eval/cli.ts baseline",
"eval:report": "node --import tsx src/eval/cli.ts report", "eval:report": "node --import tsx src/eval/cli.ts report",
@@ -24,20 +21,9 @@
"eval:calibrate": "node --import tsx src/eval/cli.ts calibrate", "eval:calibrate": "node --import tsx src/eval/cli.ts calibrate",
"eval:train-prm": "node --import tsx src/eval/cli.ts train-prm", "eval:train-prm": "node --import tsx src/eval/cli.ts train-prm",
"eval:train-router": "node --import tsx src/eval/cli.ts train-router", "eval:train-router": "node --import tsx src/eval/cli.ts train-router",
"eval:router-shadow-report": "node --import tsx src/eval/cli.ts router-shadow-report",
"eval:propose": "node --import tsx src/eval/cli.ts propose", "eval:propose": "node --import tsx src/eval/cli.ts propose",
"eval:propose:moderation": "node --import tsx src/eval/cli.ts propose --skill=moderation",
"eval:tournament": "node --import tsx src/eval/cli.ts tournament", "eval:tournament": "node --import tsx src/eval/cli.ts tournament",
"eval:tournament:moderation": "node --import tsx src/eval/cli.ts tournament --skill=moderation",
"eval:tournament-results": "node --import tsx src/eval/cli.ts tournament-results", "eval:tournament-results": "node --import tsx src/eval/cli.ts tournament-results",
"ingest:knowledge": "node --import tsx scripts/ingest-knowledge.ts",
"memory:evict": "node --import tsx scripts/memory-eviction-cron.ts",
"review:list": "node --import tsx src/operations/cli.ts review:list",
"review:show": "node --import tsx src/operations/cli.ts review:show",
"review:resolve": "node --import tsx src/operations/cli.ts review:resolve",
"review:stats": "node --import tsx src/operations/cli.ts review:stats",
"tool-audit:list": "node --import tsx src/operations/cli.ts tool-audit:list",
"tool-audit:show": "node --import tsx src/operations/cli.ts tool-audit:show",
"start:dev": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'dev'); \"./launch-server.ps1\"\"", "start:dev": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'dev'); \"./launch-server.ps1\"\"",
"start:test": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'test'); \"./launch-server.ps1\"\"", "start:test": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'test'); \"./launch-server.ps1\"\"",
"start:prod": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'prod'); \"./launch-server.ps1\"\"", "start:prod": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'prod'); \"./launch-server.ps1\"\"",
@@ -67,7 +53,6 @@
"@infisical/sdk": "^5.0.0", "@infisical/sdk": "^5.0.0",
"@opentelemetry/api": "^1.9.0", "@opentelemetry/api": "^1.9.0",
"@opentelemetry/auto-instrumentations-node": "^0.66.0", "@opentelemetry/auto-instrumentations-node": "^0.66.0",
"@opentelemetry/exporter-trace-otlp-http": "^0.207.0",
"@opentelemetry/sdk-node": "^0.207.0", "@opentelemetry/sdk-node": "^0.207.0",
"@opentelemetry/sdk-trace-base": "^2.1.0", "@opentelemetry/sdk-trace-base": "^2.1.0",
"@opentelemetry/semantic-conventions": "^1.37.0", "@opentelemetry/semantic-conventions": "^1.37.0",
-67
View File
@@ -1,67 +0,0 @@
import { loadEnvironmentFromNodeEnv } from "../src/common/env-loader.js";
loadEnvironmentFromNodeEnv();
const { initLogger, getLogger } = await import("../src/observability/logger.js");
initLogger();
const logger = getLogger();
const { GarnetClient } = await import("../src/common/garnet.js");
const { getRawSecret } = await import("../src/common/secret-manager.js");
const { initMongoDB, closeMongoDB } = await import("../src/common/mongodb.js");
const { createMemoryService } = await import("../src/memory/memory-service.js");
const { ingestMarkdownResources } = await import("../src/memory/knowledge-ingest.js");
function firstNonEmpty(
...values: Array<string | null | undefined>
): string | undefined {
for (const value of values) {
if (value?.trim()) {
return value.trim();
}
}
return undefined;
}
const dryRun = process.argv.includes("--dry-run");
const mongoConnectionString = firstNonEmpty(
getRawSecret("MONGODB_CONNECTION_STRING"),
process.env["MONGODB_CONNECTION_STRING"],
);
if (!mongoConnectionString) {
throw new Error("MONGODB_CONNECTION_STRING is required for knowledge ingest");
}
const garnetClient = new GarnetClient();
await garnetClient.connect();
try {
const mongoConnection = await initMongoDB(mongoConnectionString);
const memoryService = createMemoryService({
garnetClient,
mongooseConnection: mongoConnection,
});
const [onboarding, moderation] = await Promise.all([
ingestMarkdownResources(memoryService, {
baseDir: "src/skills/onboarding/resources",
category: "onboarding-step",
dryRun,
skill: "onboarding",
}),
ingestMarkdownResources(memoryService, {
baseDir: "src/skills/moderation/resources",
category: "rule",
dryRun,
skill: "moderation",
}),
]);
logger.info(
{ dryRun, moderation, onboarding },
"✓ Knowledge ingest completed",
);
} finally {
await closeMongoDB();
await garnetClient.close();
}
-153
View File
@@ -1,153 +0,0 @@
import { loadEnvironmentFromNodeEnv } from "../src/common/env-loader.js";
loadEnvironmentFromNodeEnv();
const { getRawSecret } = await import("../src/common/secret-manager.js");
const { initMongoDB, closeMongoDB } = await import("../src/common/mongodb.js");
const { initLogger, getLogger } = await import("../src/observability/logger.js");
const { getMemoryEvictionEnabled } = await import("../src/config.js");
const { KNOWLEDGE_COLLECTION } = await import("../src/memory/mongo-long-term-memory.js");
const { detectContradictions } = await import("../src/memory/contradiction-detector.js");
const { evaluateDocument } = await import("../src/memory/forgetting-policy.js");
initLogger();
const logger = getLogger();
interface KnowledgeRecord {
_id: string;
category: string;
text: string;
metadata?: Record<string, unknown> | undefined;
lastReferencedAt?: Date | undefined;
references?: Date[] | undefined;
updatedAt?: Date | undefined;
}
function firstNonEmpty(
...values: Array<string | null | undefined>
): string | undefined {
for (const value of values) {
if (value?.trim()) {
return value.trim();
}
}
return undefined;
}
function parseSinceDays(): number {
const raw = process.argv.find((arg) => arg.startsWith("--since="))?.split("=")[1];
if (!raw) {
return 30;
}
const match = raw.match(/^(\d+)d$/u);
return match ? Number(match[1]) : 30;
}
const dryRun = process.argv.includes("--dry-run");
const target = process.argv.find((arg) => arg.startsWith("--target="))?.split("=")[1] ?? "all";
const sinceDays = parseSinceDays();
if (!dryRun && !getMemoryEvictionEnabled()) {
throw new Error(
"AGENT_MEMORY_EVICTION_ENABLED must be true for non-dry-run eviction",
);
}
if (target !== "knowledge" && target !== "all") {
logger.info({ dryRun, target }, "memory eviction skipped: target has no Mongo knowledge work");
process.exit(0);
}
const connectionString = firstNonEmpty(
getRawSecret("MONGODB_CONNECTION_STRING"),
process.env["MONGODB_CONNECTION_STRING"],
);
if (!connectionString) {
if (dryRun) {
logger.warn(
{ dryRun },
"MONGODB_CONNECTION_STRING is not configured; memory eviction dry-run skipped",
);
process.exit(0);
}
throw new Error("MONGODB_CONNECTION_STRING is required for memory eviction");
}
const connection = await initMongoDB(connectionString);
try {
const collection = connection.collection<KnowledgeRecord>(KNOWLEDGE_COLLECTION);
const docs = await collection
.find({ "metadata.archived": { $ne: true } })
.sort({ updatedAt: 1 })
.toArray();
const contradictions = detectContradictions(
docs.map((doc) => ({
id: doc._id,
metadata: doc.metadata,
text: doc.text,
})),
);
const contradictionByExistingId = new Map(
contradictions.map((candidate) => [candidate.existingId, candidate.newId]),
);
const now = new Date();
const sinceMs = now.getTime() - sinceDays * 24 * 60 * 60 * 1000;
let archived = 0;
let kept = 0;
for (const doc of docs) {
const references = (doc.references ?? []).filter((date) => {
const time = new Date(date).getTime();
return Number.isFinite(time) && time >= sinceMs;
});
const decision = evaluateDocument(
{
id: doc._id,
category: doc.category,
metadata: doc.metadata,
text: doc.text,
},
{
contradictsId: contradictionByExistingId.get(doc._id),
lastReferencedAt: doc.lastReferencedAt ? new Date(doc.lastReferencedAt) : undefined,
now,
referenceCount30d: references.length,
updatedAt: doc.updatedAt ? new Date(doc.updatedAt) : undefined,
},
);
if (decision.action === "keep") {
kept += 1;
continue;
}
archived += 1;
if (!dryRun) {
await collection.updateOne(
{ _id: doc._id },
{
$set: {
"metadata.archived": true,
"metadata.archivedAt": now.toISOString(),
"metadata.archiveReason": decision.reason,
"metadata.archiveEvidence": decision.evidence,
},
},
);
}
}
logger.info(
{
archived,
contradictions: contradictions.length,
dryRun,
kept,
scanned: docs.length,
sinceDays,
},
"memory eviction completed",
);
} finally {
await closeMongoDB();
}
-317
View File
@@ -1,317 +0,0 @@
import type { AgentRequest, AgentResponse, ConversationMessage } from "./types.js";
import { describe, expect, it } from "vitest";
import type { Skill } from "@skills/skill";
import type { ConversationStore } from "@conv/conversation-store";
import type { NatsClient } from "@common/nats";
import type { TransitionMatrix } from "@skills/transition-matrix";
import { Agent, selectSkillForRequest } from "./agent.js";
function makeRequest(skill = "shared"): AgentRequest {
return {
chatroomId: "room-1",
nickname: "tester",
requestId: "request-1",
skill,
trigger: "user-message",
userId: "user-1",
};
}
function makeSkill(name: string, priority: number, canHandle = true): Skill {
return {
name,
priority,
canHandle: () => canHandle,
handle: (
request: AgentRequest,
_conversationHistory: ConversationMessage[],
): Promise<AgentResponse> =>
Promise.resolve({
confidence: 1,
requestId: request.requestId,
source: "template",
text: name,
}),
};
}
class MemoryConversationStore {
readonly messages: ConversationMessage[] = [];
getHistory(): Promise<ConversationMessage[]> {
return Promise.resolve([...this.messages]);
}
appendMessage(
_chatroomId: string,
_userId: string,
message: ConversationMessage,
): Promise<void> {
this.messages.push(message);
return Promise.resolve();
}
}
class FakeNatsClient {
readonly responses: AgentResponse[] = [];
respond(_msg: unknown, response: AgentResponse): Promise<void> {
this.responses.push(response);
return Promise.resolve();
}
}
async function handleThroughAgent(
skills: readonly Skill[],
request: AgentRequest,
transitionMatrix?: TransitionMatrix,
): Promise<{
nats: FakeNatsClient;
store: MemoryConversationStore;
}> {
const nats = new FakeNatsClient();
const store = new MemoryConversationStore();
const agent = new Agent(
nats as unknown as NatsClient,
store as unknown as ConversationStore,
transitionMatrix,
);
for (const skill of skills) {
agent.registerSkill(skill);
}
await (agent as unknown as {
handleRequest(msg: { data: Uint8Array }): Promise<void>;
}).handleRequest({
data: new TextEncoder().encode(JSON.stringify(request)),
});
return { nats, store };
}
describe("selectSkillForRequest", () => {
it("selects the highest-priority matching skill", () => {
const request = makeRequest();
const selected = selectSkillForRequest(
[
makeSkill("low", 10),
makeSkill("high", 90),
makeSkill("disabled", 100, false),
],
request,
);
expect(selected?.name).toBe("high");
});
it("returns null when no skill can handle the request", () => {
expect(selectSkillForRequest([makeSkill("miss", 50, false)], makeRequest())).toBeNull();
});
});
describe("Agent handoff dispatch", () => {
it("responds once with the final handoff target response", async () => {
const source: Skill = {
...makeSkill("source", 50),
handle: (request) =>
Promise.resolve({
confidence: 0.8,
metadata: {
handoffRequest: {
confidence: 0.8,
evidenceRefs: ["rule-1"],
expectedOutput: "text",
reason: "out-of-scope",
risk: "medium",
targetAgent: "target",
taskSpec: "continue",
},
},
requestId: request.requestId,
source: "template",
text: "handoff",
}),
};
const target: Skill = {
...makeSkill("target", 60),
canHandle: (request) => request.skill === "target",
handle: (request) =>
Promise.resolve({
confidence: 1,
requestId: request.requestId,
source: "template",
text: "final answer",
}),
};
const { nats, store } = await handleThroughAgent(
[source, target],
makeRequest("source"),
);
expect(nats.responses).toHaveLength(1);
expect(nats.responses[0]?.text).toBe("final answer");
expect(nats.responses[0]?.metadata).toMatchObject({
handoffChain: ["source", "target"],
handoffDepth: 1,
});
expect(store.messages.map((message) => message.content)).toEqual([
"final answer",
]);
});
it("routes handoff control tokens through metadata", async () => {
const source: Skill = {
...makeSkill("source", 50),
handle: (request) =>
Promise.resolve({
confidence: 0.9,
requestId: request.requestId,
source: "template",
text: "[HANDOFF:target:missing-tool]",
}),
};
const target: Skill = {
...makeSkill("target", 60),
canHandle: (request) => request.skill === "target",
handle: (request) =>
Promise.resolve({
confidence: 1,
metadata: {
packetSeen: Boolean(request.context?.["handoffPacket"]),
},
requestId: request.requestId,
source: "template",
text: "handled elsewhere",
}),
};
const { nats } = await handleThroughAgent(
[source, target],
makeRequest("source"),
);
expect(nats.responses[0]?.text).toBe("handled elsewhere");
expect(nats.responses[0]?.metadata?.["packetSeen"]).toBe(true);
expect(nats.responses[0]?.metadata).toMatchObject({
handoffChain: ["source", "target"],
handoffDepth: 1,
});
});
it("routes runtime verifier escalations to supervisor", async () => {
const previous = process.env["AGENT_RUNTIME_VERIFIER_ENABLED"];
process.env["AGENT_RUNTIME_VERIFIER_ENABLED"] = "true";
try {
const moderation: Skill = {
...makeSkill("moderation", 50),
canHandle: (request) => request.skill === "moderation",
handle: (request) =>
Promise.resolve({
confidence: 0.9,
requestId: request.requestId,
source: "template",
text: "https://not-allowed.example",
}),
};
const supervisor: Skill = {
...makeSkill("supervisor", 90),
canHandle: (request) => request.skill === "supervisor",
handle: (request) =>
Promise.resolve({
confidence: 0.6,
metadata: {
packetSeen: Boolean(request.context?.["handoffPacket"]),
},
requestId: request.requestId,
source: "template",
text: "sent to review",
}),
};
const { nats } = await handleThroughAgent(
[moderation, supervisor],
makeRequest("moderation"),
);
expect(nats.responses[0]?.text).toBe("sent to review");
expect(nats.responses[0]?.metadata?.["packetSeen"]).toBe(true);
expect(nats.responses[0]?.metadata).toMatchObject({
handoffChain: ["moderation", "supervisor"],
handoffDepth: 1,
});
} finally {
if (previous === undefined) {
delete process.env["AGENT_RUNTIME_VERIFIER_ENABLED"];
} else {
process.env["AGENT_RUNTIME_VERIFIER_ENABLED"] = previous;
}
}
});
it("reroutes illegal transitions to supervisor", async () => {
const source: Skill = {
...makeSkill("source", 50),
canHandle: (request) => request.skill === "source",
handle: (request) =>
Promise.resolve({
confidence: 0.9,
metadata: {
handoffRequest: {
confidence: 0.9,
evidenceRefs: [],
expectedOutput: "text",
reason: "out-of-scope",
risk: "medium",
targetAgent: "target",
taskSpec: "continue",
},
},
requestId: request.requestId,
source: "template",
text: "handoff",
}),
};
const target: Skill = {
...makeSkill("target", 60),
canHandle: (request) => request.skill === "target",
handle: (request) =>
Promise.resolve({
confidence: 1,
requestId: request.requestId,
source: "template",
text: "should not run",
}),
};
const supervisor: Skill = {
...makeSkill("supervisor", 10),
canHandle: (request) => request.skill === "supervisor",
handle: (request) =>
Promise.resolve({
confidence: 0.6,
metadata: {
packetSeen: Boolean(request.context?.["handoffPacket"]),
},
requestId: request.requestId,
source: "template",
text: "supervisor review",
}),
};
const matrix = {
isAllowed: () => false,
} as unknown as TransitionMatrix;
const { nats } = await handleThroughAgent(
[source, target, supervisor],
makeRequest("source"),
matrix,
);
expect(nats.responses[0]?.text).toBe("supervisor review");
expect(nats.responses[0]?.metadata?.["packetSeen"]).toBe(true);
expect(nats.responses[0]?.metadata).toMatchObject({
handoffChain: ["source", "supervisor"],
handoffDepth: 1,
});
});
});
+16 -83
View File
@@ -3,16 +3,10 @@ import type { ConversationStore } from "@conv/conversation-store";
import type { NatsClient } from "@common/nats"; import type { NatsClient } from "@common/nats";
import { getRequestLogger, runWithContext } from "@obs/context"; import { getRequestLogger, runWithContext } from "@obs/context";
import { getLogger } from "@obs/logger"; import { getLogger } from "@obs/logger";
import { withSpan } from "@obs/spans"; import { withSkillSpan, withSpan } from "@obs/spans";
import type { Skill } from "@skills/skill"; import type { Skill } from "@skills/skill";
import type { TransitionMatrix } from "@skills/transition-matrix";
import { getSkillJudgeConfig } from "@eval/judges/skill-config";
import { metrics } from "@obs/metrics";
import { BudgetTracker } from "./budget.js";
import { dispatch } from "./dispatch.js";
import type { AgentRequest, AgentResponse } from "./types.js"; import type { AgentRequest, AgentResponse } from "./types.js";
import { import {
getAgentRuntimeVerifierEnabled,
getAgentRequestTimeoutMs, getAgentRequestTimeoutMs,
getRequestSubject, getRequestSubject,
getSafetyMarginMs, getSafetyMarginMs,
@@ -31,17 +25,6 @@ function isValidRequest(data: unknown): data is AgentRequest {
); );
} }
export function selectSkillForRequest(
skills: readonly Skill[],
request: AgentRequest,
): Skill | null {
return (
skills
.filter((candidate) => candidate.canHandle(request))
.sort((left, right) => right.priority - left.priority)[0] ?? null
);
}
export class Agent { export class Agent {
private readonly skills: Skill[] = []; private readonly skills: Skill[] = [];
private readonly logger = getLogger(); private readonly logger = getLogger();
@@ -49,7 +32,6 @@ export class Agent {
constructor( constructor(
private readonly natsClient: NatsClient, private readonly natsClient: NatsClient,
private readonly conversationStore: ConversationStore, private readonly conversationStore: ConversationStore,
private readonly transitionMatrix?: TransitionMatrix | undefined,
) {} ) {}
registerSkill(skill: Skill): void { registerSkill(skill: Skill): void {
@@ -114,28 +96,18 @@ export class Agent {
"[Agent] Request received", "[Agent] Request received",
); );
let skill = selectSkillForRequest(this.skills, request); const skill = this.skills.find((candidate) => candidate.canHandle(request));
if (!skill) { if (!skill) {
const supervisorRequest: AgentRequest = { logger.info("[Agent] No skill found for request");
...request, const response: AgentResponse = {
skill: "supervisor", requestId: request.requestId,
trigger: "handoff", text: "",
confidence: 0,
source: "template",
}; };
skill = selectSkillForRequest(this.skills, supervisorRequest); rootSpan.setAttribute("agent.source", "template");
if (skill) { await this.natsClient.respond(msg, response);
request = supervisorRequest; return;
} else {
logger.info("[Agent] No skill found for request");
const response: AgentResponse = {
requestId: request.requestId,
text: "",
confidence: 0,
source: "template",
};
rootSpan.setAttribute("agent.source", "template");
await this.natsClient.respond(msg, response);
return;
}
} }
try { try {
@@ -156,45 +128,13 @@ export class Agent {
); );
} }
const startedAt = Date.now(); const response = await withSkillSpan(
const budget = new BudgetTracker(); skill.name,
budget.recordVisit(skill.name);
const response = await dispatch(
{ {
budget, "agent.skill": skill.name,
chain: [skill.name], "agent.trigger": request.trigger,
depth: 0,
history,
request,
skill,
},
{
getSkillJudgeConfig,
onHandoff: (event) => {
metrics.handoffTotal.inc({
from: event.from,
reason: event.reason,
skill: event.from,
to: event.to,
});
},
onHandoffLoop: (event) => {
metrics.handoffLoopTotal.inc({
from: event.from,
skill: event.from,
});
},
onValidatorFail: (event) => {
metrics.validatorFailTotal.inc({
skill: event.skill,
stage: event.stage,
});
},
selectSkillForRequest,
shouldApplyRuntimeVerifier: getAgentRuntimeVerifierEnabled,
skills: this.skills,
transitionMatrix: this.transitionMatrix,
}, },
() => skill.handle(request, history),
); );
if (response.text.trim()) { if (response.text.trim()) {
@@ -210,13 +150,6 @@ export class Agent {
} }
await this.natsClient.respond(msg, response); await this.natsClient.respond(msg, response);
metrics.handoffCount.observe(
Math.max(0, budget.snapshot().handoffs),
{ skill: skill.name },
);
metrics.timeToResolutionMs.observe(Date.now() - startedAt, {
skill: skill.name,
});
rootSpan.setAttribute( rootSpan.setAttribute(
"agent.source", "agent.source",
response.text.trim() ? response.source : "empty", response.text.trim() ? response.source : "empty",
-102
View File
@@ -1,102 +0,0 @@
import { describe, expect, it } from "vitest";
import { BudgetTracker } from "./budget.js";
describe("BudgetTracker", () => {
it("allows handoff while within limits", () => {
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 2,
maxSameAgentVisits: 2,
maxToolCalls: 2,
});
expect(budget.assertBeforeHandoff("moderation")).toBe("ok");
budget.recordHandoff("moderation");
expect(budget.snapshot()).toMatchObject({
handoffs: 1,
visits: { moderation: 1 },
});
});
it("rejects loops and exhausted handoff budgets", () => {
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 1,
maxSameAgentVisits: 1,
maxToolCalls: 5,
});
budget.recordVisit("moderation");
expect(budget.assertBeforeHandoff("moderation")).toBe("same-agent-loop");
budget.recordHandoff("onboarding");
expect(budget.assertBeforeHandoff("supervisor")).toBe("max-hops");
});
it("rejects exhausted tool and cost budgets", () => {
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 0.5,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 1,
});
budget.recordToolCall();
expect(budget.assertBeforeHandoff("supervisor")).toBe("tool-budget");
const costBudget = new BudgetTracker({
enabled: true,
maxCostUsd: 0.5,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
costBudget.recordToolCall(0.5);
expect(costBudget.assertBeforeHandoff("supervisor")).toBe("cost-budget");
});
it("tracks LLM and tool costs separately while enforcing the total budget", () => {
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 0.0055,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
for (let index = 0; index < 5; index += 1) {
budget.recordLlmCost(0.001);
}
budget.recordToolCall(0.0005);
expect(budget.snapshot()).toMatchObject({
estimatedCostUsd: 0.0055,
totalCostUsd: 0.0055,
totalLlmCostUsd: 0.005,
totalToolCostUsd: 0.0005,
toolCalls: 1,
});
expect(budget.assertBeforeHandoff("supervisor")).toBe("cost-budget");
});
it("can be disabled", () => {
const budget = new BudgetTracker({
enabled: false,
maxCostUsd: 0,
maxHandoffs: 0,
maxSameAgentVisits: 0,
maxToolCalls: 0,
});
budget.recordVisit("supervisor");
budget.recordHandoff("moderation");
budget.recordToolCall(1);
budget.recordLlmCost(1);
expect(budget.assertBeforeHandoff("supervisor")).toBe("ok");
});
});
-107
View File
@@ -1,107 +0,0 @@
import {
getAgentBudgetEnabled,
getAgentBudgetMaxCostUsd,
getAgentBudgetMaxHandoffs,
getAgentBudgetMaxSameAgentVisits,
getAgentBudgetMaxToolCalls,
} from "../config.js";
export type BudgetVerdict =
| "ok"
| "max-hops"
| "same-agent-loop"
| "tool-budget"
| "cost-budget";
export interface BudgetSnapshot {
estimatedCostUsd: number;
handoffs: number;
maxCostUsd: number;
maxHandoffs: number;
maxSameAgentVisits: number;
maxToolCalls: number;
totalCostUsd: number;
totalLlmCostUsd: number;
totalToolCostUsd: number;
toolCalls: number;
visits: Record<string, number>;
}
export class BudgetTracker {
private handoffs = 0;
private totalLlmCostUsd = 0;
private totalToolCostUsd = 0;
private toolCalls = 0;
private readonly visits = new Map<string, number>();
constructor(
private readonly limits = {
enabled: getAgentBudgetEnabled(),
maxCostUsd: getAgentBudgetMaxCostUsd(),
maxHandoffs: getAgentBudgetMaxHandoffs(),
maxSameAgentVisits: getAgentBudgetMaxSameAgentVisits(),
maxToolCalls: getAgentBudgetMaxToolCalls(),
},
) {}
assertBeforeHandoff(target: string): BudgetVerdict {
if (!this.limits.enabled) {
return "ok";
}
if (this.handoffs >= this.limits.maxHandoffs) {
return "max-hops";
}
if ((this.visits.get(target) ?? 0) >= this.limits.maxSameAgentVisits) {
return "same-agent-loop";
}
if (this.toolCalls >= this.limits.maxToolCalls) {
return "tool-budget";
}
if (this.totalCostUsd() >= this.limits.maxCostUsd) {
return "cost-budget";
}
return "ok";
}
recordHandoff(target: string): void {
this.handoffs += 1;
this.recordVisit(target);
}
recordToolCall(estimatedCostUsd = 0): void {
this.toolCalls += 1;
this.totalToolCostUsd += estimatedCostUsd;
}
recordLlmCost(costUsd: number): void {
if (!Number.isFinite(costUsd) || costUsd <= 0) {
return;
}
this.totalLlmCostUsd += costUsd;
}
recordVisit(target: string): void {
this.visits.set(target, (this.visits.get(target) ?? 0) + 1);
}
snapshot(): BudgetSnapshot {
const totalCostUsd = this.totalCostUsd();
return {
estimatedCostUsd: Number(totalCostUsd.toFixed(6)),
handoffs: this.handoffs,
maxCostUsd: this.limits.maxCostUsd,
maxHandoffs: this.limits.maxHandoffs,
maxSameAgentVisits: this.limits.maxSameAgentVisits,
maxToolCalls: this.limits.maxToolCalls,
totalCostUsd: Number(totalCostUsd.toFixed(6)),
totalLlmCostUsd: Number(this.totalLlmCostUsd.toFixed(6)),
totalToolCostUsd: Number(this.totalToolCostUsd.toFixed(6)),
toolCalls: this.toolCalls,
visits: Object.fromEntries(this.visits.entries()),
};
}
private totalCostUsd(): number {
return this.totalLlmCostUsd + this.totalToolCostUsd;
}
}
-581
View File
@@ -1,581 +0,0 @@
import { getSkillJudgeConfig as defaultGetSkillJudgeConfig } from "@eval/judges/skill-config";
import type { SkillJudgeConfig } from "@eval/judges/skill-config";
import { runWithContext } from "@obs/context";
import { withSkillSpan } from "@obs/spans";
import { parseControlTokens } from "@skills/control-token-parser";
import { decide as decideEscalation } from "@skills/escalation/trigger";
import type { Skill } from "@skills/skill";
import type { TransitionMatrix } from "@skills/transition-matrix";
import { getAgentRequestTimeoutMs, getSafetyMarginMs } from "../config.js";
import type { BudgetTracker, BudgetVerdict } from "./budget.js";
import { buildHandoffPacket } from "./handoff-packet.js";
import type {
AgentRequest,
AgentResponse,
ConversationMessage,
HandoffReason,
HandoffRequest,
} from "./types.js";
export interface HandoffEvent {
from: string;
reason: HandoffReason;
to: string;
}
export interface ValidatorFailEvent {
skill: string;
stage: string;
}
export interface EvalDispatchPlanStep {
confidence?: number | undefined;
expectedOutput?: string | undefined;
from: string;
reason: HandoffReason;
risk?: HandoffRequest["risk"] | undefined;
taskSpec?: string | undefined;
to: string;
}
export interface EvalDispatchPlan {
maxDepth?: number | undefined;
steps: EvalDispatchPlanStep[];
useActualSkills?: boolean | undefined;
}
export interface DispatchDependencies {
evalPlan?: EvalDispatchPlan | undefined;
getSkillJudgeConfig?: ((skillName: string) => SkillJudgeConfig) | undefined;
onHandoff?: ((event: HandoffEvent) => void) | undefined;
onHandoffLoop?: ((event: { from: string }) => void) | undefined;
onValidatorFail?: ((event: ValidatorFailEvent) => void) | undefined;
selectSkillForRequest: (
skills: readonly Skill[],
request: AgentRequest,
) => Skill | null;
shouldApplyRuntimeVerifier: () => boolean;
skills: readonly Skill[];
transitionMatrix?: TransitionMatrix | undefined;
}
export interface DispatchInput {
budget: BudgetTracker;
chain: string[];
depth: number;
evalPlanIndex?: number | undefined;
history: ConversationMessage[];
lastHandoff?: HandoffRequest | undefined;
request: AgentRequest;
skill: Skill;
}
export async function dispatch(
input: DispatchInput,
deps: DispatchDependencies,
): Promise<AgentResponse> {
const { budget, chain, depth, history, lastHandoff, request, skill } = input;
const contextBase = {
chatroomId: request.chatroomId,
deadlineAt: new Date(
Date.now() + getAgentRequestTimeoutMs() - getSafetyMarginMs(),
).toISOString(),
requestId: request.requestId,
skill: skill.name,
startedAt: new Date().toISOString(),
trigger: request.trigger,
userId: request.userId,
};
return runWithContext(
{
...contextBase,
budget,
handoffChain: chain,
handoffDepth: depth,
},
async () => {
const planStep = getCurrentPlanStep(input, deps);
const response = planStep
? plannedHandoffResponse(request, planStep)
: deps.evalPlan && deps.evalPlan.useActualSkills !== true
? templateEvalResponse(request, skill.name)
: await runSkillOnce({
chain,
depth,
history,
request,
skill,
});
const normalized = normalizeControlTokens(response, skill.name);
const verified = await applyRuntimeVerifier(
{
chain,
depth,
history,
request,
response: normalized,
skill,
},
deps,
);
const escalated = applyEscalationTrigger({
budget,
request,
response: verified,
skill,
});
const annotated = withHandoffMetadata(
escalated,
chain,
depth,
lastHandoff,
);
const handoff = readHandoffRequest(annotated);
if (!handoff) {
return annotated;
}
const next = prepareHandoff(
{
budget,
chain,
currentSkill: skill.name,
depth,
evalPlanIndex: input.evalPlanIndex ?? 0,
handoff,
history,
planStepUsed: Boolean(planStep),
request,
response: annotated,
},
deps,
);
if (!next) {
const fallbackHandoff: HandoffRequest = {
...handoff,
targetAgent: "supervisor",
};
return withHandoffMetadata(
makeSupervisorFallback(request, handoff.reason),
[...chain, "supervisor"],
depth + 1,
fallbackHandoff,
);
}
const maxDepth = deps.evalPlan?.maxDepth ?? 8;
if (next.depth > maxDepth) {
const fallbackHandoff: HandoffRequest = {
confidence: 0.4,
evidenceRefs: [],
expectedOutput: "text",
reason: "max-hop",
risk: "medium",
targetAgent: "supervisor",
taskSpec: "Dispatch exceeded max loop depth.",
};
const fallbackChain =
chain.at(-1) === "supervisor" ? chain : [...chain, "supervisor"];
return withHandoffMetadata(
makeSupervisorFallback(request, "max-hop"),
fallbackChain,
fallbackChain.length - 1,
fallbackHandoff,
);
}
return dispatch(next, deps);
},
);
}
export function withHandoffMetadata(
response: AgentResponse,
chain: string[],
depth: number,
lastHandoff?: HandoffRequest,
): AgentResponse {
return {
...response,
metadata: {
...(response.metadata ?? {}),
handoffChain: chain,
handoffDepth: depth,
...(lastHandoff ? { handoffLastRequest: lastHandoff } : {}),
},
};
}
export function normalizeControlTokens(
response: AgentResponse,
currentSkill: string,
): AgentResponse {
const parsed = parseControlTokens(response.text);
const handoff = parsed.tokens.find((token) => token.kind === "handoff");
const metadata = { ...(response.metadata ?? {}) };
if (handoff?.kind === "handoff") {
metadata.handoffRequest = {
confidence: response.confidence,
evidenceRefs: [],
expectedOutput: "text",
reason: handoff.reason,
risk: "medium",
targetAgent: handoff.targetAgent,
taskSpec: `Continue handling from ${currentSkill}`,
} satisfies HandoffRequest;
}
return {
...response,
metadata,
text: parsed.stripped,
};
}
export function readHandoffRequest(
response: AgentResponse,
): HandoffRequest | null {
const value = response.metadata?.["handoffRequest"];
if (!value || typeof value !== "object") {
return null;
}
const handoff = value as Partial<HandoffRequest>;
if (
typeof handoff.targetAgent !== "string" ||
typeof handoff.reason !== "string" ||
typeof handoff.taskSpec !== "string" ||
typeof handoff.confidence !== "number" ||
typeof handoff.expectedOutput !== "string"
) {
return null;
}
return {
confidence: handoff.confidence,
evidenceRefs: Array.isArray(handoff.evidenceRefs)
? handoff.evidenceRefs.filter((item): item is string => typeof item === "string")
: [],
expectedOutput: handoff.expectedOutput,
reason: handoff.reason,
risk:
handoff.risk === "low" || handoff.risk === "medium" || handoff.risk === "high"
? handoff.risk
: "medium",
targetAgent: handoff.targetAgent,
taskSpec: handoff.taskSpec,
...(handoff.userVisibleMessage
? { userVisibleMessage: handoff.userVisibleMessage }
: {}),
};
}
export function budgetVerdictToReason(verdict: BudgetVerdict): HandoffReason {
if (verdict === "same-agent-loop") {
return "same-agent-revisit";
}
if (verdict === "max-hops") {
return "max-hop";
}
return "policy-risk";
}
async function runSkillOnce(input: {
chain: string[];
depth: number;
history: ConversationMessage[];
request: AgentRequest;
skill: Skill;
}): Promise<AgentResponse> {
return withSkillSpan(
input.skill.name,
{
"agent.handoff_chain": input.chain.join(">"),
"agent.handoff_depth": input.depth,
"agent.skill": input.skill.name,
"agent.trigger": input.request.trigger,
},
() => input.skill.handle(input.request, input.history),
);
}
async function applyRuntimeVerifier(
input: {
chain: string[];
depth: number;
history: ConversationMessage[];
request: AgentRequest;
response: AgentResponse;
skill: Skill;
},
deps: DispatchDependencies,
): Promise<AgentResponse> {
if (!deps.shouldApplyRuntimeVerifier()) {
return input.response;
}
const judgeConfig = deps.getSkillJudgeConfig ?? defaultGetSkillJudgeConfig;
const verifier = judgeConfig(input.skill.name).runtimeVerifier;
if (!verifier) {
return input.response;
}
const verdict = await verifier({
request: input.request,
response: input.response,
skill: input.skill.name,
});
if (verdict === "pass") {
return input.response;
}
deps.onValidatorFail?.({ skill: input.skill.name, stage: verdict });
if (verdict === "retry" && input.depth === 0) {
const retryResponse = await runSkillOnce({
...input,
request: {
...input.request,
context: {
...(input.request.context ?? {}),
runtimeVerifierRetry: true,
},
},
});
const normalizedRetry = normalizeControlTokens(
retryResponse,
input.skill.name,
);
if (readHandoffRequest(normalizedRetry)) {
return normalizedRetry;
}
const retryVerdict = await verifier({
request: input.request,
response: normalizedRetry,
skill: input.skill.name,
});
if (retryVerdict === "pass") {
return {
...normalizedRetry,
metadata: {
...(normalizedRetry.metadata ?? {}),
runtimeVerifier: "retry-pass",
},
};
}
deps.onValidatorFail?.({
skill: input.skill.name,
stage: `retry-${retryVerdict}`,
});
return withRuntimeVerifierHandoff(normalizedRetry);
}
return withRuntimeVerifierHandoff(input.response);
}
function withRuntimeVerifierHandoff(response: AgentResponse): AgentResponse {
return {
...response,
metadata: {
...(response.metadata ?? {}),
handoffRequest: {
confidence: response.confidence,
evidenceRefs: [],
expectedOutput: "text",
reason: "validation-failed",
risk: "medium",
targetAgent: "supervisor",
taskSpec: "Runtime verifier rejected the response.",
} satisfies HandoffRequest,
},
};
}
function applyEscalationTrigger(input: {
budget: BudgetTracker;
request: AgentRequest;
response: AgentResponse;
skill: Skill;
}): AgentResponse {
if (!input.response.text.trim() || readHandoffRequest(input.response)) {
return input.response;
}
const decision = decideEscalation({
budget: input.budget.snapshot(),
capabilityCard: input.skill.capabilityCard?.(),
classified: { confidence: input.response.confidence },
targetAgent: "supervisor",
taskSpec: `Review ${input.skill.name} response for ${input.request.trigger}.`,
});
if (!decision.handoff) {
return input.response;
}
return {
...input.response,
metadata: {
...(input.response.metadata ?? {}),
handoffRequest: decision.handoff,
},
};
}
function prepareHandoff(
input: {
budget: BudgetTracker;
chain: string[];
currentSkill: string;
depth: number;
evalPlanIndex: number;
handoff: HandoffRequest;
history: ConversationMessage[];
planStepUsed: boolean;
request: AgentRequest;
response: AgentResponse;
},
deps: DispatchDependencies,
): DispatchInput | null {
const target = resolveHandoffTarget(input, deps);
const nextSkill = deps.selectSkillForRequest(deps.skills, {
...input.request,
skill: target.targetAgent,
trigger: "handoff",
});
if (!nextSkill) {
return null;
}
input.budget.recordHandoff(nextSkill.name);
deps.onHandoff?.({
from: input.currentSkill,
reason: target.reason,
to: nextSkill.name,
});
const nextChain = [...input.chain, nextSkill.name];
const lastHandoff: HandoffRequest = {
...input.handoff,
reason: target.reason,
targetAgent: nextSkill.name,
};
const packet = buildHandoffPacket({
handoff: lastHandoff,
history: input.history,
request: input.request,
response: input.response,
sourceChain: nextChain,
state: {
source: input.currentSkill,
},
});
return {
budget: input.budget,
chain: nextChain,
depth: input.depth + 1,
evalPlanIndex: input.evalPlanIndex + (input.planStepUsed ? 1 : 0),
history: input.history,
lastHandoff,
request: {
...input.request,
context: {
...(input.request.context ?? {}),
handoffPacket: packet,
},
skill: nextSkill.name,
trigger: "handoff",
},
skill: nextSkill,
};
}
function resolveHandoffTarget(
input: {
budget: BudgetTracker;
currentSkill: string;
handoff: HandoffRequest;
},
deps: DispatchDependencies,
): { reason: HandoffReason; targetAgent: string } {
const requestedTarget = input.handoff.targetAgent;
const allowed =
deps.transitionMatrix?.isAllowed(input.currentSkill, requestedTarget) ?? true;
if (!allowed) {
return {
reason: "policy-risk",
targetAgent: "supervisor",
};
}
const verdict = input.budget.assertBeforeHandoff(requestedTarget);
if (verdict === "ok") {
return {
reason: input.handoff.reason,
targetAgent: requestedTarget,
};
}
if (verdict === "same-agent-loop") {
deps.onHandoffLoop?.({ from: input.currentSkill });
}
return {
reason: budgetVerdictToReason(verdict),
targetAgent: "supervisor",
};
}
function makeSupervisorFallback(
request: AgentRequest,
reason: HandoffReason,
): AgentResponse {
return {
confidence: 0.4,
metadata: {
supervisorFallbackReason: reason,
},
requestId: request.requestId,
source: "template",
text: "확인이 필요한 내용이라 운영자에게 전달했어요.",
};
}
function getCurrentPlanStep(
input: DispatchInput,
deps: DispatchDependencies,
): EvalDispatchPlanStep | undefined {
const index = input.evalPlanIndex ?? 0;
const step = deps.evalPlan?.steps[index];
return step?.from === input.skill.name ? step : undefined;
}
function plannedHandoffResponse(
request: AgentRequest,
step: EvalDispatchPlanStep,
): AgentResponse {
return {
confidence: step.confidence ?? 0.72,
metadata: {
handoffRequest: {
confidence: step.confidence ?? 0.72,
evidenceRefs: [],
expectedOutput: step.expectedOutput ?? "text",
reason: step.reason,
risk: step.risk ?? "medium",
targetAgent: step.to,
taskSpec: step.taskSpec ?? `Continue from ${step.from} to ${step.to}`,
} satisfies HandoffRequest,
},
requestId: request.requestId,
source: "template",
text: `handoff ${step.from} to ${step.to}`,
};
}
function templateEvalResponse(
request: AgentRequest,
skillName: string,
): AgentResponse {
return {
confidence: 1,
requestId: request.requestId,
source: "template",
text: `handoff eval final: ${skillName}`,
};
}
-79
View File
@@ -1,79 +0,0 @@
import { describe, expect, it } from "vitest";
import { buildHandoffPacket, readHandoffPacket } from "./handoff-packet.js";
import type { AgentRequest, AgentResponse, ConversationMessage } from "./types.js";
const request: AgentRequest = {
chatroomId: "room-1",
nickname: "tester",
requestId: "request-1",
skill: "onboarding",
trigger: "user-message",
userId: "user-1",
userMessage: "private original text",
};
const response: AgentResponse = {
confidence: 0.7,
requestId: "request-1",
source: "template",
text: "needs handoff",
};
const history: ConversationMessage[] = [
{
content: "이전 메시지",
role: "user",
timestamp: "2026-05-18T00:00:00.000Z",
},
];
describe("handoff packet", () => {
it("builds a bounded packet without raw user message", () => {
const packet = buildHandoffPacket({
handoff: {
confidence: 0.7,
evidenceRefs: ["tool-1"],
expectedOutput: "text",
reason: "out-of-scope",
risk: "medium",
targetAgent: "moderation",
taskSpec: "Call 010-1234-5678 after 2026-05-18",
},
history,
request,
response,
sourceChain: ["onboarding", "moderation"],
});
expect(packet.userGoal).toBe("Call <phone> after <date>");
expect(packet.userGoal).not.toContain(request.userMessage);
expect(packet.sourceChain).toEqual(["onboarding", "moderation"]);
expect(packet.evidence).toEqual([
{
id: "tool-1",
kind: "tool-call",
summary: "source agent evidence reference",
},
]);
});
it("reads packet from request context", () => {
const packet = buildHandoffPacket({
handoff: {
confidence: 0.7,
evidenceRefs: [],
expectedOutput: "text",
reason: "missing-tool",
risk: "low",
targetAgent: "moderation",
taskSpec: "continue",
},
request,
response,
sourceChain: ["onboarding", "moderation"],
});
expect(readHandoffPacket({ ...request, context: { handoffPacket: packet } })).toBe(packet);
});
});
-83
View File
@@ -1,83 +0,0 @@
import type {
AgentRequest,
AgentResponse,
ConversationMessage,
HandoffReason,
HandoffRequest,
} from "./types.js";
export interface HandoffEvidence {
id: string;
kind: "message" | "tool-call" | "response";
summary: string;
}
export interface HandoffPacket {
constraints: string[];
currentState: Record<string, unknown>;
evidence: HandoffEvidence[];
expectedOutputSchema: string;
handoffReason: HandoffReason;
sourceChain: string[];
summary: string;
userGoal: string;
}
function asRecord(value: unknown): Record<string, unknown> {
return typeof value === "object" && value !== null
? (value as Record<string, unknown>)
: {};
}
function fallbackSummary(history: readonly ConversationMessage[]): string {
return history
.slice(-3)
.map((message) => `${message.role}: ${message.content.slice(0, 120)}`)
.join(" | ");
}
function sanitizeGoal(value: string | undefined): string {
if (!value?.trim()) {
return "이전 agent가 요청한 작업을 이어서 처리";
}
return value
.replace(/\b\d{2,4}[-./]\d{1,2}[-./]\d{1,2}\b/gu, "<date>")
.replace(/\b\d{2,3}[- ]?\d{3,4}[- ]?\d{4}\b/gu, "<phone>")
.slice(0, 240);
}
export function buildHandoffPacket(input: {
handoff: HandoffRequest;
history?: readonly ConversationMessage[] | undefined;
request: AgentRequest;
response: AgentResponse;
sourceChain: string[];
state?: Record<string, unknown> | undefined;
summary?: string | undefined;
}): HandoffPacket {
return {
constraints: [
"원문 사용자 메시지와 닉네임 등 PII를 새로 노출하지 않는다.",
"handoffRequest.expectedOutput 범위 안에서만 답한다.",
],
currentState: input.state ?? {},
evidence: input.handoff.evidenceRefs.map((id) => ({
id,
kind: "tool-call",
summary: "source agent evidence reference",
})),
expectedOutputSchema: input.handoff.expectedOutput,
handoffReason: input.handoff.reason,
sourceChain: input.sourceChain,
summary: input.summary?.trim() || fallbackSummary(input.history ?? []),
userGoal: sanitizeGoal(input.handoff.taskSpec),
};
}
export function readHandoffPacket(request: AgentRequest): HandoffPacket | undefined {
const packet = asRecord(request.context)["handoffPacket"];
if (!packet || typeof packet !== "object") {
return undefined;
}
return packet as HandoffPacket;
}
-21
View File
@@ -9,27 +9,6 @@ export interface AgentRequest {
context?: Record<string, unknown> | undefined; context?: Record<string, unknown> | undefined;
} }
export type HandoffReason =
| "out-of-scope"
| "missing-tool"
| "low-confidence"
| "validation-failed"
| "same-agent-revisit"
| "max-hop"
| "high-risk-action"
| "policy-risk";
export interface HandoffRequest {
targetAgent: string;
reason: HandoffReason;
taskSpec: string;
evidenceRefs: string[];
confidence: number;
risk: "low" | "medium" | "high";
expectedOutput: string;
userVisibleMessage?: string | undefined;
}
export interface AgentResponse { export interface AgentResponse {
requestId: string; requestId: string;
text: string; text: string;
-11
View File
@@ -1,5 +1,4 @@
import fs from "node:fs"; import fs from "node:fs";
import os from "node:os";
import path from "node:path"; import path from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import * as dotenv from "dotenv"; import * as dotenv from "dotenv";
@@ -35,11 +34,6 @@ function resolveEnvFile(mode: NodeEnvMode): string | null {
return null; return null;
} }
function resolveGlobalEnvFile(): string | null {
const candidate = path.resolve(os.homedir(), "source", "global-env-vars.txt");
return fs.existsSync(candidate) ? candidate : null;
}
function loadDotenvFile(filePath: string, label: string): void { function loadDotenvFile(filePath: string, label: string): void {
const result = dotenv.config({ path: filePath, override: false }); const result = dotenv.config({ path: filePath, override: false });
if (result.error) { if (result.error) {
@@ -54,11 +48,6 @@ export function loadEnvironmentFromNodeEnv(): void {
const mode = normalizeMode(process.env["NODE_ENV"]); const mode = normalizeMode(process.env["NODE_ENV"]);
process.env["NODE_ENV"] = mode; process.env["NODE_ENV"] = mode;
const globalEnvFile = resolveGlobalEnvFile();
if (globalEnvFile) {
loadDotenvFile(globalEnvFile, "global-env-vars.txt");
}
const envFile = resolveEnvFile(mode); const envFile = resolveEnvFile(mode);
if (!envFile) { if (!envFile) {
logger.warn(`[Env] env-vars/${mode}.env not found. Skipping dotenv load.`); logger.warn(`[Env] env-vars/${mode}.env not found. Skipping dotenv load.`);
-14
View File
@@ -1,14 +0,0 @@
import type { GarnetClient as GarnetClientType } from "./garnet.js";
export async function connectGarnetForCli(): Promise<{
close: () => Promise<void>;
garnetClient: GarnetClientType;
}> {
const { GarnetClient } = await import("./garnet.js");
const garnetClient = new GarnetClient();
await garnetClient.connect();
return {
close: () => garnetClient.close(),
garnetClient,
};
}
-6
View File
@@ -9,12 +9,6 @@ export const KNOWN_SECRET_KEYS = [
"GARNET_PASSWORD", "GARNET_PASSWORD",
"MONGODB_CONNECTION_STRING", "MONGODB_CONNECTION_STRING",
"MONGODB_PROJECT_ID", "MONGODB_PROJECT_ID",
"OPENROUTER_API_KEY",
"AGENT_LLM_API_KEY",
"OPENAI_API_KEY",
"EMBEDDING_API_KEY",
"PHOENIX_API_KEY",
"SLACK_ALERT_WEBHOOK_URL",
] as const; ] as const;
export type KnownSecretKey = (typeof KNOWN_SECRET_KEYS)[number]; export type KnownSecretKey = (typeof KNOWN_SECRET_KEYS)[number];
-41
View File
@@ -1,41 +0,0 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { runWithContext } from "../observability/context.js";
import { NatsClient } from "./nats.js";
import { AGENT_ALERT_SUBJECT, DISCORD_MEMBER_LOOKUP_SUBJECT } from "./nats-subjects.js";
import { checkPublishAllowed } from "./nats-skill-acl.js";
describe("NATS skill ACL", () => {
afterEach(() => {
vi.unstubAllEnvs();
});
it("allows declared publish subjects", () => {
expect(checkPublishAllowed("onboarding", DISCORD_MEMBER_LOOKUP_SUBJECT)).toBe(
true,
);
expect(checkPublishAllowed("supervisor", AGENT_ALERT_SUBJECT)).toBe(true);
expect(checkPublishAllowed("onboarding", AGENT_ALERT_SUBJECT)).toBe(false);
});
it("enforces denied publish subjects when enabled", async () => {
vi.stubEnv("AGENT_NATS_ACL_MODE", "enforce");
const client = new NatsClient();
await expect(
runWithContext(
{
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() => client.publish(AGENT_ALERT_SUBJECT, { ok: true }),
),
).rejects.toThrow(
`NATS publish denied for skill onboarding on subject ${AGENT_ALERT_SUBJECT}`,
);
});
});
-54
View File
@@ -1,54 +0,0 @@
import {
AGENT_ALERT_SUBJECT,
DISCORD_MEMBER_LOOKUP_SUBJECT,
} from "./nats-subjects.js";
export interface SkillNatsAcl {
skill: string;
allowedPublishSubjects: string[];
allowedSubscribeSubjects: string[];
}
export const DEFAULT_NATS_ACL: SkillNatsAcl[] = [
{
skill: "onboarding",
allowedPublishSubjects: [DISCORD_MEMBER_LOOKUP_SUBJECT],
allowedSubscribeSubjects: [],
},
{
skill: "moderation",
allowedPublishSubjects: [],
allowedSubscribeSubjects: [],
},
{
skill: "supervisor",
allowedPublishSubjects: [AGENT_ALERT_SUBJECT],
allowedSubscribeSubjects: [],
},
];
function wildcardToRegExp(pattern: string): RegExp {
const escaped = pattern
.split("*")
.map((part) => part.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"))
.join(".*");
return new RegExp(`^${escaped}$`, "u");
}
function matchesAny(subject: string, patterns: readonly string[]): boolean {
return patterns.some((pattern) => wildcardToRegExp(pattern).test(subject));
}
function getAcl(skill: string): SkillNatsAcl | undefined {
return DEFAULT_NATS_ACL.find((entry) => entry.skill === skill);
}
export function checkPublishAllowed(skill: string, subject: string): boolean {
const acl = getAcl(skill);
return Boolean(acl && matchesAny(subject, acl.allowedPublishSubjects));
}
export function checkSubscribeAllowed(skill: string, subject: string): boolean {
const acl = getAcl(skill);
return Boolean(acl && matchesAny(subject, acl.allowedSubscribeSubjects));
}
+1 -24
View File
@@ -13,11 +13,9 @@ export interface AlertPayload {
| "eval-drift" | "eval-drift"
| "eval-score-drop" | "eval-score-drop"
| "latency" | "latency"
| "llm-error-rate" | "llm-error-rate";
| "supervisor-handoff";
metric: string; metric: string;
severity: "info" | "warn" | "critical"; severity: "info" | "warn" | "critical";
skill?: string | undefined;
suggestedAction?: string | undefined; suggestedAction?: string | undefined;
threshold: number; threshold: number;
timestamp: string; timestamp: string;
@@ -34,12 +32,6 @@ export type DiscordMemberMatchSource =
| "username" | "username"
| "joined-after"; | "joined-after";
export type DiscordResidentApplicationMessageMatchSource =
| "author-id"
| "author-name"
| "content"
| "joined-after";
export interface DiscordMemberCandidate { export interface DiscordMemberCandidate {
memberId: string; memberId: string;
joinedAt: string; joinedAt: string;
@@ -49,20 +41,8 @@ export interface DiscordMemberCandidate {
username: string; username: string;
} }
export interface DiscordResidentApplicationMessageCandidate {
authorDisplayName?: string | undefined;
authorGuildNickname?: string | undefined;
authorId: string;
channelId: string;
content: string;
createdAt: string;
matchedBy?: DiscordResidentApplicationMessageMatchSource | undefined;
messageId: string;
}
export interface DiscordMemberLookupRequest { export interface DiscordMemberLookupRequest {
chatroomId: string; chatroomId: string;
includeResidentApplicationMessages?: boolean | undefined;
joinedAfter: string; joinedAfter: string;
searchName?: string | undefined; searchName?: string | undefined;
userId: string; userId: string;
@@ -73,9 +53,6 @@ export type DiscordMemberLookupResponse =
status: "ok"; status: "ok";
joinedAfterMatches: DiscordMemberCandidate[]; joinedAfterMatches: DiscordMemberCandidate[];
nicknameMatches: DiscordMemberCandidate[]; nicknameMatches: DiscordMemberCandidate[];
residentApplicationMessageMatches?:
| DiscordResidentApplicationMessageCandidate[]
| undefined;
} }
| { | {
status: "error"; status: "error";
-39
View File
@@ -6,13 +6,7 @@ import {
type Subscription, type Subscription,
} from "nats"; } from "nats";
import { getSecret } from "@common/secret-manager"; import { getSecret } from "@common/secret-manager";
import { getNatsAclMode } from "../config.js";
import { getContext } from "../observability/context.js";
import { getLogger } from "@obs/logger"; import { getLogger } from "@obs/logger";
import {
checkPublishAllowed,
checkSubscribeAllowed,
} from "./nats-skill-acl.js";
export interface NatsMessageContext<T = unknown> { export interface NatsMessageContext<T = unknown> {
data: T; data: T;
@@ -87,7 +81,6 @@ export class NatsClient {
subject: string, subject: string,
handler: (msg: Msg) => void | Promise<void>, handler: (msg: Msg) => void | Promise<void>,
): Promise<Subscription> { ): Promise<Subscription> {
this.checkSubjectAcl("subscribe", subject);
const subscription = this.getConnection().subscribe(subject); const subscription = this.getConnection().subscribe(subject);
void (async () => { void (async () => {
@@ -108,7 +101,6 @@ export class NatsClient {
} }
async publish<T = unknown>(subject: string, data: T): Promise<void> { async publish<T = unknown>(subject: string, data: T): Promise<void> {
this.checkSubjectAcl("publish", subject);
const client = this.getConnection(); const client = this.getConnection();
client.publish(subject, this.stringCodec.encode(JSON.stringify(data))); client.publish(subject, this.stringCodec.encode(JSON.stringify(data)));
await client.flush(); await client.flush();
@@ -119,7 +111,6 @@ export class NatsClient {
data: TRequest, data: TRequest,
options?: { timeoutMs?: number | undefined }, options?: { timeoutMs?: number | undefined },
): Promise<TResponse> { ): Promise<TResponse> {
this.checkSubjectAcl("publish", subject);
const requestOptions = const requestOptions =
options?.timeoutMs !== undefined options?.timeoutMs !== undefined
? { ? {
@@ -135,36 +126,6 @@ export class NatsClient {
return this.decodeMessage<TResponse>(response); return this.decodeMessage<TResponse>(response);
} }
private checkSubjectAcl(
operation: "publish" | "subscribe",
subject: string,
): void {
const mode = getNatsAclMode();
if (mode === "off") {
return;
}
const skill = getContext()?.skill;
if (!skill) {
return;
}
const allowed =
operation === "publish"
? checkPublishAllowed(skill, subject)
: checkSubscribeAllowed(skill, subject);
if (allowed) {
return;
}
const payload = { mode, operation, skill, subject };
if (mode === "enforce") {
this.logger.error(payload, "[NATS ACL] Subject denied");
throw new Error(
`NATS ${operation} denied for skill ${skill} on subject ${subject}`,
);
}
this.logger.warn(payload, "[NATS ACL] Subject would be denied");
}
respond(msg: Msg, data: unknown): Promise<void> { respond(msg: Msg, data: unknown): Promise<void> {
if (!msg.reply) { if (!msg.reply) {
this.logger.warn( this.logger.warn(
-250
View File
@@ -1,250 +0,0 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import {
getAlertThresholdDriftPct,
getAlertThresholdEvalScoreDropPct,
getAlertThresholdLlmErrorRate,
getAlertThresholdP95LatencyMs,
getAgentBudgetLlmCostEnabled,
getAgentCostAwareTierEnabled,
getAgentSupervisorFastPathEnabled,
getAgentToolSandboxEnabled,
getEmbeddingModel,
getEmbeddingProvider,
getEvalCaseDeadlineMs,
getEvalLlmProvider,
getEvalRuntimeLlmEnabled,
getLongTermSummaryEnabled,
getLlmModelRoundRobinEnabled,
getRuntimeLlmModelsForProvider,
getMemoryEvictionEnabled,
getNatsAclMode,
getPhoenixLogPrompts,
getPhoenixOtlpEndpoint,
getRagHybridEnabled,
getRagRerankerEnabled,
getRagRerankerTopK,
getRagEnabled,
getSlackAlertWebhookUrl,
getSlackWebhookTimeoutMs,
getSupervisorFastPathConfidence,
getSupervisorLlmResponseEnabled,
getToolAuditMaxLen,
} from "./config.js";
const ENV_KEYS = [
"ALERT_THRESHOLD_DRIFT_PCT",
"ALERT_THRESHOLD_DRIFT_PCT_ONBOARDING",
"ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT",
"ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT_MODERATION",
"ALERT_THRESHOLD_LLM_ERROR_RATE",
"ALERT_THRESHOLD_LLM_ERROR_RATE_ONBOARDING",
"ALERT_THRESHOLD_P95_LATENCY_MS",
"ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING",
"AGENT_RAG_ENABLED",
"AGENT_RAG_HYBRID_ENABLED",
"AGENT_RAG_RERANKER_ENABLED",
"AGENT_RAG_RERANKER_TOP_K",
"AGENT_LONG_TERM_SUMMARY_ENABLED",
"AGENT_MEMORY_EVICTION_ENABLED",
"AGENT_BUDGET_LLM_COST_ENABLED",
"AGENT_COST_AWARE_TIER_ENABLED",
"AGENT_SUPERVISOR_FAST_PATH_ENABLED",
"SUPERVISOR_FAST_PATH_CONFIDENCE",
"AGENT_TOOL_SANDBOX_ENABLED",
"AGENT_TOOL_AUDIT_MAX_LEN",
"AGENT_NATS_ACL_MODE",
"EMBEDDING_MODEL",
"EMBEDDING_PROVIDER",
"PHOENIX_LOG_PROMPTS",
"PHOENIX_OTLP_ENDPOINT",
"SLACK_ALERT_WEBHOOK_URL",
"SLACK_WEBHOOK_TIMEOUT_MS",
"AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED",
"CLIPROXY_TIER1_PLANNER_MODEL",
"EVAL_CASE_DEADLINE_MS",
"EVAL_LLM_PROVIDER",
"EVAL_RUNTIME_LLM_ENABLED",
"LLM_MODEL_ROUND_ROBIN_ENABLED",
"OPENROUTER_TIER1_RESPONDER_MODEL",
] as const;
describe("config", () => {
afterEach(() => {
for (const key of ENV_KEYS) {
vi.stubEnv(key, undefined);
}
vi.unstubAllEnvs();
});
it("prefers skill-specific alert thresholds", () => {
vi.stubEnv("ALERT_THRESHOLD_P95_LATENCY_MS", "3000");
vi.stubEnv("ALERT_THRESHOLD_P95_LATENCY_MS_ONBOARDING", "5000");
vi.stubEnv("ALERT_THRESHOLD_LLM_ERROR_RATE", "0.1");
vi.stubEnv("ALERT_THRESHOLD_LLM_ERROR_RATE_ONBOARDING", "0.25");
expect(getAlertThresholdP95LatencyMs("onboarding")).toBe(5000);
expect(getAlertThresholdP95LatencyMs("moderation")).toBe(3000);
expect(getAlertThresholdLlmErrorRate("onboarding")).toBe(0.25);
});
it("normalizes skill names and falls back to global values", () => {
vi.stubEnv("ALERT_THRESHOLD_DRIFT_PCT", "15");
vi.stubEnv("ALERT_THRESHOLD_DRIFT_PCT_HIGH_RISK", "7");
vi.stubEnv("ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT", "10");
vi.stubEnv("ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT_MODERATION", "6");
expect(getAlertThresholdDriftPct("high-risk")).toBe(7);
expect(getAlertThresholdDriftPct("onboarding")).toBe(15);
expect(getAlertThresholdEvalScoreDropPct("moderation")).toBe(6);
});
it("reads Slack webhook configuration", () => {
vi.stubEnv("SLACK_ALERT_WEBHOOK_URL", " https://hooks.slack.test/x ");
vi.stubEnv("SLACK_WEBHOOK_TIMEOUT_MS", "1234");
expect(getSlackAlertWebhookUrl()).toBe("https://hooks.slack.test/x");
expect(getSlackWebhookTimeoutMs()).toBe(1234);
});
it("reads Phoenix tracing configuration", () => {
expect(getPhoenixOtlpEndpoint()).toBeUndefined();
expect(getPhoenixLogPrompts()).toBe(false);
vi.stubEnv("PHOENIX_OTLP_ENDPOINT", " http://localhost:6006/v1/traces ");
vi.stubEnv("PHOENIX_LOG_PROMPTS", "true");
expect(getPhoenixOtlpEndpoint()).toBe("http://localhost:6006/v1/traces");
expect(getPhoenixLogPrompts()).toBe(true);
});
it("reads RAG and embedding configuration", () => {
expect(getRagEnabled()).toBe(true);
expect(getRagHybridEnabled()).toBe(true);
expect(getRagRerankerEnabled()).toBe(true);
expect(getRagRerankerTopK()).toBe(5);
expect(getLongTermSummaryEnabled()).toBe(true);
expect(getMemoryEvictionEnabled()).toBe(false);
expect(getEmbeddingProvider()).toBe("openrouter");
expect(getEmbeddingModel()).toBe("openai/text-embedding-3-small");
vi.stubEnv("AGENT_RAG_ENABLED", "false");
vi.stubEnv("AGENT_RAG_HYBRID_ENABLED", "false");
vi.stubEnv("AGENT_RAG_RERANKER_ENABLED", "false");
vi.stubEnv("AGENT_RAG_RERANKER_TOP_K", "2");
vi.stubEnv("AGENT_LONG_TERM_SUMMARY_ENABLED", "false");
vi.stubEnv("AGENT_MEMORY_EVICTION_ENABLED", "true");
vi.stubEnv("EMBEDDING_PROVIDER", "openai");
vi.stubEnv("EMBEDDING_MODEL", "custom-embedding");
expect(getRagEnabled()).toBe(false);
expect(getRagHybridEnabled()).toBe(false);
expect(getRagRerankerEnabled()).toBe(false);
expect(getRagRerankerTopK()).toBe(2);
expect(getLongTermSummaryEnabled()).toBe(false);
expect(getMemoryEvictionEnabled()).toBe(true);
expect(getEmbeddingProvider()).toBe("openai");
expect(getEmbeddingModel()).toBe("custom-embedding");
});
it("enables supervisor LLM user messages by default with env override", () => {
expect(getSupervisorLlmResponseEnabled()).toBe(true);
vi.stubEnv("AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED", "false");
expect(getSupervisorLlmResponseEnabled()).toBe(false);
});
it("reads operational excellence guardrail configuration", () => {
expect(getAgentBudgetLlmCostEnabled()).toBe(true);
expect(getAgentCostAwareTierEnabled()).toBe(true);
expect(getAgentSupervisorFastPathEnabled()).toBe(true);
expect(getSupervisorFastPathConfidence()).toBe(0.85);
expect(getAgentToolSandboxEnabled()).toBe(false);
expect(getToolAuditMaxLen()).toBe(100_000);
expect(getNatsAclMode()).toBe("log-only");
vi.stubEnv("AGENT_BUDGET_LLM_COST_ENABLED", "false");
vi.stubEnv("AGENT_COST_AWARE_TIER_ENABLED", "false");
vi.stubEnv("AGENT_SUPERVISOR_FAST_PATH_ENABLED", "false");
vi.stubEnv("SUPERVISOR_FAST_PATH_CONFIDENCE", "0.9");
vi.stubEnv("AGENT_TOOL_SANDBOX_ENABLED", "true");
vi.stubEnv("AGENT_TOOL_AUDIT_MAX_LEN", "42");
vi.stubEnv("AGENT_NATS_ACL_MODE", "enforce");
expect(getAgentBudgetLlmCostEnabled()).toBe(false);
expect(getAgentCostAwareTierEnabled()).toBe(false);
expect(getAgentSupervisorFastPathEnabled()).toBe(false);
expect(getSupervisorFastPathConfidence()).toBe(0.9);
expect(getAgentToolSandboxEnabled()).toBe(true);
expect(getToolAuditMaxLen()).toBe(42);
expect(getNatsAclMode()).toBe("enforce");
});
it("reads eval case deadline configuration", () => {
expect(getEvalCaseDeadlineMs()).toBe(4_500);
vi.stubEnv("EVAL_CASE_DEADLINE_MS", "45000");
expect(getEvalCaseDeadlineMs()).toBe(45_000);
});
it("enables eval runtime LLM by default with env override", () => {
expect(getEvalRuntimeLlmEnabled()).toBe(true);
vi.stubEnv("EVAL_RUNTIME_LLM_ENABLED", "false");
expect(getEvalRuntimeLlmEnabled()).toBe(false);
});
it("pins eval LLM provider to cliproxy by default", () => {
expect(getEvalLlmProvider()).toBe("cliproxy");
vi.stubEnv("EVAL_LLM_PROVIDER", "all");
expect(getEvalLlmProvider()).toBe("all");
vi.stubEnv("EVAL_LLM_PROVIDER", "openrouter");
expect(getEvalLlmProvider()).toBe("openrouter");
vi.stubEnv("EVAL_LLM_PROVIDER", "invalid");
expect(getEvalLlmProvider()).toBe("cliproxy");
});
it("reads provider-scoped runtime LLM models", () => {
vi.stubEnv("CLIPROXY_TIER1_PLANNER_MODEL", "cliproxy-planner");
vi.stubEnv(
"OPENROUTER_TIER1_RESPONDER_MODEL",
"openrouter-fast, openrouter-backup",
);
expect(
getRuntimeLlmModelsForProvider(
"cliproxy",
{ role: "planner", tier: 1 },
"local-qwen",
),
).toEqual(["cliproxy-planner"]);
expect(
getRuntimeLlmModelsForProvider(
"openrouter",
{ role: "responder", tier: 1 },
"local-qwen",
),
).toEqual(["openrouter-fast", "openrouter-backup"]);
expect(
getRuntimeLlmModelsForProvider(
"llama",
{ role: "responder", tier: 1 },
"local-qwen",
),
).toEqual(["local-qwen"]);
});
it("keeps model round-robin disabled by default", () => {
expect(getLlmModelRoundRobinEnabled()).toBe(false);
vi.stubEnv("LLM_MODEL_ROUND_ROBIN_ENABLED", "true");
expect(getLlmModelRoundRobinEnabled()).toBe(true);
});
});
+30 -371
View File
@@ -7,14 +7,20 @@ type LogLevel =
| "trace" | "trace"
| "silent"; | "silent";
function getPositiveNumber(envKey: string, fallback: number): number { function getPositiveNumber(
envKey: string,
fallback: number,
): number {
const raw = process.env[envKey]?.trim(); const raw = process.env[envKey]?.trim();
if (!raw) return fallback; if (!raw) return fallback;
const parsed = Number(raw); const parsed = Number(raw);
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
} }
function getRatioNumber(envKey: string, fallback: number): number { function getRatioNumber(
envKey: string,
fallback: number,
): number {
const raw = process.env[envKey]?.trim(); const raw = process.env[envKey]?.trim();
if (!raw) return fallback; if (!raw) return fallback;
const parsed = Number(raw); const parsed = Number(raw);
@@ -23,52 +29,6 @@ function getRatioNumber(envKey: string, fallback: number): number {
: fallback; : fallback;
} }
function normalizeSkillEnvSuffix(
skill: string | undefined,
): string | undefined {
const suffix = skill
?.trim()
.toUpperCase()
.replace(/[^A-Z0-9]/gu, "_");
return suffix || undefined;
}
function getPositiveNumberForSkill(
envKey: string,
skill: string | undefined,
fallback: number,
): number {
const skillSuffix = normalizeSkillEnvSuffix(skill);
if (skillSuffix) {
const raw = process.env[`${envKey}_${skillSuffix}`]?.trim();
if (raw) {
const parsed = Number(raw);
if (Number.isFinite(parsed) && parsed > 0) {
return parsed;
}
}
}
return getPositiveNumber(envKey, fallback);
}
function getRatioNumberForSkill(
envKey: string,
skill: string | undefined,
fallback: number,
): number {
const skillSuffix = normalizeSkillEnvSuffix(skill);
if (skillSuffix) {
const raw = process.env[`${envKey}_${skillSuffix}`]?.trim();
if (raw) {
const parsed = Number(raw);
if (Number.isFinite(parsed) && parsed >= 0 && parsed <= 1) {
return parsed;
}
}
}
return getRatioNumber(envKey, fallback);
}
function getBoolean(envKey: string, fallback: boolean): boolean { function getBoolean(envKey: string, fallback: boolean): boolean {
const raw = process.env[envKey]?.trim().toLowerCase(); const raw = process.env[envKey]?.trim().toLowerCase();
if (!raw) return fallback; if (!raw) return fallback;
@@ -77,23 +37,6 @@ function getBoolean(envKey: string, fallback: boolean): boolean {
return fallback; return fallback;
} }
function getStringListFromEnv(...envKeys: string[]): string[] | undefined {
for (const envKey of envKeys) {
const raw = process.env[envKey]?.trim();
if (!raw) continue;
const values = raw
.split(/[\n,]/u)
.map((value) => value.trim())
.filter((value) => value.length > 0);
if (values.length > 0) {
return values;
}
}
return undefined;
}
export function getNodeEnv(): "dev" | "test" | "prod" { export function getNodeEnv(): "dev" | "test" | "prod" {
const value = process.env["NODE_ENV"]?.trim(); const value = process.env["NODE_ENV"]?.trim();
return value === "test" || value === "prod" ? value : "dev"; return value === "test" || value === "prod" ? value : "dev";
@@ -138,79 +81,6 @@ export function getOtelExporter(): "stdout" | "none" {
return getNodeEnv() === "dev" ? "none" : "stdout"; return getNodeEnv() === "dev" ? "none" : "stdout";
} }
export function getPhoenixOtlpEndpoint(): string | undefined {
return process.env["PHOENIX_OTLP_ENDPOINT"]?.trim() || undefined;
}
export function getPhoenixLogPrompts(): boolean {
return getBoolean("PHOENIX_LOG_PROMPTS", false);
}
export function getRagEnabled(): boolean {
return getBoolean("AGENT_RAG_ENABLED", true);
}
export function getRagHybridEnabled(): boolean {
return getBoolean("AGENT_RAG_HYBRID_ENABLED", true);
}
export function getRagRerankerEnabled(): boolean {
return getBoolean("AGENT_RAG_RERANKER_ENABLED", true);
}
export function getRagRerankerTopK(): number {
return getPositiveNumber("AGENT_RAG_RERANKER_TOP_K", 5);
}
export function getRagCandidatePoolSize(): number {
return getPositiveNumber("AGENT_RAG_CANDIDATE_POOL_SIZE", 50);
}
export function getLongTermSummaryEnabled(): boolean {
return getBoolean("AGENT_LONG_TERM_SUMMARY_ENABLED", true);
}
export function getMemoryEvictionEnabled(): boolean {
return getBoolean("AGENT_MEMORY_EVICTION_ENABLED", false);
}
export function getEmbeddingProvider(): "openrouter" | "openai" {
const raw = process.env["EMBEDDING_PROVIDER"]?.trim().toLowerCase();
return raw === "openai" ? "openai" : "openrouter";
}
export function getEmbeddingModel(provider = getEmbeddingProvider()): string {
const configured = process.env["EMBEDDING_MODEL"]?.trim();
if (configured) {
return configured;
}
return provider === "openai"
? "text-embedding-3-small"
: "openai/text-embedding-3-small";
}
export function getEmbeddingApiUrl(provider = getEmbeddingProvider()): string {
const configured = process.env["EMBEDDING_API_URL"]?.trim();
if (configured) {
return configured;
}
return provider === "openai"
? "https://api.openai.com/v1"
: "https://openrouter.ai/api/v1";
}
export function getEmbeddingDimensions(): number {
return getPositiveNumber("EMBEDDING_DIMENSIONS", 1536);
}
export function getEmbeddingRequestTimeoutMs(): number {
return getPositiveNumber("EMBEDDING_REQUEST_TIMEOUT_MS", 30_000);
}
export function getKnowledgeSearchCandidateLimit(): number {
return getPositiveNumber("KNOWLEDGE_SEARCH_CANDIDATE_LIMIT", 500);
}
export function getRequestSubject(): string { export function getRequestSubject(): string {
return ( return (
process.env["AGENT_REQUEST_SUBJECT"]?.trim() || "lolympus-agent.request" process.env["AGENT_REQUEST_SUBJECT"]?.trim() || "lolympus-agent.request"
@@ -228,66 +98,6 @@ export function getAgentRequestTimeoutMs(): number {
return getPositiveNumber("AGENT_REQUEST_TIMEOUT_MS", 5_000); return getPositiveNumber("AGENT_REQUEST_TIMEOUT_MS", 5_000);
} }
export function getAgentBudgetEnabled(): boolean {
return getBoolean("AGENT_BUDGET_ENABLED", true);
}
export function getAgentBudgetMaxHandoffs(): number {
return getPositiveNumber("AGENT_BUDGET_MAX_HANDOFFS", 4);
}
export function getAgentBudgetMaxSameAgentVisits(): number {
return getPositiveNumber("AGENT_BUDGET_MAX_SAME_AGENT_VISITS", 2);
}
export function getAgentBudgetMaxToolCalls(): number {
return getPositiveNumber("AGENT_BUDGET_MAX_TOOL_CALLS", 20);
}
export function getAgentBudgetMaxCostUsd(): number {
return getPositiveNumber("AGENT_BUDGET_MAX_COST_USD", 0.5);
}
export function getAgentBudgetLlmCostEnabled(): boolean {
return getBoolean("AGENT_BUDGET_LLM_COST_ENABLED", true);
}
export function getAgentCostAwareTierEnabled(): boolean {
return getBoolean("AGENT_COST_AWARE_TIER_ENABLED", true);
}
export function getAgentSupervisorFastPathEnabled(): boolean {
return getBoolean("AGENT_SUPERVISOR_FAST_PATH_ENABLED", true);
}
export function getSupervisorFastPathConfidence(): number {
return getRatioNumber("SUPERVISOR_FAST_PATH_CONFIDENCE", 0.85);
}
export function getAgentToolSandboxEnabled(): boolean {
return getBoolean("AGENT_TOOL_SANDBOX_ENABLED", false);
}
export function getToolAuditMaxLen(): number {
return getPositiveNumber("AGENT_TOOL_AUDIT_MAX_LEN", 100_000);
}
export function getNatsAclMode(): "off" | "log-only" | "enforce" {
const raw = process.env["AGENT_NATS_ACL_MODE"]?.trim().toLowerCase();
if (raw === "off" || raw === "log-only" || raw === "enforce") {
return raw;
}
return "log-only";
}
export function getAgentRuntimeVerifierEnabled(): boolean {
return getBoolean("AGENT_RUNTIME_VERIFIER_ENABLED", false);
}
export function getMetricsPort(): number {
return getPositiveNumber("METRICS_PORT", 9464);
}
export function getDiscordVerifyTimeoutMs(): number { export function getDiscordVerifyTimeoutMs(): number {
return getPositiveNumber("DISCORD_VERIFY_TIMEOUT_MS", 2_500); return getPositiveNumber("DISCORD_VERIFY_TIMEOUT_MS", 2_500);
} }
@@ -380,50 +190,6 @@ export function getEvalReviewDedupeTtlSec(): number {
return getPositiveNumber("EVAL_REVIEW_DEDUPE_TTL_SEC", 604_800); return getPositiveNumber("EVAL_REVIEW_DEDUPE_TTL_SEC", 604_800);
} }
export function getSupervisorReviewQueueMaxLen(): number {
return getPositiveNumber("SUPERVISOR_REVIEW_QUEUE_MAXLEN", 10_000);
}
export function getSupervisorReviewResolutionTtlSec(): number {
return getPositiveNumber(
"SUPERVISOR_REVIEW_RESOLUTION_TTL_SEC",
90 * 24 * 60 * 60,
);
}
export function getSupervisorResolutionsStreamMaxLen(): number {
return getPositiveNumber("SUPERVISOR_RESOLUTIONS_STREAM_MAXLEN", 100_000);
}
export function getSupervisorLlmEnabled(): boolean {
return getBoolean("AGENT_SUPERVISOR_LLM_ENABLED", true);
}
export function getSupervisorLlmConfidenceThreshold(): number {
return getRatioNumber("AGENT_SUPERVISOR_LLM_CONFIDENCE_THRESHOLD", 0.6);
}
export function getSupervisorLlmResponseEnabled(): boolean {
return getBoolean("AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED", true);
}
export type EvalLlmProvider = "all" | "cliproxy" | "llama" | "openrouter";
const EVAL_LLM_PROVIDERS = new Set<EvalLlmProvider>([
"all",
"cliproxy",
"llama",
"openrouter",
]);
export function getEvalLlmProvider(): EvalLlmProvider {
const raw = process.env["EVAL_LLM_PROVIDER"]?.trim().toLowerCase();
if (raw && EVAL_LLM_PROVIDERS.has(raw as EvalLlmProvider)) {
return raw as EvalLlmProvider;
}
return "cliproxy";
}
export function getEvalCalibrationMinCorr(): number { export function getEvalCalibrationMinCorr(): number {
return getRatioNumber("EVAL_CALIBRATION_MIN_CORR", 0.5); return getRatioNumber("EVAL_CALIBRATION_MIN_CORR", 0.5);
} }
@@ -476,11 +242,7 @@ export function getAgentSkillMdMode(): boolean {
return getBoolean("AGENT_SKILL_MD_MODE", false); return getBoolean("AGENT_SKILL_MD_MODE", false);
} }
export function getAgentRouterMode(): export function getAgentRouterMode(): "off" | "static" | "learned-v1" | "learned-v2" {
| "off"
| "static"
| "learned-v1"
| "learned-v2" {
const raw = process.env["AGENT_ROUTER_MODE"]?.trim(); const raw = process.env["AGENT_ROUTER_MODE"]?.trim();
if ( if (
raw === "off" || raw === "off" ||
@@ -549,10 +311,6 @@ export function getLlmCircuitCooldownMs(): number {
return getPositiveNumber("LLM_CIRCUIT_COOLDOWN_MS", 60_000); return getPositiveNumber("LLM_CIRCUIT_COOLDOWN_MS", 60_000);
} }
export function getLlmModelRoundRobinEnabled(): boolean {
return getBoolean("LLM_MODEL_ROUND_ROBIN_ENABLED", false);
}
export function getClassifierMaxTokens(): number { export function getClassifierMaxTokens(): number {
return getPositiveNumber("CLASSIFIER_MAX_TOKENS", 16); return getPositiveNumber("CLASSIFIER_MAX_TOKENS", 16);
} }
@@ -589,14 +347,6 @@ export function getEvalPrmMinLabels(): number {
return getPositiveNumber("EVAL_PRM_MIN_LABELS", 500); return getPositiveNumber("EVAL_PRM_MIN_LABELS", 500);
} }
export function getEvalCaseDeadlineMs(): number {
return getPositiveNumber("EVAL_CASE_DEADLINE_MS", 4_500);
}
export function getEvalRuntimeLlmEnabled(): boolean {
return getBoolean("EVAL_RUNTIME_LLM_ENABLED", true);
}
export function getEvalAttributionTickMs(): number { export function getEvalAttributionTickMs(): number {
return getPositiveNumber("EVAL_ATTRIBUTION_TICK_MS", 6 * 60 * 60 * 1000); return getPositiveNumber("EVAL_ATTRIBUTION_TICK_MS", 6 * 60 * 60 * 1000);
} }
@@ -614,9 +364,7 @@ export function getOpenRouterEnabled(): boolean {
} }
export function getOpenRouterApiUrl(): string { export function getOpenRouterApiUrl(): string {
return ( return process.env["OPENROUTER_API_URL"]?.trim() || "https://openrouter.ai/api/v1";
process.env["OPENROUTER_API_URL"]?.trim() || "https://openrouter.ai/api/v1"
);
} }
export function getTier0Model(): string { export function getTier0Model(): string {
@@ -652,91 +400,15 @@ export function getTier2CriticModel(): string {
} }
export function getOpenRouterModels(): string[] { export function getOpenRouterModels(): string[] {
return ( const raw = process.env["OPENROUTER_MODEL"]?.trim();
getStringListFromEnv("OPENROUTER_MODEL", "OPENROUTER_MODELS") ?? [ if (!raw) {
"openai/gpt-5.3-codex-spark", return ["openai/gpt-5.3-codex-spark"];
]
);
}
type RuntimeLlmProvider = "cliproxy" | "llama" | "openrouter";
type RuntimeLlmRole = "critic" | "planner" | "responder" | undefined;
type RuntimeLlmTier = 0 | 1 | 2 | undefined;
function getRuntimeRouteSuffix(args: {
role?: RuntimeLlmRole;
tier?: RuntimeLlmTier;
}): string {
if (args.role === "planner") {
return "TIER1_PLANNER";
} }
if (args.role === "critic") { const models = raw
return "TIER2_CRITIC"; .split(",")
} .map((value) => value.trim())
if (args.tier === 0) { .filter((value) => value.length > 0);
return "TIER0"; return models.length > 0 ? models : ["openai/gpt-5.3-codex-spark"];
}
return "TIER1_RESPONDER";
}
function getProviderPrefix(provider: RuntimeLlmProvider): string {
return provider.toUpperCase();
}
function getProviderDefaultModels(
provider: RuntimeLlmProvider,
llamaFallbackModel: string,
): string[] {
if (provider === "cliproxy") {
return (
getStringListFromEnv(
"CLIPROXY_MODEL",
"CLIPROXY_MODELS",
"CLI_PROXY_LLM_MODEL",
) ?? ["gpt-5.3-codex-spark"]
);
}
if (provider === "openrouter") {
return getOpenRouterModels();
}
return (
getStringListFromEnv("LLAMA_MODEL", "LLAMA_MODELS", "AGENT_LLM_MODEL") ?? [
llamaFallbackModel,
]
);
}
export function getRuntimeLlmModelsForProvider(
provider: RuntimeLlmProvider,
args: {
role?: RuntimeLlmRole;
tier?: RuntimeLlmTier;
},
llamaFallbackModel: string,
): string[] {
const suffix = getRuntimeRouteSuffix(args);
const providerPrefix = getProviderPrefix(provider);
const providerScopedModels = getStringListFromEnv(
`${providerPrefix}_${suffix}_MODEL`,
`${providerPrefix}_${suffix}_MODELS`,
);
if (providerScopedModels) {
return providerScopedModels;
}
if (provider === "cliproxy") {
const legacyModels = getStringListFromEnv(
`${suffix}_MODEL`,
`${suffix}_MODELS`,
);
if (legacyModels) {
return legacyModels;
}
}
return getProviderDefaultModels(provider, llamaFallbackModel);
} }
export function getLlmMaxTokens(): number { export function getLlmMaxTokens(): number {
@@ -809,42 +481,29 @@ export function getShadowEvalSamplingRate(tier: 0 | 1 | 2): number {
getNodeEnv() === "prod" ? 0.5 : 1.0, getNodeEnv() === "prod" ? 0.5 : 1.0,
); );
case 2: case 2:
return getRatioNumber("SHADOW_EVAL_SAMPLING_RATE_TIER2", 1.0); return getRatioNumber(
"SHADOW_EVAL_SAMPLING_RATE_TIER2",
1.0,
);
} }
} }
export function getAlertThresholdP95LatencyMs(skill?: string): number { export function getAlertThresholdP95LatencyMs(): number {
return getPositiveNumberForSkill( return getPositiveNumber("ALERT_THRESHOLD_P95_LATENCY_MS", 3_000);
"ALERT_THRESHOLD_P95_LATENCY_MS",
skill,
3_000,
);
} }
export function getAlertThresholdEvalScoreDropPct(skill?: string): number { export function getAlertThresholdEvalScoreDropPct(): number {
return getPositiveNumberForSkill( return getPositiveNumber("ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT", 10);
"ALERT_THRESHOLD_EVAL_SCORE_DROP_PCT",
skill,
10,
);
} }
export function getAlertThresholdLlmErrorRate(skill?: string): number { export function getAlertThresholdLlmErrorRate(): number {
return getRatioNumberForSkill("ALERT_THRESHOLD_LLM_ERROR_RATE", skill, 0.1); return getRatioNumber("ALERT_THRESHOLD_LLM_ERROR_RATE", 0.1);
} }
export function getAlertThresholdDriftPct(skill?: string): number { export function getAlertThresholdDriftPct(): number {
return getPositiveNumberForSkill("ALERT_THRESHOLD_DRIFT_PCT", skill, 15); return getPositiveNumber("ALERT_THRESHOLD_DRIFT_PCT", 15);
} }
export function getAlertRateLimitSec(): number { export function getAlertRateLimitSec(): number {
return getPositiveNumber("ALERT_RATE_LIMIT_SEC", 300); return getPositiveNumber("ALERT_RATE_LIMIT_SEC", 300);
} }
export function getSlackAlertWebhookUrl(): string | undefined {
return process.env["SLACK_ALERT_WEBHOOK_URL"]?.trim() || undefined;
}
export function getSlackWebhookTimeoutMs(): number {
return getPositiveNumber("SLACK_WEBHOOK_TIMEOUT_MS", 3_000);
}
+6 -36
View File
@@ -9,7 +9,6 @@ export const EVAL_AGG_CURSOR_KEY = "lolympus-agent:eval:agg:cursor";
const EVAL_AGG_SLICE_KEY_PREFIX = "lolympus-agent:eval:agg:"; const EVAL_AGG_SLICE_KEY_PREFIX = "lolympus-agent:eval:agg:";
export interface SliceKey { export interface SliceKey {
skill?: string | undefined;
tier: 0 | 1 | 2; tier: 0 | 1 | 2;
intent: string; intent: string;
phase: string; phase: string;
@@ -112,14 +111,6 @@ function getNestedString(
return typeof value === "string" && value.trim() ? value : null; return typeof value === "string" && value.trim() ? value : null;
} }
function getStringField(
record: Record<string, unknown>,
key: string,
): string | null {
const value = record[key];
return typeof value === "string" && value.trim() ? value : null;
}
function extractLlmOverall(message: Record<string, string>): number | null { function extractLlmOverall(message: Record<string, string>): number | null {
const direct = parseFiniteNumber(message["llmOverall"]); const direct = parseFiniteNumber(message["llmOverall"]);
if (direct !== null) { if (direct !== null) {
@@ -186,13 +177,9 @@ function parseObservation(entry: StreamEntry): EvalResultObservation | null {
const tier = parseTier(entry.message["tier"]) ?? parseTier(derived["tier"]); const tier = parseTier(entry.message["tier"]) ?? parseTier(derived["tier"]);
const intent = const intent =
entry.message["intent"]?.trim() || entry.message["intent"]?.trim() ||
getStringField(derived, "intent") ||
getNestedString(derived, "intent", "kind"); getNestedString(derived, "intent", "kind");
const phase = const phase =
entry.message["phase"]?.trim() || entry.message["phase"]?.trim() || getNestedString(derived, "phase", "kind");
getStringField(derived, "phase") ||
getNestedString(derived, "phase", "kind");
const skill = entry.message["skill"]?.trim();
if (tier === null || !intent || !phase) { if (tier === null || !intent || !phase) {
return null; return null;
@@ -207,7 +194,6 @@ function parseObservation(entry: StreamEntry): EvalResultObservation | null {
llmOverall: extractLlmOverall(entry.message), llmOverall: extractLlmOverall(entry.message),
processJudgeScore: extractProcessJudgeScore(entry.message), processJudgeScore: extractProcessJudgeScore(entry.message),
slice: { slice: {
...(skill ? { skill } : {}),
tier, tier,
intent, intent,
phase, phase,
@@ -355,28 +341,17 @@ function average(sum: number, count: number): number | null {
function sliceKeyFromString(value: string): SliceKey | null { function sliceKeyFromString(value: string): SliceKey | null {
const parts = value.split(":"); const parts = value.split(":");
let skill: string | undefined; const [tierRaw, intent, phase] = parts;
let offset = 0;
let tierRaw = parts[0];
if (parseTier(tierRaw) === null && parts.length >= 4) {
skill = parts[0];
offset = 1;
tierRaw = parts[offset];
}
const intent = parts[offset + 1];
const phase = parts[offset + 2];
const tier = parseTier(tierRaw); const tier = parseTier(tierRaw);
if (tier === null || !intent || !phase) { if (tier === null || !intent || !phase) {
return null; return null;
} }
const tail = parts.slice(offset + 3); const routingDecision = parts.length >= 6 ? parts[3] : undefined;
const routingDecision = tail.length >= 3 ? tail[0] : undefined; const abstentionAction = parts.length >= 6 ? parts[4] : parts[3];
const abstentionAction = tail.length >= 3 ? tail[1] : tail[0]; const abstentionReason = parts.length >= 6 ? parts[5] : parts[4];
const abstentionReason = tail.length >= 3 ? tail[2] : tail[1];
return { return {
...(skill ? { skill } : {}),
tier, tier,
intent, intent,
phase, phase,
@@ -401,12 +376,7 @@ function sliceKeyFromRedisKey(key: string): SliceKey | null {
} }
export function sliceKeyToString(key: SliceKey): string { export function sliceKeyToString(key: SliceKey): string {
const base = [ const base = `${key.tier}:${key.intent}:${key.phase}`;
...(key.skill ? [key.skill] : []),
String(key.tier),
key.intent,
key.phase,
].join(":");
if (!key.routingDecision && !key.abstentionAction && !key.abstentionReason) { if (!key.routingDecision && !key.abstentionAction && !key.abstentionReason) {
return base; return base;
} }
+9 -53
View File
@@ -17,19 +17,17 @@ import {
type SliceKey, type SliceKey,
type SliceReportRow, type SliceReportRow,
} from "./aggregator.js"; } from "./aggregator.js";
import type { EvalClassifiedSnapshot, EvalDerivedSnapshot } from "./types.js";
export const EVAL_ARCHIVE_CURSOR_KEY = "lolympus-agent:eval:archive:cursor"; export const EVAL_ARCHIVE_CURSOR_KEY = "lolympus-agent:eval:archive:cursor";
export const EVAL_ARCHIVE_COLLECTION = "eval_archive"; export const EVAL_ARCHIVE_COLLECTION = "eval_archive";
export interface EvalArchiveEntry { export interface EvalArchiveEntry {
streamId: string; streamId: string;
skill?: string | undefined;
evaluatedAt: Date; evaluatedAt: Date;
request: AgentRequest; request: AgentRequest;
response: AgentResponse; response: AgentResponse;
derived: DerivedState | EvalDerivedSnapshot; derived: DerivedState;
classified: ClassifiedIntent | EvalClassifiedSnapshot; classified: ClassifiedIntent;
validation: ValidationResult; validation: ValidationResult;
llmJudge: LlmJudgeResult; llmJudge: LlmJudgeResult;
processJudge?: ProcessJudgeResult | undefined; processJudge?: ProcessJudgeResult | undefined;
@@ -129,44 +127,6 @@ function getAbstention(
}; };
} }
function getStringField(
record: Record<string, unknown>,
key: string,
): string | null {
const value = record[key];
return typeof value === "string" && value.trim() ? value : null;
}
function getNestedString(
record: Record<string, unknown>,
first: string,
second: string,
): string | null {
const nested = asRecord(record[first]);
return getStringField(nested, second);
}
export function getArchiveDerivedIntent(derived: EvalArchiveEntry["derived"]): string {
const record = asRecord(derived);
return getStringField(record, "intent") ??
getNestedString(record, "intent", "kind") ??
"unknown";
}
export function getArchiveDerivedPhase(derived: EvalArchiveEntry["derived"]): string {
const record = asRecord(derived);
return getStringField(record, "phase") ??
getNestedString(record, "phase", "kind") ??
"unknown";
}
export function getArchiveRoutingDecision(
derived: EvalArchiveEntry["derived"],
): string | null {
const record = asRecord(derived);
return getNestedString(record, "routingDecision", "kind");
}
function parseEvaluatedAt(value: string | undefined, nowMs: number): Date { function parseEvaluatedAt(value: string | undefined, nowMs: number): Date {
if (!value?.trim()) { if (!value?.trim()) {
return new Date(nowMs); return new Date(nowMs);
@@ -204,13 +164,13 @@ function calculatePercentile(values: number[], percentile: number): number | nul
function getSlice(entry: EvalArchiveEntry): SliceKey { function getSlice(entry: EvalArchiveEntry): SliceKey {
const abstention = getAbstention(entry); const abstention = getAbstention(entry);
const routingDecision = getArchiveRoutingDecision(entry.derived);
return { return {
skill: entry.skill || entry.request.skill || "onboarding",
tier: entry.tier, tier: entry.tier,
intent: getArchiveDerivedIntent(entry.derived), intent: entry.derived.intent.kind,
phase: getArchiveDerivedPhase(entry.derived), phase: entry.derived.phase.kind,
...(routingDecision ? { routingDecision } : {}), ...(entry.derived.routingDecision
? { routingDecision: entry.derived.routingDecision.kind }
: {}),
...(abstention.action ? { abstentionAction: abstention.action } : {}), ...(abstention.action ? { abstentionAction: abstention.action } : {}),
...(abstention.reason ? { abstentionReason: abstention.reason } : {}), ...(abstention.reason ? { abstentionReason: abstention.reason } : {}),
}; };
@@ -232,7 +192,7 @@ function parseEvalArchiveEntry(
entry.message["input"], entry.message["input"],
"input", "input",
); );
const derived = parseJsonField<DerivedState | EvalDerivedSnapshot>( const derived = parseJsonField<DerivedState>(
entry.message["derived"], entry.message["derived"],
"derived", "derived",
); );
@@ -248,12 +208,11 @@ function parseEvalArchiveEntry(
const archiveEntry: EvalArchiveEntry = { const archiveEntry: EvalArchiveEntry = {
streamId: entry.id, streamId: entry.id,
skill: entry.message["skill"]?.trim() || input.request.skill || "onboarding",
evaluatedAt: parseEvaluatedAt(entry.message["evaluatedAt"], nowMs), evaluatedAt: parseEvaluatedAt(entry.message["evaluatedAt"], nowMs),
request: input.request, request: input.request,
response: parseJsonField<AgentResponse>(entry.message["output"], "output"), response: parseJsonField<AgentResponse>(entry.message["output"], "output"),
derived, derived,
classified: parseJsonField<ClassifiedIntent | EvalClassifiedSnapshot>( classified: parseJsonField<ClassifiedIntent>(
entry.message["classified"], entry.message["classified"],
"classified", "classified",
), ),
@@ -289,12 +248,9 @@ async function ensureEvalArchiveIndexes(
{ expireAfterSeconds: getEvalArchiveTtlDays() * 24 * 60 * 60 }, { expireAfterSeconds: getEvalArchiveTtlDays() * 24 * 60 * 60 },
), ),
collection.createIndex({ collection.createIndex({
skill: 1,
tier: 1, tier: 1,
"derived.intent.kind": 1, "derived.intent.kind": 1,
"derived.intent": 1,
"derived.phase.kind": 1, "derived.phase.kind": 1,
"derived.phase": 1,
evaluatedAt: -1, evaluatedAt: -1,
}), }),
]); ]);
+1 -3
View File
@@ -14,7 +14,6 @@ import {
interface CaptureOptions { interface CaptureOptions {
limit?: number | undefined; limit?: number | undefined;
outputPath?: string | undefined; outputPath?: string | undefined;
skill?: string | undefined;
} }
function asRecord(value: unknown): Record<string, unknown> { function asRecord(value: unknown): Record<string, unknown> {
@@ -123,8 +122,7 @@ export async function captureGoldenSetScaffold(
const outputPath = resolve( const outputPath = resolve(
process.cwd(), process.cwd(),
options?.outputPath ?? options?.outputPath ?? "src/eval/golden/onboarding.capture.jsonl",
`src/eval/golden/${options?.skill ?? "onboarding"}/capture.jsonl`,
); );
await mkdir(dirname(outputPath), { recursive: true }); await mkdir(dirname(outputPath), { recursive: true });
await writeFile(outputPath, `${lines.join("\n")}\n`, "utf8"); await writeFile(outputPath, `${lines.join("\n")}\n`, "utf8");
-113
View File
@@ -1,113 +0,0 @@
export interface ParsedArgs {
command: string;
flags: Map<string, string | true>;
positionals: string[];
}
export function parsePositiveInteger(
value: string | undefined,
): number | undefined {
if (!value?.trim()) {
return undefined;
}
const parsed = Number(value);
return Number.isInteger(parsed) && parsed > 0 ? parsed : undefined;
}
export function parsePositiveNumber(
value: string | undefined,
): number | undefined {
if (!value?.trim()) {
return undefined;
}
const parsed = Number(value);
return Number.isFinite(parsed) && parsed > 0 ? parsed : undefined;
}
export function parseDurationMs(value: string | undefined): number {
const raw = value?.trim().toLowerCase() || "7d";
const match = raw.match(/^(\d+)(h|d)$/u);
if (!match) {
throw new Error("--since must use a duration like 24h, 7d, or 30d");
}
const amount = Number(match[1]);
const unit = match[2];
if (!Number.isInteger(amount) || amount <= 0) {
throw new Error("--since must be a positive duration");
}
return unit === "h"
? amount * 60 * 60 * 1000
: amount * 24 * 60 * 60 * 1000;
}
export function parseCsv(value: string | undefined): string[] {
if (!value?.trim()) {
return [];
}
return value
.split(",")
.map((item) => item.trim())
.filter((item) => item.length > 0);
}
export function parseArgs(argv: string[]): ParsedArgs {
const [command = "run", ...rest] = argv;
const flags = new Map<string, string | true>();
const positionals: string[] = [];
for (let index = 0; index < rest.length; index++) {
const token = rest[index];
if (!token) {
continue;
}
if (!token.startsWith("--")) {
positionals.push(token);
continue;
}
const withoutPrefix = token.slice(2);
const equalsIndex = withoutPrefix.indexOf("=");
if (equalsIndex >= 0) {
const key = withoutPrefix.slice(0, equalsIndex);
const value = withoutPrefix.slice(equalsIndex + 1);
flags.set(key, value || true);
continue;
}
const next = rest[index + 1];
if (next && !next.startsWith("--")) {
flags.set(withoutPrefix, next);
index += 1;
continue;
}
flags.set(withoutPrefix, true);
}
return {
command,
flags,
positionals,
};
}
export function getFlagString(
flags: Map<string, string | true>,
key: string,
): string | undefined {
const value = flags.get(key);
return typeof value === "string" ? value : undefined;
}
export function hasFlag(
flags: Map<string, string | true>,
key: string,
): boolean {
return flags.has(key);
}
+179 -134
View File
@@ -1,15 +1,51 @@
import type { MongoConnectionLike } from "./archive.js"; import type { MongoConnectionLike } from "./archive.js";
import type { SliceKey, SliceReportRow } from "./aggregator.js"; import type { SliceKey, SliceReportRow } from "./aggregator.js";
import type { GarnetClient } from "@common/garnet"; import type { GarnetClient } from "@common/garnet";
import {
getFlagString, function parsePositiveInteger(value: string | undefined): number | undefined {
hasFlag, if (!value?.trim()) {
parseArgs, return undefined;
parseCsv, }
parseDurationMs,
parsePositiveInteger, const parsed = Number(value);
parsePositiveNumber, return Number.isInteger(parsed) && parsed > 0 ? parsed : undefined;
} from "./cli-args.js"; }
function parsePositiveNumber(value: string | undefined): number | undefined {
if (!value?.trim()) {
return undefined;
}
const parsed = Number(value);
return Number.isFinite(parsed) && parsed > 0 ? parsed : undefined;
}
function parseDurationMs(value: string | undefined): number {
const raw = value?.trim().toLowerCase() || "7d";
const match = raw.match(/^(\d+)(h|d)$/u);
if (!match) {
throw new Error("--since must use a duration like 24h, 7d, or 30d");
}
const amount = Number(match[1]);
const unit = match[2];
if (!Number.isInteger(amount) || amount <= 0) {
throw new Error("--since must be a positive duration");
}
return unit === "h" ? amount * 60 * 60 * 1000 : amount * 24 * 60 * 60 * 1000;
}
function parseCsv(value: string | undefined): string[] {
if (!value?.trim()) {
return [];
}
return value
.split(",")
.map((item) => item.trim())
.filter((item) => item.length > 0);
}
function parseSlice(value: string | undefined): SliceKey { function parseSlice(value: string | undefined): SliceKey {
if (!value?.trim()) { if (!value?.trim()) {
@@ -29,7 +65,6 @@ function parseSlice(value: string | undefined): SliceKey {
const tier = Number(fields["tier"]); const tier = Number(fields["tier"]);
const intent = fields["intent"]; const intent = fields["intent"];
const phase = fields["phase"]; const phase = fields["phase"];
const skill = fields["skill"]?.trim();
if ( if (
(tier !== 0 && tier !== 1 && tier !== 2) || (tier !== 0 && tier !== 1 && tier !== 2) ||
@@ -42,44 +77,99 @@ function parseSlice(value: string | undefined): SliceKey {
} }
return { return {
...(skill ? { skill } : {}),
tier, tier,
intent, intent,
phase, phase,
}; };
} }
function printRunSummary( interface ParsedArgs {
report: { command: string;
generatedAt: string; flags: Map<string, string | true>;
summary: { positionals: string[];
cases: number; }
passRate: number;
ruleAverageScore: number; function parseArgs(argv: string[]): ParsedArgs {
averageLatencyMs: number; const [command = "run", ...rest] = argv;
llmAverageOverall: number | null; const flags = new Map<string, string | true>();
llmJudgedCases: number; const positionals: string[] = [];
};
baselineDelta?: for (let index = 0; index < rest.length; index++) {
| { const token = rest[index];
passRate: number | null; if (!token) {
ruleAverageScore: number | null; continue;
llmAverageOverall: number | null; }
averageLatencyMs: number | null;
} if (!token.startsWith("--")) {
| undefined; positionals.push(token);
}, continue;
reportPath: string, }
): void {
const withoutPrefix = token.slice(2);
const equalsIndex = withoutPrefix.indexOf("=");
if (equalsIndex >= 0) {
const key = withoutPrefix.slice(0, equalsIndex);
const value = withoutPrefix.slice(equalsIndex + 1);
flags.set(key, value || true);
continue;
}
const next = rest[index + 1];
if (next && !next.startsWith("--")) {
flags.set(withoutPrefix, next);
index += 1;
continue;
}
flags.set(withoutPrefix, true);
}
return {
command,
flags,
positionals,
};
}
function getFlagString(
flags: Map<string, string | true>,
key: string,
): string | undefined {
const value = flags.get(key);
return typeof value === "string" ? value : undefined;
}
function hasFlag(
flags: Map<string, string | true>,
key: string,
): boolean {
return flags.has(key);
}
function printRunSummary(report: {
generatedAt: string;
summary: {
cases: number;
passRate: number;
ruleAverageScore: number;
averageLatencyMs: number;
llmAverageOverall: number | null;
llmJudgedCases: number;
};
baselineDelta?: {
passRate: number | null;
ruleAverageScore: number | null;
llmAverageOverall: number | null;
averageLatencyMs: number | null;
} | undefined;
}, reportPath: string): void {
const llmLine = const llmLine =
report.summary.llmAverageOverall === null report.summary.llmAverageOverall === null
? "n/a" ? "n/a"
: `${report.summary.llmAverageOverall} (${report.summary.llmJudgedCases} judged)`; : `${report.summary.llmAverageOverall} (${report.summary.llmJudgedCases} judged)`;
console.info(`[eval] generatedAt=${report.generatedAt}`); console.info(`[eval] generatedAt=${report.generatedAt}`);
console.info( console.info(`[eval] cases=${report.summary.cases} passRate=${report.summary.passRate}`);
`[eval] cases=${report.summary.cases} passRate=${report.summary.passRate}`,
);
console.info( console.info(
`[eval] ruleAverageScore=${report.summary.ruleAverageScore} averageLatencyMs=${report.summary.averageLatencyMs}`, `[eval] ruleAverageScore=${report.summary.ruleAverageScore} averageLatencyMs=${report.summary.averageLatencyMs}`,
); );
@@ -229,43 +319,39 @@ async function main(): Promise<void> {
parsePositiveInteger(getFlagString(flags, "limit")) ?? parsePositiveInteger(getFlagString(flags, "limit")) ??
parsePositiveInteger(positionals[0]), parsePositiveInteger(positionals[0]),
outputPath: getFlagString(flags, "output") ?? positionals[1], outputPath: getFlagString(flags, "output") ?? positionals[1],
skill: getFlagString(flags, "skill"),
}); });
logger.info({ outputPath }, '[eval] "capture" completed'); logger.info({ outputPath }, '[eval] "capture" completed');
return; return;
} }
if (command === "run") { if (command === "run") {
const { loadEvalCases, resolveEvalSetPath, runEval } = const { loadEvalCases, resolveEvalSetPath, runOnboardingEval } = await import(
await import("./runner.js"); "./runner.js"
const { publishEvalReportIfPossible } = await import("./runner.js"); );
const setName = getFlagString(flags, "set") ?? "onboarding/cases"; const { publishEvalReportIfPossible } = await import(
const resolvedSet = resolveEvalSetPath(setName); "./runner.js"
const skill = getFlagString(flags, "skill") ?? resolvedSet.skill; );
const inputPath = getFlagString(flags, "input") ?? resolvedSet.path; const setName = getFlagString(flags, "set") ?? "golden";
const inputPath = getFlagString(flags, "input") ?? resolveEvalSetPath(setName);
const limit = parsePositiveInteger(getFlagString(flags, "limit")); const limit = parsePositiveInteger(getFlagString(flags, "limit"));
if (hasFlag(flags, "dry-run")) { if (hasFlag(flags, "dry-run")) {
const cases = await loadEvalCases(inputPath, { limit }); const cases = await loadEvalCases(inputPath, { limit });
console.info( console.info(`[eval] dry-run parsed ${cases.length} cases from ${inputPath}`);
`[eval] dry-run skill=${skill} parsed ${cases.length} cases from ${inputPath}`,
);
return; return;
} }
const { report, reportPath } = await runEval({ const { report, reportPath } = await runOnboardingEval({
enableLlmJudge: !hasFlag(flags, "skip-llm-judge"), enableLlmJudge: !hasFlag(flags, "skip-llm-judge"),
inputPath, inputPath,
limit, limit,
outputPath: getFlagString(flags, "output"), outputPath: getFlagString(flags, "output"),
setName, setName,
skill,
}); });
const publishedReport = hasFlag(flags, "no-publish-report") const publishedReport = hasFlag(flags, "no-publish-report")
? false ? false
: await publishEvalReportIfPossible(report); : await publishEvalReportIfPossible(report);
printRunSummary(report, reportPath); printRunSummary(report, reportPath);
console.info(`[eval] skill=${skill}`);
console.info(`[eval] publishedReport=${publishedReport ? "yes" : "no"}`); console.info(`[eval] publishedReport=${publishedReport ? "yes" : "no"}`);
return; return;
} }
@@ -273,8 +359,7 @@ async function main(): Promise<void> {
if (command === "baseline") { if (command === "baseline") {
const { updateEvalBaseline } = await import("./runner.js"); const { updateEvalBaseline } = await import("./runner.js");
const reportPath = getFlagString(flags, "report") ?? positionals[0]; const reportPath = getFlagString(flags, "report") ?? positionals[0];
const { baseline, baselinePath, storedInRedis } = const { baseline, baselinePath, storedInRedis } = await updateEvalBaseline(reportPath);
await updateEvalBaseline(reportPath);
console.info( console.info(
`[eval] baseline updated set=${baseline.setName} baseline=${baselinePath} redis=${storedInRedis ? "yes" : "no"}`, `[eval] baseline updated set=${baseline.setName} baseline=${baselinePath} redis=${storedInRedis ? "yes" : "no"}`,
); );
@@ -303,15 +388,14 @@ async function main(): Promise<void> {
} }
if (command === "promote") { if (command === "promote") {
const { appendCandidatesToJsonl, selectPromotionCandidates } = const { appendCandidatesToJsonl, selectPromotionCandidates } = await import(
await import("./promote.js"); "./promote.js"
);
const mongo = await connectMongoForEval(); const mongo = await connectMongoForEval();
try { try {
const candidates = await selectPromotionCandidates(mongo.connection, { const candidates = await selectPromotionCandidates(mongo.connection, {
limit: parsePositiveInteger(getFlagString(flags, "limit")), limit: parsePositiveInteger(getFlagString(flags, "limit")),
minLowScore: parsePositiveNumber( minLowScore: parsePositiveNumber(getFlagString(flags, "min-low-score")),
getFlagString(flags, "min-low-score"),
),
}); });
if (hasFlag(flags, "dry-run")) { if (hasFlag(flags, "dry-run")) {
@@ -330,7 +414,6 @@ async function main(): Promise<void> {
const result = await appendCandidatesToJsonl( const result = await appendCandidatesToJsonl(
candidates, candidates,
getFlagString(flags, "output"), getFlagString(flags, "output"),
getFlagString(flags, "skill"),
); );
console.info( console.info(
`[eval] promoted appended=${result.appendedCount} output=${result.outputPath}`, `[eval] promoted appended=${result.appendedCount} output=${result.outputPath}`,
@@ -342,8 +425,7 @@ async function main(): Promise<void> {
} }
if (command === "review") { if (command === "review") {
const { parseReviewReason, runReviewCli } = const { parseReviewReason, runReviewCli } = await import("./review/cli.js");
await import("./review/cli.js");
const garnet = await connectGarnetForEval(); const garnet = await connectGarnetForEval();
try { try {
const result = await runReviewCli(garnet.garnetClient, { const result = await runReviewCli(garnet.garnetClient, {
@@ -363,8 +445,9 @@ async function main(): Promise<void> {
if (command === "calibrate") { if (command === "calibrate") {
const { getEvalCalibrationMinCorr } = await import("../config.js"); const { getEvalCalibrationMinCorr } = await import("../config.js");
const { computeCalibrationReport } = const { computeCalibrationReport } = await import(
await import("./review/calibration.js"); "./review/calibration.js"
);
const mongo = await connectMongoForEval(); const mongo = await connectMongoForEval();
const garnet = await connectGarnetForEval(); const garnet = await connectGarnetForEval();
try { try {
@@ -398,13 +481,19 @@ async function main(): Promise<void> {
} }
if (command === "train-prm") { if (command === "train-prm") {
const { getEvalPrmMinLabels, getEvalPrmWeightsPath } = const {
await import("../config.js"); getEvalPrmMinLabels,
getEvalPrmWeightsPath,
} = await import("../config.js");
const { getEvalArchiveCollection } = await import("./archive.js"); const { getEvalArchiveCollection } = await import("./archive.js");
const { queryArchivedReviewLabels } = const { queryArchivedReviewLabels } = await import(
await import("./review/label-archive.js"); "./review/label-archive.js"
const { buildPrmTrainingSamples, savePrmModel, trainRidgePrmModel } = );
await import("./judges/prm.js"); const {
buildPrmTrainingSamples,
savePrmModel,
trainRidgePrmModel,
} = await import("./judges/prm.js");
const mongo = await connectMongoForEval(); const mongo = await connectMongoForEval();
try { try {
const labels = await queryArchivedReviewLabels(mongo.connection, { const labels = await queryArchivedReviewLabels(mongo.connection, {
@@ -460,13 +549,16 @@ async function main(): Promise<void> {
} }
if (command === "train-router") { if (command === "train-router") {
const { getAgentRouterWeightsPath, getEvalPrmMinLabels } = const {
await import("../config.js"); getAgentRouterWeightsPath,
getEvalPrmMinLabels,
} = await import("../config.js");
const { getEvalArchiveCollection } = await import("./archive.js"); const { getEvalArchiveCollection } = await import("./archive.js");
const { buildRouterTrainingSamples } = const { buildRouterTrainingSamples } = await import("./router-training.js");
await import("./router-training.js"); const {
const { saveLearnedRouterModel, trainLearnedRouterModel } = saveLearnedRouterModel,
await import("@skills/onboarding/runtime/router"); trainLearnedRouterModel,
} = await import("@skills/onboarding/runtime/router");
const mongo = await connectMongoForEval(); const mongo = await connectMongoForEval();
try { try {
const archiveLimit = const archiveLimit =
@@ -498,9 +590,7 @@ async function main(): Promise<void> {
const model = trainLearnedRouterModel(samples, { const model = trainLearnedRouterModel(samples, {
epochs: parsePositiveInteger(getFlagString(flags, "epochs")), epochs: parsePositiveInteger(getFlagString(flags, "epochs")),
learningRate: parsePositiveNumber( learningRate: parsePositiveNumber(getFlagString(flags, "learning-rate")),
getFlagString(flags, "learning-rate"),
),
l2: parsePositiveNumber(getFlagString(flags, "l2")), l2: parsePositiveNumber(getFlagString(flags, "l2")),
}); });
saveLearnedRouterModel(outputPath, model); saveLearnedRouterModel(outputPath, model);
@@ -513,46 +603,6 @@ async function main(): Promise<void> {
return; return;
} }
if (command === "router-shadow-report") {
const { getAgentRouterWeightsPath } = await import("../config.js");
const { loadLearnedRouterModel } =
await import("@skills/onboarding/runtime/router");
const { buildRouterShadowReport, formatRouterShadowReport } =
await import("./router-shadow-report.js");
const sinceMs = parseDurationMs(getFlagString(flags, "since"));
const untilFlag = getFlagString(flags, "until");
const untilMs = untilFlag ? parseDurationMs(untilFlag) : 0;
const now = Date.now();
const weightsPath =
getFlagString(flags, "weights") ?? getAgentRouterWeightsPath();
const reportOptions = {
since: new Date(now - sinceMs),
until: new Date(now - untilMs),
weightsPath,
};
if (!loadLearnedRouterModel(weightsPath)) {
const report = await buildRouterShadowReport(
{ readStreamRange: () => Promise.resolve([]) },
reportOptions,
);
console.info(formatRouterShadowReport(report));
return;
}
const garnet = await connectGarnetForEval();
try {
const report = await buildRouterShadowReport(
garnet.garnetClient,
reportOptions,
);
console.info(formatRouterShadowReport(report));
} finally {
await garnet.close();
}
return;
}
if (command === "propose") { if (command === "propose") {
const { createEvalLlmClient } = await import("./runner.js"); const { createEvalLlmClient } = await import("./runner.js");
const { const {
@@ -562,14 +612,9 @@ async function main(): Promise<void> {
} = await import("./loop/propose.js"); } = await import("./loop/propose.js");
const mongo = await connectMongoForEval(); const mongo = await connectMongoForEval();
try { try {
const slice = parseSlice(getFlagString(flags, "slice"));
const skill = getFlagString(flags, "skill");
const input = await buildProposalInputFromArchive( const input = await buildProposalInputFromArchive(
mongo.connection, mongo.connection,
{ parseSlice(getFlagString(flags, "slice")),
...slice,
...(skill ? { skill } : {}),
},
{ {
force: hasFlag(flags, "force"), force: hasFlag(flags, "force"),
limit: parsePositiveInteger(getFlagString(flags, "limit")), limit: parsePositiveInteger(getFlagString(flags, "limit")),
@@ -598,32 +643,30 @@ async function main(): Promise<void> {
if (command === "tournament") { if (command === "tournament") {
const { createEvalLlmClient } = await import("./runner.js"); const { createEvalLlmClient } = await import("./runner.js");
const { loadTournamentCases, runTournament, saveTournamentResult } = const {
await import("./loop/tournament.js"); loadTournamentCases,
runTournament,
saveTournamentResult,
} = await import("./loop/tournament.js");
const variantIds = parseCsv( const variantIds = parseCsv(
getFlagString(flags, "variants") ?? positionals[0], getFlagString(flags, "variants") ?? positionals[0],
); );
if (variantIds.length < 2) { if (variantIds.length < 2) {
throw new Error( throw new Error(
"eval:tournament requires --variants=control,<variantId>", 'eval:tournament requires --variants=control,<variantId>',
); );
} }
const mongo = await connectMongoForEval(); const mongo = await connectMongoForEval();
try { try {
const maxMatches = parsePositiveInteger( const maxMatches = parsePositiveInteger(getFlagString(flags, "max-matches"));
getFlagString(flags, "max-matches"),
);
const skill = getFlagString(flags, "skill");
const cases = await loadTournamentCases(mongo.connection, { const cases = await loadTournamentCases(mongo.connection, {
caseIds: parseCsv(getFlagString(flags, "case-ids")), caseIds: parseCsv(getFlagString(flags, "case-ids")),
limit: maxMatches, limit: maxMatches,
skill,
}); });
const result = await runTournament(createEvalLlmClient(), { const result = await runTournament(createEvalLlmClient(), {
cases, cases,
maxMatches, maxMatches,
skill,
variantIds, variantIds,
}); });
await saveTournamentResult(mongo.connection, result); await saveTournamentResult(mongo.connection, result);
@@ -640,7 +683,6 @@ async function main(): Promise<void> {
try { try {
const results = await queryTournamentResults(mongo.connection, { const results = await queryTournamentResults(mongo.connection, {
limit: parsePositiveInteger(getFlagString(flags, "limit")), limit: parsePositiveInteger(getFlagString(flags, "limit")),
skill: getFlagString(flags, "skill"),
variantId: getFlagString(flags, "variant"), variantId: getFlagString(flags, "variant"),
}); });
for (const result of results) { for (const result of results) {
@@ -654,7 +696,10 @@ async function main(): Promise<void> {
throw new Error(`Unknown eval command: ${command}`); throw new Error(`Unknown eval command: ${command}`);
} catch (error) { } catch (error) {
logger.error({ err: error }, `[eval] "${command}" failed`); logger.error(
{ err: error },
`[eval] "${command}" failed`,
);
process.exitCode = 1; process.exitCode = 1;
} }
} }
+6 -12
View File
@@ -3,12 +3,7 @@ import type {
EvalArchiveEntry, EvalArchiveEntry,
MongoConnectionLike, MongoConnectionLike,
} from "./archive.js"; } from "./archive.js";
import { import { getEvalArchiveCollection } from "./archive.js";
getArchiveDerivedIntent,
getArchiveDerivedPhase,
getArchiveRoutingDecision,
getEvalArchiveCollection,
} from "./archive.js";
import type { SliceKey } from "./aggregator.js"; import type { SliceKey } from "./aggregator.js";
const DAY_MS = 24 * 60 * 60 * 1000; const DAY_MS = 24 * 60 * 60 * 1000;
@@ -46,7 +41,6 @@ function average(values: number[]): number | null {
function sliceKeyToString(slice: SliceKey): string { function sliceKeyToString(slice: SliceKey): string {
return [ return [
slice.skill ?? "onboarding",
slice.tier, slice.tier,
slice.intent, slice.intent,
slice.phase, slice.phase,
@@ -57,13 +51,13 @@ function sliceKeyToString(slice: SliceKey): string {
} }
function getSlice(entry: EvalArchiveEntry): SliceKey { function getSlice(entry: EvalArchiveEntry): SliceKey {
const routingDecision = getArchiveRoutingDecision(entry.derived);
return { return {
skill: entry.skill || entry.request.skill || "onboarding",
tier: entry.tier, tier: entry.tier,
intent: getArchiveDerivedIntent(entry.derived), intent: entry.derived.intent.kind,
phase: getArchiveDerivedPhase(entry.derived), phase: entry.derived.phase.kind,
...(routingDecision ? { routingDecision } : {}), ...(entry.derived.routingDecision
? { routingDecision: entry.derived.routingDecision.kind }
: {}),
}; };
} }
-6
View File
@@ -1,6 +0,0 @@
{"id":"case_handoff_onboarding_to_moderation_ad","name":"onboarding routes community rule question to moderation","input":{"request":{"requestId":"handoff-1","skill":"onboarding","chatroomId":"room-handoff","userId":"user-1","nickname":"tester","trigger":"user-message","userMessage":"광고 올려도 돼요?","context":{"evalHandoffPlan":[{"from":"onboarding","to":"moderation","reason":"out-of-scope","taskSpec":"광고/홍보 규칙 문의 처리"}]}},"conversationHistory":[{"role":"user","content":"광고 올려도 돼요?","timestamp":"2026-05-18T00:00:00.000Z"}]},"expected":{"expectedHandoff":{"shouldHandoff":true,"expectedReason":"out-of-scope","expectedTarget":"moderation"},"expectedFinalSkill":"moderation","expectedSourceChain":{"mode":"contains","chain":["moderation"]}}}
{"id":"case_handoff_moderation_to_onboarding_welcome","name":"moderation routes onboarding question to onboarding","input":{"request":{"requestId":"handoff-2","skill":"moderation","chatroomId":"room-handoff","userId":"user-2","nickname":"tester","trigger":"user-message","userMessage":"신규 가입 안내랑 닉네임 규칙 알려줘","context":{"evalHandoffPlan":[{"from":"moderation","to":"onboarding","reason":"out-of-scope","taskSpec":"신규 가입 안내 처리"}]}},"conversationHistory":[{"role":"user","content":"신규 가입 안내랑 닉네임 규칙 알려줘","timestamp":"2026-05-18T00:00:00.000Z"}]},"expected":{"expectedHandoff":{"shouldHandoff":true,"expectedReason":"out-of-scope","expectedTarget":"onboarding"},"expectedFinalSkill":"onboarding","expectedSourceChain":{"mode":"exact","chain":["moderation","onboarding"]}}}
{"id":"case_handoff_low_confidence_to_supervisor","name":"unclear handoff escalates to supervisor","input":{"request":{"requestId":"handoff-3","skill":"onboarding","chatroomId":"room-handoff","userId":"user-3","nickname":"tester","trigger":"user-message","userMessage":"음... 이거랑 저거 그거 어떻게 해?","context":{"evalHandoffPlan":[{"from":"onboarding","to":"supervisor","reason":"low-confidence","taskSpec":"의도 불명확 케이스 운영자 검토"}]}}},"expected":{"expectedHandoff":{"shouldHandoff":true,"expectedReason":"low-confidence","expectedTarget":"supervisor"},"expectedFinalSkill":"supervisor","expectedSourceChain":{"mode":"exact","chain":["onboarding","supervisor"]}}}
{"id":"case_handoff_validation_retry_then_supervisor","name":"validation failure escalates to supervisor","input":{"request":{"requestId":"handoff-4","skill":"moderation","chatroomId":"room-handoff","userId":"user-4","nickname":"tester","trigger":"user-message","userMessage":"규칙 답변에 링크를 넣어도 돼?","context":{"evalHandoffPlan":[{"from":"moderation","to":"supervisor","reason":"validation-failed","taskSpec":"runtime verifier failed after retry"}]}}},"expected":{"expectedHandoff":{"shouldHandoff":true,"expectedReason":"validation-failed","expectedTarget":"supervisor"},"expectedFinalSkill":"supervisor","expectedSourceChain":{"mode":"exact","chain":["moderation","supervisor"]}}}
{"id":"case_handoff_aba_loop_blocked","name":"aba loop is rerouted to supervisor","input":{"request":{"requestId":"handoff-5","skill":"onboarding","chatroomId":"room-handoff","userId":"user-5","nickname":"tester","trigger":"user-message","userMessage":"가입인지 규칙인지 헷갈리는 질문","context":{"evalBudgetMaxSameAgentVisits":1,"evalHandoffPlan":[{"from":"onboarding","to":"moderation","reason":"out-of-scope","taskSpec":"규칙 확인"},{"from":"moderation","to":"onboarding","reason":"out-of-scope","taskSpec":"가입 상태 확인"}]}}},"expected":{"expectedHandoff":{"shouldHandoff":true,"expectedReason":"same-agent-revisit","expectedTarget":"supervisor"},"expectedFinalSkill":"supervisor","expectedSourceChain":{"mode":"exact","chain":["onboarding","moderation","supervisor"]}}}
{"id":"case_handoff_high_risk_to_supervisor","name":"high risk action escalates to supervisor","input":{"request":{"requestId":"handoff-6","skill":"moderation","chatroomId":"room-handoff","userId":"user-6","nickname":"tester","trigger":"user-message","userMessage":"이 유저 바로 밴해줘","context":{"evalHandoffPlan":[{"from":"moderation","to":"supervisor","reason":"high-risk-action","risk":"high","taskSpec":"제재성 고위험 요청 운영자 검토"}]}}},"expected":{"expectedHandoff":{"shouldHandoff":true,"expectedReason":"high-risk-action","expectedTarget":"supervisor"},"expectedFinalSkill":"supervisor","expectedSourceChain":{"mode":"exact","chain":["moderation","supervisor"]}}}
-6
View File
@@ -1,6 +0,0 @@
{"id":"memory-retrieval-accurate","name":"retrieval-accurate","input":{"request":{"requestId":"mem_mod_ads","skill":"moderation","chatroomId":"memory_room","userId":"memory_user_1","nickname":"tester","trigger":"user-message","userMessage":"광고 도배 기준이 뭐예요?"}},"expected":{"intent":"rule-question","mustIncludeAny":["광고","도배"]}}
{"id":"memory-long-range-understanding","name":"long-range-understanding","input":{"request":{"requestId":"mem_onboarding_recall","skill":"onboarding","chatroomId":"memory_room","userId":"memory_user_2","nickname":"LolName_KR/95/남/D","trigger":"user-message","userMessage":"내 닉네임 뭐라고 했지?"},"conversationHistory":[{"role":"user","content":"디스코드 닉네임은 LolName_KR입니다","timestamp":"2026-05-01T00:00:00.000Z"},{"role":"assistant","content":"확인했어요.","timestamp":"2026-05-01T00:00:01.000Z"}],"summary":"사용자가 디스코드 닉네임은 LolName_KR이라고 말했다."},"expected":{"mustIncludeAny":["LolName_KR"],"expectedMemory":{"expectedRecalledFact":"LolName_KR"}}}
{"id":"memory-test-time-learning","name":"test-time-learning","input":{"request":{"requestId":"mem_onboarding_hint","skill":"onboarding","chatroomId":"memory_room","userId":"memory_user_3","nickname":"낭만/00/남D","trigger":"nickname-invalid"},"userProfile":{"firstSeenAt":"2026-05-01T00:00:00.000Z","lastSeenAt":"2026-05-18T00:00:00.000Z","flaggedIssues":["닉네임 형식 오류","닉네임 형식 오류","닉네임 형식 오류"],"pastIntents":[{"kind":"nickname-question","count":5}],"preferredStyle":"concise","totalTurns":12,"userId":"user_memory"}},"expected":{"mustIncludeAny":["예시","형식"],"expectedMemory":{"expectedProphylacticHint":true}}}
{"id":"memory-selective-forgetting","name":"selective-forgetting","input":{"request":{"requestId":"mem_forgetting","skill":"onboarding","chatroomId":"memory_room","userId":"memory_user_4","nickname":"tester","trigger":"user-message","userMessage":"닉네임 규칙 최신 기준 알려줘"},"summary":"정정된 최신 규칙만 답해야 한다."},"expected":{"mustNotInclude":["예전 기준"],"expectedMemory":{"expectedArchivedDocPattern":"예전 기준"}}}
{"id":"memory-escalation-history","name":"escalation-history","input":{"request":{"requestId":"mem_supervisor_history","skill":"supervisor","chatroomId":"memory_room","userId":"memory_user_5","nickname":"tester","trigger":"user-message","context":{"handoffPacket":{"constraints":[],"currentState":{},"evidence":[],"expectedOutputSchema":"text","handoffReason":"out-of-scope","sourceChain":["onboarding"],"summary":"광고성 메시지로 보임","userGoal":"광고 홍보 가능 여부 확인"}}}},"expected":{"expectedMemory":{"expectedSupervisorTarget":"moderation"}}}
{"id":"memory-contradiction","name":"memory-contradiction","input":{"request":{"requestId":"mem_contradiction","skill":"onboarding","chatroomId":"memory_room","userId":"memory_user_6","nickname":"새닉/95/남/D","trigger":"user-message","userMessage":"닉네임 새닉으로 변경했어요"},"userProfile":{"firstSeenAt":"2026-05-01T00:00:00.000Z","lastSeenAt":"2026-05-18T00:00:00.000Z","flaggedIssues":[],"pastIntents":[{"kind":"nickname-question","count":4}],"preferredStyle":"neutral","totalTurns":8,"userId":"user_memory","longTermSummary":{"text":"사용자는 이전 닉네임에서 새닉으로 변경했다고 말했다.","updatedAt":"2026-05-18T00:00:00.000Z","turnSnapshot":8}}},"expected":{"mustIncludeAny":["새닉"],"expectedMemory":{"expectedContradictionResolved":true,"expectedRecalledFact":"새닉"}}}
Whitespace-only changes.
Whitespace-only changes.
-5
View File
@@ -1,5 +0,0 @@
{"id":"moderation_rule_ads","name":"moderation:rule-question:ads","input":{"request":{"requestId":"mod_req_ads","skill":"moderation","chatroomId":"mod_room","userId":"mod_user_1","nickname":"문의자","trigger":"user-message","userMessage":"광고 올려도 돼요?"}},"expected":{"intent":"rule-question","mustIncludeAny":["광고","홍보","허가","제한"],"mustNotInclude":["무조건 가능","바로 올려도"]}}
{"id":"moderation_rule_profanity","name":"moderation:rule-question:profanity","input":{"request":{"requestId":"mod_req_profanity","skill":"moderation","chatroomId":"mod_room","userId":"mod_user_2","nickname":"문의자","trigger":"user-message","userMessage":"욕설하면 어떻게 돼요?"}},"expected":{"intent":"rule-question","mustIncludeAny":["욕설","금지","운영진"],"mustNotInclude":["괜찮아요","상관없"]}}
{"id":"moderation_rule_spam","name":"moderation:rule-question:spam","input":{"request":{"requestId":"mod_req_spam","skill":"moderation","chatroomId":"mod_room","userId":"mod_user_3","nickname":"문의자","trigger":"user-message","userMessage":"도배 기준이 뭐예요?"}},"expected":{"intent":"rule-question","mustIncludeAny":["도배","반복","제재"],"mustNotInclude":["기준 없음","상관없"]}}
{"id":"moderation_report_profanity","name":"moderation:violation-report:profanity","input":{"request":{"requestId":"mod_req_report","skill":"moderation","chatroomId":"mod_room","userId":"mod_user_4","nickname":"제보자","trigger":"user-message","userMessage":"어떤 분이 욕했어요 신고하고 싶어요"}},"expected":{"intent":"violation-report","mustIncludeAny":["관련 메시지","상황","운영진"],"mustNotInclude":["제가 처리","제재 확정"]}}
{"id":"moderation_appeal_sanction","name":"moderation:appeal:sanction","input":{"request":{"requestId":"mod_req_appeal","skill":"moderation","chatroomId":"mod_room","userId":"mod_user_5","nickname":"소명자","trigger":"user-message","userMessage":"정지당했는데 억울해요"}},"expected":{"intent":"appeal","mustIncludeAny":["이의 제기","소명","운영진"],"mustNotInclude":["해제됐","풀렸"]}}
@@ -138,63 +138,3 @@
{"id":"case_acd612e0","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778740258770-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_7169cd5a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_81f2ad8f","nickname":"<nick:ba29bb75>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}} {"id":"case_acd612e0","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778740258770-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_7169cd5a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_81f2ad8f","nickname":"<nick:ba29bb75>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_6cd7106c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778718965990-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse={\"reply\":\"안녕하세요!\"}","input":{"request":{"requestId":"req_1477394f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_24cff593","nickname":"<nick:d455d0e4>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}} {"id":"case_6cd7106c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778718965990-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse={\"reply\":\"안녕하세요!\"}","input":{"request":{"requestId":"req_1477394f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_24cff593","nickname":"<nick:d455d0e4>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_de51735e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778674030965-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_32cc22d6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_cd95eff0","nickname":"<nick:7feae321>","trigger":"user-message","userMessage":"안녕하세오"}},"expected":{}} {"id":"case_de51735e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778674030965-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_32cc22d6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_cd95eff0","nickname":"<nick:7feae321>","trigger":"user-message","userMessage":"안녕하세오"}},"expected":{}}
{"id":"case_4f45294b","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778756106669-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 여기 채팅방에 올리면 돼요. \n채팅방 오른쪽 상단 메뉴의 '커버 보기'로 들어가 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷으로 보내면 됩니다.","input":{"request":{"requestId":"req_e6e21e71","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"하트인증\n여따 올리면 되요?"}},"expected":{}}
{"id":"case_e3ff18c6","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778756063255-0; llmOverall=2.75; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 확인했어요. 천천히 진행해 주세요.","input":{"request":{"requestId":"req_7b05c872","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"네"}},"expected":{}}
{"id":"case_1b906220","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778763820454-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임 순서는 이름/출생연도/성별/티어로 맞게 돼 있고, 안 되는 부분은 마지막 티어 코드뿐이에요. \n현재의 '플'은 허용 티어 코드가 아니므로 GM, I, B, S, G, P, E, D, M, C, U 중 한 글자(예: D)로 바꿔주세요.","input":{"request":{"requestId":"req_a8ba4ffe","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_e3e5b006","nickname":"<nick:405abf95>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_4ff31f85","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778756157817-0; llmOverall=n/a; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 확인했어요. 천천히 진행해 주세요.","input":{"request":{"requestId":"req_c78edc0e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"오"}},"expected":{}}
{"id":"case_0fd45691","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778874307141-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=닉네임이 안 되는 이유는 형식이 맞지 않아서예요. \n현재 \"위례\"는 이름만 있고, 연도/성별/티어가 없어 형식이 깨졌습니다. \n카카오톡 닉네임은 [이름]/[출생연도 두 자리]/[남 또는 여]/[티어]로 바꿔주세요.","input":{"request":{"requestId":"req_b6f69c24","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:16fb6789>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_408c317c","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778874400236-0; llmOverall=2.25; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=채팅방 오른쪽 상단 메뉴 버튼을 눌러 “커버 보기”로 들어가면 하트 인증 화면이 보여요.","input":{"request":{"requestId":"req_1db51665","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"하트인증은 어디다 하나요?"}},"expected":{}}
{"id":"case_cf22a9d8","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778874533535-0; llmOverall=2.5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=좋아요, 천천히 진행해 주세요.","input":{"request":{"requestId":"req_60f5e74a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"넵 감사합니다"}},"expected":{}}
{"id":"case_fd404eb4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778874508177-0; llmOverall=5; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=네, 가능합니다. 카카오톡 1단계가 완료된 상태라면 디스코드는 저녁에도 접속해 진행하시면 돼요.","input":{"request":{"requestId":"req_d814bae6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"디스코드는 저녁에 해도될까요?"}},"expected":{}}
{"id":"case_d42b8624","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778874344992-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_47d100d2","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"안녕하세여"}},"expected":{}}
{"id":"case_bf09ad3a","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778954734843-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=현재 남은 절차는 카카오톡 닉네임 변경과 방 하트 인증입니다.","input":{"request":{"requestId":"req_010646c0","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_a2e8fc0b","nickname":"<nick:a9bad7ab>","trigger":"user-message","userMessage":"혹시 이시간엔 내전 안돌아가죠?"}},"expected":{}}
{"id":"case_275402ce","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950244576-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 건 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정, 디스코드 주민신청 채널에 메시지 남기기예요.","input":{"request":{"requestId":"req_bc7c8aae","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"5대5 하려고 하는데 가능 한지 해서용"}},"expected":{}}
{"id":"case_0e56499c","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950227521-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 절차는 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정, 디스코드 주민신청 채널에 메시지 남기기예요.","input":{"request":{"requestId":"req_593ffaff","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"아 지금 들어와 있는 관리자 분이용"}},"expected":{}}
{"id":"case_c8398f09","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778949416363-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=웅/97/남/M님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_b0f0648f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"혹시 지 외전 문의 드리려 하는데"}},"expected":{}}
{"id":"case_0a460498","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950169079-0; llmOverall=2; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=관리분(관리자)은 있어요.","input":{"request":{"requestId":"req_6bffed4f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"관리분 있으신가요?"}},"expected":{}}
{"id":"case_002ac417","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778931066262-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임 \"강서/93/남\"은 티어가 빠져서 형식이 안 맞아요. \n강서/93/남은 티어가 빠져서 형식이 안 맞아요. 마지막에 티어 한 글자만 추가하면 됩니다. \n예: 강서/93/남/D","input":{"request":{"requestId":"req_ed284edb","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0d05dfd2","nickname":"<nick:1ec0deae>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_3e163e2e","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778949203127-0; llmOverall=2.75; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임은 형식이 안 맞아요. \n카톡 닉네임은 구분자로 '/'를 써야 하고 4개 항목이 필요해요. 현재는 웅 97 남 M라 안 되고, 현재 닉네임은 웅/97/남/M 이렇게 고치면 됩니다.","input":{"request":{"requestId":"req_ab630198","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:6334c82e>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_1db9c911","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778958194603-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_f3d19239","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3e4be23b","nickname":"<nick:b09afa6a>","trigger":"user-message","userMessage":"안녕하세여"}},"expected":{}}
{"id":"case_ee7ae44a","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778955694984-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_fea5fd76","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3592a95a","nickname":"<nick:437c8fc3>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_d6bd160f","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778950199447-0; llmOverall=2.25; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=운영진께, 특히 관리자님께 문의하시면 돼요.","input":{"request":{"requestId":"req_5fbe8788","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"누구한테 여쭤봐야 할까요"}},"expected":{}}
{"id":"case_36b3ac9c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778949423115-0; llmOverall=n/a; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=웅/97/남/M님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_35e659fa","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"어디에 문의 해야 할까용"}},"expected":{}}
{"id":"case_f7a5d661","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019400869-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_4861ee12","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"user-message","userMessage":"듀랭하실분들 있을까요"}},"expected":{}}
{"id":"case_74eac586","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019394267-0; llmOverall=1.25; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_743e0456","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"user-message","userMessage":"혹시"}},"expected":{}}
{"id":"case_9e9195bb","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019388776-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=@벽 넘어가는 탈론 00/남/E3님 닉네임에서 수정할 부분이 있어요!\n나이를 두 자리 숫자로 입력해주세요 (예: 25)\n성별(남/여)과 티어 코드를 '/'로 구분해서 입력해주세요","input":{"request":{"requestId":"req_568d875b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_8acd1f91","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779011411116-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임에서 안 된 부분은 마지막 티어예요. \nG2는 허용 티어가 아니라서, 뒤의 G2를 GM, I, B, S, G, P, E, D, M, C, U 중 하나(예: G)로만 바꾸면 됩니다.","input":{"request":{"requestId":"req_9860e773","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:666d8731>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_d6824fdb","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779023514843-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_4d45f87c","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bc77847e","nickname":"<nick:9ee38f43>","trigger":"user-message","userMessage":"안녕하세요 공지읽고오겠습니다"}},"expected":{}}
{"id":"case_83136947","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779023414663-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요, 반갑습니다.","input":{"request":{"requestId":"req_d55e086e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_a9ca5eb1","nickname":"<nick:ebf85b46>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_1686cd08","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779019324629-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임은 /로 구분된 4개 항목(이름/출생연도 두 자리/성별/티어)이 아니어서 안 돼요. 지금은 항목이 2개로 잘못되어 있어요. \n고쳐야 할 부분은 연도/성별/티어를 다시 /로 정확히 나눠서 넣는 거예요. \n성별은 남 또는 여 한 글자, 티어는 GM/I/B/S/G/P/E/D/M/C/U 중 한 글자여야 해요.","input":{"request":{"requestId":"req_7cb58844","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:23dc710b>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_0f722e08","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779015436203-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_a6a1d38f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bc35f985","nickname":"<nick:c3de5ead>","trigger":"user-message","userMessage":"안녕하세요~"}},"expected":{}}
{"id":"case_009d9558","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779011507015-0; llmOverall=2.25; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 절차는 \n카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 서버 프로필 닉네임 설정, 디스코드 주민신청 채널 메시지 남기기 입니다.","input":{"request":{"requestId":"req_18174c18","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:94137b1e>","trigger":"user-message","userMessage":"이러면되나요"}},"expected":{}}
{"id":"case_b51a6dfb","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779011451230-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_5d3182b0","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:94137b1e>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_8f87e018","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779562992008-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=고구마는보라색/98/남/D님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_d9ff0f8b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:b410b374>","trigger":"user-message","userMessage":"봇인가요? 어케 이렇게 빠르지"}},"expected":{}}
{"id":"case_51fec5d7","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779563007004-0; llmOverall=1.25; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=고구마는보라색/98/남/D님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_058c9557","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:b410b374>","trigger":"user-message","userMessage":"아하"}},"expected":{}}
{"id":"case_47db12bc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779544865402-0; llmOverall=1.25; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=방 하트 인증은 채팅방 오른쪽 상단 메뉴 버튼에서 \"커버 보기\"를 눌러 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷을 올려주시면 돼요.","input":{"request":{"requestId":"req_887be307","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_e35470e6","nickname":"<nick:01e9e582>","trigger":"user-message","userMessage":"어떻게 인증하나요 ?"}},"expected":{}}
{"id":"case_8aa3d0bc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779574878821-0; llmOverall=2; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=아직 승인해드리기 어려워요. 남은 항목이 카카오톡 본인인증방 입장과 디스코드 주민신청 채널 메시지라서, 이 두 가지만 먼저 완료해주시면 될 것 같아요.","input":{"request":{"requestId":"req_117cdf74","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"디코 승인 부탁드려도 될까요?"}},"expected":{}}
{"id":"case_0191603e","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779574708797-0; llmOverall=2; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=디스코드 관련 안내입니다. 디스코드 닉네임은 닉네임#태그/출생 연도 두 자리/티어 형식으로 수정해 주세요. 예시: 할리#프렌치불독/88/D \n디스코드 서버에서 이 닉네임은 본인 식별용이라 꼭 맞춰야 하고, 주민신청 채널에는 서버 소개 메시지도 남겨야 해요.","input":{"request":{"requestId":"req_17d4f38d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"/디코"}},"expected":{}}
{"id":"case_214e18f8","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779562888510-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=@고구마는보라색/98/D/남님 닉네임에서 수정할 부분이 있어요!\n성별을 '남' 또는 '여'로 입력해주세요\n티어 코드(GM, I, B, S, G, P, E, D, M, C, U 중 하나)를 입력해주세요","input":{"request":{"requestId":"req_dd3e072b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:f7254b29>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_b90d32a5","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535490115-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=네, 다시 보내주시면 돼요. \n방에서 오른쪽 상단 메뉴 버튼 열고 커버 보기로 들어간 뒤 하트를 눌러 빨간색으로 바꿔주세요. \n그 상태 화면을 채팅방에서 바로 캡처해서 편집(자르기/필터/화면 편집) 없이 원본 그대로 보내주시면 됩니다. \n참고로 회색 하트면 아직 인증으로 안 된 상태예요.","input":{"request":{"requestId":"req_8ac4ab90","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_dc095d27","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535154924-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=네, 다시 시도할 수 있어요. \n방 하트를 빨간색으로 바꾼 뒤, 채팅방에서 바로 캡처한 원본 화면(편집/자르기/필터 없음)만 새로 보내주시면 됩니다.","input":{"request":{"requestId":"req_bb9360ad","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_c33b0453","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535131374-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 하트는 눌렀다면 하트가 빨간색인 화면을 편집 없이 원본 스크린샷으로 올려주시면 돼요.","input":{"request":{"requestId":"req_5c1b0e13","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"user-message","userMessage":"하트는 눌렀는대.."}},"expected":{}}
{"id":"case_955dc7df","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1779605646714-0; llmOverall=2.5; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=오케이, 하트 인증은 화면이 제대로 올라오면 바로 확인돼요. \n지금 올리신 걸로 보냈다면 잠깐 확인 기다려볼게요. \n만약 아직 안 보인다면, 하트를 빨간색으로 바꾼 그대로의 원본 캡처만 한 번 더 보내주세요.","input":{"request":{"requestId":"req_fb38ace6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0ea265d7","nickname":"<nick:7a45b00c>","trigger":"user-message","userMessage":"위에 인증햇어여"}},"expected":{}}
{"id":"case_e75e191e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779604227776-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=다시 시도하려면 이렇게 하면 돼요.\n\n채팅방 오른쪽 상단 메뉴 버튼에서 커버 보기로 들어가서 하트를 빨간색으로 바꾼 뒤, \n그 장면이 보이게 편집 없이 원본 화면 캡처로 다시 보내주면 돼요.","input":{"request":{"requestId":"req_4368c115","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0ea265d7","nickname":"<nick:d0f3633e>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_f485439f","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779593282207-0; llmOverall=n/a; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=Summer/95/남/S님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_a33cd695","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:7ae6ee97>","trigger":"user-message","userMessage":"가입했습니다"}},"expected":{}}
{"id":"case_75bc3458","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779593274596-0; llmOverall=n/a; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=Summer/95/남/S님 이제 추가 절차만 남았어요. 카카오톡 본인인증방 입장과 디스코드 주민신청 채널에 메시지 남기기을 확인해 주세요.","input":{"request":{"requestId":"req_5bb3e408","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:7ae6ee97>","trigger":"user-message","userMessage":"디스코드"}},"expected":{}}
{"id":"case_c1016e76","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779592708891-0; llmOverall=5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 천천히 하세요.","input":{"request":{"requestId":"req_179a599b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:50cd33f9>","trigger":"user-message","userMessage":"엇 잠시만용"}},"expected":{}}
{"id":"case_12f3372d","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779574484786-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 바로 여기 채팅방에 올리면 됩니다. \n채팅방 오른쪽 상단 메뉴의 \"커버 보기\"에서 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷으로 보내주세요.","input":{"request":{"requestId":"req_b7b755bf","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"하트인증은 여기다 올리면 될까요?"}},"expected":{}}
{"id":"case_9f0cb2f4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779574363525-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_b822a53e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:90b41f68>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_885a35a9","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629981088-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_42415f0f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"진짜 잘하시네요"}},"expected":{}}
{"id":"case_4603c0cc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629976794-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_5400a31f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"챌린저 ㄷㄷ"}},"expected":{}}
{"id":"case_2b38adb8","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629930176-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=채팅방 프로필에서 하트를 눌러 빨간색으로 만든 다음, 그 화면이 보이게 원본 스크린샷을 보내주시면 인증돼요.","input":{"request":{"requestId":"req_bf140842","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"어떻게 하나여"}},"expected":{}}
{"id":"case_f04b5287","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779633055712-0; llmOverall=1.25; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요, 오늘도 편하게 진행해요.","input":{"request":{"requestId":"req_bdd1114b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:c673b3e2>","trigger":"user-message","userMessage":"안녕하세요!"}},"expected":{}}
{"id":"case_3e077513","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779630086906-0; llmOverall=1.5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 그렇게 해주시면 돼요.","input":{"request":{"requestId":"req_bbc426e2","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"아항"}},"expected":{}}
{"id":"case_274fec57","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629952313-0; llmOverall=1.5; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_c69053e8","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"보내면 되나여?"}},"expected":{}}
{"id":"case_29f02c6e","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629936899-0; llmOverall=1.5; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_be0e7f4a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"여기 채팅방에"}},"expected":{}}
{"id":"case_59c880e4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779633743117-0; llmOverall=4.25; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=카카오톡에서 디스코드 초대 안내를 받았던 메시지를 찾아 초대 수락을 눌러 들어가면 돼요.","input":{"request":{"requestId":"req_705c9e3f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:c673b3e2>","trigger":"user-message","userMessage":"저 디코는 어떻게 들어가요? 주민신청까지 해뒀습니다"}},"expected":{}}
{"id":"case_4fe5864d","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779632430014-0; llmOverall=5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=좋아요, 천천히 진행해 주세요.","input":{"request":{"requestId":"req_781682ec","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:658a6c82>","trigger":"user-message","userMessage":"넵"}},"expected":{}}
{"id":"case_9a3640ce","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779632381253-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=안 되는 이유는 티어가 허용 코드(GM, I, B, S, G, P, E, D, M, C, U) 중 하나의 대문자로 되어야 하는데 p가 소문자여서예요. \n카카오톡 닉네임의 마지막 부분만 P로 바꿔서 세나/95/여/P로 수정해 주세요.","input":{"request":{"requestId":"req_7368778f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:1f050511>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_5ea6c4d2","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779629969676-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=환영해요! 반갑습니다.","input":{"request":{"requestId":"req_9352c82d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"안녕하세요!!"}},"expected":{}}
{"id":"case_207dec80","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779629904020-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=방 하트 인증은 채팅방 오른쪽 상단 메뉴 버튼의 '커버 보기'에서 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 화면이 보이게 스크린샷을 보내주시면 돼요.","input":{"request":{"requestId":"req_37928f7d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"방하트 인증은"}},"expected":{}}
{"id":"case_b42ca588","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779622035543-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=어서오세요!","input":{"request":{"requestId":"req_e9dcff68","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3adc9936","nickname":"<nick:ccb74517>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_d96d1b98","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779619834487-0; llmOverall=n/a; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=네, 본인인증이 완료됐다면 본인인증방은 나가셔도 돼요. \n현재 남은 절차는 디스코드 주민신청 채널 메시지 남기기 하나예요.","input":{"request":{"requestId":"req_e6bc023c","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_d9c80f01","nickname":"<nick:9c5d5e3a>","trigger":"user-message","userMessage":"본인인증방은이제나가도돼요?"}},"expected":{}}
File renamed without changes.
+3 -26
View File
@@ -1,9 +1,8 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import type { PromptVariantManifest } from "@eval/prompt-variant";
import { SkillRegistry } from "@skills/skill-loader"; import { SkillRegistry } from "@skills/skill-loader";
import { createModerationToolRegistry } from "@skills/moderation/tools"; import type { PromptVariantManifest } from "@skills/onboarding/prompts/builder";
import { createOnboardingToolRegistry } from "@skills/onboarding/tools"; import { createBuiltInToolRegistry } from "@tools/builtins/index";
import { runG1Static, runG4Permission } from "./gates.js"; import { runG1Static, runG4Permission } from "./gates.js";
function manifest( function manifest(
@@ -70,7 +69,7 @@ describe("governance gates", () => {
it("checks required tool and resource permissions in G4", () => { it("checks required tool and resource permissions in G4", () => {
const skillRegistry = SkillRegistry.load(`${process.cwd()}/src/skills`); const skillRegistry = SkillRegistry.load(`${process.cwd()}/src/skills`);
const toolRegistry = createOnboardingToolRegistry(); const toolRegistry = createBuiltInToolRegistry();
expect( expect(
runG4Permission( runG4Permission(
@@ -81,7 +80,6 @@ describe("governance gates", () => {
} as PromptVariantManifest, } as PromptVariantManifest,
toolRegistry, toolRegistry,
skillRegistry, skillRegistry,
"onboarding",
), ),
).toMatchObject({ ).toMatchObject({
passed: true, passed: true,
@@ -96,31 +94,10 @@ describe("governance gates", () => {
} as PromptVariantManifest, } as PromptVariantManifest,
toolRegistry, toolRegistry,
skillRegistry, skillRegistry,
"onboarding",
), ),
).toMatchObject({ ).toMatchObject({
passed: false, passed: false,
reason: "missing-required-capability", reason: "missing-required-capability",
}); });
}); });
it("checks moderation resources against the target skill", () => {
const skillRegistry = SkillRegistry.load(`${process.cwd()}/src/skills`);
expect(
runG4Permission(
{
...manifest(),
requiredResources: ["community-rules"],
requiredTools: ["lookup-rule"],
skill: "moderation",
} as PromptVariantManifest,
createModerationToolRegistry(),
skillRegistry,
"moderation",
),
).toMatchObject({
passed: true,
});
});
}); });
+4 -10
View File
@@ -1,4 +1,5 @@
import type { PromptVariantManifest } from "@eval/prompt-variant"; import type { PromptVariantManifest } from "@skills/onboarding/prompts/builder";
import type { RuntimeFlagManifest } from "@skills/onboarding/runtime/runtime-flag-manifest";
import type { SkillRegistry } from "@skills/skill-loader"; import type { SkillRegistry } from "@skills/skill-loader";
import type { ToolRegistry } from "@tools/registry"; import type { ToolRegistry } from "@tools/registry";
@@ -9,12 +10,6 @@ export interface GateResult {
reason?: string | undefined; reason?: string | undefined;
} }
interface RuntimeFlagManifest {
description: string;
flags: Record<string, unknown>;
variantId: string;
}
type GovernedManifest = PromptVariantManifest | RuntimeFlagManifest; type GovernedManifest = PromptVariantManifest | RuntimeFlagManifest;
const riskyPatterns: Array<{ pattern: RegExp; reason: string }> = [ const riskyPatterns: Array<{ pattern: RegExp; reason: string }> = [
@@ -99,15 +94,14 @@ export function runG4Permission(
manifest: GovernedManifest, manifest: GovernedManifest,
toolRegistry: ToolRegistry, toolRegistry: ToolRegistry,
skillRegistry: SkillRegistry, skillRegistry: SkillRegistry,
targetSkill: string,
): GateResult { ): GateResult {
const record = asRecord(manifest); const record = asRecord(manifest);
const requiredTools = readStringArray(record, "requiredTools"); const requiredTools = readStringArray(record, "requiredTools");
const requiredResources = readStringArray(record, "requiredResources"); const requiredResources = readStringArray(record, "requiredResources");
const missingTools = requiredTools.filter((toolName) => !toolRegistry.has(toolName)); const missingTools = requiredTools.filter((toolName) => !toolRegistry.has(toolName));
const skillManifest = skillRegistry.get(targetSkill); const onboarding = skillRegistry.get("onboarding");
const missingResources = requiredResources.filter( const missingResources = requiredResources.filter(
(resource) => !skillManifest?.resources[resource], (resource) => !onboarding?.resources[resource],
); );
if (missingTools.length > 0 || missingResources.length > 0) { if (missingTools.length > 0 || missingResources.length > 0) {
-143
View File
@@ -1,143 +0,0 @@
import { selectSkillForRequest } from "@agent/agent";
import { BudgetTracker } from "@agent/budget";
import { dispatch, type EvalDispatchPlan, type EvalDispatchPlanStep } from "@agent/dispatch";
import type { AgentRequest, AgentResponse, ConversationMessage, HandoffReason } from "@agent/types";
import type { Skill } from "@skills/skill";
import type { TransitionMatrix } from "@skills/transition-matrix";
import { getSkillJudgeConfig } from "./judges/skill-config.js";
export interface EvalDispatchResult {
chain: string[];
finalSkill: string;
response: AgentResponse;
}
function asRecord(value: unknown): Record<string, unknown> {
return typeof value === "object" && value !== null
? (value as Record<string, unknown>)
: {};
}
const HANDOFF_REASONS = [
"out-of-scope",
"missing-tool",
"low-confidence",
"validation-failed",
"same-agent-revisit",
"max-hop",
"high-risk-action",
"policy-risk",
] as const;
function isHandoffReason(value: unknown): value is HandoffReason {
return HANDOFF_REASONS.includes(value as HandoffReason);
}
function readPositiveNumber(
context: Record<string, unknown> | undefined,
key: string,
fallback: number,
): number {
const value = context?.[key];
return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : fallback;
}
function readPlan(request: AgentRequest): EvalDispatchPlanStep[] {
const rawPlan = asRecord(request.context)["evalHandoffPlan"];
if (!Array.isArray(rawPlan)) {
return [];
}
return rawPlan.flatMap((item): EvalDispatchPlanStep[] => {
const record = asRecord(item);
const from = record["from"];
const to = record["to"];
const reason = record["reason"];
if (
typeof from !== "string" ||
typeof to !== "string" ||
!isHandoffReason(reason)
) {
return [];
}
return [
{
confidence: typeof record["confidence"] === "number" ? record["confidence"] : undefined,
expectedOutput: typeof record["expectedOutput"] === "string" ? record["expectedOutput"] : undefined,
from,
reason,
risk:
record["risk"] === "low" ||
record["risk"] === "medium" ||
record["risk"] === "high"
? record["risk"]
: undefined,
taskSpec: typeof record["taskSpec"] === "string" ? record["taskSpec"] : undefined,
to,
},
];
});
}
function asStringArray(value: unknown): string[] {
return Array.isArray(value)
? value.filter((item): item is string => typeof item === "string")
: [];
}
export async function runEvalDispatch(input: {
history?: ConversationMessage[] | undefined;
request: AgentRequest;
skills: readonly Skill[];
transitionMatrix?: TransitionMatrix | undefined;
}): Promise<EvalDispatchResult> {
const context = input.request.context;
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: readPositiveNumber(context, "evalBudgetMaxCostUsd", 0.5),
maxHandoffs: readPositiveNumber(context, "evalBudgetMaxHandoffs", 4),
maxSameAgentVisits: readPositiveNumber(
context,
"evalBudgetMaxSameAgentVisits",
2,
),
maxToolCalls: readPositiveNumber(context, "evalBudgetMaxToolCalls", 20),
});
const skill = selectSkillForRequest(input.skills, input.request);
if (!skill) {
throw new Error(`No skill can handle eval request: ${input.request.skill}`);
}
budget.recordVisit(skill.name);
const plan: EvalDispatchPlan = {
maxDepth: readPositiveNumber(context, "evalMaxDispatchDepth", 8),
steps: readPlan(input.request),
useActualSkills: asRecord(context)["evalUseActualSkills"] === true,
};
const response = await dispatch(
{
budget,
chain: [skill.name],
depth: 0,
history: input.history ?? [],
request: input.request,
skill,
},
{
evalPlan: plan,
getSkillJudgeConfig,
selectSkillForRequest,
shouldApplyRuntimeVerifier: () => false,
skills: input.skills,
transitionMatrix: input.transitionMatrix,
},
);
const chain = asStringArray(response.metadata?.["handoffChain"]);
return {
chain,
finalSkill: chain.at(-1) ?? skill.name,
response,
};
}
+6 -12
View File
@@ -2,13 +2,7 @@ import type {
ProcessTraceSnapshot, ProcessTraceSnapshot,
ProcessTraceStepName, ProcessTraceStepName,
} from "@skills/onboarding/runtime/trace"; } from "@skills/onboarding/runtime/trace";
import { import type { EvalArchiveEntry, MongoConnectionLike } from "../archive.js";
getArchiveDerivedIntent,
getArchiveDerivedPhase,
getArchiveRoutingDecision,
type EvalArchiveEntry,
type MongoConnectionLike,
} from "../archive.js";
import type { SliceKey } from "../aggregator.js"; import type { SliceKey } from "../aggregator.js";
import type { PrmModel, PrmFeatureVector } from "./prm.js"; import type { PrmModel, PrmFeatureVector } from "./prm.js";
import { import {
@@ -142,14 +136,14 @@ export function attributeTraceWithPrm(
function sliceFromEntry(entry: EvalArchiveEntry): SliceKey { function sliceFromEntry(entry: EvalArchiveEntry): SliceKey {
const abstention = (entry.process as ProcessTraceSnapshot | undefined) const abstention = (entry.process as ProcessTraceSnapshot | undefined)
?.steps.classify?.output.abstention; ?.steps.classify?.output.abstention;
const routingDecision = getArchiveRoutingDecision(entry.derived);
return { return {
skill: entry.skill || entry.request.skill || "onboarding",
tier: entry.tier, tier: entry.tier,
intent: getArchiveDerivedIntent(entry.derived), intent: entry.derived.intent.kind,
phase: getArchiveDerivedPhase(entry.derived), phase: entry.derived.phase.kind,
...(routingDecision ? { routingDecision } : {}), ...(entry.derived.routingDecision
? { routingDecision: entry.derived.routingDecision.kind }
: {}),
...(abstention?.action.kind !== undefined && ...(abstention?.action.kind !== undefined &&
abstention.action.kind !== "proceed" abstention.action.kind !== "proceed"
? { abstentionAction: abstention.action.kind } ? { abstentionAction: abstention.action.kind }
-102
View File
@@ -1,102 +0,0 @@
import { describe, expect, it } from "vitest";
import type { AgentResponse } from "@agent/types";
import { judgeHandoff } from "./handoff-judge.js";
const response: AgentResponse = {
confidence: 0.8,
metadata: {
handoffChain: ["onboarding", "moderation", "supervisor"],
handoffLastRequest: {
confidence: 0.6,
evidenceRefs: [],
expectedOutput: "text",
reason: "same-agent-revisit",
risk: "medium",
targetAgent: "supervisor",
taskSpec: "loop blocked",
},
},
requestId: "request-1",
source: "template",
text: "sent to review",
};
describe("judgeHandoff", () => {
it("passes exact chain, final skill, target, and reason checks", () => {
expect(
judgeHandoff(
{
expectedFinalSkill: "supervisor",
expectedHandoff: {
expectedReason: "same-agent-revisit",
expectedTarget: "supervisor",
shouldHandoff: true,
},
expectedSourceChain: {
chain: ["onboarding", "moderation", "supervisor"],
mode: "exact",
},
},
response,
),
).toMatchObject({
actualChain: ["onboarding", "moderation", "supervisor"],
actualFinalSkill: "supervisor",
actualHandoffReason: "same-agent-revisit",
actualHandoffTarget: "supervisor",
passed: true,
});
});
it("supports prefix chain checks", () => {
expect(
judgeHandoff(
{
expectedSourceChain: {
chain: ["onboarding", "moderation"],
mode: "prefix",
},
},
response,
).passed,
).toBe(true);
});
it("supports contains chain checks", () => {
expect(
judgeHandoff(
{
expectedSourceChain: {
chain: ["moderation"],
mode: "contains",
},
},
response,
).passed,
).toBe(true);
});
it("fails when required metadata is missing", () => {
const result = judgeHandoff(
{
expectedFinalSkill: "supervisor",
expectedHandoff: { shouldHandoff: true },
expectedSourceChain: {
chain: ["supervisor"],
mode: "contains",
},
},
{ ...response, metadata: {} },
);
expect(result.passed).toBe(false);
expect(result.checks.filter((check) => !check.passed)).toHaveLength(3);
});
it("skips unspecified checks", () => {
const result = judgeHandoff({}, response);
expect(result.passed).toBe(true);
expect(result.checks.every((check) => check.skipped)).toBe(true);
});
});
-152
View File
@@ -1,152 +0,0 @@
import type { AgentResponse, HandoffRequest } from "@agent/types";
import type {
EvalCaseExpected,
HandoffJudgeCheck,
HandoffJudgeResult,
} from "../types.js";
function asStringArray(value: unknown): string[] {
if (!Array.isArray(value)) {
return [];
}
return value.filter((item): item is string => typeof item === "string");
}
function readHandoffRequest(
value: unknown,
): Pick<HandoffRequest, "reason" | "targetAgent"> | null {
if (typeof value !== "object" || value === null) {
return null;
}
const record = value as Partial<HandoffRequest>;
if (typeof record.reason !== "string" || typeof record.targetAgent !== "string") {
return null;
}
return {
reason: record.reason,
targetAgent: record.targetAgent,
};
}
function chainMatches(
actual: readonly string[],
expected: readonly string[],
mode: "contains" | "exact" | "prefix",
): boolean {
if (mode === "exact") {
return (
actual.length === expected.length &&
expected.every((skill, index) => actual[index] === skill)
);
}
if (mode === "prefix") {
return expected.every((skill, index) => actual[index] === skill);
}
if (expected.length === 0) {
return true;
}
return actual.some((_skill, startIndex) =>
expected.every((skill, offset) => actual[startIndex + offset] === skill),
);
}
function skippedCheck(name: HandoffJudgeCheck["name"]): HandoffJudgeCheck {
return {
name,
passed: true,
skipped: true,
};
}
export function judgeHandoff(
expected: EvalCaseExpected,
response: AgentResponse,
finalSkill?: string,
): HandoffJudgeResult {
const metadata = response.metadata ?? {};
const actualChain = asStringArray(metadata["handoffChain"]);
const finalSkillFromChain = actualChain.at(-1);
const actualFinalSkill =
finalSkill ?? (typeof finalSkillFromChain === "string" ? finalSkillFromChain : "");
const handoff =
readHandoffRequest(metadata["handoffRequest"]) ??
readHandoffRequest(metadata["handoffLastRequest"]);
const actualHandoffReason = handoff?.reason;
const actualHandoffTarget = handoff?.targetAgent;
const checks: HandoffJudgeCheck[] = [];
if (expected.expectedHandoff?.shouldHandoff === undefined) {
checks.push(skippedCheck("should-handoff"));
} else {
const didHandoff = actualChain.length > 1 || Boolean(handoff);
const passed = didHandoff === expected.expectedHandoff.shouldHandoff;
checks.push({
detail: `expected=${expected.expectedHandoff.shouldHandoff} actual=${didHandoff}`,
name: "should-handoff",
passed,
skipped: false,
});
}
if (!expected.expectedHandoff?.expectedReason) {
checks.push(skippedCheck("handoff-reason"));
} else {
const passed = actualHandoffReason === expected.expectedHandoff.expectedReason;
checks.push({
detail: `expected=${expected.expectedHandoff.expectedReason} actual=${actualHandoffReason ?? "-"}`,
name: "handoff-reason",
passed,
skipped: false,
});
}
if (!expected.expectedHandoff?.expectedTarget) {
checks.push(skippedCheck("handoff-target"));
} else {
const passed = actualHandoffTarget === expected.expectedHandoff.expectedTarget;
checks.push({
detail: `expected=${expected.expectedHandoff.expectedTarget} actual=${actualHandoffTarget ?? "-"}`,
name: "handoff-target",
passed,
skipped: false,
});
}
if (!expected.expectedFinalSkill) {
checks.push(skippedCheck("final-skill"));
} else {
const passed = actualFinalSkill === expected.expectedFinalSkill;
checks.push({
detail: `expected=${expected.expectedFinalSkill} actual=${actualFinalSkill || "-"}`,
name: "final-skill",
passed,
skipped: false,
});
}
if (!expected.expectedSourceChain) {
checks.push(skippedCheck("chain"));
} else {
const passed = chainMatches(
actualChain,
expected.expectedSourceChain.chain,
expected.expectedSourceChain.mode,
);
checks.push({
detail: `mode=${expected.expectedSourceChain.mode} expected=${expected.expectedSourceChain.chain.join(">")} actual=${actualChain.join(">") || "-"}`,
name: "chain",
passed,
skipped: false,
});
}
return {
actualChain,
actualFinalSkill,
...(actualHandoffReason ? { actualHandoffReason } : {}),
...(actualHandoffTarget ? { actualHandoffTarget } : {}),
checks,
passed: checks.every((check) => check.skipped || check.passed),
};
}
-23
View File
@@ -93,29 +93,6 @@ describe("judgeOnboardingWithLlm", () => {
} }
}); });
it("keeps scores when rationale is longer than the prompt asks for", async () => {
const result = await judgeOnboardingWithLlm(
new FakeLlmClient(
JSON.stringify({
friendliness: 3,
accuracy: 4,
concision: 5,
scope: 4,
rationale: "가".repeat(800),
}),
),
evalCase,
"이제 방 하트 인증만 남았어요.",
derived,
);
expect(result.skipped).toBe(false);
if (!result.skipped) {
expect(result.scores.overall).toBe(4);
expect(result.rationale).toHaveLength(500);
}
});
it("skips when the judge returns invalid JSON", async () => { it("skips when the judge returns invalid JSON", async () => {
const result = await judgeOnboardingWithLlm( const result = await judgeOnboardingWithLlm(
new FakeLlmClient('{"friendliness":"bad"}'), new FakeLlmClient('{"friendliness":"bad"}'),
+25 -55
View File
@@ -7,43 +7,29 @@ import type {
LlmJudgeResult, LlmJudgeResult,
LlmJudgeScores, LlmJudgeScores,
} from "../types.js"; } from "../types.js";
import { getSkillJudgeConfig } from "./skill-config.js";
const MAX_RATIONALE_CHARS = 500; const llmJudgeSchema = z.object({
friendliness: z.number().int().min(1).max(5),
accuracy: z.number().int().min(1).max(5),
concision: z.number().int().min(1).max(5),
scope: z.number().int().min(1).max(5),
rationale: z.string().trim().min(1).max(200),
});
function round(value: number): number { function round(value: number): number {
return Number(value.toFixed(3)); return Number(value.toFixed(3));
} }
function createJudgeSchema(axes: readonly string[]): z.ZodObject<z.ZodRawShape> {
return z.object({
...Object.fromEntries(
axes.map((axis) => [axis, z.number().int().min(1).max(5)]),
),
rationale: z.string().trim().min(1),
});
}
function normalizeRationale(value: unknown): string {
return String(value).slice(0, MAX_RATIONALE_CHARS);
}
function buildJudgePrompt( function buildJudgePrompt(
skillName: string,
evalCase: EvalCaseDefinition, evalCase: EvalCaseDefinition,
responseText: string, responseText: string,
derived: EvalDerivedSnapshot, derived: EvalDerivedSnapshot,
): string { ): string {
const config = getSkillJudgeConfig(skillName);
const axes = config.axes.join(", ");
const scoreShape = Object.fromEntries(config.axes.map((axis) => [axis, 1]));
return [ return [
`다음 ${skillName} 에이전트 응답을 평가해 주세요.`, "다음 온보딩 에이전트 응답을 평가해 주세요.",
`평가 축은 ${axes} 입니다.`, "평가 축은 friendliness, accuracy, concision, scope 입니다.",
"각 점수는 1~5의 정수만 사용하세요.", "각 점수는 1~5의 정수만 사용하세요.",
"rationale은 200자 이하로 짧게 작성하세요.", "scope는 사용자가 묻지 않은 추가 설명을 과하게 붙이지 않았는지 포함해 평가하세요.",
config.promptTemplate ??
"accuracy는 기대 조건과 실제 답변의 사실성을, concision은 과도한 부가 설명 여부를 포함해 평가하세요.",
"", "",
"[request]", "[request]",
JSON.stringify(evalCase.input.request), JSON.stringify(evalCase.input.request),
@@ -62,13 +48,12 @@ function buildJudgePrompt(
"[assistant_response]", "[assistant_response]",
responseText, responseText,
"", "",
`JSON object only: ${JSON.stringify({ ...scoreShape, rationale: "..." })}`, 'JSON object only: {"friendliness":1,"accuracy":1,"concision":1,"scope":1,"rationale":"..."}',
].join("\n"); ].join("\n");
} }
export async function judgeSkillWithLlm( export async function judgeOnboardingWithLlm(
llmClient: LlmClient, llmClient: LlmClient,
skillName: string,
evalCase: EvalCaseDefinition, evalCase: EvalCaseDefinition,
responseText: string, responseText: string,
derived: EvalDerivedSnapshot, derived: EvalDerivedSnapshot,
@@ -92,11 +77,11 @@ export async function judgeSkillWithLlm(
{ {
role: "system", role: "system",
content: content:
`당신은 ${skillName} 응답 품질 평가자입니다. 반드시 JSON object만 출력하세요.`, "당신은 온보딩 응답 품질 평가자입니다. 반드시 JSON object만 출력하세요.",
}, },
{ {
role: "user", role: "user",
content: buildJudgePrompt(skillName, evalCase, responseText, derived), content: buildJudgePrompt(evalCase, responseText, derived),
}, },
], ],
{ {
@@ -115,18 +100,18 @@ export async function judgeSkillWithLlm(
} }
try { try {
const config = getSkillJudgeConfig(skillName); const parsed = llmJudgeSchema.parse(JSON.parse(response.content));
const parsed = createJudgeSchema(config.axes).parse(
JSON.parse(response.content),
);
const axisScores = Object.fromEntries(
config.axes.map((axis) => [axis, Number(parsed[axis])]),
);
const scores: LlmJudgeScores = { const scores: LlmJudgeScores = {
...axisScores, friendliness: parsed.friendliness,
accuracy: parsed.accuracy,
concision: parsed.concision,
scope: parsed.scope,
overall: round( overall: round(
config.axes.reduce((sum, axis) => sum + Number(parsed[axis]), 0) / (parsed.friendliness +
config.axes.length, parsed.accuracy +
parsed.concision +
parsed.scope) /
4,
), ),
}; };
@@ -136,7 +121,7 @@ export async function judgeSkillWithLlm(
endpoint: response.endpoint, endpoint: response.endpoint,
provider: response.provider, provider: response.provider,
scores, scores,
rationale: normalizeRationale(parsed["rationale"]), rationale: parsed.rationale,
raw: response.content, raw: response.content,
}; };
} catch { } catch {
@@ -147,18 +132,3 @@ export async function judgeSkillWithLlm(
}; };
} }
} }
export async function judgeOnboardingWithLlm(
llmClient: LlmClient,
evalCase: EvalCaseDefinition,
responseText: string,
derived: EvalDerivedSnapshot,
): Promise<LlmJudgeResult> {
return judgeSkillWithLlm(
llmClient,
"onboarding",
evalCase,
responseText,
derived,
);
}
-95
View File
@@ -1,95 +0,0 @@
import type { AgentResponse } from "@agent/types";
import type { RetrievedDocument } from "../../memory/memory-types.js";
import type { UserProfile } from "@skills/onboarding/state/user-memory";
export interface MemoryJudgeExpected {
expectedArchivedDocPattern?: string | undefined;
expectedContradictionResolved?: boolean | undefined;
expectedProphylacticHint?: boolean | undefined;
expectedRecalledFact?: string | undefined;
expectedRetrievedDocIds?: string[] | undefined;
expectedSupervisorTarget?: string | undefined;
}
export interface MemoryJudgeResult {
checks: Array<{
detail?: string | undefined;
name: keyof MemoryJudgeExpected;
passed: boolean;
skipped: boolean;
}>;
passed: boolean;
score: number;
}
export function judgeMemory(
expected: MemoryJudgeExpected | undefined,
response: AgentResponse,
metadata?: {
profile?: UserProfile | undefined;
retrievedDocs?: RetrievedDocument[] | undefined;
},
): MemoryJudgeResult {
if (!expected || Object.keys(expected).length === 0) {
return { checks: [], passed: true, score: 1 };
}
const checks: MemoryJudgeResult["checks"] = [];
if (expected.expectedRecalledFact) {
checks.push({
name: "expectedRecalledFact",
passed: response.text.includes(expected.expectedRecalledFact),
skipped: false,
});
}
if (expected.expectedProphylacticHint !== undefined) {
const hasHint = /(예시|형식|가이드|주의|먼저)/u.test(response.text);
checks.push({
name: "expectedProphylacticHint",
passed: expected.expectedProphylacticHint ? hasHint : !hasHint,
skipped: false,
});
}
if (expected.expectedSupervisorTarget) {
const suggestedTarget = response.metadata?.["suggestedTarget"];
checks.push({
detail: typeof suggestedTarget === "string" ? suggestedTarget : "",
name: "expectedSupervisorTarget",
passed: suggestedTarget === expected.expectedSupervisorTarget,
skipped: false,
});
}
if (expected.expectedRetrievedDocIds) {
const ids = new Set(metadata?.retrievedDocs?.map((doc) => doc.id) ?? []);
checks.push({
name: "expectedRetrievedDocIds",
passed: expected.expectedRetrievedDocIds.every((id) => ids.has(id)),
skipped: ids.size === 0,
});
}
if (expected.expectedArchivedDocPattern) {
checks.push({
name: "expectedArchivedDocPattern",
passed: !response.text.includes(expected.expectedArchivedDocPattern),
skipped: false,
});
}
if (expected.expectedContradictionResolved !== undefined) {
checks.push({
name: "expectedContradictionResolved",
passed: expected.expectedContradictionResolved
? !/(예전|이전|기존)/u.test(response.text)
: true,
skipped: false,
});
}
const activeChecks = checks.filter((check) => !check.skipped);
const passedCount = activeChecks.filter((check) => check.passed).length;
const score = activeChecks.length === 0 ? 1 : passedCount / activeChecks.length;
return {
checks,
passed: score === 1,
score,
};
}
+1 -10
View File
@@ -5,7 +5,6 @@ import { tmpdir } from "node:os";
import { afterEach, describe, expect, it } from "vitest"; import { afterEach, describe, expect, it } from "vitest";
import type { ProcessTraceSnapshot } from "@skills/onboarding/runtime/trace"; import type { ProcessTraceSnapshot } from "@skills/onboarding/runtime/trace";
import { createOnboardingToolRegistry } from "@skills/onboarding/tools";
import { judgeProcessRule } from "./process-judge.js"; import { judgeProcessRule } from "./process-judge.js";
import { PRM_FEATURE_NAMES, savePrmModel } from "./prm.js"; import { PRM_FEATURE_NAMES, savePrmModel } from "./prm.js";
@@ -87,8 +86,6 @@ function makeTrace(
} }
describe("judgeProcessRule", () => { describe("judgeProcessRule", () => {
const toolRegistry = createOnboardingToolRegistry();
afterEach(() => { afterEach(() => {
delete process.env["EVAL_PRM_ENABLED"]; delete process.env["EVAL_PRM_ENABLED"];
delete process.env["EVAL_PRM_WEIGHTS_PATH"]; delete process.env["EVAL_PRM_WEIGHTS_PATH"];
@@ -105,7 +102,6 @@ describe("judgeProcessRule", () => {
}, },
}, },
makeTrace(), makeTrace(),
toolRegistry,
); );
expect(result.score).toBe(1); expect(result.score).toBe(1);
@@ -126,7 +122,6 @@ describe("judgeProcessRule", () => {
}, },
}, },
makeTrace(), makeTrace(),
toolRegistry,
); );
expect( expect(
@@ -160,7 +155,6 @@ describe("judgeProcessRule", () => {
}, },
], ],
}), }),
toolRegistry,
); );
expect( expect(
@@ -186,7 +180,6 @@ describe("judgeProcessRule", () => {
}, },
}, },
}), }),
toolRegistry,
); );
expect( expect(
@@ -221,7 +214,6 @@ describe("judgeProcessRule", () => {
}, },
}, },
}), }),
toolRegistry,
); );
expect( expect(
@@ -234,7 +226,7 @@ describe("judgeProcessRule", () => {
}); });
it("skips optional expectations when a golden case has not opted in", () => { it("skips optional expectations when a golden case has not opted in", () => {
const result = judgeProcessRule({}, makeTrace(), toolRegistry); const result = judgeProcessRule({}, makeTrace());
expect( expect(
result.checks.find((check) => check.rule === "intent-match")?.skipped, result.checks.find((check) => check.rule === "intent-match")?.skipped,
@@ -269,7 +261,6 @@ describe("judgeProcessRule", () => {
}, },
}, },
}), }),
toolRegistry,
); );
expect(result.prmFeatures).toMatchObject({ expect(result.prmFeatures).toMatchObject({
+11 -14
View File
@@ -2,7 +2,8 @@ import { z } from "zod";
import type { AgentRequest } from "@agent/types"; import type { AgentRequest } from "@agent/types";
import type { LlmClient } from "@llm/llm-client"; import type { LlmClient } from "@llm/llm-client";
import type { ToolRegistry } from "@tools/registry"; import { createBuiltInToolRegistry } from "@tools/builtins/index";
import type { ProcessTraceSnapshot } from "@skills/onboarding/runtime/trace";
import { import {
getEvalProcessLlmJudgeMaxTokens, getEvalProcessLlmJudgeMaxTokens,
getEvalPrmEnabled, getEvalPrmEnabled,
@@ -11,7 +12,6 @@ import {
} from "../../config.js"; } from "../../config.js";
import type { import type {
EvalCaseExpected, EvalCaseExpected,
EvalProcessTraceSnapshot,
ProcessJudgeCheck, ProcessJudgeCheck,
ProcessJudgeResult, ProcessJudgeResult,
} from "../types.js"; } from "../types.js";
@@ -80,7 +80,7 @@ function finalize(checks: ProcessJudgeCheck[], rationale?: string): ProcessJudge
function enrichWithPrm( function enrichWithPrm(
result: ProcessJudgeResult, result: ProcessJudgeResult,
trace: EvalProcessTraceSnapshot, trace: ProcessTraceSnapshot,
): ProcessJudgeResult { ): ProcessJudgeResult {
if (!getEvalPrmEnabled()) { if (!getEvalPrmEnabled()) {
return result; return result;
@@ -125,7 +125,7 @@ function hasAllTools(
return subset && actual.size === expected.size; return subset && actual.size === expected.size;
} }
function countRecordedSteps(trace: EvalProcessTraceSnapshot): number { function countRecordedSteps(trace: ProcessTraceSnapshot): number {
const scalarSteps = [ const scalarSteps = [
trace.steps.classify, trace.steps.classify,
trace.steps.derive, trace.steps.derive,
@@ -138,20 +138,18 @@ function countRecordedSteps(trace: EvalProcessTraceSnapshot): number {
return scalarSteps + (trace.steps.toolCalls?.length ?? 0); return scalarSteps + (trace.steps.toolCalls?.length ?? 0);
} }
function validateToolArgs( function validateToolArgs(trace: ProcessTraceSnapshot): ProcessJudgeCheck {
trace: EvalProcessTraceSnapshot,
toolRegistry: ToolRegistry,
): ProcessJudgeCheck {
const toolCalls = trace.steps.toolCalls ?? []; const toolCalls = trace.steps.toolCalls ?? [];
if (toolCalls.length === 0) { if (toolCalls.length === 0) {
return makeSkippedCheck("tool-call", "tool-args-valid", "no tool calls"); return makeSkippedCheck("tool-call", "tool-args-valid", "no tool calls");
} }
const registry = createBuiltInToolRegistry();
const failures: string[] = []; const failures: string[] = [];
for (const call of toolCalls) { for (const call of toolCalls) {
try { try {
toolRegistry.get(call.output.name).argsSchema.parse(call.output.args); registry.get(call.output.name).argsSchema.parse(call.output.args);
} catch { } catch {
failures.push(call.output.name); failures.push(call.output.name);
} }
@@ -169,8 +167,7 @@ function validateToolArgs(
export function judgeProcessRule( export function judgeProcessRule(
expected: EvalCaseExpected, expected: EvalCaseExpected,
trace: EvalProcessTraceSnapshot, trace: ProcessTraceSnapshot,
toolRegistry: ToolRegistry,
): ProcessJudgeResult { ): ProcessJudgeResult {
const checks: ProcessJudgeCheck[] = []; const checks: ProcessJudgeCheck[] = [];
@@ -247,7 +244,7 @@ export function judgeProcessRule(
); );
} }
checks.push(validateToolArgs(trace, toolRegistry)); checks.push(validateToolArgs(trace));
const toolPlanStepCount = trace.steps.plan?.output.toolPlanStepCount; const toolPlanStepCount = trace.steps.plan?.output.toolPlanStepCount;
if (toolPlanStepCount !== undefined) { if (toolPlanStepCount !== undefined) {
@@ -321,7 +318,7 @@ export function judgeProcessRule(
} }
function buildProcessJudgePrompt( function buildProcessJudgePrompt(
trace: EvalProcessTraceSnapshot, trace: ProcessTraceSnapshot,
options: { finalText: string; request: AgentRequest }, options: { finalText: string; request: AgentRequest },
): string { ): string {
return [ return [
@@ -344,7 +341,7 @@ function buildProcessJudgePrompt(
export async function judgeProcessWithLlm( export async function judgeProcessWithLlm(
llmClient: LlmClient, llmClient: LlmClient,
trace: EvalProcessTraceSnapshot, trace: ProcessTraceSnapshot,
options: { finalText: string; request: AgentRequest }, options: { finalText: string; request: AgentRequest },
): Promise<ProcessJudgeResult> { ): Promise<ProcessJudgeResult> {
if (!llmClient.hasEndpoints()) { if (!llmClient.hasEndpoints()) {
+10 -31
View File
@@ -1,7 +1,6 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { judgeRuleCase } from "./rule-judge.js"; import { judgeOnboardingRuleCase } from "./rule-judge.js";
import { getSkillJudgeConfig } from "./skill-config.js";
import type { DerivedState } from "@skills/onboarding/state/onboarding-state"; import type { DerivedState } from "@skills/onboarding/state/onboarding-state";
function makeDerived( function makeDerived(
@@ -19,12 +18,9 @@ function makeDerived(
}; };
} }
describe("judgeRuleCase", () => { describe("judgeOnboardingRuleCase", () => {
const validator = getSkillJudgeConfig("onboarding").validator;
it("passes when active checks match", () => { it("passes when active checks match", () => {
const derived = makeDerived(); const result = judgeOnboardingRuleCase(
const result = judgeRuleCase(
{ {
intent: "status-question", intent: "status-question",
responseMode: "llm", responseMode: "llm",
@@ -32,12 +28,7 @@ describe("judgeRuleCase", () => {
mustNotInclude: ["디스코드 서버"], mustNotInclude: ["디스코드 서버"],
}, },
"이제 방 하트 인증만 남았어요.", "이제 방 하트 인증만 남았어요.",
{ makeDerived(),
derived,
intent: derived.intent.kind,
responseMode: derived.responseMode,
validator,
},
); );
expect(result.passed).toBe(true); expect(result.passed).toBe(true);
@@ -46,19 +37,13 @@ describe("judgeRuleCase", () => {
}); });
it("fails intent and mustNotInclude mismatches", () => { it("fails intent and mustNotInclude mismatches", () => {
const derived = makeDerived(); const result = judgeOnboardingRuleCase(
const result = judgeRuleCase(
{ {
intent: "help-how", intent: "help-how",
mustNotInclude: ["디스코드"], mustNotInclude: ["디스코드"],
}, },
"디스코드 서버에 먼저 들어오시면 돼요.", "디스코드 서버에 먼저 들어오시면 돼요.",
{ makeDerived(),
derived,
intent: derived.intent.kind,
responseMode: derived.responseMode,
validator,
},
); );
expect(result.passed).toBe(false); expect(result.passed).toBe(false);
@@ -71,18 +56,12 @@ describe("judgeRuleCase", () => {
}); });
it("treats rejecting validator results as a failed rule check", () => { it("treats rejecting validator results as a failed rule check", () => {
const derived = makeDerived({ const result = judgeOnboardingRuleCase(
intent: { kind: "photo-question" },
});
const result = judgeRuleCase(
{}, {},
"디스코드 서버에 들어가시면 돼요.", "디스코드 서버에 들어가시면 돼요.",
{ makeDerived({
derived, intent: { kind: "photo-question" },
intent: derived.intent.kind, }),
responseMode: derived.responseMode,
validator,
},
); );
expect(result.passed).toBe(false); expect(result.passed).toBe(false);
+23 -36
View File
@@ -1,5 +1,6 @@
import { validateOnboardingResponse } from "@skills/onboarding/runtime/validator";
import type { DerivedState } from "@skills/onboarding/state/onboarding-state";
import type { EvalCaseExpected, RuleJudgeCheck, RuleJudgeResult } from "../types.js"; import type { EvalCaseExpected, RuleJudgeCheck, RuleJudgeResult } from "../types.js";
import type { SkillJudgeConfig } from "./skill-config.js";
function normalizeText(value: string): string { function normalizeText(value: string): string {
return value.trim().toLowerCase(); return value.trim().toLowerCase();
@@ -9,47 +10,33 @@ function round(value: number): number {
return Number(value.toFixed(3)); return Number(value.toFixed(3));
} }
export function judgeRuleCase( export function judgeOnboardingRuleCase(
expected: EvalCaseExpected, expected: EvalCaseExpected,
responseText: string, responseText: string,
actual: { derived: DerivedState,
derived?: unknown;
intent: string;
responseMode?: string | undefined;
validator?: SkillJudgeConfig["validator"] | undefined;
},
): RuleJudgeResult { ): RuleJudgeResult {
const validation = validateOnboardingResponse(responseText, derived, {
allowShortResponse: derived.allowShortResponse,
});
const checks: RuleJudgeCheck[] = []; const checks: RuleJudgeCheck[] = [];
const validation = actual.validator
? actual.validator(responseText, actual.derived)
: null;
if (validation) { checks.push({
checks.push({ name: "validator",
name: "validator", passed: validation.status !== "reject",
passed: validation.status !== "reject", skipped: false,
skipped: false, detail:
detail: validation.reasons.length > 0
validation.reasons.length > 0 ? validation.reasons.join(", ")
? validation.reasons.join(", ") : validation.status,
: validation.status, });
});
} else {
checks.push({
name: "validator",
passed: true,
skipped: true,
detail: "not configured for this skill",
});
}
const expectedIntent = expected.intent?.trim(); const expectedIntent = expected.intent?.trim();
if (expectedIntent) { if (expectedIntent) {
checks.push({ checks.push({
name: "intent", name: "intent",
passed: normalizeText(expectedIntent) === normalizeText(actual.intent), passed: normalizeText(expectedIntent) === normalizeText(derived.intent.kind),
skipped: false, skipped: false,
detail: `expected=${expectedIntent}, actual=${actual.intent}`, detail: `expected=${expectedIntent}, actual=${derived.intent.kind}`,
}); });
} else { } else {
checks.push({ checks.push({
@@ -60,14 +47,14 @@ export function judgeRuleCase(
} }
const expectedResponseMode = expected.responseMode?.trim(); const expectedResponseMode = expected.responseMode?.trim();
if (expectedResponseMode && actual.responseMode) { if (expectedResponseMode) {
checks.push({ checks.push({
name: "responseMode", name: "responseMode",
passed: passed:
normalizeText(expectedResponseMode) === normalizeText(expectedResponseMode) ===
normalizeText(actual.responseMode), normalizeText(derived.responseMode),
skipped: false, skipped: false,
detail: `expected=${expectedResponseMode}, actual=${actual.responseMode}`, detail: `expected=${expectedResponseMode}, actual=${derived.responseMode}`,
}); });
} else { } else {
checks.push({ checks.push({
@@ -128,7 +115,7 @@ export function judgeRuleCase(
passed: activeChecks.every((check) => check.passed), passed: activeChecks.every((check) => check.passed),
score: activeChecks.length === 0 ? 1 : round(passedChecks / activeChecks.length), score: activeChecks.length === 0 ? 1 : round(passedChecks / activeChecks.length),
checks, checks,
validatorStatus: validation?.status ?? "pass", validatorStatus: validation.status,
validatorReasons: validation?.reasons ?? [], validatorReasons: validation.reasons,
}; };
} }
-40
View File
@@ -1,40 +0,0 @@
import type { AgentRequest, AgentResponse } from "@agent/types";
import { validateOnboardingResponse } from "@skills/onboarding/runtime/validator";
import type { DerivedState } from "@skills/onboarding/state/onboarding-state";
export type RuntimeVerdict = "pass" | "retry" | "escalate";
export interface RuntimeVerifierInput {
derived?: unknown;
packet?: unknown;
request: AgentRequest;
response: AgentResponse;
skill: string;
}
export function verifyOnboardingRuntime(
input: RuntimeVerifierInput,
): Promise<RuntimeVerdict> {
const derived = input.derived as DerivedState | undefined;
if (!derived) {
return Promise.resolve("pass");
}
const validation = validateOnboardingResponse(
input.response.text.trim() ? input.response.text : "[NO_RESPONSE]",
derived,
{
allowShortResponse: derived.allowShortResponse,
},
);
return Promise.resolve(validation.status === "reject" ? "retry" : "pass");
}
export function verifyModerationRuntime(
input: RuntimeVerifierInput,
): Promise<RuntimeVerdict> {
const text = input.response.text;
if (/https?:\/\//iu.test(text) || /\b\d{2,3}[- ]?\d{3,4}[- ]?\d{4}\b/u.test(text)) {
return Promise.resolve("escalate");
}
return Promise.resolve("pass");
}
-55
View File
@@ -1,55 +0,0 @@
import type { ToolRegistry } from "@tools/registry";
import { createModerationToolRegistry } from "@skills/moderation/tools";
import { createOnboardingToolRegistry } from "@skills/onboarding/tools";
import {
validateOnboardingResponse,
type ValidationResult,
} from "@skills/onboarding/runtime/validator";
import type { DerivedState } from "@skills/onboarding/state/onboarding-state";
import {
verifyModerationRuntime,
verifyOnboardingRuntime,
type RuntimeVerifierInput,
type RuntimeVerdict,
} from "./runtime-verifier.js";
export type SkillValidationResult = ValidationResult;
export interface SkillJudgeConfig {
axes: readonly string[];
promptTemplate?: string | undefined;
toolRegistryFactory: () => ToolRegistry;
tournamentInstruction: string;
runtimeVerifier?: ((input: RuntimeVerifierInput) => Promise<RuntimeVerdict>) | undefined;
validator?: ((text: string, derived: unknown) => SkillValidationResult) | undefined;
weights?: Record<string, number> | undefined;
}
export const SKILL_JUDGE_CONFIGS: Record<string, SkillJudgeConfig> = {
moderation: {
axes: ["appropriateness", "accuracy", "helpfulness"],
toolRegistryFactory: createModerationToolRegistry,
tournamentInstruction:
"커뮤니티 규칙/운영 문의 응답으로서 적절성, 정확성, 도움됨을 비교하세요.",
runtimeVerifier: verifyModerationRuntime,
},
onboarding: {
axes: ["friendliness", "accuracy", "concision", "scope"],
toolRegistryFactory: createOnboardingToolRegistry,
tournamentInstruction:
"온보딩 응답으로서 친절함, 정확성, 간결함, 질문 범위 준수를 비교하세요.",
runtimeVerifier: verifyOnboardingRuntime,
validator: (text, derived) =>
validateOnboardingResponse(text, derived as DerivedState, {
allowShortResponse: (derived as DerivedState).allowShortResponse,
}),
},
};
export function getSkillJudgeConfig(skillName: string): SkillJudgeConfig {
const config = SKILL_JUDGE_CONFIGS[skillName] ?? SKILL_JUDGE_CONFIGS["onboarding"];
if (!config) {
throw new Error("Missing onboarding judge config");
}
return config;
}
-45
View File
@@ -146,51 +146,6 @@ describe("proposeVariants", () => {
}); });
}); });
it("normalizes a single diff object from LLM JSON", async () => {
const llmClient = {
hasEndpoints: () => true,
chatCompletion: () =>
Promise.resolve({
content: JSON.stringify({
prompts: [
{
description: "Clarify deadline status",
diff: {
layer: "policy",
op: "append",
text: "마감 임박 상태 질문에는 남은 필수 항목과 다음 행동을 먼저 말한다.",
},
rationale: "some proposal models emit one diff as an object",
expectedImprovement: "valid proposal manifests",
},
],
tools: [],
}),
endpoint: "mock",
message: { role: "assistant", content: "{}" },
provider: "cliproxy",
raw: {},
}),
} as unknown as LlmClient;
const proposals = await proposeVariants(llmClient, {
slice: {
tier: 1,
intent: "status-question",
phase: "deadline-near",
},
recentLabels: [],
worstCases: [archiveEntry()],
currentPromptSnapshot: "snapshot",
});
expect(proposals.prompts[0]?.diff).toHaveLength(1);
expect(proposals.prompts[0]?.diff[0]).toMatchObject({
layer: "policy",
op: "append",
});
});
it("summarizes PRM root causes for proposal context", () => { it("summarizes PRM root causes for proposal context", () => {
const validateCase = archiveEntry(); const validateCase = archiveEntry();
const toolCase = { const toolCase = {
+46 -52
View File
@@ -5,11 +5,15 @@ import { dirname, resolve } from "node:path";
import { z } from "zod"; import { z } from "zod";
import type { LlmClient } from "@llm/llm-client"; import type { LlmClient } from "@llm/llm-client";
import { SkillRegistry } from "@skills/skill-loader"; import { SkillRegistry } from "@skills/skill-loader";
import { createBuiltInToolRegistry } from "@tools/builtins/index";
import type { DerivedState, Intent, OnboardingPhase } from "@skills/onboarding/state/onboarding-state";
import type { Blackboard, PromptContext } from "@skills/onboarding/types";
import { import {
PromptBuilder,
type PromptLayerDiff, type PromptLayerDiff,
type PromptLayerName, type PromptLayerName,
type PromptVariantManifest, type PromptVariantManifest,
} from "@eval/prompt-variant"; } from "@skills/onboarding/prompts/builder";
import { import {
getEvalTournamentMinCases, getEvalTournamentMinCases,
getLlmProposeMaxTokens, getLlmProposeMaxTokens,
@@ -22,8 +26,6 @@ import { queryArchivedReviewLabels } from "../review/label-archive.js";
import type { ReviewLabel } from "../review/queue.js"; import type { ReviewLabel } from "../review/queue.js";
import { runG1Static, runG4Permission } from "../governance/gates.js"; import { runG1Static, runG4Permission } from "../governance/gates.js";
import { promoteTrustTier } from "../governance/trust.js"; import { promoteTrustTier } from "../governance/trust.js";
import { getSkillJudgeConfig } from "../judges/skill-config.js";
import { getSkillProposeAdapter } from "./skill-adapters.js";
export interface ProposalInput { export interface ProposalInput {
slice: SliceKey; slice: SliceKey;
@@ -68,22 +70,12 @@ const promptDiffSchema = z.object({
text: z.string().trim().max(4_000).optional(), text: z.string().trim().max(4_000).optional(),
}); });
const promptDiffListSchema = z.preprocess(
(value) => {
if (Array.isArray(value) || value === undefined) {
return value;
}
return [value];
},
z.array(promptDiffSchema).min(1).max(10),
);
const proposalSchema = z.object({ const proposalSchema = z.object({
prompts: z prompts: z
.array( .array(
z.object({ z.object({
description: z.string().trim().min(1).max(300), description: z.string().trim().min(1).max(300),
diff: promptDiffListSchema, diff: z.array(promptDiffSchema).min(1).max(10),
expectedImprovement: z.string().trim().min(1).max(500), expectedImprovement: z.string().trim().min(1).max(500),
rationale: z.string().trim().min(1).max(500), rationale: z.string().trim().min(1).max(500),
}), }),
@@ -127,12 +119,44 @@ function getLlmOverall(entry: EvalArchiveEntry): number | null {
return entry.llmJudge.skipped ? null : entry.llmJudge.scores.overall; return entry.llmJudge.skipped ? null : entry.llmJudge.scores.overall;
} }
function emptyBlackboard(): Blackboard {
return {
flaggedIssues: [],
openQuestions: [],
recentNicknameAttempts: [],
toneMemory: {},
};
}
function derivedFromSlice(slice: SliceKey): DerivedState {
return {
allowShortResponse: false,
intent: { kind: slice.intent } as Intent,
phase: { kind: slice.phase } as OnboardingPhase,
responseMode: "llm",
shouldRespond: true,
suggestedMaxTokens: 384,
tier: slice.tier,
};
}
function defaultPromptContext(): PromptContext {
return {
blackboard: emptyBlackboard(),
conversationHistory: [],
userMessage: "현재 slice의 약점을 개선할 수 있는 기본 응답 정책을 확인해줘.",
};
}
export function buildCurrentPromptSnapshot( export function buildCurrentPromptSnapshot(
slice: SliceKey, slice: SliceKey,
archiveEntry?: EvalArchiveEntry, archiveEntry?: EvalArchiveEntry,
): string { ): string {
return getSkillProposeAdapter(slice.skill ?? archiveEntry?.skill ?? "onboarding") return PromptBuilder.forDerived(
.buildSnapshot(slice, archiveEntry); archiveEntry?.derived ?? derivedFromSlice(slice),
null,
defaultPromptContext(),
).buildSystemPrompt();
} }
function summarizeArchiveEntry(entry: EvalArchiveEntry): Record<string, unknown> { function summarizeArchiveEntry(entry: EvalArchiveEntry): Record<string, unknown> {
@@ -199,9 +223,8 @@ export function summarizeRootCauses(
} }
function buildProposalPrompt(input: ProposalInput): string { function buildProposalPrompt(input: ProposalInput): string {
const skill = input.slice.skill ?? "onboarding";
return [ return [
`${skill} 에이전트의 약점 slice를 개선할 prompt/tool variant를 제안해 주세요.`, "온보딩 에이전트의 약점 slice를 개선할 prompt/tool variant를 제안해 주세요.",
"반드시 JSON object만 출력하세요.", "반드시 JSON object만 출력하세요.",
"prompt diff는 기존 system prompt layer에 append/replace/remove 중 하나로 적용됩니다.", "prompt diff는 기존 system prompt layer에 append/replace/remove 중 하나로 적용됩니다.",
"과도한 일반론보다 아래 실패 케이스의 반복 원인을 직접 줄이는 작은 변경을 선호하세요.", "과도한 일반론보다 아래 실패 케이스의 반복 원인을 직접 줄이는 작은 변경을 선호하세요.",
@@ -238,7 +261,7 @@ export async function proposeVariants(
{ {
role: "system", role: "system",
content: content:
`당신은 ${input.slice.skill ?? "onboarding"} 에이전트 품질 개선을 돕는 prompt/tool reviewer입니다. 반드시 JSON object만 출력하세요.`, "당신은 온보딩 에이전트 품질 개선을 돕는 prompt/tool reviewer입니다. 반드시 JSON object만 출력하세요.",
}, },
{ {
role: "user", role: "user",
@@ -263,12 +286,10 @@ export async function proposeVariants(
return { return {
...proposal, ...proposal,
diff, diff,
skill: input.slice.skill ?? "onboarding",
variantId: stableHash({ variantId: stableHash({
description: proposal.description, description: proposal.description,
diff, diff,
rationale: proposal.rationale, rationale: proposal.rationale,
skill: input.slice.skill ?? "onboarding",
}), }),
}; };
}); });
@@ -294,34 +315,11 @@ export async function buildProposalInputFromArchive(
}, },
): Promise<ProposalInput> { ): Promise<ProposalInput> {
const limit = options?.limit ?? 50; const limit = options?.limit ?? 50;
const skill = slice.skill ?? "onboarding";
const skillFilter =
skill === "onboarding"
? {
$or: [
{ skill },
{ skill: { $exists: false } },
],
}
: { skill };
const entries = await getEvalArchiveCollection(mongoConnection) const entries = await getEvalArchiveCollection(mongoConnection)
.find({ .find({
$and: [ tier: slice.tier,
skillFilter, "derived.intent.kind": slice.intent,
{ tier: slice.tier }, "derived.phase.kind": slice.phase,
{
$or: [
{
"derived.intent.kind": slice.intent,
"derived.phase.kind": slice.phase,
},
{
"derived.intent": slice.intent,
"derived.phase": slice.phase,
},
],
},
],
}) })
.sort({ evaluatedAt: -1 }) .sort({ evaluatedAt: -1 })
.limit(limit) .limit(limit)
@@ -365,17 +363,13 @@ export async function writePromptVariantProposal(
await mkdir(dirname(outputPath), { recursive: true }); await mkdir(dirname(outputPath), { recursive: true });
const manifest: PromptVariantProposal = { const manifest: PromptVariantProposal = {
...proposal, ...proposal,
skill: proposal.skill ?? "onboarding",
trustTier: proposal.trustTier ?? "T1", trustTier: proposal.trustTier ?? "T1",
}; };
const g1 = runG1Static(manifest); const g1 = runG1Static(manifest);
const skill = manifest.skill ?? "onboarding";
const judgeConfig = getSkillJudgeConfig(skill);
const g4 = runG4Permission( const g4 = runG4Permission(
manifest, manifest,
judgeConfig.toolRegistryFactory(), createBuiltInToolRegistry(),
SkillRegistry.load(resolve(process.cwd(), "src", "skills")), SkillRegistry.load(resolve(process.cwd(), "src", "skills")),
skill,
); );
const gatedManifest: PromptVariantProposal = { const gatedManifest: PromptVariantProposal = {
...manifest, ...manifest,
-16
View File
@@ -1,16 +0,0 @@
import { moderationProposeAdapter } from "@skills/moderation/eval/propose-adapter";
import { onboardingProposeAdapter } from "@skills/onboarding/eval/propose-adapter";
import type { SkillProposeAdapter } from "./skill-propose-adapter.js";
const SKILL_PROPOSE_ADAPTERS: Record<string, SkillProposeAdapter> = {
[moderationProposeAdapter.skill]: moderationProposeAdapter,
[onboardingProposeAdapter.skill]: onboardingProposeAdapter,
};
export function getSkillProposeAdapter(skill: string): SkillProposeAdapter {
const adapter = SKILL_PROPOSE_ADAPTERS[skill] ?? SKILL_PROPOSE_ADAPTERS["onboarding"];
if (!adapter) {
throw new Error("Missing onboarding propose adapter");
}
return adapter;
}
-38
View File
@@ -1,38 +0,0 @@
import type { AgentRequest } from "@agent/types";
import type { LlmClient } from "@llm/llm-client";
import type { ChatMessage } from "@llm/types";
import type { PromptVariantManifest } from "@eval/prompt-variant";
import type { EvalArchiveEntry } from "../archive.js";
import type { SliceKey } from "../aggregator.js";
export interface SkillProposeAdapter {
readonly skill: string;
buildSnapshot(slice: SliceKey, archiveEntry?: EvalArchiveEntry): string;
derivedFromSlice(slice: SliceKey): unknown;
generateResponse(
llmClient: LlmClient,
entry: EvalArchiveEntry,
variantId: string,
manifests?: Record<string, PromptVariantManifest>,
): Promise<string>;
}
export function getVariantManifest(
variantId: string,
manifests?: Record<string, PromptVariantManifest>,
): PromptVariantManifest | string | null {
if (variantId === "control") {
return null;
}
return manifests?.[variantId] ?? variantId;
}
export function basicPromptSnapshot(messages: ChatMessage[]): string {
return messages
.map((message) => `[${message.role}]\n${message.content}`)
.join("\n\n");
}
export function getRequestText(request: AgentRequest): string {
return request.userMessage?.trim() || "";
}
+62 -60
View File
@@ -2,12 +2,11 @@ import { createHash } from "node:crypto";
import { z } from "zod"; import { z } from "zod";
import type { LlmClient } from "@llm/llm-client"; import type { LlmClient } from "@llm/llm-client";
import type { PromptVariantManifest } from "@eval/prompt-variant"; import { PromptBuilder, type PromptVariantManifest } from "@skills/onboarding/prompts/builder";
import type { Blackboard, PromptContext } from "@skills/onboarding/types";
import { getEvalTournamentMaxMatches } from "../../config.js"; import { getEvalTournamentMaxMatches } from "../../config.js";
import type { EvalArchiveEntry, MongoConnectionLike } from "../archive.js"; import type { EvalArchiveEntry, MongoConnectionLike } from "../archive.js";
import { getEvalArchiveCollection } from "../archive.js"; import { getEvalArchiveCollection } from "../archive.js";
import { getSkillJudgeConfig } from "../judges/skill-config.js";
import { getSkillProposeAdapter } from "./skill-adapters.js";
import { import {
createEloRating, createEloRating,
scoreForVerdict, scoreForVerdict,
@@ -31,7 +30,6 @@ export interface TournamentMatch {
export interface TournamentResult { export interface TournamentResult {
generatedAt: string; generatedAt: string;
matches: TournamentMatch[]; matches: TournamentMatch[];
skill?: string | undefined;
variants: EloRating[]; variants: EloRating[];
} }
@@ -60,10 +58,6 @@ const sideBySideJudgeSchema = z.object({
verdict: z.enum(["A", "B", "tie"]), verdict: z.enum(["A", "B", "tie"]),
}); });
function entrySkill(entry: EvalArchiveEntry): string {
return entry.skill || entry.request.skill || "onboarding";
}
function stableHash(value: unknown): string { function stableHash(value: unknown): string {
return createHash("sha256") return createHash("sha256")
.update(JSON.stringify(value)) .update(JSON.stringify(value))
@@ -71,18 +65,56 @@ function stableHash(value: unknown): string {
.slice(0, 12); .slice(0, 12);
} }
function emptyBlackboard(): Blackboard {
return {
flaggedIssues: [],
openQuestions: [],
recentNicknameAttempts: [],
toneMemory: {},
};
}
function promptContextForCase(entry: EvalArchiveEntry): PromptContext {
return {
blackboard: emptyBlackboard(),
conversationHistory: [],
userMessage: entry.request.userMessage?.trim() || "필요한 온보딩 안내만 답해줘.",
};
}
function getVariantManifest(
variantId: string,
manifests?: Record<string, PromptVariantManifest>,
): PromptVariantManifest | string | null {
if (variantId === "control") {
return null;
}
return manifests?.[variantId] ?? variantId;
}
async function generateTournamentResponse( async function generateTournamentResponse(
llmClient: LlmClient, llmClient: LlmClient,
entry: EvalArchiveEntry, entry: EvalArchiveEntry,
variantId: string, variantId: string,
manifests?: Record<string, PromptVariantManifest>, manifests?: Record<string, PromptVariantManifest>,
): Promise<string> { ): Promise<string> {
return getSkillProposeAdapter(entrySkill(entry)).generateResponse( let builder = PromptBuilder.forDerived(
llmClient, entry.derived,
entry, null,
variantId, promptContextForCase(entry),
manifests,
); );
const manifest = getVariantManifest(variantId, manifests);
if (manifest) {
builder = builder.applyVariant(manifest);
}
const response = await llmClient.chatCompletion(builder.build(), {
maxTokens: entry.derived.suggestedMaxTokens,
temperature: 0,
tier: entry.derived.tier,
});
return response?.content.trim() ?? "";
} }
function buildJudgePrompt( function buildJudgePrompt(
@@ -90,12 +122,9 @@ function buildJudgePrompt(
responseA: string, responseA: string,
responseB: string, responseB: string,
): string { ): string {
const skill = entrySkill(entry);
const config = getSkillJudgeConfig(skill);
return [ return [
`아래 두 ${skill} 응답 중 더 나은 쪽을 고르세요.`, "아래 두 온보딩 응답 중 더 나은 쪽을 고르세요.",
config.tournamentInstruction, "평가 기준은 정확성, 질문 범위 준수, 친근함, 간결함입니다.",
`평가 축: ${config.axes.join(", ")}`,
"A/B 위치 편향 없이 판단하고, 둘 다 비슷하면 tie를 선택하세요.", "A/B 위치 편향 없이 판단하고, 둘 다 비슷하면 tie를 선택하세요.",
"", "",
"[request]", "[request]",
@@ -128,7 +157,7 @@ async function judgeTournamentMatch(
{ {
role: "system", role: "system",
content: content:
`당신은 ${entrySkill(entry)} 응답 A/B 테스트 심판입니다. 반드시 JSON object만 출력하세요.`, "당신은 온보딩 응답 A/B 테스트 심판입니다. 반드시 JSON object만 출력하세요.",
}, },
{ {
role: "user", role: "user",
@@ -172,7 +201,6 @@ async function ensureTournamentIndexes(
): Promise<void> { ): Promise<void> {
await Promise.all([ await Promise.all([
collection.createIndex({ generatedAt: -1 }), collection.createIndex({ generatedAt: -1 }),
collection.createIndex({ skill: 1, generatedAt: -1 }),
collection.createIndex({ "variants.id": 1, generatedAt: -1 }), collection.createIndex({ "variants.id": 1, generatedAt: -1 }),
]); ]);
} }
@@ -197,7 +225,6 @@ export async function runTournament(
cases: EvalArchiveEntry[]; cases: EvalArchiveEntry[];
maxMatches?: number | undefined; maxMatches?: number | undefined;
nowMs?: number | undefined; nowMs?: number | undefined;
skill?: string | undefined;
variantIds: string[]; variantIds: string[];
variantManifests?: Record<string, PromptVariantManifest> | undefined; variantManifests?: Record<string, PromptVariantManifest> | undefined;
}, },
@@ -276,11 +303,9 @@ export async function runTournament(
} }
} }
const skill = options.skill ?? options.cases[0]?.skill;
return { return {
generatedAt: new Date(options.nowMs ?? Date.now()).toISOString(), generatedAt: new Date(options.nowMs ?? Date.now()).toISOString(),
matches, matches,
...(skill ? { skill } : {}),
variants: [...ratings.values()].sort((left, right) => right.elo - left.elo), variants: [...ratings.values()].sort((left, right) => right.elo - left.elo),
}; };
} }
@@ -290,35 +315,22 @@ export async function loadTournamentCases(
options: { options: {
caseIds?: string[] | undefined; caseIds?: string[] | undefined;
limit?: number | undefined; limit?: number | undefined;
skill?: string | undefined;
}, },
): Promise<EvalArchiveEntry[]> { ): Promise<EvalArchiveEntry[]> {
const filter: Record<string, unknown> = {}; const filter =
if (options.caseIds && options.caseIds.length > 0) { options.caseIds && options.caseIds.length > 0
filter.streamId = { ? {
$in: options.caseIds, streamId: {
}; $in: options.caseIds,
} },
if (options.skill) { }
if (options.skill === "onboarding") { : {};
filter.$or = [
{ skill: options.skill },
{ skill: { $exists: false } },
];
} else {
filter.skill = options.skill;
}
}
const entries = await getEvalArchiveCollection(mongoConnection) return getEvalArchiveCollection(mongoConnection)
.find(filter) .find(filter)
.sort({ evaluatedAt: -1 }) .sort({ evaluatedAt: -1 })
.limit(options.limit ?? getEvalTournamentMaxMatches()) .limit(options.limit ?? getEvalTournamentMaxMatches())
.toArray(); .toArray();
return entries.map((entry) => ({
...entry,
skill: entry.skill || entry.request.skill || "onboarding",
}));
} }
export async function saveTournamentResult( export async function saveTournamentResult(
@@ -334,24 +346,14 @@ export async function queryTournamentResults(
mongoConnection: MongoConnectionLike, mongoConnection: MongoConnectionLike,
options?: { options?: {
limit?: number | undefined; limit?: number | undefined;
skill?: string | undefined;
variantId?: string | undefined; variantId?: string | undefined;
}, },
): Promise<TournamentResult[]> { ): Promise<TournamentResult[]> {
const filter: Record<string, unknown> = {}; const filter = options?.variantId
if (options?.variantId) { ? {
filter["variants.id"] = options.variantId; "variants.id": options.variantId,
} }
if (options?.skill) { : {};
if (options.skill === "onboarding") {
filter.$or = [
{ skill: options.skill },
{ skill: { $exists: false } },
];
} else {
filter.skill = options.skill;
}
}
return getTournamentCollection(mongoConnection) return getTournamentCollection(mongoConnection)
.find(filter) .find(filter)
+7 -15
View File
@@ -16,14 +16,11 @@ import {
} from "./anonymize.js"; } from "./anonymize.js";
import type { EvalCaseDefinition } from "./types.js"; import type { EvalCaseDefinition } from "./types.js";
import type { EvalArchiveEntry, MongoConnectionLike } from "./archive.js"; import type { EvalArchiveEntry, MongoConnectionLike } from "./archive.js";
import { import { getEvalArchiveCollection } from "./archive.js";
getArchiveDerivedIntent,
getArchiveDerivedPhase,
getEvalArchiveCollection,
} from "./archive.js";
import { queryArchivedReviewLabels } from "./review/label-archive.js"; import { queryArchivedReviewLabels } from "./review/label-archive.js";
import type { ReviewLabel } from "./review/queue.js"; import type { ReviewLabel } from "./review/queue.js";
const DEFAULT_CANDIDATE_PATH = "src/eval/golden/onboarding.candidate.jsonl";
const DAY_MS = 24 * 60 * 60 * 1000; const DAY_MS = 24 * 60 * 60 * 1000;
export interface PromotionCandidate { export interface PromotionCandidate {
@@ -50,10 +47,6 @@ function stableShortHash(value: string): string {
return createHash("sha256").update(value).digest("hex").slice(0, 8); return createHash("sha256").update(value).digest("hex").slice(0, 8);
} }
function defaultCandidatePathFor(skill: string): string {
return `src/eval/golden/${skill}/candidate.jsonl`;
}
function getLlmOverall(entry: EvalArchiveEntry): number | null { function getLlmOverall(entry: EvalArchiveEntry): number | null {
if (entry.llmJudge.skipped) { if (entry.llmJudge.skipped) {
return null; return null;
@@ -63,7 +56,7 @@ function getLlmOverall(entry: EvalArchiveEntry): number | null {
} }
function sliceKey(entry: EvalArchiveEntry): string { function sliceKey(entry: EvalArchiveEntry): string {
return `${entry.skill || entry.request.skill || "onboarding"}:${entry.tier}:${getArchiveDerivedIntent(entry.derived)}:${getArchiveDerivedPhase(entry.derived)}`; return `${entry.tier}:${entry.derived.intent.kind}:${entry.derived.phase.kind}`;
} }
function sortByLowestScore( function sortByLowestScore(
@@ -290,8 +283,8 @@ function candidateToEvalCase(candidate: PromotionCandidate): EvalCaseDefinition
candidate.labelRationale === undefined candidate.labelRationale === undefined
? null ? null
: `humanRationale=${candidate.labelRationale}`, : `humanRationale=${candidate.labelRationale}`,
`derivedIntent=${getArchiveDerivedIntent(candidate.derived)}`, `derivedIntent=${candidate.derived.intent.kind}`,
`derivedPhase=${getArchiveDerivedPhase(candidate.derived)}`, `derivedPhase=${candidate.derived.phase.kind}`,
`assistantResponse=${candidate.response.text}`, `assistantResponse=${candidate.response.text}`,
candidate.betterResponse === undefined candidate.betterResponse === undefined
? null ? null
@@ -333,10 +326,9 @@ function extractExistingCaseIds(raw: string): Set<string> {
export async function appendCandidatesToJsonl( export async function appendCandidatesToJsonl(
candidates: PromotionCandidate[], candidates: PromotionCandidate[],
outputPath?: string, outputPath = DEFAULT_CANDIDATE_PATH,
skill = "onboarding",
): Promise<{ appendedCount: number; outputPath: string }> { ): Promise<{ appendedCount: number; outputPath: string }> {
const resolved = resolve(process.cwd(), outputPath ?? defaultCandidatePathFor(skill)); const resolved = resolve(process.cwd(), outputPath);
await mkdir(dirname(resolved), { recursive: true }); await mkdir(dirname(resolved), { recursive: true });
const existingRaw = existsSync(resolved) const existingRaw = existsSync(resolved)
? await readFile(resolved, "utf8") ? await readFile(resolved, "utf8")
-70
View File
@@ -1,70 +0,0 @@
import { existsSync, readFileSync } from "node:fs";
import { resolve } from "node:path";
export type PromptLayerName =
| "context"
| "examples"
| "persona"
| "policy"
| "state"
| "tool-policy";
export type PromptLayerOperation = "append" | "remove" | "replace";
export interface PromptLayerDiff {
layer: PromptLayerName;
op: PromptLayerOperation;
reason?: string | undefined;
text?: string | undefined;
}
export interface PromptVariantManifest {
description?: string | undefined;
diff: PromptLayerDiff[];
expectedImprovement?: string | undefined;
rationale?: string | undefined;
requiredResources?: string[] | undefined;
requiredTools?: string[] | undefined;
skill?: string | undefined;
trustTier?: "T1" | "T2" | "T3" | "T4" | undefined;
variantId: string;
}
const variantCache = new Map<string, PromptVariantManifest | null>();
export function loadPromptVariantManifest(
variantId: string,
): PromptVariantManifest | null {
const normalized = variantId.trim();
if (!normalized) {
return null;
}
if (variantCache.has(normalized)) {
return variantCache.get(normalized) ?? null;
}
const path = resolve(
process.cwd(),
"src",
"eval",
"proposals",
`${normalized}.json`,
);
if (!existsSync(path)) {
variantCache.set(normalized, null);
return null;
}
try {
const parsed = JSON.parse(readFileSync(path, "utf8")) as PromptVariantManifest;
if (parsed.variantId !== normalized || !Array.isArray(parsed.diff)) {
variantCache.set(normalized, null);
return null;
}
variantCache.set(normalized, parsed);
return parsed;
} catch {
variantCache.set(normalized, null);
return null;
}
}
-16
View File
@@ -1,16 +0,0 @@
{
"description": "deadline-near 상태에서도 사용자가 묻지 않은 남은 등록 절차를 자동으로 붙이지 않도록 우선순위 강화",
"diff": [
{
"layer": "policy",
"op": "append",
"reason": "첫 번째 실패에서 현재 상태의 remaining 정보가 질문보다 우선되어 무관한 절차 안내가 출력됐다. deadline-near에서도 질문 우선 원칙을 강화한다.",
"text": "[deadline-near 응답 범위 제한]\n- phase가 deadline-near여도 마지막 메시지가 가능 여부/확인 주체/확인 여부를 묻는 질문이면, 독촉성 남은 절차 안내를 붙이지 마.\n- remaining 상태는 사용자가 \"뭐가 남았나요\", \"다 했나요\", \"완료인가요\"처럼 진행 상태를 물을 때만 사용해.\n- 답변 전 마지막으로 확인: 내 응답 첫 문장이 사용자의 질문에 대한 '네/아니요/아직/관리자가 확인' 중 직접 답인지 점검해."
}
],
"expectedImprovement": "무관한 카카오톡/디스코드 절차 나열이 줄고, 사용자의 마지막 질문 중심으로 1문장 내외의 답변이 생성된다.",
"rationale": "현재 스냅샷에는 remaining 정보가 강하게 주입되어 있어, 사용자가 재등록 가능 여부만 묻는 상황에서도 남은 절차를 안내할 위험이 높다. 진행 상태 질문이 아닐 때 remaining을 쓰지 않게 하면 범위 이탈을 줄일 수 있다.",
"skill": "onboarding",
"variantId": "579dcc7d",
"trustTier": "T2"
}
-16
View File
@@ -1,16 +0,0 @@
{
"description": "디스코드 확인 주체/재등록 가능 여부 질문에는 상태 나열이나 불확실 회피 대신 짧게 직접 답하도록 하는 예외 규칙 추가",
"diff": [
{
"layer": "policy",
"op": "append",
"reason": "실패 케이스가 모두 사용자의 단일 질문에 답하지 못하고, 확인 불가 회피 또는 무관한 남은 절차 나열로 이탈했다. 해당 slice에서 반복되는 직접 질문 패턴을 좁게 처리한다.",
"text": "[discord-verify-question 직접응답 보강]\n- 사용자가 \"디코는 누가 확인하나요\", \"관리자가 확인하나요\", \"관리자 형님들이 확인해야 하나요\"처럼 디스코드 확인 주체를 물으면, 확인되지 않았다고 회피하지 말고 \"네, 디스코드 관련 확인은 관리자/운영진이 확인해요.\"라고 짧게 답해.\n- 사용자가 \"계정 확인 후 다시 등록해도 되나요\", \"제 명의 계정 확인 후 다시 등록해도 될까요\"처럼 본인 계정/명의 확인 뒤 재등록 가능 여부를 물으면, 다른 절차를 나열하지 말고 \"네, 본인 명의 계정 확인 후 다시 등록해 주시면 돼요.\"라고 답해.\n- 위 두 유형에서는 카카오톡 완료 여부, 남은 단계, 디스코드 닉네임 형식, 다음 절차를 사용자가 묻지 않는 한 덧붙이지 마."
}
],
"expectedImprovement": "디스코드 확인 주체 질문에는 '관리자/운영진이 확인한다'고 정확히 답하고, 계정 확인 후 재등록 질문에는 '가능하다'고 직접 답해 accuracy와 scope가 개선된다.",
"rationale": "현재 프롬프트의 '확인되지 않았으면 확인되지 않았다고만 답'과 '세션 상태' 정보가 과하게 작동해, 실제로는 일반 운영 규칙으로 답할 수 있는 질문에도 회피하거나 상태 목록을 출력한다. 확인 주체와 재등록 가능 여부는 온보딩 절차의 일반 질문이므로 짧은 정답 템플릿을 추가하는 것이 가장 작은 수정이다.",
"skill": "onboarding",
"variantId": "991bfb18",
"trustTier": "T2"
}
-15
View File
@@ -1,15 +0,0 @@
{
"description": "status-question에서 의도 정렬 개선: 진행상태 확인 질문과 남은 항목 열거 요청을 분리",
"diff": [
{
"layer": "policy",
"op": "append",
"reason": "현재는 status-question으로 분류되면 항상 절차 목록 템플릿을 출력해, \"내전/5대5\" 같은 비온보딩 표현이나 진행 여부 확인 질문에서도 잘못된 단계 나열이 발생함. 진행확인형과 남은 절차 요청형을 분리해 잘못된 응답을 방지함.",
"text": "[답변 분기 보강]\n- status-question으로 분류되더라도, 먼저 발화의 실제 의도를 판단한다.\n- 온보딩 상태/진행 확인형인지 판단 기준: \"완료\", \"됐다/됐나\", \"안됐나\", \"입장했/들어왔\", \"가능\", \"놓친\", \"마무리\" 등 진행/결과 확인 표현이 있는가?\n- 진행상태 확인형이면 먼저 한 줄로 상태 판정만 답한다. 예: \"아직 등록이 완료되지 않았어요.\" 또는 \"방금 들어왔을 때는 아직 미완료입니다.\"\n- 남은 절차를 열거하는 것은 사용자 요청이 명시적일 때만. (예: \"뭐 남았지?\", \"아직 뭐 해야 하지?\", \"어떤 단계가 남았어?\")\n- 온보딩 키워드(닉네임/방 하트/본인인증/구인방/디스코드/등록/완료/입장/남았)가 없고 게임/일반 잡담성 표현(내전, 5대5, 관리자 멘트 등)만 있을 때는 \"[NO_RESPONSE]\"를 그대로 출력한다."
}
],
"expectedImprovement": "의도 정합성(intentAlignment) 및 accuracy/scope/concision 실패가 반복되는 유형에서 즉시 개선되고, 오지 않은 정보 제공을 줄여 점수 하락을 완화한다.",
"rationale": "실패의 공통원인은 '무조건 남은 항목 나열' 구조이다. 새 규칙으로 status-question 응답을 상태 판정형/요청형으로 조건 분기해, 의도 불일치 답변을 줄인다.",
"variantId": "d1ed50ee",
"trustTier": "T2"
}
+4 -5
View File
@@ -81,12 +81,11 @@ function toCalibrationPairs(
pairs.push({ pairs.push({
evalStreamId: label.evalStreamId, evalStreamId: label.evalStreamId,
humanScore: label.humanScore, humanScore: label.humanScore,
llmAccuracy: archive.llmJudge.scores.accuracy ?? archive.llmJudge.scores.overall, llmAccuracy: archive.llmJudge.scores.accuracy,
llmConcision: archive.llmJudge.scores.concision ?? archive.llmJudge.scores.overall, llmConcision: archive.llmJudge.scores.concision,
llmFriendliness: llmFriendliness: archive.llmJudge.scores.friendliness,
archive.llmJudge.scores.friendliness ?? archive.llmJudge.scores.overall,
llmOverall: archive.llmJudge.scores.overall, llmOverall: archive.llmJudge.scores.overall,
llmScope: archive.llmJudge.scores.scope ?? archive.llmJudge.scores.overall, llmScope: archive.llmJudge.scores.scope,
}); });
} }
+1 -1
View File
@@ -25,7 +25,7 @@ export interface ReviewCliOptions {
resetPending?: boolean | undefined; resetPending?: boolean | undefined;
} }
export function getReviewerId(): string { function getReviewerId(): string {
const configured = process.env["REVIEWER_ID"]?.trim(); const configured = process.env["REVIEWER_ID"]?.trim();
if (configured) { if (configured) {
return configured; return configured;
-178
View File
@@ -1,178 +0,0 @@
import { mkdtempSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { describe, expect, it } from "vitest";
import type { StreamEntry } from "@common/garnet";
import {
ROUTER_FEATURE_NAMES,
ROUTING_DECISION_KINDS,
type LearnedRouterModel,
} from "@skills/onboarding/runtime/router";
import {
SUPERVISOR_RESOLUTIONS_STREAM_KEY,
SUPERVISOR_REVIEW_STREAM_KEY,
} from "@skills/supervisor/review-queue";
import { BACKGROUND_STREAM_KEY } from "./shadow-bus.js";
import {
buildRouterShadowReport,
formatRouterShadowReport,
} from "./router-shadow-report.js";
class FakeGarnet {
calls = 0;
constructor(private readonly streams: Record<string, StreamEntry[]> = {}) {}
readStreamRange(key: string): Promise<StreamEntry[]> {
this.calls += 1;
return Promise.resolve(this.streams[key] ?? []);
}
}
function writeModel(): string {
const weights = ROUTING_DECISION_KINDS.map(() =>
ROUTER_FEATURE_NAMES.map(() => 0),
);
const tier2Index = ROUTING_DECISION_KINDS.indexOf("tier2-full");
const tier2Weights = weights[tier2Index];
if (tier2Weights) {
tier2Weights[0] = 10;
}
const model: LearnedRouterModel = {
decisionKinds: [...ROUTING_DECISION_KINDS],
featureNames: [...ROUTER_FEATURE_NAMES],
trainedAt: "2026-05-18T00:00:00.000Z",
version: 1,
weights,
};
const path = join(
mkdtempSync(join(tmpdir(), "router-shadow-")),
"model.json",
);
writeFileSync(path, `${JSON.stringify(model)}\n`);
return path;
}
function streamEntry(id: string, message: Record<string, string>): StreamEntry {
return { id, message };
}
describe("buildRouterShadowReport", () => {
it("reports missing weights without reading Garnet streams", async () => {
const garnet = new FakeGarnet();
const report = await buildRouterShadowReport(garnet, {
since: new Date("2026-05-18T00:00:00.000Z"),
until: new Date("2026-05-18T01:00:00.000Z"),
weightsPath: "/path/that/does/not/exist.json",
});
expect(report).toMatchObject({
agreementRate: 1,
status: "weights-missing",
totalSamples: 0,
});
expect(report.notes[0]).toContain("Router weights missing");
expect(garnet.calls).toBe(0);
});
it("compares static and learned router decisions from supervisor outcomes", async () => {
const weightsPath = writeModel();
const garnet = new FakeGarnet({
[BACKGROUND_STREAM_KEY]: [
streamEntry("10-0", {
classified: JSON.stringify({
confidence: 0.7,
intent: { kind: "status-question" },
source: "llm",
}),
createdAt: "2026-05-18T00:00:30.000Z",
derived: JSON.stringify({
allowShortResponse: false,
intent: { kind: "status-question" },
phase: { kind: "deadline-near", remaining: [] },
responseMode: "llm",
routingDecision: { kind: "tier1-direct", skipPlanner: false },
shouldRespond: true,
suggestedMaxTokens: 512,
tier: 1,
}),
input: JSON.stringify({
request: {
chatroomId: "room-1",
nickname: "tester",
requestId: "request-1",
skill: "onboarding",
trigger: "user-message",
userId: "user-1",
userMessage: "아직 남은 절차가 있나요?",
},
}),
output: JSON.stringify({
confidence: 1,
requestId: "request-1",
source: "llm",
text: "남은 절차를 확인해드릴게요.",
}),
skill: "onboarding",
}),
],
[SUPERVISOR_RESOLUTIONS_STREAM_KEY]: [
streamEntry("2-0", {
action: "manual-reply",
resolvedAt: "2026-05-18T00:02:00.000Z",
resolvedBy: "operator@example.com",
reviewId: "1-0",
type: "resolution",
}),
],
[SUPERVISOR_REVIEW_STREAM_KEY]: [
streamEntry("1-0", {
chatroomId: "room-1",
packet: JSON.stringify({
constraints: [],
currentState: {},
evidence: [],
expectedOutputSchema: "text",
handoffReason: "low-confidence",
sourceChain: ["onboarding", "supervisor"],
summary: "deadline status question",
userGoal: "status",
}),
queuedAt: "2026-05-18T00:01:00.000Z",
reason: "low-confidence",
requestId: "request-1",
target: "human-review",
type: "supervisor-handoff",
userId: "user-1",
}),
],
});
const report = await buildRouterShadowReport(garnet, {
since: new Date("2026-05-18T00:00:00.000Z"),
until: new Date("2026-05-18T01:00:00.000Z"),
weightsPath,
});
expect(report).toMatchObject({
agreementRate: 0,
divergences: [
{
count: 1,
examples: ["request-1"],
learnedDecision: "tier2-full",
staticDecision: "tier1-direct",
},
],
skippedSamples: 0,
status: "ok",
totalSamples: 1,
});
expect(formatRouterShadowReport(report)).toContain(
"tier1-direct\ttier2-full\t1\trequest-1",
);
});
});
-249
View File
@@ -1,249 +0,0 @@
import type { AgentRequest } from "@agent/types";
import type { GarnetClient } from "@common/garnet";
import {
decideLearnedRouting,
decidePreGenerationRouting,
loadLearnedRouterModel,
routerDecisionKind,
type RoutingDecisionKind,
type RouterInput,
} from "@skills/onboarding/runtime/router";
import type {
ClassifiedIntent,
DerivedState,
} from "@skills/onboarding/state/onboarding-state";
import {
loadSupervisorOutcomes,
type SupervisorOutcome,
} from "./router-training-signals.js";
import type { ShadowEvalEnvelope } from "./shadow-bus.js";
export interface RouterShadowDivergence {
staticDecision: string;
learnedDecision: string;
count: number;
examples: string[];
}
export type RouterShadowReportStatus = "ok" | "weights-missing";
export interface RouterShadowReport {
totalSamples: number;
agreementRate: number;
divergences: RouterShadowDivergence[];
generatedAt: string;
windowFrom: string;
windowUntil: string;
status: RouterShadowReportStatus;
skippedSamples: number;
notes: string[];
}
export interface RouterShadowReportOptions {
since: Date;
until: Date;
weightsPath?: string | undefined;
}
function asRecord(value: unknown): Record<string, unknown> {
return typeof value === "object" && value !== null
? (value as Record<string, unknown>)
: {};
}
function isResponseMode(value: unknown): value is DerivedState["responseMode"] {
return (
value === "skip" ||
value === "template" ||
value === "llm" ||
value === "llm-with-tools"
);
}
function isClassifiedIntent(value: unknown): value is ClassifiedIntent {
const record = asRecord(value);
const intent = asRecord(record["intent"]);
return (
typeof intent["kind"] === "string" &&
typeof record["confidence"] === "number" &&
Number.isFinite(record["confidence"]) &&
(record["source"] === "regex" || record["source"] === "llm")
);
}
function isDerivedState(value: unknown): value is DerivedState {
const record = asRecord(value);
const phase = asRecord(record["phase"]);
const intent = asRecord(record["intent"]);
const tier = record["tier"];
return (
typeof phase["kind"] === "string" &&
typeof intent["kind"] === "string" &&
typeof record["shouldRespond"] === "boolean" &&
isResponseMode(record["responseMode"]) &&
typeof record["suggestedMaxTokens"] === "number" &&
Number.isFinite(record["suggestedMaxTokens"]) &&
typeof record["allowShortResponse"] === "boolean" &&
(tier === 0 || tier === 1 || tier === 2)
);
}
function isOnboardingShadowEntry(
shadow: ShadowEvalEnvelope,
): shadow is ShadowEvalEnvelope & {
classified: ClassifiedIntent;
derived: DerivedState;
request: AgentRequest;
} {
return (
(shadow.skill === "onboarding" || shadow.request.skill === "onboarding") &&
isClassifiedIntent(shadow.classified) &&
isDerivedState(shadow.derived)
);
}
function routerInputFromOutcome(
outcome: SupervisorOutcome,
): RouterInput | null {
const shadow = outcome.shadowEvalEntry;
if (!shadow || !isOnboardingShadowEntry(shadow)) {
return null;
}
return {
classified: shadow.classified,
derived: shadow.derived,
request: shadow.request,
session: null,
};
}
function addDivergence(
groups: Map<string, RouterShadowDivergence>,
staticDecision: RoutingDecisionKind,
learnedDecision: RoutingDecisionKind,
requestId: string,
): void {
const key = `${staticDecision}->${learnedDecision}`;
const current = groups.get(key) ?? {
count: 0,
examples: [],
learnedDecision,
staticDecision,
};
current.count += 1;
if (current.examples.length < 5) {
current.examples.push(requestId);
}
groups.set(key, current);
}
function roundRate(value: number): number {
return Number(value.toFixed(3));
}
export async function buildRouterShadowReport(
garnetClient: Pick<GarnetClient, "readStreamRange">,
options: RouterShadowReportOptions,
): Promise<RouterShadowReport> {
const generatedAt = new Date().toISOString();
const baseReport = {
generatedAt,
windowFrom: options.since.toISOString(),
windowUntil: options.until.toISOString(),
};
const model = loadLearnedRouterModel(options.weightsPath);
if (!model) {
return {
...baseReport,
agreementRate: 1,
divergences: [],
notes: [
"Router weights missing or invalid; set --weights or AGENT_ROUTER_WEIGHTS_PATH.",
],
skippedSamples: 0,
status: "weights-missing",
totalSamples: 0,
};
}
const outcomes = await loadSupervisorOutcomes(
garnetClient,
options.since,
options.until,
);
const divergences = new Map<string, RouterShadowDivergence>();
let totalSamples = 0;
let matchingSamples = 0;
let skippedSamples = 0;
for (const outcome of outcomes) {
const input = routerInputFromOutcome(outcome);
if (!input) {
skippedSamples += 1;
continue;
}
const staticDecision = routerDecisionKind(
decidePreGenerationRouting(input),
);
const learnedDecision = routerDecisionKind(
decideLearnedRouting(input, model),
);
totalSamples += 1;
if (staticDecision === learnedDecision) {
matchingSamples += 1;
continue;
}
addDivergence(
divergences,
staticDecision,
learnedDecision,
input.request.requestId,
);
}
return {
...baseReport,
agreementRate:
totalSamples === 0 ? 1 : roundRate(matchingSamples / totalSamples),
divergences: [...divergences.values()].sort(
(left, right) => right.count - left.count,
),
notes: [],
skippedSamples,
status: "ok",
totalSamples,
};
}
export function formatRouterShadowReport(report: RouterShadowReport): string {
const lines = [
`[router-shadow-report] generatedAt=${report.generatedAt}`,
`[router-shadow-report] status=${report.status} window=${report.windowFrom}..${report.windowUntil}`,
`[router-shadow-report] samples=${report.totalSamples} agreementRate=${report.agreementRate} skipped=${report.skippedSamples}`,
];
for (const note of report.notes) {
lines.push(`[router-shadow-report] note=${note}`);
}
if (report.divergences.length === 0) {
lines.push("[router-shadow-report] divergences=none");
return lines.join("\n");
}
lines.push("staticDecision\tlearnedDecision\tcount\texamples");
for (const divergence of report.divergences) {
lines.push(
[
divergence.staticDecision,
divergence.learnedDecision,
divergence.count,
divergence.examples.join(","),
].join("\t"),
);
}
return lines.join("\n");
}
-108
View File
@@ -1,108 +0,0 @@
import { describe, expect, it } from "vitest";
import type { StreamEntry } from "@common/garnet";
import { BACKGROUND_STREAM_KEY } from "./shadow-bus.js";
import { loadSupervisorOutcomes } from "./router-training-signals.js";
import {
SUPERVISOR_RESOLUTIONS_STREAM_KEY,
SUPERVISOR_REVIEW_STREAM_KEY,
} from "@skills/supervisor/review-queue";
class FakeGarnet {
constructor(private readonly streams: Record<string, StreamEntry[]>) {}
readStreamRange(key: string): Promise<StreamEntry[]> {
return Promise.resolve(this.streams[key] ?? []);
}
}
describe("loadSupervisorOutcomes", () => {
it("joins supervisor review entries, resolution audit, and shadow eval by requestId", async () => {
const garnet = new FakeGarnet({
[BACKGROUND_STREAM_KEY]: [
{
id: "10-0",
message: {
classified: "{}",
createdAt: "2026-05-18T00:00:30.000Z",
derived: "{}",
input: JSON.stringify({
request: {
chatroomId: "room-1",
nickname: "tester",
requestId: "request-1",
skill: "onboarding",
trigger: "user-message",
userId: "user-1",
},
}),
output: JSON.stringify({
confidence: 1,
requestId: "request-1",
source: "template",
text: "ok",
}),
skill: "onboarding",
},
},
],
[SUPERVISOR_RESOLUTIONS_STREAM_KEY]: [
{
id: "2-0",
message: {
action: "manual-reply",
resolvedAt: "2026-05-18T00:02:00.000Z",
resolvedBy: "operator@example.com",
reviewId: "1-0",
type: "resolution",
},
},
],
[SUPERVISOR_REVIEW_STREAM_KEY]: [
{
id: "1-0",
message: {
chatroomId: "room-1",
packet: JSON.stringify({
constraints: [],
currentState: {},
evidence: [],
expectedOutputSchema: "text",
handoffReason: "low-confidence",
sourceChain: ["onboarding", "supervisor"],
summary: "unclear",
userGoal: "help",
}),
queuedAt: "2026-05-18T00:01:00.000Z",
reason: "low-confidence",
requestId: "request-1",
target: "human-review",
userId: "user-1",
},
},
],
});
await expect(
loadSupervisorOutcomes(
garnet,
new Date("2026-05-18T00:00:00.000Z"),
new Date("2026-05-18T00:03:00.000Z"),
),
).resolves.toMatchObject([
{
reason: "low-confidence",
resolution: {
action: "manual-reply",
},
reviewId: "1-0",
shadowEvalEntry: {
request: {
requestId: "request-1",
},
},
sourceChain: ["onboarding", "supervisor"],
},
]);
});
});
-177
View File
@@ -1,177 +0,0 @@
import type { HandoffReason } from "@agent/types";
import type { GarnetClient, StreamEntry } from "@common/garnet";
import {
parseSupervisorReviewEntry,
SUPERVISOR_RESOLUTIONS_STREAM_KEY,
SUPERVISOR_REVIEW_STREAM_KEY,
type SupervisorReviewAction,
type SupervisorReviewResolution,
} from "@skills/supervisor/review-queue";
import { BACKGROUND_STREAM_KEY, type ShadowEvalEnvelope } from "./shadow-bus.js";
export interface SupervisorOutcome {
queuedAt: string;
reason: HandoffReason;
resolution: {
action: SupervisorReviewAction;
resolvedAt: string;
};
reviewId: string;
shadowEvalEntry?: ShadowEvalEnvelope | undefined;
sourceChain: string[];
}
function parseJson<T>(value: string | undefined): T | undefined {
if (!value?.trim()) {
return undefined;
}
try {
return JSON.parse(value) as T;
} catch {
return undefined;
}
}
function isWithinIso(
value: string,
since: Date,
until: Date,
): boolean {
const time = Date.parse(value);
return (
Number.isFinite(time) &&
time >= since.getTime() &&
time <= until.getTime()
);
}
function parseShadowEntry(
entry: StreamEntry,
): ShadowEvalEnvelope | null {
const input = parseJson<{ request?: ShadowEvalEnvelope["request"] }>(
entry.message["input"],
);
const response = parseJson<ShadowEvalEnvelope["response"]>(
entry.message["output"],
);
const skill = entry.message["skill"];
const createdAt = entry.message["createdAt"];
if (!input?.request || !response || !skill || !createdAt) {
return null;
}
return {
classified: parseJson(entry.message["classified"]) ?? {},
createdAt,
derived: parseJson(entry.message["derived"]) ?? {},
request: input.request,
response,
skill,
...(entry.message["latencyMs"]
? { latencyMs: Number(entry.message["latencyMs"]) }
: {}),
...(entry.message["process"]
? { process: parseJson(entry.message["process"]) }
: {}),
};
}
function isSupervisorReviewAction(
value: string | undefined,
): value is SupervisorReviewAction {
return value === "manual-reply" || value === "reroute" || value === "dismiss";
}
function parseResolutionEntry(
entry: StreamEntry,
): { resolution: SupervisorReviewResolution; reviewId: string } | null {
const reviewId = entry.message["reviewId"];
const resolvedAt = entry.message["resolvedAt"];
const resolvedBy = entry.message["resolvedBy"];
const action = entry.message["action"];
if (
!reviewId ||
!resolvedAt ||
!resolvedBy ||
!isSupervisorReviewAction(action)
) {
return null;
}
return {
resolution: {
action,
resolvedAt,
resolvedBy,
...(entry.message["note"] ? { note: entry.message["note"] } : {}),
},
reviewId,
};
}
function latestResolutionsByReviewId(
entries: StreamEntry[],
): Map<string, SupervisorReviewResolution> {
const byReviewId = new Map<string, SupervisorReviewResolution>();
for (const entry of entries) {
const parsed = parseResolutionEntry(entry);
if (!parsed) {
continue;
}
const current = byReviewId.get(parsed.reviewId);
if (!current || parsed.resolution.resolvedAt > current.resolvedAt) {
byReviewId.set(parsed.reviewId, parsed.resolution);
}
}
return byReviewId;
}
export async function loadSupervisorOutcomes(
garnetClient: Pick<GarnetClient, "readStreamRange">,
since: Date,
until: Date,
): Promise<SupervisorOutcome[]> {
const [reviewEntries, resolutionEntries, shadowEntries] = await Promise.all([
garnetClient.readStreamRange(SUPERVISOR_REVIEW_STREAM_KEY, "-", "+"),
garnetClient.readStreamRange(SUPERVISOR_RESOLUTIONS_STREAM_KEY, "-", "+"),
garnetClient.readStreamRange(BACKGROUND_STREAM_KEY, "-", "+"),
]);
const shadowByRequestId = new Map(
shadowEntries.flatMap((entry) => {
const shadow = parseShadowEntry(entry);
return shadow ? [[shadow.request.requestId, shadow] as const] : [];
}),
);
const resolutionByReviewId = latestResolutionsByReviewId(resolutionEntries);
const outcomes: SupervisorOutcome[] = [];
for (const entry of reviewEntries) {
const review = parseSupervisorReviewEntry(entry);
if (!review || !isWithinIso(review.queuedAt, since, until)) {
continue;
}
const resolution = resolutionByReviewId.get(review.id);
if (!resolution || !isWithinIso(resolution.resolvedAt, since, until)) {
continue;
}
outcomes.push({
queuedAt: review.queuedAt,
reason: review.reason,
resolution: {
action: resolution.action,
resolvedAt: resolution.resolvedAt,
},
reviewId: review.id,
shadowEvalEntry: shadowByRequestId.get(review.requestId),
sourceChain: review.packet?.sourceChain ?? [],
});
}
return outcomes;
}
export const ROUTER_TRAINING_SIGNAL_STREAMS = {
background: BACKGROUND_STREAM_KEY,
resolutions: SUPERVISOR_RESOLUTIONS_STREAM_KEY,
reviews: SUPERVISOR_REVIEW_STREAM_KEY,
} as const;
+2 -25
View File
@@ -12,26 +12,6 @@ import {
} from "@skills/onboarding/runtime/router"; } from "@skills/onboarding/runtime/router";
import type { EvalArchiveEntry } from "./archive.js"; import type { EvalArchiveEntry } from "./archive.js";
function isOnboardingArchiveEntry(
entry: EvalArchiveEntry,
): entry is EvalArchiveEntry & {
classified: ClassifiedIntent;
derived: DerivedState;
} {
return (
(entry.skill === undefined ||
entry.skill === "onboarding" ||
entry.request.skill === "onboarding") &&
typeof entry.derived === "object" &&
entry.derived !== null &&
typeof (entry.derived as { intent?: unknown }).intent === "object" &&
typeof (entry.derived as { phase?: unknown }).phase === "object" &&
typeof entry.classified === "object" &&
entry.classified !== null &&
typeof (entry.classified as { intent?: unknown }).intent === "object"
);
}
function getLlmOverall(entry: EvalArchiveEntry): number | null { function getLlmOverall(entry: EvalArchiveEntry): number | null {
return entry.llmJudge.skipped ? null : entry.llmJudge.scores.overall; return entry.llmJudge.skipped ? null : entry.llmJudge.scores.overall;
} }
@@ -41,9 +21,6 @@ function requestFromEntry(entry: EvalArchiveEntry): AgentRequest {
} }
function deriveLabel(entry: EvalArchiveEntry): RoutingDecisionKind | null { function deriveLabel(entry: EvalArchiveEntry): RoutingDecisionKind | null {
if (!isOnboardingArchiveEntry(entry)) {
return null;
}
const staticDecision = routerDecisionKind( const staticDecision = routerDecisionKind(
decidePreGenerationRouting({ decidePreGenerationRouting({
classified: entry.classified, classified: entry.classified,
@@ -80,11 +57,11 @@ function deriveLabel(entry: EvalArchiveEntry): RoutingDecisionKind | null {
} }
function toClassified(entry: EvalArchiveEntry): ClassifiedIntent { function toClassified(entry: EvalArchiveEntry): ClassifiedIntent {
return entry.classified as ClassifiedIntent; return entry.classified;
} }
function toDerived(entry: EvalArchiveEntry): DerivedState { function toDerived(entry: EvalArchiveEntry): DerivedState {
return entry.derived as DerivedState; return entry.derived;
} }
export function buildRouterTrainingSamples( export function buildRouterTrainingSamples(
+1 -27
View File
@@ -1,7 +1,7 @@
import { describe, expect, it, vi } from "vitest"; import { describe, expect, it, vi } from "vitest";
import { AGENT_EVAL_REPORT_SUBJECT } from "@common/nats-subjects"; import { AGENT_EVAL_REPORT_SUBJECT } from "@common/nats-subjects";
import { publishEvalReportIfPossible, resolveEvalSetPath } from "./runner.js"; import { publishEvalReportIfPossible } from "./runner.js";
import type { EvalReport } from "./types.js"; import type { EvalReport } from "./types.js";
const natsMocks = vi.hoisted(() => ({ const natsMocks = vi.hoisted(() => ({
@@ -54,29 +54,3 @@ describe("publishEvalReportIfPossible", () => {
expect(natsMocks.close).toHaveBeenCalledTimes(1); expect(natsMocks.close).toHaveBeenCalledTimes(1);
}); });
}); });
describe("resolveEvalSetPath", () => {
it("resolves skill/subset eval sets", () => {
const resolved = resolveEvalSetPath("moderation/cases");
expect(resolved).toMatchObject({
skill: "moderation",
subset: "cases",
});
expect(resolved.path).toContain("src/eval/golden/moderation/cases.jsonl");
});
it("maps legacy golden set to onboarding cases", () => {
const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);
const resolved = resolveEvalSetPath("golden");
expect(resolved.skill).toBe("onboarding");
expect(resolved.subset).toBe("cases");
expect(warn).toHaveBeenCalledWith(
"[eval] --set=golden is deprecated; use --set=onboarding/cases",
);
warn.mockRestore();
});
});
+138 -678
View File
File diff suppressed because it is too large. Load diff
-4
View File
@@ -61,7 +61,6 @@ describe("ShadowEvalBus", () => {
derived, derived,
request, request,
response, response,
skill: "onboarding",
}); });
expect(appendStream).toHaveBeenCalledTimes(1); expect(appendStream).toHaveBeenCalledTimes(1);
@@ -69,9 +68,7 @@ describe("ShadowEvalBus", () => {
expect(appendStream.mock.calls[0]?.[1]).toMatchObject({ expect(appendStream.mock.calls[0]?.[1]).toMatchObject({
type: "shadow-eval", type: "shadow-eval",
requestId: "req-1", requestId: "req-1",
skill: "onboarding",
tier: "0", tier: "0",
version: "2",
}); });
randomSpy.mockRestore(); randomSpy.mockRestore();
@@ -89,7 +86,6 @@ describe("ShadowEvalBus", () => {
...response, ...response,
text: "", text: "",
}, },
skill: "onboarding",
}); });
expect(appendStream).not.toHaveBeenCalled(); expect(appendStream).not.toHaveBeenCalled();
+19 -56
View File
@@ -1,23 +1,26 @@
import type { AgentRequest, AgentResponse } from "@agent/types"; import type { AgentRequest, AgentResponse } from "@agent/types";
import type { GarnetClient } from "@common/garnet"; import type { GarnetClient } from "@common/garnet";
import { getRequestLogger } from "@obs/context"; import { getRequestLogger } from "@obs/context";
import type {
ClassifiedIntent,
DerivedState,
} from "@skills/onboarding/state/onboarding-state";
import type { ProcessTraceSnapshot } from "@skills/onboarding/runtime/trace";
import { import {
getBackgroundStreamMaxLen, getBackgroundStreamMaxLen,
getShadowEvalSamplingRate, getShadowEvalSamplingRate,
} from "../config.js"; } from "../config.js";
export const BACKGROUND_STREAM_KEY = "lolympus-agent:background"; const BACKGROUND_STREAM_KEY = "lolympus-agent:background";
export interface ShadowEvalEnvelope { export interface ShadowEvalEnvelope {
classified: unknown; classified: ClassifiedIntent;
createdAt: string; createdAt: string;
derived: unknown; derived: DerivedState;
latencyMs?: number | undefined; latencyMs?: number | undefined;
process?: unknown; process?: ProcessTraceSnapshot | undefined;
request: AgentRequest; request: AgentRequest;
response: AgentResponse; response: AgentResponse;
skill: string;
version?: 2 | undefined;
} }
export interface ShadowEvalPublisher { export interface ShadowEvalPublisher {
@@ -35,46 +38,10 @@ export function shouldSampleShadowEval(
return randomValue < getShadowEvalSamplingRate(tier); return randomValue < getShadowEvalSamplingRate(tier);
} }
function asRecord(value: unknown): Record<string, unknown> { function getTraceAbstention(process: ProcessTraceSnapshot | undefined):
return typeof value === "object" && value !== null | NonNullable<ProcessTraceSnapshot["steps"]["classify"]>["output"]["abstention"]
? (value as Record<string, unknown>) | undefined {
: {}; return process?.steps.classify?.output.abstention;
}
function parseTier(value: unknown): 0 | 1 | 2 | null {
const parsed = typeof value === "number" ? value : Number(value);
if (parsed === 0 || parsed === 1 || parsed === 2) {
return parsed;
}
return null;
}
function getDerivedTier(derived: unknown): 0 | 1 | 2 | null {
return parseTier(asRecord(derived)["tier"]);
}
function getTraceAbstention(process: unknown): Record<string, unknown> | null {
const steps = asRecord(asRecord(process)["steps"]);
const classify = asRecord(steps["classify"]);
const output = asRecord(classify["output"]);
const abstention = asRecord(output["abstention"]);
return Object.keys(abstention).length > 0 ? abstention : null;
}
function getAbstentionActionKind(abstention: Record<string, unknown> | null): string | null {
const action = asRecord(abstention?.["action"]);
const kind = action["kind"];
return typeof kind === "string" && kind.trim() ? kind : null;
}
function getAbstentionReason(abstention: Record<string, unknown> | null): string | null {
const reason = abstention?.["reason"];
return typeof reason === "string" && reason.trim() ? reason : null;
}
function getAbstentionScore(abstention: Record<string, unknown> | null): number | null {
const score = abstention?.["score"];
return typeof score === "number" && Number.isFinite(score) ? score : null;
} }
export class ShadowEvalBus implements ShadowEvalPublisher { export class ShadowEvalBus implements ShadowEvalPublisher {
@@ -84,13 +51,11 @@ export class ShadowEvalBus implements ShadowEvalPublisher {
publish(envelope: ShadowEvalEnvelope): void { publish(envelope: ShadowEvalEnvelope): void {
const abstention = getTraceAbstention(envelope.process); const abstention = getTraceAbstention(envelope.process);
const actionKind = getAbstentionActionKind(abstention); if (!envelope.response.text.trim() && abstention?.action.kind !== "no-response") {
if (!envelope.response.text.trim() && actionKind !== "no-response") {
return; return;
} }
const tier = getDerivedTier(envelope.derived); if (!shouldSampleShadowEval(envelope.derived.tier)) {
if (tier === null || !shouldSampleShadowEval(tier)) {
return; return;
} }
@@ -101,9 +66,7 @@ export class ShadowEvalBus implements ShadowEvalPublisher {
requestId: envelope.request.requestId, requestId: envelope.request.requestId,
chatroomId: envelope.request.chatroomId, chatroomId: envelope.request.chatroomId,
userId: envelope.request.userId, userId: envelope.request.userId,
skill: envelope.skill, tier: String(envelope.derived.tier),
version: String(envelope.version ?? 2),
tier: String(tier),
latencyMs: latencyMs:
envelope.latencyMs !== undefined ? String(envelope.latencyMs) : "", envelope.latencyMs !== undefined ? String(envelope.latencyMs) : "",
input: JSON.stringify({ input: JSON.stringify({
@@ -116,9 +79,9 @@ export class ShadowEvalBus implements ShadowEvalPublisher {
}; };
if (abstention) { if (abstention) {
fields.abstentionAction = actionKind ?? ""; fields.abstentionAction = abstention.action.kind;
fields.abstentionReason = getAbstentionReason(abstention) ?? ""; fields.abstentionReason = abstention.reason ?? "";
fields.abstentionScore = String(getAbstentionScore(abstention) ?? ""); fields.abstentionScore = String(abstention.score);
} }
if (envelope.process) { if (envelope.process) {
+1 -62
View File
@@ -1,11 +1,9 @@
import { describe, expect, it, vi } from "vitest"; import { describe, expect, it } from "vitest";
import type { AlertPayload } from "@common/nats-subjects";
import { import {
calculateEvalScoreDropPct, calculateEvalScoreDropPct,
calculateLlmErrorRate, calculateLlmErrorRate,
calculatePercentile, calculatePercentile,
ShadowWorker,
} from "./shadow-worker.js"; } from "./shadow-worker.js";
describe("calculatePercentile", () => { describe("calculatePercentile", () => {
@@ -42,73 +40,14 @@ describe("calculateLlmErrorRate", () => {
llmOverall: null, llmOverall: null,
recordedAt: 1, recordedAt: 1,
responseLatencyMs: 100, responseLatencyMs: 100,
skill: "onboarding",
}, },
{ {
llmJudgeErrored: false, llmJudgeErrored: false,
llmOverall: 4.2, llmOverall: 4.2,
recordedAt: 2, recordedAt: 2,
responseLatencyMs: 200, responseLatencyMs: 200,
skill: "onboarding",
}, },
]), ]),
).toBe(0.5); ).toBe(0.5);
}); });
}); });
describe("ShadowWorker alerts", () => {
it("emits latency alerts with the observation skill", async () => {
const sendAlert = vi.fn().mockResolvedValue(true);
const worker = Object.assign(Object.create(ShadowWorker.prototype), {
alertSender: { sendAlert },
recentEvalObservations: [
{
llmJudgeErrored: false,
llmOverall: 4,
recordedAt: Date.now(),
responseLatencyMs: 4500,
skill: "onboarding",
},
{
llmJudgeErrored: false,
llmOverall: 4,
recordedAt: Date.now(),
responseLatencyMs: 4600,
skill: "onboarding",
},
{
llmJudgeErrored: false,
llmOverall: 4,
recordedAt: Date.now(),
responseLatencyMs: 4700,
skill: "onboarding",
},
{
llmJudgeErrored: false,
llmOverall: 4,
recordedAt: Date.now(),
responseLatencyMs: 4800,
skill: "onboarding",
},
{
llmJudgeErrored: false,
llmOverall: 4,
recordedAt: Date.now(),
responseLatencyMs: 4900,
skill: "onboarding",
},
],
}) as {
checkLatencyAlert: (now: number) => Promise<void>;
};
await worker.checkLatencyAlert(Date.now());
expect(sendAlert).toHaveBeenCalledWith(
expect.objectContaining<Partial<AlertPayload>>({
kind: "latency",
skill: "onboarding",
}),
);
});
});
+113 -218
View File
@@ -12,6 +12,12 @@ import { summarizeConversation } from "@conv/summarizer";
import type { LlmClient } from "@llm/llm-client"; import type { LlmClient } from "@llm/llm-client";
import { getLogger } from "@obs/logger"; import { getLogger } from "@obs/logger";
import type { AlertSender } from "@obs/alert"; import type { AlertSender } from "@obs/alert";
import { validateOnboardingResponse } from "@skills/onboarding/runtime/validator";
import type { ProcessTraceSnapshot } from "@skills/onboarding/runtime/trace";
import type {
ClassifiedIntent,
DerivedState,
} from "@skills/onboarding/state/onboarding-state";
import type { ConversationStore } from "@conv/conversation-store"; import type { ConversationStore } from "@conv/conversation-store";
import { import {
getAlertThresholdDriftPct, getAlertThresholdDriftPct,
@@ -43,14 +49,13 @@ import {
type MongoConnectionLike, type MongoConnectionLike,
} from "./archive.js"; } from "./archive.js";
import { computeDriftReports } from "./drift.js"; import { computeDriftReports } from "./drift.js";
import { judgeSkillWithLlm } from "./judges/llm-judge.js"; import { judgeOnboardingWithLlm } from "./judges/llm-judge.js";
import { writeProcessAttributions } from "./judges/attribution.js"; import { writeProcessAttributions } from "./judges/attribution.js";
import { loadPrmModel } from "./judges/prm.js"; import { loadPrmModel } from "./judges/prm.js";
import { import {
judgeProcessRule, judgeProcessRule,
judgeProcessWithLlm, judgeProcessWithLlm,
} from "./judges/process-judge.js"; } from "./judges/process-judge.js";
import { getSkillJudgeConfig } from "./judges/skill-config.js";
import { import {
appendCandidatesToJsonl, appendCandidatesToJsonl,
selectPromotionCandidates, selectPromotionCandidates,
@@ -67,7 +72,6 @@ import type {
EvalBaselineFile, EvalBaselineFile,
EvalCaseDefinition, EvalCaseDefinition,
EvalDerivedSnapshot, EvalDerivedSnapshot,
EvalProcessTraceSnapshot,
ProcessJudgeResult, ProcessJudgeResult,
} from "./types.js"; } from "./types.js";
@@ -76,10 +80,7 @@ const BACKGROUND_DLQ_STREAM_KEY = "lolympus-agent:background:dlq";
const BACKGROUND_GROUP = "bg-worker"; const BACKGROUND_GROUP = "bg-worker";
const EVAL_BASELINE_KEY = "lolympus-agent:eval:baseline"; const EVAL_BASELINE_KEY = "lolympus-agent:eval:baseline";
const EVAL_RESULTS_STREAM_KEY = "lolympus-agent:eval:results"; const EVAL_RESULTS_STREAM_KEY = "lolympus-agent:eval:results";
const EVAL_BASELINE_FILES = [ const EVAL_BASELINE_FILE = "eval-reports/baseline.json";
"eval-reports/baseline.onboarding.json",
"eval-reports/baseline.json",
];
const ALERT_WINDOW_MS = 5 * 60_000; const ALERT_WINDOW_MS = 5 * 60_000;
const ALERT_WINDOW_LABEL = "5m"; const ALERT_WINDOW_LABEL = "5m";
const ALERT_TICK_MS = 60_000; const ALERT_TICK_MS = 60_000;
@@ -96,15 +97,13 @@ const STOP_TIMEOUT_MS = 10_000;
type BackgroundJobType = "shadow-eval" | "summarize"; type BackgroundJobType = "shadow-eval" | "summarize";
interface ShadowEvalJob { interface ShadowEvalJob {
classified: unknown; classified: ClassifiedIntent;
createdAt: string; createdAt: string;
derived: unknown; derived: DerivedState;
latencyMs?: number | undefined; latencyMs?: number | undefined;
process?: EvalProcessTraceSnapshot | undefined; process?: ProcessTraceSnapshot | undefined;
request: AgentRequest; request: AgentRequest;
response: AgentResponse; response: AgentResponse;
skill: string;
version: number;
} }
interface EvalObservation { interface EvalObservation {
@@ -112,7 +111,6 @@ interface EvalObservation {
llmOverall: number | null; llmOverall: number | null;
recordedAt: number; recordedAt: number;
responseLatencyMs: number | null; responseLatencyMs: number | null;
skill: string;
} }
function delay(ms: number): Promise<void> { function delay(ms: number): Promise<void> {
@@ -167,75 +165,26 @@ function parseOptionalJsonField<T>(value: string | undefined): T | undefined {
return JSON.parse(value) as T; return JSON.parse(value) as T;
} }
function asRecord(value: unknown): Record<string, unknown> { function toDerivedSnapshot(derived: DerivedState): EvalDerivedSnapshot {
return typeof value === "object" && value !== null
? (value as Record<string, unknown>)
: {};
}
function getStringField(
record: Record<string, unknown>,
key: string,
): string | null {
const value = record[key];
return typeof value === "string" && value.trim() ? value : null;
}
function getNestedString(
record: Record<string, unknown>,
first: string,
second: string,
): string | null {
return getStringField(asRecord(record[first]), second);
}
function parseTier(value: unknown): 0 | 1 | 2 {
const parsed = typeof value === "number" ? value : Number(value);
return parsed === 0 || parsed === 1 || parsed === 2 ? parsed : 0;
}
function parseResponseMode(value: unknown): EvalDerivedSnapshot["responseMode"] {
return value === "skip" ||
value === "template" ||
value === "llm" ||
value === "llm-with-tools"
? value
: "llm";
}
function toDerivedSnapshot(derived: unknown): EvalDerivedSnapshot {
const record = asRecord(derived);
const intent = getStringField(record, "intent") ??
getNestedString(record, "intent", "kind") ??
"unknown";
const phase = getStringField(record, "phase") ??
getNestedString(record, "phase", "kind") ??
"unknown";
const abstentionScore = record["abstentionScore"];
const suggestedMaxTokens = record["suggestedMaxTokens"];
return { return {
...(typeof abstentionScore === "number" ...(derived.abstentionScore !== undefined
? { abstentionScore } ? { abstentionScore: derived.abstentionScore }
: {}), : {}),
allowShortResponse: record["allowShortResponse"] === true, allowShortResponse: derived.allowShortResponse,
intent, intent: derived.intent.kind,
phase, phase: derived.phase.kind,
responseMode: parseResponseMode(record["responseMode"]), responseMode: derived.responseMode,
...(record["routingDecision"] ...(derived.routingDecision
? { routingDecision: record["routingDecision"] as EvalDerivedSnapshot["routingDecision"] } ? { routingDecision: derived.routingDecision }
: {}), : {}),
shouldRespond: record["shouldRespond"] !== false, shouldRespond: derived.shouldRespond,
suggestedMaxTokens: suggestedMaxTokens: derived.suggestedMaxTokens,
typeof suggestedMaxTokens === "number" && Number.isFinite(suggestedMaxTokens) tier: derived.tier,
? suggestedMaxTokens
: 192,
tier: parseTier(record["tier"]),
}; };
} }
function getTraceAbstention(process: EvalProcessTraceSnapshot | undefined): function getTraceAbstention(process: ProcessTraceSnapshot | undefined):
| NonNullable<EvalProcessTraceSnapshot["steps"]["classify"]>["output"]["abstention"] | NonNullable<ProcessTraceSnapshot["steps"]["classify"]>["output"]["abstention"]
| undefined { | undefined {
return process?.steps.classify?.output.abstention; return process?.steps.classify?.output.abstention;
} }
@@ -256,17 +205,17 @@ function parseShadowEvalJob(message: StreamEntry): ShadowEvalJob {
message.message["input"], message.message["input"],
"input", "input",
); );
const process = parseOptionalJsonField<EvalProcessTraceSnapshot>( const process = parseOptionalJsonField<ProcessTraceSnapshot>(
message.message["process"], message.message["process"],
); );
const job: ShadowEvalJob = { const job: ShadowEvalJob = {
classified: parseJsonField<unknown>( classified: parseJsonField<ClassifiedIntent>(
message.message["classified"], message.message["classified"],
"classified", "classified",
), ),
createdAt: message.message["createdAt"] ?? new Date().toISOString(), createdAt: message.message["createdAt"] ?? new Date().toISOString(),
derived: parseJsonField<unknown>( derived: parseJsonField<DerivedState>(
message.message["derived"], message.message["derived"],
"derived", "derived",
), ),
@@ -276,8 +225,6 @@ function parseShadowEvalJob(message: StreamEntry): ShadowEvalJob {
message.message["output"], message.message["output"],
"output", "output",
), ),
skill: message.message["skill"]?.trim() || input.request.skill || "onboarding",
version: Number(message.message["version"] ?? 1),
}; };
if (process) { if (process) {
@@ -332,18 +279,6 @@ export function calculateLlmErrorRate(
return errorCount / observations.length; return errorCount / observations.length;
} }
function groupObservationsBySkill(
observations: EvalObservation[],
): Map<string, EvalObservation[]> {
const groups = new Map<string, EvalObservation[]>();
for (const observation of observations) {
const values = groups.get(observation.skill) ?? [];
values.push(observation);
groups.set(observation.skill, values);
}
return groups;
}
export class ShadowWorker { export class ShadowWorker {
private archiveFailureCount = 0; private archiveFailureCount = 0;
private lastAlertCheckAt = 0; private lastAlertCheckAt = 0;
@@ -562,8 +497,6 @@ export class ShadowWorker {
private async processShadowEval(message: StreamEntry): Promise<void> { private async processShadowEval(message: StreamEntry): Promise<void> {
const job = parseShadowEvalJob(message); const job = parseShadowEvalJob(message);
const judgeConfig = getSkillJudgeConfig(job.skill);
const derivedSnapshot = toDerivedSnapshot(job.derived);
const abstention = getTraceAbstention(job.process); const abstention = getTraceAbstention(job.process);
const responseTextForValidation = const responseTextForValidation =
job.response.text.trim() job.response.text.trim()
@@ -571,19 +504,18 @@ export class ShadowWorker {
: abstention?.action.kind === "no-response" : abstention?.action.kind === "no-response"
? "[NO_RESPONSE]" ? "[NO_RESPONSE]"
: job.response.text; : job.response.text;
const validation = judgeConfig.validator const validation = validateOnboardingResponse(
? judgeConfig.validator(responseTextForValidation, job.derived) responseTextForValidation,
: { job.derived,
reasons: [], {
status: "pass" as const, allowShortResponse: job.derived.allowShortResponse,
text: responseTextForValidation, },
}; );
const llmJudge = await judgeSkillWithLlm( const llmJudge = await judgeOnboardingWithLlm(
this.llmClient, this.llmClient,
job.skill,
toEvalCase(job, message.id), toEvalCase(job, message.id),
job.response.text, job.response.text,
derivedSnapshot, toDerivedSnapshot(job.derived),
); );
const llmOverall = llmJudge.skipped ? null : llmJudge.scores.overall; const llmOverall = llmJudge.skipped ? null : llmJudge.scores.overall;
const llmJudgeErrored = const llmJudgeErrored =
@@ -599,7 +531,6 @@ export class ShadowWorker {
llmOverall, llmOverall,
recordedAt: Date.now(), recordedAt: Date.now(),
responseLatencyMs: job.latencyMs ?? null, responseLatencyMs: job.latencyMs ?? null,
skill: job.skill,
}); });
const resultFields: Record<string, string> = { const resultFields: Record<string, string> = {
@@ -607,12 +538,10 @@ export class ShadowWorker {
requestId: job.request.requestId, requestId: job.request.requestId,
chatroomId: job.request.chatroomId, chatroomId: job.request.chatroomId,
userId: job.request.userId, userId: job.request.userId,
skill: job.skill, tier: String(job.derived.tier),
version: String(Math.max(2, job.version)), intent: job.derived.intent.kind,
tier: String(derivedSnapshot.tier), phase: job.derived.phase.kind,
intent: derivedSnapshot.intent, responseMode: job.derived.responseMode,
phase: derivedSnapshot.phase,
responseMode: derivedSnapshot.responseMode,
validatorStatus: validation.status, validatorStatus: validation.status,
latencyMs: job.latencyMs !== undefined ? String(job.latencyMs) : "", latencyMs: job.latencyMs !== undefined ? String(job.latencyMs) : "",
llmOverall: llmOverall === null ? "" : String(llmOverall), llmOverall: llmOverall === null ? "" : String(llmOverall),
@@ -657,12 +586,7 @@ export class ShadowWorker {
throw new Error("Missing process trace"); throw new Error("Missing process trace");
} }
const judgeConfig = getSkillJudgeConfig(job.skill); const ruleResult = judgeProcessRule(toEvalCase(job, messageId).expected, job.process);
const ruleResult = judgeProcessRule(
toEvalCase(job, messageId).expected,
job.process,
judgeConfig.toolRegistryFactory(),
);
if ( if (
!getEvalProcessLlmJudgeEnabled() || !getEvalProcessLlmJudgeEnabled() ||
Math.random() >= getEvalProcessLlmJudgeSamplingRate() Math.random() >= getEvalProcessLlmJudgeSamplingRate()
@@ -952,22 +876,17 @@ export class ShadowWorker {
return; return;
} }
const threshold = getAlertThresholdDriftPct();
const reports = await computeDriftReports(this.mongoConnection, { nowMs: now }); const reports = await computeDriftReports(this.mongoConnection, { nowMs: now });
const significantReports = reports.filter((report) => { const significantReports = reports.filter((report) => report.significant);
const threshold = getAlertThresholdDriftPct(report.slice.skill);
return report.recent24hCount >= 30 && report.dropPct >= threshold;
});
for (const report of significantReports) { for (const report of significantReports) {
const slice = sliceKeyToString(report.slice); const slice = sliceKeyToString(report.slice);
const skill = report.slice.skill ?? "onboarding";
const threshold = getAlertThresholdDriftPct(skill);
await this.alertSender.sendAlert( await this.alertSender.sendAlert(
{ {
kind: "eval-drift", kind: "eval-drift",
metric: "shadow_eval.drift.llm_overall_drop_pct", metric: "shadow_eval.drift.llm_overall_drop_pct",
severity: report.dropPct >= threshold * 1.5 ? "critical" : "warn", severity: report.dropPct >= threshold * 1.5 ? "critical" : "warn",
skill,
suggestedAction: `${slice} slice의 최근 실패 케이스와 prompt/tool 변경 이력을 확인해 주세요.`, suggestedAction: `${slice} slice의 최근 실패 케이스와 prompt/tool 변경 이력을 확인해 주세요.`,
threshold, threshold,
timestamp: new Date(now).toISOString(), timestamp: new Date(now).toISOString(),
@@ -1078,40 +997,29 @@ export class ShadowWorker {
} }
private async checkLatencyAlert(now: number): Promise<void> { private async checkLatencyAlert(now: number): Promise<void> {
if (!this.alertSender) { const latencySamples = this.recentEvalObservations.flatMap((observation) =>
observation.responseLatencyMs === null ? [] : [observation.responseLatencyMs],
);
if (latencySamples.length < MIN_LATENCY_OBSERVATIONS || !this.alertSender) {
return; return;
} }
for (const [skill, observations] of groupObservationsBySkill( const p95LatencyMs = calculatePercentile(latencySamples, 0.95);
this.recentEvalObservations, const threshold = getAlertThresholdP95LatencyMs();
)) { if (p95LatencyMs === null || p95LatencyMs <= threshold) {
const latencySamples = observations.flatMap((observation) => return;
observation.responseLatencyMs === null
? []
: [observation.responseLatencyMs],
);
if (latencySamples.length < MIN_LATENCY_OBSERVATIONS) {
continue;
}
const p95LatencyMs = calculatePercentile(latencySamples, 0.95);
const threshold = getAlertThresholdP95LatencyMs(skill);
if (p95LatencyMs === null || p95LatencyMs <= threshold) {
continue;
}
await this.alertSender.sendAlert({
kind: "latency",
metric: "shadow_eval.p95_latency_ms",
severity: p95LatencyMs >= threshold * 1.5 ? "critical" : "warn",
skill,
suggestedAction: "최근 LLM endpoint 지연과 tool loop 호출 수를 확인해 주세요.",
threshold,
timestamp: new Date(now).toISOString(),
value: p95LatencyMs,
window: ALERT_WINDOW_LABEL,
});
} }
await this.alertSender.sendAlert({
kind: "latency",
metric: "shadow_eval.p95_latency_ms",
severity: p95LatencyMs >= threshold * 1.5 ? "critical" : "warn",
suggestedAction: "최근 LLM endpoint 지연과 tool loop 호출 수를 확인해 주세요.",
threshold,
timestamp: new Date(now).toISOString(),
value: p95LatencyMs,
window: ALERT_WINDOW_LABEL,
});
} }
private async checkEvalScoreDropAlert(now: number): Promise<void> { private async checkEvalScoreDropAlert(now: number): Promise<void> {
@@ -1119,70 +1027,59 @@ export class ShadowWorker {
return; return;
} }
const baselineAverage = await this.loadBaselineOverallScore(); const scoreSamples = this.recentEvalObservations.flatMap((observation) =>
for (const [skill, observations] of groupObservationsBySkill( observation.llmOverall === null ? [] : [observation.llmOverall],
this.recentEvalObservations, );
)) { if (scoreSamples.length < MIN_SCORE_OBSERVATIONS) {
const scoreSamples = observations.flatMap((observation) =>
observation.llmOverall === null ? [] : [observation.llmOverall],
);
if (scoreSamples.length < MIN_SCORE_OBSERVATIONS) {
continue;
}
const currentAverage =
scoreSamples.reduce((sum, score) => sum + score, 0) / scoreSamples.length;
const dropPct = calculateEvalScoreDropPct(currentAverage, baselineAverage);
const threshold = getAlertThresholdEvalScoreDropPct(skill);
if (dropPct === null || dropPct <= threshold) {
continue;
}
await this.alertSender.sendAlert({
kind: "eval-score-drop",
metric: "shadow_eval.llm_overall_drop_pct",
severity: dropPct >= threshold * 1.5 ? "critical" : "warn",
skill,
suggestedAction: "최근 prompt 변경, tool 결과, fallback 비율을 함께 점검해 주세요.",
threshold,
timestamp: new Date(now).toISOString(),
value: Number(dropPct.toFixed(3)),
window: ALERT_WINDOW_LABEL,
});
}
}
private async checkLlmErrorRateAlert(now: number): Promise<void> {
if (!this.alertSender) {
return; return;
} }
for (const [skill, observations] of groupObservationsBySkill( const currentAverage =
this.recentEvalObservations, scoreSamples.reduce((sum, score) => sum + score, 0) / scoreSamples.length;
)) { const baselineAverage = await this.loadBaselineOverallScore();
if (observations.length < MIN_ERROR_RATE_OBSERVATIONS) { const dropPct = calculateEvalScoreDropPct(currentAverage, baselineAverage);
continue; const threshold = getAlertThresholdEvalScoreDropPct();
}
const errorRate = calculateLlmErrorRate(observations); if (dropPct === null || dropPct <= threshold) {
const threshold = getAlertThresholdLlmErrorRate(skill); return;
if (errorRate === null || errorRate <= threshold) {
continue;
}
await this.alertSender.sendAlert({
kind: "llm-error-rate",
metric: "shadow_eval.llm_error_rate",
severity: errorRate >= threshold * 1.5 ? "critical" : "warn",
skill,
suggestedAction: "judge endpoint 응답 상태와 circuit breaker open 여부를 확인해 주세요.",
threshold,
timestamp: new Date(now).toISOString(),
value: Number(errorRate.toFixed(4)),
window: ALERT_WINDOW_LABEL,
});
} }
await this.alertSender.sendAlert({
kind: "eval-score-drop",
metric: "shadow_eval.llm_overall_drop_pct",
severity: dropPct >= threshold * 1.5 ? "critical" : "warn",
suggestedAction: "최근 prompt 변경, tool 결과, fallback 비율을 함께 점검해 주세요.",
threshold,
timestamp: new Date(now).toISOString(),
value: Number(dropPct.toFixed(3)),
window: ALERT_WINDOW_LABEL,
});
}
private async checkLlmErrorRateAlert(now: number): Promise<void> {
if (
!this.alertSender ||
this.recentEvalObservations.length < MIN_ERROR_RATE_OBSERVATIONS
) {
return;
}
const errorRate = calculateLlmErrorRate(this.recentEvalObservations);
const threshold = getAlertThresholdLlmErrorRate();
if (errorRate === null || errorRate <= threshold) {
return;
}
await this.alertSender.sendAlert({
kind: "llm-error-rate",
metric: "shadow_eval.llm_error_rate",
severity: errorRate >= threshold * 1.5 ? "critical" : "warn",
suggestedAction: "judge endpoint 응답 상태와 circuit breaker open 여부를 확인해 주세요.",
threshold,
timestamp: new Date(now).toISOString(),
value: Number(errorRate.toFixed(4)),
window: ALERT_WINDOW_LABEL,
});
} }
private async checkOpenCircuitAlerts(now: number): Promise<void> { private async checkOpenCircuitAlerts(now: number): Promise<void> {
@@ -1226,10 +1123,8 @@ export class ShadowWorker {
return redisParsed; return redisParsed;
} }
const filePath = EVAL_BASELINE_FILES const filePath = resolve(process.cwd(), EVAL_BASELINE_FILE);
.map((candidate) => resolve(process.cwd(), candidate)) if (!existsSync(filePath)) {
.find((candidate) => existsSync(candidate));
if (!filePath) {
return null; return null;
} }
+4 -44
View File
@@ -2,22 +2,14 @@ import type {
AgentRequest, AgentRequest,
AgentResponse, AgentResponse,
ConversationMessage, ConversationMessage,
HandoffReason,
} from "@agent/types"; } from "@agent/types";
import type { Blackboard, OnboardingSession } from "@skills/onboarding/types"; import type { Blackboard, OnboardingSession } from "@skills/onboarding/types";
import type { UserProfile } from "@skills/onboarding/state/user-memory";
import type { ClassifiedIntent, DerivedState } from "@skills/onboarding/state/onboarding-state"; import type { ClassifiedIntent, DerivedState } from "@skills/onboarding/state/onboarding-state";
import type { import type {
ProcessTraceSnapshot, ProcessTraceSnapshot,
ProcessTraceStepName, ProcessTraceStepName,
} from "@skills/onboarding/runtime/trace"; } from "@skills/onboarding/runtime/trace";
import type { PrmFeatureVector } from "./judges/prm.js"; import type { PrmFeatureVector } from "./judges/prm.js";
import type {
MemoryJudgeExpected,
MemoryJudgeResult,
} from "./judges/memory-judge.js";
export type EvalProcessTraceSnapshot = ProcessTraceSnapshot;
export interface EvalRoomContextMessage { export interface EvalRoomContextMessage {
senderId: string; senderId: string;
@@ -33,21 +25,9 @@ export interface EvalCaseInput {
session?: OnboardingSession | null | undefined; session?: OnboardingSession | null | undefined;
blackboard?: Blackboard | null | undefined; blackboard?: Blackboard | null | undefined;
summary?: string | undefined; summary?: string | undefined;
userProfile?: UserProfile | null | undefined;
} }
export interface EvalCaseExpected { export interface EvalCaseExpected {
expectedFinalSkill?: string | undefined;
expectedHandoff?: {
expectedReason?: HandoffReason | undefined;
expectedTarget?: string | undefined;
shouldHandoff?: boolean | undefined;
} | undefined;
expectedMemory?: MemoryJudgeExpected | undefined;
expectedSourceChain?: {
chain: string[];
mode: "contains" | "exact" | "prefix";
} | undefined;
intent?: string | undefined; intent?: string | undefined;
mustIncludeAny?: string[] | undefined; mustIncludeAny?: string[] | undefined;
mustNotInclude?: string[] | undefined; mustNotInclude?: string[] | undefined;
@@ -90,7 +70,10 @@ export interface RuleJudgeResult {
} }
export interface LlmJudgeScores { export interface LlmJudgeScores {
[axis: string]: number; friendliness: number;
accuracy: number;
concision: number;
scope: number;
overall: number; overall: number;
} }
@@ -139,27 +122,6 @@ export interface ProcessJudgeResult {
totalCount: number; totalCount: number;
} }
export interface HandoffJudgeCheck {
detail?: string | undefined;
name:
| "chain"
| "final-skill"
| "handoff-reason"
| "handoff-target"
| "should-handoff";
passed: boolean;
skipped: boolean;
}
export interface HandoffJudgeResult {
actualChain: string[];
actualFinalSkill: string;
actualHandoffReason?: HandoffReason | undefined;
actualHandoffTarget?: string | undefined;
checks: HandoffJudgeCheck[];
passed: boolean;
}
export interface EvalDerivedSnapshot { export interface EvalDerivedSnapshot {
abstentionScore?: number | undefined; abstentionScore?: number | undefined;
allowShortResponse: boolean; allowShortResponse: boolean;
@@ -189,8 +151,6 @@ export interface EvalCaseResult {
response: AgentResponse; response: AgentResponse;
ruleJudge: RuleJudgeResult; ruleJudge: RuleJudgeResult;
llmJudge: LlmJudgeResult; llmJudge: LlmJudgeResult;
handoffJudge?: HandoffJudgeResult | undefined;
memoryJudge?: MemoryJudgeResult | undefined;
processJudge?: ProcessJudgeResult | undefined; processJudge?: ProcessJudgeResult | undefined;
processTrace?: ProcessTraceSnapshot | undefined; processTrace?: ProcessTraceSnapshot | undefined;
error?: string | undefined; error?: string | undefined;
-17
View File
@@ -1,17 +0,0 @@
const MODEL_RATES: Record<string, { inputPer1M: number; outputPer1M: number }> = {
"gpt-5.3-codex-spark": { inputPer1M: 0.15, outputPer1M: 0.6 },
"google/gemma-4-26b-a4b-it:free": { inputPer1M: 0, outputPer1M: 0 },
"google/gemma-4-31b-it:free": { inputPer1M: 0, outputPer1M: 0 },
};
export function estimateChatCost(
model: string,
tokensIn = 0,
tokensOut = 0,
): number {
const rate = MODEL_RATES[model] ?? { inputPer1M: 0.1, outputPer1M: 0.3 };
return Number(
((tokensIn / 1_000_000) * rate.inputPer1M +
(tokensOut / 1_000_000) * rate.outputPer1M).toFixed(6),
);
}
+1 -184
View File
@@ -1,38 +1,9 @@
import { afterEach, describe, expect, it, vi } from "vitest"; import { afterEach, describe, expect, it, vi } from "vitest";
const llmSpanMock = vi.hoisted(() => ({
initialAttributes: [] as Array<Record<string, unknown> | undefined>,
setAttributes: vi.fn((attributes: Record<string, unknown>) => attributes),
}));
vi.mock("@obs/spans", () => ({
maskTraceContent: (content: string) =>
content
.replace(
/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/giu,
"[redacted-email]",
)
.replace(/\b\d{2,3}[-.\s]?\d{3,4}[-.\s]?\d{4}\b/gu, "[redacted-phone]"),
withLlmSpan: vi.fn(
(
attributes: Record<string, unknown> | undefined,
fn: (span: { setAttributes: typeof llmSpanMock.setAttributes }) => unknown,
) => {
llmSpanMock.initialAttributes.push(attributes);
return Promise.resolve(fn({ setAttributes: llmSpanMock.setAttributes }));
},
),
}));
import { LlmClient } from "./llm-client.js"; import { LlmClient } from "./llm-client.js";
import { BudgetTracker } from "@agent/budget";
import { runWithContext } from "@obs/context";
import type { ChatCompletionResponse } from "./types.js"; import type { ChatCompletionResponse } from "./types.js";
function okResponse( function okResponse(content = "ok"): Response {
content = "ok",
overrides: Partial<ChatCompletionResponse> = {},
): Response {
const payload: ChatCompletionResponse = { const payload: ChatCompletionResponse = {
choices: [ choices: [
{ {
@@ -44,7 +15,6 @@ function okResponse(
}, },
], ],
model: "test-model", model: "test-model",
...overrides,
}; };
return new Response(JSON.stringify(payload), { return new Response(JSON.stringify(payload), {
@@ -65,10 +35,7 @@ function stringifyFetchUrl(input: string | URL | Request): string {
describe("LlmClient endpoint normalization", () => { describe("LlmClient endpoint normalization", () => {
afterEach(() => { afterEach(() => {
llmSpanMock.initialAttributes.length = 0;
llmSpanMock.setAttributes.mockClear();
vi.unstubAllGlobals(); vi.unstubAllGlobals();
vi.unstubAllEnvs();
}); });
it("normalizes OpenRouter API base URLs to chat completions", async () => { it("normalizes OpenRouter API base URLs to chat completions", async () => {
@@ -119,154 +86,4 @@ describe("LlmClient endpoint normalization", () => {
expect(result?.content).toBe("ok"); expect(result?.content).toBe("ok");
expect(urls).toEqual(["https://openrouter.ai/api/v1/chat/completions"]); expect(urls).toEqual(["https://openrouter.ai/api/v1/chat/completions"]);
}); });
it("records OpenInference token, cost, and masked prompt attributes", async () => {
vi.stubEnv("PHOENIX_LOG_PROMPTS", "true");
vi.stubGlobal(
"fetch",
vi.fn(() =>
Promise.resolve(
okResponse("reply to 010-9999-8888", {
id: "chatcmpl-test",
usage: {
completion_tokens: 50,
prompt_tokens: 100,
total_tokens: 150,
},
}),
),
),
);
const client = new LlmClient({
endpoints: [{ provider: "openrouter", url: "https://openrouter.ai/api/v1" }],
providerApiKeys: { openrouter: "test-key" },
});
const result = await client.chatCompletion(
[{ content: "hi a@example.com 010-1234-5678", role: "user" }],
{
maxTokens: 128,
modelOverride: "gpt-5.3-codex-spark",
temperature: 0.25,
},
);
expect(result?.content).toBe("reply to 010-9999-8888");
expect(llmSpanMock.initialAttributes.at(-1)).toMatchObject({
"gen_ai.request.max_tokens": 128,
"gen_ai.request.model": "gpt-5.3-codex-spark",
"gen_ai.request.temperature": 0.25,
"llm.input_messages.0.message.content":
"hi [redacted-email] [redacted-phone]",
"llm.model_name": "gpt-5.3-codex-spark",
});
expect(llmSpanMock.setAttributes).toHaveBeenCalledWith(
expect.objectContaining({
"gen_ai.response.id": "chatcmpl-test",
"gen_ai.usage.cost_usd": 0.000045,
"llm.output_messages.0.message.content": "reply to [redacted-phone]",
"llm.token_count.completion": 50,
"llm.token_count.prompt": 100,
}),
);
});
it("records LLM cost into the request budget", async () => {
vi.stubGlobal(
"fetch",
vi.fn(() =>
Promise.resolve(
okResponse("reply", {
usage: {
completion_tokens: 50,
prompt_tokens: 100,
total_tokens: 150,
},
}),
),
),
);
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
const client = new LlmClient({
endpoints: [{ provider: "openrouter", url: "https://openrouter.ai/api/v1" }],
providerApiKeys: { openrouter: "test-key" },
});
await runWithContext(
{
budget,
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() =>
client.chatCompletion([{ content: "hi", role: "user" }], {
modelOverride: "gpt-5.3-codex-spark",
}),
);
expect(budget.snapshot().totalLlmCostUsd).toBe(0.000045);
expect(llmSpanMock.setAttributes).toHaveBeenCalledWith(
expect.objectContaining({
"budget.llm_cost_usd": 0.000045,
"budget.total_cost_usd": 0.000045,
}),
);
});
it("does not record LLM cost when budget cost tracking is disabled", async () => {
vi.stubEnv("AGENT_BUDGET_LLM_COST_ENABLED", "false");
vi.stubGlobal(
"fetch",
vi.fn(() =>
Promise.resolve(
okResponse("reply", {
usage: {
completion_tokens: 50,
prompt_tokens: 100,
total_tokens: 150,
},
}),
),
),
);
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
const client = new LlmClient({
endpoints: [{ provider: "openrouter", url: "https://openrouter.ai/api/v1" }],
providerApiKeys: { openrouter: "test-key" },
});
await runWithContext(
{
budget,
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() =>
client.chatCompletion([{ content: "hi", role: "user" }], {
modelOverride: "gpt-5.3-codex-spark",
}),
);
expect(budget.snapshot().totalLlmCostUsd).toBe(0);
});
}); });
+3 -85
View File
@@ -7,20 +7,16 @@ import type {
LlmEndpointConfig, LlmEndpointConfig,
ProviderName, ProviderName,
} from "./types.js"; } from "./types.js";
import { getContext, getRequestLogger } from "@obs/context"; import { getRequestLogger } from "@obs/context";
import { maskTraceContent, withLlmSpan } from "@obs/spans"; import { withLlmSpan } from "@obs/spans";
import { import {
getAgentBudgetLlmCostEnabled,
getLlmHealthTimeoutMs, getLlmHealthTimeoutMs,
getLlmRequestTimeoutMs, getLlmRequestTimeoutMs,
getLlmTemperature, getLlmTemperature,
getPhoenixLogPrompts,
} from "../config.js"; } from "../config.js";
import { LlmCircuitBreaker } from "./circuit.js"; import { LlmCircuitBreaker } from "./circuit.js";
import { getProviderAdapter } from "./providers/index.js"; import { getProviderAdapter } from "./providers/index.js";
import { selectMaxTokensForTier, selectModelsForProvider, selectProviderOrder } from "./routing.js"; import { selectMaxTokensForTier, selectModelsForProvider, selectProviderOrder } from "./routing.js";
import { estimateChatCost } from "./cost.js";
import { metrics } from "@obs/metrics";
const DEFAULT_MODEL = "Qwen3.5-4B"; const DEFAULT_MODEL = "Qwen3.5-4B";
@@ -86,33 +82,6 @@ function getRequestSignal(
return controller.signal; return controller.signal;
} }
function buildPromptTraceAttributes(
messages: readonly ChatMessage[],
): Record<string, string> {
return Object.fromEntries(
messages.flatMap((message, index) => [
[`llm.input_messages.${index}.message.role`, message.role],
[
`llm.input_messages.${index}.message.content`,
maskTraceContent(message.content),
],
]),
);
}
function buildCompletionTraceAttributes(
result: ChatCompletionResult | null,
): Record<string, string> {
if (!result) {
return {};
}
return {
"llm.output_messages.0.message.content": maskTraceContent(result.content),
"llm.output_messages.0.message.role": result.message.role,
};
}
export class LlmClient { export class LlmClient {
private readonly circuit: LlmCircuitBreaker; private readonly circuit: LlmCircuitBreaker;
private readonly endpoints: LlmEndpointConfig[]; private readonly endpoints: LlmEndpointConfig[];
@@ -194,10 +163,9 @@ export class LlmClient {
continue; continue;
} }
const configuredRequestTimeoutMs = getLlmRequestTimeoutMs();
const requestTimeoutMs = Math.max( const requestTimeoutMs = Math.max(
1, 1,
Math.min(configuredRequestTimeoutMs, options?.deadlineMs ?? Number.POSITIVE_INFINITY), Math.min(getLlmRequestTimeoutMs(), options?.deadlineMs ?? Number.POSITIVE_INFINITY),
); );
const models = selectModelsForProvider(provider.name, options, this.llamaModel); const models = selectModelsForProvider(provider.name, options, this.llamaModel);
let endpointSucceeded = false; let endpointSucceeded = false;
@@ -212,18 +180,11 @@ export class LlmClient {
temperature: options?.temperature ?? getLlmTemperature(), temperature: options?.temperature ?? getLlmTemperature(),
tools: options?.tools, tools: options?.tools,
}); });
const shouldLogPrompts = getPhoenixLogPrompts();
const llmResult = await withLlmSpan( const llmResult = await withLlmSpan(
{ {
...(shouldLogPrompts
? buildPromptTraceAttributes(request.messages)
: {}),
"gen_ai.request.max_tokens": request.max_tokens,
"gen_ai.request.model": request.model, "gen_ai.request.model": request.model,
"gen_ai.request.temperature": request.temperature,
"gen_ai.system": provider.name, "gen_ai.system": provider.name,
"llm.model_name": request.model,
}, },
async (span) => { async (span) => {
const response = await fetch(endpoint.completionUrl, { const response = await fetch(endpoint.completionUrl, {
@@ -242,9 +203,6 @@ export class LlmClient {
model: request.model, model: request.model,
status: response.status, status: response.status,
body, body,
configuredTimeoutMs: configuredRequestTimeoutMs,
deadlineMs: options?.deadlineMs,
timeoutMs: requestTimeoutMs,
}, },
"[LLM] Request failed", "[LLM] Request failed",
); );
@@ -253,40 +211,13 @@ export class LlmClient {
const payload = (await response.json()) as ChatCompletionResponse; const payload = (await response.json()) as ChatCompletionResponse;
const result = provider.toResult(payload, endpoint.completionUrl); const result = provider.toResult(payload, endpoint.completionUrl);
const costUsd = estimateChatCost(
request.model,
payload.usage?.prompt_tokens,
payload.usage?.completion_tokens,
);
metrics.tokenCostUsd.observe(costUsd, {
model: request.model,
skill: getContext()?.skill,
});
if (getAgentBudgetLlmCostEnabled()) {
getContext()?.budget?.recordLlmCost(costUsd);
}
const budgetSnapshot = getContext()?.budget?.snapshot();
span.setAttributes({ span.setAttributes({
...(shouldLogPrompts
? buildCompletionTraceAttributes(result)
: {}),
...(budgetSnapshot
? {
"budget.llm_cost_usd": budgetSnapshot.totalLlmCostUsd,
"budget.tool_cost_usd": budgetSnapshot.totalToolCostUsd,
"budget.total_cost_usd": budgetSnapshot.totalCostUsd,
}
: {}),
"gen_ai.response.finish_reasons": payload.choices "gen_ai.response.finish_reasons": payload.choices
?.map((choice) => choice.finish_reason ?? "unknown") ?.map((choice) => choice.finish_reason ?? "unknown")
.filter((reason): reason is string => Boolean(reason)), .filter((reason): reason is string => Boolean(reason)),
"gen_ai.response.id": payload.id,
"gen_ai.usage.cost_usd": costUsd,
"gen_ai.usage.input_tokens": payload.usage?.prompt_tokens, "gen_ai.usage.input_tokens": payload.usage?.prompt_tokens,
"gen_ai.usage.output_tokens": payload.usage?.completion_tokens, "gen_ai.usage.output_tokens": payload.usage?.completion_tokens,
"llm.token_count.completion": payload.usage?.completion_tokens,
"llm.token_count.prompt": payload.usage?.prompt_tokens,
}); });
if (!result) { if (!result) {
@@ -308,16 +239,6 @@ export class LlmClient {
if (llmResult) { if (llmResult) {
await this.circuit.recordSuccess(endpoint.completionUrl); await this.circuit.recordSuccess(endpoint.completionUrl);
endpointSucceeded = true; endpointSucceeded = true;
logger.debug(
{
endpoint: endpoint.completionUrl,
model: request.model,
provider: provider.name,
role: options?.role,
tier: options?.tier,
},
"[LLM] Response received",
);
return llmResult; return llmResult;
} }
} catch (error) { } catch (error) {
@@ -327,9 +248,6 @@ export class LlmClient {
provider: provider.name, provider: provider.name,
model, model,
error: error instanceof Error ? error.message : String(error), error: error instanceof Error ? error.message : String(error),
configuredTimeoutMs: configuredRequestTimeoutMs,
deadlineMs: options?.deadlineMs,
timeoutMs: requestTimeoutMs,
}, },
"[LLM] Request error", "[LLM] Request error",
); );
+6 -63
View File
@@ -1,7 +1,6 @@
import { afterEach, describe, expect, it, vi } from "vitest"; import { afterEach, describe, expect, it, vi } from "vitest";
import type { LlmEndpointConfig } from "./types.js"; import type { LlmEndpointConfig } from "./types.js";
import { import {
resetLlmRoutingRoundRobinForTests,
selectMaxTokensForTier, selectMaxTokensForTier,
selectModelsForProvider, selectModelsForProvider,
selectProviderOrder, selectProviderOrder,
@@ -9,12 +8,6 @@ import {
describe("llm routing", () => { describe("llm routing", () => {
afterEach(() => { afterEach(() => {
delete process.env["CLIPROXY_TIER0_MODEL"];
delete process.env["CLIPROXY_TIER1_PLANNER_MODEL"];
delete process.env["CLIPROXY_TIER2_CRITIC_MODEL"];
delete process.env["LLAMA_TIER2_CRITIC_MODEL"];
delete process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"];
delete process.env["OPENROUTER_TIER1_RESPONDER_MODEL"];
delete process.env["OPENROUTER_MODEL"]; delete process.env["OPENROUTER_MODEL"];
delete process.env["TIER0_MODEL"]; delete process.env["TIER0_MODEL"];
delete process.env["TIER1_PLANNER_MODEL"]; delete process.env["TIER1_PLANNER_MODEL"];
@@ -23,7 +16,6 @@ describe("llm routing", () => {
delete process.env["TIER0_MAX_TOKENS"]; delete process.env["TIER0_MAX_TOKENS"];
delete process.env["TIER1_MAX_TOKENS"]; delete process.env["TIER1_MAX_TOKENS"];
delete process.env["TIER2_MAX_TOKENS"]; delete process.env["TIER2_MAX_TOKENS"];
resetLlmRoutingRoundRobinForTests();
vi.restoreAllMocks(); vi.restoreAllMocks();
}); });
@@ -101,12 +93,9 @@ describe("llm routing", () => {
).toEqual(["cliproxy", "openrouter"]); ).toEqual(["cliproxy", "openrouter"]);
}); });
it("returns provider-aware models for runtime roles and tiers", () => { it("returns role-aware model for cliproxy and llama fallback as single-item list", () => {
process.env["CLIPROXY_TIER1_PLANNER_MODEL"] = "cliproxy-planner"; process.env["TIER1_PLANNER_MODEL"] = "cliproxy-planner";
process.env["CLIPROXY_TIER2_CRITIC_MODEL"] = "cliproxy-critic"; process.env["TIER2_CRITIC_MODEL"] = "cliproxy-critic";
process.env["OPENROUTER_TIER1_RESPONDER_MODEL"] =
"openrouter-fast, openrouter-backup";
process.env["LLAMA_TIER2_CRITIC_MODEL"] = "llama-critic";
expect( expect(
selectModelsForProvider("cliproxy", { role: "planner", tier: 1 }, "local-qwen"), selectModelsForProvider("cliproxy", { role: "planner", tier: 1 }, "local-qwen"),
@@ -114,12 +103,9 @@ describe("llm routing", () => {
expect( expect(
selectModelsForProvider("cliproxy", { role: "critic", tier: 2 }, "local-qwen"), selectModelsForProvider("cliproxy", { role: "critic", tier: 2 }, "local-qwen"),
).toEqual(["cliproxy-critic"]); ).toEqual(["cliproxy-critic"]);
expect( expect(selectModelsForProvider("llama", { tier: 2 }, "local-qwen")).toEqual([
selectModelsForProvider("openrouter", { role: "responder", tier: 1 }, "local-qwen"), "local-qwen",
).toEqual(["openrouter-fast", "openrouter-backup"]); ]);
expect(
selectModelsForProvider("llama", { role: "critic", tier: 2 }, "local-qwen"),
).toEqual(["llama-critic"]);
}); });
it("splits OPENROUTER_MODEL by comma and preserves priority order", () => { it("splits OPENROUTER_MODEL by comma and preserves priority order", () => {
@@ -138,49 +124,6 @@ describe("llm routing", () => {
]); ]);
}); });
it("round-robins models within the same provider role and tier when enabled", () => {
process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"] = "true";
process.env["CLIPROXY_TIER0_MODEL"] =
"gemini-3.1-flash-lite, gpt-5.3-codex-spark";
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gemini-3.1-flash-lite", "gpt-5.3-codex-spark"]);
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gpt-5.3-codex-spark", "gemini-3.1-flash-lite"]);
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gemini-3.1-flash-lite", "gpt-5.3-codex-spark"]);
});
it("keeps provider order fixed even when model round-robin is enabled", () => {
process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"] = "true";
const endpoints: LlmEndpointConfig[] = [
{
completionUrl: "http://llama.test/v1/chat/completions",
healthUrl: "http://llama.test/health",
provider: "llama",
},
{
completionUrl: "http://openrouter.test/v1/chat/completions",
healthUrl: "http://openrouter.test",
provider: "openrouter",
},
{
completionUrl: "http://cliproxy.test/v1/chat/completions",
healthUrl: "http://cliproxy.test",
provider: "cliproxy",
},
];
expect(
selectProviderOrder(endpoints, { role: "responder", tier: 1 }).map(
(endpoint) => endpoint.provider,
),
).toEqual(["cliproxy", "openrouter", "llama"]);
});
it("chooses tier defaults for max tokens when explicit maxTokens is absent", () => { it("chooses tier defaults for max tokens when explicit maxTokens is absent", () => {
process.env["TIER0_MAX_TOKENS"] = "111"; process.env["TIER0_MAX_TOKENS"] = "111";
process.env["TIER1_MAX_TOKENS"] = "222"; process.env["TIER1_MAX_TOKENS"] = "222";
+26 -37
View File
@@ -1,8 +1,11 @@
import { import {
getLlmModelRoundRobinEnabled, getOpenRouterModels,
getRuntimeLlmModelsForProvider,
getTier0MaxTokens, getTier0MaxTokens,
getTier0Model,
getTier1MaxTokens, getTier1MaxTokens,
getTier1PlannerModel,
getTier1ResponderModel,
getTier2CriticModel,
getTier2MaxTokens, getTier2MaxTokens,
getLlmMaxTokens, getLlmMaxTokens,
} from "../config.js"; } from "../config.js";
@@ -10,7 +13,6 @@ import type { ChatCompletionOptions, LlmEndpointConfig, ProviderName } from "./t
const PROVIDER_PRIORITY: readonly ProviderName[] = ["cliproxy", "openrouter", "llama"]; const PROVIDER_PRIORITY: readonly ProviderName[] = ["cliproxy", "openrouter", "llama"];
const TEMPLATE_ONLY_PROVIDER_PRIORITY: readonly ProviderName[] = ["llama"]; const TEMPLATE_ONLY_PROVIDER_PRIORITY: readonly ProviderName[] = ["llama"];
const modelRoundRobinOffsets = new Map<string, number>();
function getProviderPriority( function getProviderPriority(
options: ChatCompletionOptions | undefined, options: ChatCompletionOptions | undefined,
@@ -31,15 +33,27 @@ export function selectModelsForProvider(
return [options.modelOverride.trim()]; return [options.modelOverride.trim()];
} }
const models = getRuntimeLlmModelsForProvider( if (provider === "llama") {
provider, return [llamaFallbackModel];
{ }
role: options?.role,
tier: options?.tier, if (provider === "openrouter") {
}, return getOpenRouterModels();
llamaFallbackModel, }
);
return rotateModelsForProvider(provider, options, models); if (options?.role === "planner") {
return [getTier1PlannerModel()];
}
if (options?.role === "critic") {
return [getTier2CriticModel()];
}
if (options?.tier === 1 || options?.tier === 2) {
return [getTier1ResponderModel()];
}
return [getTier0Model()];
} }
export function selectMaxTokensForTier( export function selectMaxTokensForTier(
@@ -92,28 +106,3 @@ export function selectProviderOrder(
}) })
.map((item) => item.endpoint); .map((item) => item.endpoint);
} }
function rotateModelsForProvider(
provider: ProviderName,
options: ChatCompletionOptions | undefined,
models: string[],
): string[] {
if (!getLlmModelRoundRobinEnabled() || models.length <= 1) {
return models;
}
const key = [
provider,
options?.role ?? "none",
options?.tier ?? "none",
models.join("|"),
].join(":");
const offset = modelRoundRobinOffsets.get(key) ?? 0;
modelRoundRobinOffsets.set(key, offset + 1);
const rotation = offset % models.length;
return [...models.slice(rotation), ...models.slice(0, rotation)];
}
export function resetLlmRoutingRoundRobinForTests(): void {
modelRoundRobinOffsets.clear();
}
-1
View File
@@ -30,7 +30,6 @@ export interface ChatCompletionRequest {
} }
export interface ChatCompletionResponse { export interface ChatCompletionResponse {
id?: string | undefined;
model?: string; model?: string;
choices?: Array<{ choices?: Array<{
finish_reason?: string | null; finish_reason?: string | null;
+11 -67
View File
@@ -3,7 +3,6 @@ import { getOpenRouterApiUrl, getServiceName } from "./config.js";
import type { GarnetClient } from "./common/garnet.js"; import type { GarnetClient } from "./common/garnet.js";
import type { NatsClient } from "./common/nats.js"; import type { NatsClient } from "./common/nats.js";
import type { SkillFactory } from "./skills/skill-factory.js";
loadEnvironmentFromNodeEnv(); loadEnvironmentFromNodeEnv();
@@ -21,24 +20,12 @@ const { GarnetClient: RuntimeGarnetClient } = await import(
const { NatsClient: RuntimeNatsClient } = await import("./common/nats.js"); const { NatsClient: RuntimeNatsClient } = await import("./common/nats.js");
const { Agent } = await import("./agent/index.js"); const { Agent } = await import("./agent/index.js");
const { ConversationStore } = await import("./conversation/index.js"); const { ConversationStore } = await import("./conversation/index.js");
const { createOnboardingSkill } = await import("./skills/onboarding/index.js"); const { OnboardingSkill } = await import("./skills/onboarding/index.js");
const { createModerationSkill } = await import("./skills/moderation/index.js");
const { createSupervisorSkill } = await import("./skills/supervisor/index.js");
const { SkillRegistry } = await import("./skills/skill-loader.js");
const { TransitionMatrix } = await import("./skills/transition-matrix.js");
const { ShadowEvalBus } = await import("./eval/shadow-bus.js"); const { ShadowEvalBus } = await import("./eval/shadow-bus.js");
const { LlmClient } = await import("./llm/index.js"); const { LlmClient } = await import("./llm/index.js");
const { getRawSecret, getSecret } = await import("./common/secret-manager.js"); const { getRawSecret, getSecret } = await import("./common/secret-manager.js");
const { initMongoDB, closeMongoDB } = await import("./common/mongodb.js"); const { initMongoDB, closeMongoDB } = await import("./common/mongodb.js");
const { createMemoryService } = await import("./memory/memory-service.js");
const { createEscalationHistoryService } = await import(
"./memory/mongo-escalation-history.js"
);
const { RoomContextService } = await import("./room-context/index.js"); const { RoomContextService } = await import("./room-context/index.js");
const { startMetricsServer, stopMetricsServer } = await import(
"./observability/metrics-server.js"
);
const { resolve } = await import("node:path");
function firstNonEmpty( function firstNonEmpty(
...values: Array<string | null | undefined> ...values: Array<string | null | undefined>
@@ -205,14 +192,13 @@ async function bootstrap(): Promise<void> {
} }
// MongoDB + RoomContextService setup (optional) // MongoDB + RoomContextService setup (optional)
let mongoConnection: Awaited<ReturnType<typeof initMongoDB>> | undefined;
let roomContextService: InstanceType<typeof RoomContextService> | undefined; let roomContextService: InstanceType<typeof RoomContextService> | undefined;
const mongoConnectionString = firstNonEmpty( const mongoConnectionString = firstNonEmpty(
getSecret("MONGODB_CONNECTION_STRING"), getSecret("MONGODB_CONNECTION_STRING"),
process.env["MONGODB_CONNECTION_STRING"], process.env["MONGODB_CONNECTION_STRING"],
); );
if (mongoConnectionString) { if (mongoConnectionString) {
mongoConnection = await initMongoDB(mongoConnectionString); await initMongoDB(mongoConnectionString);
// 시크릿의 projectId를 환경변수로 전파 (config.ts에서 참조) // 시크릿의 projectId를 환경변수로 전파 (config.ts에서 참조)
const mongoProjectId = firstNonEmpty( const mongoProjectId = firstNonEmpty(
getSecret("MONGODB_PROJECT_ID"), getSecret("MONGODB_PROJECT_ID"),
@@ -227,70 +213,28 @@ async function bootstrap(): Promise<void> {
logger.warn("[MongoDB] No connection string configured — room context disabled"); logger.warn("[MongoDB] No connection string configured — room context disabled");
} }
const memoryService = createMemoryService({
garnetClient,
llmClient,
mongooseConnection: mongoConnection,
});
const escalationHistoryService =
createEscalationHistoryService(mongoConnection);
// Agent setup // Agent setup
const conversationStore = new ConversationStore(garnetClient); const conversationStore = new ConversationStore(garnetClient);
const skillRegistry = SkillRegistry.load(resolve(process.cwd(), "src", "skills")); const agent = new Agent(natsClient, conversationStore);
const transitionMatrix = TransitionMatrix.fromRegistry(skillRegistry);
const agent = new Agent(natsClient, conversationStore, transitionMatrix);
// Register skills // Register skills
const shadowEvalBus = new ShadowEvalBus(garnetClient); const shadowEvalBus = new ShadowEvalBus(garnetClient);
const factories: Record<string, SkillFactory> = { const onboardingSkill = new OnboardingSkill(
moderation: createModerationSkill, garnetClient,
onboarding: createOnboardingSkill, llmClient,
supervisor: createSupervisorSkill, roomContextService,
}; shadowEvalBus,
const seenSkills = new Set<string>(); natsClient,
for (const manifest of skillRegistry.list()) { );
if (seenSkills.has(manifest.name)) { agent.registerSkill(onboardingSkill);
logger.warn(
{ skill: manifest.name },
"[Skill] Duplicate manifest ignored",
);
continue;
}
seenSkills.add(manifest.name);
const factory = factories[manifest.name];
if (!factory) {
logger.warn(
{ skill: manifest.name },
"[Skill] No factory registered",
);
continue;
}
agent.registerSkill(
factory({
garnetClient,
escalationHistoryService,
llmClient,
memoryService,
natsClient,
roomContextService,
shadowEvalBus,
skillRegistry,
}),
);
}
installShutdownHooks(async () => { installShutdownHooks(async () => {
await stopMetricsServer();
await natsClient.close(); await natsClient.close();
await garnetClient.close(); await garnetClient.close();
await closeMongoDB(); await closeMongoDB();
await shutdownTracing(); await shutdownTracing();
}); });
startMetricsServer();
await agent.start(); await agent.start();
logger.info(`✓ ${serviceName} bootstrapped`); logger.info(`✓ ${serviceName} bootstrapped`);
-16
View File
@@ -1,16 +0,0 @@
import { describe, expect, it } from "vitest";
import { buildBM25Index } from "./bm25.js";
describe("buildBM25Index", () => {
it("ranks Korean keyword matches above unrelated documents", () => {
const index = buildBM25Index([
{ id: "ads", text: "광고와 도배 홍보 링크는 제재 대상입니다." },
{ id: "age", text: "가입 가능 연령은 27세 이상입니다." },
]);
const [top] = index.rank("광고 도배 기준", 2);
expect(top?.id).toBe("ads");
expect(top?.score).toBeGreaterThan(0);
});
});
-82
View File
@@ -1,82 +0,0 @@
export interface BM25Document {
id: string;
text: string;
}
export interface BM25Index {
rank(query: string, topK: number): Array<{ id: string; score: number }>;
}
const K1 = 1.5;
const B = 0.75;
function normalize(text: string): string {
return text
.normalize("NFKC")
.toLowerCase()
.replace(/[^\p{Letter}\p{Number}\s]/gu, " ");
}
function tokenize(text: string): string[] {
const words = normalize(text).split(/\s+/u).filter((token) => token.length > 0);
const compact = normalize(text).replace(/\s+/gu, "");
const grams: string[] = [];
for (let i = 0; i < compact.length - 1; i += 1) {
grams.push(compact.slice(i, i + 2));
}
return [...new Set([...words, ...grams])];
}
export function buildBM25Index(documents: BM25Document[]): BM25Index {
const termFrequencies = new Map<string, Map<string, number>>();
const documentLengths = new Map<string, number>();
const documentFrequency = new Map<string, number>();
let totalLength = 0;
for (const document of documents) {
const tokens = tokenize(document.text);
documentLengths.set(document.id, tokens.length);
totalLength += tokens.length;
const frequencies = new Map<string, number>();
for (const token of tokens) {
frequencies.set(token, (frequencies.get(token) ?? 0) + 1);
}
termFrequencies.set(document.id, frequencies);
for (const token of frequencies.keys()) {
documentFrequency.set(token, (documentFrequency.get(token) ?? 0) + 1);
}
}
const averageLength = documents.length === 0 ? 0 : totalLength / documents.length;
return {
rank(query: string, topK: number): Array<{ id: string; score: number }> {
const queryTokens = tokenize(query);
if (queryTokens.length === 0 || documents.length === 0 || averageLength === 0) {
return [];
}
return documents
.map((document) => {
const frequencies = termFrequencies.get(document.id) ?? new Map<string, number>();
const documentLength = documentLengths.get(document.id) ?? 0;
const score = queryTokens.reduce((sum, token) => {
const tf = frequencies.get(token) ?? 0;
if (tf === 0) {
return sum;
}
const df = documentFrequency.get(token) ?? 0;
const idf = Math.log(1 + (documents.length - df + 0.5) / (df + 0.5));
const denominator = tf + K1 * (1 - B + B * (documentLength / averageLength));
return sum + idf * ((tf * (K1 + 1)) / denominator);
}, 0);
return { id: document.id, score };
})
.filter((item) => item.score > 0)
.sort((left, right) => right.score - left.score)
.slice(0, Math.max(1, topK));
},
};
}
-30
View File
@@ -1,30 +0,0 @@
import { describe, expect, it } from "vitest";
import { detectContradictions, textSimilarity } from "./contradiction-detector.js";
describe("textSimilarity", () => {
it("returns lower scores for very different text", () => {
expect(textSimilarity("닉네임 규칙", "광고 제재 기준")).toBeLessThan(0.7);
});
});
describe("detectContradictions", () => {
it("flags corrections in the same source section", () => {
const [candidate] = detectContradictions([
{
id: "old",
metadata: { sourceFile: "rules.md", sectionTitle: "nickname" },
text: "닉네임은 기존 형식을 사용합니다.",
},
{
id: "new",
metadata: { sourceFile: "rules.md", sectionTitle: "nickname" },
text: "정정: 닉네임은 새 형식을 사용합니다.",
},
]);
expect(candidate).toMatchObject({
existingId: "old",
newId: "new",
});
});
});
-83
View File
@@ -1,83 +0,0 @@
export interface ContradictionCandidate {
existingId: string;
newId: string;
reason: "keyword" | "text-distance";
similarity: number;
}
export interface ContradictionDocument {
id: string;
metadata?: Record<string, unknown> | undefined;
text: string;
}
function groupKey(doc: ContradictionDocument): string | null {
const sourceFile = doc.metadata?.["sourceFile"];
const sectionTitle = doc.metadata?.["sectionTitle"];
if (typeof sourceFile !== "string" || typeof sectionTitle !== "string") {
return null;
}
return `${sourceFile}#${sectionTitle}`;
}
function bigrams(text: string): Set<string> {
const compact = text
.normalize("NFKC")
.toLowerCase()
.replace(/\s+/gu, "");
const grams = new Set<string>();
for (let i = 0; i < compact.length - 1; i += 1) {
grams.add(compact.slice(i, i + 2));
}
return grams;
}
export function textSimilarity(left: string, right: string): number {
const leftGrams = bigrams(left);
const rightGrams = bigrams(right);
if (leftGrams.size === 0 && rightGrams.size === 0) {
return 1;
}
const intersection = [...leftGrams].filter((gram) => rightGrams.has(gram)).length;
const union = new Set([...leftGrams, ...rightGrams]).size;
return union === 0 ? 0 : intersection / union;
}
function hasCorrectionKeyword(text: string): boolean {
return /(변경|정정|수정|폐지|삭제|바뀜|업데이트)/u.test(text);
}
export function detectContradictions(
documents: ContradictionDocument[],
): ContradictionCandidate[] {
const groups = new Map<string, ContradictionDocument[]>();
for (const doc of documents) {
const key = groupKey(doc);
if (!key) {
continue;
}
groups.set(key, [...(groups.get(key) ?? []), doc]);
}
const candidates: ContradictionCandidate[] = [];
for (const docs of groups.values()) {
for (let index = 1; index < docs.length; index += 1) {
const previous = docs[index - 1];
const current = docs[index];
if (!previous || !current) {
continue;
}
const similarity = textSimilarity(previous.text, current.text);
if (similarity < 0.7 || hasCorrectionKeyword(current.text)) {
candidates.push({
existingId: previous.id,
newId: current.id,
reason: hasCorrectionKeyword(current.text) ? "keyword" : "text-distance",
similarity,
});
}
}
}
return candidates;
}
-28
View File
@@ -1,28 +0,0 @@
import { describe, expect, it } from "vitest";
import { cosineSimilarity, rankByCosineSimilarity } from "./cosine.js";
describe("cosineSimilarity", () => {
it("scores identical vectors above unrelated vectors", () => {
expect(cosineSimilarity([1, 0], [1, 0])).toBeCloseTo(1);
expect(cosineSimilarity([1, 0], [0, 1])).toBeCloseTo(0);
});
it("ranks documents by cosine score", () => {
const ranked = rankByCosineSimilarity(
[1, 0],
[
{ embedding: [0, 1], id: "b" },
{ embedding: [1, 0], id: "a" },
],
1,
);
expect(ranked).toEqual([
expect.objectContaining({
id: "a",
score: 1,
}),
]);
});
});
-40
View File
@@ -1,40 +0,0 @@
export function cosineSimilarity(a: number[], b: number[]): number {
if (a.length === 0 || a.length !== b.length) {
return 0;
}
let dot = 0;
let magnitudeA = 0;
let magnitudeB = 0;
for (let index = 0; index < a.length; index += 1) {
const left = a[index] ?? 0;
const right = b[index] ?? 0;
dot += left * right;
magnitudeA += left * left;
magnitudeB += right * right;
}
if (magnitudeA === 0 || magnitudeB === 0) {
return 0;
}
return dot / (Math.sqrt(magnitudeA) * Math.sqrt(magnitudeB));
}
export function rankByCosineSimilarity<T extends { embedding: number[] }>(
query: number[],
docs: T[],
topK: number,
): Array<T & { score: number }> {
if (topK <= 0) {
return [];
}
return docs
.map((doc) => ({
...doc,
score: cosineSimilarity(query, doc.embedding),
}))
.sort((left, right) => right.score - left.score)
.slice(0, topK);
}
-146
View File
@@ -1,146 +0,0 @@
import { getRawSecret } from "@common/secret-manager";
import { getLogger } from "@obs/logger";
import {
getEmbeddingApiUrl,
getEmbeddingDimensions,
getEmbeddingModel,
getEmbeddingProvider,
getEmbeddingRequestTimeoutMs,
} from "../config.js";
export interface EmbeddingClient {
embed(text: string): Promise<number[]>;
embedBatch(texts: string[]): Promise<number[][]>;
readonly model: string;
readonly dimensions: number;
}
interface EmbeddingResponse {
data?: Array<{
embedding?: number[] | undefined;
index?: number | undefined;
}> | undefined;
model?: string | undefined;
}
abstract class HttpEmbeddingClient implements EmbeddingClient {
constructor(
readonly model: string,
readonly dimensions: number,
private readonly apiKey: string | undefined,
private readonly baseUrl: string,
) {}
async embed(text: string): Promise<number[]> {
const [embedding] = await this.embedBatch([text]);
if (!embedding) {
throw new Error("Embedding response did not include an embedding");
}
return embedding;
}
async embedBatch(texts: string[]): Promise<number[][]> {
const input = texts.map((text) => text.trim()).filter(Boolean);
if (input.length === 0) {
return [];
}
if (!this.apiKey) {
throw new Error(`${this.constructor.name} requires an embedding API key`);
}
const response = await fetch(`${this.baseUrl.replace(/\/+$/u, "")}/embeddings`, {
body: JSON.stringify({
dimensions: this.dimensions,
input,
model: this.model,
}),
headers: {
Authorization: `Bearer ${this.apiKey}`,
"Content-Type": "application/json",
},
method: "POST",
signal: AbortSignal.timeout(getEmbeddingRequestTimeoutMs()),
});
if (!response.ok) {
const body = (await response.text()).slice(0, 500);
throw new Error(`Embedding request failed (${response.status}): ${body}`);
}
const payload = (await response.json()) as EmbeddingResponse;
const rows = payload.data ?? [];
return rows
.sort((left, right) => (left.index ?? 0) - (right.index ?? 0))
.map((row) => row.embedding ?? []);
}
}
export class OpenRouterEmbeddingClient extends HttpEmbeddingClient {
constructor() {
super(
getEmbeddingModel("openrouter"),
getEmbeddingDimensions(),
getRawSecret("EMBEDDING_API_KEY") ?? getRawSecret("OPENROUTER_API_KEY"),
getEmbeddingApiUrl("openrouter"),
);
}
}
export class OpenAIEmbeddingClient extends HttpEmbeddingClient {
constructor() {
super(
getEmbeddingModel("openai").replace(/^openai\//u, ""),
getEmbeddingDimensions(),
getRawSecret("EMBEDDING_API_KEY") ?? getRawSecret("OPENAI_API_KEY"),
getEmbeddingApiUrl("openai"),
);
}
}
class FallbackEmbeddingClient implements EmbeddingClient {
readonly model: string;
readonly dimensions: number;
private readonly logger = getLogger();
constructor(
private readonly primary: EmbeddingClient,
private readonly fallback: EmbeddingClient,
) {
this.model = primary.model;
this.dimensions = primary.dimensions;
}
embed(text: string): Promise<number[]> {
return this.embedBatch([text]).then((embeddings) => {
const [embedding] = embeddings;
if (!embedding) {
throw new Error("Embedding response did not include an embedding");
}
return embedding;
});
}
async embedBatch(texts: string[]): Promise<number[][]> {
try {
return await this.primary.embedBatch(texts);
} catch (error) {
this.logger.warn(
{ err: error, provider: "openrouter" },
"[Memory] Primary embedding provider failed; falling back",
);
return this.fallback.embedBatch(texts);
}
}
}
export function createEmbeddingClient(): EmbeddingClient {
const provider = getEmbeddingProvider();
if (provider === "openai") {
return new OpenAIEmbeddingClient();
}
return new FallbackEmbeddingClient(
new OpenRouterEmbeddingClient(),
new OpenAIEmbeddingClient(),
);
}
-58
View File
@@ -1,58 +0,0 @@
import type { HandoffReason } from "@agent/types";
import type { SupervisorReviewResolution } from "@skills/supervisor/review-queue";
export interface UserEscalationRecord {
queuedAt: string;
reason: HandoffReason;
reviewId: string;
target: string;
resolution?: {
resolvedAt: string;
action: "dismiss" | "manual-reply" | "reroute";
note?: string | undefined;
} | undefined;
packetSummary: string;
sourceChain: string[];
}
export interface UserEscalationHistory {
userId: string;
chatroomId?: string | undefined;
escalations: UserEscalationRecord[];
totalEscalations: number;
lastEscalationAt: string;
topReasons: Array<{ reason: HandoffReason; count: number }>;
}
export interface EscalationHistoryService {
load(userId: string): Promise<UserEscalationHistory | null>;
recordEscalation(record: {
userId: string;
chatroomId?: string | undefined;
queuedAt: string;
reason: HandoffReason;
reviewId: string;
target: string;
packetSummary: string;
sourceChain: string[];
}): Promise<void>;
recordResolution(record: {
userId: string;
reviewId: string;
resolution: SupervisorReviewResolution;
}): Promise<void>;
}
export class NoopEscalationHistoryService implements EscalationHistoryService {
load(): Promise<UserEscalationHistory | null> {
return Promise.resolve(null);
}
recordEscalation(): Promise<void> {
return Promise.resolve();
}
recordResolution(): Promise<void> {
return Promise.resolve();
}
}
-32
View File
@@ -1,32 +0,0 @@
import { describe, expect, it } from "vitest";
import { evaluateDocument, evaluateUserFlags } from "./forgetting-policy.js";
import type { UserProfile } from "@skills/onboarding/state/user-memory";
describe("evaluateDocument", () => {
it("archives stale low-reference documents", () => {
const decision = evaluateDocument(
{ category: "rule", id: "doc-1", text: "old rule" },
{
now: new Date("2026-05-18T00:00:00.000Z"),
referenceCount30d: 0,
updatedAt: new Date("2025-12-01T00:00:00.000Z"),
},
);
expect(decision.action).toBe("archive");
expect(decision.reason).toBe("freshness-stale");
});
});
describe("evaluateUserFlags", () => {
it("keeps the newest unique flags within the limit", () => {
const profile = {
flaggedIssues: ["a", "b", "c"],
} as UserProfile;
expect(evaluateUserFlags(profile, 2)).toEqual({
archived: ["a"],
kept: ["b", "c"],
});
});
});
Loaded 100 of 211 files, more files were not shown because too many files have changed in this diff. Show more