Compare commits

...
4 Commits
Author SHA1 Message Date
dahyeon d05de52cd9 Improve onboarding verification and routing controls 2026-05-24 23:46:59 -07:00
dahyeon bc0f8e0773 Add secret coverage and NATS subject ACL
- KNOWN_SECRET_KEYS now covers LLM, embedding, and Phoenix API keys
- Logger redaction paths extended to cover the new credential keys
- nats-skill-acl maps each skill to allowed publish/subscribe subjects
- NATS publish runs ACL check with log-only/enforce/off modes
2026-05-19 01:28:54 -07:00
dahyeon d011a2829c Add tool permission sandbox and audit stream
- Tool interface declares required permissions via requires field
- scoped-context builds a least-privileged ToolContext per call
- ToolAuditLogger appends each tool invocation to a Garnet stream
- Operations CLI gains tool-audit:list and tool-audit:show commands
- All onboarding and moderation tools annotated with explicit requires
2026-05-19 01:28:54 -07:00
dahyeon 21df14cd00 Add cost-aware routing and LLM budget tracking
- BudgetTracker tracks LLM token cost separately from tool cost
- LlmClient auto-records cost into request-scoped budget
- Router selects tier from classify confidence, intent risk, escalation count
- Supervisor short-circuits via regex fast-path when rule confidence is high
- Onboarding skill loads user escalation history to inform routing
2026-05-19 01:28:54 -07:00
60 changed files with 4267 additions and 181 deletions

No files matched your search

+30 -3
View File
@@ -257,6 +257,18 @@ npm run review:list -- --reason=low-confidence --limit=10
Resolution history는 stream에 쌓이며, compatibility hash는 sunset 전까지 read fallback으로 유지된다. Sunset 조건과 절차는 [resolution-hash-sunset-procedure.md](resolution-hash-sunset-procedure.md)를 따른다. Resolution history는 stream에 쌓이며, compatibility hash는 sunset 전까지 read fallback으로 유지된다. Sunset 조건과 절차는 [resolution-hash-sunset-procedure.md](resolution-hash-sunset-procedure.md)를 따른다.
### 10.2 Tool Audit
Tool call은 성공과 실패 모두 `lolympus-agent:tool-calls` Stream에 남긴다. 원본 args는 저장하지 않고, 민감 키를 제거한 뒤 hash만 기록한다.
```bash
npm run tool-audit:list -- --limit=20
npm run tool-audit:list -- --skill=onboarding --since=24h
npm run tool-audit:show -- <callId>
```
Stream은 `AGENT_TOOL_AUDIT_MAX_LEN` 기준으로 trim된다. 기본값은 `100000`이다.
## 11. Calibration 흐름 ## 11. Calibration 흐름
Calibration은 사람 점수와 LLM judge 점수의 상관을 계산한다. Calibration은 사람 점수와 LLM judge 점수의 상관을 계산한다.
@@ -460,6 +472,13 @@ npm test
| `EVAL_TOURNAMENT_MIN_CASES` | 10 | proposal 최소 archive cases | | `EVAL_TOURNAMENT_MIN_CASES` | 10 | proposal 최소 archive cases |
| `PROMPT_VARIANT_ID` | unset | 운영 prompt variant 선택 | | `PROMPT_VARIANT_ID` | unset | 운영 prompt variant 선택 |
| `AGENT_ROUTER_WEIGHTS_PATH` | unset | learned router weights path | | `AGENT_ROUTER_WEIGHTS_PATH` | unset | learned router weights path |
| `AGENT_BUDGET_LLM_COST_ENABLED` | true | LLM token cost를 request budget에 누적 |
| `AGENT_COST_AWARE_TIER_ENABLED` | true | confidence/cost-aware tier selection |
| `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | supervisor regex fast-path 사용 |
| `SUPERVISOR_FAST_PATH_CONFIDENCE` | 0.85 | fast-path 최소 규칙 confidence |
| `AGENT_TOOL_SANDBOX_ENABLED` | false | tool context 최소 권한 강제 여부 |
| `AGENT_TOOL_AUDIT_MAX_LEN` | 100000 | tool audit Stream maxlen |
| `AGENT_NATS_ACL_MODE` | log-only | NATS subject ACL 모드 |
| `AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED` | true | supervisor LLM userMessage 사용 여부 | | `AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED` | true | supervisor LLM userMessage 사용 여부 |
| `PHOENIX_OTLP_ENDPOINT` | unset | Phoenix OTLP trace endpoint | | `PHOENIX_OTLP_ENDPOINT` | unset | Phoenix OTLP trace endpoint |
| `PHOENIX_LOG_PROMPTS` | false | LLM prompt/completion content trace 기록 | | `PHOENIX_LOG_PROMPTS` | false | LLM prompt/completion content trace 기록 |
@@ -519,7 +538,15 @@ ALERT_THRESHOLD_DRIFT_PCT=15
예를 들어 onboarding latency alert와 moderation latency alert는 서로 throttle하지 않는다. 예를 들어 onboarding latency alert와 moderation latency alert는 서로 throttle하지 않는다.
skill이 없는 cross-system alert는 기존처럼 글로벌 dedupe를 사용한다. skill이 없는 cross-system alert는 기존처럼 글로벌 dedupe를 사용한다.
### 16.2 Phoenix trace ### 16.2 Secret, Tool Sandbox, NATS ACL
Secret은 Infisical을 우선으로 로드한다. Infisical credential은 로컬 env 또는 IMDS에서 읽고, 새 secret env를 추가할 때는 `src/common/known-secret-keys.ts`에 함께 등록해야 로그 redaction 대상에 들어간다. LLM, embedding, Phoenix API key도 redaction 대상이다.
Tool은 `requires` 선언으로 필요한 context만 받는다. `AGENT_TOOL_SANDBOX_ENABLED=false`가 기본이라 운영 초기는 호환성을 유지한다. 1주 정도 audit을 본 뒤 false positive가 없으면 true로 전환한다.
NATS subject ACL은 `src/common/nats-skill-acl.ts`의 `DEFAULT_NATS_ACL`에서 관리한다. 기본 `AGENT_NATS_ACL_MODE=log-only`는 위반을 warn으로만 남기고 publish/request는 그대로 진행한다. 1개월 운영 후 false positive가 없으면 `enforce`로 전환한다.
### 16.3 Phoenix trace
LLM 호출, skill 실행, tool 실행 span은 OpenTelemetry trace로 남긴다. LLM 호출, skill 실행, tool 실행 span은 OpenTelemetry trace로 남긴다.
`PHOENIX_OTLP_ENDPOINT`가 없으면 기존 설정을 따른다. 개발 환경 기본값은 trace exporter 비활성화이고, `OTEL_EXPORTER=stdout`이면 콘솔 exporter를 쓴다. `PHOENIX_OTLP_ENDPOINT`가 없으면 기존 설정을 따른다. 개발 환경 기본값은 trace exporter 비활성화이고, `OTEL_EXPORTER=stdout`이면 콘솔 exporter를 쓴다.
@@ -544,7 +571,7 @@ Phoenix UI에서는 `service.name=lolympus-agent`, `gen_ai.system`, `gen_ai.requ
Prompt와 completion 본문은 기본적으로 기록하지 않는다. Prompt와 completion 본문은 기본적으로 기록하지 않는다.
디버깅 때문에 필요할 때만 `PHOENIX_LOG_PROMPTS=true`를 켠다. 이 경우 email, 전화번호, token/API key 형태의 값은 span 기록 전에 masking된다. 디버깅 때문에 필요할 때만 `PHOENIX_LOG_PROMPTS=true`를 켠다. 이 경우 email, 전화번호, token/API key 형태의 값은 span 기록 전에 masking된다.
### 16.3 Prometheus scrape ### 16.4 Prometheus scrape
Agent process는 `METRICS_PORT`에서 HTTP endpoint를 연다. Agent process는 `METRICS_PORT`에서 HTTP endpoint를 연다.
@@ -596,7 +623,7 @@ sum by (skill, stage) (rate(agent_validator_fail_total[5m]))
| eval drift | 15% 이하 | | eval drift | 15% 이하 |
| archive stalled | 3회 연속 실패 0건 | | archive stalled | 3회 연속 실패 0건 |
### 16.4 Slack webhook 설정 ### 16.5 Slack webhook 설정
Slack incoming webhook URL을 env로 설정한다. Slack incoming webhook URL을 env로 설정한다.
File diff suppressed because it is too large. Load diff
+21 -5
View File
@@ -16,19 +16,35 @@ if ($isMacOS -or $isLinux) {
$env:ENV_FILE = "$PSScriptRoot\env-vars\$env:NODE_ENV.env" $env:ENV_FILE = "$PSScriptRoot\env-vars\$env:NODE_ENV.env"
} }
# 환경 변수 파일 확인 $GLOBAL_ENV_FILE = Join-Path $HOME "source/global-env-vars.txt"
if (-Not (Test-Path $env:ENV_FILE)) {
Write-Error "The file '$env:ENV_FILE' does not exist." function Import-EnvFileIfPresent {
param(
[string]$Path,
[switch]$Required
)
if (-Not (Test-Path $Path)) {
if ($Required) {
Write-Error "The file '$Path' does not exist."
exit 1 exit 1
} }
return
}
# 'dahyeon-local.env' 파일에서 환경 변수 설정 Get-Content $Path | ForEach-Object {
Get-Content $env:ENV_FILE | ForEach-Object {
if ($_ -match '^\s*([^#]+?)\s*=\s*(.*)$') { if ($_ -match '^\s*([^#]+?)\s*=\s*(.*)$') {
$key, $value = $matches[1].Trim(), $matches[2].Trim().Trim("`"") $key, $value = $matches[1].Trim(), $matches[2].Trim().Trim("`"")
if (-not [System.Environment]::GetEnvironmentVariable($key)) {
[System.Environment]::SetEnvironmentVariable($key, $value) [System.Environment]::SetEnvironmentVariable($key, $value)
} }
} }
}
}
# 환경 변수 파일 확인
Import-EnvFileIfPresent -Path $GLOBAL_ENV_FILE
Import-EnvFileIfPresent -Path $env:ENV_FILE -Required
Get-ChildItem Env: | Where-Object { Get-ChildItem Env: | Where-Object {
$_.Name -eq "NODE_ENV" -or $_.Name -eq "NODE_ENV" -or
+2
View File
@@ -36,6 +36,8 @@
"review:show": "node --import tsx src/operations/cli.ts review:show", "review:show": "node --import tsx src/operations/cli.ts review:show",
"review:resolve": "node --import tsx src/operations/cli.ts review:resolve", "review:resolve": "node --import tsx src/operations/cli.ts review:resolve",
"review:stats": "node --import tsx src/operations/cli.ts review:stats", "review:stats": "node --import tsx src/operations/cli.ts review:stats",
"tool-audit:list": "node --import tsx src/operations/cli.ts tool-audit:list",
"tool-audit:show": "node --import tsx src/operations/cli.ts tool-audit:show",
"start:dev": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'dev'); \"./launch-server.ps1\"\"", "start:dev": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'dev'); \"./launch-server.ps1\"\"",
"start:test": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'test'); \"./launch-server.ps1\"\"", "start:test": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'test'); \"./launch-server.ps1\"\"",
"start:prod": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'prod'); \"./launch-server.ps1\"\"", "start:prod": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'prod'); \"./launch-server.ps1\"\"",
+25
View File
@@ -59,6 +59,30 @@ describe("BudgetTracker", () => {
expect(costBudget.assertBeforeHandoff("supervisor")).toBe("cost-budget"); expect(costBudget.assertBeforeHandoff("supervisor")).toBe("cost-budget");
}); });
it("tracks LLM and tool costs separately while enforcing the total budget", () => {
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 0.0055,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
for (let index = 0; index < 5; index += 1) {
budget.recordLlmCost(0.001);
}
budget.recordToolCall(0.0005);
expect(budget.snapshot()).toMatchObject({
estimatedCostUsd: 0.0055,
totalCostUsd: 0.0055,
totalLlmCostUsd: 0.005,
totalToolCostUsd: 0.0005,
toolCalls: 1,
});
expect(budget.assertBeforeHandoff("supervisor")).toBe("cost-budget");
});
it("can be disabled", () => { it("can be disabled", () => {
const budget = new BudgetTracker({ const budget = new BudgetTracker({
enabled: false, enabled: false,
@@ -71,6 +95,7 @@ describe("BudgetTracker", () => {
budget.recordVisit("supervisor"); budget.recordVisit("supervisor");
budget.recordHandoff("moderation"); budget.recordHandoff("moderation");
budget.recordToolCall(1); budget.recordToolCall(1);
budget.recordLlmCost(1);
expect(budget.assertBeforeHandoff("supervisor")).toBe("ok"); expect(budget.assertBeforeHandoff("supervisor")).toBe("ok");
}); });
+23 -4
View File
@@ -20,13 +20,17 @@ export interface BudgetSnapshot {
maxHandoffs: number; maxHandoffs: number;
maxSameAgentVisits: number; maxSameAgentVisits: number;
maxToolCalls: number; maxToolCalls: number;
totalCostUsd: number;
totalLlmCostUsd: number;
totalToolCostUsd: number;
toolCalls: number; toolCalls: number;
visits: Record<string, number>; visits: Record<string, number>;
} }
export class BudgetTracker { export class BudgetTracker {
private estimatedCostUsd = 0;
private handoffs = 0; private handoffs = 0;
private totalLlmCostUsd = 0;
private totalToolCostUsd = 0;
private toolCalls = 0; private toolCalls = 0;
private readonly visits = new Map<string, number>(); private readonly visits = new Map<string, number>();
@@ -53,7 +57,7 @@ export class BudgetTracker {
if (this.toolCalls >= this.limits.maxToolCalls) { if (this.toolCalls >= this.limits.maxToolCalls) {
return "tool-budget"; return "tool-budget";
} }
if (this.estimatedCostUsd >= this.limits.maxCostUsd) { if (this.totalCostUsd() >= this.limits.maxCostUsd) {
return "cost-budget"; return "cost-budget";
} }
return "ok"; return "ok";
@@ -66,7 +70,14 @@ export class BudgetTracker {
recordToolCall(estimatedCostUsd = 0): void { recordToolCall(estimatedCostUsd = 0): void {
this.toolCalls += 1; this.toolCalls += 1;
this.estimatedCostUsd += estimatedCostUsd; this.totalToolCostUsd += estimatedCostUsd;
}
recordLlmCost(costUsd: number): void {
if (!Number.isFinite(costUsd) || costUsd <= 0) {
return;
}
this.totalLlmCostUsd += costUsd;
} }
recordVisit(target: string): void { recordVisit(target: string): void {
@@ -74,15 +85,23 @@ export class BudgetTracker {
} }
snapshot(): BudgetSnapshot { snapshot(): BudgetSnapshot {
const totalCostUsd = this.totalCostUsd();
return { return {
estimatedCostUsd: Number(this.estimatedCostUsd.toFixed(6)), estimatedCostUsd: Number(totalCostUsd.toFixed(6)),
handoffs: this.handoffs, handoffs: this.handoffs,
maxCostUsd: this.limits.maxCostUsd, maxCostUsd: this.limits.maxCostUsd,
maxHandoffs: this.limits.maxHandoffs, maxHandoffs: this.limits.maxHandoffs,
maxSameAgentVisits: this.limits.maxSameAgentVisits, maxSameAgentVisits: this.limits.maxSameAgentVisits,
maxToolCalls: this.limits.maxToolCalls, maxToolCalls: this.limits.maxToolCalls,
totalCostUsd: Number(totalCostUsd.toFixed(6)),
totalLlmCostUsd: Number(this.totalLlmCostUsd.toFixed(6)),
totalToolCostUsd: Number(this.totalToolCostUsd.toFixed(6)),
toolCalls: this.toolCalls, toolCalls: this.toolCalls,
visits: Object.fromEntries(this.visits.entries()), visits: Object.fromEntries(this.visits.entries()),
}; };
} }
private totalCostUsd(): number {
return this.totalLlmCostUsd + this.totalToolCostUsd;
}
} }
+1 -1
View File
@@ -90,7 +90,7 @@ export async function dispatch(
return runWithContext( return runWithContext(
{ {
...contextBase, ...contextBase,
budget: budget.snapshot(), budget,
handoffChain: chain, handoffChain: chain,
handoffDepth: depth, handoffDepth: depth,
}, },
+11
View File
@@ -1,4 +1,5 @@
import fs from "node:fs"; import fs from "node:fs";
import os from "node:os";
import path from "node:path"; import path from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import * as dotenv from "dotenv"; import * as dotenv from "dotenv";
@@ -34,6 +35,11 @@ function resolveEnvFile(mode: NodeEnvMode): string | null {
return null; return null;
} }
function resolveGlobalEnvFile(): string | null {
const candidate = path.resolve(os.homedir(), "source", "global-env-vars.txt");
return fs.existsSync(candidate) ? candidate : null;
}
function loadDotenvFile(filePath: string, label: string): void { function loadDotenvFile(filePath: string, label: string): void {
const result = dotenv.config({ path: filePath, override: false }); const result = dotenv.config({ path: filePath, override: false });
if (result.error) { if (result.error) {
@@ -48,6 +54,11 @@ export function loadEnvironmentFromNodeEnv(): void {
const mode = normalizeMode(process.env["NODE_ENV"]); const mode = normalizeMode(process.env["NODE_ENV"]);
process.env["NODE_ENV"] = mode; process.env["NODE_ENV"] = mode;
const globalEnvFile = resolveGlobalEnvFile();
if (globalEnvFile) {
loadDotenvFile(globalEnvFile, "global-env-vars.txt");
}
const envFile = resolveEnvFile(mode); const envFile = resolveEnvFile(mode);
if (!envFile) { if (!envFile) {
logger.warn(`[Env] env-vars/${mode}.env not found. Skipping dotenv load.`); logger.warn(`[Env] env-vars/${mode}.env not found. Skipping dotenv load.`);
+3 -2
View File
@@ -1,9 +1,10 @@
import { GarnetClient } from "./garnet.js"; import type { GarnetClient as GarnetClientType } from "./garnet.js";
export async function connectGarnetForCli(): Promise<{ export async function connectGarnetForCli(): Promise<{
close: () => Promise<void>; close: () => Promise<void>;
garnetClient: GarnetClient; garnetClient: GarnetClientType;
}> { }> {
const { GarnetClient } = await import("./garnet.js");
const garnetClient = new GarnetClient(); const garnetClient = new GarnetClient();
await garnetClient.connect(); await garnetClient.connect();
return { return {
+5
View File
@@ -9,6 +9,11 @@ export const KNOWN_SECRET_KEYS = [
"GARNET_PASSWORD", "GARNET_PASSWORD",
"MONGODB_CONNECTION_STRING", "MONGODB_CONNECTION_STRING",
"MONGODB_PROJECT_ID", "MONGODB_PROJECT_ID",
"OPENROUTER_API_KEY",
"AGENT_LLM_API_KEY",
"OPENAI_API_KEY",
"EMBEDDING_API_KEY",
"PHOENIX_API_KEY",
"SLACK_ALERT_WEBHOOK_URL", "SLACK_ALERT_WEBHOOK_URL",
] as const; ] as const;
+41
View File
@@ -0,0 +1,41 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { runWithContext } from "../observability/context.js";
import { NatsClient } from "./nats.js";
import { AGENT_ALERT_SUBJECT, DISCORD_MEMBER_LOOKUP_SUBJECT } from "./nats-subjects.js";
import { checkPublishAllowed } from "./nats-skill-acl.js";
describe("NATS skill ACL", () => {
afterEach(() => {
vi.unstubAllEnvs();
});
it("allows declared publish subjects", () => {
expect(checkPublishAllowed("onboarding", DISCORD_MEMBER_LOOKUP_SUBJECT)).toBe(
true,
);
expect(checkPublishAllowed("supervisor", AGENT_ALERT_SUBJECT)).toBe(true);
expect(checkPublishAllowed("onboarding", AGENT_ALERT_SUBJECT)).toBe(false);
});
it("enforces denied publish subjects when enabled", async () => {
vi.stubEnv("AGENT_NATS_ACL_MODE", "enforce");
const client = new NatsClient();
await expect(
runWithContext(
{
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() => client.publish(AGENT_ALERT_SUBJECT, { ok: true }),
),
).rejects.toThrow(
`NATS publish denied for skill onboarding on subject ${AGENT_ALERT_SUBJECT}`,
);
});
});
+54
View File
@@ -0,0 +1,54 @@
import {
AGENT_ALERT_SUBJECT,
DISCORD_MEMBER_LOOKUP_SUBJECT,
} from "./nats-subjects.js";
export interface SkillNatsAcl {
skill: string;
allowedPublishSubjects: string[];
allowedSubscribeSubjects: string[];
}
export const DEFAULT_NATS_ACL: SkillNatsAcl[] = [
{
skill: "onboarding",
allowedPublishSubjects: [DISCORD_MEMBER_LOOKUP_SUBJECT],
allowedSubscribeSubjects: [],
},
{
skill: "moderation",
allowedPublishSubjects: [],
allowedSubscribeSubjects: [],
},
{
skill: "supervisor",
allowedPublishSubjects: [AGENT_ALERT_SUBJECT],
allowedSubscribeSubjects: [],
},
];
function wildcardToRegExp(pattern: string): RegExp {
const escaped = pattern
.split("*")
.map((part) => part.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"))
.join(".*");
return new RegExp(`^${escaped}$`, "u");
}
function matchesAny(subject: string, patterns: readonly string[]): boolean {
return patterns.some((pattern) => wildcardToRegExp(pattern).test(subject));
}
function getAcl(skill: string): SkillNatsAcl | undefined {
return DEFAULT_NATS_ACL.find((entry) => entry.skill === skill);
}
export function checkPublishAllowed(skill: string, subject: string): boolean {
const acl = getAcl(skill);
return Boolean(acl && matchesAny(subject, acl.allowedPublishSubjects));
}
export function checkSubscribeAllowed(skill: string, subject: string): boolean {
const acl = getAcl(skill);
return Boolean(acl && matchesAny(subject, acl.allowedSubscribeSubjects));
}
+21
View File
@@ -34,6 +34,12 @@ export type DiscordMemberMatchSource =
| "username" | "username"
| "joined-after"; | "joined-after";
export type DiscordResidentApplicationMessageMatchSource =
| "author-id"
| "author-name"
| "content"
| "joined-after";
export interface DiscordMemberCandidate { export interface DiscordMemberCandidate {
memberId: string; memberId: string;
joinedAt: string; joinedAt: string;
@@ -43,8 +49,20 @@ export interface DiscordMemberCandidate {
username: string; username: string;
} }
export interface DiscordResidentApplicationMessageCandidate {
authorDisplayName?: string | undefined;
authorGuildNickname?: string | undefined;
authorId: string;
channelId: string;
content: string;
createdAt: string;
matchedBy?: DiscordResidentApplicationMessageMatchSource | undefined;
messageId: string;
}
export interface DiscordMemberLookupRequest { export interface DiscordMemberLookupRequest {
chatroomId: string; chatroomId: string;
includeResidentApplicationMessages?: boolean | undefined;
joinedAfter: string; joinedAfter: string;
searchName?: string | undefined; searchName?: string | undefined;
userId: string; userId: string;
@@ -55,6 +73,9 @@ export type DiscordMemberLookupResponse =
status: "ok"; status: "ok";
joinedAfterMatches: DiscordMemberCandidate[]; joinedAfterMatches: DiscordMemberCandidate[];
nicknameMatches: DiscordMemberCandidate[]; nicknameMatches: DiscordMemberCandidate[];
residentApplicationMessageMatches?:
| DiscordResidentApplicationMessageCandidate[]
| undefined;
} }
| { | {
status: "error"; status: "error";
+39
View File
@@ -6,7 +6,13 @@ import {
type Subscription, type Subscription,
} from "nats"; } from "nats";
import { getSecret } from "@common/secret-manager"; import { getSecret } from "@common/secret-manager";
import { getNatsAclMode } from "../config.js";
import { getContext } from "../observability/context.js";
import { getLogger } from "@obs/logger"; import { getLogger } from "@obs/logger";
import {
checkPublishAllowed,
checkSubscribeAllowed,
} from "./nats-skill-acl.js";
export interface NatsMessageContext<T = unknown> { export interface NatsMessageContext<T = unknown> {
data: T; data: T;
@@ -81,6 +87,7 @@ export class NatsClient {
subject: string, subject: string,
handler: (msg: Msg) => void | Promise<void>, handler: (msg: Msg) => void | Promise<void>,
): Promise<Subscription> { ): Promise<Subscription> {
this.checkSubjectAcl("subscribe", subject);
const subscription = this.getConnection().subscribe(subject); const subscription = this.getConnection().subscribe(subject);
void (async () => { void (async () => {
@@ -101,6 +108,7 @@ export class NatsClient {
} }
async publish<T = unknown>(subject: string, data: T): Promise<void> { async publish<T = unknown>(subject: string, data: T): Promise<void> {
this.checkSubjectAcl("publish", subject);
const client = this.getConnection(); const client = this.getConnection();
client.publish(subject, this.stringCodec.encode(JSON.stringify(data))); client.publish(subject, this.stringCodec.encode(JSON.stringify(data)));
await client.flush(); await client.flush();
@@ -111,6 +119,7 @@ export class NatsClient {
data: TRequest, data: TRequest,
options?: { timeoutMs?: number | undefined }, options?: { timeoutMs?: number | undefined },
): Promise<TResponse> { ): Promise<TResponse> {
this.checkSubjectAcl("publish", subject);
const requestOptions = const requestOptions =
options?.timeoutMs !== undefined options?.timeoutMs !== undefined
? { ? {
@@ -126,6 +135,36 @@ export class NatsClient {
return this.decodeMessage<TResponse>(response); return this.decodeMessage<TResponse>(response);
} }
private checkSubjectAcl(
operation: "publish" | "subscribe",
subject: string,
): void {
const mode = getNatsAclMode();
if (mode === "off") {
return;
}
const skill = getContext()?.skill;
if (!skill) {
return;
}
const allowed =
operation === "publish"
? checkPublishAllowed(skill, subject)
: checkSubscribeAllowed(skill, subject);
if (allowed) {
return;
}
const payload = { mode, operation, skill, subject };
if (mode === "enforce") {
this.logger.error(payload, "[NATS ACL] Subject denied");
throw new Error(
`NATS ${operation} denied for skill ${skill} on subject ${subject}`,
);
}
this.logger.warn(payload, "[NATS ACL] Subject would be denied");
}
respond(msg: Msg, data: unknown): Promise<void> { respond(msg: Msg, data: unknown): Promise<void> {
if (!msg.reply) { if (!msg.reply) {
this.logger.warn( this.logger.warn(
+118
View File
@@ -5,10 +5,20 @@ import {
getAlertThresholdEvalScoreDropPct, getAlertThresholdEvalScoreDropPct,
getAlertThresholdLlmErrorRate, getAlertThresholdLlmErrorRate,
getAlertThresholdP95LatencyMs, getAlertThresholdP95LatencyMs,
getAgentBudgetLlmCostEnabled,
getAgentCostAwareTierEnabled,
getAgentSupervisorFastPathEnabled,
getAgentToolSandboxEnabled,
getEmbeddingModel, getEmbeddingModel,
getEmbeddingProvider, getEmbeddingProvider,
getEvalCaseDeadlineMs,
getEvalLlmProvider,
getEvalRuntimeLlmEnabled,
getLongTermSummaryEnabled, getLongTermSummaryEnabled,
getLlmModelRoundRobinEnabled,
getRuntimeLlmModelsForProvider,
getMemoryEvictionEnabled, getMemoryEvictionEnabled,
getNatsAclMode,
getPhoenixLogPrompts, getPhoenixLogPrompts,
getPhoenixOtlpEndpoint, getPhoenixOtlpEndpoint,
getRagHybridEnabled, getRagHybridEnabled,
@@ -17,7 +27,9 @@ import {
getRagEnabled, getRagEnabled,
getSlackAlertWebhookUrl, getSlackAlertWebhookUrl,
getSlackWebhookTimeoutMs, getSlackWebhookTimeoutMs,
getSupervisorFastPathConfidence,
getSupervisorLlmResponseEnabled, getSupervisorLlmResponseEnabled,
getToolAuditMaxLen,
} from "./config.js"; } from "./config.js";
const ENV_KEYS = [ const ENV_KEYS = [
@@ -35,6 +47,13 @@ const ENV_KEYS = [
"AGENT_RAG_RERANKER_TOP_K", "AGENT_RAG_RERANKER_TOP_K",
"AGENT_LONG_TERM_SUMMARY_ENABLED", "AGENT_LONG_TERM_SUMMARY_ENABLED",
"AGENT_MEMORY_EVICTION_ENABLED", "AGENT_MEMORY_EVICTION_ENABLED",
"AGENT_BUDGET_LLM_COST_ENABLED",
"AGENT_COST_AWARE_TIER_ENABLED",
"AGENT_SUPERVISOR_FAST_PATH_ENABLED",
"SUPERVISOR_FAST_PATH_CONFIDENCE",
"AGENT_TOOL_SANDBOX_ENABLED",
"AGENT_TOOL_AUDIT_MAX_LEN",
"AGENT_NATS_ACL_MODE",
"EMBEDDING_MODEL", "EMBEDDING_MODEL",
"EMBEDDING_PROVIDER", "EMBEDDING_PROVIDER",
"PHOENIX_LOG_PROMPTS", "PHOENIX_LOG_PROMPTS",
@@ -42,6 +61,12 @@ const ENV_KEYS = [
"SLACK_ALERT_WEBHOOK_URL", "SLACK_ALERT_WEBHOOK_URL",
"SLACK_WEBHOOK_TIMEOUT_MS", "SLACK_WEBHOOK_TIMEOUT_MS",
"AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED", "AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED",
"CLIPROXY_TIER1_PLANNER_MODEL",
"EVAL_CASE_DEADLINE_MS",
"EVAL_LLM_PROVIDER",
"EVAL_RUNTIME_LLM_ENABLED",
"LLM_MODEL_ROUND_ROBIN_ENABLED",
"OPENROUTER_TIER1_RESPONDER_MODEL",
] as const; ] as const;
describe("config", () => { describe("config", () => {
@@ -129,4 +154,97 @@ describe("config", () => {
expect(getSupervisorLlmResponseEnabled()).toBe(false); expect(getSupervisorLlmResponseEnabled()).toBe(false);
}); });
it("reads operational excellence guardrail configuration", () => {
expect(getAgentBudgetLlmCostEnabled()).toBe(true);
expect(getAgentCostAwareTierEnabled()).toBe(true);
expect(getAgentSupervisorFastPathEnabled()).toBe(true);
expect(getSupervisorFastPathConfidence()).toBe(0.85);
expect(getAgentToolSandboxEnabled()).toBe(false);
expect(getToolAuditMaxLen()).toBe(100_000);
expect(getNatsAclMode()).toBe("log-only");
vi.stubEnv("AGENT_BUDGET_LLM_COST_ENABLED", "false");
vi.stubEnv("AGENT_COST_AWARE_TIER_ENABLED", "false");
vi.stubEnv("AGENT_SUPERVISOR_FAST_PATH_ENABLED", "false");
vi.stubEnv("SUPERVISOR_FAST_PATH_CONFIDENCE", "0.9");
vi.stubEnv("AGENT_TOOL_SANDBOX_ENABLED", "true");
vi.stubEnv("AGENT_TOOL_AUDIT_MAX_LEN", "42");
vi.stubEnv("AGENT_NATS_ACL_MODE", "enforce");
expect(getAgentBudgetLlmCostEnabled()).toBe(false);
expect(getAgentCostAwareTierEnabled()).toBe(false);
expect(getAgentSupervisorFastPathEnabled()).toBe(false);
expect(getSupervisorFastPathConfidence()).toBe(0.9);
expect(getAgentToolSandboxEnabled()).toBe(true);
expect(getToolAuditMaxLen()).toBe(42);
expect(getNatsAclMode()).toBe("enforce");
});
it("reads eval case deadline configuration", () => {
expect(getEvalCaseDeadlineMs()).toBe(4_500);
vi.stubEnv("EVAL_CASE_DEADLINE_MS", "45000");
expect(getEvalCaseDeadlineMs()).toBe(45_000);
});
it("enables eval runtime LLM by default with env override", () => {
expect(getEvalRuntimeLlmEnabled()).toBe(true);
vi.stubEnv("EVAL_RUNTIME_LLM_ENABLED", "false");
expect(getEvalRuntimeLlmEnabled()).toBe(false);
});
it("pins eval LLM provider to cliproxy by default", () => {
expect(getEvalLlmProvider()).toBe("cliproxy");
vi.stubEnv("EVAL_LLM_PROVIDER", "all");
expect(getEvalLlmProvider()).toBe("all");
vi.stubEnv("EVAL_LLM_PROVIDER", "openrouter");
expect(getEvalLlmProvider()).toBe("openrouter");
vi.stubEnv("EVAL_LLM_PROVIDER", "invalid");
expect(getEvalLlmProvider()).toBe("cliproxy");
});
it("reads provider-scoped runtime LLM models", () => {
vi.stubEnv("CLIPROXY_TIER1_PLANNER_MODEL", "cliproxy-planner");
vi.stubEnv(
"OPENROUTER_TIER1_RESPONDER_MODEL",
"openrouter-fast, openrouter-backup",
);
expect(
getRuntimeLlmModelsForProvider(
"cliproxy",
{ role: "planner", tier: 1 },
"local-qwen",
),
).toEqual(["cliproxy-planner"]);
expect(
getRuntimeLlmModelsForProvider(
"openrouter",
{ role: "responder", tier: 1 },
"local-qwen",
),
).toEqual(["openrouter-fast", "openrouter-backup"]);
expect(
getRuntimeLlmModelsForProvider(
"llama",
{ role: "responder", tier: 1 },
"local-qwen",
),
).toEqual(["local-qwen"]);
});
it("keeps model round-robin disabled by default", () => {
expect(getLlmModelRoundRobinEnabled()).toBe(false);
vi.stubEnv("LLM_MODEL_ROUND_ROBIN_ENABLED", "true");
expect(getLlmModelRoundRobinEnabled()).toBe(true);
});
}); });
+162 -8
View File
@@ -77,6 +77,23 @@ function getBoolean(envKey: string, fallback: boolean): boolean {
return fallback; return fallback;
} }
function getStringListFromEnv(...envKeys: string[]): string[] | undefined {
for (const envKey of envKeys) {
const raw = process.env[envKey]?.trim();
if (!raw) continue;
const values = raw
.split(/[\n,]/u)
.map((value) => value.trim())
.filter((value) => value.length > 0);
if (values.length > 0) {
return values;
}
}
return undefined;
}
export function getNodeEnv(): "dev" | "test" | "prod" { export function getNodeEnv(): "dev" | "test" | "prod" {
const value = process.env["NODE_ENV"]?.trim(); const value = process.env["NODE_ENV"]?.trim();
return value === "test" || value === "prod" ? value : "dev"; return value === "test" || value === "prod" ? value : "dev";
@@ -231,6 +248,38 @@ export function getAgentBudgetMaxCostUsd(): number {
return getPositiveNumber("AGENT_BUDGET_MAX_COST_USD", 0.5); return getPositiveNumber("AGENT_BUDGET_MAX_COST_USD", 0.5);
} }
export function getAgentBudgetLlmCostEnabled(): boolean {
return getBoolean("AGENT_BUDGET_LLM_COST_ENABLED", true);
}
export function getAgentCostAwareTierEnabled(): boolean {
return getBoolean("AGENT_COST_AWARE_TIER_ENABLED", true);
}
export function getAgentSupervisorFastPathEnabled(): boolean {
return getBoolean("AGENT_SUPERVISOR_FAST_PATH_ENABLED", true);
}
export function getSupervisorFastPathConfidence(): number {
return getRatioNumber("SUPERVISOR_FAST_PATH_CONFIDENCE", 0.85);
}
export function getAgentToolSandboxEnabled(): boolean {
return getBoolean("AGENT_TOOL_SANDBOX_ENABLED", false);
}
export function getToolAuditMaxLen(): number {
return getPositiveNumber("AGENT_TOOL_AUDIT_MAX_LEN", 100_000);
}
export function getNatsAclMode(): "off" | "log-only" | "enforce" {
const raw = process.env["AGENT_NATS_ACL_MODE"]?.trim().toLowerCase();
if (raw === "off" || raw === "log-only" || raw === "enforce") {
return raw;
}
return "log-only";
}
export function getAgentRuntimeVerifierEnabled(): boolean { export function getAgentRuntimeVerifierEnabled(): boolean {
return getBoolean("AGENT_RUNTIME_VERIFIER_ENABLED", false); return getBoolean("AGENT_RUNTIME_VERIFIER_ENABLED", false);
} }
@@ -358,6 +407,23 @@ export function getSupervisorLlmResponseEnabled(): boolean {
return getBoolean("AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED", true); return getBoolean("AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED", true);
} }
export type EvalLlmProvider = "all" | "cliproxy" | "llama" | "openrouter";
const EVAL_LLM_PROVIDERS = new Set<EvalLlmProvider>([
"all",
"cliproxy",
"llama",
"openrouter",
]);
export function getEvalLlmProvider(): EvalLlmProvider {
const raw = process.env["EVAL_LLM_PROVIDER"]?.trim().toLowerCase();
if (raw && EVAL_LLM_PROVIDERS.has(raw as EvalLlmProvider)) {
return raw as EvalLlmProvider;
}
return "cliproxy";
}
export function getEvalCalibrationMinCorr(): number { export function getEvalCalibrationMinCorr(): number {
return getRatioNumber("EVAL_CALIBRATION_MIN_CORR", 0.5); return getRatioNumber("EVAL_CALIBRATION_MIN_CORR", 0.5);
} }
@@ -483,6 +549,10 @@ export function getLlmCircuitCooldownMs(): number {
return getPositiveNumber("LLM_CIRCUIT_COOLDOWN_MS", 60_000); return getPositiveNumber("LLM_CIRCUIT_COOLDOWN_MS", 60_000);
} }
export function getLlmModelRoundRobinEnabled(): boolean {
return getBoolean("LLM_MODEL_ROUND_ROBIN_ENABLED", false);
}
export function getClassifierMaxTokens(): number { export function getClassifierMaxTokens(): number {
return getPositiveNumber("CLASSIFIER_MAX_TOKENS", 16); return getPositiveNumber("CLASSIFIER_MAX_TOKENS", 16);
} }
@@ -519,6 +589,14 @@ export function getEvalPrmMinLabels(): number {
return getPositiveNumber("EVAL_PRM_MIN_LABELS", 500); return getPositiveNumber("EVAL_PRM_MIN_LABELS", 500);
} }
export function getEvalCaseDeadlineMs(): number {
return getPositiveNumber("EVAL_CASE_DEADLINE_MS", 4_500);
}
export function getEvalRuntimeLlmEnabled(): boolean {
return getBoolean("EVAL_RUNTIME_LLM_ENABLED", true);
}
export function getEvalAttributionTickMs(): number { export function getEvalAttributionTickMs(): number {
return getPositiveNumber("EVAL_ATTRIBUTION_TICK_MS", 6 * 60 * 60 * 1000); return getPositiveNumber("EVAL_ATTRIBUTION_TICK_MS", 6 * 60 * 60 * 1000);
} }
@@ -574,15 +652,91 @@ export function getTier2CriticModel(): string {
} }
export function getOpenRouterModels(): string[] { export function getOpenRouterModels(): string[] {
const raw = process.env["OPENROUTER_MODEL"]?.trim(); return (
if (!raw) { getStringListFromEnv("OPENROUTER_MODEL", "OPENROUTER_MODELS") ?? [
return ["openai/gpt-5.3-codex-spark"]; "openai/gpt-5.3-codex-spark",
]
);
} }
const models = raw
.split(",") type RuntimeLlmProvider = "cliproxy" | "llama" | "openrouter";
.map((value) => value.trim()) type RuntimeLlmRole = "critic" | "planner" | "responder" | undefined;
.filter((value) => value.length > 0); type RuntimeLlmTier = 0 | 1 | 2 | undefined;
return models.length > 0 ? models : ["openai/gpt-5.3-codex-spark"];
function getRuntimeRouteSuffix(args: {
role?: RuntimeLlmRole;
tier?: RuntimeLlmTier;
}): string {
if (args.role === "planner") {
return "TIER1_PLANNER";
}
if (args.role === "critic") {
return "TIER2_CRITIC";
}
if (args.tier === 0) {
return "TIER0";
}
return "TIER1_RESPONDER";
}
function getProviderPrefix(provider: RuntimeLlmProvider): string {
return provider.toUpperCase();
}
function getProviderDefaultModels(
provider: RuntimeLlmProvider,
llamaFallbackModel: string,
): string[] {
if (provider === "cliproxy") {
return (
getStringListFromEnv(
"CLIPROXY_MODEL",
"CLIPROXY_MODELS",
"CLI_PROXY_LLM_MODEL",
) ?? ["gpt-5.3-codex-spark"]
);
}
if (provider === "openrouter") {
return getOpenRouterModels();
}
return (
getStringListFromEnv("LLAMA_MODEL", "LLAMA_MODELS", "AGENT_LLM_MODEL") ?? [
llamaFallbackModel,
]
);
}
export function getRuntimeLlmModelsForProvider(
provider: RuntimeLlmProvider,
args: {
role?: RuntimeLlmRole;
tier?: RuntimeLlmTier;
},
llamaFallbackModel: string,
): string[] {
const suffix = getRuntimeRouteSuffix(args);
const providerPrefix = getProviderPrefix(provider);
const providerScopedModels = getStringListFromEnv(
`${providerPrefix}_${suffix}_MODEL`,
`${providerPrefix}_${suffix}_MODELS`,
);
if (providerScopedModels) {
return providerScopedModels;
}
if (provider === "cliproxy") {
const legacyModels = getStringListFromEnv(
`${suffix}_MODEL`,
`${suffix}_MODELS`,
);
if (legacyModels) {
return legacyModels;
}
}
return getProviderDefaultModels(provider, llamaFallbackModel);
} }
export function getLlmMaxTokens(): number { export function getLlmMaxTokens(): number {
@@ -138,3 +138,63 @@
{"id":"case_acd612e0","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778740258770-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_7169cd5a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_81f2ad8f","nickname":"<nick:ba29bb75>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}} {"id":"case_acd612e0","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778740258770-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_7169cd5a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_81f2ad8f","nickname":"<nick:ba29bb75>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_6cd7106c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778718965990-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse={\"reply\":\"안녕하세요!\"}","input":{"request":{"requestId":"req_1477394f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_24cff593","nickname":"<nick:d455d0e4>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}} {"id":"case_6cd7106c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778718965990-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse={\"reply\":\"안녕하세요!\"}","input":{"request":{"requestId":"req_1477394f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_24cff593","nickname":"<nick:d455d0e4>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_de51735e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778674030965-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_32cc22d6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_cd95eff0","nickname":"<nick:7feae321>","trigger":"user-message","userMessage":"안녕하세오"}},"expected":{}} {"id":"case_de51735e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778674030965-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_32cc22d6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_cd95eff0","nickname":"<nick:7feae321>","trigger":"user-message","userMessage":"안녕하세오"}},"expected":{}}
{"id":"case_4f45294b","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778756106669-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 여기 채팅방에 올리면 돼요. \n채팅방 오른쪽 상단 메뉴의 '커버 보기'로 들어가 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷으로 보내면 됩니다.","input":{"request":{"requestId":"req_e6e21e71","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"하트인증\n여따 올리면 되요?"}},"expected":{}}
{"id":"case_e3ff18c6","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778756063255-0; llmOverall=2.75; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 확인했어요. 천천히 진행해 주세요.","input":{"request":{"requestId":"req_7b05c872","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"네"}},"expected":{}}
{"id":"case_1b906220","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778763820454-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임 순서는 이름/출생연도/성별/티어로 맞게 돼 있고, 안 되는 부분은 마지막 티어 코드뿐이에요. \n현재의 '플'은 허용 티어 코드가 아니므로 GM, I, B, S, G, P, E, D, M, C, U 중 한 글자(예: D)로 바꿔주세요.","input":{"request":{"requestId":"req_a8ba4ffe","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_e3e5b006","nickname":"<nick:405abf95>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_4ff31f85","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778756157817-0; llmOverall=n/a; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 확인했어요. 천천히 진행해 주세요.","input":{"request":{"requestId":"req_c78edc0e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"오"}},"expected":{}}
{"id":"case_0fd45691","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778874307141-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=닉네임이 안 되는 이유는 형식이 맞지 않아서예요. \n현재 \"위례\"는 이름만 있고, 연도/성별/티어가 없어 형식이 깨졌습니다. \n카카오톡 닉네임은 [이름]/[출생연도 두 자리]/[남 또는 여]/[티어]로 바꿔주세요.","input":{"request":{"requestId":"req_b6f69c24","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:16fb6789>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_408c317c","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778874400236-0; llmOverall=2.25; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=채팅방 오른쪽 상단 메뉴 버튼을 눌러 “커버 보기”로 들어가면 하트 인증 화면이 보여요.","input":{"request":{"requestId":"req_1db51665","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"하트인증은 어디다 하나요?"}},"expected":{}}
{"id":"case_cf22a9d8","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778874533535-0; llmOverall=2.5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=좋아요, 천천히 진행해 주세요.","input":{"request":{"requestId":"req_60f5e74a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"넵 감사합니다"}},"expected":{}}
{"id":"case_fd404eb4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778874508177-0; llmOverall=5; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=네, 가능합니다. 카카오톡 1단계가 완료된 상태라면 디스코드는 저녁에도 접속해 진행하시면 돼요.","input":{"request":{"requestId":"req_d814bae6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"디스코드는 저녁에 해도될까요?"}},"expected":{}}
{"id":"case_d42b8624","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778874344992-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_47d100d2","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"안녕하세여"}},"expected":{}}
{"id":"case_bf09ad3a","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778954734843-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=현재 남은 절차는 카카오톡 닉네임 변경과 방 하트 인증입니다.","input":{"request":{"requestId":"req_010646c0","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_a2e8fc0b","nickname":"<nick:a9bad7ab>","trigger":"user-message","userMessage":"혹시 이시간엔 내전 안돌아가죠?"}},"expected":{}}
{"id":"case_275402ce","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950244576-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 건 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정, 디스코드 주민신청 채널에 메시지 남기기예요.","input":{"request":{"requestId":"req_bc7c8aae","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"5대5 하려고 하는데 가능 한지 해서용"}},"expected":{}}
{"id":"case_0e56499c","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950227521-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 절차는 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정, 디스코드 주민신청 채널에 메시지 남기기예요.","input":{"request":{"requestId":"req_593ffaff","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"아 지금 들어와 있는 관리자 분이용"}},"expected":{}}
{"id":"case_c8398f09","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778949416363-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=웅/97/남/M님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_b0f0648f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"혹시 지 외전 문의 드리려 하는데"}},"expected":{}}
{"id":"case_0a460498","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950169079-0; llmOverall=2; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=관리분(관리자)은 있어요.","input":{"request":{"requestId":"req_6bffed4f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"관리분 있으신가요?"}},"expected":{}}
{"id":"case_002ac417","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778931066262-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임 \"강서/93/남\"은 티어가 빠져서 형식이 안 맞아요. \n강서/93/남은 티어가 빠져서 형식이 안 맞아요. 마지막에 티어 한 글자만 추가하면 됩니다. \n예: 강서/93/남/D","input":{"request":{"requestId":"req_ed284edb","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0d05dfd2","nickname":"<nick:1ec0deae>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_3e163e2e","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778949203127-0; llmOverall=2.75; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임은 형식이 안 맞아요. \n카톡 닉네임은 구분자로 '/'를 써야 하고 4개 항목이 필요해요. 현재는 웅 97 남 M라 안 되고, 현재 닉네임은 웅/97/남/M 이렇게 고치면 됩니다.","input":{"request":{"requestId":"req_ab630198","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:6334c82e>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_1db9c911","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778958194603-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_f3d19239","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3e4be23b","nickname":"<nick:b09afa6a>","trigger":"user-message","userMessage":"안녕하세여"}},"expected":{}}
{"id":"case_ee7ae44a","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778955694984-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_fea5fd76","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3592a95a","nickname":"<nick:437c8fc3>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_d6bd160f","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778950199447-0; llmOverall=2.25; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=운영진께, 특히 관리자님께 문의하시면 돼요.","input":{"request":{"requestId":"req_5fbe8788","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"누구한테 여쭤봐야 할까요"}},"expected":{}}
{"id":"case_36b3ac9c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778949423115-0; llmOverall=n/a; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=웅/97/남/M님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_35e659fa","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"어디에 문의 해야 할까용"}},"expected":{}}
{"id":"case_f7a5d661","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019400869-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_4861ee12","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"user-message","userMessage":"듀랭하실분들 있을까요"}},"expected":{}}
{"id":"case_74eac586","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019394267-0; llmOverall=1.25; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_743e0456","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"user-message","userMessage":"혹시"}},"expected":{}}
{"id":"case_9e9195bb","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019388776-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=@벽 넘어가는 탈론 00/남/E3님 닉네임에서 수정할 부분이 있어요!\n나이를 두 자리 숫자로 입력해주세요 (예: 25)\n성별(남/여)과 티어 코드를 '/'로 구분해서 입력해주세요","input":{"request":{"requestId":"req_568d875b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_8acd1f91","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779011411116-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임에서 안 된 부분은 마지막 티어예요. \nG2는 허용 티어가 아니라서, 뒤의 G2를 GM, I, B, S, G, P, E, D, M, C, U 중 하나(예: G)로만 바꾸면 됩니다.","input":{"request":{"requestId":"req_9860e773","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:666d8731>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_d6824fdb","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779023514843-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_4d45f87c","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bc77847e","nickname":"<nick:9ee38f43>","trigger":"user-message","userMessage":"안녕하세요 공지읽고오겠습니다"}},"expected":{}}
{"id":"case_83136947","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779023414663-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요, 반갑습니다.","input":{"request":{"requestId":"req_d55e086e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_a9ca5eb1","nickname":"<nick:ebf85b46>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_1686cd08","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779019324629-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임은 /로 구분된 4개 항목(이름/출생연도 두 자리/성별/티어)이 아니어서 안 돼요. 지금은 항목이 2개로 잘못되어 있어요. \n고쳐야 할 부분은 연도/성별/티어를 다시 /로 정확히 나눠서 넣는 거예요. \n성별은 남 또는 여 한 글자, 티어는 GM/I/B/S/G/P/E/D/M/C/U 중 한 글자여야 해요.","input":{"request":{"requestId":"req_7cb58844","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:23dc710b>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_0f722e08","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779015436203-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_a6a1d38f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bc35f985","nickname":"<nick:c3de5ead>","trigger":"user-message","userMessage":"안녕하세요~"}},"expected":{}}
{"id":"case_009d9558","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779011507015-0; llmOverall=2.25; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 절차는 \n카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 서버 프로필 닉네임 설정, 디스코드 주민신청 채널 메시지 남기기 입니다.","input":{"request":{"requestId":"req_18174c18","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:94137b1e>","trigger":"user-message","userMessage":"이러면되나요"}},"expected":{}}
{"id":"case_b51a6dfb","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779011451230-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_5d3182b0","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:94137b1e>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_8f87e018","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779562992008-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=고구마는보라색/98/남/D님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_d9ff0f8b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:b410b374>","trigger":"user-message","userMessage":"봇인가요? 어케 이렇게 빠르지"}},"expected":{}}
{"id":"case_51fec5d7","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779563007004-0; llmOverall=1.25; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=고구마는보라색/98/남/D님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_058c9557","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:b410b374>","trigger":"user-message","userMessage":"아하"}},"expected":{}}
{"id":"case_47db12bc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779544865402-0; llmOverall=1.25; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=방 하트 인증은 채팅방 오른쪽 상단 메뉴 버튼에서 \"커버 보기\"를 눌러 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷을 올려주시면 돼요.","input":{"request":{"requestId":"req_887be307","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_e35470e6","nickname":"<nick:01e9e582>","trigger":"user-message","userMessage":"어떻게 인증하나요 ?"}},"expected":{}}
{"id":"case_8aa3d0bc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779574878821-0; llmOverall=2; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=아직 승인해드리기 어려워요. 남은 항목이 카카오톡 본인인증방 입장과 디스코드 주민신청 채널 메시지라서, 이 두 가지만 먼저 완료해주시면 될 것 같아요.","input":{"request":{"requestId":"req_117cdf74","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"디코 승인 부탁드려도 될까요?"}},"expected":{}}
{"id":"case_0191603e","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779574708797-0; llmOverall=2; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=디스코드 관련 안내입니다. 디스코드 닉네임은 닉네임#태그/출생 연도 두 자리/티어 형식으로 수정해 주세요. 예시: 할리#프렌치불독/88/D \n디스코드 서버에서 이 닉네임은 본인 식별용이라 꼭 맞춰야 하고, 주민신청 채널에는 서버 소개 메시지도 남겨야 해요.","input":{"request":{"requestId":"req_17d4f38d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"/디코"}},"expected":{}}
{"id":"case_214e18f8","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779562888510-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=@고구마는보라색/98/D/남님 닉네임에서 수정할 부분이 있어요!\n성별을 '남' 또는 '여'로 입력해주세요\n티어 코드(GM, I, B, S, G, P, E, D, M, C, U 중 하나)를 입력해주세요","input":{"request":{"requestId":"req_dd3e072b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:f7254b29>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_b90d32a5","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535490115-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=네, 다시 보내주시면 돼요. \n방에서 오른쪽 상단 메뉴 버튼 열고 커버 보기로 들어간 뒤 하트를 눌러 빨간색으로 바꿔주세요. \n그 상태 화면을 채팅방에서 바로 캡처해서 편집(자르기/필터/화면 편집) 없이 원본 그대로 보내주시면 됩니다. \n참고로 회색 하트면 아직 인증으로 안 된 상태예요.","input":{"request":{"requestId":"req_8ac4ab90","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_dc095d27","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535154924-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=네, 다시 시도할 수 있어요. \n방 하트를 빨간색으로 바꾼 뒤, 채팅방에서 바로 캡처한 원본 화면(편집/자르기/필터 없음)만 새로 보내주시면 됩니다.","input":{"request":{"requestId":"req_bb9360ad","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_c33b0453","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535131374-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 하트는 눌렀다면 하트가 빨간색인 화면을 편집 없이 원본 스크린샷으로 올려주시면 돼요.","input":{"request":{"requestId":"req_5c1b0e13","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"user-message","userMessage":"하트는 눌렀는대.."}},"expected":{}}
{"id":"case_955dc7df","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1779605646714-0; llmOverall=2.5; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=오케이, 하트 인증은 화면이 제대로 올라오면 바로 확인돼요. \n지금 올리신 걸로 보냈다면 잠깐 확인 기다려볼게요. \n만약 아직 안 보인다면, 하트를 빨간색으로 바꾼 그대로의 원본 캡처만 한 번 더 보내주세요.","input":{"request":{"requestId":"req_fb38ace6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0ea265d7","nickname":"<nick:7a45b00c>","trigger":"user-message","userMessage":"위에 인증햇어여"}},"expected":{}}
{"id":"case_e75e191e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779604227776-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=다시 시도하려면 이렇게 하면 돼요.\n\n채팅방 오른쪽 상단 메뉴 버튼에서 커버 보기로 들어가서 하트를 빨간색으로 바꾼 뒤, \n그 장면이 보이게 편집 없이 원본 화면 캡처로 다시 보내주면 돼요.","input":{"request":{"requestId":"req_4368c115","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0ea265d7","nickname":"<nick:d0f3633e>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_f485439f","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779593282207-0; llmOverall=n/a; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=Summer/95/남/S님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_a33cd695","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:7ae6ee97>","trigger":"user-message","userMessage":"가입했습니다"}},"expected":{}}
{"id":"case_75bc3458","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779593274596-0; llmOverall=n/a; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=Summer/95/남/S님 이제 추가 절차만 남았어요. 카카오톡 본인인증방 입장과 디스코드 주민신청 채널에 메시지 남기기을 확인해 주세요.","input":{"request":{"requestId":"req_5bb3e408","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:7ae6ee97>","trigger":"user-message","userMessage":"디스코드"}},"expected":{}}
{"id":"case_c1016e76","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779592708891-0; llmOverall=5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 천천히 하세요.","input":{"request":{"requestId":"req_179a599b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:50cd33f9>","trigger":"user-message","userMessage":"엇 잠시만용"}},"expected":{}}
{"id":"case_12f3372d","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779574484786-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 바로 여기 채팅방에 올리면 됩니다. \n채팅방 오른쪽 상단 메뉴의 \"커버 보기\"에서 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷으로 보내주세요.","input":{"request":{"requestId":"req_b7b755bf","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"하트인증은 여기다 올리면 될까요?"}},"expected":{}}
{"id":"case_9f0cb2f4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779574363525-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_b822a53e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:90b41f68>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_885a35a9","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629981088-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_42415f0f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"진짜 잘하시네요"}},"expected":{}}
{"id":"case_4603c0cc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629976794-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_5400a31f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"챌린저 ㄷㄷ"}},"expected":{}}
{"id":"case_2b38adb8","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629930176-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=채팅방 프로필에서 하트를 눌러 빨간색으로 만든 다음, 그 화면이 보이게 원본 스크린샷을 보내주시면 인증돼요.","input":{"request":{"requestId":"req_bf140842","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"어떻게 하나여"}},"expected":{}}
{"id":"case_f04b5287","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779633055712-0; llmOverall=1.25; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요, 오늘도 편하게 진행해요.","input":{"request":{"requestId":"req_bdd1114b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:c673b3e2>","trigger":"user-message","userMessage":"안녕하세요!"}},"expected":{}}
{"id":"case_3e077513","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779630086906-0; llmOverall=1.5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 그렇게 해주시면 돼요.","input":{"request":{"requestId":"req_bbc426e2","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"아항"}},"expected":{}}
{"id":"case_274fec57","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629952313-0; llmOverall=1.5; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_c69053e8","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"보내면 되나여?"}},"expected":{}}
{"id":"case_29f02c6e","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629936899-0; llmOverall=1.5; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_be0e7f4a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"여기 채팅방에"}},"expected":{}}
{"id":"case_59c880e4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779633743117-0; llmOverall=4.25; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=카카오톡에서 디스코드 초대 안내를 받았던 메시지를 찾아 초대 수락을 눌러 들어가면 돼요.","input":{"request":{"requestId":"req_705c9e3f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:c673b3e2>","trigger":"user-message","userMessage":"저 디코는 어떻게 들어가요? 주민신청까지 해뒀습니다"}},"expected":{}}
{"id":"case_4fe5864d","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779632430014-0; llmOverall=5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=좋아요, 천천히 진행해 주세요.","input":{"request":{"requestId":"req_781682ec","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:658a6c82>","trigger":"user-message","userMessage":"넵"}},"expected":{}}
{"id":"case_9a3640ce","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779632381253-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=안 되는 이유는 티어가 허용 코드(GM, I, B, S, G, P, E, D, M, C, U) 중 하나의 대문자로 되어야 하는데 p가 소문자여서예요. \n카카오톡 닉네임의 마지막 부분만 P로 바꿔서 세나/95/여/P로 수정해 주세요.","input":{"request":{"requestId":"req_7368778f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:1f050511>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_5ea6c4d2","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779629969676-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=환영해요! 반갑습니다.","input":{"request":{"requestId":"req_9352c82d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"안녕하세요!!"}},"expected":{}}
{"id":"case_207dec80","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779629904020-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=방 하트 인증은 채팅방 오른쪽 상단 메뉴 버튼의 '커버 보기'에서 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 화면이 보이게 스크린샷을 보내주시면 돼요.","input":{"request":{"requestId":"req_37928f7d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"방하트 인증은"}},"expected":{}}
{"id":"case_b42ca588","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779622035543-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=어서오세요!","input":{"request":{"requestId":"req_e9dcff68","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3adc9936","nickname":"<nick:ccb74517>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_d96d1b98","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779619834487-0; llmOverall=n/a; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=네, 본인인증이 완료됐다면 본인인증방은 나가셔도 돼요. \n현재 남은 절차는 디스코드 주민신청 채널 메시지 남기기 하나예요.","input":{"request":{"requestId":"req_e6bc023c","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_d9c80f01","nickname":"<nick:9c5d5e3a>","trigger":"user-message","userMessage":"본인인증방은이제나가도돼요?"}},"expected":{}}
+23
View File
@@ -93,6 +93,29 @@ describe("judgeOnboardingWithLlm", () => {
} }
}); });
it("keeps scores when rationale is longer than the prompt asks for", async () => {
const result = await judgeOnboardingWithLlm(
new FakeLlmClient(
JSON.stringify({
friendliness: 3,
accuracy: 4,
concision: 5,
scope: 4,
rationale: "가".repeat(800),
}),
),
evalCase,
"이제 방 하트 인증만 남았어요.",
derived,
);
expect(result.skipped).toBe(false);
if (!result.skipped) {
expect(result.scores.overall).toBe(4);
expect(result.rationale).toHaveLength(500);
}
});
it("skips when the judge returns invalid JSON", async () => { it("skips when the judge returns invalid JSON", async () => {
const result = await judgeOnboardingWithLlm( const result = await judgeOnboardingWithLlm(
new FakeLlmClient('{"friendliness":"bad"}'), new FakeLlmClient('{"friendliness":"bad"}'),
+9 -2
View File
@@ -9,6 +9,8 @@ import type {
} from "../types.js"; } from "../types.js";
import { getSkillJudgeConfig } from "./skill-config.js"; import { getSkillJudgeConfig } from "./skill-config.js";
const MAX_RATIONALE_CHARS = 500;
function round(value: number): number { function round(value: number): number {
return Number(value.toFixed(3)); return Number(value.toFixed(3));
} }
@@ -18,10 +20,14 @@ function createJudgeSchema(axes: readonly string[]): z.ZodObject<z.ZodRawShape>
...Object.fromEntries( ...Object.fromEntries(
axes.map((axis) => [axis, z.number().int().min(1).max(5)]), axes.map((axis) => [axis, z.number().int().min(1).max(5)]),
), ),
rationale: z.string().trim().min(1).max(200), rationale: z.string().trim().min(1),
}); });
} }
function normalizeRationale(value: unknown): string {
return String(value).slice(0, MAX_RATIONALE_CHARS);
}
function buildJudgePrompt( function buildJudgePrompt(
skillName: string, skillName: string,
evalCase: EvalCaseDefinition, evalCase: EvalCaseDefinition,
@@ -35,6 +41,7 @@ function buildJudgePrompt(
`다음 ${skillName} 에이전트 응답을 평가해 주세요.`, `다음 ${skillName} 에이전트 응답을 평가해 주세요.`,
`평가 축은 ${axes} 입니다.`, `평가 축은 ${axes} 입니다.`,
"각 점수는 1~5의 정수만 사용하세요.", "각 점수는 1~5의 정수만 사용하세요.",
"rationale은 200자 이하로 짧게 작성하세요.",
config.promptTemplate ?? config.promptTemplate ??
"accuracy는 기대 조건과 실제 답변의 사실성을, concision은 과도한 부가 설명 여부를 포함해 평가하세요.", "accuracy는 기대 조건과 실제 답변의 사실성을, concision은 과도한 부가 설명 여부를 포함해 평가하세요.",
"", "",
@@ -129,7 +136,7 @@ export async function judgeSkillWithLlm(
endpoint: response.endpoint, endpoint: response.endpoint,
provider: response.provider, provider: response.provider,
scores, scores,
rationale: String(parsed["rationale"]), rationale: normalizeRationale(parsed["rationale"]),
raw: response.content, raw: response.content,
}; };
} catch { } catch {
+16
View File
@@ -0,0 +1,16 @@
{
"description": "deadline-near 상태에서도 사용자가 묻지 않은 남은 등록 절차를 자동으로 붙이지 않도록 우선순위 강화",
"diff": [
{
"layer": "policy",
"op": "append",
"reason": "첫 번째 실패에서 현재 상태의 remaining 정보가 질문보다 우선되어 무관한 절차 안내가 출력됐다. deadline-near에서도 질문 우선 원칙을 강화한다.",
"text": "[deadline-near 응답 범위 제한]\n- phase가 deadline-near여도 마지막 메시지가 가능 여부/확인 주체/확인 여부를 묻는 질문이면, 독촉성 남은 절차 안내를 붙이지 마.\n- remaining 상태는 사용자가 \"뭐가 남았나요\", \"다 했나요\", \"완료인가요\"처럼 진행 상태를 물을 때만 사용해.\n- 답변 전 마지막으로 확인: 내 응답 첫 문장이 사용자의 질문에 대한 '네/아니요/아직/관리자가 확인' 중 직접 답인지 점검해."
}
],
"expectedImprovement": "무관한 카카오톡/디스코드 절차 나열이 줄고, 사용자의 마지막 질문 중심으로 1문장 내외의 답변이 생성된다.",
"rationale": "현재 스냅샷에는 remaining 정보가 강하게 주입되어 있어, 사용자가 재등록 가능 여부만 묻는 상황에서도 남은 절차를 안내할 위험이 높다. 진행 상태 질문이 아닐 때 remaining을 쓰지 않게 하면 범위 이탈을 줄일 수 있다.",
"skill": "onboarding",
"variantId": "579dcc7d",
"trustTier": "T2"
}
+16
View File
@@ -0,0 +1,16 @@
{
"description": "디스코드 확인 주체/재등록 가능 여부 질문에는 상태 나열이나 불확실 회피 대신 짧게 직접 답하도록 하는 예외 규칙 추가",
"diff": [
{
"layer": "policy",
"op": "append",
"reason": "실패 케이스가 모두 사용자의 단일 질문에 답하지 못하고, 확인 불가 회피 또는 무관한 남은 절차 나열로 이탈했다. 해당 slice에서 반복되는 직접 질문 패턴을 좁게 처리한다.",
"text": "[discord-verify-question 직접응답 보강]\n- 사용자가 \"디코는 누가 확인하나요\", \"관리자가 확인하나요\", \"관리자 형님들이 확인해야 하나요\"처럼 디스코드 확인 주체를 물으면, 확인되지 않았다고 회피하지 말고 \"네, 디스코드 관련 확인은 관리자/운영진이 확인해요.\"라고 짧게 답해.\n- 사용자가 \"계정 확인 후 다시 등록해도 되나요\", \"제 명의 계정 확인 후 다시 등록해도 될까요\"처럼 본인 계정/명의 확인 뒤 재등록 가능 여부를 물으면, 다른 절차를 나열하지 말고 \"네, 본인 명의 계정 확인 후 다시 등록해 주시면 돼요.\"라고 답해.\n- 위 두 유형에서는 카카오톡 완료 여부, 남은 단계, 디스코드 닉네임 형식, 다음 절차를 사용자가 묻지 않는 한 덧붙이지 마."
}
],
"expectedImprovement": "디스코드 확인 주체 질문에는 '관리자/운영진이 확인한다'고 정확히 답하고, 계정 확인 후 재등록 질문에는 '가능하다'고 직접 답해 accuracy와 scope가 개선된다.",
"rationale": "현재 프롬프트의 '확인되지 않았으면 확인되지 않았다고만 답'과 '세션 상태' 정보가 과하게 작동해, 실제로는 일반 운영 규칙으로 답할 수 있는 질문에도 회피하거나 상태 목록을 출력한다. 확인 주체와 재등록 가능 여부는 온보딩 절차의 일반 질문이므로 짧은 정답 템플릿을 추가하는 것이 가장 작은 수정이다.",
"skill": "onboarding",
"variantId": "991bfb18",
"trustTier": "T2"
}
+52 -16
View File
@@ -31,7 +31,13 @@ import type {
DerivedState, DerivedState,
} from "@skills/onboarding/state/onboarding-state"; } from "@skills/onboarding/state/onboarding-state";
import { noopShadowEvalPublisher } from "./shadow-bus.js"; import { noopShadowEvalPublisher } from "./shadow-bus.js";
import { getOnboardingSessionKeyPrefix, getOpenRouterApiUrl } from "../config.js"; import {
getEvalCaseDeadlineMs,
getEvalLlmProvider,
getEvalRuntimeLlmEnabled,
getOnboardingSessionKeyPrefix,
getOpenRouterApiUrl,
} from "../config.js";
import { runEvalDispatch } from "./handoff-dispatch.js"; import { runEvalDispatch } from "./handoff-dispatch.js";
import { judgeHandoff } from "./judges/handoff-judge.js"; import { judgeHandoff } from "./judges/handoff-judge.js";
import { judgeSkillWithLlm } from "./judges/llm-judge.js"; import { judgeSkillWithLlm } from "./judges/llm-judge.js";
@@ -285,6 +291,9 @@ function toProviderEndpoints(
} }
export function createEvalLlmClient(): LlmClient { export function createEvalLlmClient(): LlmClient {
const evalProvider = getEvalLlmProvider();
const includeProvider = (provider: ProviderName): boolean =>
evalProvider === "all" || evalProvider === provider;
const cliproxyEndpoints = parseEndpointList(process.env["CLI_PROXY_API_URLS"]); const cliproxyEndpoints = parseEndpointList(process.env["CLI_PROXY_API_URLS"]);
const llamaEndpoints = parseEndpointList( const llamaEndpoints = parseEndpointList(
process.env["AGENT_LLM_API_URLS"], process.env["AGENT_LLM_API_URLS"],
@@ -301,7 +310,14 @@ export function createEvalLlmClient(): LlmClient {
); );
const providerApiKeys = Object.fromEntries( const providerApiKeys = Object.fromEntries(
[ [
["cliproxy", firstNonEmpty(process.env["CLI_PROXY_API_KEY"])], [
"cliproxy",
firstNonEmpty(
process.env["CLI_PROXY_API_KEY"],
process.env["LLAMA_SERVER_API_KEY"],
process.env["AGENT_LLM_API_KEY"],
),
],
["llama", llmApiKey], ["llama", llmApiKey],
["openrouter", openRouterApiKey], ["openrouter", openRouterApiKey],
].filter((entry): entry is [string, string] => typeof entry[1] === "string"), ].filter((entry): entry is [string, string] => typeof entry[1] === "string"),
@@ -309,9 +325,15 @@ export function createEvalLlmClient(): LlmClient {
return new LlmClient({ return new LlmClient({
endpoints: [ endpoints: [
...toProviderEndpoints(cliproxyEndpoints, "cliproxy"), ...(includeProvider("cliproxy")
...toProviderEndpoints(openRouterEndpoints, "openrouter"), ? toProviderEndpoints(cliproxyEndpoints, "cliproxy")
...toProviderEndpoints(llamaEndpoints, "llama"), : []),
...(includeProvider("openrouter")
? toProviderEndpoints(openRouterEndpoints, "openrouter")
: []),
...(includeProvider("llama")
? toProviderEndpoints(llamaEndpoints, "llama")
: []),
], ],
apiKey: llmApiKey, apiKey: llmApiKey,
model: firstNonEmpty(process.env["AGENT_LLM_MODEL"]), model: firstNonEmpty(process.env["AGENT_LLM_MODEL"]),
@@ -846,7 +868,8 @@ export async function loadEvalCases(
async function evaluateCase( async function evaluateCase(
evalCase: EvalCaseDefinition, evalCase: EvalCaseDefinition,
llmClient: LlmClient, runtimeLlmClient: LlmClient,
judgeLlmClient: LlmClient,
enableLlmJudge: boolean, enableLlmJudge: boolean,
skillName: string, skillName: string,
): Promise<EvalCaseResult> { ): Promise<EvalCaseResult> {
@@ -860,7 +883,7 @@ async function evaluateCase(
: undefined; : undefined;
const deps = { const deps = {
garnet: inMemoryGarnet, garnet: inMemoryGarnet,
llm: llmClient, llm: runtimeLlmClient,
room: roomContextService, room: roomContextService,
}; };
if (skillName === "handoff") { if (skillName === "handoff") {
@@ -868,7 +891,7 @@ async function evaluateCase(
evalCase, evalCase,
input, input,
Object.values(SKILL_RUNNER_FACTORIES).map((factory) => factory(deps)), Object.values(SKILL_RUNNER_FACTORIES).map((factory) => factory(deps)),
llmClient, judgeLlmClient,
enableLlmJudge, enableLlmJudge,
); );
} }
@@ -883,7 +906,7 @@ async function evaluateCase(
return runWithContext( return runWithContext(
{ {
chatroomId: input.request.chatroomId, chatroomId: input.request.chatroomId,
deadlineAt: new Date(Date.now() + 4_500).toISOString(), deadlineAt: new Date(Date.now() + getEvalCaseDeadlineMs()).toISOString(),
requestId: input.request.requestId, requestId: input.request.requestId,
skill: input.request.skill, skill: input.request.skill,
startedAt: new Date().toISOString(), startedAt: new Date().toISOString(),
@@ -896,7 +919,8 @@ async function evaluateCase(
evalCase, evalCase,
input, input,
skill, skill,
llmClient, runtimeLlmClient,
judgeLlmClient,
enableLlmJudge, enableLlmJudge,
effectiveSkill, effectiveSkill,
) )
@@ -904,7 +928,7 @@ async function evaluateCase(
evalCase, evalCase,
input, input,
skill, skill,
llmClient, judgeLlmClient,
enableLlmJudge, enableLlmJudge,
effectiveSkill, effectiveSkill,
), ),
@@ -915,11 +939,12 @@ async function evaluateOnboardingCase(
evalCase: EvalCaseDefinition, evalCase: EvalCaseDefinition,
input: EvalCaseInput, input: EvalCaseInput,
skill: Skill, skill: Skill,
llmClient: LlmClient, runtimeLlmClient: LlmClient,
judgeLlmClient: LlmClient,
enableLlmJudge: boolean, enableLlmJudge: boolean,
skillName: string, skillName: string,
): Promise<EvalCaseResult> { ): Promise<EvalCaseResult> {
const classifier = new OnboardingIntentClassifier(llmClient); const classifier = new OnboardingIntentClassifier(runtimeLlmClient);
const processTrace = new ProcessTraceCollector(); const processTrace = new ProcessTraceCollector();
const judgeConfig = getSkillJudgeConfig(skillName); const judgeConfig = getSkillJudgeConfig(skillName);
@@ -950,7 +975,7 @@ async function evaluateOnboardingCase(
}); });
const llmJudge = enableLlmJudge const llmJudge = enableLlmJudge
? await judgeSkillWithLlm( ? await judgeSkillWithLlm(
llmClient, judgeLlmClient,
skillName, skillName,
evalCase, evalCase,
response.text, response.text,
@@ -1262,12 +1287,23 @@ export async function runEval(
limit: options?.limit, limit: options?.limit,
}); });
const llmClient = createEvalLlmClient(); const judgeLlmClient = createEvalLlmClient();
const runtimeLlmClient = getEvalRuntimeLlmEnabled()
? judgeLlmClient
: new LlmClient({ endpoints: [] });
const enableLlmJudge = options?.enableLlmJudge ?? true; const enableLlmJudge = options?.enableLlmJudge ?? true;
const caseResults: EvalCaseResult[] = []; const caseResults: EvalCaseResult[] = [];
for (const evalCase of evalCases) { for (const evalCase of evalCases) {
caseResults.push(await evaluateCase(evalCase, llmClient, enableLlmJudge, skill)); caseResults.push(
await evaluateCase(
evalCase,
runtimeLlmClient,
judgeLlmClient,
enableLlmJudge,
skill,
),
);
} }
const summary = createSummary(caseResults); const summary = createSummary(caseResults);
+101
View File
@@ -25,6 +25,8 @@ vi.mock("@obs/spans", () => ({
})); }));
import { LlmClient } from "./llm-client.js"; import { LlmClient } from "./llm-client.js";
import { BudgetTracker } from "@agent/budget";
import { runWithContext } from "@obs/context";
import type { ChatCompletionResponse } from "./types.js"; import type { ChatCompletionResponse } from "./types.js";
function okResponse( function okResponse(
@@ -168,4 +170,103 @@ describe("LlmClient endpoint normalization", () => {
}), }),
); );
}); });
it("records LLM cost into the request budget", async () => {
vi.stubGlobal(
"fetch",
vi.fn(() =>
Promise.resolve(
okResponse("reply", {
usage: {
completion_tokens: 50,
prompt_tokens: 100,
total_tokens: 150,
},
}),
),
),
);
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
const client = new LlmClient({
endpoints: [{ provider: "openrouter", url: "https://openrouter.ai/api/v1" }],
providerApiKeys: { openrouter: "test-key" },
});
await runWithContext(
{
budget,
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() =>
client.chatCompletion([{ content: "hi", role: "user" }], {
modelOverride: "gpt-5.3-codex-spark",
}),
);
expect(budget.snapshot().totalLlmCostUsd).toBe(0.000045);
expect(llmSpanMock.setAttributes).toHaveBeenCalledWith(
expect.objectContaining({
"budget.llm_cost_usd": 0.000045,
"budget.total_cost_usd": 0.000045,
}),
);
});
it("does not record LLM cost when budget cost tracking is disabled", async () => {
vi.stubEnv("AGENT_BUDGET_LLM_COST_ENABLED", "false");
vi.stubGlobal(
"fetch",
vi.fn(() =>
Promise.resolve(
okResponse("reply", {
usage: {
completion_tokens: 50,
prompt_tokens: 100,
total_tokens: 150,
},
}),
),
),
);
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
const client = new LlmClient({
endpoints: [{ provider: "openrouter", url: "https://openrouter.ai/api/v1" }],
providerApiKeys: { openrouter: "test-key" },
});
await runWithContext(
{
budget,
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() =>
client.chatCompletion([{ content: "hi", role: "user" }], {
modelOverride: "gpt-5.3-codex-spark",
}),
);
expect(budget.snapshot().totalLlmCostUsd).toBe(0);
});
}); });
+30 -1
View File
@@ -10,6 +10,7 @@ import type {
import { getContext, getRequestLogger } from "@obs/context"; import { getContext, getRequestLogger } from "@obs/context";
import { maskTraceContent, withLlmSpan } from "@obs/spans"; import { maskTraceContent, withLlmSpan } from "@obs/spans";
import { import {
getAgentBudgetLlmCostEnabled,
getLlmHealthTimeoutMs, getLlmHealthTimeoutMs,
getLlmRequestTimeoutMs, getLlmRequestTimeoutMs,
getLlmTemperature, getLlmTemperature,
@@ -193,9 +194,10 @@ export class LlmClient {
continue; continue;
} }
const configuredRequestTimeoutMs = getLlmRequestTimeoutMs();
const requestTimeoutMs = Math.max( const requestTimeoutMs = Math.max(
1, 1,
Math.min(getLlmRequestTimeoutMs(), options?.deadlineMs ?? Number.POSITIVE_INFINITY), Math.min(configuredRequestTimeoutMs, options?.deadlineMs ?? Number.POSITIVE_INFINITY),
); );
const models = selectModelsForProvider(provider.name, options, this.llamaModel); const models = selectModelsForProvider(provider.name, options, this.llamaModel);
let endpointSucceeded = false; let endpointSucceeded = false;
@@ -240,6 +242,9 @@ export class LlmClient {
model: request.model, model: request.model,
status: response.status, status: response.status,
body, body,
configuredTimeoutMs: configuredRequestTimeoutMs,
deadlineMs: options?.deadlineMs,
timeoutMs: requestTimeoutMs,
}, },
"[LLM] Request failed", "[LLM] Request failed",
); );
@@ -257,11 +262,22 @@ export class LlmClient {
model: request.model, model: request.model,
skill: getContext()?.skill, skill: getContext()?.skill,
}); });
if (getAgentBudgetLlmCostEnabled()) {
getContext()?.budget?.recordLlmCost(costUsd);
}
const budgetSnapshot = getContext()?.budget?.snapshot();
span.setAttributes({ span.setAttributes({
...(shouldLogPrompts ...(shouldLogPrompts
? buildCompletionTraceAttributes(result) ? buildCompletionTraceAttributes(result)
: {}), : {}),
...(budgetSnapshot
? {
"budget.llm_cost_usd": budgetSnapshot.totalLlmCostUsd,
"budget.tool_cost_usd": budgetSnapshot.totalToolCostUsd,
"budget.total_cost_usd": budgetSnapshot.totalCostUsd,
}
: {}),
"gen_ai.response.finish_reasons": payload.choices "gen_ai.response.finish_reasons": payload.choices
?.map((choice) => choice.finish_reason ?? "unknown") ?.map((choice) => choice.finish_reason ?? "unknown")
.filter((reason): reason is string => Boolean(reason)), .filter((reason): reason is string => Boolean(reason)),
@@ -292,6 +308,16 @@ export class LlmClient {
if (llmResult) { if (llmResult) {
await this.circuit.recordSuccess(endpoint.completionUrl); await this.circuit.recordSuccess(endpoint.completionUrl);
endpointSucceeded = true; endpointSucceeded = true;
logger.debug(
{
endpoint: endpoint.completionUrl,
model: request.model,
provider: provider.name,
role: options?.role,
tier: options?.tier,
},
"[LLM] Response received",
);
return llmResult; return llmResult;
} }
} catch (error) { } catch (error) {
@@ -301,6 +327,9 @@ export class LlmClient {
provider: provider.name, provider: provider.name,
model, model,
error: error instanceof Error ? error.message : String(error), error: error instanceof Error ? error.message : String(error),
configuredTimeoutMs: configuredRequestTimeoutMs,
deadlineMs: options?.deadlineMs,
timeoutMs: requestTimeoutMs,
}, },
"[LLM] Request error", "[LLM] Request error",
); );
+63 -6
View File
@@ -1,6 +1,7 @@
import { afterEach, describe, expect, it, vi } from "vitest"; import { afterEach, describe, expect, it, vi } from "vitest";
import type { LlmEndpointConfig } from "./types.js"; import type { LlmEndpointConfig } from "./types.js";
import { import {
resetLlmRoutingRoundRobinForTests,
selectMaxTokensForTier, selectMaxTokensForTier,
selectModelsForProvider, selectModelsForProvider,
selectProviderOrder, selectProviderOrder,
@@ -8,6 +9,12 @@ import {
describe("llm routing", () => { describe("llm routing", () => {
afterEach(() => { afterEach(() => {
delete process.env["CLIPROXY_TIER0_MODEL"];
delete process.env["CLIPROXY_TIER1_PLANNER_MODEL"];
delete process.env["CLIPROXY_TIER2_CRITIC_MODEL"];
delete process.env["LLAMA_TIER2_CRITIC_MODEL"];
delete process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"];
delete process.env["OPENROUTER_TIER1_RESPONDER_MODEL"];
delete process.env["OPENROUTER_MODEL"]; delete process.env["OPENROUTER_MODEL"];
delete process.env["TIER0_MODEL"]; delete process.env["TIER0_MODEL"];
delete process.env["TIER1_PLANNER_MODEL"]; delete process.env["TIER1_PLANNER_MODEL"];
@@ -16,6 +23,7 @@ describe("llm routing", () => {
delete process.env["TIER0_MAX_TOKENS"]; delete process.env["TIER0_MAX_TOKENS"];
delete process.env["TIER1_MAX_TOKENS"]; delete process.env["TIER1_MAX_TOKENS"];
delete process.env["TIER2_MAX_TOKENS"]; delete process.env["TIER2_MAX_TOKENS"];
resetLlmRoutingRoundRobinForTests();
vi.restoreAllMocks(); vi.restoreAllMocks();
}); });
@@ -93,9 +101,12 @@ describe("llm routing", () => {
).toEqual(["cliproxy", "openrouter"]); ).toEqual(["cliproxy", "openrouter"]);
}); });
it("returns role-aware model for cliproxy and llama fallback as single-item list", () => { it("returns provider-aware models for runtime roles and tiers", () => {
process.env["TIER1_PLANNER_MODEL"] = "cliproxy-planner"; process.env["CLIPROXY_TIER1_PLANNER_MODEL"] = "cliproxy-planner";
process.env["TIER2_CRITIC_MODEL"] = "cliproxy-critic"; process.env["CLIPROXY_TIER2_CRITIC_MODEL"] = "cliproxy-critic";
process.env["OPENROUTER_TIER1_RESPONDER_MODEL"] =
"openrouter-fast, openrouter-backup";
process.env["LLAMA_TIER2_CRITIC_MODEL"] = "llama-critic";
expect( expect(
selectModelsForProvider("cliproxy", { role: "planner", tier: 1 }, "local-qwen"), selectModelsForProvider("cliproxy", { role: "planner", tier: 1 }, "local-qwen"),
@@ -103,9 +114,12 @@ describe("llm routing", () => {
expect( expect(
selectModelsForProvider("cliproxy", { role: "critic", tier: 2 }, "local-qwen"), selectModelsForProvider("cliproxy", { role: "critic", tier: 2 }, "local-qwen"),
).toEqual(["cliproxy-critic"]); ).toEqual(["cliproxy-critic"]);
expect(selectModelsForProvider("llama", { tier: 2 }, "local-qwen")).toEqual([ expect(
"local-qwen", selectModelsForProvider("openrouter", { role: "responder", tier: 1 }, "local-qwen"),
]); ).toEqual(["openrouter-fast", "openrouter-backup"]);
expect(
selectModelsForProvider("llama", { role: "critic", tier: 2 }, "local-qwen"),
).toEqual(["llama-critic"]);
}); });
it("splits OPENROUTER_MODEL by comma and preserves priority order", () => { it("splits OPENROUTER_MODEL by comma and preserves priority order", () => {
@@ -124,6 +138,49 @@ describe("llm routing", () => {
]); ]);
}); });
it("round-robins models within the same provider role and tier when enabled", () => {
process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"] = "true";
process.env["CLIPROXY_TIER0_MODEL"] =
"gemini-3.1-flash-lite, gpt-5.3-codex-spark";
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gemini-3.1-flash-lite", "gpt-5.3-codex-spark"]);
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gpt-5.3-codex-spark", "gemini-3.1-flash-lite"]);
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gemini-3.1-flash-lite", "gpt-5.3-codex-spark"]);
});
it("keeps provider order fixed even when model round-robin is enabled", () => {
process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"] = "true";
const endpoints: LlmEndpointConfig[] = [
{
completionUrl: "http://llama.test/v1/chat/completions",
healthUrl: "http://llama.test/health",
provider: "llama",
},
{
completionUrl: "http://openrouter.test/v1/chat/completions",
healthUrl: "http://openrouter.test",
provider: "openrouter",
},
{
completionUrl: "http://cliproxy.test/v1/chat/completions",
healthUrl: "http://cliproxy.test",
provider: "cliproxy",
},
];
expect(
selectProviderOrder(endpoints, { role: "responder", tier: 1 }).map(
(endpoint) => endpoint.provider,
),
).toEqual(["cliproxy", "openrouter", "llama"]);
});
it("chooses tier defaults for max tokens when explicit maxTokens is absent", () => { it("chooses tier defaults for max tokens when explicit maxTokens is absent", () => {
process.env["TIER0_MAX_TOKENS"] = "111"; process.env["TIER0_MAX_TOKENS"] = "111";
process.env["TIER1_MAX_TOKENS"] = "222"; process.env["TIER1_MAX_TOKENS"] = "222";
+37 -26
View File
@@ -1,11 +1,8 @@
import { import {
getOpenRouterModels, getLlmModelRoundRobinEnabled,
getRuntimeLlmModelsForProvider,
getTier0MaxTokens, getTier0MaxTokens,
getTier0Model,
getTier1MaxTokens, getTier1MaxTokens,
getTier1PlannerModel,
getTier1ResponderModel,
getTier2CriticModel,
getTier2MaxTokens, getTier2MaxTokens,
getLlmMaxTokens, getLlmMaxTokens,
} from "../config.js"; } from "../config.js";
@@ -13,6 +10,7 @@ import type { ChatCompletionOptions, LlmEndpointConfig, ProviderName } from "./t
const PROVIDER_PRIORITY: readonly ProviderName[] = ["cliproxy", "openrouter", "llama"]; const PROVIDER_PRIORITY: readonly ProviderName[] = ["cliproxy", "openrouter", "llama"];
const TEMPLATE_ONLY_PROVIDER_PRIORITY: readonly ProviderName[] = ["llama"]; const TEMPLATE_ONLY_PROVIDER_PRIORITY: readonly ProviderName[] = ["llama"];
const modelRoundRobinOffsets = new Map<string, number>();
function getProviderPriority( function getProviderPriority(
options: ChatCompletionOptions | undefined, options: ChatCompletionOptions | undefined,
@@ -33,27 +31,15 @@ export function selectModelsForProvider(
return [options.modelOverride.trim()]; return [options.modelOverride.trim()];
} }
if (provider === "llama") { const models = getRuntimeLlmModelsForProvider(
return [llamaFallbackModel]; provider,
} {
role: options?.role,
if (provider === "openrouter") { tier: options?.tier,
return getOpenRouterModels(); },
} llamaFallbackModel,
);
if (options?.role === "planner") { return rotateModelsForProvider(provider, options, models);
return [getTier1PlannerModel()];
}
if (options?.role === "critic") {
return [getTier2CriticModel()];
}
if (options?.tier === 1 || options?.tier === 2) {
return [getTier1ResponderModel()];
}
return [getTier0Model()];
} }
export function selectMaxTokensForTier( export function selectMaxTokensForTier(
@@ -106,3 +92,28 @@ export function selectProviderOrder(
}) })
.map((item) => item.endpoint); .map((item) => item.endpoint);
} }
function rotateModelsForProvider(
provider: ProviderName,
options: ChatCompletionOptions | undefined,
models: string[],
): string[] {
if (!getLlmModelRoundRobinEnabled() || models.length <= 1) {
return models;
}
const key = [
provider,
options?.role ?? "none",
options?.tier ?? "none",
models.join("|"),
].join(":");
const offset = modelRoundRobinOffsets.get(key) ?? 0;
modelRoundRobinOffsets.set(key, offset + 1);
const rotation = offset % models.length;
return [...models.slice(rotation), ...models.slice(0, rotation)];
}
export function resetLlmRoutingRoundRobinForTests(): void {
modelRoundRobinOffsets.clear();
}
+2 -2
View File
@@ -3,12 +3,12 @@ import { AsyncLocalStorage } from 'node:async_hooks';
import type { Logger } from 'pino'; import type { Logger } from 'pino';
import { getLogger } from './logger.js'; import { getLogger } from './logger.js';
import type { BudgetSnapshot } from '@agent/budget'; import type { BudgetTracker } from '@agent/budget';
export interface RequestContext { export interface RequestContext {
chatroomId: string; chatroomId: string;
deadlineAt?: string; deadlineAt?: string;
budget?: BudgetSnapshot | undefined; budget?: BudgetTracker | undefined;
handoffChain?: string[] | undefined; handoffChain?: string[] | undefined;
handoffDepth?: number | undefined; handoffDepth?: number | undefined;
logger: Logger; logger: Logger;
+12
View File
@@ -15,9 +15,21 @@ type RequestBindings = {
function buildRedactionPaths(): string[] { function buildRedactionPaths(): string[] {
const commonPaths = [ const commonPaths = [
'authorization', 'authorization',
'apiKey',
'api_key',
'headers.authorization', 'headers.authorization',
'headers.x-api-key',
'headers["x-api-key"]',
'req.headers.authorization', 'req.headers.authorization',
'req.headers.x-api-key',
'req.headers["x-api-key"]',
'request.headers.authorization', 'request.headers.authorization',
'request.headers.x-api-key',
'request.headers["x-api-key"]',
'request.apiKey',
'request.api_key',
'response.apiKey',
'response.api_key',
]; ];
const secretPaths = KNOWN_SECRET_KEYS.flatMap((key) => [ const secretPaths = KNOWN_SECRET_KEYS.flatMap((key) => [
+23
View File
@@ -4,6 +4,8 @@ import {
formatReviewList, formatReviewList,
formatReviewShow, formatReviewShow,
formatReviewStats, formatReviewStats,
formatToolAuditList,
formatToolAuditShow,
} from "./cli.js"; } from "./cli.js";
describe("operations review CLI formatters", () => { describe("operations review CLI formatters", () => {
@@ -81,4 +83,25 @@ describe("operations review CLI formatters", () => {
}), }),
).toContain("low-confidence\ttotal=3\tpending=1\tresolved=2"); ).toContain("low-confidence\ttotal=3\tpending=1\tresolved=2");
}); });
it("formats tool audit rows", () => {
const entry = {
argsHash: "abc123",
calledAt: "2026-05-18T00:00:00.000Z",
chatroomId: "room-1",
durationMs: 12,
id: "1-0",
ok: true,
resultSummary: "{\"ok\":true}",
skill: "onboarding",
toolName: "validate-nickname",
type: "tool-call" as const,
userId: "user-1",
};
expect(formatToolAuditList([entry])).toContain(
"1-0\t2026-05-18T00:00:00.000Z\tonboarding\tvalidate-nickname\tyes\t12",
);
expect(formatToolAuditShow(entry)).toContain("argsHash: abc123");
});
}); });
+99 -1
View File
@@ -15,6 +15,10 @@ import {
type SupervisorReviewResolution, type SupervisorReviewResolution,
type SupervisorReviewStats, type SupervisorReviewStats,
} from "@skills/supervisor/review-queue"; } from "@skills/supervisor/review-queue";
import {
ToolAuditLogger,
type ListedToolAuditEntry,
} from "@tools/audit";
import { import {
getFlagString, getFlagString,
hasFlag, hasFlag,
@@ -164,6 +168,46 @@ export function formatReviewStats(stats: SupervisorReviewStats): string {
].join("\n"); ].join("\n");
} }
export function formatToolAuditList(items: ListedToolAuditEntry[]): string {
const rows = [
[
"id",
"calledAt",
"skill",
"toolName",
"ok",
"durationMs",
"errorKind",
],
...items.map((item) => [
item.id,
item.calledAt,
item.skill,
item.toolName,
item.ok ? "yes" : "no",
String(item.durationMs),
item.errorKind ?? "-",
]),
];
return rows.map((row) => row.join("\t")).join("\n");
}
export function formatToolAuditShow(item: ListedToolAuditEntry): string {
return [
`id: ${item.id}`,
`calledAt: ${item.calledAt}`,
`skill: ${item.skill}`,
`toolName: ${item.toolName}`,
`chatroomId: ${item.chatroomId}`,
`userId: ${item.userId}`,
`ok: ${item.ok ? "yes" : "no"}`,
`durationMs: ${item.durationMs}`,
`errorKind: ${item.errorKind ?? "-"}`,
`argsHash: ${item.argsHash}`,
`resultSummary: ${item.resultSummary || "-"}`,
].join("\n");
}
async function handleReviewList( async function handleReviewList(
flags: Map<string, string | true>, flags: Map<string, string | true>,
): Promise<void> { ): Promise<void> {
@@ -277,6 +321,52 @@ async function handleReviewStats(
} }
} }
async function handleToolAuditList(
flags: Map<string, string | true>,
): Promise<void> {
const connection = await connectGarnetForCli();
try {
const items = await new ToolAuditLogger(connection.garnetClient).list({
limit: parsePositiveInteger(getFlagString(flags, "limit")) ?? 50,
since: sinceDurationToIso(getFlagString(flags, "since") ?? "24h"),
skill: getFlagString(flags, "skill"),
});
console.info(
hasFlag(flags, "json")
? JSON.stringify(items, null, 2)
: formatToolAuditList(items),
);
} finally {
await connection.close();
}
}
async function handleToolAuditShow(
id: string | undefined,
flags: Map<string, string | true>,
): Promise<void> {
if (!id?.trim()) {
throw new Error("tool-audit:show requires an entry id");
}
const connection = await connectGarnetForCli();
try {
const item = await new ToolAuditLogger(connection.garnetClient).get(id);
if (!item) {
process.exitCode = 2;
console.error(`[tool-audit] entry not found: ${id}`);
return;
}
console.info(
hasFlag(flags, "json")
? JSON.stringify(item, null, 2)
: formatToolAuditShow(item),
);
} finally {
await connection.close();
}
}
async function main(): Promise<void> { async function main(): Promise<void> {
loadEnvironmentFromNodeEnv(); loadEnvironmentFromNodeEnv();
initLogger(); initLogger();
@@ -298,9 +388,17 @@ async function main(): Promise<void> {
await handleReviewStats(flags); await handleReviewStats(flags);
return; return;
} }
if (command === "tool-audit:list") {
await handleToolAuditList(flags);
return;
}
if (command === "tool-audit:show") {
await handleToolAuditShow(positionals[0], flags);
return;
}
throw new Error( throw new Error(
"Unknown command. Use review:list, review:show, review:resolve, or review:stats.", "Unknown command. Use review:list, review:show, review:resolve, review:stats, tool-audit:list, or tool-audit:show.",
); );
} }
+3
View File
@@ -46,6 +46,9 @@ describe("escalation trigger", () => {
maxHandoffs: 4, maxHandoffs: 4,
maxSameAgentVisits: 2, maxSameAgentVisits: 2,
maxToolCalls: 20, maxToolCalls: 20,
totalCostUsd: 0,
totalLlmCostUsd: 0,
totalToolCostUsd: 0,
toolCalls: 0, toolCalls: 0,
visits: {}, visits: {},
}, },
+1
View File
@@ -16,5 +16,6 @@ export function createOnboardingSkill(deps: SkillDeps): Skill {
deps.shadowEvalBus, deps.shadowEvalBus,
deps.natsClient, deps.natsClient,
deps.memoryService, deps.memoryService,
deps.escalationHistoryService,
); );
} }
+42 -33
View File
@@ -38,6 +38,48 @@ export const ONBOARDING_INTENT_TABLE = {
match: ({ trimmed }) => isRemainingStatusQuestion(trimmed), match: ({ trimmed }) => isRemainingStatusQuestion(trimmed),
triggers: ["progress-update", "timeout-warning"], triggers: ["progress-update", "timeout-warning"],
}, },
"discord-verify-question": {
confidence: 0.96,
keywords: ["디스코드", "확인", "닉네임", "주민신청"],
llm: true,
match: ({ lower, trimmed }) => {
const mentionsDiscord = /(디스코드|디코|discord)/u.test(lower);
const asksDiscordVerification =
/(확인|조회|찾아|검색|검사|바뀌|변경|닉변|맞(?:아|나요)?|됐(?:나|나요)?|완료됐(?:나|나요)?|승인|반영)/u.test(
trimmed,
);
const reportsDiscordNicknameCompletion =
/(닉네임|닉변|프로필\s*닉네임|서버\s*닉네임)/u.test(trimmed) &&
/(완료했(?:어|어요|습니다)?|끝냈(?:어|어요|습니다)?|설정했(?:어|어요|습니다)?|바꿨(?:어|어요|습니다)?|변경했(?:어|어요|습니다)?|다\s*했(?:어|어요|습니다)?|해놨(?:어|어요|습니다)?|해뒀(?:어|어요|습니다)?|적용됐(?:어|어요|습니다)?)/u.test(
trimmed,
);
const reportsResidentApplicationCompletion =
/(주민\s*신청|주민신청|주민\s*등록|주민등록|프로필\s*메시지|메시지)/u.test(
trimmed,
) &&
/(완료했(?:어|어요|습니다)?|끝냈(?:어|어요|습니다)?|남겼(?:어|어요|습니다)?|썼(?:어|어요|습니다)?|작성했(?:어|어요|습니다)?|다\s*했(?:어|어요|습니다)?|해놨(?:어|어요|습니다)?|해뒀(?:어|어요|습니다)?|했(?:어|어요|습니다)?)/u.test(
trimmed,
);
return (
mentionsDiscord &&
(asksDiscordVerification ||
reportsDiscordNicknameCompletion ||
reportsResidentApplicationCompletion)
);
},
},
"discord-question": {
confidence: 0.95,
keywords: ["디스코드", "주민신청", "프로필", "권한", "설정"],
llm: true,
match: ({ lower }) => /(디스코드|디코|discord)/u.test(lower),
regex: [
{
confidence: 0.94,
pattern: /(주민신청\s*채널|주민신청채널|서버\s*프로필)/u,
},
],
},
"help-where": { "help-where": {
confidence: 0.9, confidence: 0.9,
keywords: [ keywords: [
@@ -75,39 +117,6 @@ export const ONBOARDING_INTENT_TABLE = {
/(가입|등록|절차|도움|도와|뭐해야|뭐 해야|해야 돼|해야해|무엇을 해야)/u, /(가입|등록|절차|도움|도와|뭐해야|뭐 해야|해야 돼|해야해|무엇을 해야)/u,
], ],
}, },
"discord-verify-question": {
confidence: 0.96,
keywords: ["디스코드", "확인", "닉네임"],
llm: true,
match: ({ lower, trimmed }) => {
const mentionsDiscord = /(디스코드|디코|discord)/u.test(lower);
const asksDiscordVerification =
/(확인|조회|찾아|검색|검사|바뀌|변경|닉변|맞(?:아|나요)?|됐(?:나|나요)?|완료됐(?:나|나요)?)/u.test(
trimmed,
);
const reportsDiscordNicknameCompletion =
/(닉네임|닉변|프로필\s*닉네임|서버\s*닉네임)/u.test(trimmed) &&
/(완료했(?:어|어요|습니다)?|끝냈(?:어|어요|습니다)?|설정했(?:어|어요|습니다)?|바꿨(?:어|어요|습니다)?|변경했(?:어|어요|습니다)?|다\s*했(?:어|어요|습니다)?|해놨(?:어|어요|습니다)?|해뒀(?:어|어요|습니다)?|적용됐(?:어|어요|습니다)?)/u.test(
trimmed,
);
return (
mentionsDiscord &&
(asksDiscordVerification || reportsDiscordNicknameCompletion)
);
},
},
"discord-question": {
confidence: 0.95,
keywords: ["디스코드", "주민신청", "프로필", "권한", "설정"],
llm: true,
match: ({ lower }) => /(디스코드|디코|discord)/u.test(lower),
regex: [
{
confidence: 0.94,
pattern: /(주민신청\s*채널|주민신청채널|서버\s*프로필)/u,
},
],
},
"age-question": { "age-question": {
confidence: 0.93, confidence: 0.93,
keywords: ["27", "가입", "가능"], keywords: ["27", "가입", "가능"],
+4 -4
View File
@@ -145,12 +145,12 @@ export const ONBOARDING_PROMPT_DATA = {
{ {
user: "디스코드 프로필 닉네임 설정 완료했어요", user: "디스코드 프로필 닉네임 설정 완료했어요",
assistant: assistant:
"디스코드 닉네임을 확인해봤는데 아직 올바른 형식으로 확인되지는 않았어요. 디스코드 서버 닉네임은 이름#태그/출생 연도/티어 형식이에요. 예: 할리#프렌치불독/88/D", "디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '날으는꼴뚜기/D' 으로 확인됩니다. 할리#프렌치불독/88/D 와 같이 닉네임#태그/출생연도/티어로 수정해 주세요.",
}, },
{ {
user: "디스코드 닉네임 바꿨어요", user: "디스코드 닉네임 바꿨어요",
assistant: assistant:
"디스코드 닉네임이 아직 이름#태그/출생 연도/티어 형식으로 확인되지 않아요. 예: 할리#프렌치불독/88/D처럼 태그를 포함해서 설정해 주세요.", "디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '날으는꼴뚜기/D' 으로 확인됩니다. 할리#프렌치불독/88/D 와 같이 닉네임#태그/출생연도/티어로 수정해 주세요.",
}, },
], ],
greeting: [ greeting: [
@@ -236,10 +236,10 @@ export const ONBOARDING_PROMPT_DATA = {
].join("\n"), ].join("\n"),
"discord-question": [ "discord-question": [
"- 디스코드 일반 안내나 규칙 질문이면 lookup-community-rule 도구를 사용해 정확한 규칙만 답해.", "- 디스코드 일반 안내나 규칙 질문이면 lookup-community-rule 도구를 사용해 정확한 규칙만 답해.",
"- 사용자가 디스코드 닉네임이나 프로필 설정을 끝냈다고 말하거나 반영 여부를 묻는다면 verify-discord-nickname 도구를 먼저 사용해 직접 확인해.", "- 사용자가 디스코드 닉네임, 프로필 설정, 주민신청 메시지를 끝냈다고 말하거나 반영 여부를 묻는다면 verify-discord-nickname 도구를 먼저 사용해 직접 확인해.",
].join("\n"), ].join("\n"),
"discord-verify-question": "discord-verify-question":
"- 디스코드 닉네임 확인 요청은 verify-discord-nickname 도구를 먼저 사용하고, 조회 결과만 짧게 답해.", "- 디스코드 닉네임/주민신청 확인 요청은 verify-discord-nickname 도구와 check-onboarding-progress 도구를 먼저 사용하고, 조회 결과만 짧게 답해. 확인 없이 초대 안내만 반복하지 마. 닉네임이 틀렸거나 미설정이면 recommendedNickname/correctionGuide를 우선 사용해 정확히 어떻게 고칠지 안내해.",
"help-where": "help-where":
"- 위치 안내가 모호할 때만 get-room-context 도구를 고려하고, 기본적으로는 절차 위치만 답해.", "- 위치 안내가 모호할 때만 get-room-context 도구를 고려하고, 기본적으로는 절차 위치만 답해.",
"status-question": "status-question":
-3
View File
@@ -4,7 +4,6 @@ import { z } from "zod";
import { import {
getCriticMaxTokens, getCriticMaxTokens,
getLlmTemperature, getLlmTemperature,
getTier2CriticModel,
} from "../../../config.js"; } from "../../../config.js";
import type { DerivedState } from "../state/onboarding-state.js"; import type { DerivedState } from "../state/onboarding-state.js";
import { getActiveProcessTrace } from "./trace.js"; import { getActiveProcessTrace } from "./trace.js";
@@ -147,7 +146,6 @@ export class OnboardingCritic {
{ {
deadlineMs: options?.deadlineMs, deadlineMs: options?.deadlineMs,
maxTokens: getCriticMaxTokens(), maxTokens: getCriticMaxTokens(),
modelOverride: getTier2CriticModel(),
role: "critic", role: "critic",
temperature: Math.max(0, getLlmTemperature() - 0.15), temperature: Math.max(0, getLlmTemperature() - 0.15),
tier: 2, tier: 2,
@@ -205,7 +203,6 @@ export class OnboardingCritic {
deadlineMs: options.deadlineMs, deadlineMs: options.deadlineMs,
jsonMode: true, jsonMode: true,
maxTokens: getCriticMaxTokens(), maxTokens: getCriticMaxTokens(),
modelOverride: getTier2CriticModel(),
role: "critic", role: "critic",
temperature: Math.max(0, getLlmTemperature() - 0.15), temperature: Math.max(0, getLlmTemperature() - 0.15),
tier: 2, tier: 2,
@@ -207,6 +207,317 @@ describe("OnboardingOrchestrator", () => {
expect(calls[1]?.options?.tools).toBeUndefined(); expect(calls[1]?.options?.tools).toBeUndefined();
}); });
it("forces discord verification tools before answering completion reports", async () => {
const calls: Array<{
messages: ChatMessage[];
options: ChatCompletionOptions | undefined;
}> = [];
const executedTools: string[] = [];
const llmClientStub = {
chatCompletion: (
messages: ChatMessage[],
options?: ChatCompletionOptions,
) => {
calls.push({ messages, options });
if (calls.length === 1) {
return Promise.resolve({
content: JSON.stringify({
approach: "answer-direct",
targetPhase: "deadline-near",
mustInclude: [],
mustAvoid: [],
suggestedTools: [],
tone: "friendly",
targetLengthChars: 120,
rationale: "잘못된 직접 답변 계획",
}),
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "{}" } }] },
message: { role: "assistant" as const, content: "{}" },
});
}
return Promise.resolve({
content: "디스코드 닉네임은 확인됐고, 주민신청 메시지도 확인됐어요.",
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "확인됐어요." } }] },
message: { role: "assistant" as const, content: "확인됐어요." },
});
},
};
const verifyTool: Tool<Record<string, never>, { status: string }> = {
name: "verify-discord-nickname",
description: "디스코드 닉네임 확인",
argsSchema: z.object({}),
resultSchema: z.object({
status: z.string(),
}),
execute: () => {
executedTools.push("verify-discord-nickname");
return Promise.resolve({ status: "matched-renamed" });
},
};
const progressTool: Tool<
{ chatroomId: string; userId: string },
{ remainingSteps: string[] }
> = {
name: "check-onboarding-progress",
description: "진행 상태 확인",
argsSchema: z.object({
chatroomId: z.string(),
userId: z.string(),
}),
resultSchema: z.object({
remainingSteps: z.array(z.string()),
}),
execute: (args) => {
executedTools.push(`${args.chatroomId}:${args.userId}`);
return Promise.resolve({ remainingSteps: [] });
},
};
const orchestrator = new OnboardingOrchestrator(llmClientStub as never);
const result = await orchestrator.run(
createDerivedState({
allowShortResponse: false,
intent: { kind: "discord-verify-question" },
phase: { kind: "deadline-near", remaining: ["discord-profile-message"] },
responseMode: "llm-with-tools",
tier: 1,
}),
[
{ role: "system", content: "기본 시스템 프롬프트" },
{
role: "user",
content: "저 디코는 어떻게 들어가요? 주민신청까지 해뒀습니다",
},
],
{
request: {
requestId: "req-1",
skill: "onboarding",
chatroomId: "room-1",
userId: "user-1",
nickname: "프리리/95/여/D",
trigger: "user-message",
},
toolContext: {},
tools: [verifyTool, progressTool],
},
);
expect(result).toContain("주민신청");
expect(executedTools).toEqual([
"verify-discord-nickname",
"room-1:user-1",
]);
expect(calls[1]?.messages[0]?.content).toContain("[TOOL_PLAN_RESULTS]");
expect(calls[1]?.messages[0]?.content).toContain("주민신청 메시지 확인 상태");
expect(calls[1]?.messages[0]?.content).toContain(
"디스코드 닉네임 상태를 독립 항목처럼 길게 설명",
);
expect(calls[1]?.options?.tools).toBeUndefined();
});
it("does not expose verification tool again after partial discord tool-plan success", async () => {
const calls: Array<{
messages: ChatMessage[];
options: ChatCompletionOptions | undefined;
}> = [];
let verifyCallCount = 0;
const llmClientStub = {
chatCompletion: (
messages: ChatMessage[],
options?: ChatCompletionOptions,
) => {
calls.push({ messages, options });
if (calls.length === 1) {
return Promise.resolve({
content: JSON.stringify({
approach: "validate-then-explain",
targetPhase: "deadline-near",
mustInclude: [],
mustAvoid: [],
suggestedTools: ["verify-discord-nickname"],
tone: "friendly",
targetLengthChars: 120,
rationale: "닉네임 확인",
}),
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "{}" } }] },
message: { role: "assistant" as const, content: "{}" },
});
}
return Promise.resolve({
content: "디스코드 닉네임은 확인됐어요.",
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "확인됐어요." } }] },
message: { role: "assistant" as const, content: "확인됐어요." },
});
},
};
const verifyTool: Tool<Record<string, never>, { status: string }> = {
name: "verify-discord-nickname",
description: "디스코드 닉네임 확인",
argsSchema: z.object({}),
resultSchema: z.object({
status: z.string(),
}),
execute: () => {
verifyCallCount += 1;
return Promise.resolve({ status: "matched-renamed" });
},
};
const orchestrator = new OnboardingOrchestrator(llmClientStub as never);
const result = await orchestrator.run(
createDerivedState({
allowShortResponse: false,
intent: { kind: "discord-verify-question" },
phase: { kind: "deadline-near", remaining: ["discord-profile-nickname"] },
responseMode: "llm-with-tools",
tier: 1,
}),
[
{ role: "system", content: "기본 시스템 프롬프트" },
{
role: "user",
content: "디스코드 닉네임 변경했어요",
},
],
{
request: {
requestId: "req-1",
skill: "onboarding",
chatroomId: "room-1",
userId: "user-1",
nickname: "프리리/95/여/D",
trigger: "user-message",
},
toolContext: {},
tools: [verifyTool],
},
);
expect(result).toContain("닉네임");
expect(verifyCallCount).toBe(1);
expect(calls[1]?.messages[0]?.content).toContain("[TOOL_PLAN_RESULTS]");
expect(calls[1]?.messages[0]?.content).toContain("matched-renamed");
expect(calls[1]?.messages[0]?.content).toContain(
"Tool not registered for step: check-onboarding-progress",
);
expect(calls[1]?.options?.tools).toBeUndefined();
});
it("focuses discord nickname answers when only nickname verification was asked", async () => {
const calls: Array<{
messages: ChatMessage[];
options: ChatCompletionOptions | undefined;
}> = [];
const llmClientStub = {
chatCompletion: (
messages: ChatMessage[],
options?: ChatCompletionOptions,
) => {
calls.push({ messages, options });
if (calls.length === 1) {
return Promise.resolve({
content: JSON.stringify({
approach: "validate-then-explain",
targetPhase: "deadline-near",
mustInclude: [],
mustAvoid: [],
suggestedTools: [],
tone: "friendly",
targetLengthChars: 120,
rationale: "닉네임 확인",
}),
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "{}" } }] },
message: { role: "assistant" as const, content: "{}" },
});
}
return Promise.resolve({
content: "디스코드 닉네임은 확인됐어요.",
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "확인됐어요." } }] },
message: { role: "assistant" as const, content: "확인됐어요." },
});
},
};
const verifyTool: Tool<Record<string, never>, { status: string }> = {
name: "verify-discord-nickname",
description: "디스코드 닉네임 확인",
argsSchema: z.object({}),
resultSchema: z.object({
status: z.string(),
}),
execute: () => Promise.resolve({ status: "matched-renamed" }),
};
const progressTool: Tool<
{ chatroomId: string; userId: string },
{ remainingSteps: string[] }
> = {
name: "check-onboarding-progress",
description: "진행 상태 확인",
argsSchema: z.object({
chatroomId: z.string(),
userId: z.string(),
}),
resultSchema: z.object({
remainingSteps: z.array(z.string()),
}),
execute: () => Promise.resolve({ remainingSteps: [] }),
};
const orchestrator = new OnboardingOrchestrator(llmClientStub as never);
const result = await orchestrator.run(
createDerivedState({
allowShortResponse: false,
intent: { kind: "discord-verify-question" },
phase: { kind: "deadline-near", remaining: ["discord-profile-nickname"] },
responseMode: "llm-with-tools",
tier: 1,
}),
[
{ role: "system", content: "기본 시스템 프롬프트" },
{
role: "user",
content: "디스코드 닉네임 변경했어요",
},
],
{
request: {
requestId: "req-1",
skill: "onboarding",
chatroomId: "room-1",
userId: "user-1",
nickname: "프리리/95/여/D",
trigger: "user-message",
},
toolContext: {},
tools: [verifyTool, progressTool],
},
);
expect(result).toContain("닉네임");
expect(calls[1]?.messages[0]?.content).toContain("디스코드 닉네임 조회 결과");
expect(calls[1]?.messages[0]?.content).toContain(
"주민신청 메시지 상태 언급",
);
});
it("injects requested SKILL.md resources before responder when enabled", async () => { it("injects requested SKILL.md resources before responder when enabled", async () => {
process.env["AGENT_SKILL_MD_MODE"] = "true"; process.env["AGENT_SKILL_MD_MODE"] = "true";
const calls: Array<{ const calls: Array<{
+232 -24
View File
@@ -10,11 +10,8 @@ import {
getPlannerMaxTokens, getPlannerMaxTokens,
getTier0EstimatedCallMs, getTier0EstimatedCallMs,
getTier0MaxTokens, getTier0MaxTokens,
getTier0Model,
getTier1EstimatedCallMs, getTier1EstimatedCallMs,
getTier1MaxTokens, getTier1MaxTokens,
getTier1PlannerModel,
getTier1ResponderModel,
getTier2EstimatedCallMs, getTier2EstimatedCallMs,
getTier2MaxTokens, getTier2MaxTokens,
} from "../../../config.js"; } from "../../../config.js";
@@ -75,6 +72,48 @@ function summarizeMessages(messages: ChatMessage[]): string {
.join("\n\n"); .join("\n\n");
} }
type DiscordVerificationFocus =
| "both"
| "nickname"
| "resident-application-message";
function getLatestUserMessage(messages: ChatMessage[]): string {
for (let index = messages.length - 1; index >= 0; index -= 1) {
const message = messages[index];
if (message?.role === "user") {
return message.content;
}
}
return "";
}
function getDiscordVerificationFocus(
derived: DerivedState,
messages: ChatMessage[],
): DiscordVerificationFocus {
if (derived.intent.kind !== "discord-verify-question") {
return "both";
}
const latestUserMessage = getLatestUserMessage(messages).toLowerCase();
const asksResidentApplication =
/(주민\s*신청|주민신청|주민\s*등록|주민등록|프로필\s*메시지|채널\s*메시지|메시지\s*(작성|남김|보냄|썼|완료|했습니다|했어요))/u.test(
latestUserMessage,
);
const asksNickname =
/(닉네임|닉변|서버\s*프로필|서버프로필|프로필\s*닉네임|서버\s*표시명|서버표시명)/u.test(
latestUserMessage,
);
if (asksResidentApplication && !asksNickname) {
return "resident-application-message";
}
if (asksNickname && !asksResidentApplication) {
return "nickname";
}
return "both";
}
function tryParsePlannerOutput(raw: string): PlannerOutput | null { function tryParsePlannerOutput(raw: string): PlannerOutput | null {
try { try {
const parsed = JSON.parse(raw) as unknown; const parsed = JSON.parse(raw) as unknown;
@@ -95,7 +134,24 @@ function tryParsePlannerOutput(raw: string): PlannerOutput | null {
} }
} }
function getDefaultPlan(derived: DerivedState): PlannerOutput { function getDefaultPlan(
derived: DerivedState,
focus: DiscordVerificationFocus = "both",
): PlannerOutput {
if (derived.intent.kind === "discord-verify-question") {
return enforceRequiredToolPlan(derived, {
approach: "validate-then-explain",
targetPhase: derived.phase.kind,
mustInclude: getDiscordVerificationMustInclude(focus),
mustAvoid: getDiscordVerificationMustAvoid(focus),
requestResources: [],
suggestedTools: [],
tone: "friendly",
targetLengthChars: 150,
rationale: "discord-verification-fallback-plan",
}, focus);
}
return { return {
approach: approach:
derived.responseMode === "llm-with-tools" derived.responseMode === "llm-with-tools"
@@ -115,6 +171,104 @@ function getDefaultPlan(derived: DerivedState): PlannerOutput {
}; };
} }
function hasToolPlanStep(plan: PlannerOutput, toolName: string): boolean {
return plan.toolPlan?.steps.some((step) => step.tool === toolName) ?? false;
}
function getDiscordVerificationMustInclude(
focus: DiscordVerificationFocus,
): string[] {
if (focus === "nickname") {
return ["디스코드 닉네임 조회 결과"];
}
if (focus === "resident-application-message") {
return [
"주민신청 메시지 확인 상태",
"메시지 형식이 틀리면 서버프로필 닉네임과 동일하게 다시 작성하라는 안내",
];
}
return [
"디스코드 닉네임 조회 결과",
"주민신청 메시지 확인 상태",
];
}
function getDiscordVerificationMustAvoid(
focus: DiscordVerificationFocus,
): string[] {
const base = ["확인 없이 완료 단정", "초대 안내만 반복"];
if (focus === "nickname") {
return [...base, "주민신청 메시지 상태 언급"];
}
if (focus === "resident-application-message") {
return [
...base,
"디스코드 닉네임 상태를 독립 항목처럼 길게 설명",
"묻지 않은 남은 절차 나열",
];
}
return base;
}
function enforceRequiredToolPlan(
derived: DerivedState,
plan: PlannerOutput,
focus: DiscordVerificationFocus = "both",
): PlannerOutput {
if (derived.intent.kind !== "discord-verify-question") {
return plan;
}
const suggestedTools = [
...new Set([
...plan.suggestedTools,
"verify-discord-nickname" as const,
"check-onboarding-progress" as const,
]),
];
const steps = [...(plan.toolPlan?.steps ?? [])];
if (!hasToolPlanStep(plan, "verify-discord-nickname")) {
steps.unshift({
id: "verify-discord",
tool: "verify-discord-nickname",
on: { error: "fallback-llm" },
});
}
if (!hasToolPlanStep(plan, "check-onboarding-progress")) {
steps.push({
id: "check-progress",
tool: "check-onboarding-progress",
argsFromContext: {
chatroomId: "$request.chatroomId",
userId: "$request.userId",
},
on: { error: "fallback-llm" },
});
}
return {
...plan,
approach: "validate-then-explain",
mustAvoid: [
...new Set([
...plan.mustAvoid,
...getDiscordVerificationMustAvoid(focus),
]),
].slice(0, 6),
mustInclude: [
...new Set([
...plan.mustInclude,
...getDiscordVerificationMustInclude(focus),
]),
].slice(0, 6),
suggestedTools: suggestedTools.slice(0, 3),
toolPlan: {
composeWith: "all-then-llm",
steps: steps.slice(0, 4),
},
};
}
function getRemainingDeadlineMs(): number | undefined { function getRemainingDeadlineMs(): number | undefined {
const deadlineAt = getContext()?.deadlineAt; const deadlineAt = getContext()?.deadlineAt;
if (!deadlineAt) { if (!deadlineAt) {
@@ -224,6 +378,49 @@ function buildToolPlanAddendum(result: ToolPlanResult): string {
].join("\n"); ].join("\n");
} }
function hasToolPlanResultForTool(
plan: PlannerOutput,
result: ToolPlanResult,
toolName: string,
): boolean {
return (
plan.toolPlan?.steps.some(
(step) =>
step.tool === toolName &&
Object.hasOwn(result.results, step.id),
) ?? false
);
}
function shouldInjectToolPlanResult(
derived: DerivedState,
result: ToolPlanResult,
): boolean {
if (!result.fallbackToLlm) {
return true;
}
return (
derived.intent.kind === "discord-verify-question" &&
Object.keys(result.results).length > 0
);
}
function shouldSuppressResponderTools(
derived: DerivedState,
plan: PlannerOutput,
result: ToolPlanResult,
): boolean {
if (!result.fallbackToLlm) {
return true;
}
return (
derived.intent.kind === "discord-verify-question" &&
hasToolPlanResultForTool(plan, result, "verify-discord-nickname")
);
}
function pickToolsForPlan( function pickToolsForPlan(
plan: PlannerOutput, plan: PlannerOutput,
tools: readonly Tool<unknown, unknown>[], tools: readonly Tool<unknown, unknown>[],
@@ -254,7 +451,6 @@ export class OnboardingOrchestrator {
options: OrchestratorRunOptions, options: OrchestratorRunOptions,
responderTools: readonly Tool<unknown, unknown>[], responderTools: readonly Tool<unknown, unknown>[],
maxTokens: number, maxTokens: number,
modelOverride: string,
): Promise<string | null> { ): Promise<string | null> {
const logger = getRequestLogger(); const logger = getRequestLogger();
const deadlineMs = getRemainingDeadlineMs(); const deadlineMs = getRemainingDeadlineMs();
@@ -275,7 +471,6 @@ export class OnboardingOrchestrator {
llmClient: this.llmClient, llmClient: this.llmClient,
maxTokens, maxTokens,
messages, messages,
modelOverride,
role: "responder", role: "responder",
routingDecision: derived.routingDecision, routingDecision: derived.routingDecision,
tier: derived.tier, tier: derived.tier,
@@ -288,7 +483,6 @@ export class OnboardingOrchestrator {
const response = await this.llmClient.chatCompletion(messages, { const response = await this.llmClient.chatCompletion(messages, {
deadlineMs, deadlineMs,
maxTokens, maxTokens,
modelOverride,
role: "responder", role: "responder",
routingDecision: derived.routingDecision, routingDecision: derived.routingDecision,
temperature: options.temperature ?? getLlmTemperature(), temperature: options.temperature ?? getLlmTemperature(),
@@ -304,24 +498,29 @@ export class OnboardingOrchestrator {
): Promise<PlannerOutput> { ): Promise<PlannerOutput> {
const logger = getRequestLogger(); const logger = getRequestLogger();
const traceStep = getActiveProcessTrace()?.startStep("plan"); const traceStep = getActiveProcessTrace()?.startStep("plan");
const verificationFocus = getDiscordVerificationFocus(derived, messages);
const finishTrace = (plan: PlannerOutput): PlannerOutput => { const finishTrace = (plan: PlannerOutput): PlannerOutput => {
const normalizedPlan = enforceRequiredToolPlan(
derived,
plan,
verificationFocus,
);
traceStep?.finish({ traceStep?.finish({
approach: plan.approach, approach: normalizedPlan.approach,
mustAvoid: plan.mustAvoid, mustAvoid: normalizedPlan.mustAvoid,
mustInclude: plan.mustInclude, mustInclude: normalizedPlan.mustInclude,
requestResources: plan.requestResources, requestResources: normalizedPlan.requestResources,
suggestedTools: plan.suggestedTools, suggestedTools: normalizedPlan.suggestedTools,
targetLengthChars: plan.targetLengthChars, targetLengthChars: normalizedPlan.targetLengthChars,
toolPlanStepCount: plan.toolPlan?.steps.length, toolPlanStepCount: normalizedPlan.toolPlan?.steps.length,
}); });
return plan; return normalizedPlan;
}; };
const plannerMessages = buildPlannerMessages(derived, messages); const plannerMessages = buildPlannerMessages(derived, messages);
const firstAttempt = await this.llmClient.chatCompletion(plannerMessages, { const firstAttempt = await this.llmClient.chatCompletion(plannerMessages, {
deadlineMs: getRemainingDeadlineMs(), deadlineMs: getRemainingDeadlineMs(),
jsonMode: true, jsonMode: true,
maxTokens: getPlannerMaxTokens(), maxTokens: getPlannerMaxTokens(),
modelOverride: getTier1PlannerModel(),
role: "planner", role: "planner",
routingDecision: derived.routingDecision, routingDecision: derived.routingDecision,
temperature: 0, temperature: 0,
@@ -356,7 +555,6 @@ export class OnboardingOrchestrator {
deadlineMs: getRemainingDeadlineMs(), deadlineMs: getRemainingDeadlineMs(),
jsonMode: true, jsonMode: true,
maxTokens: getPlannerMaxTokens(), maxTokens: getPlannerMaxTokens(),
modelOverride: getTier1PlannerModel(),
role: "planner", role: "planner",
routingDecision: derived.routingDecision, routingDecision: derived.routingDecision,
temperature: 0, temperature: 0,
@@ -376,7 +574,7 @@ export class OnboardingOrchestrator {
return finishTrace(retryPlan); return finishTrace(retryPlan);
} }
const fallbackPlan = getDefaultPlan(derived); const fallbackPlan = getDefaultPlan(derived, verificationFocus);
logger.debug( logger.debug(
{ {
intent: derived.intent.kind, intent: derived.intent.kind,
@@ -403,7 +601,6 @@ export class OnboardingOrchestrator {
options, options,
tools, tools,
maxTokens, maxTokens,
getTier0Model(),
); );
} }
@@ -428,16 +625,19 @@ export class OnboardingOrchestrator {
); );
} }
const toolPlanResult = await this.runProgrammaticToolPlan( const toolPlanResult = await this.runProgrammaticToolPlan(
derived,
plan, plan,
finalResponderMessages, finalResponderMessages,
options, options,
); );
if (toolPlanResult && !toolPlanResult.fallbackToLlm) { if (toolPlanResult && shouldInjectToolPlanResult(derived, toolPlanResult)) {
finalResponderMessages = injectSystemAddendum( finalResponderMessages = injectSystemAddendum(
finalResponderMessages, finalResponderMessages,
buildToolPlanAddendum(toolPlanResult), buildToolPlanAddendum(toolPlanResult),
); );
}
if (toolPlanResult && shouldSuppressResponderTools(derived, plan, toolPlanResult)) {
responderTools = []; responderTools = [];
} }
@@ -447,7 +647,6 @@ export class OnboardingOrchestrator {
options, options,
responderTools, responderTools,
options.maxTokens ?? getTier1MaxTokens(), options.maxTokens ?? getTier1MaxTokens(),
getTier1ResponderModel(),
); );
} }
@@ -472,16 +671,19 @@ export class OnboardingOrchestrator {
); );
} }
const toolPlanResult = await this.runProgrammaticToolPlan( const toolPlanResult = await this.runProgrammaticToolPlan(
derived,
plan, plan,
finalResponderMessages, finalResponderMessages,
options, options,
); );
if (toolPlanResult && !toolPlanResult.fallbackToLlm) { if (toolPlanResult && shouldInjectToolPlanResult(derived, toolPlanResult)) {
finalResponderMessages = injectSystemAddendum( finalResponderMessages = injectSystemAddendum(
finalResponderMessages, finalResponderMessages,
buildToolPlanAddendum(toolPlanResult), buildToolPlanAddendum(toolPlanResult),
); );
}
if (toolPlanResult && shouldSuppressResponderTools(derived, plan, toolPlanResult)) {
responderTools = []; responderTools = [];
} }
@@ -491,7 +693,6 @@ export class OnboardingOrchestrator {
options, options,
responderTools, responderTools,
options.maxTokens ?? getTier2MaxTokens(), options.maxTokens ?? getTier2MaxTokens(),
getTier1ResponderModel(),
); );
if (!draft) { if (!draft) {
@@ -517,12 +718,19 @@ export class OnboardingOrchestrator {
} }
private async runProgrammaticToolPlan( private async runProgrammaticToolPlan(
derived: DerivedState,
plan: PlannerOutput, plan: PlannerOutput,
_messages: ChatMessage[], _messages: ChatMessage[],
options: OrchestratorRunOptions, options: OrchestratorRunOptions,
): Promise<ToolPlanResult | null> { ): Promise<ToolPlanResult | null> {
if ( if (
!getAgentProgrammaticToolCalling() || !getAgentProgrammaticToolCalling() &&
derived.intent.kind !== "discord-verify-question"
) {
return null;
}
if (
!plan.toolPlan || !plan.toolPlan ||
!options.request || !options.request ||
!options.tools?.length !options.tools?.length
+45 -2
View File
@@ -102,9 +102,9 @@ describe("decidePreGenerationRouting", () => {
}, },
{ {
confidence: 0.3, confidence: 0.3,
expected: { kind: "tier0-only" }, expected: { kind: "tier1-direct", skipPlanner: false },
intentKind: "off-topic-to-other" as const, intentKind: "off-topic-to-other" as const,
name: "keeps low-confidence off-topic on tier0", name: "promotes low-confidence off-topic to tier1",
phase: { kind: "fresh" } as OnboardingPhase, phase: { kind: "fresh" } as OnboardingPhase,
}, },
{ {
@@ -319,6 +319,49 @@ describe("decidePreGenerationRouting", () => {
).toEqual(expected); ).toEqual(expected);
}); });
it("keeps high-confidence low-risk requests on tier0", () => {
expect(
decidePreGenerationRouting({
classified: classified("age-question", 0.95),
derived: derived({ kind: "deadline-near", remaining: ["photo"] }, intent("age-question"), {
tier: 1,
}),
intentRiskTier: "low",
request: request(),
session: session(),
userEscalationCount: 0,
}),
).toEqual({ kind: "tier0-only" });
});
it("promotes low-confidence high-risk requests with escalation history", () => {
expect(
decidePreGenerationRouting({
classified: classified("unknown", 0.49),
derived: derived({ kind: "fresh" }, intent("unknown")),
intentRiskTier: "high",
request: request(),
session: session(),
userEscalationCount: 4,
}),
).toEqual({ kind: "tier1-direct", skipPlanner: false });
});
it("prefers tier0 during latency degradation unless safety already promoted", () => {
expect(
decidePreGenerationRouting({
classified: classified("age-question", 0.8),
derived: derived({ kind: "deadline-near", remaining: ["nickname"] }, intent("age-question"), {
tier: 1,
}),
intentRiskTier: "medium",
recentLatencyP95Ms: 10_000,
request: request(),
session: session(),
}),
).toEqual({ kind: "tier0-only" });
});
it("trains, saves, loads, and applies a learned router model", () => { it("trains, saves, loads, and applies a learned router model", () => {
const lowConfidenceInput = { const lowConfidenceInput = {
classified: classified("age-question", 0.25), classified: classified("age-question", 0.25),
+49
View File
@@ -3,6 +3,10 @@ import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname } from "node:path"; import { dirname } from "node:path";
import type { AgentRequest } from "@agent/types"; import type { AgentRequest } from "@agent/types";
import {
getAgentCostAwareTierEnabled,
getAlertThresholdP95LatencyMs,
} from "../../../config.js";
import type { import type {
ClassifiedIntent, ClassifiedIntent,
DerivedState, DerivedState,
@@ -58,8 +62,11 @@ export interface RouterTrainingSample {
export interface RouterInput { export interface RouterInput {
classified: ClassifiedIntent; classified: ClassifiedIntent;
derived: DerivedState; derived: DerivedState;
intentRiskTier?: "low" | "medium" | "high" | undefined;
recentLatencyP95Ms?: number | undefined;
request: AgentRequest; request: AgentRequest;
session: OnboardingSession | null; session: OnboardingSession | null;
userEscalationCount?: number | undefined;
} }
export const ROUTER_FEATURE_NAMES = [ export const ROUTER_FEATURE_NAMES = [
@@ -294,6 +301,43 @@ function needsStepOneHelp(phase: OnboardingPhase, intent: Intent): boolean {
); );
} }
function isLatencyDegraded(input: RouterInput): boolean {
const p95 = input.recentLatencyP95Ms;
return (
typeof p95 === "number" &&
Number.isFinite(p95) &&
p95 > getAlertThresholdP95LatencyMs("onboarding")
);
}
function maybeDecideCostAwareRouting(input: RouterInput): RoutingDecision | null {
if (!getAgentCostAwareTierEnabled()) {
return null;
}
const confidence = clamp01(input.classified.confidence);
const riskTier = input.intentRiskTier ?? "medium";
const escalationCount = Math.max(0, input.userEscalationCount ?? 0);
const lowLatency = !isLatencyDegraded(input);
if (confidence < 0.5 || (riskTier === "high" && escalationCount > 3)) {
if (input.derived.responseMode === "llm-with-tools") {
return { kind: "tier1-with-tools" };
}
return { kind: "tier1-direct", skipPlanner: false };
}
if (!lowLatency) {
return { kind: "tier0-only" };
}
if (confidence >= 0.9 && riskTier === "low") {
return { kind: "tier0-only" };
}
return null;
}
export function decidePreGenerationRouting(input: RouterInput): RoutingDecision { export function decidePreGenerationRouting(input: RouterInput): RoutingDecision {
const { classified, derived, request, session } = input; const { classified, derived, request, session } = input;
const { confidence, intent } = classified; const { confidence, intent } = classified;
@@ -334,6 +378,11 @@ export function decidePreGenerationRouting(input: RouterInput): RoutingDecision
}; };
} }
const costAwareDecision = maybeDecideCostAwareRouting(input);
if (costAwareDecision) {
return costAwareDecision;
}
if ( if (
phase.kind === "nickname-rejected" || phase.kind === "nickname-rejected" ||
intent.kind === "discord-verify-question" || intent.kind === "discord-verify-question" ||
@@ -142,4 +142,71 @@ describe("runToolPlan", () => {
toolName: "lookup-community-rule", toolName: "lookup-community-rule",
}); });
}); });
it("appends audit entries for successful tool calls", async () => {
const streams: Array<{
fields: Record<string, string>;
key: string;
options?: { maxLen?: number | undefined };
}> = [];
const tool: Tool<{ topic: string }, { text: string }> = {
name: "lookup-community-rule",
description: "규칙 조회",
requires: [],
argsSchema: z.object({
topic: z.string(),
}),
resultSchema: z.object({
text: z.string(),
}),
execute: () => Promise.resolve({ text: "하트 규칙" }),
};
await runToolPlan(
{
composeWith: "all-then-llm",
steps: [
{
id: "rule",
tool: "lookup-community-rule",
argsLiteral: {
topic: "heart",
},
},
],
},
[tool],
{
garnetClient: {
appendStream: (
key: string,
fields: Record<string, string>,
options?: { maxLen?: number | undefined },
) => {
streams.push({
fields,
key,
...(options ? { options } : {}),
});
return Promise.resolve("1-0");
},
} as never,
},
request(),
);
expect(streams[0]).toMatchObject({
key: "lolympus-agent:tool-calls",
fields: {
ok: "true",
skill: "onboarding",
toolName: "lookup-community-rule",
type: "tool-call",
},
options: {
maxLen: 100_000,
},
});
expect(streams[0]?.fields["argsHash"]).toEqual(expect.any(String));
});
}); });
@@ -1,5 +1,8 @@
import type { AgentRequest } from "@agent/types"; import type { AgentRequest } from "@agent/types";
import { getRequestLogger } from "@obs/context";
import { withToolSpan } from "@obs/spans"; import { withToolSpan } from "@obs/spans";
import { hashToolArgs, ToolAuditLogger } from "@tools/audit";
import { buildScopedContext } from "@tools/scoped-context";
import type { import type {
Tool, Tool,
ToolContext, ToolContext,
@@ -208,6 +211,41 @@ function makePlanFailure(message: string, details?: unknown): ToolExecutionFailu
}); });
} }
async function appendToolAudit(input: {
args: unknown;
ctx: ToolContext;
durationMs: number;
errorKind?: string | undefined;
ok: boolean;
request: AgentRequest;
resultSummary: string;
toolName: string;
}): Promise<void> {
if (!input.ctx.garnetClient) {
return;
}
try {
await new ToolAuditLogger(input.ctx.garnetClient).append({
type: "tool-call",
argsHash: hashToolArgs(input.args),
calledAt: new Date().toISOString(),
chatroomId: input.request.chatroomId,
durationMs: input.durationMs,
...(input.errorKind ? { errorKind: input.errorKind } : {}),
ok: input.ok,
resultSummary: input.resultSummary,
skill: input.request.skill,
toolName: input.toolName,
userId: input.request.userId,
});
} catch (error) {
getRequestLogger().warn(
{ err: error, toolName: input.toolName },
"[ToolAudit] Failed to append tool audit entry",
);
}
}
function orderPlan(plan: ToolPlan): OrderedPlan { function orderPlan(plan: ToolPlan): OrderedPlan {
const failures: ToolExecutionFailure[] = []; const failures: ToolExecutionFailure[] = [];
const byId = new Map<string, ToolPlanStep>(); const byId = new Map<string, ToolPlanStep>();
@@ -278,19 +316,31 @@ async function runStep(input: {
const traceStep = getActiveProcessTrace()?.startStep("tool-call"); const traceStep = getActiveProcessTrace()?.startStep("tool-call");
let args: Record<string, unknown> = {}; let args: Record<string, unknown> = {};
const startedAt = Date.now();
try { try {
args = buildArgs(input); args = buildArgs(input);
const validated = tool.argsSchema.parse(args); const validated = tool.argsSchema.parse(args);
const scopedContext = buildScopedContext(tool.requires, input.ctx, tool.name);
const result = await withToolSpan( const result = await withToolSpan(
tool.name, tool.name,
{ {
"tool.name": tool.name, "tool.name": tool.name,
"tool.plan.step_id": input.step.id, "tool.plan.step_id": input.step.id,
}, },
() => tool.execute(validated, input.ctx), () => tool.execute(validated, scopedContext),
); );
const structuredFailure = getStructuredFailure(input.step, result); const structuredFailure = getStructuredFailure(input.step, result);
if (structuredFailure) { if (structuredFailure) {
await appendToolAudit({
args,
ctx: input.ctx,
durationMs: Math.max(0, Date.now() - startedAt),
errorKind: structuredFailure.kind,
ok: false,
request: input.request,
resultSummary: structuredFailure.message,
toolName: tool.name,
});
traceStep?.finish({ traceStep?.finish({
args, args,
errorKind: structuredFailure.kind, errorKind: structuredFailure.kind,
@@ -302,15 +352,35 @@ async function runStep(input: {
} }
input.results[input.step.id] = result; input.results[input.step.id] = result;
const resultSummary = summarizeToolResult(result);
await appendToolAudit({
args,
ctx: input.ctx,
durationMs: Math.max(0, Date.now() - startedAt),
ok: true,
request: input.request,
resultSummary,
toolName: tool.name,
});
traceStep?.finish({ traceStep?.finish({
args, args,
name: tool.name, name: tool.name,
ok: true, ok: true,
resultSummary: summarizeToolResult(result), resultSummary,
}); });
return null; return null;
} catch (error) { } catch (error) {
const failure = classifyError(tool.name, error); const failure = classifyError(tool.name, error);
await appendToolAudit({
args,
ctx: input.ctx,
durationMs: Math.max(0, Date.now() - startedAt),
errorKind: failure.kind,
ok: false,
request: input.request,
resultSummary: failure.message,
toolName: tool.name,
});
traceStep?.finish({ traceStep?.finish({
args, args,
errorKind: failure.kind, errorKind: failure.kind,
@@ -26,7 +26,11 @@ export function selectToolsForDerived(
if (derived.intent.kind === 'discord-verify-question') { if (derived.intent.kind === 'discord-verify-question') {
return maybeUseToolSearch( return maybeUseToolSearch(
tools.filter((tool) => tool.name === 'verify-discord-nickname'), tools.filter(
(tool) =>
tool.name === 'verify-discord-nickname' ||
tool.name === 'check-onboarding-progress',
),
); );
} }
+26
View File
@@ -1,6 +1,7 @@
import type { AgentRequest, AgentResponse, ConversationMessage } from '@agent/types'; import type { AgentRequest, AgentResponse, ConversationMessage } from '@agent/types';
import type { GarnetClient } from '@common/garnet'; import type { GarnetClient } from '@common/garnet';
import type { NatsClient } from '@common/nats'; import type { NatsClient } from '@common/nats';
import type { EscalationHistoryService } from '../../memory/escalation-history.js';
import type { MemoryService } from '../../memory/memory-types.js'; import type { MemoryService } from '../../memory/memory-types.js';
import { import {
buildLongTermSummary, buildLongTermSummary,
@@ -118,6 +119,7 @@ export class OnboardingSkill implements Skill {
shadowEvalPublisher?: ShadowEvalPublisher, shadowEvalPublisher?: ShadowEvalPublisher,
private readonly natsClient?: Pick<NatsClient, 'request'>, private readonly natsClient?: Pick<NatsClient, 'request'>,
private readonly memoryService?: MemoryService, private readonly memoryService?: MemoryService,
private readonly escalationHistoryService?: EscalationHistoryService,
) { ) {
this.classifier = new OnboardingIntentClassifier(llmClient); this.classifier = new OnboardingIntentClassifier(llmClient);
this.blackboardStore = new OnboardingBlackboardStore( this.blackboardStore = new OnboardingBlackboardStore(
@@ -186,13 +188,17 @@ export class OnboardingSkill implements Skill {
const deriveStep = trace?.startStep('derive'); const deriveStep = trace?.startStep('derive');
const derived = deriveState(session, request.trigger, classified.intent); const derived = deriveState(session, request.trigger, classified.intent);
const escalationHistory = await this.loadEscalationHistory(request.userId);
const routerMode = getAgentRouterMode(); const routerMode = getAgentRouterMode();
if (routerMode === 'static' || routerMode === 'learned-v1') { if (routerMode === 'static' || routerMode === 'learned-v1') {
const routerInput = { const routerInput = {
classified, classified,
derived, derived,
intentRiskTier: this.capabilityCard().riskTier,
recentLatencyP95Ms: this.getRecentLatencyP95Ms(request),
request, request,
session, session,
userEscalationCount: escalationHistory?.totalEscalations ?? 0,
}; };
derived.routingDecision = derived.routingDecision =
routerMode === 'learned-v1' routerMode === 'learned-v1'
@@ -353,6 +359,26 @@ export class OnboardingSkill implements Skill {
return normalizeOnboardingSession(rawSession); return normalizeOnboardingSession(rawSession);
} }
private async loadEscalationHistory(userId: string) {
if (!this.escalationHistoryService) {
return null;
}
try {
return await this.escalationHistoryService.load(userId);
} catch (error) {
getRequestLogger().warn(
{ err: error },
'[OnboardingSkill] Failed to load escalation history',
);
return null;
}
}
private getRecentLatencyP95Ms(request: AgentRequest): number | undefined {
const raw = request.context?.['recentLatencyP95Ms'];
return typeof raw === 'number' && Number.isFinite(raw) ? raw : undefined;
}
private async decideEarlyAbstention( private async decideEarlyAbstention(
request: AgentRequest, request: AgentRequest,
classified: ClassifiedIntent, classified: ClassifiedIntent,
@@ -52,6 +52,15 @@ describe("OnboardingIntentClassifier", () => {
expect(result.source).toBe("regex"); expect(result.source).toBe("regex");
}); });
it("classifies resident application completion reports as discord verification", async () => {
const result = await classifier.classify(
"저 디코는 어떻게 들어가요? 주민신청까지 해뒀습니다",
"user-message",
);
expect(result.intent.kind).toBe("discord-verify-question");
expect(result.source).toBe("regex");
});
it("classifies identity room questions as help-how", async () => { it("classifies identity room questions as help-how", async () => {
const result = await classifier.classify( const result = await classifier.classify(
"본인인증 방은 뭐에요?", "본인인증 방은 뭐에요?",
+43 -7
View File
@@ -1,17 +1,53 @@
import type { HandoffPacket } from "@agent/handoff-packet"; import type { HandoffPacket } from "@agent/handoff-packet";
export function routeSupervisorTarget(packet: HandoffPacket | undefined): string | null { export interface SupervisorRuleRoute {
const text = [ confidence: number;
target: string | null;
}
const RULES = [
{
pattern: /(규칙|신고|제재|경고|욕설|광고|홍보|appeal|policy|moderation)/iu,
target: "moderation",
},
{
pattern: /(가입|닉네임|하트|인증|디스코드|온보딩|onboarding)/iu,
target: "onboarding",
},
] as const;
function packetText(packet: HandoffPacket | undefined): string {
return [
packet?.userGoal, packet?.userGoal,
packet?.summary, packet?.summary,
packet?.handoffReason, packet?.handoffReason,
].join(" "); ].join(" ");
}
if (/(규칙|신고|제재|경고|욕설|광고|홍보|appeal|policy|moderation)/iu.test(text)) { export function routeSupervisorByRules(
return "moderation"; packet: HandoffPacket | undefined,
): SupervisorRuleRoute {
const text = packetText(packet);
const matches = RULES.filter((rule) => rule.pattern.test(text));
if (matches.length === 1) {
return {
confidence: 0.9,
target: matches[0]?.target ?? null,
};
} }
if (/(가입|닉네임|하트|인증|디스코드|온보딩|onboarding)/iu.test(text)) { if (matches.length > 1) {
return "onboarding"; return {
confidence: 0.6,
target: matches[0]?.target ?? null,
};
} }
return null; return {
confidence: 0,
target: null,
};
}
export function routeSupervisorTarget(packet: HandoffPacket | undefined): string | null {
return routeSupervisorByRules(packet).target;
} }
+41 -1
View File
@@ -119,7 +119,7 @@ describe("SupervisorSkill", () => {
expect(response.text).toContain("운영자"); expect(response.text).toContain("운영자");
expect(response.metadata).toMatchObject({ expect(response.metadata).toMatchObject({
routedBy: "rules", routedBy: "regex-fast-path",
source: "template", source: "template",
supervisorReason: "max-hop", supervisorReason: "max-hop",
suggestedTarget: "onboarding", suggestedTarget: "onboarding",
@@ -149,6 +149,7 @@ describe("SupervisorSkill", () => {
}); });
it("uses LLM routing when the decision is confident and valid", async () => { it("uses LLM routing when the decision is confident and valid", async () => {
vi.stubEnv("AGENT_SUPERVISOR_FAST_PATH_ENABLED", "false");
const garnet = new FakeGarnet(); const garnet = new FakeGarnet();
const nats = new FakeNats(); const nats = new FakeNats();
const skill = new SupervisorSkill( const skill = new SupervisorSkill(
@@ -189,6 +190,45 @@ describe("SupervisorSkill", () => {
}); });
}); });
it("uses regex fast-path before calling the supervisor LLM", async () => {
const garnet = new FakeGarnet();
const nats = new FakeNats();
const llm = {
chatCompletion: vi.fn(),
};
const skill = new SupervisorSkill(
garnet,
nats,
llm as unknown as Pick<LlmClient, "chatCompletion">,
registry,
);
const packet = buildHandoffPacket({
handoff: {
confidence: 0.7,
evidenceRefs: [],
expectedOutput: "text",
reason: "out-of-scope",
risk: "medium",
targetAgent: "supervisor",
taskSpec: "광고 규칙 문의",
},
request: baseRequest,
response: sourceResponse,
sourceChain: ["onboarding", "supervisor"],
});
const response = await skill.handle(
{ ...baseRequest, context: { handoffPacket: packet } },
[],
);
expect(llm.chatCompletion).not.toHaveBeenCalled();
expect(response.metadata).toMatchObject({
routedBy: "regex-fast-path",
suggestedTarget: "moderation",
});
});
it("uses safe LLM userMessage for supervisor replies", async () => { it("uses safe LLM userMessage for supervisor replies", async () => {
const garnet = new FakeGarnet(); const garnet = new FakeGarnet();
const nats = new FakeNats(); const nats = new FakeNats();
+29 -5
View File
@@ -3,6 +3,7 @@ import type {
AgentResponse, AgentResponse,
ConversationMessage, ConversationMessage,
} from "@agent/types"; } from "@agent/types";
import { trace } from "@opentelemetry/api";
import { readHandoffPacket } from "@agent/handoff-packet"; import { readHandoffPacket } from "@agent/handoff-packet";
import type { HandoffPacket } from "@agent/handoff-packet"; import type { HandoffPacket } from "@agent/handoff-packet";
import type { GarnetClient } from "@common/garnet"; import type { GarnetClient } from "@common/garnet";
@@ -19,6 +20,8 @@ import type { Skill } from "@skills/skill";
import type { SkillCapabilityCard, SkillRegistry } from "@skills/skill-loader"; import type { SkillCapabilityCard, SkillRegistry } from "@skills/skill-loader";
import { getLogger } from "@obs/logger"; import { getLogger } from "@obs/logger";
import { import {
getAgentSupervisorFastPathEnabled,
getSupervisorFastPathConfidence,
getSupervisorLlmConfidenceThreshold, getSupervisorLlmConfidenceThreshold,
getSupervisorLlmEnabled, getSupervisorLlmEnabled,
getSupervisorLlmResponseEnabled, getSupervisorLlmResponseEnabled,
@@ -28,7 +31,7 @@ import {
buildSupervisorUserPrompt, buildSupervisorUserPrompt,
} from "./capability-prompt.js"; } from "./capability-prompt.js";
import { appendSupervisorReview } from "./review-queue.js"; import { appendSupervisorReview } from "./review-queue.js";
import { routeSupervisorTarget } from "./routing-rules.js"; import { routeSupervisorByRules } from "./routing-rules.js";
import { import {
supervisorRoutingSchema, supervisorRoutingSchema,
type SupervisorRoutingDecision, type SupervisorRoutingDecision,
@@ -114,13 +117,30 @@ export class SupervisorSkill implements Skill {
): Promise<AgentResponse> { ): Promise<AgentResponse> {
const packet = readHandoffPacket(request); const packet = readHandoffPacket(request);
const escalationHistory = await this.loadEscalationHistory(request.userId); const escalationHistory = await this.loadEscalationHistory(request.userId);
const llmDecision = await this.routeByLlm(packet, escalationHistory); const rulesDecision = routeSupervisorByRules(packet);
const useRegexFastPath =
getAgentSupervisorFastPathEnabled() &&
rulesDecision.target !== null &&
rulesDecision.confidence >= getSupervisorFastPathConfidence() &&
this.isKnownTarget(rulesDecision.target);
const llmDecision = useRegexFastPath
? null
: await this.routeByLlm(packet, escalationHistory);
const useLlmRoute = const useLlmRoute =
llmDecision !== null && llmDecision !== null &&
llmDecision.confidence >= getSupervisorLlmConfidenceThreshold(); llmDecision.confidence >= getSupervisorLlmConfidenceThreshold();
const target = useLlmRoute const target = useLlmRoute
? llmDecision.target ? llmDecision.target
: routeSupervisorTarget(packet); : rulesDecision.target;
const routedBy = useRegexFastPath
? "regex-fast-path"
: useLlmRoute
? "llm"
: "rules-fallback";
trace.getActiveSpan()?.setAttributes({
"supervisor.routed_by": routedBy,
"supervisor.rules_confidence": rulesDecision.confidence,
});
const reason = packet?.handoffReason ?? "out-of-scope"; const reason = packet?.handoffReason ?? "out-of-scope";
const queuedAt = new Date().toISOString(); const queuedAt = new Date().toISOString();
@@ -169,7 +189,8 @@ export class SupervisorSkill implements Skill {
return { return {
confidence: 0.6, confidence: 0.6,
metadata: { metadata: {
routedBy: useLlmRoute ? "llm" : "rules", routedBy,
rulesConfidence: rulesDecision.confidence,
source: userMessage.source, source: userMessage.source,
supervisorReason: useLlmRoute ? llmDecision.reason : reason, supervisorReason: useLlmRoute ? llmDecision.reason : reason,
supervisorEscalationTotal: escalationHistory?.totalEscalations ?? 0, supervisorEscalationTotal: escalationHistory?.totalEscalations ?? 0,
@@ -234,7 +255,10 @@ export class SupervisorSkill implements Skill {
private isKnownTarget(target: string): boolean { private isKnownTarget(target: string): boolean {
return ( return (
target === "human-review" || Boolean(this.skillRegistry?.get(target)) target === "human-review" ||
target === "onboarding" ||
target === "moderation" ||
Boolean(this.skillRegistry?.get(target))
); );
} }
+135
View File
@@ -0,0 +1,135 @@
import { createHash } from "node:crypto";
import type { GarnetClient } from "@common/garnet";
import { getToolAuditMaxLen } from "../config.js";
export const TOOL_AUDIT_STREAM = "lolympus-agent:tool-calls";
export interface ToolAuditEntry {
type: "tool-call";
toolName: string;
chatroomId: string;
userId: string;
argsHash: string;
resultSummary: string;
ok: boolean;
errorKind?: string | undefined;
durationMs: number;
skill: string;
traceId?: string | undefined;
calledAt: string;
}
export interface ListedToolAuditEntry extends ToolAuditEntry {
id: string;
}
function normalizeForHash(value: unknown): unknown {
if (Array.isArray(value)) {
return value.map(normalizeForHash);
}
if (typeof value !== "object" || value === null) {
return value;
}
return Object.fromEntries(
Object.entries(value as Record<string, unknown>)
.filter(([key]) => !/(password|secret|token|api[_-]?key|authorization)/iu.test(key))
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, entry]) => [key, normalizeForHash(entry)]),
);
}
function asString(value: string | undefined): string | undefined {
return value?.trim() ? value : undefined;
}
function parseAuditEntry(
id: string,
fields: Record<string, string>,
): ListedToolAuditEntry | null {
if (fields["type"] !== "tool-call") {
return null;
}
return {
id,
type: "tool-call",
argsHash: fields["argsHash"] ?? "",
calledAt: fields["calledAt"] ?? "",
chatroomId: fields["chatroomId"] ?? "",
durationMs: Number(fields["durationMs"] ?? 0),
...(asString(fields["errorKind"]) ? { errorKind: fields["errorKind"] } : {}),
ok: fields["ok"] === "true",
resultSummary: fields["resultSummary"] ?? "",
skill: fields["skill"] ?? "",
...(asString(fields["traceId"]) ? { traceId: fields["traceId"] } : {}),
toolName: fields["toolName"] ?? "",
userId: fields["userId"] ?? "",
};
}
export function hashToolArgs(args: unknown): string {
return createHash("sha256")
.update(JSON.stringify(normalizeForHash(args)))
.digest("hex")
.slice(0, 16);
}
export class ToolAuditLogger {
constructor(
private readonly garnetClient: Pick<
GarnetClient,
"appendStream" | "readStreamRange"
>,
) {}
async append(entry: ToolAuditEntry): Promise<void> {
await this.garnetClient.appendStream(
TOOL_AUDIT_STREAM,
Object.fromEntries(
Object.entries(entry).map(([key, value]) => [key, String(value)]),
),
{ maxLen: getToolAuditMaxLen() },
);
}
async list(opts: {
skill?: string | undefined;
since?: string | undefined;
limit?: number | undefined;
}): Promise<ListedToolAuditEntry[]> {
const limit = Math.max(1, Math.min(opts.limit ?? 50, 500));
const entries = await this.garnetClient.readStreamRange(
TOOL_AUDIT_STREAM,
"-",
"+",
{ count: limit * 4 },
);
const sinceTime = opts.since ? Date.parse(opts.since) : NaN;
return entries
.map((entry) => parseAuditEntry(entry.id, entry.message))
.filter((entry): entry is ListedToolAuditEntry => entry !== null)
.filter((entry) => !opts.skill || entry.skill === opts.skill)
.filter((entry) => {
if (!Number.isFinite(sinceTime)) {
return true;
}
const calledAtTime = Date.parse(entry.calledAt);
return Number.isFinite(calledAtTime) && calledAtTime >= sinceTime;
})
.slice(-limit)
.reverse();
}
async get(id: string): Promise<ListedToolAuditEntry | null> {
const entries = await this.garnetClient.readStreamRange(
TOOL_AUDIT_STREAM,
id,
id,
{ count: 1 },
);
const first = entries[0];
return first ? parseAuditEntry(first.id, first.message) : null;
}
}
+28 -1
View File
@@ -1,6 +1,13 @@
import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import type { MemoryService } from "../../memory/memory-types.js"; import type { MemoryService } from "../../memory/memory-types.js";
import { lookupRuleTool } from "./lookup-rule.js"; import {
lookupRuleTool,
resolveCommunityRulesPath,
} from "./lookup-rule.js";
const failingMemoryService = { const failingMemoryService = {
longTerm: { longTerm: {
@@ -9,6 +16,26 @@ const failingMemoryService = {
} as unknown as MemoryService; } as unknown as MemoryService;
describe("lookupRuleTool", () => { describe("lookupRuleTool", () => {
it("falls back to the source rules resource when imported from dist", () => {
const root = mkdtempSync(join(tmpdir(), "community-rules-resource-"));
const sourceResourceDir = join(
root,
"src",
"skills",
"moderation",
"resources",
);
mkdirSync(sourceResourceDir, { recursive: true });
const sourceRulesPath = join(sourceResourceDir, "community-rules.md");
writeFileSync(sourceRulesPath, "## ads\n광고 금지\n");
const moduleUrl = pathToFileURL(
join(root, "dist", "tools", "moderation", "lookup-rule.js"),
).href;
expect(resolveCommunityRulesPath(moduleUrl, root)).toBe(sourceRulesPath);
});
it("returns the matching community rule section", async () => { it("returns the matching community rule section", async () => {
const result = await lookupRuleTool.execute({ topic: "ads" }, {}); const result = await lookupRuleTool.execute({ topic: "ads" }, {});
+27 -4
View File
@@ -1,4 +1,5 @@
import { readFileSync } from "node:fs"; import { existsSync, readFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { z } from "zod"; import { z } from "zod";
import { import {
@@ -40,10 +41,31 @@ type RuleTopic =
| "sanctions" | "sanctions"
| "spam"; | "spam";
const RULES_PATH = fileURLToPath( const RULES_RELATIVE_PATH = "skills/moderation/resources/community-rules.md";
new URL("../../skills/moderation/resources/community-rules.md", import.meta.url),
export function resolveCommunityRulesPath(
moduleUrl = import.meta.url,
cwd = process.cwd(),
): string {
const moduleDir = dirname(fileURLToPath(moduleUrl));
const candidates = [
resolve(moduleDir, "../../", RULES_RELATIVE_PATH),
resolve(moduleDir, "../../../src", RULES_RELATIVE_PATH),
resolve(cwd, "dist", RULES_RELATIVE_PATH),
resolve(cwd, "src", RULES_RELATIVE_PATH),
];
const found = candidates.find((candidate) => existsSync(candidate));
if (!found) {
throw new Error(
`Missing moderation community rules resource. Tried: ${candidates.join(", ")}`,
); );
const RULES_TEXT = readFileSync(RULES_PATH, "utf8"); }
return found;
}
const RULES_TEXT = readFileSync(resolveCommunityRulesPath(), "utf8");
function extractRuleSection(topic: RuleTopic): string { function extractRuleSection(topic: RuleTopic): string {
const pattern = new RegExp(`^## ${topic}\\n([\\s\\S]*?)(?=\\n## |$)`, "mu"); const pattern = new RegExp(`^## ${topic}\\n([\\s\\S]*?)(?=\\n## |$)`, "mu");
@@ -93,6 +115,7 @@ export const lookupRuleTool: Tool<
name: "lookup-rule", name: "lookup-rule",
description: description:
"광고, 도배, 욕설, 신고, 제재, 이의 제기, 개인정보 같은 moderation 규칙을 조회한다.", "광고, 도배, 욕설, 신고, 제재, 이의 제기, 개인정보 같은 moderation 규칙을 조회한다.",
requires: ["memory-search"],
argsSchema, argsSchema,
resultSchema, resultSchema,
async execute(args, ctx) { async execute(args, ctx) {
@@ -44,6 +44,8 @@ describe("checkOnboardingProgressTool", () => {
expect(result.step1Done).toBe(true); expect(result.step1Done).toBe(true);
expect(result.overallDone).toBe(false); expect(result.overallDone).toBe(false);
expect(result.discordProfileNicknameDone).toBe(false);
expect(result.discordProfileMessageDone).toBe(false);
expect(result.remainingSteps).toEqual([ expect(result.remainingSteps).toEqual([
"discord-profile-nickname", "discord-profile-nickname",
"discord-profile-message", "discord-profile-message",
@@ -104,6 +106,8 @@ describe("checkOnboardingProgressTool", () => {
expect(result.step1Done).toBe(true); expect(result.step1Done).toBe(true);
expect(result.overallDone).toBe(true); expect(result.overallDone).toBe(true);
expect(result.discordProfileNicknameDone).toBe(null);
expect(result.discordProfileMessageDone).toBe(null);
expect(result.remainingSteps).toEqual([]); expect(result.remainingSteps).toEqual([]);
}); });
}); });
@@ -18,6 +18,8 @@ const argsSchema = z.object({
}); });
const resultSchema = z.object({ const resultSchema = z.object({
discordProfileMessageDone: z.boolean().nullable(),
discordProfileNicknameDone: z.boolean().nullable(),
nicknameDone: z.boolean(), nicknameDone: z.boolean(),
photoDone: z.boolean(), photoDone: z.boolean(),
step1Done: z.boolean(), step1Done: z.boolean(),
@@ -41,6 +43,7 @@ export const checkOnboardingProgressTool: Tool<
name: "check-onboarding-progress", name: "check-onboarding-progress",
description: description:
"현재 유저의 온보딩 진행 상태를 읽어 카카오톡 단계와 디스코드 단계를 포함한 남은 절차를 구조화해서 반환한다.", "현재 유저의 온보딩 진행 상태를 읽어 카카오톡 단계와 디스코드 단계를 포함한 남은 절차를 구조화해서 반환한다.",
requires: ["garnet", "user-session"],
argsSchema, argsSchema,
resultSchema, resultSchema,
async execute(args, ctx) { async execute(args, ctx) {
@@ -61,6 +64,8 @@ export const checkOnboardingProgressTool: Tool<
: null; : null;
return { return {
discordProfileMessageDone: session?.discordProfileMessageVerified ?? null,
discordProfileNicknameDone: session?.discordProfileNicknameVerified ?? null,
nicknameDone: session?.nicknameVerified ?? false, nicknameDone: session?.nicknameVerified ?? false,
photoDone: session?.photoVerified ?? false, photoDone: session?.photoVerified ?? false,
step1Done: isStep1Completed(session), step1Done: isStep1Completed(session),
+1
View File
@@ -24,6 +24,7 @@ export const getRoomContextTool: Tool<
name: "get-room-context", name: "get-room-context",
description: description:
"최근 채팅방 대화를 조회해서 현재 유저 메시지가 다른 사람에게 향한 말인지, 방 분위기가 어떤지 판단할 수 있게 돕는다.", "최근 채팅방 대화를 조회해서 현재 유저 메시지가 다른 사람에게 향한 말인지, 방 분위기가 어떤지 판단할 수 있게 돕는다.",
requires: ["room-context"],
argsSchema, argsSchema,
resultSchema, resultSchema,
async execute(args, ctx) { async execute(args, ctx) {
@@ -90,6 +90,7 @@ export const lookupCommunityRuleTool: Tool<
name: "lookup-community-rule", name: "lookup-community-rule",
description: description:
"나이 제한, 닉네임 형식, 티어 코드, 하트 인증 규칙, 디스코드 안내 같은 커뮤니티 규칙을 조회한다.", "나이 제한, 닉네임 형식, 티어 코드, 하트 인증 규칙, 디스코드 안내 같은 커뮤니티 규칙을 조회한다.",
requires: ["memory-search"],
argsSchema, argsSchema,
resultSchema, resultSchema,
async execute(args, ctx) { async execute(args, ctx) {
@@ -32,6 +32,7 @@ export const validateNicknameTool: Tool<
name: "validate-nickname", name: "validate-nickname",
description: description:
"새내기 닉네임 형식이 이름/출생 연도/성별/티어 규칙에 맞는지 검사하고, 틀린 부분을 구조화해서 반환한다.", "새내기 닉네임 형식이 이름/출생 연도/성별/티어 규칙에 맞는지 검사하고, 틀린 부분을 구조화해서 반환한다.",
requires: [],
argsSchema, argsSchema,
resultSchema, resultSchema,
execute(args) { execute(args) {
@@ -54,6 +54,16 @@ describe("verifyDiscordNicknameTool", () => {
username: "priri", username: "priri",
}, },
], ],
residentApplicationMessageMatches: [
{
authorId: "discord-1",
channelId: "resident-channel-1",
content: "프리리#KR1/95/E",
createdAt: "2026-04-17T00:12:00.000Z",
matchedBy: "author-id",
messageId: "message-1",
},
],
joinedAfterMatches: [], joinedAfterMatches: [],
}; };
const request = vi.fn( const request = vi.fn(
@@ -81,12 +91,88 @@ describe("verifyDiscordNicknameTool", () => {
expect(result.status).toBe("matched-renamed"); expect(result.status).toBe("matched-renamed");
expect(result.message).toContain("올바른 형식으로 설정되어 있어요"); expect(result.message).toContain("올바른 형식으로 설정되어 있어요");
expect(result.message).toContain("주민신청 메시지도 확인됐어요");
expect(result.discordMemberId).toBe("discord-1"); expect(result.discordMemberId).toBe("discord-1");
expect(result.residentApplicationMessageStatus).toBe("confirmed");
expect(result.residentApplicationMessageId).toBe("message-1");
expect(result.residentApplicationMessageContent).toBe("프리리#KR1/95/E");
expect(result.residentApplicationMessageExpectedContent).toBe("프리리#KR1/95/E");
expect(session.discordMemberId).toBe("discord-1"); expect(session.discordMemberId).toBe("discord-1");
expect(session.discordProfileNicknameVerified).toBe(true);
expect(session.discordProfileMessageVerified).toBe(true);
expect(session.discordProfileMessageVerifiedAt).toBe(
"2026-04-17T00:12:00.000Z",
);
expect(request).toHaveBeenCalledTimes(1); expect(request).toHaveBeenCalledTimes(1);
expect(request.mock.calls[0]?.[1]).toMatchObject({
includeResidentApplicationMessages: true,
searchName: "프리리",
});
expect(setJson).toHaveBeenCalledTimes(1); expect(setJson).toHaveBeenCalledTimes(1);
}); });
it("does not confirm resident application messages that differ from the server nickname", async () => {
const setJson = vi.fn().mockResolvedValue(undefined);
const response: DiscordMemberLookupResponse = {
status: "ok",
nicknameMatches: [
{
memberId: "discord-1",
joinedAt: "2026-04-17T00:10:00.000Z",
globalName: "프리리",
guildNickname: "프리리#KR1/95/E",
matchedBy: "guild-nickname",
username: "priri",
},
],
residentApplicationMessageMatches: [
{
authorId: "discord-1",
channelId: "resident-channel-1",
content: "프리리/E",
createdAt: "2026-04-17T00:12:00.000Z",
matchedBy: "author-id",
messageId: "message-1",
},
],
joinedAfterMatches: [],
};
const request = vi.fn(
<TRequest = unknown, TResponse = unknown>(
_subject: string,
_payload: TRequest,
_options?: { timeoutMs?: number },
): Promise<TResponse> => Promise.resolve(response as TResponse),
);
const session = makeSession();
const result = await verifyDiscordNicknameTool.execute(
{},
{
chatroomId: session.chatroomId,
garnetClient: {
setJson,
} as never,
natsClient: {
request: request as never,
},
session,
userId: session.userId,
},
);
expect(result.status).toBe("matched-renamed");
expect(result.residentApplicationMessageStatus).toBe("content-mismatch");
expect(result.residentApplicationMessageContent).toBe("프리리/E");
expect(result.residentApplicationMessageExpectedContent).toBe("프리리#KR1/95/E");
expect(result.residentApplicationMessageCorrectionGuide).toContain(
"주민신청 메시지는 서버프로필 닉네임과 동일하게 작성해야 해요",
);
expect(result.message).toContain("현재 메시지는 '프리리/E'");
expect(session.discordProfileNicknameVerified).toBe(true);
expect(session.discordProfileMessageVerified).toBeUndefined();
});
it("does not mark renamed when only the raw kakao nickname matches literally", async () => { it("does not mark renamed when only the raw kakao nickname matches literally", async () => {
const response: DiscordMemberLookupResponse = { const response: DiscordMemberLookupResponse = {
status: "ok", status: "ok",
@@ -121,7 +207,51 @@ describe("verifyDiscordNicknameTool", () => {
); );
expect(result.status).toBe("matched-pending-rename"); expect(result.status).toBe("matched-pending-rename");
expect(result.message).toContain("이름#태그/출생연도/티어"); expect(result.message).toContain("디스코드 닉네임이 형식과 맞지 않습니다.");
expect(result.message).toContain("현재 소환사님의 서버프로필은 '프리리/95/여/E'");
expect(result.message).toContain("닉네임#태그/출생연도/티어로 수정해 주세요.");
expect(result.recommendedNickname).toBe("프리리#태그/95/E");
expect(result.correctionGuide).toBe(result.message);
expect(result.guidance).toContain("이름#태그/출생연도/티어");
});
it("preserves an existing discord tag when suggesting a corrected nickname", async () => {
const response: DiscordMemberLookupResponse = {
status: "ok",
nicknameMatches: [
{
memberId: "discord-1",
globalName: "프리리",
guildNickname: "프리리#KR1/95/여/E",
joinedAt: "2026-04-17T00:10:00.000Z",
matchedBy: "global-name",
username: "priri",
},
],
joinedAfterMatches: [],
};
const request = vi.fn(
<TRequest = unknown, TResponse = unknown>(
_subject: string,
_payload: TRequest,
_options?: { timeoutMs?: number },
): Promise<TResponse> => Promise.resolve(response as TResponse),
);
const result = await verifyDiscordNicknameTool.execute(
{},
{
natsClient: {
request: request as never,
},
session: makeSession(),
},
);
expect(result.status).toBe("matched-pending-rename");
expect(result.recommendedNickname).toBe("프리리#KR1/95/E");
expect(result.correctionGuide).toContain("현재 소환사님의 서버프로필은 '프리리#KR1/95/여/E'");
expect(result.correctionGuide).toContain("할리#프렌치불독/88/D 와 같이");
}); });
it("falls back to joined-after candidates and returns not-confirmed when ambiguous", async () => { it("falls back to joined-after candidates and returns not-confirmed when ambiguous", async () => {
+306 -16
View File
@@ -8,6 +8,7 @@ import type {
DiscordMemberCandidate, DiscordMemberCandidate,
DiscordMemberLookupRequest, DiscordMemberLookupRequest,
DiscordMemberLookupResponse, DiscordMemberLookupResponse,
DiscordResidentApplicationMessageCandidate,
} from '@common/nats-subjects'; } from '@common/nats-subjects';
import { getContext, getRequestLogger } from '@obs/context'; import { getContext, getRequestLogger } from '@obs/context';
import { import {
@@ -20,6 +21,7 @@ import {
getKakaotalkNickname, getKakaotalkNickname,
matchesDiscordNicknameRule, matchesDiscordNicknameRule,
parseKakaoNickname, parseKakaoNickname,
type ParsedKakaoNickname,
} from '@skills/onboarding/session'; } from '@skills/onboarding/session';
import type { Tool, ToolContext } from '../tool.js'; import type { Tool, ToolContext } from '../tool.js';
@@ -50,6 +52,8 @@ const resultSchema = z.object({
searchName: z.string().optional(), searchName: z.string().optional(),
discordMemberId: z.string().optional(), discordMemberId: z.string().optional(),
currentDisplayName: z.string().optional(), currentDisplayName: z.string().optional(),
recommendedNickname: z.string().optional(),
correctionGuide: z.string().optional(),
formatExample: z.string().optional(), formatExample: z.string().optional(),
guidance: z.string().optional(), guidance: z.string().optional(),
issueMessages: z.array(z.string()).optional(), issueMessages: z.array(z.string()).optional(),
@@ -57,6 +61,19 @@ const resultSchema = z.object({
matchSource: z.enum(['joined-after', 'nickname']).optional(), matchSource: z.enum(['joined-after', 'nickname']).optional(),
matchedBy: z.enum(['global-name', 'guild-nickname', 'joined-after', 'username']).optional(), matchedBy: z.enum(['global-name', 'guild-nickname', 'joined-after', 'username']).optional(),
nicknameMatchCount: z.number().int().nonnegative().optional(), nicknameMatchCount: z.number().int().nonnegative().optional(),
residentApplicationMessageAuthorId: z.string().optional(),
residentApplicationMessageContent: z.string().optional(),
residentApplicationMessageCorrectionGuide: z.string().optional(),
residentApplicationMessageCreatedAt: z.string().optional(),
residentApplicationMessageExpectedContent: z.string().optional(),
residentApplicationMessageId: z.string().optional(),
residentApplicationMessageMatchCount: z.number().int().nonnegative().optional(),
residentApplicationMessageMatchedBy: z
.enum(['author-id', 'author-name', 'content', 'joined-after'])
.optional(),
residentApplicationMessageStatus: z
.enum(['ambiguous', 'confirmed', 'content-mismatch', 'lookup-unavailable', 'not-confirmed'])
.optional(),
}); });
function pickDisplayName(candidate: DiscordMemberCandidate): string { function pickDisplayName(candidate: DiscordMemberCandidate): string {
@@ -65,34 +82,249 @@ function pickDisplayName(candidate: DiscordMemberCandidate): string {
); );
} }
function resolveTimeoutMs(): number | null { function extractExistingDiscordTag(
currentGuildNickname: string | undefined,
parsed: ParsedKakaoNickname,
): string | null {
const normalized = currentGuildNickname?.trim();
if (!normalized) {
return null;
}
const prefix = `${parsed.baseName}#`;
if (!normalized.startsWith(prefix)) {
return null;
}
const tag = normalized.slice(prefix.length).split('/')[0]?.trim();
return tag || null;
}
function buildRecommendedNickname(
parsed: ParsedKakaoNickname,
currentGuildNickname: string | undefined,
): string {
const tag = extractExistingDiscordTag(currentGuildNickname, parsed) ?? '태그';
return `${parsed.baseName}#${tag}/${parsed.year}/${parsed.tier}`;
}
function buildCorrectionGuide(args: {
currentDisplayName: string;
currentGuildNickname?: string | undefined;
}): string {
const formatGuide = `${DISCORD_NICKNAME_FORMAT_EXAMPLE} 와 같이 닉네임#태그/출생연도/티어로 수정해 주세요.`;
if (args.currentGuildNickname?.trim()) {
return `디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '${args.currentGuildNickname.trim()}' 으로 확인됩니다. ${formatGuide}`;
}
return `디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '${args.currentDisplayName}' 으로 확인됩니다. ${formatGuide}`;
}
type ResidentApplicationMessageStatus =
| 'ambiguous'
| 'confirmed'
| 'content-mismatch'
| 'lookup-unavailable'
| 'not-confirmed';
interface ResidentApplicationMessageResult {
candidate?: DiscordResidentApplicationMessageCandidate | undefined;
correctionGuide?: string | undefined;
expectedContent?: string | undefined;
matchCount?: number | undefined;
status: ResidentApplicationMessageStatus;
}
function resolveResidentApplicationMessageResult(
matches: DiscordResidentApplicationMessageCandidate[] | undefined,
): ResidentApplicationMessageResult {
if (!matches) {
return { status: 'lookup-unavailable' };
}
if (matches.length === 0) {
return { matchCount: 0, status: 'not-confirmed' };
}
if (matches.length > 1) {
return { matchCount: matches.length, status: 'ambiguous' };
}
return {
candidate: matches[0],
matchCount: 1,
status: 'confirmed',
};
}
function normalizeMessageContent(content: string | undefined): string {
return content?.replace(/\s+/gu, ' ').trim() ?? '';
}
function buildResidentApplicationMessageCorrectionGuide(args: {
actualContent: string;
expectedContent: string;
}): string {
return `주민신청 메시지는 서버프로필 닉네임과 동일하게 작성해야 해요. 현재 메시지는 '${args.actualContent}' 로 확인됩니다. '${args.expectedContent}' 로 다시 남겨주세요.`;
}
function validateResidentApplicationMessageContent(
result: ResidentApplicationMessageResult,
expectedContent: string | undefined,
): ResidentApplicationMessageResult {
if (result.status !== 'confirmed' || !result.candidate || !expectedContent) {
return expectedContent ? { ...result, expectedContent } : result;
}
const actualContent = normalizeMessageContent(result.candidate.content);
const normalizedExpected = normalizeMessageContent(expectedContent);
if (actualContent === normalizedExpected) {
return {
...result,
expectedContent: normalizedExpected,
};
}
return {
...result,
correctionGuide: buildResidentApplicationMessageCorrectionGuide({
actualContent,
expectedContent: normalizedExpected,
}),
expectedContent: normalizedExpected,
status: 'content-mismatch',
};
}
function truncateMessageContent(content: string): string {
const normalized = content.replace(/\s+/gu, ' ').trim();
return normalized.length > 160 ? `${normalized.slice(0, 157)}...` : normalized;
}
function buildResidentApplicationMessageSummary(
result: ResidentApplicationMessageResult,
): string | null {
if (result.status === 'lookup-unavailable') {
return null;
}
if (result.status === 'confirmed' && result.candidate) {
return `주민신청 메시지도 확인됐어요.`;
}
if (result.status === 'content-mismatch' && result.correctionGuide) {
return result.correctionGuide;
}
if (result.status === 'ambiguous') {
return '주민신청 메시지 후보가 여러 개라 하나로 확정하지 못했어요.';
}
return '주민신청 채널 메시지는 아직 확인되지 않았어요.';
}
function residentApplicationMessageFields(
result: ResidentApplicationMessageResult,
): Pick<
z.infer<typeof resultSchema>,
| 'residentApplicationMessageAuthorId'
| 'residentApplicationMessageContent'
| 'residentApplicationMessageCorrectionGuide'
| 'residentApplicationMessageCreatedAt'
| 'residentApplicationMessageExpectedContent'
| 'residentApplicationMessageId'
| 'residentApplicationMessageMatchCount'
| 'residentApplicationMessageMatchedBy'
| 'residentApplicationMessageStatus'
> {
const candidate = result.candidate;
return {
residentApplicationMessageCorrectionGuide: result.correctionGuide,
residentApplicationMessageExpectedContent: result.expectedContent,
residentApplicationMessageMatchCount: result.matchCount,
residentApplicationMessageStatus: result.status,
...(candidate
? {
residentApplicationMessageAuthorId: candidate.authorId,
residentApplicationMessageContent: truncateMessageContent(candidate.content),
residentApplicationMessageCreatedAt: candidate.createdAt,
residentApplicationMessageId: candidate.messageId,
residentApplicationMessageMatchedBy: candidate.matchedBy,
}
: {}),
};
}
interface TimeoutBudget {
configuredTimeoutMs: number;
deadlineAt?: string | undefined;
remainingMs?: number | undefined;
timeoutMs: number | null;
}
function resolveTimeoutBudget(): TimeoutBudget {
const configured = getDiscordVerifyTimeoutMs(); const configured = getDiscordVerifyTimeoutMs();
const deadlineAt = getContext()?.deadlineAt; const deadlineAt = getContext()?.deadlineAt;
if (!deadlineAt) { if (!deadlineAt) {
return configured; return {
configuredTimeoutMs: configured,
timeoutMs: configured,
};
} }
const remainingMs = Date.parse(deadlineAt) - Date.now() - 100; const remainingMs = Date.parse(deadlineAt) - Date.now() - 100;
if (!Number.isFinite(remainingMs) || remainingMs <= 0) { if (!Number.isFinite(remainingMs) || remainingMs <= 0) {
return null; return {
configuredTimeoutMs: configured,
deadlineAt,
remainingMs,
timeoutMs: null,
};
} }
return Math.min(configured, remainingMs); return {
configuredTimeoutMs: configured,
deadlineAt,
remainingMs,
timeoutMs: Math.min(configured, remainingMs),
};
} }
async function persistDiscordMemberId(memberId: string, ctx: ToolContext): Promise<void> { async function persistDiscordVerification(
input: {
memberId: string;
nicknameVerified: boolean;
residentApplicationMessageVerifiedAt?: string | undefined;
},
ctx: ToolContext,
): Promise<void> {
if (!ctx.garnetClient || !ctx.session) { if (!ctx.garnetClient || !ctx.session) {
return; return;
} }
const now = new Date().toISOString();
const nextSession = { const nextSession = {
...ctx.session, ...ctx.session,
discordMemberId: memberId, discordMemberId: input.memberId,
...(input.nicknameVerified
? {
discordProfileNicknameVerified: true,
discordProfileNicknameVerifiedAt:
ctx.session.discordProfileNicknameVerifiedAt ?? now,
}
: {}),
...(input.residentApplicationMessageVerifiedAt
? {
discordProfileMessageVerified: true,
discordProfileMessageVerifiedAt: input.residentApplicationMessageVerifiedAt,
}
: {}),
}; };
const key = `${getOnboardingSessionKeyPrefix()}:onboarding:${nextSession.chatroomId}:${nextSession.userId}`; const key = `${getOnboardingSessionKeyPrefix()}:onboarding:${nextSession.chatroomId}:${nextSession.userId}`;
await ctx.garnetClient.setJson(key, nextSession); await ctx.garnetClient.setJson(key, nextSession);
ctx.session.discordMemberId = memberId; Object.assign(ctx.session, nextSession);
} }
export const verifyDiscordNicknameTool: Tool< export const verifyDiscordNicknameTool: Tool<
@@ -102,6 +334,7 @@ export const verifyDiscordNicknameTool: Tool<
name: 'verify-discord-nickname', name: 'verify-discord-nickname',
description: description:
'현재 유저의 카카오톡 인증 닉네임과 세션 입장 시각을 기준으로 디스코드 멤버를 찾아, 닉네임 변경이 확인됐는지 구조화해서 반환한다.', '현재 유저의 카카오톡 인증 닉네임과 세션 입장 시각을 기준으로 디스코드 멤버를 찾아, 닉네임 변경이 확인됐는지 구조화해서 반환한다.',
requires: ['garnet', 'nats', 'user-session'],
argsSchema, argsSchema,
resultSchema, resultSchema,
async execute(_args, ctx) { async execute(_args, ctx) {
@@ -145,8 +378,7 @@ export const verifyDiscordNicknameTool: Tool<
} }
const parsedKakaoNickname = parseKakaoNickname(expectedNickname); const parsedKakaoNickname = parseKakaoNickname(expectedNickname);
const searchName = parsedKakaoNickname?.baseName ?? ''; if (!parsedKakaoNickname?.baseName) {
if (!searchName) {
logger.debug( logger.debug(
{ {
expectedNickname, expectedNickname,
@@ -160,6 +392,7 @@ export const verifyDiscordNicknameTool: Tool<
expectedNickname, expectedNickname,
}; };
} }
const searchName = parsedKakaoNickname.baseName;
if (!ctx.natsClient) { if (!ctx.natsClient) {
logger.debug( logger.debug(
@@ -178,11 +411,15 @@ export const verifyDiscordNicknameTool: Tool<
}; };
} }
const timeoutMs = resolveTimeoutMs(); const timeoutBudget = resolveTimeoutBudget();
const timeoutMs = timeoutBudget.timeoutMs;
if (!timeoutMs) { if (!timeoutMs) {
logger.debug( logger.debug(
{ {
configuredTimeoutMs: timeoutBudget.configuredTimeoutMs,
deadlineAt: timeoutBudget.deadlineAt,
expectedNickname, expectedNickname,
remainingMs: timeoutBudget.remainingMs,
searchName, searchName,
}, },
'[VerifyDiscordNickname] No timeout budget left for lookup', '[VerifyDiscordNickname] No timeout budget left for lookup',
@@ -200,6 +437,7 @@ export const verifyDiscordNicknameTool: Tool<
try { try {
const payload: DiscordMemberLookupRequest = { const payload: DiscordMemberLookupRequest = {
chatroomId: ctx.chatroomId ?? ctx.session.chatroomId, chatroomId: ctx.chatroomId ?? ctx.session.chatroomId,
includeResidentApplicationMessages: true,
joinedAfter: ctx.session.joinedAt, joinedAfter: ctx.session.joinedAt,
searchName, searchName,
userId: ctx.userId ?? ctx.session.userId, userId: ctx.userId ?? ctx.session.userId,
@@ -208,6 +446,9 @@ export const verifyDiscordNicknameTool: Tool<
logger.debug( logger.debug(
{ {
payload, payload,
configuredTimeoutMs: timeoutBudget.configuredTimeoutMs,
deadlineAt: timeoutBudget.deadlineAt,
remainingMs: timeoutBudget.remainingMs,
subject: getDiscordMemberLookupSubject(), subject: getDiscordMemberLookupSubject(),
timeoutMs, timeoutMs,
}, },
@@ -256,11 +497,18 @@ export const verifyDiscordNicknameTool: Tool<
const nicknameMatches = lookupResult.nicknameMatches; const nicknameMatches = lookupResult.nicknameMatches;
const joinedAfterMatches = lookupResult.joinedAfterMatches; const joinedAfterMatches = lookupResult.joinedAfterMatches;
const unvalidatedResidentApplicationMessageResult = resolveResidentApplicationMessageResult(
lookupResult.residentApplicationMessageMatches,
);
logger.debug( logger.debug(
{ {
expectedNickname, expectedNickname,
joinedAfterMatches: joinedAfterMatches.length, joinedAfterMatches: joinedAfterMatches.length,
nicknameMatches: nicknameMatches.length, nicknameMatches: nicknameMatches.length,
residentApplicationMessageMatchCount:
unvalidatedResidentApplicationMessageResult.matchCount,
residentApplicationMessageStatus:
unvalidatedResidentApplicationMessageResult.status,
searchName, searchName,
}, },
'[VerifyDiscordNickname] Lookup response received', '[VerifyDiscordNickname] Lookup response received',
@@ -292,6 +540,10 @@ export const verifyDiscordNicknameTool: Tool<
expectedNickname, expectedNickname,
joinedAfterMatchCount: joinedAfterMatches.length, joinedAfterMatchCount: joinedAfterMatches.length,
nicknameMatchCount: nicknameMatches.length, nicknameMatchCount: nicknameMatches.length,
residentApplicationMessageMatchCount:
unvalidatedResidentApplicationMessageResult.matchCount,
residentApplicationMessageStatus:
unvalidatedResidentApplicationMessageResult.status,
reason: reason:
reason ?? reason ??
(nicknameMatches.length === 0 ? 'nickname-search-none' : 'nickname-search-ambiguous'), (nicknameMatches.length === 0 ? 'nickname-search-none' : 'nickname-search-ambiguous'),
@@ -309,29 +561,63 @@ export const verifyDiscordNicknameTool: Tool<
searchName, searchName,
joinedAfterMatchCount: joinedAfterMatches.length, joinedAfterMatchCount: joinedAfterMatches.length,
nicknameMatchCount: nicknameMatches.length, nicknameMatchCount: nicknameMatches.length,
...residentApplicationMessageFields(unvalidatedResidentApplicationMessageResult),
}; };
} }
await persistDiscordMemberId(candidate.memberId, ctx);
const currentDisplayName = pickDisplayName(candidate); const currentDisplayName = pickDisplayName(candidate);
const currentGuildNickname = candidate.guildNickname?.trim(); const currentGuildNickname = candidate.guildNickname?.trim();
const isRenamed = matchesDiscordNicknameRule(currentGuildNickname, expectedNickname); const isRenamed = matchesDiscordNicknameRule(currentGuildNickname, expectedNickname);
const status = isRenamed ? 'matched-renamed' : 'matched-pending-rename'; const status = isRenamed ? 'matched-renamed' : 'matched-pending-rename';
const recommendedNickname = buildRecommendedNickname(parsedKakaoNickname, currentGuildNickname);
const residentApplicationMessageExpectedContent = isRenamed
? currentGuildNickname
: recommendedNickname.includes('#태그/')
? undefined
: recommendedNickname;
const residentApplicationMessageResult = validateResidentApplicationMessageContent(
unvalidatedResidentApplicationMessageResult,
residentApplicationMessageExpectedContent,
);
await persistDiscordVerification(
{
memberId: candidate.memberId,
nicknameVerified: isRenamed,
residentApplicationMessageVerifiedAt:
residentApplicationMessageResult.status === 'confirmed'
? residentApplicationMessageResult.candidate?.createdAt
: undefined,
},
ctx,
);
const correctionGuide = buildCorrectionGuide({
currentDisplayName,
currentGuildNickname,
});
const discordFormatGuide = `디스코드 서버 닉네임은 이름#태그/출생연도/티어 형식이에요. 카카오톡 형식과 달리 성별은 빼고 태그(#)가 들어가요. 예: ${DISCORD_NICKNAME_FORMAT_EXAMPLE}`; const discordFormatGuide = `디스코드 서버 닉네임은 이름#태그/출생연도/티어 형식이에요. 카카오톡 형식과 달리 성별은 빼고 태그(#)가 들어가요. 예: ${DISCORD_NICKNAME_FORMAT_EXAMPLE}`;
const residentApplicationMessageSummary = buildResidentApplicationMessageSummary(
residentApplicationMessageResult,
);
const message = isRenamed const message = isRenamed
? '디스코드 서버 닉네임이 올바른 형식으로 설정되어 있어요. 확인 완료!' ? '디스코드 서버 닉네임이 올바른 형식으로 설정되어 있어요. 확인 완료!'
: currentGuildNickname : correctionGuide;
? `디스코드에서 유저를 찾았지만 서버 닉네임이 아직 올바른 형식으로 확인되지는 않았어요. 현재 서버 닉네임은 "${currentGuildNickname}"으로 보여요. ${discordFormatGuide}` const finalMessage = [message, residentApplicationMessageSummary]
: `디스코드에서 유저를 찾았지만 서버 닉네임이 아직 설정되지 않았거나 확인되지 않았어요. 현재 표시 이름은 "${currentDisplayName}"으로 보여요. ${discordFormatGuide}`; .filter(Boolean)
.join(' ');
logger.debug( logger.debug(
{ {
currentDisplayName, currentDisplayName,
currentGuildNickname,
discordMemberId: candidate.memberId, discordMemberId: candidate.memberId,
expectedNickname, expectedNickname,
matchSource, matchSource,
matchedBy: candidate.matchedBy, matchedBy: candidate.matchedBy,
recommendedNickname,
residentApplicationMessageMatchCount:
residentApplicationMessageResult.matchCount,
residentApplicationMessageStatus: residentApplicationMessageResult.status,
status, status,
}, },
'[VerifyDiscordNickname] Verification resolved', '[VerifyDiscordNickname] Verification resolved',
@@ -339,12 +625,16 @@ export const verifyDiscordNicknameTool: Tool<
return { return {
status, status,
message, message: finalMessage,
currentDisplayName, currentDisplayName,
correctionGuide: isRenamed ? undefined : correctionGuide,
discordMemberId: candidate.memberId, discordMemberId: candidate.memberId,
expectedNickname, expectedNickname,
guidance: isRenamed ? undefined : discordFormatGuide,
matchSource, matchSource,
matchedBy: candidate.matchedBy, matchedBy: candidate.matchedBy,
recommendedNickname: isRenamed ? undefined : recommendedNickname,
...residentApplicationMessageFields(residentApplicationMessageResult),
searchName, searchName,
}; };
}, },
+62
View File
@@ -0,0 +1,62 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { buildScopedContext } from "./scoped-context.js";
import type { ToolContext } from "./tool.js";
describe("buildScopedContext", () => {
afterEach(() => {
vi.unstubAllEnvs();
});
it("returns the full context when sandbox is disabled", () => {
const ctx: ToolContext = {
chatroomId: "room-1",
garnetClient: {} as ToolContext["garnetClient"],
};
expect(buildScopedContext([], ctx, "pure-tool")).toBe(ctx);
});
it("throws when sandboxed tools access undeclared permissions", () => {
vi.stubEnv("AGENT_TOOL_SANDBOX_ENABLED", "true");
const ctx: ToolContext = {
chatroomId: "room-1",
garnetClient: {} as ToolContext["garnetClient"],
userId: "user-1",
};
const scoped = buildScopedContext([], ctx, "validate-nickname");
expect(scoped.chatroomId).toBe("room-1");
expect(() => scoped.garnetClient).toThrow(
"Tool validate-nickname does not declare requires:garnet",
);
});
it("exposes declared memory search without profile access", async () => {
vi.stubEnv("AGENT_TOOL_SANDBOX_ENABLED", "true");
const search = vi.fn(() => Promise.resolve([]));
const ctx = {
memoryService: {
longTerm: {
search,
loadUserProfile: vi.fn(),
},
session: {},
assembleWorkingMemory: vi.fn(),
},
} as unknown as ToolContext;
const scoped = buildScopedContext(
["memory-search"],
ctx,
"lookup-community-rule",
);
await scoped.memoryService?.longTerm.search("heart", { topK: 1 });
expect(search).toHaveBeenCalledWith("heart", { topK: 1 });
expect(() => scoped.memoryService?.longTerm.loadUserProfile("user-1")).toThrow(
"Tool lookup-community-rule does not declare requires:memory-profile",
);
});
});
+149
View File
@@ -0,0 +1,149 @@
import { getAgentToolSandboxEnabled } from "../config.js";
import type { LongTermMemory, MemoryService } from "../memory/memory-types.js";
import { getLogger } from "../observability/logger.js";
import type { ToolContext, ToolPermission } from "./tool.js";
const PERMISSION_BY_FIELD: Record<string, ToolPermission> = {
garnetClient: "garnet",
memoryService: "memory-search",
natsClient: "nats",
roomContextService: "room-context",
session: "user-session",
toolRegistry: "tool-registry",
};
function deniedAccess(toolName: string, permission: ToolPermission): never {
throw new Error(`Tool ${toolName} does not declare requires:${permission}`);
}
function defineDeniedGetter<T extends object>(
target: T,
key: string,
toolName: string,
permission: ToolPermission,
): void {
Object.defineProperty(target, key, {
enumerable: true,
get: () => deniedAccess(toolName, permission),
});
}
function buildScopedMemoryService(
requires: ReadonlySet<ToolPermission>,
fullContext: ToolContext,
toolName: string,
): MemoryService | undefined {
if (!fullContext.memoryService) {
return undefined;
}
const longTerm: Partial<LongTermMemory> = {};
if (requires.has("memory-search")) {
longTerm.search = fullContext.memoryService.longTerm.search.bind(
fullContext.memoryService.longTerm,
);
} else {
defineDeniedGetter(longTerm, "search", toolName, "memory-search");
}
if (requires.has("memory-profile")) {
longTerm.loadUserProfile =
fullContext.memoryService.longTerm.loadUserProfile.bind(
fullContext.memoryService.longTerm,
);
} else {
defineDeniedGetter(longTerm, "loadUserProfile", toolName, "memory-profile");
}
defineDeniedGetter(longTerm, "updateUserProfile", toolName, "memory-profile");
defineDeniedGetter(longTerm, "upsertDocument", toolName, "memory-search");
defineDeniedGetter(longTerm, "deleteDocument", toolName, "memory-search");
return {
longTerm: longTerm as LongTermMemory,
session: fullContext.memoryService.session,
assembleWorkingMemory: fullContext.memoryService.assembleWorkingMemory.bind(
fullContext.memoryService,
),
};
}
function findUndeclaredContextPermissions(
requires: ReadonlySet<ToolPermission>,
fullContext: ToolContext,
): ToolPermission[] {
return [
...new Set(
Object.entries(PERMISSION_BY_FIELD)
.filter(
([field, permission]) =>
!requires.has(permission) &&
fullContext[field as keyof ToolContext] !== undefined,
)
.map(([, permission]) => permission),
),
];
}
export function buildScopedContext(
requires: readonly ToolPermission[] | undefined,
fullContext: ToolContext,
toolName = "unknown",
): ToolContext {
const sandboxEnabled = getAgentToolSandboxEnabled();
if (!requires) {
if (sandboxEnabled) {
getLogger().warn(
{ toolName },
"[ToolSandbox] Tool has no requires declaration; using full context",
);
}
return fullContext;
}
if (!sandboxEnabled) {
const undeclared = findUndeclaredContextPermissions(
new Set(requires),
fullContext,
);
if (undeclared.length > 0) {
getLogger().warn(
{ toolName, undeclared },
"[ToolSandbox] Tool context includes undeclared permissions",
);
}
return fullContext;
}
const required = new Set(requires);
const scoped: ToolContext = {
chatroomId: fullContext.chatroomId,
userId: fullContext.userId,
};
if (required.has("garnet")) {
scoped.garnetClient = fullContext.garnetClient;
}
if (required.has("nats")) {
scoped.natsClient = fullContext.natsClient;
}
if (required.has("room-context")) {
scoped.roomContextService = fullContext.roomContextService;
}
if (required.has("tool-registry")) {
scoped.toolRegistry = fullContext.toolRegistry;
}
if (required.has("user-session")) {
scoped.session = fullContext.session;
}
if (required.has("memory-search") || required.has("memory-profile")) {
scoped.memoryService = buildScopedMemoryService(required, fullContext, toolName);
}
for (const [field, permission] of Object.entries(PERMISSION_BY_FIELD)) {
if (field in scoped) {
continue;
}
defineDeniedGetter(scoped, field, toolName, permission);
}
return scoped;
}
+10
View File
@@ -17,6 +17,15 @@ export interface ToolContext {
toolRegistry?: ToolRegistry | undefined; toolRegistry?: ToolRegistry | undefined;
} }
export type ToolPermission =
| "garnet"
| "memory-profile"
| "memory-search"
| "nats"
| "room-context"
| "tool-registry"
| "user-session";
export type ToolErrorKind = export type ToolErrorKind =
| "args-invalid" | "args-invalid"
| "transient" | "transient"
@@ -34,6 +43,7 @@ export interface ToolExecutionFailure {
export interface Tool<A, R> { export interface Tool<A, R> {
name: string; name: string;
description: string; description: string;
requires?: readonly ToolPermission[] | undefined;
argsSchema: ZodType<A>; argsSchema: ZodType<A>;
resultSchema: ZodType<R>; resultSchema: ZodType<R>;
execute(args: A, ctx: ToolContext): Promise<R>; execute(args: A, ctx: ToolContext): Promise<R>;