Compare commits

...
4 Commits
Author SHA1 Message Date
dahyeon d05de52cd9 Improve onboarding verification and routing controls 2026-05-24 23:46:59 -07:00
dahyeon bc0f8e0773 Add secret coverage and NATS subject ACL
- KNOWN_SECRET_KEYS now covers LLM, embedding, and Phoenix API keys
- Logger redaction paths extended to cover the new credential keys
- nats-skill-acl maps each skill to allowed publish/subscribe subjects
- NATS publish runs ACL check with log-only/enforce/off modes
2026-05-19 01:28:54 -07:00
dahyeon d011a2829c Add tool permission sandbox and audit stream
- Tool interface declares required permissions via requires field
- scoped-context builds a least-privileged ToolContext per call
- ToolAuditLogger appends each tool invocation to a Garnet stream
- Operations CLI gains tool-audit:list and tool-audit:show commands
- All onboarding and moderation tools annotated with explicit requires
2026-05-19 01:28:54 -07:00
dahyeon 21df14cd00 Add cost-aware routing and LLM budget tracking
- BudgetTracker tracks LLM token cost separately from tool cost
- LlmClient auto-records cost into request-scoped budget
- Router selects tier from classify confidence, intent risk, escalation count
- Supervisor short-circuits via regex fast-path when rule confidence is high
- Onboarding skill loads user escalation history to inform routing
2026-05-19 01:28:54 -07:00
60 changed files with 4273 additions and 187 deletions

No files matched your search

+30 -3
View File
@@ -257,6 +257,18 @@ npm run review:list -- --reason=low-confidence --limit=10
Resolution history는 stream에 쌓이며, compatibility hash는 sunset 전까지 read fallback으로 유지된다. Sunset 조건과 절차는 [resolution-hash-sunset-procedure.md](resolution-hash-sunset-procedure.md)를 따른다.
### 10.2 Tool Audit
Tool call은 성공과 실패 모두 `lolympus-agent:tool-calls` Stream에 남긴다. 원본 args는 저장하지 않고, 민감 키를 제거한 뒤 hash만 기록한다.
```bash
npm run tool-audit:list -- --limit=20
npm run tool-audit:list -- --skill=onboarding --since=24h
npm run tool-audit:show -- <callId>
```
Stream은 `AGENT_TOOL_AUDIT_MAX_LEN` 기준으로 trim된다. 기본값은 `100000`이다.
## 11. Calibration 흐름
Calibration은 사람 점수와 LLM judge 점수의 상관을 계산한다.
@@ -460,6 +472,13 @@ npm test
| `EVAL_TOURNAMENT_MIN_CASES` | 10 | proposal 최소 archive cases |
| `PROMPT_VARIANT_ID` | unset | 운영 prompt variant 선택 |
| `AGENT_ROUTER_WEIGHTS_PATH` | unset | learned router weights path |
| `AGENT_BUDGET_LLM_COST_ENABLED` | true | LLM token cost를 request budget에 누적 |
| `AGENT_COST_AWARE_TIER_ENABLED` | true | confidence/cost-aware tier selection |
| `AGENT_SUPERVISOR_FAST_PATH_ENABLED` | true | supervisor regex fast-path 사용 |
| `SUPERVISOR_FAST_PATH_CONFIDENCE` | 0.85 | fast-path 최소 규칙 confidence |
| `AGENT_TOOL_SANDBOX_ENABLED` | false | tool context 최소 권한 강제 여부 |
| `AGENT_TOOL_AUDIT_MAX_LEN` | 100000 | tool audit Stream maxlen |
| `AGENT_NATS_ACL_MODE` | log-only | NATS subject ACL 모드 |
| `AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED` | true | supervisor LLM userMessage 사용 여부 |
| `PHOENIX_OTLP_ENDPOINT` | unset | Phoenix OTLP trace endpoint |
| `PHOENIX_LOG_PROMPTS` | false | LLM prompt/completion content trace 기록 |
@@ -519,7 +538,15 @@ ALERT_THRESHOLD_DRIFT_PCT=15
예를 들어 onboarding latency alert와 moderation latency alert는 서로 throttle하지 않는다.
skill이 없는 cross-system alert는 기존처럼 글로벌 dedupe를 사용한다.
### 16.2 Phoenix trace
### 16.2 Secret, Tool Sandbox, NATS ACL
Secret은 Infisical을 우선으로 로드한다. Infisical credential은 로컬 env 또는 IMDS에서 읽고, 새 secret env를 추가할 때는 `src/common/known-secret-keys.ts`에 함께 등록해야 로그 redaction 대상에 들어간다. LLM, embedding, Phoenix API key도 redaction 대상이다.
Tool은 `requires` 선언으로 필요한 context만 받는다. `AGENT_TOOL_SANDBOX_ENABLED=false`가 기본이라 운영 초기는 호환성을 유지한다. 1주 정도 audit을 본 뒤 false positive가 없으면 true로 전환한다.
NATS subject ACL은 `src/common/nats-skill-acl.ts`의 `DEFAULT_NATS_ACL`에서 관리한다. 기본 `AGENT_NATS_ACL_MODE=log-only`는 위반을 warn으로만 남기고 publish/request는 그대로 진행한다. 1개월 운영 후 false positive가 없으면 `enforce`로 전환한다.
### 16.3 Phoenix trace
LLM 호출, skill 실행, tool 실행 span은 OpenTelemetry trace로 남긴다.
`PHOENIX_OTLP_ENDPOINT`가 없으면 기존 설정을 따른다. 개발 환경 기본값은 trace exporter 비활성화이고, `OTEL_EXPORTER=stdout`이면 콘솔 exporter를 쓴다.
@@ -544,7 +571,7 @@ Phoenix UI에서는 `service.name=lolympus-agent`, `gen_ai.system`, `gen_ai.requ
Prompt와 completion 본문은 기본적으로 기록하지 않는다.
디버깅 때문에 필요할 때만 `PHOENIX_LOG_PROMPTS=true`를 켠다. 이 경우 email, 전화번호, token/API key 형태의 값은 span 기록 전에 masking된다.
### 16.3 Prometheus scrape
### 16.4 Prometheus scrape
Agent process는 `METRICS_PORT`에서 HTTP endpoint를 연다.
@@ -596,7 +623,7 @@ sum by (skill, stage) (rate(agent_validator_fail_total[5m]))
| eval drift | 15% 이하 |
| archive stalled | 3회 연속 실패 0건 |
### 16.4 Slack webhook 설정
### 16.5 Slack webhook 설정
Slack incoming webhook URL을 env로 설정한다.
File diff suppressed because it is too large. Load diff
+26 -10
View File
@@ -16,20 +16,36 @@ if ($isMacOS -or $isLinux) {
$env:ENV_FILE = "$PSScriptRoot\env-vars\$env:NODE_ENV.env"
}
# 환경 변수 파일 확인
if (-Not (Test-Path $env:ENV_FILE)) {
Write-Error "The file '$env:ENV_FILE' does not exist."
exit 1
}
$GLOBAL_ENV_FILE = Join-Path $HOME "source/global-env-vars.txt"
# 'dahyeon-local.env' 파일에서 환경 변수 설정
Get-Content $env:ENV_FILE | ForEach-Object {
if ($_ -match '^\s*([^#]+?)\s*=\s*(.*)$') {
$key, $value = $matches[1].Trim(), $matches[2].Trim().Trim("`"")
[System.Environment]::SetEnvironmentVariable($key, $value)
function Import-EnvFileIfPresent {
param(
[string]$Path,
[switch]$Required
)
if (-Not (Test-Path $Path)) {
if ($Required) {
Write-Error "The file '$Path' does not exist."
exit 1
}
return
}
Get-Content $Path | ForEach-Object {
if ($_ -match '^\s*([^#]+?)\s*=\s*(.*)$') {
$key, $value = $matches[1].Trim(), $matches[2].Trim().Trim("`"")
if (-not [System.Environment]::GetEnvironmentVariable($key)) {
[System.Environment]::SetEnvironmentVariable($key, $value)
}
}
}
}
# 환경 변수 파일 확인
Import-EnvFileIfPresent -Path $GLOBAL_ENV_FILE
Import-EnvFileIfPresent -Path $env:ENV_FILE -Required
Get-ChildItem Env: | Where-Object {
$_.Name -eq "NODE_ENV" -or
$_.Name -like "IMDS_*" -or
+2
View File
@@ -36,6 +36,8 @@
"review:show": "node --import tsx src/operations/cli.ts review:show",
"review:resolve": "node --import tsx src/operations/cli.ts review:resolve",
"review:stats": "node --import tsx src/operations/cli.ts review:stats",
"tool-audit:list": "node --import tsx src/operations/cli.ts tool-audit:list",
"tool-audit:show": "node --import tsx src/operations/cli.ts tool-audit:show",
"start:dev": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'dev'); \"./launch-server.ps1\"\"",
"start:test": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'test'); \"./launch-server.ps1\"\"",
"start:prod": "pwsh -Command \"[System.Environment]::SetEnvironmentVariable('NODE_ENV', 'prod'); \"./launch-server.ps1\"\"",
+25
View File
@@ -59,6 +59,30 @@ describe("BudgetTracker", () => {
expect(costBudget.assertBeforeHandoff("supervisor")).toBe("cost-budget");
});
it("tracks LLM and tool costs separately while enforcing the total budget", () => {
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 0.0055,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
for (let index = 0; index < 5; index += 1) {
budget.recordLlmCost(0.001);
}
budget.recordToolCall(0.0005);
expect(budget.snapshot()).toMatchObject({
estimatedCostUsd: 0.0055,
totalCostUsd: 0.0055,
totalLlmCostUsd: 0.005,
totalToolCostUsd: 0.0005,
toolCalls: 1,
});
expect(budget.assertBeforeHandoff("supervisor")).toBe("cost-budget");
});
it("can be disabled", () => {
const budget = new BudgetTracker({
enabled: false,
@@ -71,6 +95,7 @@ describe("BudgetTracker", () => {
budget.recordVisit("supervisor");
budget.recordHandoff("moderation");
budget.recordToolCall(1);
budget.recordLlmCost(1);
expect(budget.assertBeforeHandoff("supervisor")).toBe("ok");
});
+23 -4
View File
@@ -20,13 +20,17 @@ export interface BudgetSnapshot {
maxHandoffs: number;
maxSameAgentVisits: number;
maxToolCalls: number;
totalCostUsd: number;
totalLlmCostUsd: number;
totalToolCostUsd: number;
toolCalls: number;
visits: Record<string, number>;
}
export class BudgetTracker {
private estimatedCostUsd = 0;
private handoffs = 0;
private totalLlmCostUsd = 0;
private totalToolCostUsd = 0;
private toolCalls = 0;
private readonly visits = new Map<string, number>();
@@ -53,7 +57,7 @@ export class BudgetTracker {
if (this.toolCalls >= this.limits.maxToolCalls) {
return "tool-budget";
}
if (this.estimatedCostUsd >= this.limits.maxCostUsd) {
if (this.totalCostUsd() >= this.limits.maxCostUsd) {
return "cost-budget";
}
return "ok";
@@ -66,7 +70,14 @@ export class BudgetTracker {
recordToolCall(estimatedCostUsd = 0): void {
this.toolCalls += 1;
this.estimatedCostUsd += estimatedCostUsd;
this.totalToolCostUsd += estimatedCostUsd;
}
recordLlmCost(costUsd: number): void {
if (!Number.isFinite(costUsd) || costUsd <= 0) {
return;
}
this.totalLlmCostUsd += costUsd;
}
recordVisit(target: string): void {
@@ -74,15 +85,23 @@ export class BudgetTracker {
}
snapshot(): BudgetSnapshot {
const totalCostUsd = this.totalCostUsd();
return {
estimatedCostUsd: Number(this.estimatedCostUsd.toFixed(6)),
estimatedCostUsd: Number(totalCostUsd.toFixed(6)),
handoffs: this.handoffs,
maxCostUsd: this.limits.maxCostUsd,
maxHandoffs: this.limits.maxHandoffs,
maxSameAgentVisits: this.limits.maxSameAgentVisits,
maxToolCalls: this.limits.maxToolCalls,
totalCostUsd: Number(totalCostUsd.toFixed(6)),
totalLlmCostUsd: Number(this.totalLlmCostUsd.toFixed(6)),
totalToolCostUsd: Number(this.totalToolCostUsd.toFixed(6)),
toolCalls: this.toolCalls,
visits: Object.fromEntries(this.visits.entries()),
};
}
private totalCostUsd(): number {
return this.totalLlmCostUsd + this.totalToolCostUsd;
}
}
+1 -1
View File
@@ -90,7 +90,7 @@ export async function dispatch(
return runWithContext(
{
...contextBase,
budget: budget.snapshot(),
budget,
handoffChain: chain,
handoffDepth: depth,
},
+11
View File
@@ -1,4 +1,5 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import * as dotenv from "dotenv";
@@ -34,6 +35,11 @@ function resolveEnvFile(mode: NodeEnvMode): string | null {
return null;
}
function resolveGlobalEnvFile(): string | null {
const candidate = path.resolve(os.homedir(), "source", "global-env-vars.txt");
return fs.existsSync(candidate) ? candidate : null;
}
function loadDotenvFile(filePath: string, label: string): void {
const result = dotenv.config({ path: filePath, override: false });
if (result.error) {
@@ -48,6 +54,11 @@ export function loadEnvironmentFromNodeEnv(): void {
const mode = normalizeMode(process.env["NODE_ENV"]);
process.env["NODE_ENV"] = mode;
const globalEnvFile = resolveGlobalEnvFile();
if (globalEnvFile) {
loadDotenvFile(globalEnvFile, "global-env-vars.txt");
}
const envFile = resolveEnvFile(mode);
if (!envFile) {
logger.warn(`[Env] env-vars/${mode}.env not found. Skipping dotenv load.`);
+3 -2
View File
@@ -1,9 +1,10 @@
import { GarnetClient } from "./garnet.js";
import type { GarnetClient as GarnetClientType } from "./garnet.js";
export async function connectGarnetForCli(): Promise<{
close: () => Promise<void>;
garnetClient: GarnetClient;
garnetClient: GarnetClientType;
}> {
const { GarnetClient } = await import("./garnet.js");
const garnetClient = new GarnetClient();
await garnetClient.connect();
return {
+5
View File
@@ -9,6 +9,11 @@ export const KNOWN_SECRET_KEYS = [
"GARNET_PASSWORD",
"MONGODB_CONNECTION_STRING",
"MONGODB_PROJECT_ID",
"OPENROUTER_API_KEY",
"AGENT_LLM_API_KEY",
"OPENAI_API_KEY",
"EMBEDDING_API_KEY",
"PHOENIX_API_KEY",
"SLACK_ALERT_WEBHOOK_URL",
] as const;
+41
View File
@@ -0,0 +1,41 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { runWithContext } from "../observability/context.js";
import { NatsClient } from "./nats.js";
import { AGENT_ALERT_SUBJECT, DISCORD_MEMBER_LOOKUP_SUBJECT } from "./nats-subjects.js";
import { checkPublishAllowed } from "./nats-skill-acl.js";
describe("NATS skill ACL", () => {
afterEach(() => {
vi.unstubAllEnvs();
});
it("allows declared publish subjects", () => {
expect(checkPublishAllowed("onboarding", DISCORD_MEMBER_LOOKUP_SUBJECT)).toBe(
true,
);
expect(checkPublishAllowed("supervisor", AGENT_ALERT_SUBJECT)).toBe(true);
expect(checkPublishAllowed("onboarding", AGENT_ALERT_SUBJECT)).toBe(false);
});
it("enforces denied publish subjects when enabled", async () => {
vi.stubEnv("AGENT_NATS_ACL_MODE", "enforce");
const client = new NatsClient();
await expect(
runWithContext(
{
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() => client.publish(AGENT_ALERT_SUBJECT, { ok: true }),
),
).rejects.toThrow(
`NATS publish denied for skill onboarding on subject ${AGENT_ALERT_SUBJECT}`,
);
});
});
+54
View File
@@ -0,0 +1,54 @@
import {
AGENT_ALERT_SUBJECT,
DISCORD_MEMBER_LOOKUP_SUBJECT,
} from "./nats-subjects.js";
export interface SkillNatsAcl {
skill: string;
allowedPublishSubjects: string[];
allowedSubscribeSubjects: string[];
}
export const DEFAULT_NATS_ACL: SkillNatsAcl[] = [
{
skill: "onboarding",
allowedPublishSubjects: [DISCORD_MEMBER_LOOKUP_SUBJECT],
allowedSubscribeSubjects: [],
},
{
skill: "moderation",
allowedPublishSubjects: [],
allowedSubscribeSubjects: [],
},
{
skill: "supervisor",
allowedPublishSubjects: [AGENT_ALERT_SUBJECT],
allowedSubscribeSubjects: [],
},
];
function wildcardToRegExp(pattern: string): RegExp {
const escaped = pattern
.split("*")
.map((part) => part.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"))
.join(".*");
return new RegExp(`^${escaped}$`, "u");
}
function matchesAny(subject: string, patterns: readonly string[]): boolean {
return patterns.some((pattern) => wildcardToRegExp(pattern).test(subject));
}
function getAcl(skill: string): SkillNatsAcl | undefined {
return DEFAULT_NATS_ACL.find((entry) => entry.skill === skill);
}
export function checkPublishAllowed(skill: string, subject: string): boolean {
const acl = getAcl(skill);
return Boolean(acl && matchesAny(subject, acl.allowedPublishSubjects));
}
export function checkSubscribeAllowed(skill: string, subject: string): boolean {
const acl = getAcl(skill);
return Boolean(acl && matchesAny(subject, acl.allowedSubscribeSubjects));
}
+21
View File
@@ -34,6 +34,12 @@ export type DiscordMemberMatchSource =
| "username"
| "joined-after";
export type DiscordResidentApplicationMessageMatchSource =
| "author-id"
| "author-name"
| "content"
| "joined-after";
export interface DiscordMemberCandidate {
memberId: string;
joinedAt: string;
@@ -43,8 +49,20 @@ export interface DiscordMemberCandidate {
username: string;
}
export interface DiscordResidentApplicationMessageCandidate {
authorDisplayName?: string | undefined;
authorGuildNickname?: string | undefined;
authorId: string;
channelId: string;
content: string;
createdAt: string;
matchedBy?: DiscordResidentApplicationMessageMatchSource | undefined;
messageId: string;
}
export interface DiscordMemberLookupRequest {
chatroomId: string;
includeResidentApplicationMessages?: boolean | undefined;
joinedAfter: string;
searchName?: string | undefined;
userId: string;
@@ -55,6 +73,9 @@ export type DiscordMemberLookupResponse =
status: "ok";
joinedAfterMatches: DiscordMemberCandidate[];
nicknameMatches: DiscordMemberCandidate[];
residentApplicationMessageMatches?:
| DiscordResidentApplicationMessageCandidate[]
| undefined;
}
| {
status: "error";
+39
View File
@@ -6,7 +6,13 @@ import {
type Subscription,
} from "nats";
import { getSecret } from "@common/secret-manager";
import { getNatsAclMode } from "../config.js";
import { getContext } from "../observability/context.js";
import { getLogger } from "@obs/logger";
import {
checkPublishAllowed,
checkSubscribeAllowed,
} from "./nats-skill-acl.js";
export interface NatsMessageContext<T = unknown> {
data: T;
@@ -81,6 +87,7 @@ export class NatsClient {
subject: string,
handler: (msg: Msg) => void | Promise<void>,
): Promise<Subscription> {
this.checkSubjectAcl("subscribe", subject);
const subscription = this.getConnection().subscribe(subject);
void (async () => {
@@ -101,6 +108,7 @@ export class NatsClient {
}
async publish<T = unknown>(subject: string, data: T): Promise<void> {
this.checkSubjectAcl("publish", subject);
const client = this.getConnection();
client.publish(subject, this.stringCodec.encode(JSON.stringify(data)));
await client.flush();
@@ -111,6 +119,7 @@ export class NatsClient {
data: TRequest,
options?: { timeoutMs?: number | undefined },
): Promise<TResponse> {
this.checkSubjectAcl("publish", subject);
const requestOptions =
options?.timeoutMs !== undefined
? {
@@ -126,6 +135,36 @@ export class NatsClient {
return this.decodeMessage<TResponse>(response);
}
private checkSubjectAcl(
operation: "publish" | "subscribe",
subject: string,
): void {
const mode = getNatsAclMode();
if (mode === "off") {
return;
}
const skill = getContext()?.skill;
if (!skill) {
return;
}
const allowed =
operation === "publish"
? checkPublishAllowed(skill, subject)
: checkSubscribeAllowed(skill, subject);
if (allowed) {
return;
}
const payload = { mode, operation, skill, subject };
if (mode === "enforce") {
this.logger.error(payload, "[NATS ACL] Subject denied");
throw new Error(
`NATS ${operation} denied for skill ${skill} on subject ${subject}`,
);
}
this.logger.warn(payload, "[NATS ACL] Subject would be denied");
}
respond(msg: Msg, data: unknown): Promise<void> {
if (!msg.reply) {
this.logger.warn(
+118
View File
@@ -5,10 +5,20 @@ import {
getAlertThresholdEvalScoreDropPct,
getAlertThresholdLlmErrorRate,
getAlertThresholdP95LatencyMs,
getAgentBudgetLlmCostEnabled,
getAgentCostAwareTierEnabled,
getAgentSupervisorFastPathEnabled,
getAgentToolSandboxEnabled,
getEmbeddingModel,
getEmbeddingProvider,
getEvalCaseDeadlineMs,
getEvalLlmProvider,
getEvalRuntimeLlmEnabled,
getLongTermSummaryEnabled,
getLlmModelRoundRobinEnabled,
getRuntimeLlmModelsForProvider,
getMemoryEvictionEnabled,
getNatsAclMode,
getPhoenixLogPrompts,
getPhoenixOtlpEndpoint,
getRagHybridEnabled,
@@ -17,7 +27,9 @@ import {
getRagEnabled,
getSlackAlertWebhookUrl,
getSlackWebhookTimeoutMs,
getSupervisorFastPathConfidence,
getSupervisorLlmResponseEnabled,
getToolAuditMaxLen,
} from "./config.js";
const ENV_KEYS = [
@@ -35,6 +47,13 @@ const ENV_KEYS = [
"AGENT_RAG_RERANKER_TOP_K",
"AGENT_LONG_TERM_SUMMARY_ENABLED",
"AGENT_MEMORY_EVICTION_ENABLED",
"AGENT_BUDGET_LLM_COST_ENABLED",
"AGENT_COST_AWARE_TIER_ENABLED",
"AGENT_SUPERVISOR_FAST_PATH_ENABLED",
"SUPERVISOR_FAST_PATH_CONFIDENCE",
"AGENT_TOOL_SANDBOX_ENABLED",
"AGENT_TOOL_AUDIT_MAX_LEN",
"AGENT_NATS_ACL_MODE",
"EMBEDDING_MODEL",
"EMBEDDING_PROVIDER",
"PHOENIX_LOG_PROMPTS",
@@ -42,6 +61,12 @@ const ENV_KEYS = [
"SLACK_ALERT_WEBHOOK_URL",
"SLACK_WEBHOOK_TIMEOUT_MS",
"AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED",
"CLIPROXY_TIER1_PLANNER_MODEL",
"EVAL_CASE_DEADLINE_MS",
"EVAL_LLM_PROVIDER",
"EVAL_RUNTIME_LLM_ENABLED",
"LLM_MODEL_ROUND_ROBIN_ENABLED",
"OPENROUTER_TIER1_RESPONDER_MODEL",
] as const;
describe("config", () => {
@@ -129,4 +154,97 @@ describe("config", () => {
expect(getSupervisorLlmResponseEnabled()).toBe(false);
});
it("reads operational excellence guardrail configuration", () => {
expect(getAgentBudgetLlmCostEnabled()).toBe(true);
expect(getAgentCostAwareTierEnabled()).toBe(true);
expect(getAgentSupervisorFastPathEnabled()).toBe(true);
expect(getSupervisorFastPathConfidence()).toBe(0.85);
expect(getAgentToolSandboxEnabled()).toBe(false);
expect(getToolAuditMaxLen()).toBe(100_000);
expect(getNatsAclMode()).toBe("log-only");
vi.stubEnv("AGENT_BUDGET_LLM_COST_ENABLED", "false");
vi.stubEnv("AGENT_COST_AWARE_TIER_ENABLED", "false");
vi.stubEnv("AGENT_SUPERVISOR_FAST_PATH_ENABLED", "false");
vi.stubEnv("SUPERVISOR_FAST_PATH_CONFIDENCE", "0.9");
vi.stubEnv("AGENT_TOOL_SANDBOX_ENABLED", "true");
vi.stubEnv("AGENT_TOOL_AUDIT_MAX_LEN", "42");
vi.stubEnv("AGENT_NATS_ACL_MODE", "enforce");
expect(getAgentBudgetLlmCostEnabled()).toBe(false);
expect(getAgentCostAwareTierEnabled()).toBe(false);
expect(getAgentSupervisorFastPathEnabled()).toBe(false);
expect(getSupervisorFastPathConfidence()).toBe(0.9);
expect(getAgentToolSandboxEnabled()).toBe(true);
expect(getToolAuditMaxLen()).toBe(42);
expect(getNatsAclMode()).toBe("enforce");
});
it("reads eval case deadline configuration", () => {
expect(getEvalCaseDeadlineMs()).toBe(4_500);
vi.stubEnv("EVAL_CASE_DEADLINE_MS", "45000");
expect(getEvalCaseDeadlineMs()).toBe(45_000);
});
it("enables eval runtime LLM by default with env override", () => {
expect(getEvalRuntimeLlmEnabled()).toBe(true);
vi.stubEnv("EVAL_RUNTIME_LLM_ENABLED", "false");
expect(getEvalRuntimeLlmEnabled()).toBe(false);
});
it("pins eval LLM provider to cliproxy by default", () => {
expect(getEvalLlmProvider()).toBe("cliproxy");
vi.stubEnv("EVAL_LLM_PROVIDER", "all");
expect(getEvalLlmProvider()).toBe("all");
vi.stubEnv("EVAL_LLM_PROVIDER", "openrouter");
expect(getEvalLlmProvider()).toBe("openrouter");
vi.stubEnv("EVAL_LLM_PROVIDER", "invalid");
expect(getEvalLlmProvider()).toBe("cliproxy");
});
it("reads provider-scoped runtime LLM models", () => {
vi.stubEnv("CLIPROXY_TIER1_PLANNER_MODEL", "cliproxy-planner");
vi.stubEnv(
"OPENROUTER_TIER1_RESPONDER_MODEL",
"openrouter-fast, openrouter-backup",
);
expect(
getRuntimeLlmModelsForProvider(
"cliproxy",
{ role: "planner", tier: 1 },
"local-qwen",
),
).toEqual(["cliproxy-planner"]);
expect(
getRuntimeLlmModelsForProvider(
"openrouter",
{ role: "responder", tier: 1 },
"local-qwen",
),
).toEqual(["openrouter-fast", "openrouter-backup"]);
expect(
getRuntimeLlmModelsForProvider(
"llama",
{ role: "responder", tier: 1 },
"local-qwen",
),
).toEqual(["local-qwen"]);
});
it("keeps model round-robin disabled by default", () => {
expect(getLlmModelRoundRobinEnabled()).toBe(false);
vi.stubEnv("LLM_MODEL_ROUND_ROBIN_ENABLED", "true");
expect(getLlmModelRoundRobinEnabled()).toBe(true);
});
});
+162 -8
View File
@@ -77,6 +77,23 @@ function getBoolean(envKey: string, fallback: boolean): boolean {
return fallback;
}
function getStringListFromEnv(...envKeys: string[]): string[] | undefined {
for (const envKey of envKeys) {
const raw = process.env[envKey]?.trim();
if (!raw) continue;
const values = raw
.split(/[\n,]/u)
.map((value) => value.trim())
.filter((value) => value.length > 0);
if (values.length > 0) {
return values;
}
}
return undefined;
}
export function getNodeEnv(): "dev" | "test" | "prod" {
const value = process.env["NODE_ENV"]?.trim();
return value === "test" || value === "prod" ? value : "dev";
@@ -231,6 +248,38 @@ export function getAgentBudgetMaxCostUsd(): number {
return getPositiveNumber("AGENT_BUDGET_MAX_COST_USD", 0.5);
}
export function getAgentBudgetLlmCostEnabled(): boolean {
return getBoolean("AGENT_BUDGET_LLM_COST_ENABLED", true);
}
export function getAgentCostAwareTierEnabled(): boolean {
return getBoolean("AGENT_COST_AWARE_TIER_ENABLED", true);
}
export function getAgentSupervisorFastPathEnabled(): boolean {
return getBoolean("AGENT_SUPERVISOR_FAST_PATH_ENABLED", true);
}
export function getSupervisorFastPathConfidence(): number {
return getRatioNumber("SUPERVISOR_FAST_PATH_CONFIDENCE", 0.85);
}
export function getAgentToolSandboxEnabled(): boolean {
return getBoolean("AGENT_TOOL_SANDBOX_ENABLED", false);
}
export function getToolAuditMaxLen(): number {
return getPositiveNumber("AGENT_TOOL_AUDIT_MAX_LEN", 100_000);
}
export function getNatsAclMode(): "off" | "log-only" | "enforce" {
const raw = process.env["AGENT_NATS_ACL_MODE"]?.trim().toLowerCase();
if (raw === "off" || raw === "log-only" || raw === "enforce") {
return raw;
}
return "log-only";
}
export function getAgentRuntimeVerifierEnabled(): boolean {
return getBoolean("AGENT_RUNTIME_VERIFIER_ENABLED", false);
}
@@ -358,6 +407,23 @@ export function getSupervisorLlmResponseEnabled(): boolean {
return getBoolean("AGENT_SUPERVISOR_LLM_RESPONSE_ENABLED", true);
}
export type EvalLlmProvider = "all" | "cliproxy" | "llama" | "openrouter";
const EVAL_LLM_PROVIDERS = new Set<EvalLlmProvider>([
"all",
"cliproxy",
"llama",
"openrouter",
]);
export function getEvalLlmProvider(): EvalLlmProvider {
const raw = process.env["EVAL_LLM_PROVIDER"]?.trim().toLowerCase();
if (raw && EVAL_LLM_PROVIDERS.has(raw as EvalLlmProvider)) {
return raw as EvalLlmProvider;
}
return "cliproxy";
}
export function getEvalCalibrationMinCorr(): number {
return getRatioNumber("EVAL_CALIBRATION_MIN_CORR", 0.5);
}
@@ -483,6 +549,10 @@ export function getLlmCircuitCooldownMs(): number {
return getPositiveNumber("LLM_CIRCUIT_COOLDOWN_MS", 60_000);
}
export function getLlmModelRoundRobinEnabled(): boolean {
return getBoolean("LLM_MODEL_ROUND_ROBIN_ENABLED", false);
}
export function getClassifierMaxTokens(): number {
return getPositiveNumber("CLASSIFIER_MAX_TOKENS", 16);
}
@@ -519,6 +589,14 @@ export function getEvalPrmMinLabels(): number {
return getPositiveNumber("EVAL_PRM_MIN_LABELS", 500);
}
export function getEvalCaseDeadlineMs(): number {
return getPositiveNumber("EVAL_CASE_DEADLINE_MS", 4_500);
}
export function getEvalRuntimeLlmEnabled(): boolean {
return getBoolean("EVAL_RUNTIME_LLM_ENABLED", true);
}
export function getEvalAttributionTickMs(): number {
return getPositiveNumber("EVAL_ATTRIBUTION_TICK_MS", 6 * 60 * 60 * 1000);
}
@@ -574,15 +652,91 @@ export function getTier2CriticModel(): string {
}
export function getOpenRouterModels(): string[] {
const raw = process.env["OPENROUTER_MODEL"]?.trim();
if (!raw) {
return ["openai/gpt-5.3-codex-spark"];
return (
getStringListFromEnv("OPENROUTER_MODEL", "OPENROUTER_MODELS") ?? [
"openai/gpt-5.3-codex-spark",
]
);
}
type RuntimeLlmProvider = "cliproxy" | "llama" | "openrouter";
type RuntimeLlmRole = "critic" | "planner" | "responder" | undefined;
type RuntimeLlmTier = 0 | 1 | 2 | undefined;
function getRuntimeRouteSuffix(args: {
role?: RuntimeLlmRole;
tier?: RuntimeLlmTier;
}): string {
if (args.role === "planner") {
return "TIER1_PLANNER";
}
const models = raw
.split(",")
.map((value) => value.trim())
.filter((value) => value.length > 0);
return models.length > 0 ? models : ["openai/gpt-5.3-codex-spark"];
if (args.role === "critic") {
return "TIER2_CRITIC";
}
if (args.tier === 0) {
return "TIER0";
}
return "TIER1_RESPONDER";
}
function getProviderPrefix(provider: RuntimeLlmProvider): string {
return provider.toUpperCase();
}
function getProviderDefaultModels(
provider: RuntimeLlmProvider,
llamaFallbackModel: string,
): string[] {
if (provider === "cliproxy") {
return (
getStringListFromEnv(
"CLIPROXY_MODEL",
"CLIPROXY_MODELS",
"CLI_PROXY_LLM_MODEL",
) ?? ["gpt-5.3-codex-spark"]
);
}
if (provider === "openrouter") {
return getOpenRouterModels();
}
return (
getStringListFromEnv("LLAMA_MODEL", "LLAMA_MODELS", "AGENT_LLM_MODEL") ?? [
llamaFallbackModel,
]
);
}
export function getRuntimeLlmModelsForProvider(
provider: RuntimeLlmProvider,
args: {
role?: RuntimeLlmRole;
tier?: RuntimeLlmTier;
},
llamaFallbackModel: string,
): string[] {
const suffix = getRuntimeRouteSuffix(args);
const providerPrefix = getProviderPrefix(provider);
const providerScopedModels = getStringListFromEnv(
`${providerPrefix}_${suffix}_MODEL`,
`${providerPrefix}_${suffix}_MODELS`,
);
if (providerScopedModels) {
return providerScopedModels;
}
if (provider === "cliproxy") {
const legacyModels = getStringListFromEnv(
`${suffix}_MODEL`,
`${suffix}_MODELS`,
);
if (legacyModels) {
return legacyModels;
}
}
return getProviderDefaultModels(provider, llamaFallbackModel);
}
export function getLlmMaxTokens(): number {
@@ -138,3 +138,63 @@
{"id":"case_acd612e0","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778740258770-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_7169cd5a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_81f2ad8f","nickname":"<nick:ba29bb75>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_6cd7106c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778718965990-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse={\"reply\":\"안녕하세요!\"}","input":{"request":{"requestId":"req_1477394f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_24cff593","nickname":"<nick:d455d0e4>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_de51735e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778674030965-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_32cc22d6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_cd95eff0","nickname":"<nick:7feae321>","trigger":"user-message","userMessage":"안녕하세오"}},"expected":{}}
{"id":"case_4f45294b","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778756106669-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 여기 채팅방에 올리면 돼요. \n채팅방 오른쪽 상단 메뉴의 '커버 보기'로 들어가 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷으로 보내면 됩니다.","input":{"request":{"requestId":"req_e6e21e71","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"하트인증\n여따 올리면 되요?"}},"expected":{}}
{"id":"case_e3ff18c6","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778756063255-0; llmOverall=2.75; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 확인했어요. 천천히 진행해 주세요.","input":{"request":{"requestId":"req_7b05c872","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"네"}},"expected":{}}
{"id":"case_1b906220","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778763820454-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임 순서는 이름/출생연도/성별/티어로 맞게 돼 있고, 안 되는 부분은 마지막 티어 코드뿐이에요. \n현재의 '플'은 허용 티어 코드가 아니므로 GM, I, B, S, G, P, E, D, M, C, U 중 한 글자(예: D)로 바꿔주세요.","input":{"request":{"requestId":"req_a8ba4ffe","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_e3e5b006","nickname":"<nick:405abf95>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_4ff31f85","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778756157817-0; llmOverall=n/a; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 확인했어요. 천천히 진행해 주세요.","input":{"request":{"requestId":"req_c78edc0e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0dd56d1c","nickname":"<nick:7724addd>","trigger":"user-message","userMessage":"오"}},"expected":{}}
{"id":"case_0fd45691","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778874307141-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=닉네임이 안 되는 이유는 형식이 맞지 않아서예요. \n현재 \"위례\"는 이름만 있고, 연도/성별/티어가 없어 형식이 깨졌습니다. \n카카오톡 닉네임은 [이름]/[출생연도 두 자리]/[남 또는 여]/[티어]로 바꿔주세요.","input":{"request":{"requestId":"req_b6f69c24","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:16fb6789>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_408c317c","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778874400236-0; llmOverall=2.25; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=채팅방 오른쪽 상단 메뉴 버튼을 눌러 “커버 보기”로 들어가면 하트 인증 화면이 보여요.","input":{"request":{"requestId":"req_1db51665","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"하트인증은 어디다 하나요?"}},"expected":{}}
{"id":"case_cf22a9d8","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778874533535-0; llmOverall=2.5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=좋아요, 천천히 진행해 주세요.","input":{"request":{"requestId":"req_60f5e74a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"넵 감사합니다"}},"expected":{}}
{"id":"case_fd404eb4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778874508177-0; llmOverall=5; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=네, 가능합니다. 카카오톡 1단계가 완료된 상태라면 디스코드는 저녁에도 접속해 진행하시면 돼요.","input":{"request":{"requestId":"req_d814bae6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"디스코드는 저녁에 해도될까요?"}},"expected":{}}
{"id":"case_d42b8624","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778874344992-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_47d100d2","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0558bf70","nickname":"<nick:ad75d929>","trigger":"user-message","userMessage":"안녕하세여"}},"expected":{}}
{"id":"case_bf09ad3a","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778954734843-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=현재 남은 절차는 카카오톡 닉네임 변경과 방 하트 인증입니다.","input":{"request":{"requestId":"req_010646c0","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_a2e8fc0b","nickname":"<nick:a9bad7ab>","trigger":"user-message","userMessage":"혹시 이시간엔 내전 안돌아가죠?"}},"expected":{}}
{"id":"case_275402ce","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950244576-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 건 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정, 디스코드 주민신청 채널에 메시지 남기기예요.","input":{"request":{"requestId":"req_bc7c8aae","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"5대5 하려고 하는데 가능 한지 해서용"}},"expected":{}}
{"id":"case_0e56499c","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950227521-0; llmOverall=1; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 절차는 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정, 디스코드 주민신청 채널에 메시지 남기기예요.","input":{"request":{"requestId":"req_593ffaff","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"아 지금 들어와 있는 관리자 분이용"}},"expected":{}}
{"id":"case_c8398f09","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778949416363-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=웅/97/남/M님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_b0f0648f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"혹시 지 외전 문의 드리려 하는데"}},"expected":{}}
{"id":"case_0a460498","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778950169079-0; llmOverall=2; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=관리분(관리자)은 있어요.","input":{"request":{"requestId":"req_6bffed4f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"관리분 있으신가요?"}},"expected":{}}
{"id":"case_002ac417","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1778931066262-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임 \"강서/93/남\"은 티어가 빠져서 형식이 안 맞아요. \n강서/93/남은 티어가 빠져서 형식이 안 맞아요. 마지막에 티어 한 글자만 추가하면 됩니다. \n예: 강서/93/남/D","input":{"request":{"requestId":"req_ed284edb","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0d05dfd2","nickname":"<nick:1ec0deae>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_3e163e2e","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1778949203127-0; llmOverall=2.75; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임은 형식이 안 맞아요. \n카톡 닉네임은 구분자로 '/'를 써야 하고 4개 항목이 필요해요. 현재는 웅 97 남 M라 안 되고, 현재 닉네임은 웅/97/남/M 이렇게 고치면 됩니다.","input":{"request":{"requestId":"req_ab630198","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:6334c82e>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_1db9c911","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778958194603-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_f3d19239","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3e4be23b","nickname":"<nick:b09afa6a>","trigger":"user-message","userMessage":"안녕하세여"}},"expected":{}}
{"id":"case_ee7ae44a","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778955694984-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_fea5fd76","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3592a95a","nickname":"<nick:437c8fc3>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_d6bd160f","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778950199447-0; llmOverall=2.25; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=운영진께, 특히 관리자님께 문의하시면 돼요.","input":{"request":{"requestId":"req_5fbe8788","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"누구한테 여쭤봐야 할까요"}},"expected":{}}
{"id":"case_36b3ac9c","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1778949423115-0; llmOverall=n/a; derivedIntent=help-where; derivedPhase=deadline-near; assistantResponse=웅/97/남/M님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_35e659fa","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ecdda046","nickname":"<nick:e448bd8b>","trigger":"user-message","userMessage":"어디에 문의 해야 할까용"}},"expected":{}}
{"id":"case_f7a5d661","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019400869-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_4861ee12","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"user-message","userMessage":"듀랭하실분들 있을까요"}},"expected":{}}
{"id":"case_74eac586","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019394267-0; llmOverall=1.25; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_743e0456","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"user-message","userMessage":"혹시"}},"expected":{}}
{"id":"case_9e9195bb","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779019388776-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=@벽 넘어가는 탈론 00/남/E3님 닉네임에서 수정할 부분이 있어요!\n나이를 두 자리 숫자로 입력해주세요 (예: 25)\n성별(남/여)과 티어 코드를 '/'로 구분해서 입력해주세요","input":{"request":{"requestId":"req_568d875b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:444d812a>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_8acd1f91","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779011411116-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임에서 안 된 부분은 마지막 티어예요. \nG2는 허용 티어가 아니라서, 뒤의 G2를 GM, I, B, S, G, P, E, D, M, C, U 중 하나(예: G)로만 바꾸면 됩니다.","input":{"request":{"requestId":"req_9860e773","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:666d8731>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_d6824fdb","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779023514843-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_4d45f87c","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bc77847e","nickname":"<nick:9ee38f43>","trigger":"user-message","userMessage":"안녕하세요 공지읽고오겠습니다"}},"expected":{}}
{"id":"case_83136947","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779023414663-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요, 반갑습니다.","input":{"request":{"requestId":"req_d55e086e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_a9ca5eb1","nickname":"<nick:ebf85b46>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_1686cd08","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779019324629-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=현재 닉네임은 /로 구분된 4개 항목(이름/출생연도 두 자리/성별/티어)이 아니어서 안 돼요. 지금은 항목이 2개로 잘못되어 있어요. \n고쳐야 할 부분은 연도/성별/티어를 다시 /로 정확히 나눠서 넣는 거예요. \n성별은 남 또는 여 한 글자, 티어는 GM/I/B/S/G/P/E/D/M/C/U 중 한 글자여야 해요.","input":{"request":{"requestId":"req_7cb58844","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bea0f163","nickname":"<nick:23dc710b>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_0f722e08","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779015436203-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_a6a1d38f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_bc35f985","nickname":"<nick:c3de5ead>","trigger":"user-message","userMessage":"안녕하세요~"}},"expected":{}}
{"id":"case_009d9558","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779011507015-0; llmOverall=2.25; derivedIntent=status-question; derivedPhase=deadline-near; assistantResponse=남은 절차는 \n카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 서버 프로필 닉네임 설정, 디스코드 주민신청 채널 메시지 남기기 입니다.","input":{"request":{"requestId":"req_18174c18","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:94137b1e>","trigger":"user-message","userMessage":"이러면되나요"}},"expected":{}}
{"id":"case_b51a6dfb","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779011451230-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요!","input":{"request":{"requestId":"req_5d3182b0","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_40745cd2","nickname":"<nick:94137b1e>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_8f87e018","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779562992008-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=고구마는보라색/98/남/D님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_d9ff0f8b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:b410b374>","trigger":"user-message","userMessage":"봇인가요? 어케 이렇게 빠르지"}},"expected":{}}
{"id":"case_51fec5d7","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779563007004-0; llmOverall=1.25; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=고구마는보라색/98/남/D님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장, 카카오톡 구인방 입장, 디스코드 프로필 닉네임 설정과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_058c9557","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:b410b374>","trigger":"user-message","userMessage":"아하"}},"expected":{}}
{"id":"case_47db12bc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779544865402-0; llmOverall=1.25; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=방 하트 인증은 채팅방 오른쪽 상단 메뉴 버튼에서 \"커버 보기\"를 눌러 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷을 올려주시면 돼요.","input":{"request":{"requestId":"req_887be307","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_e35470e6","nickname":"<nick:01e9e582>","trigger":"user-message","userMessage":"어떻게 인증하나요 ?"}},"expected":{}}
{"id":"case_8aa3d0bc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779574878821-0; llmOverall=2; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=아직 승인해드리기 어려워요. 남은 항목이 카카오톡 본인인증방 입장과 디스코드 주민신청 채널 메시지라서, 이 두 가지만 먼저 완료해주시면 될 것 같아요.","input":{"request":{"requestId":"req_117cdf74","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"디코 승인 부탁드려도 될까요?"}},"expected":{}}
{"id":"case_0191603e","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779574708797-0; llmOverall=2; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=디스코드 관련 안내입니다. 디스코드 닉네임은 닉네임#태그/출생 연도 두 자리/티어 형식으로 수정해 주세요. 예시: 할리#프렌치불독/88/D \n디스코드 서버에서 이 닉네임은 본인 식별용이라 꼭 맞춰야 하고, 주민신청 채널에는 서버 소개 메시지도 남겨야 해요.","input":{"request":{"requestId":"req_17d4f38d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"/디코"}},"expected":{}}
{"id":"case_214e18f8","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779562888510-0; llmOverall=2; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=@고구마는보라색/98/D/남님 닉네임에서 수정할 부분이 있어요!\n성별을 '남' 또는 '여'로 입력해주세요\n티어 코드(GM, I, B, S, G, P, E, D, M, C, U 중 하나)를 입력해주세요","input":{"request":{"requestId":"req_dd3e072b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ab97b973","nickname":"<nick:f7254b29>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_b90d32a5","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535490115-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=네, 다시 보내주시면 돼요. \n방에서 오른쪽 상단 메뉴 버튼 열고 커버 보기로 들어간 뒤 하트를 눌러 빨간색으로 바꿔주세요. \n그 상태 화면을 채팅방에서 바로 캡처해서 편집(자르기/필터/화면 편집) 없이 원본 그대로 보내주시면 됩니다. \n참고로 회색 하트면 아직 인증으로 안 된 상태예요.","input":{"request":{"requestId":"req_8ac4ab90","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_dc095d27","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535154924-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=네, 다시 시도할 수 있어요. \n방 하트를 빨간색으로 바꾼 뒤, 채팅방에서 바로 캡처한 원본 화면(편집/자르기/필터 없음)만 새로 보내주시면 됩니다.","input":{"request":{"requestId":"req_bb9360ad","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_c33b0453","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779535131374-0; llmOverall=2; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 하트는 눌렀다면 하트가 빨간색인 화면을 편집 없이 원본 스크린샷으로 올려주시면 돼요.","input":{"request":{"requestId":"req_5c1b0e13","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_7d8e9f5b","nickname":"<nick:949aeac9>","trigger":"user-message","userMessage":"하트는 눌렀는대.."}},"expected":{}}
{"id":"case_955dc7df","lineNumber":0,"name":"promotion:rule-llm-disagree","notes":"promotionReason=rule-llm-disagree; sourceStreamId=1779605646714-0; llmOverall=2.5; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=오케이, 하트 인증은 화면이 제대로 올라오면 바로 확인돼요. \n지금 올리신 걸로 보냈다면 잠깐 확인 기다려볼게요. \n만약 아직 안 보인다면, 하트를 빨간색으로 바꾼 그대로의 원본 캡처만 한 번 더 보내주세요.","input":{"request":{"requestId":"req_fb38ace6","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0ea265d7","nickname":"<nick:7a45b00c>","trigger":"user-message","userMessage":"위에 인증햇어여"}},"expected":{}}
{"id":"case_e75e191e","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779604227776-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=photo-rejected; assistantResponse=다시 시도하려면 이렇게 하면 돼요.\n\n채팅방 오른쪽 상단 메뉴 버튼에서 커버 보기로 들어가서 하트를 빨간색으로 바꾼 뒤, \n그 장면이 보이게 편집 없이 원본 화면 캡처로 다시 보내주면 돼요.","input":{"request":{"requestId":"req_4368c115","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_0ea265d7","nickname":"<nick:d0f3633e>","trigger":"photo-failed"}},"expected":{}}
{"id":"case_f485439f","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779593282207-0; llmOverall=n/a; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=Summer/95/남/S님 카카오톡 절차는 완료됐고, 이제 카카오톡 본인인증방 입장과 디스코드 주민신청 채널에 메시지 남기기만 진행하시면 돼요!","input":{"request":{"requestId":"req_a33cd695","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:7ae6ee97>","trigger":"user-message","userMessage":"가입했습니다"}},"expected":{}}
{"id":"case_75bc3458","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779593274596-0; llmOverall=n/a; derivedIntent=discord-question; derivedPhase=deadline-near; assistantResponse=Summer/95/남/S님 이제 추가 절차만 남았어요. 카카오톡 본인인증방 입장과 디스코드 주민신청 채널에 메시지 남기기을 확인해 주세요.","input":{"request":{"requestId":"req_5bb3e408","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:7ae6ee97>","trigger":"user-message","userMessage":"디스코드"}},"expected":{}}
{"id":"case_c1016e76","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779592708891-0; llmOverall=5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 천천히 하세요.","input":{"request":{"requestId":"req_179a599b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_18313eb7","nickname":"<nick:50cd33f9>","trigger":"user-message","userMessage":"엇 잠시만용"}},"expected":{}}
{"id":"case_12f3372d","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779574484786-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=네, 바로 여기 채팅방에 올리면 됩니다. \n채팅방 오른쪽 상단 메뉴의 \"커버 보기\"에서 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 스크린샷으로 보내주세요.","input":{"request":{"requestId":"req_b7b755bf","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:f29257ca>","trigger":"user-message","userMessage":"하트인증은 여기다 올리면 될까요?"}},"expected":{}}
{"id":"case_9f0cb2f4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779574363525-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요~","input":{"request":{"requestId":"req_b822a53e","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_53e8e877","nickname":"<nick:90b41f68>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_885a35a9","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629981088-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_42415f0f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"진짜 잘하시네요"}},"expected":{}}
{"id":"case_4603c0cc","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629976794-0; llmOverall=1; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_5400a31f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"챌린저 ㄷㄷ"}},"expected":{}}
{"id":"case_2b38adb8","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629930176-0; llmOverall=1; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=채팅방 프로필에서 하트를 눌러 빨간색으로 만든 다음, 그 화면이 보이게 원본 스크린샷을 보내주시면 인증돼요.","input":{"request":{"requestId":"req_bf140842","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"어떻게 하나여"}},"expected":{}}
{"id":"case_f04b5287","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779633055712-0; llmOverall=1.25; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=안녕하세요, 오늘도 편하게 진행해요.","input":{"request":{"requestId":"req_bdd1114b","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:c673b3e2>","trigger":"user-message","userMessage":"안녕하세요!"}},"expected":{}}
{"id":"case_3e077513","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779630086906-0; llmOverall=1.5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=네, 그렇게 해주시면 돼요.","input":{"request":{"requestId":"req_bbc426e2","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"아항"}},"expected":{}}
{"id":"case_274fec57","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629952313-0; llmOverall=1.5; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=먼저 닉네임 변경과 하트 인증 두 가지를 완료하시면 돼요. 톡 게시글의 공지를 확인해 주세요!","input":{"request":{"requestId":"req_c69053e8","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"보내면 되나여?"}},"expected":{}}
{"id":"case_29f02c6e","lineNumber":0,"name":"promotion:low-score","notes":"promotionReason=low-score; sourceStreamId=1779629936899-0; llmOverall=1.5; derivedIntent=unknown; derivedPhase=deadline-near; assistantResponse=톡 게시글의 공지를 참고하셔서 먼저 카카오톡 닉네임 변경과 하트 인증을 진행해 주세요!","input":{"request":{"requestId":"req_be0e7f4a","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"여기 채팅방에"}},"expected":{}}
{"id":"case_59c880e4","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779633743117-0; llmOverall=4.25; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=카카오톡에서 디스코드 초대 안내를 받았던 메시지를 찾아 초대 수락을 눌러 들어가면 돼요.","input":{"request":{"requestId":"req_705c9e3f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:c673b3e2>","trigger":"user-message","userMessage":"저 디코는 어떻게 들어가요? 주민신청까지 해뒀습니다"}},"expected":{}}
{"id":"case_4fe5864d","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779632430014-0; llmOverall=5; derivedIntent=acknowledgement; derivedPhase=deadline-near; assistantResponse=좋아요, 천천히 진행해 주세요.","input":{"request":{"requestId":"req_781682ec","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:658a6c82>","trigger":"user-message","userMessage":"넵"}},"expected":{}}
{"id":"case_9a3640ce","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779632381253-0; llmOverall=n/a; derivedIntent=nickname-question; derivedPhase=nickname-rejected; assistantResponse=안 되는 이유는 티어가 허용 코드(GM, I, B, S, G, P, E, D, M, C, U) 중 하나의 대문자로 되어야 하는데 p가 소문자여서예요. \n카카오톡 닉네임의 마지막 부분만 P로 바꿔서 세나/95/여/P로 수정해 주세요.","input":{"request":{"requestId":"req_7368778f","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_ade16621","nickname":"<nick:1f050511>","trigger":"nickname-invalid"}},"expected":{}}
{"id":"case_5ea6c4d2","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779629969676-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=환영해요! 반갑습니다.","input":{"request":{"requestId":"req_9352c82d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_41e5d293","nickname":"<nick:6110a7e7>","trigger":"user-message","userMessage":"안녕하세요!!"}},"expected":{}}
{"id":"case_207dec80","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779629904020-0; llmOverall=n/a; derivedIntent=photo-question; derivedPhase=deadline-near; assistantResponse=방 하트 인증은 채팅방 오른쪽 상단 메뉴 버튼의 '커버 보기'에서 하트를 빨간색으로 바꾼 뒤, 편집 없이 원본 화면이 보이게 스크린샷을 보내주시면 돼요.","input":{"request":{"requestId":"req_37928f7d","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_80c06d53","nickname":"<nick:68a7fbf0>","trigger":"user-message","userMessage":"방하트 인증은"}},"expected":{}}
{"id":"case_b42ca588","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779622035543-0; llmOverall=5; derivedIntent=greeting; derivedPhase=deadline-near; assistantResponse=어서오세요!","input":{"request":{"requestId":"req_e9dcff68","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_3adc9936","nickname":"<nick:ccb74517>","trigger":"user-message","userMessage":"안녕하세요"}},"expected":{}}
{"id":"case_d96d1b98","lineNumber":0,"name":"promotion:rare-slice","notes":"promotionReason=rare-slice; sourceStreamId=1779619834487-0; llmOverall=n/a; derivedIntent=help-how; derivedPhase=deadline-near; assistantResponse=네, 본인인증이 완료됐다면 본인인증방은 나가셔도 돼요. \n현재 남은 절차는 디스코드 주민신청 채널 메시지 남기기 하나예요.","input":{"request":{"requestId":"req_e6bc023c","skill":"onboarding","chatroomId":"room_0575da24","userId":"user_d9c80f01","nickname":"<nick:9c5d5e3a>","trigger":"user-message","userMessage":"본인인증방은이제나가도돼요?"}},"expected":{}}
+23
View File
@@ -93,6 +93,29 @@ describe("judgeOnboardingWithLlm", () => {
}
});
it("keeps scores when rationale is longer than the prompt asks for", async () => {
const result = await judgeOnboardingWithLlm(
new FakeLlmClient(
JSON.stringify({
friendliness: 3,
accuracy: 4,
concision: 5,
scope: 4,
rationale: "가".repeat(800),
}),
),
evalCase,
"이제 방 하트 인증만 남았어요.",
derived,
);
expect(result.skipped).toBe(false);
if (!result.skipped) {
expect(result.scores.overall).toBe(4);
expect(result.rationale).toHaveLength(500);
}
});
it("skips when the judge returns invalid JSON", async () => {
const result = await judgeOnboardingWithLlm(
new FakeLlmClient('{"friendliness":"bad"}'),
+9 -2
View File
@@ -9,6 +9,8 @@ import type {
} from "../types.js";
import { getSkillJudgeConfig } from "./skill-config.js";
const MAX_RATIONALE_CHARS = 500;
function round(value: number): number {
return Number(value.toFixed(3));
}
@@ -18,10 +20,14 @@ function createJudgeSchema(axes: readonly string[]): z.ZodObject<z.ZodRawShape>
...Object.fromEntries(
axes.map((axis) => [axis, z.number().int().min(1).max(5)]),
),
rationale: z.string().trim().min(1).max(200),
rationale: z.string().trim().min(1),
});
}
function normalizeRationale(value: unknown): string {
return String(value).slice(0, MAX_RATIONALE_CHARS);
}
function buildJudgePrompt(
skillName: string,
evalCase: EvalCaseDefinition,
@@ -35,6 +41,7 @@ function buildJudgePrompt(
`다음 ${skillName} 에이전트 응답을 평가해 주세요.`,
`평가 축은 ${axes} 입니다.`,
"각 점수는 1~5의 정수만 사용하세요.",
"rationale은 200자 이하로 짧게 작성하세요.",
config.promptTemplate ??
"accuracy는 기대 조건과 실제 답변의 사실성을, concision은 과도한 부가 설명 여부를 포함해 평가하세요.",
"",
@@ -129,7 +136,7 @@ export async function judgeSkillWithLlm(
endpoint: response.endpoint,
provider: response.provider,
scores,
rationale: String(parsed["rationale"]),
rationale: normalizeRationale(parsed["rationale"]),
raw: response.content,
};
} catch {
+16
View File
@@ -0,0 +1,16 @@
{
"description": "deadline-near 상태에서도 사용자가 묻지 않은 남은 등록 절차를 자동으로 붙이지 않도록 우선순위 강화",
"diff": [
{
"layer": "policy",
"op": "append",
"reason": "첫 번째 실패에서 현재 상태의 remaining 정보가 질문보다 우선되어 무관한 절차 안내가 출력됐다. deadline-near에서도 질문 우선 원칙을 강화한다.",
"text": "[deadline-near 응답 범위 제한]\n- phase가 deadline-near여도 마지막 메시지가 가능 여부/확인 주체/확인 여부를 묻는 질문이면, 독촉성 남은 절차 안내를 붙이지 마.\n- remaining 상태는 사용자가 \"뭐가 남았나요\", \"다 했나요\", \"완료인가요\"처럼 진행 상태를 물을 때만 사용해.\n- 답변 전 마지막으로 확인: 내 응답 첫 문장이 사용자의 질문에 대한 '네/아니요/아직/관리자가 확인' 중 직접 답인지 점검해."
}
],
"expectedImprovement": "무관한 카카오톡/디스코드 절차 나열이 줄고, 사용자의 마지막 질문 중심으로 1문장 내외의 답변이 생성된다.",
"rationale": "현재 스냅샷에는 remaining 정보가 강하게 주입되어 있어, 사용자가 재등록 가능 여부만 묻는 상황에서도 남은 절차를 안내할 위험이 높다. 진행 상태 질문이 아닐 때 remaining을 쓰지 않게 하면 범위 이탈을 줄일 수 있다.",
"skill": "onboarding",
"variantId": "579dcc7d",
"trustTier": "T2"
}
+16
View File
@@ -0,0 +1,16 @@
{
"description": "디스코드 확인 주체/재등록 가능 여부 질문에는 상태 나열이나 불확실 회피 대신 짧게 직접 답하도록 하는 예외 규칙 추가",
"diff": [
{
"layer": "policy",
"op": "append",
"reason": "실패 케이스가 모두 사용자의 단일 질문에 답하지 못하고, 확인 불가 회피 또는 무관한 남은 절차 나열로 이탈했다. 해당 slice에서 반복되는 직접 질문 패턴을 좁게 처리한다.",
"text": "[discord-verify-question 직접응답 보강]\n- 사용자가 \"디코는 누가 확인하나요\", \"관리자가 확인하나요\", \"관리자 형님들이 확인해야 하나요\"처럼 디스코드 확인 주체를 물으면, 확인되지 않았다고 회피하지 말고 \"네, 디스코드 관련 확인은 관리자/운영진이 확인해요.\"라고 짧게 답해.\n- 사용자가 \"계정 확인 후 다시 등록해도 되나요\", \"제 명의 계정 확인 후 다시 등록해도 될까요\"처럼 본인 계정/명의 확인 뒤 재등록 가능 여부를 물으면, 다른 절차를 나열하지 말고 \"네, 본인 명의 계정 확인 후 다시 등록해 주시면 돼요.\"라고 답해.\n- 위 두 유형에서는 카카오톡 완료 여부, 남은 단계, 디스코드 닉네임 형식, 다음 절차를 사용자가 묻지 않는 한 덧붙이지 마."
}
],
"expectedImprovement": "디스코드 확인 주체 질문에는 '관리자/운영진이 확인한다'고 정확히 답하고, 계정 확인 후 재등록 질문에는 '가능하다'고 직접 답해 accuracy와 scope가 개선된다.",
"rationale": "현재 프롬프트의 '확인되지 않았으면 확인되지 않았다고만 답'과 '세션 상태' 정보가 과하게 작동해, 실제로는 일반 운영 규칙으로 답할 수 있는 질문에도 회피하거나 상태 목록을 출력한다. 확인 주체와 재등록 가능 여부는 온보딩 절차의 일반 질문이므로 짧은 정답 템플릿을 추가하는 것이 가장 작은 수정이다.",
"skill": "onboarding",
"variantId": "991bfb18",
"trustTier": "T2"
}
+52 -16
View File
@@ -31,7 +31,13 @@ import type {
DerivedState,
} from "@skills/onboarding/state/onboarding-state";
import { noopShadowEvalPublisher } from "./shadow-bus.js";
import { getOnboardingSessionKeyPrefix, getOpenRouterApiUrl } from "../config.js";
import {
getEvalCaseDeadlineMs,
getEvalLlmProvider,
getEvalRuntimeLlmEnabled,
getOnboardingSessionKeyPrefix,
getOpenRouterApiUrl,
} from "../config.js";
import { runEvalDispatch } from "./handoff-dispatch.js";
import { judgeHandoff } from "./judges/handoff-judge.js";
import { judgeSkillWithLlm } from "./judges/llm-judge.js";
@@ -285,6 +291,9 @@ function toProviderEndpoints(
}
export function createEvalLlmClient(): LlmClient {
const evalProvider = getEvalLlmProvider();
const includeProvider = (provider: ProviderName): boolean =>
evalProvider === "all" || evalProvider === provider;
const cliproxyEndpoints = parseEndpointList(process.env["CLI_PROXY_API_URLS"]);
const llamaEndpoints = parseEndpointList(
process.env["AGENT_LLM_API_URLS"],
@@ -301,7 +310,14 @@ export function createEvalLlmClient(): LlmClient {
);
const providerApiKeys = Object.fromEntries(
[
["cliproxy", firstNonEmpty(process.env["CLI_PROXY_API_KEY"])],
[
"cliproxy",
firstNonEmpty(
process.env["CLI_PROXY_API_KEY"],
process.env["LLAMA_SERVER_API_KEY"],
process.env["AGENT_LLM_API_KEY"],
),
],
["llama", llmApiKey],
["openrouter", openRouterApiKey],
].filter((entry): entry is [string, string] => typeof entry[1] === "string"),
@@ -309,9 +325,15 @@ export function createEvalLlmClient(): LlmClient {
return new LlmClient({
endpoints: [
...toProviderEndpoints(cliproxyEndpoints, "cliproxy"),
...toProviderEndpoints(openRouterEndpoints, "openrouter"),
...toProviderEndpoints(llamaEndpoints, "llama"),
...(includeProvider("cliproxy")
? toProviderEndpoints(cliproxyEndpoints, "cliproxy")
: []),
...(includeProvider("openrouter")
? toProviderEndpoints(openRouterEndpoints, "openrouter")
: []),
...(includeProvider("llama")
? toProviderEndpoints(llamaEndpoints, "llama")
: []),
],
apiKey: llmApiKey,
model: firstNonEmpty(process.env["AGENT_LLM_MODEL"]),
@@ -846,7 +868,8 @@ export async function loadEvalCases(
async function evaluateCase(
evalCase: EvalCaseDefinition,
llmClient: LlmClient,
runtimeLlmClient: LlmClient,
judgeLlmClient: LlmClient,
enableLlmJudge: boolean,
skillName: string,
): Promise<EvalCaseResult> {
@@ -860,7 +883,7 @@ async function evaluateCase(
: undefined;
const deps = {
garnet: inMemoryGarnet,
llm: llmClient,
llm: runtimeLlmClient,
room: roomContextService,
};
if (skillName === "handoff") {
@@ -868,7 +891,7 @@ async function evaluateCase(
evalCase,
input,
Object.values(SKILL_RUNNER_FACTORIES).map((factory) => factory(deps)),
llmClient,
judgeLlmClient,
enableLlmJudge,
);
}
@@ -883,7 +906,7 @@ async function evaluateCase(
return runWithContext(
{
chatroomId: input.request.chatroomId,
deadlineAt: new Date(Date.now() + 4_500).toISOString(),
deadlineAt: new Date(Date.now() + getEvalCaseDeadlineMs()).toISOString(),
requestId: input.request.requestId,
skill: input.request.skill,
startedAt: new Date().toISOString(),
@@ -896,7 +919,8 @@ async function evaluateCase(
evalCase,
input,
skill,
llmClient,
runtimeLlmClient,
judgeLlmClient,
enableLlmJudge,
effectiveSkill,
)
@@ -904,7 +928,7 @@ async function evaluateCase(
evalCase,
input,
skill,
llmClient,
judgeLlmClient,
enableLlmJudge,
effectiveSkill,
),
@@ -915,11 +939,12 @@ async function evaluateOnboardingCase(
evalCase: EvalCaseDefinition,
input: EvalCaseInput,
skill: Skill,
llmClient: LlmClient,
runtimeLlmClient: LlmClient,
judgeLlmClient: LlmClient,
enableLlmJudge: boolean,
skillName: string,
): Promise<EvalCaseResult> {
const classifier = new OnboardingIntentClassifier(llmClient);
const classifier = new OnboardingIntentClassifier(runtimeLlmClient);
const processTrace = new ProcessTraceCollector();
const judgeConfig = getSkillJudgeConfig(skillName);
@@ -950,7 +975,7 @@ async function evaluateOnboardingCase(
});
const llmJudge = enableLlmJudge
? await judgeSkillWithLlm(
llmClient,
judgeLlmClient,
skillName,
evalCase,
response.text,
@@ -1262,12 +1287,23 @@ export async function runEval(
limit: options?.limit,
});
const llmClient = createEvalLlmClient();
const judgeLlmClient = createEvalLlmClient();
const runtimeLlmClient = getEvalRuntimeLlmEnabled()
? judgeLlmClient
: new LlmClient({ endpoints: [] });
const enableLlmJudge = options?.enableLlmJudge ?? true;
const caseResults: EvalCaseResult[] = [];
for (const evalCase of evalCases) {
caseResults.push(await evaluateCase(evalCase, llmClient, enableLlmJudge, skill));
caseResults.push(
await evaluateCase(
evalCase,
runtimeLlmClient,
judgeLlmClient,
enableLlmJudge,
skill,
),
);
}
const summary = createSummary(caseResults);
+101
View File
@@ -25,6 +25,8 @@ vi.mock("@obs/spans", () => ({
}));
import { LlmClient } from "./llm-client.js";
import { BudgetTracker } from "@agent/budget";
import { runWithContext } from "@obs/context";
import type { ChatCompletionResponse } from "./types.js";
function okResponse(
@@ -168,4 +170,103 @@ describe("LlmClient endpoint normalization", () => {
}),
);
});
it("records LLM cost into the request budget", async () => {
vi.stubGlobal(
"fetch",
vi.fn(() =>
Promise.resolve(
okResponse("reply", {
usage: {
completion_tokens: 50,
prompt_tokens: 100,
total_tokens: 150,
},
}),
),
),
);
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
const client = new LlmClient({
endpoints: [{ provider: "openrouter", url: "https://openrouter.ai/api/v1" }],
providerApiKeys: { openrouter: "test-key" },
});
await runWithContext(
{
budget,
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() =>
client.chatCompletion([{ content: "hi", role: "user" }], {
modelOverride: "gpt-5.3-codex-spark",
}),
);
expect(budget.snapshot().totalLlmCostUsd).toBe(0.000045);
expect(llmSpanMock.setAttributes).toHaveBeenCalledWith(
expect.objectContaining({
"budget.llm_cost_usd": 0.000045,
"budget.total_cost_usd": 0.000045,
}),
);
});
it("does not record LLM cost when budget cost tracking is disabled", async () => {
vi.stubEnv("AGENT_BUDGET_LLM_COST_ENABLED", "false");
vi.stubGlobal(
"fetch",
vi.fn(() =>
Promise.resolve(
okResponse("reply", {
usage: {
completion_tokens: 50,
prompt_tokens: 100,
total_tokens: 150,
},
}),
),
),
);
const budget = new BudgetTracker({
enabled: true,
maxCostUsd: 1,
maxHandoffs: 5,
maxSameAgentVisits: 5,
maxToolCalls: 5,
});
const client = new LlmClient({
endpoints: [{ provider: "openrouter", url: "https://openrouter.ai/api/v1" }],
providerApiKeys: { openrouter: "test-key" },
});
await runWithContext(
{
budget,
chatroomId: "room-1",
requestId: "req-1",
skill: "onboarding",
startedAt: "2026-05-18T00:00:00.000Z",
trigger: "user-message",
userId: "user-1",
},
() =>
client.chatCompletion([{ content: "hi", role: "user" }], {
modelOverride: "gpt-5.3-codex-spark",
}),
);
expect(budget.snapshot().totalLlmCostUsd).toBe(0);
});
});
+30 -1
View File
@@ -10,6 +10,7 @@ import type {
import { getContext, getRequestLogger } from "@obs/context";
import { maskTraceContent, withLlmSpan } from "@obs/spans";
import {
getAgentBudgetLlmCostEnabled,
getLlmHealthTimeoutMs,
getLlmRequestTimeoutMs,
getLlmTemperature,
@@ -193,9 +194,10 @@ export class LlmClient {
continue;
}
const configuredRequestTimeoutMs = getLlmRequestTimeoutMs();
const requestTimeoutMs = Math.max(
1,
Math.min(getLlmRequestTimeoutMs(), options?.deadlineMs ?? Number.POSITIVE_INFINITY),
Math.min(configuredRequestTimeoutMs, options?.deadlineMs ?? Number.POSITIVE_INFINITY),
);
const models = selectModelsForProvider(provider.name, options, this.llamaModel);
let endpointSucceeded = false;
@@ -240,6 +242,9 @@ export class LlmClient {
model: request.model,
status: response.status,
body,
configuredTimeoutMs: configuredRequestTimeoutMs,
deadlineMs: options?.deadlineMs,
timeoutMs: requestTimeoutMs,
},
"[LLM] Request failed",
);
@@ -257,11 +262,22 @@ export class LlmClient {
model: request.model,
skill: getContext()?.skill,
});
if (getAgentBudgetLlmCostEnabled()) {
getContext()?.budget?.recordLlmCost(costUsd);
}
const budgetSnapshot = getContext()?.budget?.snapshot();
span.setAttributes({
...(shouldLogPrompts
? buildCompletionTraceAttributes(result)
: {}),
...(budgetSnapshot
? {
"budget.llm_cost_usd": budgetSnapshot.totalLlmCostUsd,
"budget.tool_cost_usd": budgetSnapshot.totalToolCostUsd,
"budget.total_cost_usd": budgetSnapshot.totalCostUsd,
}
: {}),
"gen_ai.response.finish_reasons": payload.choices
?.map((choice) => choice.finish_reason ?? "unknown")
.filter((reason): reason is string => Boolean(reason)),
@@ -292,6 +308,16 @@ export class LlmClient {
if (llmResult) {
await this.circuit.recordSuccess(endpoint.completionUrl);
endpointSucceeded = true;
logger.debug(
{
endpoint: endpoint.completionUrl,
model: request.model,
provider: provider.name,
role: options?.role,
tier: options?.tier,
},
"[LLM] Response received",
);
return llmResult;
}
} catch (error) {
@@ -301,6 +327,9 @@ export class LlmClient {
provider: provider.name,
model,
error: error instanceof Error ? error.message : String(error),
configuredTimeoutMs: configuredRequestTimeoutMs,
deadlineMs: options?.deadlineMs,
timeoutMs: requestTimeoutMs,
},
"[LLM] Request error",
);
+63 -6
View File
@@ -1,6 +1,7 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { LlmEndpointConfig } from "./types.js";
import {
resetLlmRoutingRoundRobinForTests,
selectMaxTokensForTier,
selectModelsForProvider,
selectProviderOrder,
@@ -8,6 +9,12 @@ import {
describe("llm routing", () => {
afterEach(() => {
delete process.env["CLIPROXY_TIER0_MODEL"];
delete process.env["CLIPROXY_TIER1_PLANNER_MODEL"];
delete process.env["CLIPROXY_TIER2_CRITIC_MODEL"];
delete process.env["LLAMA_TIER2_CRITIC_MODEL"];
delete process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"];
delete process.env["OPENROUTER_TIER1_RESPONDER_MODEL"];
delete process.env["OPENROUTER_MODEL"];
delete process.env["TIER0_MODEL"];
delete process.env["TIER1_PLANNER_MODEL"];
@@ -16,6 +23,7 @@ describe("llm routing", () => {
delete process.env["TIER0_MAX_TOKENS"];
delete process.env["TIER1_MAX_TOKENS"];
delete process.env["TIER2_MAX_TOKENS"];
resetLlmRoutingRoundRobinForTests();
vi.restoreAllMocks();
});
@@ -93,9 +101,12 @@ describe("llm routing", () => {
).toEqual(["cliproxy", "openrouter"]);
});
it("returns role-aware model for cliproxy and llama fallback as single-item list", () => {
process.env["TIER1_PLANNER_MODEL"] = "cliproxy-planner";
process.env["TIER2_CRITIC_MODEL"] = "cliproxy-critic";
it("returns provider-aware models for runtime roles and tiers", () => {
process.env["CLIPROXY_TIER1_PLANNER_MODEL"] = "cliproxy-planner";
process.env["CLIPROXY_TIER2_CRITIC_MODEL"] = "cliproxy-critic";
process.env["OPENROUTER_TIER1_RESPONDER_MODEL"] =
"openrouter-fast, openrouter-backup";
process.env["LLAMA_TIER2_CRITIC_MODEL"] = "llama-critic";
expect(
selectModelsForProvider("cliproxy", { role: "planner", tier: 1 }, "local-qwen"),
@@ -103,9 +114,12 @@ describe("llm routing", () => {
expect(
selectModelsForProvider("cliproxy", { role: "critic", tier: 2 }, "local-qwen"),
).toEqual(["cliproxy-critic"]);
expect(selectModelsForProvider("llama", { tier: 2 }, "local-qwen")).toEqual([
"local-qwen",
]);
expect(
selectModelsForProvider("openrouter", { role: "responder", tier: 1 }, "local-qwen"),
).toEqual(["openrouter-fast", "openrouter-backup"]);
expect(
selectModelsForProvider("llama", { role: "critic", tier: 2 }, "local-qwen"),
).toEqual(["llama-critic"]);
});
it("splits OPENROUTER_MODEL by comma and preserves priority order", () => {
@@ -124,6 +138,49 @@ describe("llm routing", () => {
]);
});
it("round-robins models within the same provider role and tier when enabled", () => {
process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"] = "true";
process.env["CLIPROXY_TIER0_MODEL"] =
"gemini-3.1-flash-lite, gpt-5.3-codex-spark";
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gemini-3.1-flash-lite", "gpt-5.3-codex-spark"]);
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gpt-5.3-codex-spark", "gemini-3.1-flash-lite"]);
expect(
selectModelsForProvider("cliproxy", { tier: 0 }, "local-qwen"),
).toEqual(["gemini-3.1-flash-lite", "gpt-5.3-codex-spark"]);
});
it("keeps provider order fixed even when model round-robin is enabled", () => {
process.env["LLM_MODEL_ROUND_ROBIN_ENABLED"] = "true";
const endpoints: LlmEndpointConfig[] = [
{
completionUrl: "http://llama.test/v1/chat/completions",
healthUrl: "http://llama.test/health",
provider: "llama",
},
{
completionUrl: "http://openrouter.test/v1/chat/completions",
healthUrl: "http://openrouter.test",
provider: "openrouter",
},
{
completionUrl: "http://cliproxy.test/v1/chat/completions",
healthUrl: "http://cliproxy.test",
provider: "cliproxy",
},
];
expect(
selectProviderOrder(endpoints, { role: "responder", tier: 1 }).map(
(endpoint) => endpoint.provider,
),
).toEqual(["cliproxy", "openrouter", "llama"]);
});
it("chooses tier defaults for max tokens when explicit maxTokens is absent", () => {
process.env["TIER0_MAX_TOKENS"] = "111";
process.env["TIER1_MAX_TOKENS"] = "222";
+37 -26
View File
@@ -1,11 +1,8 @@
import {
getOpenRouterModels,
getLlmModelRoundRobinEnabled,
getRuntimeLlmModelsForProvider,
getTier0MaxTokens,
getTier0Model,
getTier1MaxTokens,
getTier1PlannerModel,
getTier1ResponderModel,
getTier2CriticModel,
getTier2MaxTokens,
getLlmMaxTokens,
} from "../config.js";
@@ -13,6 +10,7 @@ import type { ChatCompletionOptions, LlmEndpointConfig, ProviderName } from "./t
const PROVIDER_PRIORITY: readonly ProviderName[] = ["cliproxy", "openrouter", "llama"];
const TEMPLATE_ONLY_PROVIDER_PRIORITY: readonly ProviderName[] = ["llama"];
const modelRoundRobinOffsets = new Map<string, number>();
function getProviderPriority(
options: ChatCompletionOptions | undefined,
@@ -33,27 +31,15 @@ export function selectModelsForProvider(
return [options.modelOverride.trim()];
}
if (provider === "llama") {
return [llamaFallbackModel];
}
if (provider === "openrouter") {
return getOpenRouterModels();
}
if (options?.role === "planner") {
return [getTier1PlannerModel()];
}
if (options?.role === "critic") {
return [getTier2CriticModel()];
}
if (options?.tier === 1 || options?.tier === 2) {
return [getTier1ResponderModel()];
}
return [getTier0Model()];
const models = getRuntimeLlmModelsForProvider(
provider,
{
role: options?.role,
tier: options?.tier,
},
llamaFallbackModel,
);
return rotateModelsForProvider(provider, options, models);
}
export function selectMaxTokensForTier(
@@ -106,3 +92,28 @@ export function selectProviderOrder(
})
.map((item) => item.endpoint);
}
function rotateModelsForProvider(
provider: ProviderName,
options: ChatCompletionOptions | undefined,
models: string[],
): string[] {
if (!getLlmModelRoundRobinEnabled() || models.length <= 1) {
return models;
}
const key = [
provider,
options?.role ?? "none",
options?.tier ?? "none",
models.join("|"),
].join(":");
const offset = modelRoundRobinOffsets.get(key) ?? 0;
modelRoundRobinOffsets.set(key, offset + 1);
const rotation = offset % models.length;
return [...models.slice(rotation), ...models.slice(0, rotation)];
}
export function resetLlmRoutingRoundRobinForTests(): void {
modelRoundRobinOffsets.clear();
}
+2 -2
View File
@@ -3,12 +3,12 @@ import { AsyncLocalStorage } from 'node:async_hooks';
import type { Logger } from 'pino';
import { getLogger } from './logger.js';
import type { BudgetSnapshot } from '@agent/budget';
import type { BudgetTracker } from '@agent/budget';
export interface RequestContext {
chatroomId: string;
deadlineAt?: string;
budget?: BudgetSnapshot | undefined;
budget?: BudgetTracker | undefined;
handoffChain?: string[] | undefined;
handoffDepth?: number | undefined;
logger: Logger;
+12
View File
@@ -15,9 +15,21 @@ type RequestBindings = {
function buildRedactionPaths(): string[] {
const commonPaths = [
'authorization',
'apiKey',
'api_key',
'headers.authorization',
'headers.x-api-key',
'headers["x-api-key"]',
'req.headers.authorization',
'req.headers.x-api-key',
'req.headers["x-api-key"]',
'request.headers.authorization',
'request.headers.x-api-key',
'request.headers["x-api-key"]',
'request.apiKey',
'request.api_key',
'response.apiKey',
'response.api_key',
];
const secretPaths = KNOWN_SECRET_KEYS.flatMap((key) => [
+23
View File
@@ -4,6 +4,8 @@ import {
formatReviewList,
formatReviewShow,
formatReviewStats,
formatToolAuditList,
formatToolAuditShow,
} from "./cli.js";
describe("operations review CLI formatters", () => {
@@ -81,4 +83,25 @@ describe("operations review CLI formatters", () => {
}),
).toContain("low-confidence\ttotal=3\tpending=1\tresolved=2");
});
it("formats tool audit rows", () => {
const entry = {
argsHash: "abc123",
calledAt: "2026-05-18T00:00:00.000Z",
chatroomId: "room-1",
durationMs: 12,
id: "1-0",
ok: true,
resultSummary: "{\"ok\":true}",
skill: "onboarding",
toolName: "validate-nickname",
type: "tool-call" as const,
userId: "user-1",
};
expect(formatToolAuditList([entry])).toContain(
"1-0\t2026-05-18T00:00:00.000Z\tonboarding\tvalidate-nickname\tyes\t12",
);
expect(formatToolAuditShow(entry)).toContain("argsHash: abc123");
});
});
+99 -1
View File
@@ -15,6 +15,10 @@ import {
type SupervisorReviewResolution,
type SupervisorReviewStats,
} from "@skills/supervisor/review-queue";
import {
ToolAuditLogger,
type ListedToolAuditEntry,
} from "@tools/audit";
import {
getFlagString,
hasFlag,
@@ -164,6 +168,46 @@ export function formatReviewStats(stats: SupervisorReviewStats): string {
].join("\n");
}
export function formatToolAuditList(items: ListedToolAuditEntry[]): string {
const rows = [
[
"id",
"calledAt",
"skill",
"toolName",
"ok",
"durationMs",
"errorKind",
],
...items.map((item) => [
item.id,
item.calledAt,
item.skill,
item.toolName,
item.ok ? "yes" : "no",
String(item.durationMs),
item.errorKind ?? "-",
]),
];
return rows.map((row) => row.join("\t")).join("\n");
}
export function formatToolAuditShow(item: ListedToolAuditEntry): string {
return [
`id: ${item.id}`,
`calledAt: ${item.calledAt}`,
`skill: ${item.skill}`,
`toolName: ${item.toolName}`,
`chatroomId: ${item.chatroomId}`,
`userId: ${item.userId}`,
`ok: ${item.ok ? "yes" : "no"}`,
`durationMs: ${item.durationMs}`,
`errorKind: ${item.errorKind ?? "-"}`,
`argsHash: ${item.argsHash}`,
`resultSummary: ${item.resultSummary || "-"}`,
].join("\n");
}
async function handleReviewList(
flags: Map<string, string | true>,
): Promise<void> {
@@ -277,6 +321,52 @@ async function handleReviewStats(
}
}
async function handleToolAuditList(
flags: Map<string, string | true>,
): Promise<void> {
const connection = await connectGarnetForCli();
try {
const items = await new ToolAuditLogger(connection.garnetClient).list({
limit: parsePositiveInteger(getFlagString(flags, "limit")) ?? 50,
since: sinceDurationToIso(getFlagString(flags, "since") ?? "24h"),
skill: getFlagString(flags, "skill"),
});
console.info(
hasFlag(flags, "json")
? JSON.stringify(items, null, 2)
: formatToolAuditList(items),
);
} finally {
await connection.close();
}
}
async function handleToolAuditShow(
id: string | undefined,
flags: Map<string, string | true>,
): Promise<void> {
if (!id?.trim()) {
throw new Error("tool-audit:show requires an entry id");
}
const connection = await connectGarnetForCli();
try {
const item = await new ToolAuditLogger(connection.garnetClient).get(id);
if (!item) {
process.exitCode = 2;
console.error(`[tool-audit] entry not found: ${id}`);
return;
}
console.info(
hasFlag(flags, "json")
? JSON.stringify(item, null, 2)
: formatToolAuditShow(item),
);
} finally {
await connection.close();
}
}
async function main(): Promise<void> {
loadEnvironmentFromNodeEnv();
initLogger();
@@ -298,9 +388,17 @@ async function main(): Promise<void> {
await handleReviewStats(flags);
return;
}
if (command === "tool-audit:list") {
await handleToolAuditList(flags);
return;
}
if (command === "tool-audit:show") {
await handleToolAuditShow(positionals[0], flags);
return;
}
throw new Error(
"Unknown command. Use review:list, review:show, review:resolve, or review:stats.",
"Unknown command. Use review:list, review:show, review:resolve, review:stats, tool-audit:list, or tool-audit:show.",
);
}
+3
View File
@@ -46,6 +46,9 @@ describe("escalation trigger", () => {
maxHandoffs: 4,
maxSameAgentVisits: 2,
maxToolCalls: 20,
totalCostUsd: 0,
totalLlmCostUsd: 0,
totalToolCostUsd: 0,
toolCalls: 0,
visits: {},
},
+1
View File
@@ -16,5 +16,6 @@ export function createOnboardingSkill(deps: SkillDeps): Skill {
deps.shadowEvalBus,
deps.natsClient,
deps.memoryService,
deps.escalationHistoryService,
);
}
+42 -33
View File
@@ -38,6 +38,48 @@ export const ONBOARDING_INTENT_TABLE = {
match: ({ trimmed }) => isRemainingStatusQuestion(trimmed),
triggers: ["progress-update", "timeout-warning"],
},
"discord-verify-question": {
confidence: 0.96,
keywords: ["디스코드", "확인", "닉네임", "주민신청"],
llm: true,
match: ({ lower, trimmed }) => {
const mentionsDiscord = /(디스코드|디코|discord)/u.test(lower);
const asksDiscordVerification =
/(확인|조회|찾아|검색|검사|바뀌|변경|닉변|맞(?:아|나요)?|됐(?:나|나요)?|완료됐(?:나|나요)?|승인|반영)/u.test(
trimmed,
);
const reportsDiscordNicknameCompletion =
/(닉네임|닉변|프로필\s*닉네임|서버\s*닉네임)/u.test(trimmed) &&
/(완료했(?:어|어요|습니다)?|끝냈(?:어|어요|습니다)?|설정했(?:어|어요|습니다)?|바꿨(?:어|어요|습니다)?|변경했(?:어|어요|습니다)?|다\s*했(?:어|어요|습니다)?|해놨(?:어|어요|습니다)?|해뒀(?:어|어요|습니다)?|적용됐(?:어|어요|습니다)?)/u.test(
trimmed,
);
const reportsResidentApplicationCompletion =
/(주민\s*신청|주민신청|주민\s*등록|주민등록|프로필\s*메시지|메시지)/u.test(
trimmed,
) &&
/(완료했(?:어|어요|습니다)?|끝냈(?:어|어요|습니다)?|남겼(?:어|어요|습니다)?|썼(?:어|어요|습니다)?|작성했(?:어|어요|습니다)?|다\s*했(?:어|어요|습니다)?|해놨(?:어|어요|습니다)?|해뒀(?:어|어요|습니다)?|했(?:어|어요|습니다)?)/u.test(
trimmed,
);
return (
mentionsDiscord &&
(asksDiscordVerification ||
reportsDiscordNicknameCompletion ||
reportsResidentApplicationCompletion)
);
},
},
"discord-question": {
confidence: 0.95,
keywords: ["디스코드", "주민신청", "프로필", "권한", "설정"],
llm: true,
match: ({ lower }) => /(디스코드|디코|discord)/u.test(lower),
regex: [
{
confidence: 0.94,
pattern: /(주민신청\s*채널|주민신청채널|서버\s*프로필)/u,
},
],
},
"help-where": {
confidence: 0.9,
keywords: [
@@ -75,39 +117,6 @@ export const ONBOARDING_INTENT_TABLE = {
/(가입|등록|절차|도움|도와|뭐해야|뭐 해야|해야 돼|해야해|무엇을 해야)/u,
],
},
"discord-verify-question": {
confidence: 0.96,
keywords: ["디스코드", "확인", "닉네임"],
llm: true,
match: ({ lower, trimmed }) => {
const mentionsDiscord = /(디스코드|디코|discord)/u.test(lower);
const asksDiscordVerification =
/(확인|조회|찾아|검색|검사|바뀌|변경|닉변|맞(?:아|나요)?|됐(?:나|나요)?|완료됐(?:나|나요)?)/u.test(
trimmed,
);
const reportsDiscordNicknameCompletion =
/(닉네임|닉변|프로필\s*닉네임|서버\s*닉네임)/u.test(trimmed) &&
/(완료했(?:어|어요|습니다)?|끝냈(?:어|어요|습니다)?|설정했(?:어|어요|습니다)?|바꿨(?:어|어요|습니다)?|변경했(?:어|어요|습니다)?|다\s*했(?:어|어요|습니다)?|해놨(?:어|어요|습니다)?|해뒀(?:어|어요|습니다)?|적용됐(?:어|어요|습니다)?)/u.test(
trimmed,
);
return (
mentionsDiscord &&
(asksDiscordVerification || reportsDiscordNicknameCompletion)
);
},
},
"discord-question": {
confidence: 0.95,
keywords: ["디스코드", "주민신청", "프로필", "권한", "설정"],
llm: true,
match: ({ lower }) => /(디스코드|디코|discord)/u.test(lower),
regex: [
{
confidence: 0.94,
pattern: /(주민신청\s*채널|주민신청채널|서버\s*프로필)/u,
},
],
},
"age-question": {
confidence: 0.93,
keywords: ["27", "가입", "가능"],
+4 -4
View File
@@ -145,12 +145,12 @@ export const ONBOARDING_PROMPT_DATA = {
{
user: "디스코드 프로필 닉네임 설정 완료했어요",
assistant:
"디스코드 닉네임을 확인해봤는데 아직 올바른 형식으로 확인되지는 않았어요. 디스코드 서버 닉네임은 이름#태그/출생 연도/티어 형식이에요. 예: 할리#프렌치불독/88/D",
"디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '날으는꼴뚜기/D' 으로 확인됩니다. 할리#프렌치불독/88/D 와 같이 닉네임#태그/출생연도/티어로 수정해 주세요.",
},
{
user: "디스코드 닉네임 바꿨어요",
assistant:
"디스코드 닉네임이 아직 이름#태그/출생 연도/티어 형식으로 확인되지 않아요. 예: 할리#프렌치불독/88/D처럼 태그를 포함해서 설정해 주세요.",
"디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '날으는꼴뚜기/D' 으로 확인됩니다. 할리#프렌치불독/88/D 와 같이 닉네임#태그/출생연도/티어로 수정해 주세요.",
},
],
greeting: [
@@ -236,10 +236,10 @@ export const ONBOARDING_PROMPT_DATA = {
].join("\n"),
"discord-question": [
"- 디스코드 일반 안내나 규칙 질문이면 lookup-community-rule 도구를 사용해 정확한 규칙만 답해.",
"- 사용자가 디스코드 닉네임이나 프로필 설정을 끝냈다고 말하거나 반영 여부를 묻는다면 verify-discord-nickname 도구를 먼저 사용해 직접 확인해.",
"- 사용자가 디스코드 닉네임, 프로필 설정, 주민신청 메시지를 끝냈다고 말하거나 반영 여부를 묻는다면 verify-discord-nickname 도구를 먼저 사용해 직접 확인해.",
].join("\n"),
"discord-verify-question":
"- 디스코드 닉네임 확인 요청은 verify-discord-nickname 도구를 먼저 사용하고, 조회 결과만 짧게 답해.",
"- 디스코드 닉네임/주민신청 확인 요청은 verify-discord-nickname 도구와 check-onboarding-progress 도구를 먼저 사용하고, 조회 결과만 짧게 답해. 확인 없이 초대 안내만 반복하지 마. 닉네임이 틀렸거나 미설정이면 recommendedNickname/correctionGuide를 우선 사용해 정확히 어떻게 고칠지 안내해.",
"help-where":
"- 위치 안내가 모호할 때만 get-room-context 도구를 고려하고, 기본적으로는 절차 위치만 답해.",
"status-question":
-3
View File
@@ -4,7 +4,6 @@ import { z } from "zod";
import {
getCriticMaxTokens,
getLlmTemperature,
getTier2CriticModel,
} from "../../../config.js";
import type { DerivedState } from "../state/onboarding-state.js";
import { getActiveProcessTrace } from "./trace.js";
@@ -147,7 +146,6 @@ export class OnboardingCritic {
{
deadlineMs: options?.deadlineMs,
maxTokens: getCriticMaxTokens(),
modelOverride: getTier2CriticModel(),
role: "critic",
temperature: Math.max(0, getLlmTemperature() - 0.15),
tier: 2,
@@ -205,7 +203,6 @@ export class OnboardingCritic {
deadlineMs: options.deadlineMs,
jsonMode: true,
maxTokens: getCriticMaxTokens(),
modelOverride: getTier2CriticModel(),
role: "critic",
temperature: Math.max(0, getLlmTemperature() - 0.15),
tier: 2,
@@ -207,6 +207,317 @@ describe("OnboardingOrchestrator", () => {
expect(calls[1]?.options?.tools).toBeUndefined();
});
it("forces discord verification tools before answering completion reports", async () => {
const calls: Array<{
messages: ChatMessage[];
options: ChatCompletionOptions | undefined;
}> = [];
const executedTools: string[] = [];
const llmClientStub = {
chatCompletion: (
messages: ChatMessage[],
options?: ChatCompletionOptions,
) => {
calls.push({ messages, options });
if (calls.length === 1) {
return Promise.resolve({
content: JSON.stringify({
approach: "answer-direct",
targetPhase: "deadline-near",
mustInclude: [],
mustAvoid: [],
suggestedTools: [],
tone: "friendly",
targetLengthChars: 120,
rationale: "잘못된 직접 답변 계획",
}),
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "{}" } }] },
message: { role: "assistant" as const, content: "{}" },
});
}
return Promise.resolve({
content: "디스코드 닉네임은 확인됐고, 주민신청 메시지도 확인됐어요.",
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "확인됐어요." } }] },
message: { role: "assistant" as const, content: "확인됐어요." },
});
},
};
const verifyTool: Tool<Record<string, never>, { status: string }> = {
name: "verify-discord-nickname",
description: "디스코드 닉네임 확인",
argsSchema: z.object({}),
resultSchema: z.object({
status: z.string(),
}),
execute: () => {
executedTools.push("verify-discord-nickname");
return Promise.resolve({ status: "matched-renamed" });
},
};
const progressTool: Tool<
{ chatroomId: string; userId: string },
{ remainingSteps: string[] }
> = {
name: "check-onboarding-progress",
description: "진행 상태 확인",
argsSchema: z.object({
chatroomId: z.string(),
userId: z.string(),
}),
resultSchema: z.object({
remainingSteps: z.array(z.string()),
}),
execute: (args) => {
executedTools.push(`${args.chatroomId}:${args.userId}`);
return Promise.resolve({ remainingSteps: [] });
},
};
const orchestrator = new OnboardingOrchestrator(llmClientStub as never);
const result = await orchestrator.run(
createDerivedState({
allowShortResponse: false,
intent: { kind: "discord-verify-question" },
phase: { kind: "deadline-near", remaining: ["discord-profile-message"] },
responseMode: "llm-with-tools",
tier: 1,
}),
[
{ role: "system", content: "기본 시스템 프롬프트" },
{
role: "user",
content: "저 디코는 어떻게 들어가요? 주민신청까지 해뒀습니다",
},
],
{
request: {
requestId: "req-1",
skill: "onboarding",
chatroomId: "room-1",
userId: "user-1",
nickname: "프리리/95/여/D",
trigger: "user-message",
},
toolContext: {},
tools: [verifyTool, progressTool],
},
);
expect(result).toContain("주민신청");
expect(executedTools).toEqual([
"verify-discord-nickname",
"room-1:user-1",
]);
expect(calls[1]?.messages[0]?.content).toContain("[TOOL_PLAN_RESULTS]");
expect(calls[1]?.messages[0]?.content).toContain("주민신청 메시지 확인 상태");
expect(calls[1]?.messages[0]?.content).toContain(
"디스코드 닉네임 상태를 독립 항목처럼 길게 설명",
);
expect(calls[1]?.options?.tools).toBeUndefined();
});
it("does not expose verification tool again after partial discord tool-plan success", async () => {
const calls: Array<{
messages: ChatMessage[];
options: ChatCompletionOptions | undefined;
}> = [];
let verifyCallCount = 0;
const llmClientStub = {
chatCompletion: (
messages: ChatMessage[],
options?: ChatCompletionOptions,
) => {
calls.push({ messages, options });
if (calls.length === 1) {
return Promise.resolve({
content: JSON.stringify({
approach: "validate-then-explain",
targetPhase: "deadline-near",
mustInclude: [],
mustAvoid: [],
suggestedTools: ["verify-discord-nickname"],
tone: "friendly",
targetLengthChars: 120,
rationale: "닉네임 확인",
}),
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "{}" } }] },
message: { role: "assistant" as const, content: "{}" },
});
}
return Promise.resolve({
content: "디스코드 닉네임은 확인됐어요.",
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "확인됐어요." } }] },
message: { role: "assistant" as const, content: "확인됐어요." },
});
},
};
const verifyTool: Tool<Record<string, never>, { status: string }> = {
name: "verify-discord-nickname",
description: "디스코드 닉네임 확인",
argsSchema: z.object({}),
resultSchema: z.object({
status: z.string(),
}),
execute: () => {
verifyCallCount += 1;
return Promise.resolve({ status: "matched-renamed" });
},
};
const orchestrator = new OnboardingOrchestrator(llmClientStub as never);
const result = await orchestrator.run(
createDerivedState({
allowShortResponse: false,
intent: { kind: "discord-verify-question" },
phase: { kind: "deadline-near", remaining: ["discord-profile-nickname"] },
responseMode: "llm-with-tools",
tier: 1,
}),
[
{ role: "system", content: "기본 시스템 프롬프트" },
{
role: "user",
content: "디스코드 닉네임 변경했어요",
},
],
{
request: {
requestId: "req-1",
skill: "onboarding",
chatroomId: "room-1",
userId: "user-1",
nickname: "프리리/95/여/D",
trigger: "user-message",
},
toolContext: {},
tools: [verifyTool],
},
);
expect(result).toContain("닉네임");
expect(verifyCallCount).toBe(1);
expect(calls[1]?.messages[0]?.content).toContain("[TOOL_PLAN_RESULTS]");
expect(calls[1]?.messages[0]?.content).toContain("matched-renamed");
expect(calls[1]?.messages[0]?.content).toContain(
"Tool not registered for step: check-onboarding-progress",
);
expect(calls[1]?.options?.tools).toBeUndefined();
});
it("focuses discord nickname answers when only nickname verification was asked", async () => {
const calls: Array<{
messages: ChatMessage[];
options: ChatCompletionOptions | undefined;
}> = [];
const llmClientStub = {
chatCompletion: (
messages: ChatMessage[],
options?: ChatCompletionOptions,
) => {
calls.push({ messages, options });
if (calls.length === 1) {
return Promise.resolve({
content: JSON.stringify({
approach: "validate-then-explain",
targetPhase: "deadline-near",
mustInclude: [],
mustAvoid: [],
suggestedTools: [],
tone: "friendly",
targetLengthChars: 120,
rationale: "닉네임 확인",
}),
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "{}" } }] },
message: { role: "assistant" as const, content: "{}" },
});
}
return Promise.resolve({
content: "디스코드 닉네임은 확인됐어요.",
endpoint: "mock://llm",
provider: "cliproxy" as const,
raw: { choices: [{ message: { content: "확인됐어요." } }] },
message: { role: "assistant" as const, content: "확인됐어요." },
});
},
};
const verifyTool: Tool<Record<string, never>, { status: string }> = {
name: "verify-discord-nickname",
description: "디스코드 닉네임 확인",
argsSchema: z.object({}),
resultSchema: z.object({
status: z.string(),
}),
execute: () => Promise.resolve({ status: "matched-renamed" }),
};
const progressTool: Tool<
{ chatroomId: string; userId: string },
{ remainingSteps: string[] }
> = {
name: "check-onboarding-progress",
description: "진행 상태 확인",
argsSchema: z.object({
chatroomId: z.string(),
userId: z.string(),
}),
resultSchema: z.object({
remainingSteps: z.array(z.string()),
}),
execute: () => Promise.resolve({ remainingSteps: [] }),
};
const orchestrator = new OnboardingOrchestrator(llmClientStub as never);
const result = await orchestrator.run(
createDerivedState({
allowShortResponse: false,
intent: { kind: "discord-verify-question" },
phase: { kind: "deadline-near", remaining: ["discord-profile-nickname"] },
responseMode: "llm-with-tools",
tier: 1,
}),
[
{ role: "system", content: "기본 시스템 프롬프트" },
{
role: "user",
content: "디스코드 닉네임 변경했어요",
},
],
{
request: {
requestId: "req-1",
skill: "onboarding",
chatroomId: "room-1",
userId: "user-1",
nickname: "프리리/95/여/D",
trigger: "user-message",
},
toolContext: {},
tools: [verifyTool, progressTool],
},
);
expect(result).toContain("닉네임");
expect(calls[1]?.messages[0]?.content).toContain("디스코드 닉네임 조회 결과");
expect(calls[1]?.messages[0]?.content).toContain(
"주민신청 메시지 상태 언급",
);
});
it("injects requested SKILL.md resources before responder when enabled", async () => {
process.env["AGENT_SKILL_MD_MODE"] = "true";
const calls: Array<{
+232 -24
View File
@@ -10,11 +10,8 @@ import {
getPlannerMaxTokens,
getTier0EstimatedCallMs,
getTier0MaxTokens,
getTier0Model,
getTier1EstimatedCallMs,
getTier1MaxTokens,
getTier1PlannerModel,
getTier1ResponderModel,
getTier2EstimatedCallMs,
getTier2MaxTokens,
} from "../../../config.js";
@@ -75,6 +72,48 @@ function summarizeMessages(messages: ChatMessage[]): string {
.join("\n\n");
}
type DiscordVerificationFocus =
| "both"
| "nickname"
| "resident-application-message";
function getLatestUserMessage(messages: ChatMessage[]): string {
for (let index = messages.length - 1; index >= 0; index -= 1) {
const message = messages[index];
if (message?.role === "user") {
return message.content;
}
}
return "";
}
function getDiscordVerificationFocus(
derived: DerivedState,
messages: ChatMessage[],
): DiscordVerificationFocus {
if (derived.intent.kind !== "discord-verify-question") {
return "both";
}
const latestUserMessage = getLatestUserMessage(messages).toLowerCase();
const asksResidentApplication =
/(주민\s*신청|주민신청|주민\s*등록|주민등록|프로필\s*메시지|채널\s*메시지|메시지\s*(작성|남김|보냄|썼|완료|했습니다|했어요))/u.test(
latestUserMessage,
);
const asksNickname =
/(닉네임|닉변|서버\s*프로필|서버프로필|프로필\s*닉네임|서버\s*표시명|서버표시명)/u.test(
latestUserMessage,
);
if (asksResidentApplication && !asksNickname) {
return "resident-application-message";
}
if (asksNickname && !asksResidentApplication) {
return "nickname";
}
return "both";
}
function tryParsePlannerOutput(raw: string): PlannerOutput | null {
try {
const parsed = JSON.parse(raw) as unknown;
@@ -95,7 +134,24 @@ function tryParsePlannerOutput(raw: string): PlannerOutput | null {
}
}
function getDefaultPlan(derived: DerivedState): PlannerOutput {
function getDefaultPlan(
derived: DerivedState,
focus: DiscordVerificationFocus = "both",
): PlannerOutput {
if (derived.intent.kind === "discord-verify-question") {
return enforceRequiredToolPlan(derived, {
approach: "validate-then-explain",
targetPhase: derived.phase.kind,
mustInclude: getDiscordVerificationMustInclude(focus),
mustAvoid: getDiscordVerificationMustAvoid(focus),
requestResources: [],
suggestedTools: [],
tone: "friendly",
targetLengthChars: 150,
rationale: "discord-verification-fallback-plan",
}, focus);
}
return {
approach:
derived.responseMode === "llm-with-tools"
@@ -115,6 +171,104 @@ function getDefaultPlan(derived: DerivedState): PlannerOutput {
};
}
function hasToolPlanStep(plan: PlannerOutput, toolName: string): boolean {
return plan.toolPlan?.steps.some((step) => step.tool === toolName) ?? false;
}
function getDiscordVerificationMustInclude(
focus: DiscordVerificationFocus,
): string[] {
if (focus === "nickname") {
return ["디스코드 닉네임 조회 결과"];
}
if (focus === "resident-application-message") {
return [
"주민신청 메시지 확인 상태",
"메시지 형식이 틀리면 서버프로필 닉네임과 동일하게 다시 작성하라는 안내",
];
}
return [
"디스코드 닉네임 조회 결과",
"주민신청 메시지 확인 상태",
];
}
function getDiscordVerificationMustAvoid(
focus: DiscordVerificationFocus,
): string[] {
const base = ["확인 없이 완료 단정", "초대 안내만 반복"];
if (focus === "nickname") {
return [...base, "주민신청 메시지 상태 언급"];
}
if (focus === "resident-application-message") {
return [
...base,
"디스코드 닉네임 상태를 독립 항목처럼 길게 설명",
"묻지 않은 남은 절차 나열",
];
}
return base;
}
function enforceRequiredToolPlan(
derived: DerivedState,
plan: PlannerOutput,
focus: DiscordVerificationFocus = "both",
): PlannerOutput {
if (derived.intent.kind !== "discord-verify-question") {
return plan;
}
const suggestedTools = [
...new Set([
...plan.suggestedTools,
"verify-discord-nickname" as const,
"check-onboarding-progress" as const,
]),
];
const steps = [...(plan.toolPlan?.steps ?? [])];
if (!hasToolPlanStep(plan, "verify-discord-nickname")) {
steps.unshift({
id: "verify-discord",
tool: "verify-discord-nickname",
on: { error: "fallback-llm" },
});
}
if (!hasToolPlanStep(plan, "check-onboarding-progress")) {
steps.push({
id: "check-progress",
tool: "check-onboarding-progress",
argsFromContext: {
chatroomId: "$request.chatroomId",
userId: "$request.userId",
},
on: { error: "fallback-llm" },
});
}
return {
...plan,
approach: "validate-then-explain",
mustAvoid: [
...new Set([
...plan.mustAvoid,
...getDiscordVerificationMustAvoid(focus),
]),
].slice(0, 6),
mustInclude: [
...new Set([
...plan.mustInclude,
...getDiscordVerificationMustInclude(focus),
]),
].slice(0, 6),
suggestedTools: suggestedTools.slice(0, 3),
toolPlan: {
composeWith: "all-then-llm",
steps: steps.slice(0, 4),
},
};
}
function getRemainingDeadlineMs(): number | undefined {
const deadlineAt = getContext()?.deadlineAt;
if (!deadlineAt) {
@@ -224,6 +378,49 @@ function buildToolPlanAddendum(result: ToolPlanResult): string {
].join("\n");
}
function hasToolPlanResultForTool(
plan: PlannerOutput,
result: ToolPlanResult,
toolName: string,
): boolean {
return (
plan.toolPlan?.steps.some(
(step) =>
step.tool === toolName &&
Object.hasOwn(result.results, step.id),
) ?? false
);
}
function shouldInjectToolPlanResult(
derived: DerivedState,
result: ToolPlanResult,
): boolean {
if (!result.fallbackToLlm) {
return true;
}
return (
derived.intent.kind === "discord-verify-question" &&
Object.keys(result.results).length > 0
);
}
function shouldSuppressResponderTools(
derived: DerivedState,
plan: PlannerOutput,
result: ToolPlanResult,
): boolean {
if (!result.fallbackToLlm) {
return true;
}
return (
derived.intent.kind === "discord-verify-question" &&
hasToolPlanResultForTool(plan, result, "verify-discord-nickname")
);
}
function pickToolsForPlan(
plan: PlannerOutput,
tools: readonly Tool<unknown, unknown>[],
@@ -254,7 +451,6 @@ export class OnboardingOrchestrator {
options: OrchestratorRunOptions,
responderTools: readonly Tool<unknown, unknown>[],
maxTokens: number,
modelOverride: string,
): Promise<string | null> {
const logger = getRequestLogger();
const deadlineMs = getRemainingDeadlineMs();
@@ -275,7 +471,6 @@ export class OnboardingOrchestrator {
llmClient: this.llmClient,
maxTokens,
messages,
modelOverride,
role: "responder",
routingDecision: derived.routingDecision,
tier: derived.tier,
@@ -288,7 +483,6 @@ export class OnboardingOrchestrator {
const response = await this.llmClient.chatCompletion(messages, {
deadlineMs,
maxTokens,
modelOverride,
role: "responder",
routingDecision: derived.routingDecision,
temperature: options.temperature ?? getLlmTemperature(),
@@ -304,24 +498,29 @@ export class OnboardingOrchestrator {
): Promise<PlannerOutput> {
const logger = getRequestLogger();
const traceStep = getActiveProcessTrace()?.startStep("plan");
const verificationFocus = getDiscordVerificationFocus(derived, messages);
const finishTrace = (plan: PlannerOutput): PlannerOutput => {
const normalizedPlan = enforceRequiredToolPlan(
derived,
plan,
verificationFocus,
);
traceStep?.finish({
approach: plan.approach,
mustAvoid: plan.mustAvoid,
mustInclude: plan.mustInclude,
requestResources: plan.requestResources,
suggestedTools: plan.suggestedTools,
targetLengthChars: plan.targetLengthChars,
toolPlanStepCount: plan.toolPlan?.steps.length,
approach: normalizedPlan.approach,
mustAvoid: normalizedPlan.mustAvoid,
mustInclude: normalizedPlan.mustInclude,
requestResources: normalizedPlan.requestResources,
suggestedTools: normalizedPlan.suggestedTools,
targetLengthChars: normalizedPlan.targetLengthChars,
toolPlanStepCount: normalizedPlan.toolPlan?.steps.length,
});
return plan;
return normalizedPlan;
};
const plannerMessages = buildPlannerMessages(derived, messages);
const firstAttempt = await this.llmClient.chatCompletion(plannerMessages, {
deadlineMs: getRemainingDeadlineMs(),
jsonMode: true,
maxTokens: getPlannerMaxTokens(),
modelOverride: getTier1PlannerModel(),
role: "planner",
routingDecision: derived.routingDecision,
temperature: 0,
@@ -356,7 +555,6 @@ export class OnboardingOrchestrator {
deadlineMs: getRemainingDeadlineMs(),
jsonMode: true,
maxTokens: getPlannerMaxTokens(),
modelOverride: getTier1PlannerModel(),
role: "planner",
routingDecision: derived.routingDecision,
temperature: 0,
@@ -376,7 +574,7 @@ export class OnboardingOrchestrator {
return finishTrace(retryPlan);
}
const fallbackPlan = getDefaultPlan(derived);
const fallbackPlan = getDefaultPlan(derived, verificationFocus);
logger.debug(
{
intent: derived.intent.kind,
@@ -403,7 +601,6 @@ export class OnboardingOrchestrator {
options,
tools,
maxTokens,
getTier0Model(),
);
}
@@ -428,16 +625,19 @@ export class OnboardingOrchestrator {
);
}
const toolPlanResult = await this.runProgrammaticToolPlan(
derived,
plan,
finalResponderMessages,
options,
);
if (toolPlanResult && !toolPlanResult.fallbackToLlm) {
if (toolPlanResult && shouldInjectToolPlanResult(derived, toolPlanResult)) {
finalResponderMessages = injectSystemAddendum(
finalResponderMessages,
buildToolPlanAddendum(toolPlanResult),
);
}
if (toolPlanResult && shouldSuppressResponderTools(derived, plan, toolPlanResult)) {
responderTools = [];
}
@@ -447,7 +647,6 @@ export class OnboardingOrchestrator {
options,
responderTools,
options.maxTokens ?? getTier1MaxTokens(),
getTier1ResponderModel(),
);
}
@@ -472,16 +671,19 @@ export class OnboardingOrchestrator {
);
}
const toolPlanResult = await this.runProgrammaticToolPlan(
derived,
plan,
finalResponderMessages,
options,
);
if (toolPlanResult && !toolPlanResult.fallbackToLlm) {
if (toolPlanResult && shouldInjectToolPlanResult(derived, toolPlanResult)) {
finalResponderMessages = injectSystemAddendum(
finalResponderMessages,
buildToolPlanAddendum(toolPlanResult),
);
}
if (toolPlanResult && shouldSuppressResponderTools(derived, plan, toolPlanResult)) {
responderTools = [];
}
@@ -491,7 +693,6 @@ export class OnboardingOrchestrator {
options,
responderTools,
options.maxTokens ?? getTier2MaxTokens(),
getTier1ResponderModel(),
);
if (!draft) {
@@ -517,12 +718,19 @@ export class OnboardingOrchestrator {
}
private async runProgrammaticToolPlan(
derived: DerivedState,
plan: PlannerOutput,
_messages: ChatMessage[],
options: OrchestratorRunOptions,
): Promise<ToolPlanResult | null> {
if (
!getAgentProgrammaticToolCalling() ||
!getAgentProgrammaticToolCalling() &&
derived.intent.kind !== "discord-verify-question"
) {
return null;
}
if (
!plan.toolPlan ||
!options.request ||
!options.tools?.length
+45 -2
View File
@@ -102,9 +102,9 @@ describe("decidePreGenerationRouting", () => {
},
{
confidence: 0.3,
expected: { kind: "tier0-only" },
expected: { kind: "tier1-direct", skipPlanner: false },
intentKind: "off-topic-to-other" as const,
name: "keeps low-confidence off-topic on tier0",
name: "promotes low-confidence off-topic to tier1",
phase: { kind: "fresh" } as OnboardingPhase,
},
{
@@ -319,6 +319,49 @@ describe("decidePreGenerationRouting", () => {
).toEqual(expected);
});
it("keeps high-confidence low-risk requests on tier0", () => {
expect(
decidePreGenerationRouting({
classified: classified("age-question", 0.95),
derived: derived({ kind: "deadline-near", remaining: ["photo"] }, intent("age-question"), {
tier: 1,
}),
intentRiskTier: "low",
request: request(),
session: session(),
userEscalationCount: 0,
}),
).toEqual({ kind: "tier0-only" });
});
it("promotes low-confidence high-risk requests with escalation history", () => {
expect(
decidePreGenerationRouting({
classified: classified("unknown", 0.49),
derived: derived({ kind: "fresh" }, intent("unknown")),
intentRiskTier: "high",
request: request(),
session: session(),
userEscalationCount: 4,
}),
).toEqual({ kind: "tier1-direct", skipPlanner: false });
});
it("prefers tier0 during latency degradation unless safety already promoted", () => {
expect(
decidePreGenerationRouting({
classified: classified("age-question", 0.8),
derived: derived({ kind: "deadline-near", remaining: ["nickname"] }, intent("age-question"), {
tier: 1,
}),
intentRiskTier: "medium",
recentLatencyP95Ms: 10_000,
request: request(),
session: session(),
}),
).toEqual({ kind: "tier0-only" });
});
it("trains, saves, loads, and applies a learned router model", () => {
const lowConfidenceInput = {
classified: classified("age-question", 0.25),
+49
View File
@@ -3,6 +3,10 @@ import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import type { AgentRequest } from "@agent/types";
import {
getAgentCostAwareTierEnabled,
getAlertThresholdP95LatencyMs,
} from "../../../config.js";
import type {
ClassifiedIntent,
DerivedState,
@@ -58,8 +62,11 @@ export interface RouterTrainingSample {
export interface RouterInput {
classified: ClassifiedIntent;
derived: DerivedState;
intentRiskTier?: "low" | "medium" | "high" | undefined;
recentLatencyP95Ms?: number | undefined;
request: AgentRequest;
session: OnboardingSession | null;
userEscalationCount?: number | undefined;
}
export const ROUTER_FEATURE_NAMES = [
@@ -294,6 +301,43 @@ function needsStepOneHelp(phase: OnboardingPhase, intent: Intent): boolean {
);
}
function isLatencyDegraded(input: RouterInput): boolean {
const p95 = input.recentLatencyP95Ms;
return (
typeof p95 === "number" &&
Number.isFinite(p95) &&
p95 > getAlertThresholdP95LatencyMs("onboarding")
);
}
function maybeDecideCostAwareRouting(input: RouterInput): RoutingDecision | null {
if (!getAgentCostAwareTierEnabled()) {
return null;
}
const confidence = clamp01(input.classified.confidence);
const riskTier = input.intentRiskTier ?? "medium";
const escalationCount = Math.max(0, input.userEscalationCount ?? 0);
const lowLatency = !isLatencyDegraded(input);
if (confidence < 0.5 || (riskTier === "high" && escalationCount > 3)) {
if (input.derived.responseMode === "llm-with-tools") {
return { kind: "tier1-with-tools" };
}
return { kind: "tier1-direct", skipPlanner: false };
}
if (!lowLatency) {
return { kind: "tier0-only" };
}
if (confidence >= 0.9 && riskTier === "low") {
return { kind: "tier0-only" };
}
return null;
}
export function decidePreGenerationRouting(input: RouterInput): RoutingDecision {
const { classified, derived, request, session } = input;
const { confidence, intent } = classified;
@@ -334,6 +378,11 @@ export function decidePreGenerationRouting(input: RouterInput): RoutingDecision
};
}
const costAwareDecision = maybeDecideCostAwareRouting(input);
if (costAwareDecision) {
return costAwareDecision;
}
if (
phase.kind === "nickname-rejected" ||
intent.kind === "discord-verify-question" ||
@@ -142,4 +142,71 @@ describe("runToolPlan", () => {
toolName: "lookup-community-rule",
});
});
it("appends audit entries for successful tool calls", async () => {
const streams: Array<{
fields: Record<string, string>;
key: string;
options?: { maxLen?: number | undefined };
}> = [];
const tool: Tool<{ topic: string }, { text: string }> = {
name: "lookup-community-rule",
description: "규칙 조회",
requires: [],
argsSchema: z.object({
topic: z.string(),
}),
resultSchema: z.object({
text: z.string(),
}),
execute: () => Promise.resolve({ text: "하트 규칙" }),
};
await runToolPlan(
{
composeWith: "all-then-llm",
steps: [
{
id: "rule",
tool: "lookup-community-rule",
argsLiteral: {
topic: "heart",
},
},
],
},
[tool],
{
garnetClient: {
appendStream: (
key: string,
fields: Record<string, string>,
options?: { maxLen?: number | undefined },
) => {
streams.push({
fields,
key,
...(options ? { options } : {}),
});
return Promise.resolve("1-0");
},
} as never,
},
request(),
);
expect(streams[0]).toMatchObject({
key: "lolympus-agent:tool-calls",
fields: {
ok: "true",
skill: "onboarding",
toolName: "lookup-community-rule",
type: "tool-call",
},
options: {
maxLen: 100_000,
},
});
expect(streams[0]?.fields["argsHash"]).toEqual(expect.any(String));
});
});
@@ -1,5 +1,8 @@
import type { AgentRequest } from "@agent/types";
import { getRequestLogger } from "@obs/context";
import { withToolSpan } from "@obs/spans";
import { hashToolArgs, ToolAuditLogger } from "@tools/audit";
import { buildScopedContext } from "@tools/scoped-context";
import type {
Tool,
ToolContext,
@@ -208,6 +211,41 @@ function makePlanFailure(message: string, details?: unknown): ToolExecutionFailu
});
}
async function appendToolAudit(input: {
args: unknown;
ctx: ToolContext;
durationMs: number;
errorKind?: string | undefined;
ok: boolean;
request: AgentRequest;
resultSummary: string;
toolName: string;
}): Promise<void> {
if (!input.ctx.garnetClient) {
return;
}
try {
await new ToolAuditLogger(input.ctx.garnetClient).append({
type: "tool-call",
argsHash: hashToolArgs(input.args),
calledAt: new Date().toISOString(),
chatroomId: input.request.chatroomId,
durationMs: input.durationMs,
...(input.errorKind ? { errorKind: input.errorKind } : {}),
ok: input.ok,
resultSummary: input.resultSummary,
skill: input.request.skill,
toolName: input.toolName,
userId: input.request.userId,
});
} catch (error) {
getRequestLogger().warn(
{ err: error, toolName: input.toolName },
"[ToolAudit] Failed to append tool audit entry",
);
}
}
function orderPlan(plan: ToolPlan): OrderedPlan {
const failures: ToolExecutionFailure[] = [];
const byId = new Map<string, ToolPlanStep>();
@@ -278,19 +316,31 @@ async function runStep(input: {
const traceStep = getActiveProcessTrace()?.startStep("tool-call");
let args: Record<string, unknown> = {};
const startedAt = Date.now();
try {
args = buildArgs(input);
const validated = tool.argsSchema.parse(args);
const scopedContext = buildScopedContext(tool.requires, input.ctx, tool.name);
const result = await withToolSpan(
tool.name,
{
"tool.name": tool.name,
"tool.plan.step_id": input.step.id,
},
() => tool.execute(validated, input.ctx),
() => tool.execute(validated, scopedContext),
);
const structuredFailure = getStructuredFailure(input.step, result);
if (structuredFailure) {
await appendToolAudit({
args,
ctx: input.ctx,
durationMs: Math.max(0, Date.now() - startedAt),
errorKind: structuredFailure.kind,
ok: false,
request: input.request,
resultSummary: structuredFailure.message,
toolName: tool.name,
});
traceStep?.finish({
args,
errorKind: structuredFailure.kind,
@@ -302,15 +352,35 @@ async function runStep(input: {
}
input.results[input.step.id] = result;
const resultSummary = summarizeToolResult(result);
await appendToolAudit({
args,
ctx: input.ctx,
durationMs: Math.max(0, Date.now() - startedAt),
ok: true,
request: input.request,
resultSummary,
toolName: tool.name,
});
traceStep?.finish({
args,
name: tool.name,
ok: true,
resultSummary: summarizeToolResult(result),
resultSummary,
});
return null;
} catch (error) {
const failure = classifyError(tool.name, error);
await appendToolAudit({
args,
ctx: input.ctx,
durationMs: Math.max(0, Date.now() - startedAt),
errorKind: failure.kind,
ok: false,
request: input.request,
resultSummary: failure.message,
toolName: tool.name,
});
traceStep?.finish({
args,
errorKind: failure.kind,
@@ -26,7 +26,11 @@ export function selectToolsForDerived(
if (derived.intent.kind === 'discord-verify-question') {
return maybeUseToolSearch(
tools.filter((tool) => tool.name === 'verify-discord-nickname'),
tools.filter(
(tool) =>
tool.name === 'verify-discord-nickname' ||
tool.name === 'check-onboarding-progress',
),
);
}
+26
View File
@@ -1,6 +1,7 @@
import type { AgentRequest, AgentResponse, ConversationMessage } from '@agent/types';
import type { GarnetClient } from '@common/garnet';
import type { NatsClient } from '@common/nats';
import type { EscalationHistoryService } from '../../memory/escalation-history.js';
import type { MemoryService } from '../../memory/memory-types.js';
import {
buildLongTermSummary,
@@ -118,6 +119,7 @@ export class OnboardingSkill implements Skill {
shadowEvalPublisher?: ShadowEvalPublisher,
private readonly natsClient?: Pick<NatsClient, 'request'>,
private readonly memoryService?: MemoryService,
private readonly escalationHistoryService?: EscalationHistoryService,
) {
this.classifier = new OnboardingIntentClassifier(llmClient);
this.blackboardStore = new OnboardingBlackboardStore(
@@ -186,13 +188,17 @@ export class OnboardingSkill implements Skill {
const deriveStep = trace?.startStep('derive');
const derived = deriveState(session, request.trigger, classified.intent);
const escalationHistory = await this.loadEscalationHistory(request.userId);
const routerMode = getAgentRouterMode();
if (routerMode === 'static' || routerMode === 'learned-v1') {
const routerInput = {
classified,
derived,
intentRiskTier: this.capabilityCard().riskTier,
recentLatencyP95Ms: this.getRecentLatencyP95Ms(request),
request,
session,
userEscalationCount: escalationHistory?.totalEscalations ?? 0,
};
derived.routingDecision =
routerMode === 'learned-v1'
@@ -353,6 +359,26 @@ export class OnboardingSkill implements Skill {
return normalizeOnboardingSession(rawSession);
}
private async loadEscalationHistory(userId: string) {
if (!this.escalationHistoryService) {
return null;
}
try {
return await this.escalationHistoryService.load(userId);
} catch (error) {
getRequestLogger().warn(
{ err: error },
'[OnboardingSkill] Failed to load escalation history',
);
return null;
}
}
private getRecentLatencyP95Ms(request: AgentRequest): number | undefined {
const raw = request.context?.['recentLatencyP95Ms'];
return typeof raw === 'number' && Number.isFinite(raw) ? raw : undefined;
}
private async decideEarlyAbstention(
request: AgentRequest,
classified: ClassifiedIntent,
@@ -52,6 +52,15 @@ describe("OnboardingIntentClassifier", () => {
expect(result.source).toBe("regex");
});
it("classifies resident application completion reports as discord verification", async () => {
const result = await classifier.classify(
"저 디코는 어떻게 들어가요? 주민신청까지 해뒀습니다",
"user-message",
);
expect(result.intent.kind).toBe("discord-verify-question");
expect(result.source).toBe("regex");
});
it("classifies identity room questions as help-how", async () => {
const result = await classifier.classify(
"본인인증 방은 뭐에요?",
+43 -7
View File
@@ -1,17 +1,53 @@
import type { HandoffPacket } from "@agent/handoff-packet";
export function routeSupervisorTarget(packet: HandoffPacket | undefined): string | null {
const text = [
export interface SupervisorRuleRoute {
confidence: number;
target: string | null;
}
const RULES = [
{
pattern: /(규칙|신고|제재|경고|욕설|광고|홍보|appeal|policy|moderation)/iu,
target: "moderation",
},
{
pattern: /(가입|닉네임|하트|인증|디스코드|온보딩|onboarding)/iu,
target: "onboarding",
},
] as const;
function packetText(packet: HandoffPacket | undefined): string {
return [
packet?.userGoal,
packet?.summary,
packet?.handoffReason,
].join(" ");
}
if (/(규칙|신고|제재|경고|욕설|광고|홍보|appeal|policy|moderation)/iu.test(text)) {
return "moderation";
export function routeSupervisorByRules(
packet: HandoffPacket | undefined,
): SupervisorRuleRoute {
const text = packetText(packet);
const matches = RULES.filter((rule) => rule.pattern.test(text));
if (matches.length === 1) {
return {
confidence: 0.9,
target: matches[0]?.target ?? null,
};
}
if (/(가입|닉네임|하트|인증|디스코드|온보딩|onboarding)/iu.test(text)) {
return "onboarding";
if (matches.length > 1) {
return {
confidence: 0.6,
target: matches[0]?.target ?? null,
};
}
return null;
return {
confidence: 0,
target: null,
};
}
export function routeSupervisorTarget(packet: HandoffPacket | undefined): string | null {
return routeSupervisorByRules(packet).target;
}
+41 -1
View File
@@ -119,7 +119,7 @@ describe("SupervisorSkill", () => {
expect(response.text).toContain("운영자");
expect(response.metadata).toMatchObject({
routedBy: "rules",
routedBy: "regex-fast-path",
source: "template",
supervisorReason: "max-hop",
suggestedTarget: "onboarding",
@@ -149,6 +149,7 @@ describe("SupervisorSkill", () => {
});
it("uses LLM routing when the decision is confident and valid", async () => {
vi.stubEnv("AGENT_SUPERVISOR_FAST_PATH_ENABLED", "false");
const garnet = new FakeGarnet();
const nats = new FakeNats();
const skill = new SupervisorSkill(
@@ -189,6 +190,45 @@ describe("SupervisorSkill", () => {
});
});
it("uses regex fast-path before calling the supervisor LLM", async () => {
const garnet = new FakeGarnet();
const nats = new FakeNats();
const llm = {
chatCompletion: vi.fn(),
};
const skill = new SupervisorSkill(
garnet,
nats,
llm as unknown as Pick<LlmClient, "chatCompletion">,
registry,
);
const packet = buildHandoffPacket({
handoff: {
confidence: 0.7,
evidenceRefs: [],
expectedOutput: "text",
reason: "out-of-scope",
risk: "medium",
targetAgent: "supervisor",
taskSpec: "광고 규칙 문의",
},
request: baseRequest,
response: sourceResponse,
sourceChain: ["onboarding", "supervisor"],
});
const response = await skill.handle(
{ ...baseRequest, context: { handoffPacket: packet } },
[],
);
expect(llm.chatCompletion).not.toHaveBeenCalled();
expect(response.metadata).toMatchObject({
routedBy: "regex-fast-path",
suggestedTarget: "moderation",
});
});
it("uses safe LLM userMessage for supervisor replies", async () => {
const garnet = new FakeGarnet();
const nats = new FakeNats();
+29 -5
View File
@@ -3,6 +3,7 @@ import type {
AgentResponse,
ConversationMessage,
} from "@agent/types";
import { trace } from "@opentelemetry/api";
import { readHandoffPacket } from "@agent/handoff-packet";
import type { HandoffPacket } from "@agent/handoff-packet";
import type { GarnetClient } from "@common/garnet";
@@ -19,6 +20,8 @@ import type { Skill } from "@skills/skill";
import type { SkillCapabilityCard, SkillRegistry } from "@skills/skill-loader";
import { getLogger } from "@obs/logger";
import {
getAgentSupervisorFastPathEnabled,
getSupervisorFastPathConfidence,
getSupervisorLlmConfidenceThreshold,
getSupervisorLlmEnabled,
getSupervisorLlmResponseEnabled,
@@ -28,7 +31,7 @@ import {
buildSupervisorUserPrompt,
} from "./capability-prompt.js";
import { appendSupervisorReview } from "./review-queue.js";
import { routeSupervisorTarget } from "./routing-rules.js";
import { routeSupervisorByRules } from "./routing-rules.js";
import {
supervisorRoutingSchema,
type SupervisorRoutingDecision,
@@ -114,13 +117,30 @@ export class SupervisorSkill implements Skill {
): Promise<AgentResponse> {
const packet = readHandoffPacket(request);
const escalationHistory = await this.loadEscalationHistory(request.userId);
const llmDecision = await this.routeByLlm(packet, escalationHistory);
const rulesDecision = routeSupervisorByRules(packet);
const useRegexFastPath =
getAgentSupervisorFastPathEnabled() &&
rulesDecision.target !== null &&
rulesDecision.confidence >= getSupervisorFastPathConfidence() &&
this.isKnownTarget(rulesDecision.target);
const llmDecision = useRegexFastPath
? null
: await this.routeByLlm(packet, escalationHistory);
const useLlmRoute =
llmDecision !== null &&
llmDecision.confidence >= getSupervisorLlmConfidenceThreshold();
const target = useLlmRoute
? llmDecision.target
: routeSupervisorTarget(packet);
: rulesDecision.target;
const routedBy = useRegexFastPath
? "regex-fast-path"
: useLlmRoute
? "llm"
: "rules-fallback";
trace.getActiveSpan()?.setAttributes({
"supervisor.routed_by": routedBy,
"supervisor.rules_confidence": rulesDecision.confidence,
});
const reason = packet?.handoffReason ?? "out-of-scope";
const queuedAt = new Date().toISOString();
@@ -169,7 +189,8 @@ export class SupervisorSkill implements Skill {
return {
confidence: 0.6,
metadata: {
routedBy: useLlmRoute ? "llm" : "rules",
routedBy,
rulesConfidence: rulesDecision.confidence,
source: userMessage.source,
supervisorReason: useLlmRoute ? llmDecision.reason : reason,
supervisorEscalationTotal: escalationHistory?.totalEscalations ?? 0,
@@ -234,7 +255,10 @@ export class SupervisorSkill implements Skill {
private isKnownTarget(target: string): boolean {
return (
target === "human-review" || Boolean(this.skillRegistry?.get(target))
target === "human-review" ||
target === "onboarding" ||
target === "moderation" ||
Boolean(this.skillRegistry?.get(target))
);
}
+135
View File
@@ -0,0 +1,135 @@
import { createHash } from "node:crypto";
import type { GarnetClient } from "@common/garnet";
import { getToolAuditMaxLen } from "../config.js";
export const TOOL_AUDIT_STREAM = "lolympus-agent:tool-calls";
export interface ToolAuditEntry {
type: "tool-call";
toolName: string;
chatroomId: string;
userId: string;
argsHash: string;
resultSummary: string;
ok: boolean;
errorKind?: string | undefined;
durationMs: number;
skill: string;
traceId?: string | undefined;
calledAt: string;
}
export interface ListedToolAuditEntry extends ToolAuditEntry {
id: string;
}
function normalizeForHash(value: unknown): unknown {
if (Array.isArray(value)) {
return value.map(normalizeForHash);
}
if (typeof value !== "object" || value === null) {
return value;
}
return Object.fromEntries(
Object.entries(value as Record<string, unknown>)
.filter(([key]) => !/(password|secret|token|api[_-]?key|authorization)/iu.test(key))
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, entry]) => [key, normalizeForHash(entry)]),
);
}
function asString(value: string | undefined): string | undefined {
return value?.trim() ? value : undefined;
}
function parseAuditEntry(
id: string,
fields: Record<string, string>,
): ListedToolAuditEntry | null {
if (fields["type"] !== "tool-call") {
return null;
}
return {
id,
type: "tool-call",
argsHash: fields["argsHash"] ?? "",
calledAt: fields["calledAt"] ?? "",
chatroomId: fields["chatroomId"] ?? "",
durationMs: Number(fields["durationMs"] ?? 0),
...(asString(fields["errorKind"]) ? { errorKind: fields["errorKind"] } : {}),
ok: fields["ok"] === "true",
resultSummary: fields["resultSummary"] ?? "",
skill: fields["skill"] ?? "",
...(asString(fields["traceId"]) ? { traceId: fields["traceId"] } : {}),
toolName: fields["toolName"] ?? "",
userId: fields["userId"] ?? "",
};
}
export function hashToolArgs(args: unknown): string {
return createHash("sha256")
.update(JSON.stringify(normalizeForHash(args)))
.digest("hex")
.slice(0, 16);
}
export class ToolAuditLogger {
constructor(
private readonly garnetClient: Pick<
GarnetClient,
"appendStream" | "readStreamRange"
>,
) {}
async append(entry: ToolAuditEntry): Promise<void> {
await this.garnetClient.appendStream(
TOOL_AUDIT_STREAM,
Object.fromEntries(
Object.entries(entry).map(([key, value]) => [key, String(value)]),
),
{ maxLen: getToolAuditMaxLen() },
);
}
async list(opts: {
skill?: string | undefined;
since?: string | undefined;
limit?: number | undefined;
}): Promise<ListedToolAuditEntry[]> {
const limit = Math.max(1, Math.min(opts.limit ?? 50, 500));
const entries = await this.garnetClient.readStreamRange(
TOOL_AUDIT_STREAM,
"-",
"+",
{ count: limit * 4 },
);
const sinceTime = opts.since ? Date.parse(opts.since) : NaN;
return entries
.map((entry) => parseAuditEntry(entry.id, entry.message))
.filter((entry): entry is ListedToolAuditEntry => entry !== null)
.filter((entry) => !opts.skill || entry.skill === opts.skill)
.filter((entry) => {
if (!Number.isFinite(sinceTime)) {
return true;
}
const calledAtTime = Date.parse(entry.calledAt);
return Number.isFinite(calledAtTime) && calledAtTime >= sinceTime;
})
.slice(-limit)
.reverse();
}
async get(id: string): Promise<ListedToolAuditEntry | null> {
const entries = await this.garnetClient.readStreamRange(
TOOL_AUDIT_STREAM,
id,
id,
{ count: 1 },
);
const first = entries[0];
return first ? parseAuditEntry(first.id, first.message) : null;
}
}
+28 -1
View File
@@ -1,6 +1,13 @@
import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import { describe, expect, it } from "vitest";
import type { MemoryService } from "../../memory/memory-types.js";
import { lookupRuleTool } from "./lookup-rule.js";
import {
lookupRuleTool,
resolveCommunityRulesPath,
} from "./lookup-rule.js";
const failingMemoryService = {
longTerm: {
@@ -9,6 +16,26 @@ const failingMemoryService = {
} as unknown as MemoryService;
describe("lookupRuleTool", () => {
it("falls back to the source rules resource when imported from dist", () => {
const root = mkdtempSync(join(tmpdir(), "community-rules-resource-"));
const sourceResourceDir = join(
root,
"src",
"skills",
"moderation",
"resources",
);
mkdirSync(sourceResourceDir, { recursive: true });
const sourceRulesPath = join(sourceResourceDir, "community-rules.md");
writeFileSync(sourceRulesPath, "## ads\n광고 금지\n");
const moduleUrl = pathToFileURL(
join(root, "dist", "tools", "moderation", "lookup-rule.js"),
).href;
expect(resolveCommunityRulesPath(moduleUrl, root)).toBe(sourceRulesPath);
});
it("returns the matching community rule section", async () => {
const result = await lookupRuleTool.execute({ topic: "ads" }, {});
+28 -5
View File
@@ -1,4 +1,5 @@
import { readFileSync } from "node:fs";
import { existsSync, readFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { z } from "zod";
import {
@@ -40,10 +41,31 @@ type RuleTopic =
| "sanctions"
| "spam";
const RULES_PATH = fileURLToPath(
new URL("../../skills/moderation/resources/community-rules.md", import.meta.url),
);
const RULES_TEXT = readFileSync(RULES_PATH, "utf8");
const RULES_RELATIVE_PATH = "skills/moderation/resources/community-rules.md";
export function resolveCommunityRulesPath(
moduleUrl = import.meta.url,
cwd = process.cwd(),
): string {
const moduleDir = dirname(fileURLToPath(moduleUrl));
const candidates = [
resolve(moduleDir, "../../", RULES_RELATIVE_PATH),
resolve(moduleDir, "../../../src", RULES_RELATIVE_PATH),
resolve(cwd, "dist", RULES_RELATIVE_PATH),
resolve(cwd, "src", RULES_RELATIVE_PATH),
];
const found = candidates.find((candidate) => existsSync(candidate));
if (!found) {
throw new Error(
`Missing moderation community rules resource. Tried: ${candidates.join(", ")}`,
);
}
return found;
}
const RULES_TEXT = readFileSync(resolveCommunityRulesPath(), "utf8");
function extractRuleSection(topic: RuleTopic): string {
const pattern = new RegExp(`^## ${topic}\\n([\\s\\S]*?)(?=\\n## |$)`, "mu");
@@ -93,6 +115,7 @@ export const lookupRuleTool: Tool<
name: "lookup-rule",
description:
"광고, 도배, 욕설, 신고, 제재, 이의 제기, 개인정보 같은 moderation 규칙을 조회한다.",
requires: ["memory-search"],
argsSchema,
resultSchema,
async execute(args, ctx) {
@@ -44,6 +44,8 @@ describe("checkOnboardingProgressTool", () => {
expect(result.step1Done).toBe(true);
expect(result.overallDone).toBe(false);
expect(result.discordProfileNicknameDone).toBe(false);
expect(result.discordProfileMessageDone).toBe(false);
expect(result.remainingSteps).toEqual([
"discord-profile-nickname",
"discord-profile-message",
@@ -104,6 +106,8 @@ describe("checkOnboardingProgressTool", () => {
expect(result.step1Done).toBe(true);
expect(result.overallDone).toBe(true);
expect(result.discordProfileNicknameDone).toBe(null);
expect(result.discordProfileMessageDone).toBe(null);
expect(result.remainingSteps).toEqual([]);
});
});
@@ -18,6 +18,8 @@ const argsSchema = z.object({
});
const resultSchema = z.object({
discordProfileMessageDone: z.boolean().nullable(),
discordProfileNicknameDone: z.boolean().nullable(),
nicknameDone: z.boolean(),
photoDone: z.boolean(),
step1Done: z.boolean(),
@@ -41,6 +43,7 @@ export const checkOnboardingProgressTool: Tool<
name: "check-onboarding-progress",
description:
"현재 유저의 온보딩 진행 상태를 읽어 카카오톡 단계와 디스코드 단계를 포함한 남은 절차를 구조화해서 반환한다.",
requires: ["garnet", "user-session"],
argsSchema,
resultSchema,
async execute(args, ctx) {
@@ -61,6 +64,8 @@ export const checkOnboardingProgressTool: Tool<
: null;
return {
discordProfileMessageDone: session?.discordProfileMessageVerified ?? null,
discordProfileNicknameDone: session?.discordProfileNicknameVerified ?? null,
nicknameDone: session?.nicknameVerified ?? false,
photoDone: session?.photoVerified ?? false,
step1Done: isStep1Completed(session),
+1
View File
@@ -24,6 +24,7 @@ export const getRoomContextTool: Tool<
name: "get-room-context",
description:
"최근 채팅방 대화를 조회해서 현재 유저 메시지가 다른 사람에게 향한 말인지, 방 분위기가 어떤지 판단할 수 있게 돕는다.",
requires: ["room-context"],
argsSchema,
resultSchema,
async execute(args, ctx) {
@@ -90,6 +90,7 @@ export const lookupCommunityRuleTool: Tool<
name: "lookup-community-rule",
description:
"나이 제한, 닉네임 형식, 티어 코드, 하트 인증 규칙, 디스코드 안내 같은 커뮤니티 규칙을 조회한다.",
requires: ["memory-search"],
argsSchema,
resultSchema,
async execute(args, ctx) {
@@ -32,6 +32,7 @@ export const validateNicknameTool: Tool<
name: "validate-nickname",
description:
"새내기 닉네임 형식이 이름/출생 연도/성별/티어 규칙에 맞는지 검사하고, 틀린 부분을 구조화해서 반환한다.",
requires: [],
argsSchema,
resultSchema,
execute(args) {
@@ -54,6 +54,16 @@ describe("verifyDiscordNicknameTool", () => {
username: "priri",
},
],
residentApplicationMessageMatches: [
{
authorId: "discord-1",
channelId: "resident-channel-1",
content: "프리리#KR1/95/E",
createdAt: "2026-04-17T00:12:00.000Z",
matchedBy: "author-id",
messageId: "message-1",
},
],
joinedAfterMatches: [],
};
const request = vi.fn(
@@ -81,12 +91,88 @@ describe("verifyDiscordNicknameTool", () => {
expect(result.status).toBe("matched-renamed");
expect(result.message).toContain("올바른 형식으로 설정되어 있어요");
expect(result.message).toContain("주민신청 메시지도 확인됐어요");
expect(result.discordMemberId).toBe("discord-1");
expect(result.residentApplicationMessageStatus).toBe("confirmed");
expect(result.residentApplicationMessageId).toBe("message-1");
expect(result.residentApplicationMessageContent).toBe("프리리#KR1/95/E");
expect(result.residentApplicationMessageExpectedContent).toBe("프리리#KR1/95/E");
expect(session.discordMemberId).toBe("discord-1");
expect(session.discordProfileNicknameVerified).toBe(true);
expect(session.discordProfileMessageVerified).toBe(true);
expect(session.discordProfileMessageVerifiedAt).toBe(
"2026-04-17T00:12:00.000Z",
);
expect(request).toHaveBeenCalledTimes(1);
expect(request.mock.calls[0]?.[1]).toMatchObject({
includeResidentApplicationMessages: true,
searchName: "프리리",
});
expect(setJson).toHaveBeenCalledTimes(1);
});
it("does not confirm resident application messages that differ from the server nickname", async () => {
const setJson = vi.fn().mockResolvedValue(undefined);
const response: DiscordMemberLookupResponse = {
status: "ok",
nicknameMatches: [
{
memberId: "discord-1",
joinedAt: "2026-04-17T00:10:00.000Z",
globalName: "프리리",
guildNickname: "프리리#KR1/95/E",
matchedBy: "guild-nickname",
username: "priri",
},
],
residentApplicationMessageMatches: [
{
authorId: "discord-1",
channelId: "resident-channel-1",
content: "프리리/E",
createdAt: "2026-04-17T00:12:00.000Z",
matchedBy: "author-id",
messageId: "message-1",
},
],
joinedAfterMatches: [],
};
const request = vi.fn(
<TRequest = unknown, TResponse = unknown>(
_subject: string,
_payload: TRequest,
_options?: { timeoutMs?: number },
): Promise<TResponse> => Promise.resolve(response as TResponse),
);
const session = makeSession();
const result = await verifyDiscordNicknameTool.execute(
{},
{
chatroomId: session.chatroomId,
garnetClient: {
setJson,
} as never,
natsClient: {
request: request as never,
},
session,
userId: session.userId,
},
);
expect(result.status).toBe("matched-renamed");
expect(result.residentApplicationMessageStatus).toBe("content-mismatch");
expect(result.residentApplicationMessageContent).toBe("프리리/E");
expect(result.residentApplicationMessageExpectedContent).toBe("프리리#KR1/95/E");
expect(result.residentApplicationMessageCorrectionGuide).toContain(
"주민신청 메시지는 서버프로필 닉네임과 동일하게 작성해야 해요",
);
expect(result.message).toContain("현재 메시지는 '프리리/E'");
expect(session.discordProfileNicknameVerified).toBe(true);
expect(session.discordProfileMessageVerified).toBeUndefined();
});
it("does not mark renamed when only the raw kakao nickname matches literally", async () => {
const response: DiscordMemberLookupResponse = {
status: "ok",
@@ -121,7 +207,51 @@ describe("verifyDiscordNicknameTool", () => {
);
expect(result.status).toBe("matched-pending-rename");
expect(result.message).toContain("이름#태그/출생연도/티어");
expect(result.message).toContain("디스코드 닉네임이 형식과 맞지 않습니다.");
expect(result.message).toContain("현재 소환사님의 서버프로필은 '프리리/95/여/E'");
expect(result.message).toContain("닉네임#태그/출생연도/티어로 수정해 주세요.");
expect(result.recommendedNickname).toBe("프리리#태그/95/E");
expect(result.correctionGuide).toBe(result.message);
expect(result.guidance).toContain("이름#태그/출생연도/티어");
});
it("preserves an existing discord tag when suggesting a corrected nickname", async () => {
const response: DiscordMemberLookupResponse = {
status: "ok",
nicknameMatches: [
{
memberId: "discord-1",
globalName: "프리리",
guildNickname: "프리리#KR1/95/여/E",
joinedAt: "2026-04-17T00:10:00.000Z",
matchedBy: "global-name",
username: "priri",
},
],
joinedAfterMatches: [],
};
const request = vi.fn(
<TRequest = unknown, TResponse = unknown>(
_subject: string,
_payload: TRequest,
_options?: { timeoutMs?: number },
): Promise<TResponse> => Promise.resolve(response as TResponse),
);
const result = await verifyDiscordNicknameTool.execute(
{},
{
natsClient: {
request: request as never,
},
session: makeSession(),
},
);
expect(result.status).toBe("matched-pending-rename");
expect(result.recommendedNickname).toBe("프리리#KR1/95/E");
expect(result.correctionGuide).toContain("현재 소환사님의 서버프로필은 '프리리#KR1/95/여/E'");
expect(result.correctionGuide).toContain("할리#프렌치불독/88/D 와 같이");
});
it("falls back to joined-after candidates and returns not-confirmed when ambiguous", async () => {
+306 -16
View File
@@ -8,6 +8,7 @@ import type {
DiscordMemberCandidate,
DiscordMemberLookupRequest,
DiscordMemberLookupResponse,
DiscordResidentApplicationMessageCandidate,
} from '@common/nats-subjects';
import { getContext, getRequestLogger } from '@obs/context';
import {
@@ -20,6 +21,7 @@ import {
getKakaotalkNickname,
matchesDiscordNicknameRule,
parseKakaoNickname,
type ParsedKakaoNickname,
} from '@skills/onboarding/session';
import type { Tool, ToolContext } from '../tool.js';
@@ -50,6 +52,8 @@ const resultSchema = z.object({
searchName: z.string().optional(),
discordMemberId: z.string().optional(),
currentDisplayName: z.string().optional(),
recommendedNickname: z.string().optional(),
correctionGuide: z.string().optional(),
formatExample: z.string().optional(),
guidance: z.string().optional(),
issueMessages: z.array(z.string()).optional(),
@@ -57,6 +61,19 @@ const resultSchema = z.object({
matchSource: z.enum(['joined-after', 'nickname']).optional(),
matchedBy: z.enum(['global-name', 'guild-nickname', 'joined-after', 'username']).optional(),
nicknameMatchCount: z.number().int().nonnegative().optional(),
residentApplicationMessageAuthorId: z.string().optional(),
residentApplicationMessageContent: z.string().optional(),
residentApplicationMessageCorrectionGuide: z.string().optional(),
residentApplicationMessageCreatedAt: z.string().optional(),
residentApplicationMessageExpectedContent: z.string().optional(),
residentApplicationMessageId: z.string().optional(),
residentApplicationMessageMatchCount: z.number().int().nonnegative().optional(),
residentApplicationMessageMatchedBy: z
.enum(['author-id', 'author-name', 'content', 'joined-after'])
.optional(),
residentApplicationMessageStatus: z
.enum(['ambiguous', 'confirmed', 'content-mismatch', 'lookup-unavailable', 'not-confirmed'])
.optional(),
});
function pickDisplayName(candidate: DiscordMemberCandidate): string {
@@ -65,34 +82,249 @@ function pickDisplayName(candidate: DiscordMemberCandidate): string {
);
}
function resolveTimeoutMs(): number | null {
function extractExistingDiscordTag(
currentGuildNickname: string | undefined,
parsed: ParsedKakaoNickname,
): string | null {
const normalized = currentGuildNickname?.trim();
if (!normalized) {
return null;
}
const prefix = `${parsed.baseName}#`;
if (!normalized.startsWith(prefix)) {
return null;
}
const tag = normalized.slice(prefix.length).split('/')[0]?.trim();
return tag || null;
}
function buildRecommendedNickname(
parsed: ParsedKakaoNickname,
currentGuildNickname: string | undefined,
): string {
const tag = extractExistingDiscordTag(currentGuildNickname, parsed) ?? '태그';
return `${parsed.baseName}#${tag}/${parsed.year}/${parsed.tier}`;
}
function buildCorrectionGuide(args: {
currentDisplayName: string;
currentGuildNickname?: string | undefined;
}): string {
const formatGuide = `${DISCORD_NICKNAME_FORMAT_EXAMPLE} 와 같이 닉네임#태그/출생연도/티어로 수정해 주세요.`;
if (args.currentGuildNickname?.trim()) {
return `디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '${args.currentGuildNickname.trim()}' 으로 확인됩니다. ${formatGuide}`;
}
return `디스코드 닉네임이 형식과 맞지 않습니다. 현재 소환사님의 서버프로필은 '${args.currentDisplayName}' 으로 확인됩니다. ${formatGuide}`;
}
type ResidentApplicationMessageStatus =
| 'ambiguous'
| 'confirmed'
| 'content-mismatch'
| 'lookup-unavailable'
| 'not-confirmed';
interface ResidentApplicationMessageResult {
candidate?: DiscordResidentApplicationMessageCandidate | undefined;
correctionGuide?: string | undefined;
expectedContent?: string | undefined;
matchCount?: number | undefined;
status: ResidentApplicationMessageStatus;
}
function resolveResidentApplicationMessageResult(
matches: DiscordResidentApplicationMessageCandidate[] | undefined,
): ResidentApplicationMessageResult {
if (!matches) {
return { status: 'lookup-unavailable' };
}
if (matches.length === 0) {
return { matchCount: 0, status: 'not-confirmed' };
}
if (matches.length > 1) {
return { matchCount: matches.length, status: 'ambiguous' };
}
return {
candidate: matches[0],
matchCount: 1,
status: 'confirmed',
};
}
function normalizeMessageContent(content: string | undefined): string {
return content?.replace(/\s+/gu, ' ').trim() ?? '';
}
function buildResidentApplicationMessageCorrectionGuide(args: {
actualContent: string;
expectedContent: string;
}): string {
return `주민신청 메시지는 서버프로필 닉네임과 동일하게 작성해야 해요. 현재 메시지는 '${args.actualContent}' 로 확인됩니다. '${args.expectedContent}' 로 다시 남겨주세요.`;
}
function validateResidentApplicationMessageContent(
result: ResidentApplicationMessageResult,
expectedContent: string | undefined,
): ResidentApplicationMessageResult {
if (result.status !== 'confirmed' || !result.candidate || !expectedContent) {
return expectedContent ? { ...result, expectedContent } : result;
}
const actualContent = normalizeMessageContent(result.candidate.content);
const normalizedExpected = normalizeMessageContent(expectedContent);
if (actualContent === normalizedExpected) {
return {
...result,
expectedContent: normalizedExpected,
};
}
return {
...result,
correctionGuide: buildResidentApplicationMessageCorrectionGuide({
actualContent,
expectedContent: normalizedExpected,
}),
expectedContent: normalizedExpected,
status: 'content-mismatch',
};
}
function truncateMessageContent(content: string): string {
const normalized = content.replace(/\s+/gu, ' ').trim();
return normalized.length > 160 ? `${normalized.slice(0, 157)}...` : normalized;
}
function buildResidentApplicationMessageSummary(
result: ResidentApplicationMessageResult,
): string | null {
if (result.status === 'lookup-unavailable') {
return null;
}
if (result.status === 'confirmed' && result.candidate) {
return `주민신청 메시지도 확인됐어요.`;
}
if (result.status === 'content-mismatch' && result.correctionGuide) {
return result.correctionGuide;
}
if (result.status === 'ambiguous') {
return '주민신청 메시지 후보가 여러 개라 하나로 확정하지 못했어요.';
}
return '주민신청 채널 메시지는 아직 확인되지 않았어요.';
}
function residentApplicationMessageFields(
result: ResidentApplicationMessageResult,
): Pick<
z.infer<typeof resultSchema>,
| 'residentApplicationMessageAuthorId'
| 'residentApplicationMessageContent'
| 'residentApplicationMessageCorrectionGuide'
| 'residentApplicationMessageCreatedAt'
| 'residentApplicationMessageExpectedContent'
| 'residentApplicationMessageId'
| 'residentApplicationMessageMatchCount'
| 'residentApplicationMessageMatchedBy'
| 'residentApplicationMessageStatus'
> {
const candidate = result.candidate;
return {
residentApplicationMessageCorrectionGuide: result.correctionGuide,
residentApplicationMessageExpectedContent: result.expectedContent,
residentApplicationMessageMatchCount: result.matchCount,
residentApplicationMessageStatus: result.status,
...(candidate
? {
residentApplicationMessageAuthorId: candidate.authorId,
residentApplicationMessageContent: truncateMessageContent(candidate.content),
residentApplicationMessageCreatedAt: candidate.createdAt,
residentApplicationMessageId: candidate.messageId,
residentApplicationMessageMatchedBy: candidate.matchedBy,
}
: {}),
};
}
interface TimeoutBudget {
configuredTimeoutMs: number;
deadlineAt?: string | undefined;
remainingMs?: number | undefined;
timeoutMs: number | null;
}
function resolveTimeoutBudget(): TimeoutBudget {
const configured = getDiscordVerifyTimeoutMs();
const deadlineAt = getContext()?.deadlineAt;
if (!deadlineAt) {
return configured;
return {
configuredTimeoutMs: configured,
timeoutMs: configured,
};
}
const remainingMs = Date.parse(deadlineAt) - Date.now() - 100;
if (!Number.isFinite(remainingMs) || remainingMs <= 0) {
return null;
return {
configuredTimeoutMs: configured,
deadlineAt,
remainingMs,
timeoutMs: null,
};
}
return Math.min(configured, remainingMs);
return {
configuredTimeoutMs: configured,
deadlineAt,
remainingMs,
timeoutMs: Math.min(configured, remainingMs),
};
}
async function persistDiscordMemberId(memberId: string, ctx: ToolContext): Promise<void> {
async function persistDiscordVerification(
input: {
memberId: string;
nicknameVerified: boolean;
residentApplicationMessageVerifiedAt?: string | undefined;
},
ctx: ToolContext,
): Promise<void> {
if (!ctx.garnetClient || !ctx.session) {
return;
}
const now = new Date().toISOString();
const nextSession = {
...ctx.session,
discordMemberId: memberId,
discordMemberId: input.memberId,
...(input.nicknameVerified
? {
discordProfileNicknameVerified: true,
discordProfileNicknameVerifiedAt:
ctx.session.discordProfileNicknameVerifiedAt ?? now,
}
: {}),
...(input.residentApplicationMessageVerifiedAt
? {
discordProfileMessageVerified: true,
discordProfileMessageVerifiedAt: input.residentApplicationMessageVerifiedAt,
}
: {}),
};
const key = `${getOnboardingSessionKeyPrefix()}:onboarding:${nextSession.chatroomId}:${nextSession.userId}`;
await ctx.garnetClient.setJson(key, nextSession);
ctx.session.discordMemberId = memberId;
Object.assign(ctx.session, nextSession);
}
export const verifyDiscordNicknameTool: Tool<
@@ -102,6 +334,7 @@ export const verifyDiscordNicknameTool: Tool<
name: 'verify-discord-nickname',
description:
'현재 유저의 카카오톡 인증 닉네임과 세션 입장 시각을 기준으로 디스코드 멤버를 찾아, 닉네임 변경이 확인됐는지 구조화해서 반환한다.',
requires: ['garnet', 'nats', 'user-session'],
argsSchema,
resultSchema,
async execute(_args, ctx) {
@@ -145,8 +378,7 @@ export const verifyDiscordNicknameTool: Tool<
}
const parsedKakaoNickname = parseKakaoNickname(expectedNickname);
const searchName = parsedKakaoNickname?.baseName ?? '';
if (!searchName) {
if (!parsedKakaoNickname?.baseName) {
logger.debug(
{
expectedNickname,
@@ -160,6 +392,7 @@ export const verifyDiscordNicknameTool: Tool<
expectedNickname,
};
}
const searchName = parsedKakaoNickname.baseName;
if (!ctx.natsClient) {
logger.debug(
@@ -178,11 +411,15 @@ export const verifyDiscordNicknameTool: Tool<
};
}
const timeoutMs = resolveTimeoutMs();
const timeoutBudget = resolveTimeoutBudget();
const timeoutMs = timeoutBudget.timeoutMs;
if (!timeoutMs) {
logger.debug(
{
configuredTimeoutMs: timeoutBudget.configuredTimeoutMs,
deadlineAt: timeoutBudget.deadlineAt,
expectedNickname,
remainingMs: timeoutBudget.remainingMs,
searchName,
},
'[VerifyDiscordNickname] No timeout budget left for lookup',
@@ -200,6 +437,7 @@ export const verifyDiscordNicknameTool: Tool<
try {
const payload: DiscordMemberLookupRequest = {
chatroomId: ctx.chatroomId ?? ctx.session.chatroomId,
includeResidentApplicationMessages: true,
joinedAfter: ctx.session.joinedAt,
searchName,
userId: ctx.userId ?? ctx.session.userId,
@@ -208,6 +446,9 @@ export const verifyDiscordNicknameTool: Tool<
logger.debug(
{
payload,
configuredTimeoutMs: timeoutBudget.configuredTimeoutMs,
deadlineAt: timeoutBudget.deadlineAt,
remainingMs: timeoutBudget.remainingMs,
subject: getDiscordMemberLookupSubject(),
timeoutMs,
},
@@ -256,11 +497,18 @@ export const verifyDiscordNicknameTool: Tool<
const nicknameMatches = lookupResult.nicknameMatches;
const joinedAfterMatches = lookupResult.joinedAfterMatches;
const unvalidatedResidentApplicationMessageResult = resolveResidentApplicationMessageResult(
lookupResult.residentApplicationMessageMatches,
);
logger.debug(
{
expectedNickname,
joinedAfterMatches: joinedAfterMatches.length,
nicknameMatches: nicknameMatches.length,
residentApplicationMessageMatchCount:
unvalidatedResidentApplicationMessageResult.matchCount,
residentApplicationMessageStatus:
unvalidatedResidentApplicationMessageResult.status,
searchName,
},
'[VerifyDiscordNickname] Lookup response received',
@@ -292,6 +540,10 @@ export const verifyDiscordNicknameTool: Tool<
expectedNickname,
joinedAfterMatchCount: joinedAfterMatches.length,
nicknameMatchCount: nicknameMatches.length,
residentApplicationMessageMatchCount:
unvalidatedResidentApplicationMessageResult.matchCount,
residentApplicationMessageStatus:
unvalidatedResidentApplicationMessageResult.status,
reason:
reason ??
(nicknameMatches.length === 0 ? 'nickname-search-none' : 'nickname-search-ambiguous'),
@@ -309,29 +561,63 @@ export const verifyDiscordNicknameTool: Tool<
searchName,
joinedAfterMatchCount: joinedAfterMatches.length,
nicknameMatchCount: nicknameMatches.length,
...residentApplicationMessageFields(unvalidatedResidentApplicationMessageResult),
};
}
await persistDiscordMemberId(candidate.memberId, ctx);
const currentDisplayName = pickDisplayName(candidate);
const currentGuildNickname = candidate.guildNickname?.trim();
const isRenamed = matchesDiscordNicknameRule(currentGuildNickname, expectedNickname);
const status = isRenamed ? 'matched-renamed' : 'matched-pending-rename';
const recommendedNickname = buildRecommendedNickname(parsedKakaoNickname, currentGuildNickname);
const residentApplicationMessageExpectedContent = isRenamed
? currentGuildNickname
: recommendedNickname.includes('#태그/')
? undefined
: recommendedNickname;
const residentApplicationMessageResult = validateResidentApplicationMessageContent(
unvalidatedResidentApplicationMessageResult,
residentApplicationMessageExpectedContent,
);
await persistDiscordVerification(
{
memberId: candidate.memberId,
nicknameVerified: isRenamed,
residentApplicationMessageVerifiedAt:
residentApplicationMessageResult.status === 'confirmed'
? residentApplicationMessageResult.candidate?.createdAt
: undefined,
},
ctx,
);
const correctionGuide = buildCorrectionGuide({
currentDisplayName,
currentGuildNickname,
});
const discordFormatGuide = `디스코드 서버 닉네임은 이름#태그/출생연도/티어 형식이에요. 카카오톡 형식과 달리 성별은 빼고 태그(#)가 들어가요. 예: ${DISCORD_NICKNAME_FORMAT_EXAMPLE}`;
const residentApplicationMessageSummary = buildResidentApplicationMessageSummary(
residentApplicationMessageResult,
);
const message = isRenamed
? '디스코드 서버 닉네임이 올바른 형식으로 설정되어 있어요. 확인 완료!'
: currentGuildNickname
? `디스코드에서 유저를 찾았지만 서버 닉네임이 아직 올바른 형식으로 확인되지는 않았어요. 현재 서버 닉네임은 "${currentGuildNickname}"으로 보여요. ${discordFormatGuide}`
: `디스코드에서 유저를 찾았지만 서버 닉네임이 아직 설정되지 않았거나 확인되지 않았어요. 현재 표시 이름은 "${currentDisplayName}"으로 보여요. ${discordFormatGuide}`;
: correctionGuide;
const finalMessage = [message, residentApplicationMessageSummary]
.filter(Boolean)
.join(' ');
logger.debug(
{
currentDisplayName,
currentGuildNickname,
discordMemberId: candidate.memberId,
expectedNickname,
matchSource,
matchedBy: candidate.matchedBy,
recommendedNickname,
residentApplicationMessageMatchCount:
residentApplicationMessageResult.matchCount,
residentApplicationMessageStatus: residentApplicationMessageResult.status,
status,
},
'[VerifyDiscordNickname] Verification resolved',
@@ -339,12 +625,16 @@ export const verifyDiscordNicknameTool: Tool<
return {
status,
message,
message: finalMessage,
currentDisplayName,
correctionGuide: isRenamed ? undefined : correctionGuide,
discordMemberId: candidate.memberId,
expectedNickname,
guidance: isRenamed ? undefined : discordFormatGuide,
matchSource,
matchedBy: candidate.matchedBy,
recommendedNickname: isRenamed ? undefined : recommendedNickname,
...residentApplicationMessageFields(residentApplicationMessageResult),
searchName,
};
},
+62
View File
@@ -0,0 +1,62 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { buildScopedContext } from "./scoped-context.js";
import type { ToolContext } from "./tool.js";
describe("buildScopedContext", () => {
afterEach(() => {
vi.unstubAllEnvs();
});
it("returns the full context when sandbox is disabled", () => {
const ctx: ToolContext = {
chatroomId: "room-1",
garnetClient: {} as ToolContext["garnetClient"],
};
expect(buildScopedContext([], ctx, "pure-tool")).toBe(ctx);
});
it("throws when sandboxed tools access undeclared permissions", () => {
vi.stubEnv("AGENT_TOOL_SANDBOX_ENABLED", "true");
const ctx: ToolContext = {
chatroomId: "room-1",
garnetClient: {} as ToolContext["garnetClient"],
userId: "user-1",
};
const scoped = buildScopedContext([], ctx, "validate-nickname");
expect(scoped.chatroomId).toBe("room-1");
expect(() => scoped.garnetClient).toThrow(
"Tool validate-nickname does not declare requires:garnet",
);
});
it("exposes declared memory search without profile access", async () => {
vi.stubEnv("AGENT_TOOL_SANDBOX_ENABLED", "true");
const search = vi.fn(() => Promise.resolve([]));
const ctx = {
memoryService: {
longTerm: {
search,
loadUserProfile: vi.fn(),
},
session: {},
assembleWorkingMemory: vi.fn(),
},
} as unknown as ToolContext;
const scoped = buildScopedContext(
["memory-search"],
ctx,
"lookup-community-rule",
);
await scoped.memoryService?.longTerm.search("heart", { topK: 1 });
expect(search).toHaveBeenCalledWith("heart", { topK: 1 });
expect(() => scoped.memoryService?.longTerm.loadUserProfile("user-1")).toThrow(
"Tool lookup-community-rule does not declare requires:memory-profile",
);
});
});
+149
View File
@@ -0,0 +1,149 @@
import { getAgentToolSandboxEnabled } from "../config.js";
import type { LongTermMemory, MemoryService } from "../memory/memory-types.js";
import { getLogger } from "../observability/logger.js";
import type { ToolContext, ToolPermission } from "./tool.js";
const PERMISSION_BY_FIELD: Record<string, ToolPermission> = {
garnetClient: "garnet",
memoryService: "memory-search",
natsClient: "nats",
roomContextService: "room-context",
session: "user-session",
toolRegistry: "tool-registry",
};
function deniedAccess(toolName: string, permission: ToolPermission): never {
throw new Error(`Tool ${toolName} does not declare requires:${permission}`);
}
function defineDeniedGetter<T extends object>(
target: T,
key: string,
toolName: string,
permission: ToolPermission,
): void {
Object.defineProperty(target, key, {
enumerable: true,
get: () => deniedAccess(toolName, permission),
});
}
function buildScopedMemoryService(
requires: ReadonlySet<ToolPermission>,
fullContext: ToolContext,
toolName: string,
): MemoryService | undefined {
if (!fullContext.memoryService) {
return undefined;
}
const longTerm: Partial<LongTermMemory> = {};
if (requires.has("memory-search")) {
longTerm.search = fullContext.memoryService.longTerm.search.bind(
fullContext.memoryService.longTerm,
);
} else {
defineDeniedGetter(longTerm, "search", toolName, "memory-search");
}
if (requires.has("memory-profile")) {
longTerm.loadUserProfile =
fullContext.memoryService.longTerm.loadUserProfile.bind(
fullContext.memoryService.longTerm,
);
} else {
defineDeniedGetter(longTerm, "loadUserProfile", toolName, "memory-profile");
}
defineDeniedGetter(longTerm, "updateUserProfile", toolName, "memory-profile");
defineDeniedGetter(longTerm, "upsertDocument", toolName, "memory-search");
defineDeniedGetter(longTerm, "deleteDocument", toolName, "memory-search");
return {
longTerm: longTerm as LongTermMemory,
session: fullContext.memoryService.session,
assembleWorkingMemory: fullContext.memoryService.assembleWorkingMemory.bind(
fullContext.memoryService,
),
};
}
function findUndeclaredContextPermissions(
requires: ReadonlySet<ToolPermission>,
fullContext: ToolContext,
): ToolPermission[] {
return [
...new Set(
Object.entries(PERMISSION_BY_FIELD)
.filter(
([field, permission]) =>
!requires.has(permission) &&
fullContext[field as keyof ToolContext] !== undefined,
)
.map(([, permission]) => permission),
),
];
}
export function buildScopedContext(
requires: readonly ToolPermission[] | undefined,
fullContext: ToolContext,
toolName = "unknown",
): ToolContext {
const sandboxEnabled = getAgentToolSandboxEnabled();
if (!requires) {
if (sandboxEnabled) {
getLogger().warn(
{ toolName },
"[ToolSandbox] Tool has no requires declaration; using full context",
);
}
return fullContext;
}
if (!sandboxEnabled) {
const undeclared = findUndeclaredContextPermissions(
new Set(requires),
fullContext,
);
if (undeclared.length > 0) {
getLogger().warn(
{ toolName, undeclared },
"[ToolSandbox] Tool context includes undeclared permissions",
);
}
return fullContext;
}
const required = new Set(requires);
const scoped: ToolContext = {
chatroomId: fullContext.chatroomId,
userId: fullContext.userId,
};
if (required.has("garnet")) {
scoped.garnetClient = fullContext.garnetClient;
}
if (required.has("nats")) {
scoped.natsClient = fullContext.natsClient;
}
if (required.has("room-context")) {
scoped.roomContextService = fullContext.roomContextService;
}
if (required.has("tool-registry")) {
scoped.toolRegistry = fullContext.toolRegistry;
}
if (required.has("user-session")) {
scoped.session = fullContext.session;
}
if (required.has("memory-search") || required.has("memory-profile")) {
scoped.memoryService = buildScopedMemoryService(required, fullContext, toolName);
}
for (const [field, permission] of Object.entries(PERMISSION_BY_FIELD)) {
if (field in scoped) {
continue;
}
defineDeniedGetter(scoped, field, toolName, permission);
}
return scoped;
}
+10
View File
@@ -17,6 +17,15 @@ export interface ToolContext {
toolRegistry?: ToolRegistry | undefined;
}
export type ToolPermission =
| "garnet"
| "memory-profile"
| "memory-search"
| "nats"
| "room-context"
| "tool-registry"
| "user-session";
export type ToolErrorKind =
| "args-invalid"
| "transient"
@@ -34,6 +43,7 @@ export interface ToolExecutionFailure {
export interface Tool<A, R> {
name: string;
description: string;
requires?: readonly ToolPermission[] | undefined;
argsSchema: ZodType<A>;
resultSchema: ZodType<R>;
execute(args: A, ctx: ToolContext): Promise<R>;