dahyeon 12e731a428 Iterate skill SSOT routing and eval support
Add skill priority routing and duplicate manifest handling.

Split eval golden sets by skill and add moderation rule lookup coverage.

Move onboarding runtime helpers out of the main skill class.
2026-05-18 09:19:17 -07:00
2026-05-14 14:58:29 -07:00

lolympus-agent

Overview

lolympus-agent is an LLM-based onboarding agent for the lolympus gaming community. It receives onboarding events, checks the user's current state, and replies with a short helpful message through the community's backend messaging pipeline.

Purpose

I created this service to automate repetitive onboarding guidance while keeping the response quality close to a human community staff member. The agent can answer common questions, validate onboarding progress, and route more complex cases through LLM-powered reasoning and tool calls.

Functionalities

1. Onboarding responses:
    * Handles new member, nickname, photo, progress, timeout, and completion events
    * Generates concise responses for user messages
    * Falls back to deterministic templates when an LLM response is not needed

2. Nickname and progress checks:
    * Validates community nickname format
    * Checks onboarding session progress from Redis/Garnet
    * Looks up room context when MongoDB is configured

3. Tool-aware LLM runtime:
    * Supports planner, responder, critic, and tool-loop execution modes
    * Routes requests across local, proxy, or OpenRouter-compatible LLM providers
    * Uses circuit breaking, health checks, and response validation

4. Evaluation pipeline:
    * Captures golden and shadow evaluation cases
    * Runs rule, LLM, process, attribution, and PRM-style judges
    * Supports review queues, calibration, drift checks, promotion, and tournaments

5. Background worker:
    * Processes shadow evaluation jobs asynchronously
    * Archives evaluation results and publishes alerts
    * Keeps real-time responses separate from slower analysis work

Used technologies

    * TypeScript
    * Node.js
    * NATS
    * Redis / Garnet
    * MongoDB / Mongoose
    * OpenTelemetry
    * Pino
    * Zod
    * Vitest
    * ESLint
    * Prettier

How to run

  1. Install dependencies:
npm install
  1. Create a local environment file:
cp env-vars/env.example env-vars/dev.env
  1. Fill in the required NATS, Redis/Garnet, MongoDB, and LLM provider values in env-vars/dev.env.

  2. Run checks:

npm run typecheck
npm test
  1. Start the agent:
npm run start:dev

Useful commands

npm run build
npm run lint
npm run eval
npm run eval:report
npm run eval:review

The service is designed to run as two processes in production: the real-time agent (src/main.ts) and the background worker (src/workers/background.ts).

S
Description
No description provided
Readme
16 MiB
0 Stars 1 Watchers 0 Forks
Languages
TypeScript 99.7%
PowerShell 0.2%
JavaScript 0.1%