Add skill priority routing and duplicate manifest handling. Split eval golden sets by skill and add moderation rule lookup coverage. Move onboarding runtime helpers out of the main skill class.
lolympus-agent
Overview
lolympus-agent is an LLM-based onboarding agent for the lolympus gaming community. It receives onboarding events, checks the user's current state, and replies with a short helpful message through the community's backend messaging pipeline.
Purpose
I created this service to automate repetitive onboarding guidance while keeping the response quality close to a human community staff member. The agent can answer common questions, validate onboarding progress, and route more complex cases through LLM-powered reasoning and tool calls.
Functionalities
1. Onboarding responses:
* Handles new member, nickname, photo, progress, timeout, and completion events
* Generates concise responses for user messages
* Falls back to deterministic templates when an LLM response is not needed
2. Nickname and progress checks:
* Validates community nickname format
* Checks onboarding session progress from Redis/Garnet
* Looks up room context when MongoDB is configured
3. Tool-aware LLM runtime:
* Supports planner, responder, critic, and tool-loop execution modes
* Routes requests across local, proxy, or OpenRouter-compatible LLM providers
* Uses circuit breaking, health checks, and response validation
4. Evaluation pipeline:
* Captures golden and shadow evaluation cases
* Runs rule, LLM, process, attribution, and PRM-style judges
* Supports review queues, calibration, drift checks, promotion, and tournaments
5. Background worker:
* Processes shadow evaluation jobs asynchronously
* Archives evaluation results and publishes alerts
* Keeps real-time responses separate from slower analysis work
Used technologies
* TypeScript
* Node.js
* NATS
* Redis / Garnet
* MongoDB / Mongoose
* OpenTelemetry
* Pino
* Zod
* Vitest
* ESLint
* Prettier
How to run
- Install dependencies:
npm install
- Create a local environment file:
cp env-vars/env.example env-vars/dev.env
-
Fill in the required NATS, Redis/Garnet, MongoDB, and LLM provider values in
env-vars/dev.env. -
Run checks:
npm run typecheck
npm test
- Start the agent:
npm run start:dev
Useful commands
npm run build
npm run lint
npm run eval
npm run eval:report
npm run eval:review
The service is designed to run as two processes in production: the real-time
agent (src/main.ts) and the background worker (src/workers/background.ts).