diff --git a/README.md b/README.md
index 8f0f65f..ae6d607 100644
--- a/README.md
+++ b/README.md
@@ -1,98 +1,774 @@
-
-
-
+# MSA 기반 웹 크롤러 서비스
-[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
-[circleci-url]: https://circleci.com/gh/nestjs/nest
+NestJS 기반의 마이크로서비스 아키텍처 웹 크롤러 시스템입니다. 정적/동적 웹페이지 크롤링, Pipeline/Stage 기반 병렬 실행, 다중 트리거 지원, Rate Limiting 등의 기능을 제공합니다.
- A progressive Node.js framework for building efficient and scalable server-side applications.
-
-
-
-
-
-
-
-
-
-
-
-
-
+## 주요 기능
-## Description
+### 크롤링 엔진
+- **정적 페이지 크롤링**: axios + cheerio 기반 HTML 파싱
+- **동적 SPA 크롤링**: Playwright 기반 JavaScript 렌더링 지원
+- **셀렉터 기반 데이터 추출**: CSS 셀렉터로 원하는 데이터만 추출
+- **속성 추출**: 이미지 URL 등 HTML 속성 추출 지원
-[Nest](https://github.com/nestjs/nest) framework TypeScript starter repository.
+### Pipeline/Stage 아키텍처
+- **병렬 실행**: 여러 Stage를 동시에 실행하여 성능 최적화
+- **실행 기록**: 각 Stage의 실행 이력 및 결과 저장
+- **메타데이터 기반**: 유연한 설정 및 확장 가능한 구조
-## Project setup
+### 트리거 시스템
+- **스케줄 트리거**: Cron 표현식 기반 정기 실행
+- **이벤트 트리거**: NATS 메시지 기반 이벤트 드리븐 실행
+- **API 트리거**: REST API 호출로 수동 실행
-```bash
-$ npm install
+### 안정성 및 성능
+- **Rate Limiting**: 도메인별 토큰 버킷 알고리즘으로 요청 제한
+- **재시도 로직**: Exponential backoff 기반 자동 재시도
+- **에러 알림**: 실패 시 NATS를 통한 알림 발송
+- **결과 저장**: MinIO에 크롤링 결과 자동 저장
+
+## 기술 스택
+
+### 프레임워크 및 런타임
+- **NestJS 11**: TypeScript 기반 서버 프레임워크
+- **Node.js**: JavaScript 런타임
+
+### 크롤링 및 파싱
+- **axios**: HTTP 클라이언트
+- **cheerio**: 서버사이드 jQuery (HTML 파싱)
+- **playwright**: 브라우저 자동화 (동적 페이지 크롤링)
+
+### 데이터베이스 및 스토리지
+- **MongoDB + Mongoose**: NoSQL 데이터베이스
+- **MinIO**: S3 호환 객체 스토리지
+
+### 메시징 및 이벤트
+- **NATS**: 메시지 브로커 (이벤트 트리거 및 알림)
+
+### 스케줄링 및 인증
+- **@nestjs/schedule + cron**: 작업 스케줄링
+- **JWT (jose)**: 토큰 기반 인증 (Microsoft Entra ID 지원)
+- **Infisical**: 중앙화된 비밀값 관리
+
+## 아키텍처
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ REST API │
+│ (JWT Authentication Required) │
+└────────────────────────┬────────────────────────────────────┘
+ │
+ ┌────────────────┼────────────────┐
+ │ │ │
+ ┌────▼─────┐ ┌──────▼──────┐ ┌────▼─────┐
+ │ Cron │ │ NATS Event │ │ API │
+ │ Trigger │ │ Trigger │ │ Trigger │
+ └────┬─────┘ └──────┬──────┘ └────┬─────┘
+ │ │ │
+ └────────────────┼────────────────┘
+ │
+ ┌──────────▼──────────┐
+ │ Pipeline Engine │
+ │ (병렬 실행 관리) │
+ └──────────┬──────────┘
+ │
+ ┌────────────────┼────────────────┐
+ │ │ │
+ ┌────▼─────┐ ┌──────▼──────┐ ┌────▼─────┐
+ │ Stage 1 │ │ Stage 2 │ │ Stage 3 │
+ │(EventSrc)│ │ (EventSrc) │ │(EventSrc)│
+ └────┬─────┘ └──────┬──────┘ └────┬─────┘
+ │ │ │
+ ┌────▼─────┐ ┌──────▼──────┐ ┌────▼─────┐
+ │ Crawler │ │ Crawler │ │ Crawler │
+ │ Service │ │ Service │ │ Service │
+ └────┬─────┘ └──────┬──────┘ └────┬─────┘
+ │ │ │
+ └────────────────┼────────────────┘
+ │
+ ┌──────────▼──────────┐
+ │ MinIO Storage │
+ │ (크롤링 결과 저장) │
+ └─────────────────────┘
```
-## Compile and run the project
+## 설치
+
+### 필수 요구사항
+
+- **Node.js**: 18.x 이상
+- **MongoDB**: 4.x 이상
+- **NATS**: 2.x 이상
+- **MinIO**: S3 호환 객체 스토리지
+- **Infisical**: 비밀값 관리 (선택)
+
+### 1. 저장소 클론 및 패키지 설치
```bash
-# development
-$ npm run start
-
-# watch mode
-$ npm run start:dev
-
-# production mode
-$ npm run start:prod
+git clone https://git.dahyeon.us/dahyeon/crawler.git
+cd crawler
+npm install
```
-## Run tests
+### 2. Playwright 브라우저 설치
```bash
-# unit tests
-$ npm run test
-
-# e2e tests
-$ npm run test:e2e
-
-# test coverage
-$ npm run test:cov
+npx playwright install chromium
```
-## Deployment
+### 3. 환경 변수 설정
-When you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the [deployment documentation](https://docs.nestjs.com/deployment) for more information.
-
-If you are looking for a cloud-based platform to deploy your NestJS application, check out [Mau](https://mau.nestjs.com), our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps:
+환경별 `.env` 파일을 생성합니다:
```bash
-$ npm install -g @nestjs/mau
-$ mau deploy
+# 개발 환경
+cp env-vars/example.env .env.dev
+
+# 테스트 환경
+cp env-vars/example.env .env.test
+
+# 프로덕션 환경
+cp env-vars/example.env .env.prod
```
-With Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure.
+#### 필수 환경 변수 (Infisical 사용 시)
-## Resources
+```env
+# Infisical 설정
+INFISICAL_SITE_URL=https://app.infisical.com
+INFISICAL_CLIENT_ID=your-client-id
+INFISICAL_CLIENT_SECRET=your-client-secret
+INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
+```
-Check out a few resources that may come in handy when working with NestJS:
+Infisical을 통해 다음 값들을 관리:
+- `MONGODB_CONNECTION_STRING`: MongoDB 연결 문자열
+- `NATS_ENDPOINT`: NATS 서버 주소
+- `NATS_USER`: NATS 사용자명
+- `NATS_PASS`: NATS 비밀번호
+- `JWT_CLAIM_AUDIENCE`: JWT 검증 audience
+- `MICROSOFT_ENTRAID_TENANT_ID`: Azure AD 테넌트 ID
+- `MINIO_ENDPOINT`: MinIO 서버 주소
+- `MINIO_ACCESS_KEY`: MinIO 액세스 키
+- `MINIO_SECRET_KEY`: MinIO 시크릿 키
-- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework.
-- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy).
-- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/).
-- Deploy your application to AWS with the help of [NestJS Mau](https://mau.nestjs.com) in just a few clicks.
-- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com).
-- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com).
-- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs).
-- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com).
+### 4. TLS 인증서 (선택)
-## Support
+NATS TLS 연결을 위해 `internal-service-ca.pem` 파일을 프로젝트 루트에 배치합니다.
-Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support).
+## 실행
-## Stay in touch
+### 개발 모드
-- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec)
-- Website - [https://nestjs.com](https://nestjs.com/)
-- Twitter - [@nestframework](https://twitter.com/nestframework)
+```bash
+npm run start:dev
+```
-## License
+### 프로덕션 빌드
-Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE).
+```bash
+npm run build
+npm run start:prod
+```
+
+### 환경 전환
+
+```bash
+# 테스트 환경
+NODE_ENV=test npm run start:dev
+
+# 프로덕션 환경
+NODE_ENV=prod npm run start:prod
+```
+
+## REST API
+
+서버 실행 후 Swagger 문서: `http://localhost:3000/api`
+
+### 인증
+
+모든 API는 JWT 인증이 필요합니다 (일부 Public 엔드포인트 제외).
+
+```http
+Authorization: Bearer {JWT_TOKEN}
+```
+
+### Pipelinedef 관리
+
+#### 1. Pipelinedef 생성
+
+```http
+POST /api/v1/crawler/pipelinedefs
+Content-Type: application/json
+
+{
+ "apiVersion": "v1",
+ "trigger": {
+ "type": "always"
+ },
+ "stageIds": ["stage-id-1", "stage-id-2"],
+ "schedule": {
+ "cron": "0 0 * * *",
+ "timezone": "Asia/Seoul"
+ }
+}
+```
+
+#### 2. Pipelinedef 목록 조회
+
+```http
+GET /api/v1/crawler/pipelinedefs?page=1&limit=20
+```
+
+#### 3. Pipelinedef 상세 조회
+
+```http
+GET /api/v1/crawler/pipelinedefs/{id}
+```
+
+#### 4. Pipelinedef 수정
+
+```http
+PUT /api/v1/crawler/pipelinedefs/{id}
+Content-Type: application/json
+
+{
+ "schedule": {
+ "cron": "0 12 * * *"
+ }
+}
+```
+
+#### 5. Pipelinedef 삭제
+
+```http
+DELETE /api/v1/crawler/pipelinedefs/{id}
+```
+
+### Stagedef 관리
+
+#### 1. Stagedef 생성
+
+```http
+POST /api/v1/crawler/stagedefs
+Content-Type: application/json
+
+{
+ "apiVersion": "v1",
+ "name": "Example Crawler",
+ "topic": "crawler.results.example",
+ "eventSource": [
+ {
+ "type": "web",
+ "name": "example-source",
+ "metadatas": [
+ {
+ "key": "url",
+ "value": "https://example.com",
+ "dataType": "string"
+ },
+ {
+ "key": "selectors",
+ "value": "{\"title\":\"h1\",\"content\":\".main\"}",
+ "dataType": "json"
+ },
+ {
+ "key": "dynamic",
+ "value": "false",
+ "dataType": "string"
+ }
+ ]
+ }
+ ]
+}
+```
+
+#### 2. Stagedef 목록 조회
+
+```http
+GET /api/v1/crawler/stagedefs?page=1&limit=20
+```
+
+### Pipeline 실행
+
+#### 1. Pipeline 수동 실행
+
+```http
+POST /api/v1/crawler/pipelines/trigger
+Content-Type: application/json
+
+{
+ "pipelineDefId": "pipeline-def-id",
+ "variables": {
+ "customVar": "value"
+ }
+}
+```
+
+#### 2. 실행된 Pipeline 목록
+
+```http
+GET /api/v1/crawler/pipelines?page=1&limit=20&status=completed
+```
+
+**status 옵션**: `completed`, `failed`, `partial`
+
+#### 3. Pipeline 실행 상세
+
+```http
+GET /api/v1/crawler/pipelines/{pipelineId}
+```
+
+#### 4. Pipeline의 Stage 목록
+
+```http
+GET /api/v1/crawler/pipelines/{pipelineId}/stages
+```
+
+### 모니터링
+
+#### 1. 등록된 스케줄 목록
+
+```http
+GET /api/v1/crawler/schedules
+```
+
+#### 2. Rate Limit 상태
+
+```http
+GET /api/v1/crawler/rate-limits
+```
+
+#### 3. 트리거 재로드
+
+```http
+POST /api/v1/crawler/triggers/reload
+```
+
+#### 4. Health Check (Public)
+
+```http
+GET /api/v1/crawler/health
+```
+
+## 사용 예시
+
+### 1. 정적 웹페이지 크롤링
+
+```json
+{
+ "type": "web",
+ "name": "news-crawler",
+ "metadatas": [
+ {
+ "key": "url",
+ "value": "https://news.example.com/latest",
+ "dataType": "string"
+ },
+ {
+ "key": "selectors",
+ "value": "{\"title\":\"h2.article-title\",\"summary\":\".article-summary\",\"author\":\".author-name\"}",
+ "dataType": "json"
+ },
+ {
+ "key": "dynamic",
+ "value": "false",
+ "dataType": "string"
+ }
+ ]
+}
+```
+
+### 2. 동적 SPA 페이지 크롤링
+
+```json
+{
+ "type": "web",
+ "name": "spa-crawler",
+ "metadatas": [
+ {
+ "key": "url",
+ "value": "https://spa-app.example.com/products",
+ "dataType": "string"
+ },
+ {
+ "key": "selectors",
+ "value": "{\"products\":\".product-item\",\"price\":\".price\"}",
+ "dataType": "json"
+ },
+ {
+ "key": "dynamic",
+ "value": "true",
+ "dataType": "string"
+ },
+ {
+ "key": "waitSelector",
+ "value": ".products-loaded",
+ "dataType": "string"
+ }
+ ]
+}
+```
+
+### 3. 이미지 URL 추출
+
+```json
+{
+ "type": "web",
+ "name": "image-crawler",
+ "metadatas": [
+ {
+ "key": "url",
+ "value": "https://gallery.example.com",
+ "dataType": "string"
+ },
+ {
+ "key": "selectors",
+ "value": "{\"title\":\"h1.gallery-title\"}",
+ "dataType": "json"
+ },
+ {
+ "key": "extractAttributes",
+ "value": "{\"imageUrl\":\"img.gallery-image@src\",\"thumbnail\":\"img.thumb@data-src\"}",
+ "dataType": "json"
+ },
+ {
+ "key": "dynamic",
+ "value": "false",
+ "dataType": "string"
+ }
+ ]
+}
+```
+
+### 4. 매일 자정 실행하는 스케줄 설정
+
+```json
+{
+ "apiVersion": "v1",
+ "trigger": {
+ "type": "always"
+ },
+ "stageIds": ["stage-id-1"],
+ "schedule": {
+ "cron": "0 0 * * *",
+ "timezone": "Asia/Seoul"
+ }
+}
+```
+
+### 5. NATS 이벤트로 트리거
+
+```json
+{
+ "apiVersion": "v1",
+ "trigger": {
+ "type": "kafka-event",
+ "kafkaEvent": {
+ "topic": "crawler.trigger.news"
+ }
+ },
+ "stageIds": ["stage-id-1"]
+}
+```
+
+## 프로젝트 구조
+
+```
+src/
+├── main.ts # 애플리케이션 진입점
+├── app.module.ts # 루트 모듈
+│
+├── global/ # 글로벌 공유 모듈
+│ ├── auth/ # JWT 인증
+│ │ ├── jwt.service.ts
+│ │ ├── jwt.guard.ts
+│ │ └── decorators...
+│ ├── secret/ # Infisical 비밀값 관리
+│ │ └── secret.service.ts
+│ ├── database/ # MongoDB 연결
+│ │ ├── connection.service.ts
+│ │ ├── schema.ts # Mongoose 스키마
+│ │ └── types.d.ts
+│ ├── nats/ # NATS 메시징
+│ │ └── nats.service.ts
+│ └── minio/ # MinIO 파일 저장소
+│ └── minio.service.ts
+│
+└── crawler/ # 크롤러 비즈니스 로직
+ ├── crawler.module.ts
+ ├── crawler.controller.ts # REST API 컨트롤러
+ │
+ ├── services/
+ │ ├── crawler.service.ts # HTTP + Playwright 크롤링
+ │ ├── rate-limiter.service.ts # Rate Limiting
+ │ ├── retry.service.ts # 재시도 로직
+ │ ├── stage-executor.service.ts # Stage 실행
+ │ ├── pipeline-engine.service.ts # Pipeline 관리
+ │ └── trigger.service.ts # 트리거 시스템
+ │
+ ├── dto/
+ │ ├── common.dto.ts
+ │ ├── create-pipelinedef.dto.ts
+ │ ├── create-stagedef.dto.ts
+ │ └── trigger-pipeline.dto.ts
+ │
+ └── interfaces/
+ └── crawler-result.interface.ts
+```
+
+## 데이터베이스 스키마
+
+### Pipelinedef
+파이프라인 정의 (재사용 가능한 템플릿)
+
+```typescript
+{
+ apiVersion: string; // "v1"
+ trigger: PipelineTrigger; // always | kafka-event
+ stage: ObjectId[]; // Stagedef ID 배열
+ metadatas: Metadata[]; // 커스텀 메타데이터
+}
+```
+
+### Pipeline
+실행된 파이프라인 인스턴스
+
+```typescript
+{
+ apiVersion: string;
+ name: string; // "pipeline-1234567890"
+ stage: Stage[]; // 실행 중인 Stage 배열
+ metadatas: Metadata[]; // 실행 정보 (triggeredBy, status 등)
+ createdAt: Date;
+ updatedAt: Date;
+}
+```
+
+### Stagedef
+스테이지 정의
+
+```typescript
+{
+ apiVersion: string;
+ name: string;
+ taxonomy?: string;
+ eventSource: EventSource[]; // 크롤링 대상 설정
+ topic: string; // NATS 발행 토픽
+ metadatas: Metadata[];
+}
+```
+
+### Stage
+실행된 스테이지
+
+```typescript
+{
+ apiVersion: string;
+ name: string;
+ executions: Execution[]; // 실행 기록
+ status: "pending" | "in-progress" | "completed" | "failed" | "cancelled" | "skipped";
+ metadatas: Metadata[];
+}
+```
+
+### Execution
+개별 실행 기록
+
+```typescript
+{
+ status: "pending" | "in-progress" | "completed" | "failed" | "cancelled" | "skipped";
+ metadatas: Metadata[]; // crawlResult, storageKey 등
+ createdAt: Date;
+ updatedAt: Date;
+}
+```
+
+## Rate Limiting
+
+### 도메인별 제한 설정
+
+```typescript
+// 기본값: 10 req, 초당 2개 보충
+rateLimiter.setLimit('example.com', {
+ maxTokens: 20, // 최대 20개 토큰
+ refillRate: 5 // 초당 5개 보충
+});
+```
+
+### 현재 상태 조회
+
+```http
+GET /api/v1/crawler/rate-limits
+```
+
+응답:
+```json
+{
+ "example.com": {
+ "config": {
+ "maxTokens": 20,
+ "refillRate": 5
+ },
+ "available": 15
+ }
+}
+```
+
+## 재시도 전략
+
+### RetryStrategy 설정
+
+```typescript
+{
+ limit: 3, // 최대 3회 재시도
+ delay: 1000, // 초기 대기 시간 1초
+ backoff: 2, // 2배씩 증가 (1초 → 2초 → 4초)
+ when: [
+ "failed", // 모든 실패
+ "5xx", // 5xx 에러
+ "network" // 네트워크 에러
+ ],
+ notify: [
+ "admin-channel" // 실패 시 알림 채널
+ ]
+}
+```
+
+## NATS 이벤트
+
+### 발행되는 이벤트
+
+#### Pipeline 시작
+```
+Topic: crawler.pipeline.started
+Payload: {
+ pipelineId: string;
+ pipelineDefId: string;
+ triggeredBy: "api" | "schedule" | "nats";
+ timestamp: string;
+}
+```
+
+#### Pipeline 완료
+```
+Topic: crawler.pipeline.completed
+Payload: {
+ pipelineId: string;
+ status: "completed" | "failed" | "partial";
+ triggeredBy: "api" | "schedule" | "nats";
+ stageResults: Array<{
+ stageName: string;
+ status: string;
+ error?: string;
+ }>;
+ completedAt: string;
+}
+```
+
+#### Stage 시작
+```
+Topic: crawler.stage.started
+Payload: {
+ pipelineId: string;
+ stageIndex: number;
+ stageName: string;
+ timestamp: string;
+}
+```
+
+#### Stage 완료 (커스텀 토픽)
+```
+Topic: {stagedef.topic}
+Payload: {
+ pipelineId: string;
+ stageIndex: number;
+ stageName: string;
+ status: "completed" | "failed";
+ results: Array<{
+ url: string;
+ success: boolean;
+ parsedData: object;
+ }>;
+ timestamp: string;
+}
+```
+
+## 개발
+
+### 테스트
+
+```bash
+# 단위 테스트
+npm run test
+
+# E2E 테스트
+npm run test:e2e
+
+# 커버리지
+npm run test:cov
+```
+
+### 린트
+
+```bash
+npm run lint
+npm run format
+```
+
+### 빌드
+
+```bash
+npm run build
+```
+
+## 트러블슈팅
+
+### Playwright 브라우저 설치 실패
+
+```bash
+# 수동 설치
+npx playwright install-deps
+npx playwright install chromium
+```
+
+### MongoDB 연결 실패
+
+- `MONGODB_CONNECTION_STRING` 환경 변수 확인
+- MongoDB 서버 상태 확인
+- 네트워크 연결 및 방화벽 설정 확인
+
+### NATS 연결 실패
+
+- `NATS_ENDPOINT`, `NATS_USER`, `NATS_PASS` 확인
+- TLS 인증서 경로 확인 (`internal-service-ca.pem`)
+- NATS 서버 상태 확인
+
+### Rate Limit 초과
+
+```typescript
+// 도메인별 제한 조정
+await fetch('http://localhost:3000/api/v1/crawler/rate-limits');
+// 또는 코드에서 직접 설정
+rateLimiter.setLimit('slow-domain.com', { maxTokens: 5, refillRate: 1 });
+```
+
+## 보안 고려사항
+
+1. **JWT 인증**: 모든 API 엔드포인트는 JWT 인증 필요
+2. **비밀값 관리**: Infisical을 통한 중앙화된 비밀값 관리
+3. **TLS 통신**: NATS 연결 시 TLS 인증서 사용
+4. **Rate Limiting**: 크롤링 대상 서버 보호
+5. **Input Validation**: DTO 기반 요청 검증
+
+## 라이선스
+
+MIT
+
+## 작성자
+
+- Dahyeon
+- Repository: https://git.dahyeon.us/dahyeon/crawler
+
+## 기술 지원
+
+이슈 및 문의: Repository Issues 페이지
diff --git a/env-vars/dev.env b/env-vars/dev.env
index 6f0a153..bf37204 100644
--- a/env-vars/dev.env
+++ b/env-vars/dev.env
@@ -1 +1 @@
-INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510
+INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
\ No newline at end of file
diff --git a/env-vars/example.env b/env-vars/example.env
index 36bce0a..fbebf71 100644
--- a/env-vars/example.env
+++ b/env-vars/example.env
@@ -1,3 +1,3 @@
INFISICAL_CLIENT_SECRET=
INFISICAL_CLIENT_ID=
-INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510
\ No newline at end of file
+INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
\ No newline at end of file
diff --git a/env-vars/prod.env b/env-vars/prod.env
index 0543dd8..bf37204 100644
--- a/env-vars/prod.env
+++ b/env-vars/prod.env
@@ -1 +1 @@
-INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510
\ No newline at end of file
+INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
\ No newline at end of file
diff --git a/env-vars/test.env b/env-vars/test.env
index 0543dd8..bf37204 100644
--- a/env-vars/test.env
+++ b/env-vars/test.env
@@ -1 +1 @@
-INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510
\ No newline at end of file
+INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
\ No newline at end of file
diff --git a/package-lock.json b/package-lock.json
index 2046176..e8d18b5 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -14,10 +14,18 @@
"@nestjs/config": "^4.0.2",
"@nestjs/core": "^11.0.1",
"@nestjs/platform-express": "^11.0.1",
+ "@nestjs/schedule": "^6.1.0",
"@nestjs/swagger": "^11.2.3",
+ "axios": "^1.13.2",
+ "cheerio": "^1.1.2",
+ "class-transformer": "^0.5.1",
+ "class-validator": "^0.14.3",
+ "cron": "^4.3.5",
+ "jose": "^6.1.3",
"minio": "^8.0.6",
"mongoose": "^9.0.2",
"nats": "^2.29.3",
+ "playwright": "^1.57.0",
"reflect-metadata": "^0.2.2",
"rxjs": "^7.8.1",
"temporal-polyfill": "^0.3.0"
@@ -28,8 +36,10 @@
"@nestjs/cli": "^11.0.0",
"@nestjs/schematics": "^11.0.0",
"@nestjs/testing": "^11.0.1",
+ "@types/cheerio": "^0.22.35",
"@types/express": "^5.0.0",
"@types/jest": "^30.0.0",
+ "@types/multer": "^2.0.0",
"@types/node": "^24.10.4",
"@types/supertest": "^6.0.2",
"eslint": "^9.18.0",
@@ -5818,6 +5828,19 @@
"@nestjs/core": "^11.0.0"
}
},
+ "node_modules/@nestjs/schedule": {
+ "version": "6.1.0",
+ "resolved": "https://registry.npmjs.org/@nestjs/schedule/-/schedule-6.1.0.tgz",
+ "integrity": "sha512-W25Ydc933Gzb1/oo7+bWzzDiOissE+h/dhIAPugA39b9MuIzBbLybuXpc1AjoQLczO3v0ldmxaffVl87W0uqoQ==",
+ "license": "MIT",
+ "dependencies": {
+ "cron": "4.3.5"
+ },
+ "peerDependencies": {
+ "@nestjs/common": "^10.0.0 || ^11.0.0",
+ "@nestjs/core": "^10.0.0 || ^11.0.0"
+ }
+ },
"node_modules/@nestjs/schematics": {
"version": "11.0.9",
"resolved": "https://registry.npmjs.org/@nestjs/schematics/-/schematics-11.0.9.tgz",
@@ -6243,6 +6266,16 @@
"@types/node": "*"
}
},
+ "node_modules/@types/cheerio": {
+ "version": "0.22.35",
+ "resolved": "https://registry.npmjs.org/@types/cheerio/-/cheerio-0.22.35.tgz",
+ "integrity": "sha512-yD57BchKRvTV+JD53UZ6PD8KWY5g5rvvMLRnZR3EQBCZXiDT/HR+pKpMzFGlWNhFrXlo7VPZXtKvIEwZkAWOIA==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/node": "*"
+ }
+ },
"node_modules/@types/connect": {
"version": "3.4.38",
"resolved": "https://registry.npmjs.org/@types/connect/-/connect-3.4.38.tgz",
@@ -6367,6 +6400,12 @@
"dev": true,
"license": "MIT"
},
+ "node_modules/@types/luxon": {
+ "version": "3.7.1",
+ "resolved": "https://registry.npmjs.org/@types/luxon/-/luxon-3.7.1.tgz",
+ "integrity": "sha512-H3iskjFIAn5SlJU7OuxUmTEpebK6TKB8rxZShDslBMZJ5u9S//KM1sbdAisiSrqwLQncVjnpi2OK2J51h+4lsg==",
+ "license": "MIT"
+ },
"node_modules/@types/methods": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/@types/methods/-/methods-1.1.4.tgz",
@@ -6374,6 +6413,16 @@
"dev": true,
"license": "MIT"
},
+ "node_modules/@types/multer": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/@types/multer/-/multer-2.0.0.tgz",
+ "integrity": "sha512-C3Z9v9Evij2yST3RSBktxP9STm6OdMc5uR1xF1SGr98uv8dUlAL2hqwrZ3GVB3uyMyiegnscEK6PGtYvNrjTjw==",
+ "dev": true,
+ "license": "MIT",
+ "dependencies": {
+ "@types/express": "*"
+ }
+ },
"node_modules/@types/node": {
"version": "24.10.4",
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.10.4.tgz",
@@ -6451,6 +6500,12 @@
"@types/superagent": "^8.1.0"
}
},
+ "node_modules/@types/validator": {
+ "version": "13.15.10",
+ "resolved": "https://registry.npmjs.org/@types/validator/-/validator-13.15.10.tgz",
+ "integrity": "sha512-T8L6i7wCuyoK8A/ZeLYt1+q0ty3Zb9+qbSSvrIVitzT3YjZqkTZ40IbRsPanlB4h1QB3JVL1SYCdR6ngtFYcuA==",
+ "license": "MIT"
+ },
"node_modules/@types/webidl-conversions": {
"version": "7.0.3",
"resolved": "https://registry.npmjs.org/@types/webidl-conversions/-/webidl-conversions-7.0.3.tgz",
@@ -7673,6 +7728,12 @@
"url": "https://opencollective.com/express"
}
},
+ "node_modules/boolbase": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/boolbase/-/boolbase-1.0.0.tgz",
+ "integrity": "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==",
+ "license": "ISC"
+ },
"node_modules/bowser": {
"version": "2.13.1",
"resolved": "https://registry.npmjs.org/bowser/-/bowser-2.13.1.tgz",
@@ -7958,6 +8019,48 @@
"dev": true,
"license": "MIT"
},
+ "node_modules/cheerio": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/cheerio/-/cheerio-1.1.2.tgz",
+ "integrity": "sha512-IkxPpb5rS/d1IiLbHMgfPuS0FgiWTtFIm/Nj+2woXDLTZ7fOT2eqzgYbdMlLweqlHbsZjxEChoVK+7iph7jyQg==",
+ "license": "MIT",
+ "dependencies": {
+ "cheerio-select": "^2.1.0",
+ "dom-serializer": "^2.0.0",
+ "domhandler": "^5.0.3",
+ "domutils": "^3.2.2",
+ "encoding-sniffer": "^0.2.1",
+ "htmlparser2": "^10.0.0",
+ "parse5": "^7.3.0",
+ "parse5-htmlparser2-tree-adapter": "^7.1.0",
+ "parse5-parser-stream": "^7.1.2",
+ "undici": "^7.12.0",
+ "whatwg-mimetype": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=20.18.1"
+ },
+ "funding": {
+ "url": "https://github.com/cheeriojs/cheerio?sponsor=1"
+ }
+ },
+ "node_modules/cheerio-select": {
+ "version": "2.1.0",
+ "resolved": "https://registry.npmjs.org/cheerio-select/-/cheerio-select-2.1.0.tgz",
+ "integrity": "sha512-9v9kG0LvzrlcungtnJtpGNxY+fzECQKhK4EGJX2vByejiMX84MFNQw4UxPJl3bFbTMw+Dfs37XaIkCwTZfLh4g==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "boolbase": "^1.0.0",
+ "css-select": "^5.1.0",
+ "css-what": "^6.1.0",
+ "domelementtype": "^2.3.0",
+ "domhandler": "^5.0.3",
+ "domutils": "^3.0.1"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/fb55"
+ }
+ },
"node_modules/chokidar": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz",
@@ -8008,6 +8111,25 @@
"dev": true,
"license": "MIT"
},
+ "node_modules/class-transformer": {
+ "version": "0.5.1",
+ "resolved": "https://registry.npmjs.org/class-transformer/-/class-transformer-0.5.1.tgz",
+ "integrity": "sha512-SQa1Ws6hUbfC98vKGxZH3KFY0Y1lm5Zm0SY8XX9zbK7FJCyVEac3ATW0RIpwzW+oOfmHE5PMPufDG9hCfoEOMw==",
+ "license": "MIT",
+ "peer": true
+ },
+ "node_modules/class-validator": {
+ "version": "0.14.3",
+ "resolved": "https://registry.npmjs.org/class-validator/-/class-validator-0.14.3.tgz",
+ "integrity": "sha512-rXXekcjofVN1LTOSw+u4u9WXVEUvNBVjORW154q/IdmYWy1nMbOU9aNtZB0t8m+FJQ9q91jlr2f9CwwUFdFMRA==",
+ "license": "MIT",
+ "peer": true,
+ "dependencies": {
+ "@types/validator": "^13.15.3",
+ "libphonenumber-js": "^1.11.1",
+ "validator": "^13.15.20"
+ }
+ },
"node_modules/cli-cursor": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/cli-cursor/-/cli-cursor-3.1.0.tgz",
@@ -8327,6 +8449,19 @@
"dev": true,
"license": "MIT"
},
+ "node_modules/cron": {
+ "version": "4.3.5",
+ "resolved": "https://registry.npmjs.org/cron/-/cron-4.3.5.tgz",
+ "integrity": "sha512-hKPP7fq1+OfyCqoePkKfVq7tNAdFwiQORr4lZUHwrf0tebC65fYEeWgOrXOL6prn1/fegGOdTfrM6e34PJfksg==",
+ "license": "MIT",
+ "dependencies": {
+ "@types/luxon": "~3.7.0",
+ "luxon": "~3.7.0"
+ },
+ "engines": {
+ "node": ">=18.x"
+ }
+ },
"node_modules/cross-spawn": {
"version": "7.0.6",
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
@@ -8342,6 +8477,34 @@
"node": ">= 8"
}
},
+ "node_modules/css-select": {
+ "version": "5.2.2",
+ "resolved": "https://registry.npmjs.org/css-select/-/css-select-5.2.2.tgz",
+ "integrity": "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "boolbase": "^1.0.0",
+ "css-what": "^6.1.0",
+ "domhandler": "^5.0.2",
+ "domutils": "^3.0.1",
+ "nth-check": "^2.0.1"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/fb55"
+ }
+ },
+ "node_modules/css-what": {
+ "version": "6.2.2",
+ "resolved": "https://registry.npmjs.org/css-what/-/css-what-6.2.2.tgz",
+ "integrity": "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA==",
+ "license": "BSD-2-Clause",
+ "engines": {
+ "node": ">= 6"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/fb55"
+ }
+ },
"node_modules/debug": {
"version": "4.4.3",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
@@ -8479,6 +8642,61 @@
"node": ">=0.3.1"
}
},
+ "node_modules/dom-serializer": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-2.0.0.tgz",
+ "integrity": "sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==",
+ "license": "MIT",
+ "dependencies": {
+ "domelementtype": "^2.3.0",
+ "domhandler": "^5.0.2",
+ "entities": "^4.2.0"
+ },
+ "funding": {
+ "url": "https://github.com/cheeriojs/dom-serializer?sponsor=1"
+ }
+ },
+ "node_modules/domelementtype": {
+ "version": "2.3.0",
+ "resolved": "https://registry.npmjs.org/domelementtype/-/domelementtype-2.3.0.tgz",
+ "integrity": "sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/fb55"
+ }
+ ],
+ "license": "BSD-2-Clause"
+ },
+ "node_modules/domhandler": {
+ "version": "5.0.3",
+ "resolved": "https://registry.npmjs.org/domhandler/-/domhandler-5.0.3.tgz",
+ "integrity": "sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "domelementtype": "^2.3.0"
+ },
+ "engines": {
+ "node": ">= 4"
+ },
+ "funding": {
+ "url": "https://github.com/fb55/domhandler?sponsor=1"
+ }
+ },
+ "node_modules/domutils": {
+ "version": "3.2.2",
+ "resolved": "https://registry.npmjs.org/domutils/-/domutils-3.2.2.tgz",
+ "integrity": "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "dom-serializer": "^2.0.0",
+ "domelementtype": "^2.3.0",
+ "domhandler": "^5.0.3"
+ },
+ "funding": {
+ "url": "https://github.com/fb55/domutils?sponsor=1"
+ }
+ },
"node_modules/dotenv": {
"version": "16.4.7",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.4.7.tgz",
@@ -8569,6 +8787,31 @@
"node": ">= 0.8"
}
},
+ "node_modules/encoding-sniffer": {
+ "version": "0.2.1",
+ "resolved": "https://registry.npmjs.org/encoding-sniffer/-/encoding-sniffer-0.2.1.tgz",
+ "integrity": "sha512-5gvq20T6vfpekVtqrYQsSCFZ1wEg5+wW0/QaZMWkFr6BqD3NfKs0rLCx4rrVlSWJeZb5NBJgVLswK/w2MWU+Gw==",
+ "license": "MIT",
+ "dependencies": {
+ "iconv-lite": "^0.6.3",
+ "whatwg-encoding": "^3.1.1"
+ },
+ "funding": {
+ "url": "https://github.com/fb55/encoding-sniffer?sponsor=1"
+ }
+ },
+ "node_modules/encoding-sniffer/node_modules/iconv-lite": {
+ "version": "0.6.3",
+ "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
+ "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==",
+ "license": "MIT",
+ "dependencies": {
+ "safer-buffer": ">= 2.1.2 < 3.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/enhanced-resolve": {
"version": "5.18.4",
"resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.18.4.tgz",
@@ -8583,6 +8826,18 @@
"node": ">=10.13.0"
}
},
+ "node_modules/entities": {
+ "version": "4.5.0",
+ "resolved": "https://registry.npmjs.org/entities/-/entities-4.5.0.tgz",
+ "integrity": "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==",
+ "license": "BSD-2-Clause",
+ "engines": {
+ "node": ">=0.12"
+ },
+ "funding": {
+ "url": "https://github.com/fb55/entities?sponsor=1"
+ }
+ },
"node_modules/error-ex": {
"version": "1.3.4",
"resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.4.tgz",
@@ -9713,6 +9968,37 @@
"dev": true,
"license": "MIT"
},
+ "node_modules/htmlparser2": {
+ "version": "10.0.0",
+ "resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-10.0.0.tgz",
+ "integrity": "sha512-TwAZM+zE5Tq3lrEHvOlvwgj1XLWQCtaaibSN11Q+gGBAS7Y1uZSWwXXRe4iF6OXnaq1riyQAPFOBtYc77Mxq0g==",
+ "funding": [
+ "https://github.com/fb55/htmlparser2?sponsor=1",
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/fb55"
+ }
+ ],
+ "license": "MIT",
+ "dependencies": {
+ "domelementtype": "^2.3.0",
+ "domhandler": "^5.0.3",
+ "domutils": "^3.2.1",
+ "entities": "^6.0.0"
+ }
+ },
+ "node_modules/htmlparser2/node_modules/entities": {
+ "version": "6.0.1",
+ "resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
+ "integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==",
+ "license": "BSD-2-Clause",
+ "engines": {
+ "node": ">=0.12"
+ },
+ "funding": {
+ "url": "https://github.com/fb55/entities?sponsor=1"
+ }
+ },
"node_modules/http-errors": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
@@ -10892,6 +11178,15 @@
"url": "https://github.com/chalk/supports-color?sponsor=1"
}
},
+ "node_modules/jose": {
+ "version": "6.1.3",
+ "resolved": "https://registry.npmjs.org/jose/-/jose-6.1.3.tgz",
+ "integrity": "sha512-0TpaTfihd4QMNwrz/ob2Bp7X04yuxJkjRGi4aKmOqwhov54i6u79oCv7T+C7lo70MKH6BesI3vscD1yb/yzKXQ==",
+ "license": "MIT",
+ "funding": {
+ "url": "https://github.com/sponsors/panva"
+ }
+ },
"node_modules/js-tokens": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
@@ -11028,6 +11323,12 @@
"node": ">= 0.8.0"
}
},
+ "node_modules/libphonenumber-js": {
+ "version": "1.12.33",
+ "resolved": "https://registry.npmjs.org/libphonenumber-js/-/libphonenumber-js-1.12.33.tgz",
+ "integrity": "sha512-r9kw4OA6oDO4dPXkOrXTkArQAafIKAU71hChInV4FxZ69dxCfbwQGDPzqR5/vea94wU705/3AZroEbSoeVWrQw==",
+ "license": "MIT"
+ },
"node_modules/lines-and-columns": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz",
@@ -11131,6 +11432,15 @@
"yallist": "^3.0.2"
}
},
+ "node_modules/luxon": {
+ "version": "3.7.2",
+ "resolved": "https://registry.npmjs.org/luxon/-/luxon-3.7.2.tgz",
+ "integrity": "sha512-vtEhXh/gNjI9Yg1u4jX/0YVPMvxzHuGgCm6tC5kZyb08yjGWGnqAjGJvcXbqQR2P3MyMEFnRbpcdFS6PBcLqew==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=12"
+ }
+ },
"node_modules/magic-string": {
"version": "0.30.17",
"resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.17.tgz",
@@ -11706,6 +12016,18 @@
"node": ">=8"
}
},
+ "node_modules/nth-check": {
+ "version": "2.1.1",
+ "resolved": "https://registry.npmjs.org/nth-check/-/nth-check-2.1.1.tgz",
+ "integrity": "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w==",
+ "license": "BSD-2-Clause",
+ "dependencies": {
+ "boolbase": "^1.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/fb55/nth-check?sponsor=1"
+ }
+ },
"node_modules/object-assign": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
@@ -11887,6 +12209,55 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
+ "node_modules/parse5": {
+ "version": "7.3.0",
+ "resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz",
+ "integrity": "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==",
+ "license": "MIT",
+ "dependencies": {
+ "entities": "^6.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/inikulin/parse5?sponsor=1"
+ }
+ },
+ "node_modules/parse5-htmlparser2-tree-adapter": {
+ "version": "7.1.0",
+ "resolved": "https://registry.npmjs.org/parse5-htmlparser2-tree-adapter/-/parse5-htmlparser2-tree-adapter-7.1.0.tgz",
+ "integrity": "sha512-ruw5xyKs6lrpo9x9rCZqZZnIUntICjQAd0Wsmp396Ul9lN/h+ifgVV1x1gZHi8euej6wTfpqX8j+BFQxF0NS/g==",
+ "license": "MIT",
+ "dependencies": {
+ "domhandler": "^5.0.3",
+ "parse5": "^7.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/inikulin/parse5?sponsor=1"
+ }
+ },
+ "node_modules/parse5-parser-stream": {
+ "version": "7.1.2",
+ "resolved": "https://registry.npmjs.org/parse5-parser-stream/-/parse5-parser-stream-7.1.2.tgz",
+ "integrity": "sha512-JyeQc9iwFLn5TbvvqACIF/VXG6abODeB3Fwmv/TGdLk2LfbWkaySGY72at4+Ty7EkPZj854u4CrICqNk2qIbow==",
+ "license": "MIT",
+ "dependencies": {
+ "parse5": "^7.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/inikulin/parse5?sponsor=1"
+ }
+ },
+ "node_modules/parse5/node_modules/entities": {
+ "version": "6.0.1",
+ "resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
+ "integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==",
+ "license": "BSD-2-Clause",
+ "engines": {
+ "node": ">=0.12"
+ },
+ "funding": {
+ "url": "https://github.com/fb55/entities?sponsor=1"
+ }
+ },
"node_modules/parseurl": {
"version": "1.3.3",
"resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
@@ -12072,6 +12443,50 @@
"node": ">=8"
}
},
+ "node_modules/playwright": {
+ "version": "1.57.0",
+ "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.57.0.tgz",
+ "integrity": "sha512-ilYQj1s8sr2ppEJ2YVadYBN0Mb3mdo9J0wQ+UuDhzYqURwSoW4n1Xs5vs7ORwgDGmyEh33tRMeS8KhdkMoLXQw==",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "playwright-core": "1.57.0"
+ },
+ "bin": {
+ "playwright": "cli.js"
+ },
+ "engines": {
+ "node": ">=18"
+ },
+ "optionalDependencies": {
+ "fsevents": "2.3.2"
+ }
+ },
+ "node_modules/playwright-core": {
+ "version": "1.57.0",
+ "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.57.0.tgz",
+ "integrity": "sha512-agTcKlMw/mjBWOnD6kFZttAAGHgi/Nw0CZ2o6JqWSbMlI219lAFLZZCyqByTsvVAJq5XA5H8cA6PrvBRpBWEuQ==",
+ "license": "Apache-2.0",
+ "bin": {
+ "playwright-core": "cli.js"
+ },
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/playwright/node_modules/fsevents": {
+ "version": "2.3.2",
+ "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
+ "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
+ "hasInstallScript": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^8.16.0 || ^10.6.0 || >=11.0.0"
+ }
+ },
"node_modules/pluralize": {
"version": "8.0.0",
"resolved": "https://registry.npmjs.org/pluralize/-/pluralize-8.0.0.tgz",
@@ -12408,6 +12823,7 @@
"resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.2.tgz",
"integrity": "sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==",
"license": "Apache-2.0",
+ "peer": true,
"dependencies": {
"tslib": "^2.1.0"
}
@@ -13702,6 +14118,15 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
+ "node_modules/undici": {
+ "version": "7.16.0",
+ "resolved": "https://registry.npmjs.org/undici/-/undici-7.16.0.tgz",
+ "integrity": "sha512-QEg3HPMll0o3t2ourKwOeUAZ159Kn9mx5pnzHRQO8+Wixmh88YdZRiIwat0iNzNNXn0yoEtXJqFpyW7eM8BV7g==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=20.18.1"
+ }
+ },
"node_modules/undici-types": {
"version": "7.16.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.16.0.tgz",
@@ -13858,6 +14283,15 @@
"node": ">=10.12.0"
}
},
+ "node_modules/validator": {
+ "version": "13.15.26",
+ "resolved": "https://registry.npmjs.org/validator/-/validator-13.15.26.tgz",
+ "integrity": "sha512-spH26xU080ydGggxRyR1Yhcbgx+j3y5jbNXk/8L+iRvdIEQ4uTRH2Sgf2dokud6Q4oAtsbNvJ1Ft+9xmm6IZcA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.10"
+ }
+ },
"node_modules/vary": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
@@ -14115,6 +14549,39 @@
"url": "https://opencollective.com/webpack"
}
},
+ "node_modules/whatwg-encoding": {
+ "version": "3.1.1",
+ "resolved": "https://registry.npmjs.org/whatwg-encoding/-/whatwg-encoding-3.1.1.tgz",
+ "integrity": "sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ==",
+ "license": "MIT",
+ "dependencies": {
+ "iconv-lite": "0.6.3"
+ },
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/whatwg-encoding/node_modules/iconv-lite": {
+ "version": "0.6.3",
+ "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
+ "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==",
+ "license": "MIT",
+ "dependencies": {
+ "safer-buffer": ">= 2.1.2 < 3.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/whatwg-mimetype": {
+ "version": "4.0.0",
+ "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-4.0.0.tgz",
+ "integrity": "sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ }
+ },
"node_modules/whatwg-url": {
"version": "14.2.0",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-14.2.0.tgz",
diff --git a/package.json b/package.json
index e6d60f1..b7620ac 100644
--- a/package.json
+++ b/package.json
@@ -29,10 +29,18 @@
"@nestjs/config": "^4.0.2",
"@nestjs/core": "^11.0.1",
"@nestjs/platform-express": "^11.0.1",
+ "@nestjs/schedule": "^6.1.0",
"@nestjs/swagger": "^11.2.3",
+ "axios": "^1.13.2",
+ "cheerio": "^1.1.2",
+ "class-transformer": "^0.5.1",
+ "class-validator": "^0.14.3",
+ "cron": "^4.3.5",
+ "jose": "^6.1.3",
"minio": "^8.0.6",
"mongoose": "^9.0.2",
"nats": "^2.29.3",
+ "playwright": "^1.57.0",
"reflect-metadata": "^0.2.2",
"rxjs": "^7.8.1",
"temporal-polyfill": "^0.3.0"
@@ -43,8 +51,10 @@
"@nestjs/cli": "^11.0.0",
"@nestjs/schematics": "^11.0.0",
"@nestjs/testing": "^11.0.1",
+ "@types/cheerio": "^0.22.35",
"@types/express": "^5.0.0",
"@types/jest": "^30.0.0",
+ "@types/multer": "^2.0.0",
"@types/node": "^24.10.4",
"@types/supertest": "^6.0.2",
"eslint": "^9.18.0",
diff --git a/src/app.module.ts b/src/app.module.ts
index 435870f..0f0f7bd 100644
--- a/src/app.module.ts
+++ b/src/app.module.ts
@@ -1,6 +1,9 @@
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
+import { ConfigModule } from '@nestjs/config';
+import { GlobalModule } from './global/global.module';
+import { CrawlerModule } from './crawler/crawler.module';
@Module({
imports: [
@@ -13,6 +16,8 @@ import { AppService } from './app.service';
? '.env.test'
: '.env.dev',
}),
+ GlobalModule,
+ CrawlerModule,
],
controllers: [AppController],
providers: [AppService],
diff --git a/src/crawler/crawler.controller.ts b/src/crawler/crawler.controller.ts
new file mode 100644
index 0000000..9c7f7a3
--- /dev/null
+++ b/src/crawler/crawler.controller.ts
@@ -0,0 +1,338 @@
+import {
+ Controller,
+ Get,
+ Post,
+ Put,
+ Delete,
+ Body,
+ Param,
+ Query,
+ HttpStatus,
+ HttpCode,
+ UseGuards,
+ NotFoundException,
+} from '@nestjs/common';
+import mongoose from 'mongoose';
+import {
+ ApiTags,
+ ApiOperation,
+ ApiResponse,
+ ApiBearerAuth,
+ ApiQuery,
+} from '@nestjs/swagger';
+import { JwtAuthGuard } from '../global/auth/jwt.guard';
+import { AllowPublicAccess } from '../global/auth/allow-public-access.decorator';
+import { PipelineEngineService } from './services/pipeline-engine.service';
+import { TriggerService, ScheduledJob } from './services/trigger.service';
+import { RateLimiterService } from './services/rate-limiter.service';
+import { ConnectionService } from '../global/database/connection.service';
+import {
+ CreatePipelinedefDto,
+ UpdatePipelinedefDto,
+ CreateStagedefDto,
+ TriggerPipelineDto,
+} from './dto';
+
+@ApiTags('Crawler')
+@Controller('api/v1/crawler')
+@UseGuards(JwtAuthGuard)
+@ApiBearerAuth()
+export class CrawlerController {
+ constructor(
+ private readonly pipelineEngine: PipelineEngineService,
+ private readonly triggerService: TriggerService,
+ private readonly rateLimiter: RateLimiterService,
+ private readonly connectionService: ConnectionService,
+ ) {}
+
+ // ============ Pipelinedef CRUD ============
+
+ @Post('pipelinedefs')
+ @ApiOperation({ summary: 'Create a new pipeline definition' })
+ @ApiResponse({ status: 201, description: 'Pipeline definition created' })
+ async createPipelinedef(@Body() dto: CreatePipelinedefDto) {
+ const model = await this.connectionService.getMoeumModels('pipelinedefs');
+
+ const metadatas = dto.metadatas || [];
+
+ // 스케줄 정보가 있으면 메타데이터에 추가
+ if (dto.schedule) {
+ metadatas.push({
+ key: 'schedule',
+ value: JSON.stringify(dto.schedule),
+ dataType: 'json',
+ isReserved: true,
+ });
+ }
+
+ const pipelinedef = await model.create({
+ apiVersion: dto.apiVersion || 'v1',
+ trigger: dto.trigger,
+ stage: dto.stageIds.map((id) => new mongoose.Types.ObjectId(id)),
+ metadatas,
+ });
+
+ // 트리거 등록
+ if (dto.schedule) {
+ await this.triggerService.registerScheduleTrigger(
+ pipelinedef._id.toString(),
+ dto.schedule,
+ );
+ }
+
+ if (dto.trigger.type === 'kafka-event' && dto.trigger.kafkaEvent) {
+ await this.triggerService.registerNatsTrigger(
+ pipelinedef._id.toString(),
+ dto.trigger.kafkaEvent.topic,
+ );
+ }
+
+ return {
+ id: pipelinedef._id,
+ message: 'Pipeline definition created',
+ };
+ }
+
+ @Get('pipelinedefs')
+ @ApiOperation({ summary: 'List all pipeline definitions' })
+ @ApiQuery({ name: 'page', required: false, type: Number })
+ @ApiQuery({ name: 'limit', required: false, type: Number })
+ async listPipelinedefs(
+ @Query('page') page: number = 1,
+ @Query('limit') limit: number = 20,
+ ) {
+ const { items, total } = await this.pipelineEngine.listPipelinedefs(
+ page,
+ limit,
+ );
+
+ return {
+ items,
+ pagination: {
+ page: Number(page),
+ limit: Number(limit),
+ total,
+ totalPages: Math.ceil(total / limit),
+ },
+ };
+ }
+
+ @Get('pipelinedefs/:id')
+ @ApiOperation({ summary: 'Get a pipeline definition by ID' })
+ async getPipelinedef(@Param('id') id: string) {
+ const model = await this.connectionService.getMoeumModels('pipelinedefs');
+ const pipelinedef = await model.findById(id).exec();
+
+ if (!pipelinedef) {
+ throw new NotFoundException('Pipeline definition not found');
+ }
+
+ return pipelinedef;
+ }
+
+ @Put('pipelinedefs/:id')
+ @ApiOperation({ summary: 'Update a pipeline definition' })
+ async updatePipelinedef(
+ @Param('id') id: string,
+ @Body() dto: UpdatePipelinedefDto,
+ ) {
+ const model = await this.connectionService.getMoeumModels('pipelinedefs');
+
+ const updateData: any = {};
+ if (dto.apiVersion) updateData.apiVersion = dto.apiVersion;
+ if (dto.trigger) updateData.trigger = dto.trigger;
+ if (dto.stageIds) updateData.stage = dto.stageIds;
+ if (dto.metadatas) updateData.metadatas = dto.metadatas;
+
+ const updated = await model
+ .findByIdAndUpdate(id, updateData, { new: true })
+ .exec();
+
+ if (!updated) {
+ throw new NotFoundException('Pipeline definition not found');
+ }
+
+ // 스케줄 업데이트
+ if (dto.schedule) {
+ await this.triggerService.registerScheduleTrigger(id, dto.schedule);
+ }
+
+ return updated;
+ }
+
+ @Delete('pipelinedefs/:id')
+ @HttpCode(HttpStatus.NO_CONTENT)
+ @ApiOperation({ summary: 'Delete a pipeline definition' })
+ async deletePipelinedef(@Param('id') id: string) {
+ const model = await this.connectionService.getMoeumModels('pipelinedefs');
+ await model.findByIdAndDelete(id).exec();
+ await this.triggerService.unregisterScheduleTrigger(id);
+ }
+
+ // ============ Stagedef CRUD ============
+
+ @Post('stagedefs')
+ @ApiOperation({ summary: 'Create a new stage definition' })
+ async createStagedef(@Body() dto: CreateStagedefDto) {
+ const model = await this.connectionService.getMoeumModels('stagedefs');
+
+ const stagedef = await model.create({
+ apiVersion: dto.apiVersion || 'v1',
+ name: dto.name,
+ taxonomy: dto.taxonomy,
+ eventSource: dto.eventSource,
+ schema: dto.schema,
+ topic: dto.topic,
+ metadatas: dto.metadatas || [],
+ });
+
+ return {
+ id: stagedef._id,
+ message: 'Stage definition created',
+ };
+ }
+
+ @Get('stagedefs')
+ @ApiOperation({ summary: 'List all stage definitions' })
+ @ApiQuery({ name: 'page', required: false, type: Number })
+ @ApiQuery({ name: 'limit', required: false, type: Number })
+ async listStagedefs(
+ @Query('page') page: number = 1,
+ @Query('limit') limit: number = 20,
+ ) {
+ const model = await this.connectionService.getMoeumModels('stagedefs');
+ const skip = (Number(page) - 1) * Number(limit);
+
+ const [items, total] = await Promise.all([
+ model.find().skip(skip).limit(Number(limit)).exec(),
+ model.countDocuments().exec(),
+ ]);
+
+ return {
+ items,
+ pagination: {
+ page: Number(page),
+ limit: Number(limit),
+ total,
+ totalPages: Math.ceil(total / Number(limit)),
+ },
+ };
+ }
+
+ @Get('stagedefs/:id')
+ @ApiOperation({ summary: 'Get a stage definition by ID' })
+ async getStagedef(@Param('id') id: string) {
+ const model = await this.connectionService.getMoeumModels('stagedefs');
+ const stagedef = await model.findById(id).exec();
+
+ if (!stagedef) {
+ throw new NotFoundException('Stage definition not found');
+ }
+
+ return stagedef;
+ }
+
+ @Delete('stagedefs/:id')
+ @HttpCode(HttpStatus.NO_CONTENT)
+ @ApiOperation({ summary: 'Delete a stage definition' })
+ async deleteStagedef(@Param('id') id: string) {
+ const model = await this.connectionService.getMoeumModels('stagedefs');
+ await model.findByIdAndDelete(id).exec();
+ }
+
+ // ============ Pipeline 실행 ============
+
+ @Post('pipelines/trigger')
+ @ApiOperation({ summary: 'Trigger a pipeline execution' })
+ async triggerPipeline(@Body() dto: TriggerPipelineDto) {
+ const pipelineId = await this.triggerService.triggerPipelineManually(
+ dto.pipelineDefId,
+ );
+ return {
+ pipelineId,
+ message: 'Pipeline triggered successfully',
+ };
+ }
+
+ @Get('pipelines')
+ @ApiOperation({ summary: 'List executed pipelines' })
+ @ApiQuery({ name: 'page', required: false, type: Number })
+ @ApiQuery({ name: 'limit', required: false, type: Number })
+ @ApiQuery({ name: 'status', required: false, type: String })
+ async listPipelines(
+ @Query('page') page: number = 1,
+ @Query('limit') limit: number = 20,
+ @Query('status') status?: string,
+ ) {
+ const { items, total } = await this.pipelineEngine.listPipelines(
+ Number(page),
+ Number(limit),
+ status,
+ );
+
+ return {
+ items,
+ pagination: {
+ page: Number(page),
+ limit: Number(limit),
+ total,
+ totalPages: Math.ceil(total / Number(limit)),
+ },
+ };
+ }
+
+ @Get('pipelines/:id')
+ @ApiOperation({ summary: 'Get pipeline execution details' })
+ async getPipeline(@Param('id') id: string) {
+ const pipeline = await this.pipelineEngine.getPipelineStatus(id);
+
+ if (!pipeline) {
+ throw new NotFoundException('Pipeline not found');
+ }
+
+ return pipeline;
+ }
+
+ @Get('pipelines/:id/stages')
+ @ApiOperation({ summary: 'Get stages of a pipeline' })
+ async getPipelineStages(@Param('id') id: string) {
+ const pipeline = await this.pipelineEngine.getPipelineStatus(id);
+
+ if (!pipeline) {
+ throw new NotFoundException('Pipeline not found');
+ }
+
+ return pipeline.stage;
+ }
+
+ // ============ 모니터링 ============
+
+ @Get('schedules')
+ @ApiOperation({ summary: 'List registered schedules' })
+ async listSchedules(): Promise {
+ return this.triggerService.getRegisteredSchedules();
+ }
+
+ @Post('triggers/reload')
+ @ApiOperation({ summary: 'Reload all triggers' })
+ async reloadTriggers() {
+ await this.triggerService.reloadTriggers();
+ return { message: 'Triggers reloaded successfully' };
+ }
+
+ @Get('rate-limits')
+ @ApiOperation({ summary: 'Get rate limit status for all domains' })
+ async getRateLimits() {
+ return this.rateLimiter.getAllLimits();
+ }
+
+ @Get('health')
+ @AllowPublicAccess()
+ @ApiOperation({ summary: 'Health check' })
+ async healthCheck() {
+ return {
+ status: 'ok',
+ timestamp: new Date().toISOString(),
+ };
+ }
+}
diff --git a/src/crawler/crawler.module.ts b/src/crawler/crawler.module.ts
new file mode 100644
index 0000000..24a412c
--- /dev/null
+++ b/src/crawler/crawler.module.ts
@@ -0,0 +1,29 @@
+import { Module } from '@nestjs/common';
+import { ScheduleModule } from '@nestjs/schedule';
+import { CrawlerController } from './crawler.controller';
+import { CrawlerService } from './services/crawler.service';
+import { RateLimiterService } from './services/rate-limiter.service';
+import { RetryService } from './services/retry.service';
+import { StageExecutorService } from './services/stage-executor.service';
+import { PipelineEngineService } from './services/pipeline-engine.service';
+import { TriggerService } from './services/trigger.service';
+
+@Module({
+ imports: [ScheduleModule.forRoot()],
+ controllers: [CrawlerController],
+ providers: [
+ RateLimiterService,
+ CrawlerService,
+ RetryService,
+ StageExecutorService,
+ PipelineEngineService,
+ TriggerService,
+ ],
+ exports: [
+ CrawlerService,
+ PipelineEngineService,
+ TriggerService,
+ RateLimiterService,
+ ],
+})
+export class CrawlerModule {}
diff --git a/src/crawler/crawler.service.ts b/src/crawler/crawler.service.ts
new file mode 100644
index 0000000..e69de29
diff --git a/src/crawler/dto/common.dto.ts b/src/crawler/dto/common.dto.ts
new file mode 100644
index 0000000..656a5f4
--- /dev/null
+++ b/src/crawler/dto/common.dto.ts
@@ -0,0 +1,103 @@
+import {
+ IsString,
+ IsOptional,
+ IsEnum,
+ IsBoolean,
+ ValidateNested,
+ IsArray,
+} from 'class-validator';
+import { Type } from 'class-transformer';
+import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
+
+export class MetadataDto {
+ @ApiProperty({ description: '메타데이터 키' })
+ @IsString()
+ key: string;
+
+ @ApiProperty({ description: '메타데이터 값' })
+ value: string | any[];
+
+ @ApiPropertyOptional({
+ enum: ['string', 'number', 'boolean', 'array', 'json'],
+ default: 'string',
+ })
+ @IsOptional()
+ @IsEnum(['string', 'number', 'boolean', 'array', 'json'])
+ dataType?: 'string' | 'number' | 'boolean' | 'array' | 'json';
+
+ @ApiPropertyOptional({ default: false })
+ @IsOptional()
+ @IsBoolean()
+ isReserved?: boolean;
+}
+
+export class KafkaEventDto {
+ @ApiProperty({ description: 'Kafka/NATS 토픽명' })
+ @IsString()
+ topic: string;
+
+ @ApiPropertyOptional({ description: '메시지 키' })
+ @IsOptional()
+ @IsString()
+ key?: string;
+}
+
+export class PipelineTriggerDto {
+ @ApiProperty({ enum: ['always', 'kafka-event'] })
+ @IsEnum(['always', 'kafka-event'])
+ type: 'always' | 'kafka-event';
+
+ @ApiPropertyOptional({ type: KafkaEventDto })
+ @IsOptional()
+ @ValidateNested()
+ @Type(() => KafkaEventDto)
+ kafkaEvent?: KafkaEventDto;
+}
+
+export class ScheduleDto {
+ @ApiProperty({ description: 'Cron 표현식', example: '0 0 * * *' })
+ @IsString()
+ cron: string;
+
+ @ApiPropertyOptional({ description: '타임존', default: 'Asia/Seoul' })
+ @IsOptional()
+ @IsString()
+ timezone?: string;
+}
+
+export class EventSourceDto {
+ @ApiProperty({ description: '이벤트 소스 타입', example: 'web' })
+ @IsString()
+ type: string;
+
+ @ApiProperty({ description: '이벤트 소스 이름' })
+ @IsString()
+ name: string;
+
+ @ApiPropertyOptional({ type: [MetadataDto], description: '이벤트 소스 메타데이터' })
+ @IsOptional()
+ @IsArray()
+ @ValidateNested({ each: true })
+ @Type(() => MetadataDto)
+ metadatas?: MetadataDto[];
+}
+
+export class PaginationQueryDto {
+ @ApiPropertyOptional({ default: 1, description: '페이지 번호' })
+ @IsOptional()
+ page?: number = 1;
+
+ @ApiPropertyOptional({ default: 20, description: '페이지당 항목 수' })
+ @IsOptional()
+ limit?: number = 20;
+}
+
+export class PaginationResultDto {
+ items: T[];
+ pagination: {
+ page: number;
+ limit: number;
+ total: number;
+ totalPages: number;
+ };
+}
diff --git a/src/crawler/dto/create-pipelinedef.dto.ts b/src/crawler/dto/create-pipelinedef.dto.ts
new file mode 100644
index 0000000..5879c91
--- /dev/null
+++ b/src/crawler/dto/create-pipelinedef.dto.ts
@@ -0,0 +1,39 @@
+import {
+ IsString,
+ IsOptional,
+ IsArray,
+ ValidateNested,
+} from 'class-validator';
+import { Type } from 'class-transformer';
+import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
+import { MetadataDto, PipelineTriggerDto, ScheduleDto } from './common.dto';
+
+export class CreatePipelinedefDto {
+ @ApiPropertyOptional({ default: 'v1' })
+ @IsOptional()
+ @IsString()
+ apiVersion?: string;
+
+ @ApiProperty({ type: PipelineTriggerDto })
+ @ValidateNested()
+ @Type(() => PipelineTriggerDto)
+ trigger: PipelineTriggerDto;
+
+ @ApiProperty({ type: [String], description: 'Stagedef ID 목록' })
+ @IsArray()
+ @IsString({ each: true })
+ stageIds: string[];
+
+ @ApiPropertyOptional({ type: ScheduleDto, description: '스케줄 설정 (선택)' })
+ @IsOptional()
+ @ValidateNested()
+ @Type(() => ScheduleDto)
+ schedule?: ScheduleDto;
+
+ @ApiPropertyOptional({ type: [MetadataDto] })
+ @IsOptional()
+ @IsArray()
+ @ValidateNested({ each: true })
+ @Type(() => MetadataDto)
+ metadatas?: MetadataDto[];
+}
diff --git a/src/crawler/dto/create-stagedef.dto.ts b/src/crawler/dto/create-stagedef.dto.ts
new file mode 100644
index 0000000..59ca603
--- /dev/null
+++ b/src/crawler/dto/create-stagedef.dto.ts
@@ -0,0 +1,48 @@
+import {
+ IsString,
+ IsOptional,
+ IsArray,
+ ValidateNested,
+ IsObject,
+} from 'class-validator';
+import { Type } from 'class-transformer';
+import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
+import { MetadataDto, EventSourceDto } from './common.dto';
+
+export class CreateStagedefDto {
+ @ApiPropertyOptional({ default: 'v1' })
+ @IsOptional()
+ @IsString()
+ apiVersion?: string;
+
+ @ApiProperty({ description: 'Stage 이름' })
+ @IsString()
+ name: string;
+
+ @ApiPropertyOptional({ description: '분류 체계' })
+ @IsOptional()
+ @IsString()
+ taxonomy?: string;
+
+ @ApiProperty({ type: [EventSourceDto], description: '이벤트 소스 목록' })
+ @IsArray()
+ @ValidateNested({ each: true })
+ @Type(() => EventSourceDto)
+ eventSource: EventSourceDto[];
+
+ @ApiPropertyOptional({ description: '데이터 스키마 (JSON Schema)' })
+ @IsOptional()
+ @IsObject()
+ schema?: any;
+
+ @ApiProperty({ description: '결과 발행할 NATS 토픽' })
+ @IsString()
+ topic: string;
+
+ @ApiPropertyOptional({ type: [MetadataDto] })
+ @IsOptional()
+ @IsArray()
+ @ValidateNested({ each: true })
+ @Type(() => MetadataDto)
+ metadatas?: MetadataDto[];
+}
diff --git a/src/crawler/dto/index.ts b/src/crawler/dto/index.ts
new file mode 100644
index 0000000..38ab08c
--- /dev/null
+++ b/src/crawler/dto/index.ts
@@ -0,0 +1,5 @@
+export * from './create-pipelinedef.dto';
+export * from './update-pipelinedef.dto';
+export * from './create-stagedef.dto';
+export * from './trigger-pipeline.dto';
+export * from './common.dto';
diff --git a/src/crawler/dto/trigger-pipeline.dto.ts b/src/crawler/dto/trigger-pipeline.dto.ts
new file mode 100644
index 0000000..9a7714a
--- /dev/null
+++ b/src/crawler/dto/trigger-pipeline.dto.ts
@@ -0,0 +1,21 @@
+import { IsString, IsOptional, IsObject } from 'class-validator';
+import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
+
+export class TriggerPipelineDto {
+ @ApiProperty({ description: 'Pipelinedef ID' })
+ @IsString()
+ pipelineDefId: string;
+
+ @ApiPropertyOptional({ description: '추가 변수 (선택)' })
+ @IsOptional()
+ @IsObject()
+ variables?: Record;
+}
+
+export class TriggerPipelineResponseDto {
+ @ApiProperty({ description: '생성된 Pipeline ID' })
+ pipelineId: string;
+
+ @ApiProperty()
+ message: string;
+}
diff --git a/src/crawler/dto/update-pipelinedef.dto.ts b/src/crawler/dto/update-pipelinedef.dto.ts
new file mode 100644
index 0000000..3522a37
--- /dev/null
+++ b/src/crawler/dto/update-pipelinedef.dto.ts
@@ -0,0 +1,4 @@
+import { PartialType } from '@nestjs/swagger';
+import { CreatePipelinedefDto } from './create-pipelinedef.dto';
+
+export class UpdatePipelinedefDto extends PartialType(CreatePipelinedefDto) {}
diff --git a/src/crawler/interfaces/crawler-result.interface.ts b/src/crawler/interfaces/crawler-result.interface.ts
new file mode 100644
index 0000000..0284c75
--- /dev/null
+++ b/src/crawler/interfaces/crawler-result.interface.ts
@@ -0,0 +1,35 @@
+export interface CrawlResult {
+ success: boolean;
+ url: string;
+ statusCode?: number;
+ html?: string;
+ parsedData?: Record;
+ error?: string;
+ timestamp: Date;
+}
+
+export interface ParserConfig {
+ selectors: Record; // { title: 'h1.title', content: '.article-body' }
+ extractAttributes?: Record; // { imageUrl: 'img.main@src' }
+}
+
+export interface RateLimitConfig {
+ maxTokens: number; // 최대 토큰 수
+ refillRate: number; // 초당 토큰 보충 속도
+}
+
+export interface ExecutionContext {
+ pipelineId: string;
+ pipelineDefId: string;
+ triggeredBy: 'api' | 'schedule' | 'nats';
+ triggeredAt: Date;
+ variables: Map;
+}
+
+export interface StageExecutionResult {
+ stageIndex: number;
+ stageName: string;
+ status: 'completed' | 'failed';
+ crawlResults?: CrawlResult[];
+ error?: string;
+}
diff --git a/src/crawler/services/crawler.service.ts b/src/crawler/services/crawler.service.ts
new file mode 100644
index 0000000..3bb868e
--- /dev/null
+++ b/src/crawler/services/crawler.service.ts
@@ -0,0 +1,291 @@
+import { Injectable, Logger, OnModuleDestroy } from '@nestjs/common';
+import axios, { AxiosInstance, AxiosRequestConfig } from 'axios';
+import * as cheerio from 'cheerio';
+import { chromium, Browser, BrowserContext } from 'playwright';
+import { MinioService } from '../../global/minio/minio.service';
+import { RESTRequest, EventSource } from '../../global/database/schema';
+import { CrawlResult, ParserConfig } from '../interfaces/crawler-result.interface';
+import { RateLimiterService } from './rate-limiter.service';
+
+@Injectable()
+export class CrawlerService implements OnModuleDestroy {
+ private readonly logger = new Logger(CrawlerService.name);
+ private readonly httpClient: AxiosInstance;
+ private browser: Browser | null = null;
+
+ constructor(
+ private readonly minioService: MinioService,
+ private readonly rateLimiter: RateLimiterService,
+ ) {
+ this.httpClient = axios.create({
+ timeout: 30000,
+ headers: {
+ 'User-Agent':
+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
+ },
+ });
+ }
+
+ async onModuleDestroy() {
+ if (this.browser) {
+ await this.browser.close();
+ this.logger.log('Playwright browser closed');
+ }
+ }
+
+ /**
+ * Playwright 브라우저 인스턴스 초기화 (Lazy)
+ */
+ private async getBrowser(): Promise {
+ if (!this.browser) {
+ this.browser = await chromium.launch({
+ headless: true,
+ args: ['--no-sandbox', '--disable-setuid-sandbox'],
+ });
+ this.logger.log('Playwright browser launched');
+ }
+ return this.browser;
+ }
+
+ /**
+ * RESTRequest 스키마 기반 HTTP 요청 실행 (정적 페이지용)
+ */
+ async executeRequest(request: RESTRequest): Promise {
+ const config = this.buildAxiosConfig(request);
+ const domain = new URL(request.url).hostname;
+
+ try {
+ // Rate limiting 적용
+ await this.rateLimiter.acquire(domain);
+
+ const response = await this.httpClient.request(config);
+ return {
+ success: true,
+ url: request.url,
+ statusCode: response.status,
+ html:
+ typeof response.data === 'string'
+ ? response.data
+ : JSON.stringify(response.data),
+ timestamp: new Date(),
+ };
+ } catch (error) {
+ this.logger.error(`HTTP request failed: ${request.url}`, error);
+ return {
+ success: false,
+ url: request.url,
+ error: error instanceof Error ? error.message : 'Unknown error',
+ timestamp: new Date(),
+ };
+ }
+ }
+
+ /**
+ * Playwright로 동적 페이지 크롤링 (SPA용)
+ */
+ async crawlDynamicPage(
+ url: string,
+ options?: {
+ waitSelector?: string;
+ waitTimeout?: number;
+ javascript?: boolean;
+ },
+ ): Promise {
+ const domain = new URL(url).hostname;
+
+ try {
+ // Rate limiting 적용
+ await this.rateLimiter.acquire(domain);
+
+ const browser = await this.getBrowser();
+ const context: BrowserContext = await browser.newContext({
+ userAgent:
+ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
+ });
+ const page = await context.newPage();
+
+ try {
+ await page.goto(url, {
+ waitUntil: 'networkidle',
+ timeout: options?.waitTimeout ?? 30000,
+ });
+
+ // 특정 셀렉터가 나타날 때까지 대기
+ if (options?.waitSelector) {
+ await page.waitForSelector(options.waitSelector, {
+ timeout: options?.waitTimeout ?? 10000,
+ });
+ }
+
+ const html = await page.content();
+
+ return {
+ success: true,
+ url,
+ statusCode: 200,
+ html,
+ timestamp: new Date(),
+ };
+ } finally {
+ await context.close();
+ }
+ } catch (error) {
+ this.logger.error(`Dynamic page crawl failed: ${url}`, error);
+ return {
+ success: false,
+ url,
+ error: error instanceof Error ? error.message : 'Unknown error',
+ timestamp: new Date(),
+ };
+ }
+ }
+
+ /**
+ * HTML 파싱 및 데이터 추출
+ */
+ parseHtml(html: string, config: ParserConfig): Record {
+ const $ = cheerio.load(html);
+ const result: Record = {};
+
+ // 셀렉터 기반 텍스트 추출
+ for (const [key, selector] of Object.entries(config.selectors)) {
+ const elements = $(selector);
+ result[key] =
+ elements.length > 1
+ ? elements
+ .map((_, el) => $(el).text().trim())
+ .get()
+ : elements.first().text().trim();
+ }
+
+ // 속성 추출 (예: img@src)
+ if (config.extractAttributes) {
+ for (const [key, selectorAttr] of Object.entries(
+ config.extractAttributes,
+ )) {
+ const [selector, attr] = selectorAttr.split('@');
+ const elements = $(selector);
+ result[key] =
+ elements.length > 1
+ ? elements.map((_, el) => $(el).attr(attr)).get()
+ : elements.first().attr(attr);
+ }
+ }
+
+ return result;
+ }
+
+ /**
+ * EventSource 기반 크롤링 실행
+ */
+ async crawlFromEventSource(eventSource: EventSource): Promise {
+ const urlMeta = eventSource.metadatas.find((m) => m.key === 'url');
+ const selectorsMeta = eventSource.metadatas.find(
+ (m) => m.key === 'selectors',
+ );
+ const dynamicMeta = eventSource.metadatas.find((m) => m.key === 'dynamic');
+ const waitSelectorMeta = eventSource.metadatas.find(
+ (m) => m.key === 'waitSelector',
+ );
+
+ if (!urlMeta) {
+ return {
+ success: false,
+ url: '',
+ error: 'URL not found in EventSource metadatas',
+ timestamp: new Date(),
+ };
+ }
+
+ const url = urlMeta.value as string;
+ const isDynamic = dynamicMeta?.value === 'true' || String(dynamicMeta?.value) === 'true';
+
+ let crawlResult: CrawlResult;
+
+ if (isDynamic) {
+ // 동적 페이지 크롤링 (Playwright)
+ crawlResult = await this.crawlDynamicPage(url, {
+ waitSelector: waitSelectorMeta?.value as string | undefined,
+ });
+ } else {
+ // 정적 페이지 크롤링 (axios)
+ const request: RESTRequest = {
+ url,
+ method: 'GET',
+ };
+ crawlResult = await this.executeRequest(request);
+ }
+
+ // HTML 파싱
+ if (crawlResult.success && crawlResult.html && selectorsMeta) {
+ try {
+ const parserConfig: ParserConfig =
+ typeof selectorsMeta.value === 'string'
+ ? JSON.parse(selectorsMeta.value)
+ : selectorsMeta.value;
+ crawlResult.parsedData = this.parseHtml(crawlResult.html, parserConfig);
+ } catch (error) {
+ this.logger.warn(`Failed to parse selectors config: ${error}`);
+ }
+ }
+
+ return crawlResult;
+ }
+
+ /**
+ * 크롤링 결과를 MinIO에 저장
+ */
+ async saveResultToStorage(
+ result: CrawlResult,
+ stageId: string,
+ ): Promise {
+ const buffer = Buffer.from(JSON.stringify(result, null, 2));
+ const file = {
+ buffer,
+ originalname: `crawl-result-${stageId}-${Date.now()}.json`,
+ mimetype: 'application/json',
+ size: buffer.length,
+ } as Express.Multer.File;
+
+ return this.minioService.uploadFile(file);
+ }
+
+ /**
+ * axios 설정 빌드
+ */
+ private buildAxiosConfig(request: RESTRequest): AxiosRequestConfig {
+ const config: AxiosRequestConfig = {
+ url: request.url,
+ method: request.method.toLowerCase() as
+ | 'get'
+ | 'post'
+ | 'put'
+ | 'delete'
+ | 'patch',
+ headers: request.headers || {},
+ params: request.queryParams,
+ data: request.body,
+ timeout: request.timeout || 30000,
+ };
+
+ // 인증 처리
+ if (request.auth) {
+ switch (request.auth.type) {
+ case 'BearerToken':
+ config.headers!['Authorization'] = `Bearer ${request.auth.token}`;
+ break;
+ case 'BasicAuth':
+ config.auth = {
+ username: request.auth.username!,
+ password: request.auth.password!,
+ };
+ break;
+ case 'APIKey':
+ config.headers!['X-API-Key'] = request.auth.token;
+ break;
+ }
+ }
+
+ return config;
+ }
+}
diff --git a/src/crawler/services/pipeline-engine.service.ts b/src/crawler/services/pipeline-engine.service.ts
new file mode 100644
index 0000000..80a7e7e
--- /dev/null
+++ b/src/crawler/services/pipeline-engine.service.ts
@@ -0,0 +1,289 @@
+import { Injectable, Logger } from '@nestjs/common';
+import { ConnectionService } from '../../global/database/connection.service';
+import { NatsService } from '../../global/nats/nats.service';
+import { StageExecutorService } from './stage-executor.service';
+import {
+ Pipelinedef,
+ Pipeline,
+ Stagedef,
+ Stage,
+ PipelineDocument,
+} from '../../global/database/schema';
+
+export type TriggerType = 'api' | 'schedule' | 'nats';
+
+export interface PipelineExecutionSummary {
+ pipelineId: string;
+ pipelineDefId: string;
+ status: 'completed' | 'failed' | 'partial';
+ triggeredBy: TriggerType;
+ stageResults: {
+ stageName: string;
+ status: string;
+ error?: string;
+ }[];
+ startedAt: Date;
+ completedAt?: Date;
+}
+
+@Injectable()
+export class PipelineEngineService {
+ private readonly logger = new Logger(PipelineEngineService.name);
+
+ constructor(
+ private readonly connectionService: ConnectionService,
+ private readonly natsService: NatsService,
+ private readonly stageExecutor: StageExecutorService,
+ ) {}
+
+ /**
+ * Pipelinedef -> Pipeline 인스턴스 생성 및 실행
+ */
+ async createAndExecutePipeline(
+ pipelineDefId: string,
+ triggeredBy: TriggerType,
+ ): Promise {
+ const startedAt = new Date();
+
+ // 1. Pipelinedef 조회
+ const pipelinedefModel =
+ await this.connectionService.getMoeumModels('pipelinedefs');
+ const pipelinedef = await pipelinedefModel.findById(pipelineDefId).exec();
+
+ if (!pipelinedef) {
+ throw new Error(`Pipelinedef not found: ${pipelineDefId}`);
+ }
+
+ // 2. Stagedef들 조회
+ const stagedefModel =
+ await this.connectionService.getMoeumModels('stagedefs');
+ const stagedefs = await stagedefModel
+ .find({ _id: { $in: pipelinedef.stage } })
+ .exec();
+
+ if (stagedefs.length === 0) {
+ throw new Error(`No stagedefs found for pipelinedef: ${pipelineDefId}`);
+ }
+
+ // 3. Stage 인스턴스들 생성
+ const stages: Stage[] = stagedefs.map((stagedef) => ({
+ apiVersion: stagedef.apiVersion,
+ name: stagedef.name,
+ schema: stagedef.schema,
+ taxonomy: stagedef.taxonomy,
+ executions: [],
+ status: 'pending' as const,
+ metadatas: [...stagedef.metadatas],
+ retryStrategy: {
+ limit: 3,
+ delay: 1000,
+ backoff: 2,
+ when: ['failed'],
+ notify: [],
+ },
+ }));
+
+ // 4. Pipeline 인스턴스 생성
+ const pipelineModel =
+ await this.connectionService.getMoeumModels('pipelines');
+ const pipeline = await pipelineModel.create({
+ apiVersion: pipelinedef.apiVersion,
+ name: `pipeline-${Date.now()}`,
+ stage: stages,
+ metadatas: [
+ ...pipelinedef.metadatas,
+ {
+ key: 'triggeredBy',
+ value: triggeredBy,
+ dataType: 'string',
+ isReserved: true,
+ },
+ {
+ key: 'triggeredAt',
+ value: startedAt.toISOString(),
+ dataType: 'string',
+ isReserved: true,
+ },
+ {
+ key: 'pipelineDefId',
+ value: pipelineDefId,
+ dataType: 'string',
+ isReserved: true,
+ },
+ ],
+ });
+
+ this.logger.log(`Pipeline created: ${pipeline._id}`);
+
+ // 5. 시작 이벤트 발행
+ await this.natsService.publish('crawler.pipeline.started', {
+ pipelineId: pipeline._id.toString(),
+ pipelineDefId,
+ triggeredBy,
+ timestamp: startedAt.toISOString(),
+ });
+
+ // 6. Stage들 병렬 실행 (비동기)
+ this.executeStagesInParallel(
+ pipeline._id.toString(),
+ stagedefs as Stagedef[],
+ triggeredBy,
+ );
+
+ return pipeline;
+ }
+
+ /**
+ * Stage들을 병렬로 실행
+ */
+ private async executeStagesInParallel(
+ pipelineId: string,
+ stagedefs: Stagedef[],
+ triggeredBy: TriggerType,
+ ): Promise {
+ const pipelineModel =
+ await this.connectionService.getMoeumModels('pipelines');
+
+ try {
+ // Promise.allSettled로 모든 Stage 병렬 실행
+ const results = await Promise.allSettled(
+ stagedefs.map((stagedef, index) =>
+ this.stageExecutor.executeStage(pipelineId, index, stagedef),
+ ),
+ );
+
+ // 결과 집계
+ const stageResults = results.map((r, i) => ({
+ stageName: stagedefs[i].name,
+ status: r.status === 'fulfilled' ? r.value.status : 'failed',
+ error:
+ r.status === 'rejected'
+ ? r.reason?.message || 'Unknown error'
+ : undefined,
+ }));
+
+ const allSucceeded = results.every(
+ (r) => r.status === 'fulfilled' && r.value.status === 'completed',
+ );
+ const allFailed = results.every(
+ (r) =>
+ r.status === 'rejected' ||
+ (r.status === 'fulfilled' && r.value.status === 'failed'),
+ );
+
+ let pipelineStatus: 'completed' | 'failed' | 'partial';
+ if (allSucceeded) {
+ pipelineStatus = 'completed';
+ } else if (allFailed) {
+ pipelineStatus = 'failed';
+ } else {
+ pipelineStatus = 'partial';
+ }
+
+ const completedAt = new Date();
+
+ // Pipeline 상태 업데이트
+ await pipelineModel.findByIdAndUpdate(pipelineId, {
+ $push: {
+ metadatas: {
+ $each: [
+ {
+ key: 'completedAt',
+ value: completedAt.toISOString(),
+ dataType: 'string',
+ isReserved: true,
+ },
+ {
+ key: 'pipelineStatus',
+ value: pipelineStatus,
+ dataType: 'string',
+ isReserved: true,
+ },
+ ],
+ },
+ },
+ });
+
+ // 완료 이벤트 발행
+ await this.natsService.publish('crawler.pipeline.completed', {
+ pipelineId,
+ status: pipelineStatus,
+ triggeredBy,
+ stageResults,
+ completedAt: completedAt.toISOString(),
+ });
+
+ this.logger.log(
+ `Pipeline ${pipelineId} completed with status: ${pipelineStatus}`,
+ );
+ } catch (error) {
+ this.logger.error(`Pipeline execution failed: ${pipelineId}`, error);
+
+ await this.natsService.publish('crawler.pipeline.failed', {
+ pipelineId,
+ error: error instanceof Error ? error.message : 'Unknown error',
+ timestamp: new Date().toISOString(),
+ });
+ }
+ }
+
+ /**
+ * Pipeline 실행 상태 조회
+ */
+ async getPipelineStatus(pipelineId: string): Promise {
+ const pipelineModel =
+ await this.connectionService.getMoeumModels('pipelines');
+ return pipelineModel.findById(pipelineId).exec();
+ }
+
+ /**
+ * Pipelinedef 목록 조회
+ */
+ async listPipelinedefs(
+ page: number = 1,
+ limit: number = 20,
+ ): Promise<{ items: Pipelinedef[]; total: number }> {
+ const pipelinedefModel =
+ await this.connectionService.getMoeumModels('pipelinedefs');
+ const skip = (page - 1) * limit;
+
+ const [items, total] = await Promise.all([
+ pipelinedefModel.find().skip(skip).limit(limit).exec(),
+ pipelinedefModel.countDocuments().exec(),
+ ]);
+
+ return { items: items as Pipelinedef[], total };
+ }
+
+ /**
+ * 실행된 Pipeline 목록 조회
+ */
+ async listPipelines(
+ page: number = 1,
+ limit: number = 20,
+ status?: string,
+ ): Promise<{ items: Pipeline[]; total: number }> {
+ const pipelineModel =
+ await this.connectionService.getMoeumModels('pipelines');
+ const skip = (page - 1) * limit;
+
+ const filter: Record = {};
+ if (status) {
+ filter['metadatas'] = {
+ $elemMatch: { key: 'pipelineStatus', value: status },
+ };
+ }
+
+ const [items, total] = await Promise.all([
+ pipelineModel
+ .find(filter)
+ .sort({ createdAt: -1 })
+ .skip(skip)
+ .limit(limit)
+ .exec(),
+ pipelineModel.countDocuments(filter).exec(),
+ ]);
+
+ return { items: items as Pipeline[], total };
+ }
+}
diff --git a/src/crawler/services/rate-limiter.service.ts b/src/crawler/services/rate-limiter.service.ts
new file mode 100644
index 0000000..bed9005
--- /dev/null
+++ b/src/crawler/services/rate-limiter.service.ts
@@ -0,0 +1,138 @@
+import { Injectable, Logger } from '@nestjs/common';
+import { RateLimitConfig } from '../interfaces/crawler-result.interface';
+
+interface TokenBucket {
+ tokens: number;
+ lastRefill: number;
+ config: RateLimitConfig;
+}
+
+@Injectable()
+export class RateLimiterService {
+ private readonly logger = new Logger(RateLimiterService.name);
+ private readonly buckets: Map = new Map();
+ private readonly defaultConfig: RateLimitConfig = {
+ maxTokens: 10, // 최대 10개 토큰
+ refillRate: 2, // 초당 2개 토큰 보충
+ };
+
+ /**
+ * 도메인별 Rate Limit 설정
+ */
+ setLimit(domain: string, config: RateLimitConfig): void {
+ const existing = this.buckets.get(domain);
+ this.buckets.set(domain, {
+ tokens: config.maxTokens,
+ lastRefill: Date.now(),
+ config,
+ });
+ this.logger.log(
+ `Rate limit set for ${domain}: ${config.maxTokens} tokens, ${config.refillRate} req/s`,
+ );
+ }
+
+ /**
+ * 설정된 Rate Limit 조회
+ */
+ getLimit(domain: string): RateLimitConfig {
+ const bucket = this.buckets.get(domain);
+ return bucket?.config ?? this.defaultConfig;
+ }
+
+ /**
+ * 토큰 획득 (대기 필요시 대기)
+ */
+ async acquire(domain: string): Promise {
+ let bucket = this.buckets.get(domain);
+
+ // 버킷이 없으면 기본 설정으로 생성
+ if (!bucket) {
+ bucket = {
+ tokens: this.defaultConfig.maxTokens,
+ lastRefill: Date.now(),
+ config: this.defaultConfig,
+ };
+ this.buckets.set(domain, bucket);
+ }
+
+ // 토큰 보충
+ this.refill(bucket);
+
+ // 토큰이 없으면 대기
+ if (bucket.tokens <= 0) {
+ const waitTime = Math.ceil(1000 / bucket.config.refillRate);
+ this.logger.debug(`Rate limit hit for ${domain}, waiting ${waitTime}ms`);
+ await this.delay(waitTime);
+ this.refill(bucket);
+ }
+
+ // 토큰 소비
+ bucket.tokens--;
+ this.logger.debug(
+ `Token acquired for ${domain}, remaining: ${bucket.tokens}`,
+ );
+ }
+
+ /**
+ * 토큰 보충 (시간 경과에 따라)
+ */
+ private refill(bucket: TokenBucket): void {
+ const now = Date.now();
+ const elapsed = (now - bucket.lastRefill) / 1000; // 초 단위
+ const tokensToAdd = elapsed * bucket.config.refillRate;
+
+ bucket.tokens = Math.min(
+ bucket.config.maxTokens,
+ bucket.tokens + tokensToAdd,
+ );
+ bucket.lastRefill = now;
+ }
+
+ /**
+ * 현재 사용 가능한 토큰 수 조회
+ */
+ getAvailableTokens(domain: string): number {
+ const bucket = this.buckets.get(domain);
+ if (!bucket) {
+ return this.defaultConfig.maxTokens;
+ }
+ this.refill(bucket);
+ return Math.floor(bucket.tokens);
+ }
+
+ /**
+ * 모든 도메인의 Rate Limit 상태 조회
+ */
+ getAllLimits(): Record {
+ const result: Record = {};
+
+ for (const [domain, bucket] of this.buckets) {
+ this.refill(bucket);
+ result[domain] = {
+ config: bucket.config,
+ available: Math.floor(bucket.tokens),
+ };
+ }
+
+ return result;
+ }
+
+ /**
+ * 특정 도메인의 Rate Limit 초기화
+ */
+ resetLimit(domain: string): void {
+ const bucket = this.buckets.get(domain);
+ if (bucket) {
+ bucket.tokens = bucket.config.maxTokens;
+ bucket.lastRefill = Date.now();
+ this.logger.log(`Rate limit reset for ${domain}`);
+ }
+ }
+
+ /**
+ * 대기 함수
+ */
+ private delay(ms: number): Promise {
+ return new Promise((resolve) => setTimeout(resolve, ms));
+ }
+}
diff --git a/src/crawler/services/retry.service.ts b/src/crawler/services/retry.service.ts
new file mode 100644
index 0000000..3a92672
--- /dev/null
+++ b/src/crawler/services/retry.service.ts
@@ -0,0 +1,197 @@
+import { Injectable, Logger } from '@nestjs/common';
+import { NatsService } from '../../global/nats/nats.service';
+import { RetryStrategy } from '../../global/database/schema';
+
+export type RetryableOperation = () => Promise;
+
+export interface RetryResult {
+ success: boolean;
+ result?: T;
+ attempts: number;
+ errors: string[];
+}
+
+@Injectable()
+export class RetryService {
+ private readonly logger = new Logger(RetryService.name);
+
+ constructor(private readonly natsService: NatsService) {}
+
+ /**
+ * 재시도 로직과 함께 작업 실행
+ */
+ async executeWithRetry(
+ operation: RetryableOperation,
+ strategy: RetryStrategy,
+ ): Promise {
+ let lastError: Error | null = null;
+ let currentDelay = strategy.delay;
+ const errors: string[] = [];
+ const maxAttempts = strategy.limit + 1;
+
+ for (let attempt = 1; attempt <= maxAttempts; attempt++) {
+ try {
+ this.logger.debug(`Attempt ${attempt}/${maxAttempts}`);
+ const result = await operation();
+
+ // 성공 시 반환
+ if (attempt > 1) {
+ this.logger.log(`Operation succeeded on attempt ${attempt}`);
+ }
+ return result;
+ } catch (error) {
+ lastError = error instanceof Error ? error : new Error(String(error));
+ errors.push(`Attempt ${attempt}: ${lastError.message}`);
+ this.logger.warn(`Attempt ${attempt} failed: ${lastError.message}`);
+
+ // 재시도 조건 확인
+ if (!this.shouldRetry(lastError, strategy.when)) {
+ this.logger.debug(
+ 'Error does not match retry conditions, not retrying',
+ );
+ break;
+ }
+
+ // 마지막 시도가 아니면 대기
+ if (attempt < maxAttempts) {
+ this.logger.debug(`Waiting ${currentDelay}ms before next attempt`);
+ await this.delay(currentDelay);
+ currentDelay = Math.floor(currentDelay * strategy.backoff); // Exponential backoff
+ }
+ }
+ }
+
+ // 모든 재시도 실패 - 알림 발송
+ if (strategy.notify && strategy.notify.length > 0) {
+ await this.sendFailureNotification(strategy.notify, errors);
+ }
+
+ throw lastError || new Error('Operation failed after all retries');
+ }
+
+ /**
+ * 재시도 없이 결과를 래핑하여 반환
+ */
+ async executeWithResult(
+ operation: RetryableOperation,
+ strategy: RetryStrategy,
+ ): Promise> {
+ const errors: string[] = [];
+ let currentDelay = strategy.delay;
+ const maxAttempts = strategy.limit + 1;
+
+ for (let attempt = 1; attempt <= maxAttempts; attempt++) {
+ try {
+ const result = await operation();
+ return {
+ success: true,
+ result,
+ attempts: attempt,
+ errors,
+ };
+ } catch (error) {
+ const errorMessage =
+ error instanceof Error ? error.message : String(error);
+ errors.push(`Attempt ${attempt}: ${errorMessage}`);
+
+ if (!this.shouldRetry(error as Error, strategy.when)) {
+ break;
+ }
+
+ if (attempt < maxAttempts) {
+ await this.delay(currentDelay);
+ currentDelay = Math.floor(currentDelay * strategy.backoff);
+ }
+ }
+ }
+
+ // 모든 재시도 실패
+ if (strategy.notify && strategy.notify.length > 0) {
+ await this.sendFailureNotification(strategy.notify, errors);
+ }
+
+ return {
+ success: false,
+ attempts: errors.length,
+ errors,
+ };
+ }
+
+ /**
+ * 재시도 조건 확인
+ */
+ private shouldRetry(error: Error, conditions: string[]): boolean {
+ // 모든 실패에 대해 재시도
+ if (conditions.includes('failed')) {
+ return true;
+ }
+
+ const errorMessage = error.message.toLowerCase();
+
+ // 특정 에러 타입에 대해서만 재시도
+ for (const condition of conditions) {
+ if (errorMessage.includes(condition.toLowerCase())) {
+ return true;
+ }
+
+ // HTTP 상태 코드 기반 조건 (예: '5xx', '429', '500')
+ if (condition.match(/^\d{3}$/) && errorMessage.includes(condition)) {
+ return true;
+ }
+
+ // 5xx 패턴 매칭
+ if (condition === '5xx' && errorMessage.match(/5\d{2}/)) {
+ return true;
+ }
+
+ // 4xx 패턴 매칭
+ if (condition === '4xx' && errorMessage.match(/4\d{2}/)) {
+ return true;
+ }
+
+ // 네트워크 에러
+ if (
+ condition === 'network' &&
+ (errorMessage.includes('econnrefused') ||
+ errorMessage.includes('enotfound') ||
+ errorMessage.includes('timeout') ||
+ errorMessage.includes('socket'))
+ ) {
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ /**
+ * 실패 알림 발송
+ */
+ private async sendFailureNotification(
+ notifyChannels: string[],
+ errors: string[],
+ ): Promise {
+ const notification = {
+ type: 'crawler.execution.failed',
+ timestamp: new Date().toISOString(),
+ errors,
+ message: `Operation failed after ${errors.length} attempts`,
+ };
+
+ for (const channel of notifyChannels) {
+ try {
+ await this.natsService.publish(`notifications.${channel}`, notification);
+ this.logger.log(`Failure notification sent to: ${channel}`);
+ } catch (error) {
+ this.logger.error(`Failed to send notification to ${channel}`, error);
+ }
+ }
+ }
+
+ /**
+ * 대기 함수
+ */
+ private delay(ms: number): Promise {
+ return new Promise((resolve) => setTimeout(resolve, ms));
+ }
+}
diff --git a/src/crawler/services/stage-executor.service.ts b/src/crawler/services/stage-executor.service.ts
new file mode 100644
index 0000000..46f97c2
--- /dev/null
+++ b/src/crawler/services/stage-executor.service.ts
@@ -0,0 +1,255 @@
+import { Injectable, Logger } from '@nestjs/common';
+import { ConnectionService } from '../../global/database/connection.service';
+import { NatsService } from '../../global/nats/nats.service';
+import { CrawlerService } from './crawler.service';
+import { RetryService } from './retry.service';
+import {
+ Stagedef,
+ Execution,
+ RetryStrategy,
+} from '../../global/database/schema';
+import {
+ CrawlResult,
+ StageExecutionResult,
+} from '../interfaces/crawler-result.interface';
+
+const DEFAULT_RETRY_STRATEGY: RetryStrategy = {
+ limit: 3,
+ delay: 1000,
+ backoff: 2,
+ when: ['failed', '5xx', 'network'],
+ notify: [],
+};
+
+@Injectable()
+export class StageExecutorService {
+ private readonly logger = new Logger(StageExecutorService.name);
+
+ constructor(
+ private readonly connectionService: ConnectionService,
+ private readonly natsService: NatsService,
+ private readonly crawlerService: CrawlerService,
+ private readonly retryService: RetryService,
+ ) {}
+
+ /**
+ * 단일 Stage 실행
+ */
+ async executeStage(
+ pipelineId: string,
+ stageIndex: number,
+ stagedef: Stagedef,
+ ): Promise {
+ const pipelineModel =
+ await this.connectionService.getMoeumModels('pipelines');
+
+ // Stage 상태를 'in-progress'로 변경
+ await pipelineModel.findByIdAndUpdate(pipelineId, {
+ $set: { [`stage.${stageIndex}.status`]: 'in-progress' },
+ });
+
+ await this.natsService.publish('crawler.stage.started', {
+ pipelineId,
+ stageIndex,
+ stageName: stagedef.name,
+ timestamp: new Date().toISOString(),
+ });
+
+ const executions: Execution[] = [];
+ const crawlResults: CrawlResult[] = [];
+ let finalStatus: 'completed' | 'failed' = 'completed';
+
+ try {
+ // EventSource별로 크롤링 실행
+ for (const eventSource of stagedef.eventSource) {
+ const execution = await this.executeEventSource(
+ pipelineId,
+ stageIndex,
+ eventSource,
+ stagedef,
+ );
+ executions.push(execution);
+
+ if (execution.status === 'failed') {
+ finalStatus = 'failed';
+ }
+
+ // 크롤링 결과 추출
+ const resultMeta = execution.metadatas.find(
+ (m) => m.key === 'crawlResult',
+ );
+ if (resultMeta) {
+ try {
+ const result = JSON.parse(resultMeta.value as string);
+ crawlResults.push(result);
+ } catch {
+ // JSON 파싱 실패 무시
+ }
+ }
+ }
+
+ // Stage 상태 및 실행 기록 업데이트
+ await pipelineModel.findByIdAndUpdate(pipelineId, {
+ $set: {
+ [`stage.${stageIndex}.status`]: finalStatus,
+ [`stage.${stageIndex}.executions`]: executions,
+ },
+ });
+
+ // 완료 이벤트 발행 (Stagedef의 topic으로)
+ await this.natsService.publish(stagedef.topic, {
+ pipelineId,
+ stageIndex,
+ stageName: stagedef.name,
+ status: finalStatus,
+ results: crawlResults.map((r) => ({
+ url: r.url,
+ success: r.success,
+ parsedData: r.parsedData,
+ })),
+ timestamp: new Date().toISOString(),
+ });
+
+ this.logger.log(
+ `Stage ${stagedef.name} completed with status: ${finalStatus}`,
+ );
+ } catch (error) {
+ finalStatus = 'failed';
+ this.logger.error(`Stage ${stagedef.name} failed`, error);
+
+ await pipelineModel.findByIdAndUpdate(pipelineId, {
+ $set: { [`stage.${stageIndex}.status`]: 'failed' },
+ $push: {
+ [`stage.${stageIndex}.executions`]: {
+ status: 'failed',
+ metadatas: [
+ {
+ key: 'error',
+ value: error instanceof Error ? error.message : 'Unknown error',
+ dataType: 'string',
+ isReserved: true,
+ },
+ ],
+ createdAt: new Date(),
+ updatedAt: new Date(),
+ },
+ },
+ });
+ }
+
+ return {
+ stageIndex,
+ stageName: stagedef.name,
+ status: finalStatus,
+ crawlResults,
+ };
+ }
+
+ /**
+ * 단일 EventSource 실행 (재시도 로직 포함)
+ */
+ private async executeEventSource(
+ pipelineId: string,
+ stageIndex: number,
+ eventSource: any,
+ stagedef: Stagedef,
+ ): Promise {
+ const startedAt = new Date();
+ const execution: Execution = {
+ status: 'in-progress',
+ metadatas: [
+ {
+ key: 'eventSourceName',
+ value: eventSource.name,
+ dataType: 'string',
+ isReserved: true,
+ },
+ {
+ key: 'eventSourceType',
+ value: eventSource.type,
+ dataType: 'string',
+ isReserved: true,
+ },
+ {
+ key: 'startedAt',
+ value: startedAt.toISOString(),
+ dataType: 'string',
+ isReserved: true,
+ },
+ ],
+ createdAt: startedAt,
+ updatedAt: startedAt,
+ };
+
+ // RetryStrategy 추출 (Stagedef 메타데이터에서 또는 기본값)
+ const retryStrategyMeta = stagedef.metadatas.find(
+ (m) => m.key === 'retryStrategy',
+ );
+ let retryStrategy: RetryStrategy = DEFAULT_RETRY_STRATEGY;
+
+ if (retryStrategyMeta) {
+ try {
+ retryStrategy =
+ typeof retryStrategyMeta.value === 'string'
+ ? JSON.parse(retryStrategyMeta.value)
+ : retryStrategyMeta.value;
+ } catch {
+ // 파싱 실패 시 기본값 사용
+ }
+ }
+
+ try {
+ // 재시도 로직과 함께 크롤링 실행
+ const crawlResult = await this.retryService.executeWithRetry(
+ () => this.crawlerService.crawlFromEventSource(eventSource),
+ retryStrategy,
+ );
+
+ execution.status = crawlResult.success ? 'completed' : 'failed';
+ execution.metadatas.push({
+ key: 'crawlResult',
+ value: JSON.stringify(crawlResult),
+ dataType: 'json',
+ isReserved: false,
+ });
+
+ // MinIO에 결과 저장
+ if (crawlResult.success) {
+ try {
+ const storageKey = await this.crawlerService.saveResultToStorage(
+ crawlResult,
+ `${pipelineId}-${stageIndex}`,
+ );
+ execution.metadatas.push({
+ key: 'storageKey',
+ value: storageKey,
+ dataType: 'string',
+ isReserved: false,
+ });
+ } catch (error) {
+ this.logger.warn(`Failed to save result to MinIO: ${error}`);
+ }
+ }
+ } catch (error) {
+ execution.status = 'failed';
+ execution.metadatas.push({
+ key: 'error',
+ value: error instanceof Error ? error.message : 'Unknown error',
+ dataType: 'string',
+ isReserved: true,
+ });
+ }
+
+ // 완료 시간 기록
+ const completedAt = new Date();
+ execution.metadatas.push({
+ key: 'completedAt',
+ value: completedAt.toISOString(),
+ dataType: 'string',
+ isReserved: true,
+ });
+ execution.updatedAt = completedAt;
+
+ return execution;
+ }
+}
diff --git a/src/crawler/services/trigger.service.ts b/src/crawler/services/trigger.service.ts
new file mode 100644
index 0000000..64c0bd2
--- /dev/null
+++ b/src/crawler/services/trigger.service.ts
@@ -0,0 +1,327 @@
+import {
+ Injectable,
+ Logger,
+ OnModuleInit,
+ OnModuleDestroy,
+} from '@nestjs/common';
+import { SchedulerRegistry } from '@nestjs/schedule';
+import { CronJob } from 'cron';
+import { NatsConnection, StringCodec, Subscription, connect } from 'nats';
+import * as fs from 'fs/promises';
+import { ConnectionService } from '../../global/database/connection.service';
+import { SecretService } from '../../global/secret/secret.service';
+import { PipelineEngineService } from './pipeline-engine.service';
+import { Schedule } from '../../global/database/schema';
+
+export interface ScheduledJob {
+ pipelineDefId: string;
+ jobName: string;
+ cronExpression: string;
+ timezone: string;
+}
+
+@Injectable()
+export class TriggerService implements OnModuleInit, OnModuleDestroy {
+ private readonly logger = new Logger(TriggerService.name);
+ private scheduledJobs: Map = new Map();
+ private natsSubscriptions: Subscription[] = [];
+ private natsConnection: NatsConnection | null = null;
+ private readonly stringCodec = StringCodec();
+
+ constructor(
+ private readonly schedulerRegistry: SchedulerRegistry,
+ private readonly connectionService: ConnectionService,
+ private readonly pipelineEngine: PipelineEngineService,
+ private readonly secretService: SecretService,
+ ) {}
+
+ async onModuleInit() {
+ try {
+ // NATS 연결 초기화
+ await this.initializeNatsConnection();
+
+ // 기존 트리거 로드
+ await this.loadTriggers();
+
+ this.logger.log('TriggerService initialized');
+ } catch (error) {
+ this.logger.error('Failed to initialize TriggerService', error);
+ }
+ }
+
+ async onModuleDestroy() {
+ // 스케줄 정리
+ for (const [jobName] of this.scheduledJobs) {
+ try {
+ this.schedulerRegistry.deleteCronJob(jobName);
+ } catch {
+ // 무시
+ }
+ }
+ this.scheduledJobs.clear();
+
+ // NATS 구독 정리
+ for (const sub of this.natsSubscriptions) {
+ try {
+ await sub.drain();
+ } catch {
+ // 무시
+ }
+ }
+ this.natsSubscriptions = [];
+
+ // NATS 연결 종료
+ if (this.natsConnection) {
+ try {
+ await this.natsConnection.drain();
+ await this.natsConnection.close();
+ } catch {
+ // 무시
+ }
+ this.natsConnection = null;
+ }
+
+ this.logger.log('TriggerService destroyed');
+ }
+
+ /**
+ * NATS 연결 초기화
+ */
+ private async initializeNatsConnection(): Promise {
+ await this.secretService.ready();
+
+ const endpoint = await this.secretService.get('NATS_ENDPOINT');
+ const user = await this.secretService.get('NATS_USER');
+ const password = await this.secretService.get('NATS_PASS');
+
+ let tlsConfig: { ca?: string } | undefined;
+ try {
+ const cacert = (await fs.readFile('internal-service-ca.pem')).toString();
+ tlsConfig = { ca: cacert };
+ } catch {
+ this.logger.warn('TLS certificate not found, connecting without TLS');
+ }
+
+ this.natsConnection = await connect({
+ name: 'crawler-trigger-service',
+ servers: [endpoint],
+ tls: tlsConfig,
+ user,
+ pass: password,
+ maxReconnectAttempts: -1,
+ reconnectTimeWait: 500,
+ reconnect: true,
+ pingInterval: 20000,
+ timeout: 5000,
+ });
+
+ this.logger.log('NATS connection established for TriggerService');
+ }
+
+ /**
+ * 모든 트리거 로드 (스케줄 + 이벤트 구독)
+ */
+ async loadTriggers(): Promise {
+ const pipelinedefModel =
+ await this.connectionService.getMoeumModels('pipelinedefs');
+ const pipelinedefs = await pipelinedefModel.find().exec();
+
+ for (const pipelinedef of pipelinedefs) {
+ // 스케줄 트리거 확인 (메타데이터에서)
+ const scheduleMeta = pipelinedef.metadatas?.find(
+ (m) => m.key === 'schedule',
+ );
+ if (scheduleMeta) {
+ try {
+ const schedule: Schedule =
+ typeof scheduleMeta.value === 'string'
+ ? JSON.parse(scheduleMeta.value)
+ : scheduleMeta.value;
+ await this.registerScheduleTrigger(
+ pipelinedef._id.toString(),
+ schedule,
+ );
+ } catch (error) {
+ this.logger.warn(
+ `Failed to parse schedule for pipelinedef ${pipelinedef._id}`,
+ );
+ }
+ }
+
+ // Kafka/NATS 이벤트 트리거
+ if (
+ pipelinedef.trigger?.type === 'kafka-event' &&
+ pipelinedef.trigger.kafkaEvent?.topic
+ ) {
+ await this.registerNatsTrigger(
+ pipelinedef._id.toString(),
+ pipelinedef.trigger.kafkaEvent.topic,
+ );
+ }
+ }
+
+ this.logger.log(
+ `Loaded ${this.scheduledJobs.size} schedule triggers and ${this.natsSubscriptions.length} event triggers`,
+ );
+ }
+
+ /**
+ * 스케줄 트리거 등록
+ */
+ async registerScheduleTrigger(
+ pipelineDefId: string,
+ schedule: Schedule,
+ ): Promise {
+ const jobName = `schedule-${pipelineDefId}`;
+
+ // 기존 작업이 있으면 제거
+ try {
+ if (this.schedulerRegistry.doesExist('cron', jobName)) {
+ this.schedulerRegistry.deleteCronJob(jobName);
+ this.scheduledJobs.delete(jobName);
+ }
+ } catch {
+ // 무시
+ }
+
+ const job = new CronJob(
+ schedule.cron,
+ async () => {
+ this.logger.log(`Scheduled trigger fired for pipeline: ${pipelineDefId}`);
+ try {
+ await this.pipelineEngine.createAndExecutePipeline(
+ pipelineDefId,
+ 'schedule',
+ );
+ } catch (error) {
+ this.logger.error(
+ `Scheduled trigger failed: ${pipelineDefId}`,
+ error,
+ );
+ }
+ },
+ null,
+ true,
+ schedule.timezone || 'Asia/Seoul',
+ );
+
+ this.schedulerRegistry.addCronJob(jobName, job);
+ this.scheduledJobs.set(jobName, {
+ pipelineDefId,
+ jobName,
+ cronExpression: schedule.cron,
+ timezone: schedule.timezone || 'Asia/Seoul',
+ });
+
+ this.logger.log(
+ `Registered schedule trigger: ${jobName} (${schedule.cron})`,
+ );
+ }
+
+ /**
+ * NATS 이벤트 트리거 등록
+ */
+ async registerNatsTrigger(
+ pipelineDefId: string,
+ topic: string,
+ ): Promise {
+ if (!this.natsConnection) {
+ this.logger.warn('NATS connection not available');
+ return;
+ }
+
+ const subscription = this.natsConnection.subscribe(topic, {
+ callback: async (err, msg) => {
+ if (err) {
+ this.logger.error(`NATS subscription error on ${topic}`, err);
+ return;
+ }
+
+ this.logger.log(`NATS trigger received on ${topic}`);
+ try {
+ // 메시지 페이로드 파싱 (필요시 사용)
+ let payload: any = {};
+ try {
+ payload = JSON.parse(this.stringCodec.decode(msg.data));
+ } catch {
+ // JSON 아닌 경우 무시
+ }
+
+ await this.pipelineEngine.createAndExecutePipeline(
+ pipelineDefId,
+ 'nats',
+ );
+ } catch (error) {
+ this.logger.error(
+ `NATS trigger processing failed: ${pipelineDefId}`,
+ error,
+ );
+ }
+ },
+ });
+
+ this.natsSubscriptions.push(subscription);
+ this.logger.log(`Registered NATS trigger: ${topic} -> ${pipelineDefId}`);
+ }
+
+ /**
+ * API 기반 수동 트리거
+ */
+ async triggerPipelineManually(pipelineDefId: string): Promise {
+ const pipeline = await this.pipelineEngine.createAndExecutePipeline(
+ pipelineDefId,
+ 'api',
+ );
+ return pipeline._id.toString();
+ }
+
+ /**
+ * 스케줄 제거
+ */
+ async unregisterScheduleTrigger(pipelineDefId: string): Promise {
+ const jobName = `schedule-${pipelineDefId}`;
+ try {
+ if (this.schedulerRegistry.doesExist('cron', jobName)) {
+ this.schedulerRegistry.deleteCronJob(jobName);
+ this.scheduledJobs.delete(jobName);
+ this.logger.log(`Unregistered schedule trigger: ${jobName}`);
+ }
+ } catch {
+ // 무시
+ }
+ }
+
+ /**
+ * 현재 등록된 스케줄 목록 조회
+ */
+ getRegisteredSchedules(): ScheduledJob[] {
+ return Array.from(this.scheduledJobs.values());
+ }
+
+ /**
+ * 트리거 재로드
+ */
+ async reloadTriggers(): Promise {
+ // 기존 트리거 정리
+ for (const [jobName] of this.scheduledJobs) {
+ try {
+ this.schedulerRegistry.deleteCronJob(jobName);
+ } catch {
+ // 무시
+ }
+ }
+ this.scheduledJobs.clear();
+
+ for (const sub of this.natsSubscriptions) {
+ try {
+ await sub.drain();
+ } catch {
+ // 무시
+ }
+ }
+ this.natsSubscriptions = [];
+
+ // 다시 로드
+ await this.loadTriggers();
+ }
+}
diff --git a/src/global/database/connection.service.ts b/src/global/database/connection.service.ts
index cf84ef3..8c7768c 100644
--- a/src/global/database/connection.service.ts
+++ b/src/global/database/connection.service.ts
@@ -1,7 +1,7 @@
import { Injectable, Scope, OnModuleInit } from '@nestjs/common';
import mongoose from 'mongoose';
import { SecretService } from '../secret/secret.service.js';
-import { moeumModelDefinitions, MoeumModelTypes } from './schema.js';
+import { CrawlerModelDefinitions, CrawlerModelTypes } from './schema.js';
@Injectable({ scope: Scope.DEFAULT })
export class ConnectionService implements OnModuleInit {
@@ -31,19 +31,19 @@ export class ConnectionService implements OnModuleInit {
return this.connection;
}
- public async getMoeumModels(
+ public async getMoeumModels(
modelKey: K,
- ): Promise> {
+ ): Promise> {
if (this.moeumModels)
return this.moeumModels.get(modelKey) as mongoose.Model<
- MoeumModelTypes[K]
+ CrawlerModelTypes[K]
>;
- const moeumModels = new Map>();
+ const moeumModels = new Map>();
// const connection = await this.getTenantConnection(tenantId);
const connection = await this.getConnection();
- moeumModelDefinitions.forEach((modelDefinition) => {
+ CrawlerModelDefinitions.forEach((modelDefinition) => {
// 이미 생성된 경우 컴파일 스킵.
if (
Object.prototype.hasOwnProperty.call(
@@ -59,7 +59,7 @@ export class ConnectionService implements OnModuleInit {
} else {
moeumModels.set(
modelDefinition.name as K,
- connection.model(
+ connection.model(
modelDefinition.name,
modelDefinition.schema,
modelDefinition.schemaName,
@@ -69,7 +69,7 @@ export class ConnectionService implements OnModuleInit {
});
this.moeumModels = moeumModels;
- return moeumModels.get(modelKey) as mongoose.Model;
+ return moeumModels.get(modelKey) as mongoose.Model;
}
public async createSession(
diff --git a/src/global/database/schema.ts b/src/global/database/schema.ts
index 28340ee..c757699 100644
--- a/src/global/database/schema.ts
+++ b/src/global/database/schema.ts
@@ -8,236 +8,1045 @@ export interface MongooseTimestamps {
export interface Metadata {
key: string;
- value: string | any[]
+ value: string | any[];
dataType: 'string' | 'number' | 'boolean' | 'array' | 'json';
isReserved: boolean;
}
-export interface TaxonomyChildItem {
- title: string;
- order: number;
- articleId?: mongoose.Types.ObjectId;
- slug?: string; //article document의 slug 참조 (임베딩)
- children?: TaxonomyChildItem[];
+export interface Execution {
+ // executionId: string; // 각 실행의 고유 ID
+ status:
+ | 'pending'
+ | 'in-progress'
+ | 'completed'
+ | 'failed'
+ | 'cancelled'
+ | 'skipped';
+ // artifact?: [Artifact];
+ metadatas: Metadata[];
+ createdAt: Date;
+ updatedAt: Date;
}
-export interface Taxonomy
- extends MongooseTimestamps {
- label: string;
- items: TaxonomyChildItem[];
+export type ExecutionDocument = HydratedDocument;
+
+export type RESTRequest = {
+ url: string;
+ method: 'GET' | 'POST' | 'PUT' | 'DELETE' | 'PATCH';
+ headers?: { [key: string]: string };
+ queryParams?: { [key: string]: any };
+ body?: any;
+ auth?: {
+ type: 'BearerToken' | 'BasicAuth' | 'APIKey';
+ token?: string; // BearerToken 또는 APIKey에 사용
+ username?: string; // BasicAuth에 사용
+ password?: string; // BasicAuth에 사용
+ };
+ timeout?: number; // 요청 타임아웃 (밀리초)
+ retryOptions?: {
+ retries: number;
+ delay: number; // 밀리초
+ };
+};
+
+export type KafkaEvent = {
+ topic: string;
+ key?: string;
+};
+
+export type PipelineTrigger = {
+ type: 'always' | 'kafka-event';
+ kafkaEvent?: KafkaEvent;
+};
+
+export type StageTrigger = {
+ type: 'always' | 'condition' | 'schedule';
+ condition?: Condition | CompositeCondition;
+ schedule?: Schedule;
+};
+
+export type Condition = {
+ variable: string;
+ operator: 'eq' | 'ne' | 'gt' | 'lt' | 'gte' | 'lte';
+ value: any;
+};
+
+export type CompositeCondition = {
+ logic: 'AND' | 'OR';
+ conditions: (Condition | CompositeCondition)[];
+};
+
+export type Schedule = {
+ cron: string; // 크론 표현식
+ timezone?: string;
+};
+
+// export type Artifact = {
+// key: string;
+// value: any;
+// dataType: DataType;
+// isReserved: boolean;
+// };
+
+export type Variable = {
+ name: string;
+ value: any;
+ scope: 'pipeline' | 'stage';
+};
+
+export interface Pipelinedef {
+ apiVersion: string;
+ trigger: PipelineTrigger;
+ stage: mongoose.Types.ObjectId[];
+ revision: number;
metadatas: Metadata[];
}
+export type PipelinedefDocument = HydratedDocument;
-export type TaxonomyDocument = HydratedDocument
-
-export interface ContentSection {
- headline: string;
- taxonomies: ContentSectionEmbeddedTaxonomy[];
-}
-
-export interface ContentSectionEmbeddedTaxonomy {
- taxonomyId: mongoose.Types.ObjectId;
- label: string; // 성능 최적화를 위해 임베딩된 label
-}
-
-export interface Content
- extends MongooseTimestamps {
- key: string;
- label: string;
- sections: ContentSection[];
-}
-
-export type ContentDocument = HydratedDocument
-
-export interface Article
- extends MongooseTimestamps {
- title: string;
- slug: string;
- content: string;
- status: 'draft' | 'published' | 'archived';
+export interface Pipeline {
+ apiVersion: string;
+ name?: string;
metadatas: Metadata[];
+ stage: [Stage & { _id: mongoose.Types.ObjectId }];
}
-export type ArticleDocument = HydratedDocument
+export type PipelineDocument = HydratedDocument;
+export interface Stagedef {
+ apiVersion: string;
+ name: string;
+ taxonomy?: string;
+ eventSource: EventSource[];
+ schema?: any;
-export const MetadataSubSchema = new mongoose.Schema({
- key: { type: String, required: true },
- value: {
- type: mongoose.SchemaTypes.Mixed,
- required: false, // required: true로 하면 value가 null인 경우 에러 발생
- default: null,
- validate: {
- validator: function (v: string | any[]) {
- // 특정 키인 경우 value가 null 또는 빈 문자열이 아니어야 함
- // if (this.key === 'requiredKey') {
- // return typeof v === 'string' && v.trim().length > 0;
- // }
- // 그렇지 않은 경우 value는 null 또는 문자열, Array이어야 함
- return typeof v === 'string' || Array.isArray(v);
- },
- message: function (props: any) {
- if (this.key === 'requiredKey') {
- return `Value is required for key "${this.key}" and cannot be null or empty.`;
- }
- return `Value must be a string or array for key "${this.key}".`;
+ topic: string;
+ // lifecycleHooks?: LifecycleHooks;
+ metadatas: Metadata[];
+ /**
+ * 실제 존재하는 값이 아닌 가상 필드입니다.
+ * 실제 데이터베이스에 저장되지 않습니다.
+ * 메타데이터를 조작하기 쉽도록 Metadatas 인스턴스로 변환하여 반환합니다.
+ * Query.prototype.lean()을 사용할 때는 가상 필드가 반환되지 않습니다.
+ * lean()을 사용하지 않는 경우에만 사용하세요. (exec() 사용)
+ */
+ // metadatas?: Metadatas; //Virtual
+}
+
+export type StagedefDocument = HydratedDocument;
+
+export interface Stage {
+ apiVersion: string;
+ name: string;
+ schema?: any;
+ taxonomy?: string;
+ // lifecycleHooks?: LifecycleHooks;
+ executions: Execution[];
+ // outputData?: any;
+ status:
+ | 'pending' // 작업 대기
+ | 'in-progress' // 작업이 실행 중
+ | 'completed' // 작업이 완료됨(성공, 실패)
+ | 'failed' // 작업이 실패함
+ | 'cancelled' // 작업이 취소됨
+ | 'skipped'; // 작업이 스킵됨
+ // artifact?: Artifact;
+ metadatas: Metadata[];
+ /**
+ * 실제 존재하는 값이 아닌 가상 필드입니다.
+ * 실제 데이터베이스에 저장되지 않습니다.
+ * 메타데이터를 조작하기 쉽도록 Metadatas 인스턴스로 변환하여 반환합니다.
+ * Query.prototype.lean()을 사용할 때는 가상 필드가 반환되지 않습니다.
+ * lean()을 사용하지 않는 경우에만 사용하세요. (exec() 사용)
+ */
+ // metadatas?: Metadatas; //Virtual
+ retryStrategy?: RetryStrategy;
+ // variables?: Variable[];
+}
+export type StageDocument = HydratedDocument;
+
+export type EventSource = {
+ type: string;
+ name: string;
+ metadatas: Metadata[];
+};
+export type StagedefEventSourceDocument = HydratedDocument;
+
+export type RetryStrategy = {
+ limit: number;
+ delay: number;
+ backoff: number;
+ when: string[];
+ notify: string[];
+};
+
+const RESTRequestSchema = new mongoose.Schema(
+ {
+ url: { type: String, required: true },
+ method: {
+ type: String,
+ enum: ['GET', 'POST', 'PUT', 'DELETE', 'PATCH'],
+ required: true,
+ },
+ headers: { type: Map, of: String },
+ queryParams: { type: Map, of: mongoose.Schema.Types.Mixed },
+ body: { type: mongoose.Schema.Types.Mixed },
+ auth: {
+ type: new mongoose.Schema(
+ {
+ type: {
+ type: String,
+ enum: ['BearerToken', 'BasicAuth', 'APIKey'],
+ required: true,
+ },
+ token: { type: String },
+ username: { type: String },
+ password: { type: String },
+ },
+ { _id: false },
+ ),
+ },
+ timeout: { type: Number },
+ retryOptions: {
+ type: new mongoose.Schema(
+ {
+ retries: { type: Number, default: 0 },
+ delay: { type: Number, default: 1000 },
+ },
+ { _id: false },
+ ),
+ },
+ },
+ { _id: false },
+);
+
+// const LifecycleHooksSchema = new mongoose.Schema(
+// {
+// preExecution: { type: String },
+// postExecution: { type: RESTRequestSchema },
+// },
+// { _id: false },
+// );
+
+const KafkaEventSchema = new mongoose.Schema(
+ {
+ topic: { type: String, required: true },
+ key: { type: String },
+ },
+ { _id: false },
+);
+
+export const PipelineTriggerSchema = new mongoose.Schema(
+ {
+ type: {
+ type: String,
+ enum: ['always', 'kafka-event'],
+ required: true,
+ },
+ kafkaEvent: {
+ type: KafkaEventSchema,
+ required: function (this: PipelineTrigger) {
+ return this.type === 'kafka-event';
},
},
},
- dataType: { type: String, required: false, default: 'string', enum: ['string', 'number', 'boolean', 'array', 'json'] },
- isReserved: { type: Boolean, required: false, default: false },
-}, {
- _id: false,
+ { _id: false },
+);
+
+const ConditionSchema = new mongoose.Schema(
+ {
+ variable: { type: String },
+ operator: {
+ type: String,
+ enum: ['eq', 'ne', 'gt', 'lt', 'gte', 'lte'],
+ },
+ value: { type: mongoose.Schema.Types.Mixed },
+ logic: {
+ type: String,
+ enum: ['AND', 'OR'],
+ },
+ conditions: [
+ {
+ type: mongoose.Schema.Types.Mixed, // 재귀적 구조를 위한 설정
+ },
+ ],
+ },
+ { _id: false },
+);
+
+export const ScheduleSchema = new mongoose.Schema({
+ cron: { type: String, required: true },
+ timezone: { type: String },
});
-export const TaxonomyChildItemSubSchema = new Schema(
+export const StageTriggerSchema = new mongoose.Schema(
{
- title: { type: String, required: true },
- order: { type: Number, required: true, default: 100 },
- slug: { type: String }, // slug는 없을 수도 있으므로 required 제외
- articleId: { type: Schema.Types.ObjectId, ref: 'article' },
+ type: {
+ type: String,
+ enum: ['always', 'condition', 'schedule'],
+ required: true,
+ },
+ condition: {
+ type: ConditionSchema,
+ required: function (this: StageTrigger) {
+ return this.type === 'condition';
+ },
+ },
+ schedule: {
+ type: ScheduleSchema,
+ required: function (this: StageTrigger) {
+ return this.type === 'schedule';
+ },
+ },
+ },
+ { _id: false },
+);
+
+// export const ArtifactSchema = new mongoose.Schema({
+// key: { type: String },
+// value: { type: mongoose.SchemaTypes.Mixed },
+// dataType: {
+// type: String,
+// enum: dataTypeEnum,
+// required: true,
+// },
+// isReserved: { type: Boolean },
+// });
+
+export const MetadataSubSchema = new mongoose.Schema(
+ {
+ key: { type: String, required: true },
+ value: {
+ type: mongoose.SchemaTypes.Mixed,
+ required: false, // required: true로 하면 value가 null인 경우 에러 발생
+ default: null,
+ validate: {
+ validator: function (v: string | any[]) {
+ // 특정 키인 경우 value가 null 또는 빈 문자열이 아니어야 함
+ // if (this.key === 'requiredKey') {
+ // return typeof v === 'string' && v.trim().length > 0;
+ // }
+ // 그렇지 않은 경우 value는 null 또는 문자열, Array이어야 함
+ return typeof v === 'string' || Array.isArray(v);
+ },
+ message: function (props: any) {
+ if (this.key === 'requiredKey') {
+ return `Value is required for key "${this.key}" and cannot be null or empty.`;
+ }
+ return `Value must be a string or array for key "${this.key}".`;
+ },
+ },
+ },
+ dataType: {
+ type: String,
+ required: false,
+ default: 'string',
+ enum: ['string', 'number', 'boolean', 'array', 'json'],
+ },
+ isReserved: { type: Boolean, required: false, default: false },
},
{
- _id: false, // 하위 아이템마다 _id가 생성되는 것을 방지 (선택 사항)
+ _id: false,
},
);
+// export const MANDATORY_ARTIFACT_KEYS = {
+// test: 'test',
+// } as const;
+
+// export type MandatoryArtifactKey =
+// (typeof MANDATORY_ARTIFACT_KEYS)[keyof typeof MANDATORY_ARTIFACT_KEYS];
+
+// class Artifacts extends Array {
+// constructor(artifacts: Artifact[] = []) {
+// // artifacts를 순회하며 dataType을 설정
+// const processedArtifacts = artifacts.map((artifact) => ({
+// ...artifact,
+// dataType: determineDataType(artifact.value),
+// }));
+// super(...processedArtifacts);
+// }
+
+// // 특정 key로 태그를 찾는 메서드 추가
+// private findArtifactByKey(key: string): Artifact | undefined {
+// return this.find((artifact) => artifact.key === key);
+// }
+
+// // getStringByKey(key: ReservedDeviceTagKey): string;
+// getStringByKey(key: string): string;
+// getStringByKey(key: MandatoryArtifactKey | string): string {
+// return this.findArtifactByKey(key)?.value ?? '';
+// }
+
+// // getValueByKey(key: ReservedDeviceTagKey): string | null;
+// getValueByKey(key: string): string | null;
+// getValueByKey(key: MandatoryArtifactKey | string): string | null {
+// return this.findArtifactByKey(key)?.value ?? null;
+// }
+// }
+
+export const MANDATORY_METADATA_KEYS = {
+ test: 'test',
+} as const;
+
+export type MandatoryMetadataKey =
+ (typeof MANDATORY_METADATA_KEYS)[keyof typeof MANDATORY_METADATA_KEYS];
+
+// class Metadatas extends Array {
+// constructor(metadatas: Metadata[] | mongoose.FilterQuery<'FindOne'>[] | []) {
+// // metadatas를 순회하며 dataType을 설정
+// // console.log(Object.prototype.toString.call(metadatas), 'metadatas의 프로토타입');
+// const processedMetadatas = metadatas.map((metadata) => ({
+// ...metadata,
+// dataType: determineDataType(metadata.dataType),
+// }));
+// // console.log(processedMetadatas, 'processedMetadatas');
+// super(...processedMetadatas);
+// }
+
+// // 특정 key로 태그를 찾는 메서드 추가
+// private findMetadataByKey(key: string): Metadata | undefined {
+// return this.find((metadata) => metadata.key === key);
+// }
+
+// // getStringByKey(key: ReservedDeviceTagKey): string;
+// getStringByKey(key: string): string;
+// getStringByKey(key: MandatoryMetadataKey | string): string {
+// console.log('key:', key);
+// console.log('value', this.findMetadataByKey(key));
+// return this.findMetadataByKey(key)?.value ?? '';
+// }
+
+// // getValueByKey(key: ReservedDeviceTagKey): string | null;
+// getValueByKey(key: string): T | null;
+// getValueByKey(key: MandatoryMetadataKey | string): T | null {
+// return this.findMetadataByKey(key)?.value ?? null;
+// }
+// }
+
+export const ExecutionSchema = new mongoose.Schema(
+ {
+ // executionId: { type: String, required: true },
+ status: {
+ type: String,
+ enum: [
+ 'pending',
+ 'in-progress',
+ 'completed',
+ 'failed',
+ 'cancelled',
+ 'skipped',
+ ],
+ required: true,
+ },
+ // artifact: { type: [ArtifactSchema], default: [] },
+ metadatas: {
+ type: [MetadataSubSchema],
+ required: true,
+ default: [],
+ validator: function (
+ v: string | any[] | number | boolean | object | null,
+ ) {
+ // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
+ const dataType = (this as Metadata).dataType;
+
+ if (v === null) {
+ return true; // null은 항상 허용 (required: false이므로)
+ }
+
+ switch (dataType) {
+ case 'string':
+ return typeof v === 'string';
+ case 'number':
+ return typeof v === 'number';
+ case 'boolean':
+ return typeof v === 'boolean';
+ case 'array':
+ return Array.isArray(v);
+ case 'json':
+ // JSON은 객체 형태일 수 있습니다.
+ return typeof v === 'object' && v !== null && !Array.isArray(v);
+ default:
+ // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
+ return typeof v === 'string';
+ }
+ },
+ message: function (props: any) {
+ const dataType = (this as Metadata).dataType;
+ return `Value "${props.value}" does not match the expected data type "${dataType}".`;
+ },
+ },
+ },
+ {
+ // _id: false, // 서브 문서로 사용되므로 _id 필드를 생성하지 않습니다.
+ timestamps: true,
+ // discriminatorKey: 'kind', // 'kind'는 원하는 키 이름으로 변경 가능합니다.
+ },
+);
+// ExecutionSchema.post('find', makeMetadatasInstanceHook);
+// ExecutionSchema.post('findOne', makeMetadatasInstanceHook);
+// ExecutionSchema.virtual('metadatas').get(function () {
+// return new Metadatas(this.toObject().metadata);
+// });
+
+export const VariableSchema = new mongoose.Schema({
+ name: { type: String, required: true },
+ value: { type: mongoose.Schema.Types.Mixed, required: true },
+ scope: { type: String, enum: ['pipeline', 'stage'], required: true },
+});
+
+export const RetryStrategySchema = new mongoose.Schema({
+ limit: { type: Number, required: true },
+ delay: { type: Number, required: true },
+ backoff: { type: Number, required: true },
+ when: { type: [String], required: true },
+ notify: { type: [String], required: true },
+});
+
+export const EventSourceSubSchema = new mongoose.Schema(
+ {
+ type: { type: String, required: true },
+ name: { type: String, required: true },
+ metadatas: {
+ type: [MetadataSubSchema],
+ required: true,
+ default: [],
+ validator: function (
+ v: string | any[] | number | boolean | object | null,
+ ) {
+ // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
+ const dataType = (this as Metadata).dataType;
+
+ if (v === null) {
+ return true; // null은 항상 허용 (required: false이므로)
+ }
+
+ switch (dataType) {
+ case 'string':
+ return typeof v === 'string';
+ case 'number':
+ return typeof v === 'number';
+ case 'boolean':
+ return typeof v === 'boolean';
+ case 'array':
+ return Array.isArray(v);
+ case 'json':
+ // JSON은 객체 형태일 수 있습니다.
+ return typeof v === 'object' && v !== null && !Array.isArray(v);
+ default:
+ // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
+ return typeof v === 'string';
+ }
+ },
+ message: function (props: any) {
+ const dataType = (this as Metadata).dataType;
+ return `Value "${props.value}" does not match the expected data type "${dataType}".`;
+ },
+ },
+ },
+ {
+ _id: false,
+ },
+);
+
+export const StagedefSchema = new mongoose.Schema(
+ {
+ apiVersion: { type: String, required: true },
+ eventSource: { type: [EventSourceSubSchema], required: true },
+ // lifecycleHooks: { type: LifecycleHooksSchema },
+ schema: { type: mongoose.Schema.Types.Mixed },
+ name: { type: String, required: true },
+ taxonomy: { type: String },
+ topic: { type: String, required: true },
+ metadatas: {
+ type: [MetadataSubSchema],
+ required: true,
+ default: [],
+ validator: function (
+ v: string | any[] | number | boolean | object | null,
+ ) {
+ // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
+ const dataType = (this as Metadata).dataType;
+
+ if (v === null) {
+ return true; // null은 항상 허용 (required: false이므로)
+ }
+
+ switch (dataType) {
+ case 'string':
+ return typeof v === 'string';
+ case 'number':
+ return typeof v === 'number';
+ case 'boolean':
+ return typeof v === 'boolean';
+ case 'array':
+ return Array.isArray(v);
+ case 'json':
+ // JSON은 객체 형태일 수 있습니다.
+ return typeof v === 'object' && v !== null && !Array.isArray(v);
+ default:
+ // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
+ return typeof v === 'string';
+ }
+ },
+ message: function (props: any) {
+ const dataType = (this as Metadata).dataType;
+ return `Value "${props.value}" does not match the expected data type "${dataType}".`;
+ },
+ },
+ },
+ {
+ timestamps: true,
+ },
+);
+// StagedefSchema.post('find', makeMetadatasInstanceHook);
+// StagedefSchema.post('findOne', makeMetadatasInstanceHook);
+// StagedefSchema.virtual('metadatas').get(function () {
+// return new Metadatas(this.toObject().metadata);
+// });
+
+export const StageSchema = new mongoose.Schema(
+ {
+ apiVersion: { type: String, required: true },
+ name: { type: String, required: true },
+ schema: { type: mongoose.Schema.Types.Mixed },
+ taxonomy: { type: String },
+ // lifecycleHooks: { type: LifecycleHooksSchema, default: {} },
+ executions: { type: [ExecutionSchema], default: [] },
+ // outputData: { type: mongoose.Schema.Types.Mixed, default: {} },
+ status: {
+ type: String,
+ enum: [
+ 'pending',
+ 'in-progress',
+ 'completed',
+ 'failed',
+ 'cancelled',
+ 'skipped',
+ ],
+ required: true,
+ },
+ retryStrategy: { type: RetryStrategySchema },
+ // artifact: { type: ArtifactSchema, default: {} },
+ metadatas: {
+ type: [MetadataSubSchema],
+ required: true,
+ default: [],
+ validator: function (
+ v: string | any[] | number | boolean | object | null,
+ ) {
+ // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
+ const dataType = (this as Metadata).dataType;
+
+ if (v === null) {
+ return true; // null은 항상 허용 (required: false이므로)
+ }
+
+ switch (dataType) {
+ case 'string':
+ return typeof v === 'string';
+ case 'number':
+ return typeof v === 'number';
+ case 'boolean':
+ return typeof v === 'boolean';
+ case 'array':
+ return Array.isArray(v);
+ case 'json':
+ // JSON은 객체 형태일 수 있습니다.
+ return typeof v === 'object' && v !== null && !Array.isArray(v);
+ default:
+ // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
+ return typeof v === 'string';
+ }
+ },
+ message: function (props: any) {
+ const dataType = (this as Metadata).dataType;
+ return `Value "${props.value}" does not match the expected data type "${dataType}".`;
+ },
+ },
+ // variables: { type: [VariableSchema], default: [] },
+ },
+ {
+ // 여기서 discriminatorKey 옵션을 추가합니다.
+ discriminatorKey: 'kind', // 'kind'는 원하는 키 이름으로 변경 가능합니다.
+ },
+);
+// StageSchema.post('find', makeMetadatasInstanceHook);
+// StageSchema.post('findOne', makeMetadatasInstanceHook);
+// StageSchema.virtual('metadatas').get(function () {
+// return new Metadatas(this.toObject().metadata);
+// });
+
+export const PipelinedefSchema = new mongoose.Schema({
+ apiVersion: { type: String, required: true },
+ trigger: { type: PipelineTriggerSchema, required: true },
+ stage: { type: [mongoose.Types.ObjectId], required: true },
+ // revision: { type: Number },
+ metadatas: {
+ type: [MetadataSubSchema],
+ required: true,
+ default: [],
+ validator: function (v: string | any[] | number | boolean | object | null) {
+ // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
+ const dataType = (this as Metadata).dataType;
+
+ if (v === null) {
+ return true; // null은 항상 허용 (required: false이므로)
+ }
+
+ switch (dataType) {
+ case 'string':
+ return typeof v === 'string';
+ case 'number':
+ return typeof v === 'number';
+ case 'boolean':
+ return typeof v === 'boolean';
+ case 'array':
+ return Array.isArray(v);
+ case 'json':
+ // JSON은 객체 형태일 수 있습니다.
+ return typeof v === 'object' && v !== null && !Array.isArray(v);
+ default:
+ // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
+ return typeof v === 'string';
+ }
+ },
+ message: function (props: any) {
+ const dataType = (this as Metadata).dataType;
+ return `Value "${props.value}" does not match the expected data type "${dataType}".`;
+ },
+ },
+});
+// PipelinedefSchema.post('find', makeMetadatasInstanceHook);
+// PipelinedefSchema.post('findOne', makeMetadatasInstanceHook);
+// PipelinedefSchema.virtual('metadatas').get(function () {
+// return new Metadatas(this.toObject().metadata);
+// });
+
+export const PipelineSchema = new mongoose.Schema(
+ {
+ apiVersion: { type: String, required: true },
+ name: { type: String, required: false },
+ // revision: { type: Number, required: true },
+ stage: [{ type: StageSchema, required: true }],
+ metadatas: {
+ type: [MetadataSubSchema],
+ required: true,
+ default: [],
+ validator: function (
+ v: string | any[] | number | boolean | object | null,
+ ) {
+ // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
+ const dataType = (this as Metadata).dataType;
+
+ if (v === null) {
+ return true; // null은 항상 허용 (required: false이므로)
+ }
+
+ switch (dataType) {
+ case 'string':
+ return typeof v === 'string';
+ case 'number':
+ return typeof v === 'number';
+ case 'boolean':
+ return typeof v === 'boolean';
+ case 'array':
+ return Array.isArray(v);
+ case 'json':
+ // JSON은 객체 형태일 수 있습니다.
+ return typeof v === 'object' && v !== null && !Array.isArray(v);
+ default:
+ // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
+ return typeof v === 'string';
+ }
+ },
+ message: function (props: any) {
+ const dataType = (this as Metadata).dataType;
+ return `Value "${props.value}" does not match the expected data type "${dataType}".`;
+ },
+ },
+ },
+ {
+ timestamps: true,
+ },
+);
+// PipelineSchema.post('find', makeMetadatasInstanceHook);
+// PipelineSchema.post('findOne', makeMetadatasInstanceHook);
+// PipelineSchema.virtual('metadatas').get(function () {
+// return new Metadatas(this.toObject().metadata);
+// });
+
+// StageSchema.index({ status: 1 });
+
+export const AIModelDefinitions: ModelDefinition[] = [
+ { name: 'pipelinedefs', schema: PipelinedefSchema },
+ { name: 'pipelines', schema: PipelineSchema },
+ { name: 'stagedefs', schema: StagedefSchema },
+ { name: 'stages', schema: StageSchema },
+];
+
+export interface AIModelTypes {
+ pipelinedefs: Pipelinedef;
+ pipelines: Pipeline;
+ stagedefs: Stagedef;
+ stages: Stage;
+}
+
+// async function makeArtifactsInstanceHook(
+// res: any,
+// next: CallbackWithoutResultAndOptionalError,
+// ) {
+// if (Array.isArray(res)) {
+// for (let doc of res) {
+// if (doc.tags && Array.isArray(doc.tags)) {
+// // 수동으로 배열의 프로토타입을 Tags로 변환
+// doc.tags = new Artifacts(doc.tags);
+// Object.setPrototypeOf(doc.tags, Artifacts.prototype);
+// console.log('doc.tags prototype:', Object.getPrototypeOf(doc.tags));
+// console.log('doc.tags constructor name:', doc.tags.constructor.name);
+// if (doc.tags instanceof Artifacts) {
+// console.log('instanceof doc.tags', doc.tags);
+// } else {
+// console.log('Not an instance of Tags');
+// }
+// }
+// }
+// } else {
+// if (res && res.tags && Array.isArray(res.tags)) {
+// res.tags = new Artifacts(res.tags);
+// Object.setPrototypeOf(res.tags, Artifacts.prototype);
+// console.log('res.tags prototype:', Object.getPrototypeOf(res.tags));
+// console.log('res.tags constructor name:', res.tags.constructor.name);
+// if (res.tags instanceof Artifacts) {
+// console.log('instanceof res.tags', res.tags);
+// } else {
+// console.log('Not an instance of Tags');
+// }
+// }
+// }
+// next();
+// }
+
+// async function makeMetadatasInstanceHook(
+// res: any,
+// next: CallbackWithoutResultAndOptionalError,
+// ) {
+// const visited = new WeakSet();
+
+// function instantiateMetadatas(doc: any) {
+// if (doc && typeof doc === 'object') {
+// if (visited.has(doc)) {
+// return;
+// }
+// visited.add(doc);
+
+// for (const key in doc) {
+// if (doc.hasOwnProperty(key)) {
+// const value = doc[key];
+// if (key === 'metadata' && Array.isArray(value)) {
+// // Instantiate metadata arrays into Metadatas instances
+// doc[key] = new Metadatas(value);
+// Object.setPrototypeOf(doc[key], Metadatas.prototype);
+// } else if (Array.isArray(value)) {
+// // Recursively handle arrays
+// for (let item of value) {
+// instantiateMetadatas(item);
+// }
+// } else if (typeof value === 'object' && value !== null) {
+// // Recursively handle nested objects
+// instantiateMetadatas(value);
+// }
+// }
+// }
+// }
+// }
+
+// if (Array.isArray(res)) {
+// for (let doc of res) {
+// instantiateMetadatas(doc);
+// }
+// } else {
+// instantiateMetadatas(res);
+// }
+// next();
+// }
+
+// export function determineDataType(value: any): DataType {
+// if (typeof value === 'string') {
+// return 'string';
+// } else if (typeof value === 'number') {
+// return 'number';
+// } else if (typeof value === 'boolean') {
+// return 'boolean';
+// } else if (Array.isArray(value)) {
+// // 배열 내 모든 요소가 문자열인지 확인
+// if (value.every((item) => typeof item === 'string')) {
+// return 'array';
+// }
+// return 'mixed'; // 다양한 타입이 섞인 배열
+// } else if (typeof value === 'object' && value !== null) {
+// return 'mixed';
+// }
+// return 'mixed'; // 기본값
+// }
+
+// export const TaxonomyChildItemSubSchema = new Schema(
+// {
+// title: { type: String, required: true },
+// order: { type: Number, required: true, default: 100 },
+// slug: { type: String }, // slug는 없을 수도 있으므로 required 제외
+// articleId: { type: Schema.Types.ObjectId, ref: 'article' },
+// },
+// {
+// _id: false, // 하위 아이템마다 _id가 생성되는 것을 방지 (선택 사항)
+// },
+// );
+
// **중요: 재귀 참조 설정**
// 스키마 정의 후 .add()를 통해 자기 자신을 children 배열로 추가합니다.
-TaxonomyChildItemSubSchema.add({
- children: { type: [TaxonomyChildItemSubSchema], default: [] },
-});
+// TaxonomyChildItemSubSchema.add({
+// children: { type: [TaxonomyChildItemSubSchema], default: [] },
+// });
-// 2. 메인 도큐먼트 스키마 (Root)
-export const TaxonomySchema = new Schema(
+// // 2. 메인 도큐먼트 스키마 (Root)
+// export const TaxonomySchema = new Schema(
+// {
+// // items 배열은 위에서 정의한 재귀 스키마를 따름
+// label: { type: String, required: true },
+// items: {
+// type: [TaxonomyChildItemSubSchema],
+// default: [],
+// },
+
+// // metadata는 구조가 가변적이므로 Mixed 타입 사용
+// metadatas: {
+// type: [MetadataSubSchema],
+// required: true,
+// default: [],
+// validator: function (
+// v: string | any[] | number | boolean | object | null,
+// ) {
+// // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
+// const dataType = (this as Metadata).dataType;
+
+// if (v === null) {
+// return true; // null은 항상 허용 (required: false이므로)
+// }
+
+// switch (dataType) {
+// case 'string':
+// return typeof v === 'string';
+// case 'number':
+// return typeof v === 'number';
+// case 'boolean':
+// return typeof v === 'boolean';
+// case 'array':
+// return Array.isArray(v);
+// case 'json':
+// // JSON은 객체 형태일 수 있습니다.
+// return typeof v === 'object' && v !== null && !Array.isArray(v);
+// default:
+// // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
+// return typeof v === 'string';
+// }
+// },
+// message: function (props: any) {
+// const dataType = (this as Metadata).dataType;
+// return `Value "${props.value}" does not match the expected data type "${dataType}".`;
+// },
+// },
+// },
+// {
+// timestamps: true, // createAt, updatedAt 자동 생성
+// collection: 'taxonomy', // 컬렉션 이름 지정
+// },
+// );
+
+// TaxonomySchema.index({ label: 1 }, { unique: true });
+
+// export const ContentSectionEmbeddedTaxonomySubSchema =
+// new Schema(
+// {
+// taxonomyId: {
+// type: Schema.Types.ObjectId,
+// ref: 'taxonomy',
+// required: true,
+// },
+// label: { type: String, required: true },
+// },
+// { _id: false },
+// );
+
+// export const ContentSectionSubSchema = new Schema(
+// {
+// headline: { type: String, required: true },
+// taxonomies: {
+// type: [ContentSectionEmbeddedTaxonomySubSchema],
+// required: true,
+// default: [],
+// },
+// },
+// { _id: false },
+// );
+
+// export const ContentSchema = new Schema(
+// {
+// key: { type: String, required: true, unique: true },
+// label: { type: String, required: true },
+// sections: {
+// type: [ContentSectionSubSchema],
+// default: [],
+// },
+// },
+// {
+// timestamps: true,
+// collection: 'content',
+// },
+// );
+
+// export const ArticleSchema = new Schema(
+// {
+// title: { type: String, required: true },
+// content: { type: String, required: true },
+// slug: { type: String, required: true, unique: true },
+// status: {
+// type: String,
+// enum: ['draft', 'published', 'archived'],
+// default: 'draft',
+// },
+// metadatas: {
+// type: [MetadataSubSchema],
+// required: true,
+// default: [],
+// validate: {
+// validator: function (metas: Metadata[]) {
+// const keys = metas.map((m) => m.key);
+// return keys.length === new Set(keys).size;
+// },
+// message: 'Duplicate metadata keys are not allowed.',
+// },
+// },
+// },
+// {
+// timestamps: true,
+// collection: 'article',
+// },
+// );
+
+// ArticleSchema.index({ title: 'text', content: 'text' });
+
+export const CrawlerModelDefinitions: ModelDefinition[] = [
{
- // items 배열은 위에서 정의한 재귀 스키마를 따름
- label: { type: String, required: true },
- items: {
- type: [TaxonomyChildItemSubSchema],
- default: [],
- },
-
- // metadata는 구조가 가변적이므로 Mixed 타입 사용
- metadatas: {
- type: [MetadataSubSchema],
- required: true,
- default: [],
- validator: function (v: string | any[] | number | boolean | object | null) {
- // 'this'는 현재 검증 중인 Metadata 서브 도큐먼트를 가리킵니다.
- const dataType = (this as Metadata).dataType;
-
- if (v === null) {
- return true; // null은 항상 허용 (required: false이므로)
- }
-
- switch (dataType) {
- case 'string':
- return typeof v === 'string';
- case 'number':
- return typeof v === 'number';
- case 'boolean':
- return typeof v === 'boolean';
- case 'array':
- return Array.isArray(v);
- case 'json':
- // JSON은 객체 형태일 수 있습니다.
- return typeof v === 'object' && v !== null && !Array.isArray(v);
- default:
- // 'string'이 기본값이므로 이쪽으로 빠지는 경우는 드물지만, 방어 코드
- return typeof v === 'string';
- }
- },
- message: function (props: any) {
- const dataType = (this as Metadata).dataType;
- return `Value "${props.value}" does not match the expected data type "${dataType}".`;
- },
- },
+ name: 'pipelinedefs',
+ schema: PipelinedefSchema,
+ schemaName: 'CrawlerPipelinedef',
},
- {
- timestamps: true, // createAt, updatedAt 자동 생성
- collection: 'taxonomy', // 컬렉션 이름 지정
- },
-);
-
-TaxonomySchema.index({ label: 1 }, { unique: true });
-
-export const ContentSectionEmbeddedTaxonomySubSchema = new Schema(
- {
- taxonomyId: { type: Schema.Types.ObjectId, ref: 'taxonomy', required: true },
- label: { type: String, required: true },
- },
- { _id: false },
-);
-
-export const ContentSectionSubSchema = new Schema(
- {
- headline: { type: String, required: true },
- taxonomies: {
- type: [ContentSectionEmbeddedTaxonomySubSchema],
- required: true,
- default: [],
- },
- },
- { _id: false },
-);
-
-export const ContentSchema = new Schema(
- {
- key: { type: String, required: true, unique: true },
- label: { type: String, required: true },
- sections: {
-
- type: [
- ContentSectionSubSchema
- ],
- default: [],
- },
- },
- {
- timestamps: true,
- collection: 'content',
- },
-);
-
-export const ArticleSchema = new Schema(
- {
- title: { type: String, required: true },
- content: { type: String, required: true },
- slug: { type: String, required: true, unique: true },
- status: {
- type: String,
- enum: ['draft', 'published', 'archived'],
- default: 'draft',
- },
- metadatas: {
- type: [MetadataSubSchema],
- required: true,
- default: [],
- validate: {
- validator: function (metas: Metadata[]) {
- const keys = metas.map((m) => m.key);
- return keys.length === new Set(keys).size;
- },
- message: 'Duplicate metadata keys are not allowed.',
- },
- },
- },
- {
- timestamps: true,
- collection: 'article',
- },
-);
-
-ArticleSchema.index({ title: 'text', content: 'text' });
-
-export const moeumModelDefinitions: ModelDefinition[] = [
- { name: 'taxonomy', schema: TaxonomySchema, schemaName: 'taxonomy' },
- { name: 'content', schema: ContentSchema, schemaName: 'content' },
- { name: 'article', schema: ArticleSchema, schemaName: 'article' },
+ { name: 'pipelines', schema: PipelineSchema, schemaName: 'CrawlerPipeline' },
+ { name: 'stagedefs', schema: StagedefSchema, schemaName: 'CrawlerStagedef' },
+ // { name: 'stages', schema: StageSchema, schemaName: 'CrawlerStage' },
];
-export interface MoeumModelTypes {
- taxonomy: Taxonomy;
- content: Content;
- article: Article;
+export interface CrawlerModelTypes {
+ pipelinedefs: Pipelinedef;
+ pipelines: Pipeline;
+ stagedefs: Stagedef;
+ // stages: Stage;
}