Implements a comprehensive web crawler microservice

Introduces a complete NestJS-based web crawler microservice. This release establishes a robust system for web data extraction through a flexible Pipeline/Stage architecture.

Key features include:
- Supports both static (axios, cheerio) and dynamic (Playwright) page crawling.
- Implements diverse triggering options: cron-based scheduling, NATS event-driven, and manual API calls.
- Enhances system resilience with domain-specific rate limiting and configurable exponential backoff retry mechanisms.
- Provides dedicated API endpoints and database schemas for managing pipelines, stages, and execution history.
- Ensures data persistence by saving crawl results to MinIO.

The `README.md` is entirely rewritten to provide comprehensive documentation on features, architecture, setup, API usage, and troubleshooting. Core dependencies for crawling, scheduling, and API validation are added, and database models are refactored to align with the new crawler system's data structures.
This commit is contained in:
dahyeon committed 2025-12-21 14:29:25 +00:00
1 parent 60f8e249fa
commit a0d218a169
26 files changed
+4302 -216

No files matched your search

+744 -68
View File
@@ -1,98 +1,774 @@
<p align="center"> # MSA 기반 웹 크롤러 서비스
<a href="http://nestjs.com/" target="blank"><img src="https://nestjs.com/img/logo-small.svg" width="120" alt="Nest Logo" /></a>
</p>
[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456 NestJS 기반의 마이크로서비스 아키텍처 웹 크롤러 시스템입니다. 정적/동적 웹페이지 크롤링, Pipeline/Stage 기반 병렬 실행, 다중 트리거 지원, Rate Limiting 등의 기능을 제공합니다.
[circleci-url]: https://circleci.com/gh/nestjs/nest
<p align="center">A progressive <a href="http://nodejs.org" target="_blank">Node.js</a> framework for building efficient and scalable server-side applications.</p> ## 주요 기능
<p align="center">
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/v/@nestjs/core.svg" alt="NPM Version" /></a>
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/l/@nestjs/core.svg" alt="Package License" /></a>
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/dm/@nestjs/common.svg" alt="NPM Downloads" /></a>
<a href="https://circleci.com/gh/nestjs/nest" target="_blank"><img src="https://img.shields.io/circleci/build/github/nestjs/nest/master" alt="CircleCI" /></a>
<a href="https://discord.gg/G7Qnnhy" target="_blank"><img src="https://img.shields.io/badge/discord-online-brightgreen.svg" alt="Discord"/></a>
<a href="https://opencollective.com/nest#backer" target="_blank"><img src="https://opencollective.com/nest/backers/badge.svg" alt="Backers on Open Collective" /></a>
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://opencollective.com/nest/sponsors/badge.svg" alt="Sponsors on Open Collective" /></a>
<a href="https://paypal.me/kamilmysliwiec" target="_blank"><img src="https://img.shields.io/badge/Donate-PayPal-ff3f59.svg" alt="Donate us"/></a>
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://img.shields.io/badge/Support%20us-Open%20Collective-41B883.svg" alt="Support us"></a>
<a href="https://twitter.com/nestframework" target="_blank"><img src="https://img.shields.io/twitter/follow/nestframework.svg?style=social&label=Follow" alt="Follow us on Twitter"></a>
</p>
<!--[![Backers on Open Collective](https://opencollective.com/nest/backers/badge.svg)](https://opencollective.com/nest#backer)
[![Sponsors on Open Collective](https://opencollective.com/nest/sponsors/badge.svg)](https://opencollective.com/nest#sponsor)-->
## Description ### 크롤링 엔진
- **정적 페이지 크롤링**: axios + cheerio 기반 HTML 파싱
- **동적 SPA 크롤링**: Playwright 기반 JavaScript 렌더링 지원
- **셀렉터 기반 데이터 추출**: CSS 셀렉터로 원하는 데이터만 추출
- **속성 추출**: 이미지 URL 등 HTML 속성 추출 지원
[Nest](https://github.com/nestjs/nest) framework TypeScript starter repository. ### Pipeline/Stage 아키텍처
- **병렬 실행**: 여러 Stage를 동시에 실행하여 성능 최적화
- **실행 기록**: 각 Stage의 실행 이력 및 결과 저장
- **메타데이터 기반**: 유연한 설정 및 확장 가능한 구조
## Project setup ### 트리거 시스템
- **스케줄 트리거**: Cron 표현식 기반 정기 실행
- **이벤트 트리거**: NATS 메시지 기반 이벤트 드리븐 실행
- **API 트리거**: REST API 호출로 수동 실행
```bash ### 안정성 및 성능
$ npm install - **Rate Limiting**: 도메인별 토큰 버킷 알고리즘으로 요청 제한
- **재시도 로직**: Exponential backoff 기반 자동 재시도
- **에러 알림**: 실패 시 NATS를 통한 알림 발송
- **결과 저장**: MinIO에 크롤링 결과 자동 저장
## 기술 스택
### 프레임워크 및 런타임
- **NestJS 11**: TypeScript 기반 서버 프레임워크
- **Node.js**: JavaScript 런타임
### 크롤링 및 파싱
- **axios**: HTTP 클라이언트
- **cheerio**: 서버사이드 jQuery (HTML 파싱)
- **playwright**: 브라우저 자동화 (동적 페이지 크롤링)
### 데이터베이스 및 스토리지
- **MongoDB + Mongoose**: NoSQL 데이터베이스
- **MinIO**: S3 호환 객체 스토리지
### 메시징 및 이벤트
- **NATS**: 메시지 브로커 (이벤트 트리거 및 알림)
### 스케줄링 및 인증
- **@nestjs/schedule + cron**: 작업 스케줄링
- **JWT (jose)**: 토큰 기반 인증 (Microsoft Entra ID 지원)
- **Infisical**: 중앙화된 비밀값 관리
## 아키텍처
```
┌─────────────────────────────────────────────────────────────┐
│ REST API │
│ (JWT Authentication Required) │
└────────────────────────┬────────────────────────────────────┘
│
┌────────────────┼────────────────┐
│ │ │
┌────▼─────┐ ┌──────▼──────┐ ┌────▼─────┐
│ Cron │ │ NATS Event │ │ API │
│ Trigger │ │ Trigger │ │ Trigger │
└────┬─────┘ └──────┬──────┘ └────┬─────┘
│ │ │
└────────────────┼────────────────┘
│
┌──────────▼──────────┐
│ Pipeline Engine │
│ (병렬 실행 관리) │
└──────────┬──────────┘
│
┌────────────────┼────────────────┐
│ │ │
┌────▼─────┐ ┌──────▼──────┐ ┌────▼─────┐
│ Stage 1 │ │ Stage 2 │ │ Stage 3 │
│(EventSrc)│ │ (EventSrc) │ │(EventSrc)│
└────┬─────┘ └──────┬──────┘ └────┬─────┘
│ │ │
┌────▼─────┐ ┌──────▼──────┐ ┌────▼─────┐
│ Crawler │ │ Crawler │ │ Crawler │
│ Service │ │ Service │ │ Service │
└────┬─────┘ └──────┬──────┘ └────┬─────┘
│ │ │
└────────────────┼────────────────┘
│
┌──────────▼──────────┐
│ MinIO Storage │
│ (크롤링 결과 저장) │
└─────────────────────┘
``` ```
## Compile and run the project ## 설치
### 필수 요구사항
- **Node.js**: 18.x 이상
- **MongoDB**: 4.x 이상
- **NATS**: 2.x 이상
- **MinIO**: S3 호환 객체 스토리지
- **Infisical**: 비밀값 관리 (선택)
### 1. 저장소 클론 및 패키지 설치
```bash ```bash
# development git clone https://git.dahyeon.us/dahyeon/crawler.git
$ npm run start cd crawler
npm install
# watch mode
$ npm run start:dev
# production mode
$ npm run start:prod
``` ```
## Run tests ### 2. Playwright 브라우저 설치
```bash ```bash
# unit tests npx playwright install chromium
$ npm run test
# e2e tests
$ npm run test:e2e
# test coverage
$ npm run test:cov
``` ```
## Deployment ### 3. 환경 변수 설정
When you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the [deployment documentation](https://docs.nestjs.com/deployment) for more information. 환경별 `.env` 파일을 생성합니다:
If you are looking for a cloud-based platform to deploy your NestJS application, check out [Mau](https://mau.nestjs.com), our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps:
```bash ```bash
$ npm install -g @nestjs/mau # 개발 환경
$ mau deploy cp env-vars/example.env .env.dev
# 테스트 환경
cp env-vars/example.env .env.test
# 프로덕션 환경
cp env-vars/example.env .env.prod
``` ```
With Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure. #### 필수 환경 변수 (Infisical 사용 시)
## Resources ```env
# Infisical 설정
INFISICAL_SITE_URL=https://app.infisical.com
INFISICAL_CLIENT_ID=your-client-id
INFISICAL_CLIENT_SECRET=your-client-secret
INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
```
Check out a few resources that may come in handy when working with NestJS: Infisical을 통해 다음 값들을 관리:
- `MONGODB_CONNECTION_STRING`: MongoDB 연결 문자열
- `NATS_ENDPOINT`: NATS 서버 주소
- `NATS_USER`: NATS 사용자명
- `NATS_PASS`: NATS 비밀번호
- `JWT_CLAIM_AUDIENCE`: JWT 검증 audience
- `MICROSOFT_ENTRAID_TENANT_ID`: Azure AD 테넌트 ID
- `MINIO_ENDPOINT`: MinIO 서버 주소
- `MINIO_ACCESS_KEY`: MinIO 액세스 키
- `MINIO_SECRET_KEY`: MinIO 시크릿 키
- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework. ### 4. TLS 인증서 (선택)
- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy).
- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/).
- Deploy your application to AWS with the help of [NestJS Mau](https://mau.nestjs.com) in just a few clicks.
- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com).
- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com).
- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs).
- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com).
## Support NATS TLS 연결을 위해 `internal-service-ca.pem` 파일을 프로젝트 루트에 배치합니다.
Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support). ## 실행
## Stay in touch ### 개발 모드
- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec) ```bash
- Website - [https://nestjs.com](https://nestjs.com/) npm run start:dev
- Twitter - [@nestframework](https://twitter.com/nestframework) ```
## License ### 프로덕션 빌드
Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE). ```bash
npm run build
npm run start:prod
```
### 환경 전환
```bash
# 테스트 환경
NODE_ENV=test npm run start:dev
# 프로덕션 환경
NODE_ENV=prod npm run start:prod
```
## REST API
서버 실행 후 Swagger 문서: `http://localhost:3000/api`
### 인증
모든 API는 JWT 인증이 필요합니다 (일부 Public 엔드포인트 제외).
```http
Authorization: Bearer {JWT_TOKEN}
```
### Pipelinedef 관리
#### 1. Pipelinedef 생성
```http
POST /api/v1/crawler/pipelinedefs
Content-Type: application/json
{
"apiVersion": "v1",
"trigger": {
"type": "always"
},
"stageIds": ["stage-id-1", "stage-id-2"],
"schedule": {
"cron": "0 0 * * *",
"timezone": "Asia/Seoul"
}
}
```
#### 2. Pipelinedef 목록 조회
```http
GET /api/v1/crawler/pipelinedefs?page=1&limit=20
```
#### 3. Pipelinedef 상세 조회
```http
GET /api/v1/crawler/pipelinedefs/{id}
```
#### 4. Pipelinedef 수정
```http
PUT /api/v1/crawler/pipelinedefs/{id}
Content-Type: application/json
{
"schedule": {
"cron": "0 12 * * *"
}
}
```
#### 5. Pipelinedef 삭제
```http
DELETE /api/v1/crawler/pipelinedefs/{id}
```
### Stagedef 관리
#### 1. Stagedef 생성
```http
POST /api/v1/crawler/stagedefs
Content-Type: application/json
{
"apiVersion": "v1",
"name": "Example Crawler",
"topic": "crawler.results.example",
"eventSource": [
{
"type": "web",
"name": "example-source",
"metadatas": [
{
"key": "url",
"value": "https://example.com",
"dataType": "string"
},
{
"key": "selectors",
"value": "{\"title\":\"h1\",\"content\":\".main\"}",
"dataType": "json"
},
{
"key": "dynamic",
"value": "false",
"dataType": "string"
}
]
}
]
}
```
#### 2. Stagedef 목록 조회
```http
GET /api/v1/crawler/stagedefs?page=1&limit=20
```
### Pipeline 실행
#### 1. Pipeline 수동 실행
```http
POST /api/v1/crawler/pipelines/trigger
Content-Type: application/json
{
"pipelineDefId": "pipeline-def-id",
"variables": {
"customVar": "value"
}
}
```
#### 2. 실행된 Pipeline 목록
```http
GET /api/v1/crawler/pipelines?page=1&limit=20&status=completed
```
**status 옵션**: `completed`, `failed`, `partial`
#### 3. Pipeline 실행 상세
```http
GET /api/v1/crawler/pipelines/{pipelineId}
```
#### 4. Pipeline의 Stage 목록
```http
GET /api/v1/crawler/pipelines/{pipelineId}/stages
```
### 모니터링
#### 1. 등록된 스케줄 목록
```http
GET /api/v1/crawler/schedules
```
#### 2. Rate Limit 상태
```http
GET /api/v1/crawler/rate-limits
```
#### 3. 트리거 재로드
```http
POST /api/v1/crawler/triggers/reload
```
#### 4. Health Check (Public)
```http
GET /api/v1/crawler/health
```
## 사용 예시
### 1. 정적 웹페이지 크롤링
```json
{
"type": "web",
"name": "news-crawler",
"metadatas": [
{
"key": "url",
"value": "https://news.example.com/latest",
"dataType": "string"
},
{
"key": "selectors",
"value": "{\"title\":\"h2.article-title\",\"summary\":\".article-summary\",\"author\":\".author-name\"}",
"dataType": "json"
},
{
"key": "dynamic",
"value": "false",
"dataType": "string"
}
]
}
```
### 2. 동적 SPA 페이지 크롤링
```json
{
"type": "web",
"name": "spa-crawler",
"metadatas": [
{
"key": "url",
"value": "https://spa-app.example.com/products",
"dataType": "string"
},
{
"key": "selectors",
"value": "{\"products\":\".product-item\",\"price\":\".price\"}",
"dataType": "json"
},
{
"key": "dynamic",
"value": "true",
"dataType": "string"
},
{
"key": "waitSelector",
"value": ".products-loaded",
"dataType": "string"
}
]
}
```
### 3. 이미지 URL 추출
```json
{
"type": "web",
"name": "image-crawler",
"metadatas": [
{
"key": "url",
"value": "https://gallery.example.com",
"dataType": "string"
},
{
"key": "selectors",
"value": "{\"title\":\"h1.gallery-title\"}",
"dataType": "json"
},
{
"key": "extractAttributes",
"value": "{\"imageUrl\":\"img.gallery-image@src\",\"thumbnail\":\"img.thumb@data-src\"}",
"dataType": "json"
},
{
"key": "dynamic",
"value": "false",
"dataType": "string"
}
]
}
```
### 4. 매일 자정 실행하는 스케줄 설정
```json
{
"apiVersion": "v1",
"trigger": {
"type": "always"
},
"stageIds": ["stage-id-1"],
"schedule": {
"cron": "0 0 * * *",
"timezone": "Asia/Seoul"
}
}
```
### 5. NATS 이벤트로 트리거
```json
{
"apiVersion": "v1",
"trigger": {
"type": "kafka-event",
"kafkaEvent": {
"topic": "crawler.trigger.news"
}
},
"stageIds": ["stage-id-1"]
}
```
## 프로젝트 구조
```
src/
├── main.ts # 애플리케이션 진입점
├── app.module.ts # 루트 모듈
│
├── global/ # 글로벌 공유 모듈
│ ├── auth/ # JWT 인증
│ │ ├── jwt.service.ts
│ │ ├── jwt.guard.ts
│ │ └── decorators...
│ ├── secret/ # Infisical 비밀값 관리
│ │ └── secret.service.ts
│ ├── database/ # MongoDB 연결
│ │ ├── connection.service.ts
│ │ ├── schema.ts # Mongoose 스키마
│ │ └── types.d.ts
│ ├── nats/ # NATS 메시징
│ │ └── nats.service.ts
│ └── minio/ # MinIO 파일 저장소
│ └── minio.service.ts
│
└── crawler/ # 크롤러 비즈니스 로직
├── crawler.module.ts
├── crawler.controller.ts # REST API 컨트롤러
│
├── services/
│ ├── crawler.service.ts # HTTP + Playwright 크롤링
│ ├── rate-limiter.service.ts # Rate Limiting
│ ├── retry.service.ts # 재시도 로직
│ ├── stage-executor.service.ts # Stage 실행
│ ├── pipeline-engine.service.ts # Pipeline 관리
│ └── trigger.service.ts # 트리거 시스템
│
├── dto/
│ ├── common.dto.ts
│ ├── create-pipelinedef.dto.ts
│ ├── create-stagedef.dto.ts
│ └── trigger-pipeline.dto.ts
│
└── interfaces/
└── crawler-result.interface.ts
```
## 데이터베이스 스키마
### Pipelinedef
파이프라인 정의 (재사용 가능한 템플릿)
```typescript
{
apiVersion: string; // "v1"
trigger: PipelineTrigger; // always | kafka-event
stage: ObjectId[]; // Stagedef ID 배열
metadatas: Metadata[]; // 커스텀 메타데이터
}
```
### Pipeline
실행된 파이프라인 인스턴스
```typescript
{
apiVersion: string;
name: string; // "pipeline-1234567890"
stage: Stage[]; // 실행 중인 Stage 배열
metadatas: Metadata[]; // 실행 정보 (triggeredBy, status 등)
createdAt: Date;
updatedAt: Date;
}
```
### Stagedef
스테이지 정의
```typescript
{
apiVersion: string;
name: string;
taxonomy?: string;
eventSource: EventSource[]; // 크롤링 대상 설정
topic: string; // NATS 발행 토픽
metadatas: Metadata[];
}
```
### Stage
실행된 스테이지
```typescript
{
apiVersion: string;
name: string;
executions: Execution[]; // 실행 기록
status: "pending" | "in-progress" | "completed" | "failed" | "cancelled" | "skipped";
metadatas: Metadata[];
}
```
### Execution
개별 실행 기록
```typescript
{
status: "pending" | "in-progress" | "completed" | "failed" | "cancelled" | "skipped";
metadatas: Metadata[]; // crawlResult, storageKey 등
createdAt: Date;
updatedAt: Date;
}
```
## Rate Limiting
### 도메인별 제한 설정
```typescript
// 기본값: 10 req, 초당 2개 보충
rateLimiter.setLimit('example.com', {
maxTokens: 20, // 최대 20개 토큰
refillRate: 5 // 초당 5개 보충
});
```
### 현재 상태 조회
```http
GET /api/v1/crawler/rate-limits
```
응답:
```json
{
"example.com": {
"config": {
"maxTokens": 20,
"refillRate": 5
},
"available": 15
}
}
```
## 재시도 전략
### RetryStrategy 설정
```typescript
{
limit: 3, // 최대 3회 재시도
delay: 1000, // 초기 대기 시간 1초
backoff: 2, // 2배씩 증가 (1초 → 2초 → 4초)
when: [
"failed", // 모든 실패
"5xx", // 5xx 에러
"network" // 네트워크 에러
],
notify: [
"admin-channel" // 실패 시 알림 채널
]
}
```
## NATS 이벤트
### 발행되는 이벤트
#### Pipeline 시작
```
Topic: crawler.pipeline.started
Payload: {
pipelineId: string;
pipelineDefId: string;
triggeredBy: "api" | "schedule" | "nats";
timestamp: string;
}
```
#### Pipeline 완료
```
Topic: crawler.pipeline.completed
Payload: {
pipelineId: string;
status: "completed" | "failed" | "partial";
triggeredBy: "api" | "schedule" | "nats";
stageResults: Array<{
stageName: string;
status: string;
error?: string;
}>;
completedAt: string;
}
```
#### Stage 시작
```
Topic: crawler.stage.started
Payload: {
pipelineId: string;
stageIndex: number;
stageName: string;
timestamp: string;
}
```
#### Stage 완료 (커스텀 토픽)
```
Topic: {stagedef.topic}
Payload: {
pipelineId: string;
stageIndex: number;
stageName: string;
status: "completed" | "failed";
results: Array<{
url: string;
success: boolean;
parsedData: object;
}>;
timestamp: string;
}
```
## 개발
### 테스트
```bash
# 단위 테스트
npm run test
# E2E 테스트
npm run test:e2e
# 커버리지
npm run test:cov
```
### 린트
```bash
npm run lint
npm run format
```
### 빌드
```bash
npm run build
```
## 트러블슈팅
### Playwright 브라우저 설치 실패
```bash
# 수동 설치
npx playwright install-deps
npx playwright install chromium
```
### MongoDB 연결 실패
- `MONGODB_CONNECTION_STRING` 환경 변수 확인
- MongoDB 서버 상태 확인
- 네트워크 연결 및 방화벽 설정 확인
### NATS 연결 실패
- `NATS_ENDPOINT`, `NATS_USER`, `NATS_PASS` 확인
- TLS 인증서 경로 확인 (`internal-service-ca.pem`)
- NATS 서버 상태 확인
### Rate Limit 초과
```typescript
// 도메인별 제한 조정
await fetch('http://localhost:3000/api/v1/crawler/rate-limits');
// 또는 코드에서 직접 설정
rateLimiter.setLimit('slow-domain.com', { maxTokens: 5, refillRate: 1 });
```
## 보안 고려사항
1. **JWT 인증**: 모든 API 엔드포인트는 JWT 인증 필요
2. **비밀값 관리**: Infisical을 통한 중앙화된 비밀값 관리
3. **TLS 통신**: NATS 연결 시 TLS 인증서 사용
4. **Rate Limiting**: 크롤링 대상 서버 보호
5. **Input Validation**: DTO 기반 요청 검증
## 라이선스
MIT
## 작성자
- Dahyeon
- Repository: https://git.dahyeon.us/dahyeon/crawler
## 기술 지원
이슈 및 문의: Repository Issues 페이지
+1 -1
View File
@@ -1 +1 @@
INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510 INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
+1 -1
View File
@@ -1,3 +1,3 @@
INFISICAL_CLIENT_SECRET= INFISICAL_CLIENT_SECRET=
INFISICAL_CLIENT_ID= INFISICAL_CLIENT_ID=
INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510 INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
+1 -1
View File
@@ -1 +1 @@
INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510 INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
+1 -1
View File
@@ -1 +1 @@
INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510 INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
+467
View File
@@ -14,10 +14,18 @@
"@nestjs/config": "^4.0.2", "@nestjs/config": "^4.0.2",
"@nestjs/core": "^11.0.1", "@nestjs/core": "^11.0.1",
"@nestjs/platform-express": "^11.0.1", "@nestjs/platform-express": "^11.0.1",
"@nestjs/schedule": "^6.1.0",
"@nestjs/swagger": "^11.2.3", "@nestjs/swagger": "^11.2.3",
"axios": "^1.13.2",
"cheerio": "^1.1.2",
"class-transformer": "^0.5.1",
"class-validator": "^0.14.3",
"cron": "^4.3.5",
"jose": "^6.1.3",
"minio": "^8.0.6", "minio": "^8.0.6",
"mongoose": "^9.0.2", "mongoose": "^9.0.2",
"nats": "^2.29.3", "nats": "^2.29.3",
"playwright": "^1.57.0",
"reflect-metadata": "^0.2.2", "reflect-metadata": "^0.2.2",
"rxjs": "^7.8.1", "rxjs": "^7.8.1",
"temporal-polyfill": "^0.3.0" "temporal-polyfill": "^0.3.0"
@@ -28,8 +36,10 @@
"@nestjs/cli": "^11.0.0", "@nestjs/cli": "^11.0.0",
"@nestjs/schematics": "^11.0.0", "@nestjs/schematics": "^11.0.0",
"@nestjs/testing": "^11.0.1", "@nestjs/testing": "^11.0.1",
"@types/cheerio": "^0.22.35",
"@types/express": "^5.0.0", "@types/express": "^5.0.0",
"@types/jest": "^30.0.0", "@types/jest": "^30.0.0",
"@types/multer": "^2.0.0",
"@types/node": "^24.10.4", "@types/node": "^24.10.4",
"@types/supertest": "^6.0.2", "@types/supertest": "^6.0.2",
"eslint": "^9.18.0", "eslint": "^9.18.0",
@@ -5818,6 +5828,19 @@
"@nestjs/core": "^11.0.0" "@nestjs/core": "^11.0.0"
} }
}, },
"node_modules/@nestjs/schedule": {
"version": "6.1.0",
"resolved": "https://registry.npmjs.org/@nestjs/schedule/-/schedule-6.1.0.tgz",
"integrity": "sha512-W25Ydc933Gzb1/oo7+bWzzDiOissE+h/dhIAPugA39b9MuIzBbLybuXpc1AjoQLczO3v0ldmxaffVl87W0uqoQ==",
"license": "MIT",
"dependencies": {
"cron": "4.3.5"
},
"peerDependencies": {
"@nestjs/common": "^10.0.0 || ^11.0.0",
"@nestjs/core": "^10.0.0 || ^11.0.0"
}
},
"node_modules/@nestjs/schematics": { "node_modules/@nestjs/schematics": {
"version": "11.0.9", "version": "11.0.9",
"resolved": "https://registry.npmjs.org/@nestjs/schematics/-/schematics-11.0.9.tgz", "resolved": "https://registry.npmjs.org/@nestjs/schematics/-/schematics-11.0.9.tgz",
@@ -6243,6 +6266,16 @@
"@types/node": "*" "@types/node": "*"
} }
}, },
"node_modules/@types/cheerio": {
"version": "0.22.35",
"resolved": "https://registry.npmjs.org/@types/cheerio/-/cheerio-0.22.35.tgz",
"integrity": "sha512-yD57BchKRvTV+JD53UZ6PD8KWY5g5rvvMLRnZR3EQBCZXiDT/HR+pKpMzFGlWNhFrXlo7VPZXtKvIEwZkAWOIA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": "*"
}
},
"node_modules/@types/connect": { "node_modules/@types/connect": {
"version": "3.4.38", "version": "3.4.38",
"resolved": "https://registry.npmjs.org/@types/connect/-/connect-3.4.38.tgz", "resolved": "https://registry.npmjs.org/@types/connect/-/connect-3.4.38.tgz",
@@ -6367,6 +6400,12 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/@types/luxon": {
"version": "3.7.1",
"resolved": "https://registry.npmjs.org/@types/luxon/-/luxon-3.7.1.tgz",
"integrity": "sha512-H3iskjFIAn5SlJU7OuxUmTEpebK6TKB8rxZShDslBMZJ5u9S//KM1sbdAisiSrqwLQncVjnpi2OK2J51h+4lsg==",
"license": "MIT"
},
"node_modules/@types/methods": { "node_modules/@types/methods": {
"version": "1.1.4", "version": "1.1.4",
"resolved": "https://registry.npmjs.org/@types/methods/-/methods-1.1.4.tgz", "resolved": "https://registry.npmjs.org/@types/methods/-/methods-1.1.4.tgz",
@@ -6374,6 +6413,16 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/@types/multer": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/@types/multer/-/multer-2.0.0.tgz",
"integrity": "sha512-C3Z9v9Evij2yST3RSBktxP9STm6OdMc5uR1xF1SGr98uv8dUlAL2hqwrZ3GVB3uyMyiegnscEK6PGtYvNrjTjw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/express": "*"
}
},
"node_modules/@types/node": { "node_modules/@types/node": {
"version": "24.10.4", "version": "24.10.4",
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.10.4.tgz", "resolved": "https://registry.npmjs.org/@types/node/-/node-24.10.4.tgz",
@@ -6451,6 +6500,12 @@
"@types/superagent": "^8.1.0" "@types/superagent": "^8.1.0"
} }
}, },
"node_modules/@types/validator": {
"version": "13.15.10",
"resolved": "https://registry.npmjs.org/@types/validator/-/validator-13.15.10.tgz",
"integrity": "sha512-T8L6i7wCuyoK8A/ZeLYt1+q0ty3Zb9+qbSSvrIVitzT3YjZqkTZ40IbRsPanlB4h1QB3JVL1SYCdR6ngtFYcuA==",
"license": "MIT"
},
"node_modules/@types/webidl-conversions": { "node_modules/@types/webidl-conversions": {
"version": "7.0.3", "version": "7.0.3",
"resolved": "https://registry.npmjs.org/@types/webidl-conversions/-/webidl-conversions-7.0.3.tgz", "resolved": "https://registry.npmjs.org/@types/webidl-conversions/-/webidl-conversions-7.0.3.tgz",
@@ -7673,6 +7728,12 @@
"url": "https://opencollective.com/express" "url": "https://opencollective.com/express"
} }
}, },
"node_modules/boolbase": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/boolbase/-/boolbase-1.0.0.tgz",
"integrity": "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==",
"license": "ISC"
},
"node_modules/bowser": { "node_modules/bowser": {
"version": "2.13.1", "version": "2.13.1",
"resolved": "https://registry.npmjs.org/bowser/-/bowser-2.13.1.tgz", "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.13.1.tgz",
@@ -7958,6 +8019,48 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/cheerio": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/cheerio/-/cheerio-1.1.2.tgz",
"integrity": "sha512-IkxPpb5rS/d1IiLbHMgfPuS0FgiWTtFIm/Nj+2woXDLTZ7fOT2eqzgYbdMlLweqlHbsZjxEChoVK+7iph7jyQg==",
"license": "MIT",
"dependencies": {
"cheerio-select": "^2.1.0",
"dom-serializer": "^2.0.0",
"domhandler": "^5.0.3",
"domutils": "^3.2.2",
"encoding-sniffer": "^0.2.1",
"htmlparser2": "^10.0.0",
"parse5": "^7.3.0",
"parse5-htmlparser2-tree-adapter": "^7.1.0",
"parse5-parser-stream": "^7.1.2",
"undici": "^7.12.0",
"whatwg-mimetype": "^4.0.0"
},
"engines": {
"node": ">=20.18.1"
},
"funding": {
"url": "https://github.com/cheeriojs/cheerio?sponsor=1"
}
},
"node_modules/cheerio-select": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/cheerio-select/-/cheerio-select-2.1.0.tgz",
"integrity": "sha512-9v9kG0LvzrlcungtnJtpGNxY+fzECQKhK4EGJX2vByejiMX84MFNQw4UxPJl3bFbTMw+Dfs37XaIkCwTZfLh4g==",
"license": "BSD-2-Clause",
"dependencies": {
"boolbase": "^1.0.0",
"css-select": "^5.1.0",
"css-what": "^6.1.0",
"domelementtype": "^2.3.0",
"domhandler": "^5.0.3",
"domutils": "^3.0.1"
},
"funding": {
"url": "https://github.com/sponsors/fb55"
}
},
"node_modules/chokidar": { "node_modules/chokidar": {
"version": "4.0.3", "version": "4.0.3",
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz", "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz",
@@ -8008,6 +8111,25 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/class-transformer": {
"version": "0.5.1",
"resolved": "https://registry.npmjs.org/class-transformer/-/class-transformer-0.5.1.tgz",
"integrity": "sha512-SQa1Ws6hUbfC98vKGxZH3KFY0Y1lm5Zm0SY8XX9zbK7FJCyVEac3ATW0RIpwzW+oOfmHE5PMPufDG9hCfoEOMw==",
"license": "MIT",
"peer": true
},
"node_modules/class-validator": {
"version": "0.14.3",
"resolved": "https://registry.npmjs.org/class-validator/-/class-validator-0.14.3.tgz",
"integrity": "sha512-rXXekcjofVN1LTOSw+u4u9WXVEUvNBVjORW154q/IdmYWy1nMbOU9aNtZB0t8m+FJQ9q91jlr2f9CwwUFdFMRA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/validator": "^13.15.3",
"libphonenumber-js": "^1.11.1",
"validator": "^13.15.20"
}
},
"node_modules/cli-cursor": { "node_modules/cli-cursor": {
"version": "3.1.0", "version": "3.1.0",
"resolved": "https://registry.npmjs.org/cli-cursor/-/cli-cursor-3.1.0.tgz", "resolved": "https://registry.npmjs.org/cli-cursor/-/cli-cursor-3.1.0.tgz",
@@ -8327,6 +8449,19 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/cron": {
"version": "4.3.5",
"resolved": "https://registry.npmjs.org/cron/-/cron-4.3.5.tgz",
"integrity": "sha512-hKPP7fq1+OfyCqoePkKfVq7tNAdFwiQORr4lZUHwrf0tebC65fYEeWgOrXOL6prn1/fegGOdTfrM6e34PJfksg==",
"license": "MIT",
"dependencies": {
"@types/luxon": "~3.7.0",
"luxon": "~3.7.0"
},
"engines": {
"node": ">=18.x"
}
},
"node_modules/cross-spawn": { "node_modules/cross-spawn": {
"version": "7.0.6", "version": "7.0.6",
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
@@ -8342,6 +8477,34 @@
"node": ">= 8" "node": ">= 8"
} }
}, },
"node_modules/css-select": {
"version": "5.2.2",
"resolved": "https://registry.npmjs.org/css-select/-/css-select-5.2.2.tgz",
"integrity": "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw==",
"license": "BSD-2-Clause",
"dependencies": {
"boolbase": "^1.0.0",
"css-what": "^6.1.0",
"domhandler": "^5.0.2",
"domutils": "^3.0.1",
"nth-check": "^2.0.1"
},
"funding": {
"url": "https://github.com/sponsors/fb55"
}
},
"node_modules/css-what": {
"version": "6.2.2",
"resolved": "https://registry.npmjs.org/css-what/-/css-what-6.2.2.tgz",
"integrity": "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA==",
"license": "BSD-2-Clause",
"engines": {
"node": ">= 6"
},
"funding": {
"url": "https://github.com/sponsors/fb55"
}
},
"node_modules/debug": { "node_modules/debug": {
"version": "4.4.3", "version": "4.4.3",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
@@ -8479,6 +8642,61 @@
"node": ">=0.3.1" "node": ">=0.3.1"
} }
}, },
"node_modules/dom-serializer": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-2.0.0.tgz",
"integrity": "sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==",
"license": "MIT",
"dependencies": {
"domelementtype": "^2.3.0",
"domhandler": "^5.0.2",
"entities": "^4.2.0"
},
"funding": {
"url": "https://github.com/cheeriojs/dom-serializer?sponsor=1"
}
},
"node_modules/domelementtype": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/domelementtype/-/domelementtype-2.3.0.tgz",
"integrity": "sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/fb55"
}
],
"license": "BSD-2-Clause"
},
"node_modules/domhandler": {
"version": "5.0.3",
"resolved": "https://registry.npmjs.org/domhandler/-/domhandler-5.0.3.tgz",
"integrity": "sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==",
"license": "BSD-2-Clause",
"dependencies": {
"domelementtype": "^2.3.0"
},
"engines": {
"node": ">= 4"
},
"funding": {
"url": "https://github.com/fb55/domhandler?sponsor=1"
}
},
"node_modules/domutils": {
"version": "3.2.2",
"resolved": "https://registry.npmjs.org/domutils/-/domutils-3.2.2.tgz",
"integrity": "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==",
"license": "BSD-2-Clause",
"dependencies": {
"dom-serializer": "^2.0.0",
"domelementtype": "^2.3.0",
"domhandler": "^5.0.3"
},
"funding": {
"url": "https://github.com/fb55/domutils?sponsor=1"
}
},
"node_modules/dotenv": { "node_modules/dotenv": {
"version": "16.4.7", "version": "16.4.7",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.4.7.tgz", "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.4.7.tgz",
@@ -8569,6 +8787,31 @@
"node": ">= 0.8" "node": ">= 0.8"
} }
}, },
"node_modules/encoding-sniffer": {
"version": "0.2.1",
"resolved": "https://registry.npmjs.org/encoding-sniffer/-/encoding-sniffer-0.2.1.tgz",
"integrity": "sha512-5gvq20T6vfpekVtqrYQsSCFZ1wEg5+wW0/QaZMWkFr6BqD3NfKs0rLCx4rrVlSWJeZb5NBJgVLswK/w2MWU+Gw==",
"license": "MIT",
"dependencies": {
"iconv-lite": "^0.6.3",
"whatwg-encoding": "^3.1.1"
},
"funding": {
"url": "https://github.com/fb55/encoding-sniffer?sponsor=1"
}
},
"node_modules/encoding-sniffer/node_modules/iconv-lite": {
"version": "0.6.3",
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
"integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==",
"license": "MIT",
"dependencies": {
"safer-buffer": ">= 2.1.2 < 3.0.0"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/enhanced-resolve": { "node_modules/enhanced-resolve": {
"version": "5.18.4", "version": "5.18.4",
"resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.18.4.tgz", "resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.18.4.tgz",
@@ -8583,6 +8826,18 @@
"node": ">=10.13.0" "node": ">=10.13.0"
} }
}, },
"node_modules/entities": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/entities/-/entities-4.5.0.tgz",
"integrity": "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=0.12"
},
"funding": {
"url": "https://github.com/fb55/entities?sponsor=1"
}
},
"node_modules/error-ex": { "node_modules/error-ex": {
"version": "1.3.4", "version": "1.3.4",
"resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.4.tgz", "resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.4.tgz",
@@ -9713,6 +9968,37 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/htmlparser2": {
"version": "10.0.0",
"resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-10.0.0.tgz",
"integrity": "sha512-TwAZM+zE5Tq3lrEHvOlvwgj1XLWQCtaaibSN11Q+gGBAS7Y1uZSWwXXRe4iF6OXnaq1riyQAPFOBtYc77Mxq0g==",
"funding": [
"https://github.com/fb55/htmlparser2?sponsor=1",
{
"type": "github",
"url": "https://github.com/sponsors/fb55"
}
],
"license": "MIT",
"dependencies": {
"domelementtype": "^2.3.0",
"domhandler": "^5.0.3",
"domutils": "^3.2.1",
"entities": "^6.0.0"
}
},
"node_modules/htmlparser2/node_modules/entities": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
"integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=0.12"
},
"funding": {
"url": "https://github.com/fb55/entities?sponsor=1"
}
},
"node_modules/http-errors": { "node_modules/http-errors": {
"version": "2.0.1", "version": "2.0.1",
"resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
@@ -10892,6 +11178,15 @@
"url": "https://github.com/chalk/supports-color?sponsor=1" "url": "https://github.com/chalk/supports-color?sponsor=1"
} }
}, },
"node_modules/jose": {
"version": "6.1.3",
"resolved": "https://registry.npmjs.org/jose/-/jose-6.1.3.tgz",
"integrity": "sha512-0TpaTfihd4QMNwrz/ob2Bp7X04yuxJkjRGi4aKmOqwhov54i6u79oCv7T+C7lo70MKH6BesI3vscD1yb/yzKXQ==",
"license": "MIT",
"funding": {
"url": "https://github.com/sponsors/panva"
}
},
"node_modules/js-tokens": { "node_modules/js-tokens": {
"version": "4.0.0", "version": "4.0.0",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
@@ -11028,6 +11323,12 @@
"node": ">= 0.8.0" "node": ">= 0.8.0"
} }
}, },
"node_modules/libphonenumber-js": {
"version": "1.12.33",
"resolved": "https://registry.npmjs.org/libphonenumber-js/-/libphonenumber-js-1.12.33.tgz",
"integrity": "sha512-r9kw4OA6oDO4dPXkOrXTkArQAafIKAU71hChInV4FxZ69dxCfbwQGDPzqR5/vea94wU705/3AZroEbSoeVWrQw==",
"license": "MIT"
},
"node_modules/lines-and-columns": { "node_modules/lines-and-columns": {
"version": "1.2.4", "version": "1.2.4",
"resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz", "resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz",
@@ -11131,6 +11432,15 @@
"yallist": "^3.0.2" "yallist": "^3.0.2"
} }
}, },
"node_modules/luxon": {
"version": "3.7.2",
"resolved": "https://registry.npmjs.org/luxon/-/luxon-3.7.2.tgz",
"integrity": "sha512-vtEhXh/gNjI9Yg1u4jX/0YVPMvxzHuGgCm6tC5kZyb08yjGWGnqAjGJvcXbqQR2P3MyMEFnRbpcdFS6PBcLqew==",
"license": "MIT",
"engines": {
"node": ">=12"
}
},
"node_modules/magic-string": { "node_modules/magic-string": {
"version": "0.30.17", "version": "0.30.17",
"resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.17.tgz", "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.17.tgz",
@@ -11706,6 +12016,18 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/nth-check": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/nth-check/-/nth-check-2.1.1.tgz",
"integrity": "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w==",
"license": "BSD-2-Clause",
"dependencies": {
"boolbase": "^1.0.0"
},
"funding": {
"url": "https://github.com/fb55/nth-check?sponsor=1"
}
},
"node_modules/object-assign": { "node_modules/object-assign": {
"version": "4.1.1", "version": "4.1.1",
"resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
@@ -11887,6 +12209,55 @@
"url": "https://github.com/sponsors/sindresorhus" "url": "https://github.com/sponsors/sindresorhus"
} }
}, },
"node_modules/parse5": {
"version": "7.3.0",
"resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz",
"integrity": "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==",
"license": "MIT",
"dependencies": {
"entities": "^6.0.0"
},
"funding": {
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/parse5-htmlparser2-tree-adapter": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/parse5-htmlparser2-tree-adapter/-/parse5-htmlparser2-tree-adapter-7.1.0.tgz",
"integrity": "sha512-ruw5xyKs6lrpo9x9rCZqZZnIUntICjQAd0Wsmp396Ul9lN/h+ifgVV1x1gZHi8euej6wTfpqX8j+BFQxF0NS/g==",
"license": "MIT",
"dependencies": {
"domhandler": "^5.0.3",
"parse5": "^7.0.0"
},
"funding": {
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/parse5-parser-stream": {
"version": "7.1.2",
"resolved": "https://registry.npmjs.org/parse5-parser-stream/-/parse5-parser-stream-7.1.2.tgz",
"integrity": "sha512-JyeQc9iwFLn5TbvvqACIF/VXG6abODeB3Fwmv/TGdLk2LfbWkaySGY72at4+Ty7EkPZj854u4CrICqNk2qIbow==",
"license": "MIT",
"dependencies": {
"parse5": "^7.0.0"
},
"funding": {
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/parse5/node_modules/entities": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
"integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=0.12"
},
"funding": {
"url": "https://github.com/fb55/entities?sponsor=1"
}
},
"node_modules/parseurl": { "node_modules/parseurl": {
"version": "1.3.3", "version": "1.3.3",
"resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
@@ -12072,6 +12443,50 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/playwright": {
"version": "1.57.0",
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.57.0.tgz",
"integrity": "sha512-ilYQj1s8sr2ppEJ2YVadYBN0Mb3mdo9J0wQ+UuDhzYqURwSoW4n1Xs5vs7ORwgDGmyEh33tRMeS8KhdkMoLXQw==",
"license": "Apache-2.0",
"dependencies": {
"playwright-core": "1.57.0"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=18"
},
"optionalDependencies": {
"fsevents": "2.3.2"
}
},
"node_modules/playwright-core": {
"version": "1.57.0",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.57.0.tgz",
"integrity": "sha512-agTcKlMw/mjBWOnD6kFZttAAGHgi/Nw0CZ2o6JqWSbMlI219lAFLZZCyqByTsvVAJq5XA5H8cA6PrvBRpBWEuQ==",
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=18"
}
},
"node_modules/playwright/node_modules/fsevents": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
},
"node_modules/pluralize": { "node_modules/pluralize": {
"version": "8.0.0", "version": "8.0.0",
"resolved": "https://registry.npmjs.org/pluralize/-/pluralize-8.0.0.tgz", "resolved": "https://registry.npmjs.org/pluralize/-/pluralize-8.0.0.tgz",
@@ -12408,6 +12823,7 @@
"resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.2.tgz", "resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.2.tgz",
"integrity": "sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==", "integrity": "sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==",
"license": "Apache-2.0", "license": "Apache-2.0",
"peer": true,
"dependencies": { "dependencies": {
"tslib": "^2.1.0" "tslib": "^2.1.0"
} }
@@ -13702,6 +14118,15 @@
"url": "https://github.com/sponsors/sindresorhus" "url": "https://github.com/sponsors/sindresorhus"
} }
}, },
"node_modules/undici": {
"version": "7.16.0",
"resolved": "https://registry.npmjs.org/undici/-/undici-7.16.0.tgz",
"integrity": "sha512-QEg3HPMll0o3t2ourKwOeUAZ159Kn9mx5pnzHRQO8+Wixmh88YdZRiIwat0iNzNNXn0yoEtXJqFpyW7eM8BV7g==",
"license": "MIT",
"engines": {
"node": ">=20.18.1"
}
},
"node_modules/undici-types": { "node_modules/undici-types": {
"version": "7.16.0", "version": "7.16.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.16.0.tgz", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.16.0.tgz",
@@ -13858,6 +14283,15 @@
"node": ">=10.12.0" "node": ">=10.12.0"
} }
}, },
"node_modules/validator": {
"version": "13.15.26",
"resolved": "https://registry.npmjs.org/validator/-/validator-13.15.26.tgz",
"integrity": "sha512-spH26xU080ydGggxRyR1Yhcbgx+j3y5jbNXk/8L+iRvdIEQ4uTRH2Sgf2dokud6Q4oAtsbNvJ1Ft+9xmm6IZcA==",
"license": "MIT",
"engines": {
"node": ">= 0.10"
}
},
"node_modules/vary": { "node_modules/vary": {
"version": "1.1.2", "version": "1.1.2",
"resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
@@ -14115,6 +14549,39 @@
"url": "https://opencollective.com/webpack" "url": "https://opencollective.com/webpack"
} }
}, },
"node_modules/whatwg-encoding": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/whatwg-encoding/-/whatwg-encoding-3.1.1.tgz",
"integrity": "sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ==",
"license": "MIT",
"dependencies": {
"iconv-lite": "0.6.3"
},
"engines": {
"node": ">=18"
}
},
"node_modules/whatwg-encoding/node_modules/iconv-lite": {
"version": "0.6.3",
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
"integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==",
"license": "MIT",
"dependencies": {
"safer-buffer": ">= 2.1.2 < 3.0.0"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/whatwg-mimetype": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-4.0.0.tgz",
"integrity": "sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg==",
"license": "MIT",
"engines": {
"node": ">=18"
}
},
"node_modules/whatwg-url": { "node_modules/whatwg-url": {
"version": "14.2.0", "version": "14.2.0",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-14.2.0.tgz", "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-14.2.0.tgz",
+10
View File
@@ -29,10 +29,18 @@
"@nestjs/config": "^4.0.2", "@nestjs/config": "^4.0.2",
"@nestjs/core": "^11.0.1", "@nestjs/core": "^11.0.1",
"@nestjs/platform-express": "^11.0.1", "@nestjs/platform-express": "^11.0.1",
"@nestjs/schedule": "^6.1.0",
"@nestjs/swagger": "^11.2.3", "@nestjs/swagger": "^11.2.3",
"axios": "^1.13.2",
"cheerio": "^1.1.2",
"class-transformer": "^0.5.1",
"class-validator": "^0.14.3",
"cron": "^4.3.5",
"jose": "^6.1.3",
"minio": "^8.0.6", "minio": "^8.0.6",
"mongoose": "^9.0.2", "mongoose": "^9.0.2",
"nats": "^2.29.3", "nats": "^2.29.3",
"playwright": "^1.57.0",
"reflect-metadata": "^0.2.2", "reflect-metadata": "^0.2.2",
"rxjs": "^7.8.1", "rxjs": "^7.8.1",
"temporal-polyfill": "^0.3.0" "temporal-polyfill": "^0.3.0"
@@ -43,8 +51,10 @@
"@nestjs/cli": "^11.0.0", "@nestjs/cli": "^11.0.0",
"@nestjs/schematics": "^11.0.0", "@nestjs/schematics": "^11.0.0",
"@nestjs/testing": "^11.0.1", "@nestjs/testing": "^11.0.1",
"@types/cheerio": "^0.22.35",
"@types/express": "^5.0.0", "@types/express": "^5.0.0",
"@types/jest": "^30.0.0", "@types/jest": "^30.0.0",
"@types/multer": "^2.0.0",
"@types/node": "^24.10.4", "@types/node": "^24.10.4",
"@types/supertest": "^6.0.2", "@types/supertest": "^6.0.2",
"eslint": "^9.18.0", "eslint": "^9.18.0",
+5
View File
@@ -1,6 +1,9 @@
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { AppController } from './app.controller'; import { AppController } from './app.controller';
import { AppService } from './app.service'; import { AppService } from './app.service';
import { ConfigModule } from '@nestjs/config';
import { GlobalModule } from './global/global.module';
import { CrawlerModule } from './crawler/crawler.module';
@Module({ @Module({
imports: [ imports: [
@@ -13,6 +16,8 @@ import { AppService } from './app.service';
? '.env.test' ? '.env.test'
: '.env.dev', : '.env.dev',
}), }),
GlobalModule,
CrawlerModule,
], ],
controllers: [AppController], controllers: [AppController],
providers: [AppService], providers: [AppService],
+338
View File
@@ -0,0 +1,338 @@
import {
Controller,
Get,
Post,
Put,
Delete,
Body,
Param,
Query,
HttpStatus,
HttpCode,
UseGuards,
NotFoundException,
} from '@nestjs/common';
import mongoose from 'mongoose';
import {
ApiTags,
ApiOperation,
ApiResponse,
ApiBearerAuth,
ApiQuery,
} from '@nestjs/swagger';
import { JwtAuthGuard } from '../global/auth/jwt.guard';
import { AllowPublicAccess } from '../global/auth/allow-public-access.decorator';
import { PipelineEngineService } from './services/pipeline-engine.service';
import { TriggerService, ScheduledJob } from './services/trigger.service';
import { RateLimiterService } from './services/rate-limiter.service';
import { ConnectionService } from '../global/database/connection.service';
import {
CreatePipelinedefDto,
UpdatePipelinedefDto,
CreateStagedefDto,
TriggerPipelineDto,
} from './dto';
@ApiTags('Crawler')
@Controller('api/v1/crawler')
@UseGuards(JwtAuthGuard)
@ApiBearerAuth()
export class CrawlerController {
constructor(
private readonly pipelineEngine: PipelineEngineService,
private readonly triggerService: TriggerService,
private readonly rateLimiter: RateLimiterService,
private readonly connectionService: ConnectionService,
) {}
// ============ Pipelinedef CRUD ============
@Post('pipelinedefs')
@ApiOperation({ summary: 'Create a new pipeline definition' })
@ApiResponse({ status: 201, description: 'Pipeline definition created' })
async createPipelinedef(@Body() dto: CreatePipelinedefDto) {
const model = await this.connectionService.getMoeumModels('pipelinedefs');
const metadatas = dto.metadatas || [];
// 스케줄 정보가 있으면 메타데이터에 추가
if (dto.schedule) {
metadatas.push({
key: 'schedule',
value: JSON.stringify(dto.schedule),
dataType: 'json',
isReserved: true,
});
}
const pipelinedef = await model.create({
apiVersion: dto.apiVersion || 'v1',
trigger: dto.trigger,
stage: dto.stageIds.map((id) => new mongoose.Types.ObjectId(id)),
metadatas,
});
// 트리거 등록
if (dto.schedule) {
await this.triggerService.registerScheduleTrigger(
pipelinedef._id.toString(),
dto.schedule,
);
}
if (dto.trigger.type === 'kafka-event' && dto.trigger.kafkaEvent) {
await this.triggerService.registerNatsTrigger(
pipelinedef._id.toString(),
dto.trigger.kafkaEvent.topic,
);
}
return {
id: pipelinedef._id,
message: 'Pipeline definition created',
};
}
@Get('pipelinedefs')
@ApiOperation({ summary: 'List all pipeline definitions' })
@ApiQuery({ name: 'page', required: false, type: Number })
@ApiQuery({ name: 'limit', required: false, type: Number })
async listPipelinedefs(
@Query('page') page: number = 1,
@Query('limit') limit: number = 20,
) {
const { items, total } = await this.pipelineEngine.listPipelinedefs(
page,
limit,
);
return {
items,
pagination: {
page: Number(page),
limit: Number(limit),
total,
totalPages: Math.ceil(total / limit),
},
};
}
@Get('pipelinedefs/:id')
@ApiOperation({ summary: 'Get a pipeline definition by ID' })
async getPipelinedef(@Param('id') id: string) {
const model = await this.connectionService.getMoeumModels('pipelinedefs');
const pipelinedef = await model.findById(id).exec();
if (!pipelinedef) {
throw new NotFoundException('Pipeline definition not found');
}
return pipelinedef;
}
@Put('pipelinedefs/:id')
@ApiOperation({ summary: 'Update a pipeline definition' })
async updatePipelinedef(
@Param('id') id: string,
@Body() dto: UpdatePipelinedefDto,
) {
const model = await this.connectionService.getMoeumModels('pipelinedefs');
const updateData: any = {};
if (dto.apiVersion) updateData.apiVersion = dto.apiVersion;
if (dto.trigger) updateData.trigger = dto.trigger;
if (dto.stageIds) updateData.stage = dto.stageIds;
if (dto.metadatas) updateData.metadatas = dto.metadatas;
const updated = await model
.findByIdAndUpdate(id, updateData, { new: true })
.exec();
if (!updated) {
throw new NotFoundException('Pipeline definition not found');
}
// 스케줄 업데이트
if (dto.schedule) {
await this.triggerService.registerScheduleTrigger(id, dto.schedule);
}
return updated;
}
@Delete('pipelinedefs/:id')
@HttpCode(HttpStatus.NO_CONTENT)
@ApiOperation({ summary: 'Delete a pipeline definition' })
async deletePipelinedef(@Param('id') id: string) {
const model = await this.connectionService.getMoeumModels('pipelinedefs');
await model.findByIdAndDelete(id).exec();
await this.triggerService.unregisterScheduleTrigger(id);
}
// ============ Stagedef CRUD ============
@Post('stagedefs')
@ApiOperation({ summary: 'Create a new stage definition' })
async createStagedef(@Body() dto: CreateStagedefDto) {
const model = await this.connectionService.getMoeumModels('stagedefs');
const stagedef = await model.create({
apiVersion: dto.apiVersion || 'v1',
name: dto.name,
taxonomy: dto.taxonomy,
eventSource: dto.eventSource,
schema: dto.schema,
topic: dto.topic,
metadatas: dto.metadatas || [],
});
return {
id: stagedef._id,
message: 'Stage definition created',
};
}
@Get('stagedefs')
@ApiOperation({ summary: 'List all stage definitions' })
@ApiQuery({ name: 'page', required: false, type: Number })
@ApiQuery({ name: 'limit', required: false, type: Number })
async listStagedefs(
@Query('page') page: number = 1,
@Query('limit') limit: number = 20,
) {
const model = await this.connectionService.getMoeumModels('stagedefs');
const skip = (Number(page) - 1) * Number(limit);
const [items, total] = await Promise.all([
model.find().skip(skip).limit(Number(limit)).exec(),
model.countDocuments().exec(),
]);
return {
items,
pagination: {
page: Number(page),
limit: Number(limit),
total,
totalPages: Math.ceil(total / Number(limit)),
},
};
}
@Get('stagedefs/:id')
@ApiOperation({ summary: 'Get a stage definition by ID' })
async getStagedef(@Param('id') id: string) {
const model = await this.connectionService.getMoeumModels('stagedefs');
const stagedef = await model.findById(id).exec();
if (!stagedef) {
throw new NotFoundException('Stage definition not found');
}
return stagedef;
}
@Delete('stagedefs/:id')
@HttpCode(HttpStatus.NO_CONTENT)
@ApiOperation({ summary: 'Delete a stage definition' })
async deleteStagedef(@Param('id') id: string) {
const model = await this.connectionService.getMoeumModels('stagedefs');
await model.findByIdAndDelete(id).exec();
}
// ============ Pipeline 실행 ============
@Post('pipelines/trigger')
@ApiOperation({ summary: 'Trigger a pipeline execution' })
async triggerPipeline(@Body() dto: TriggerPipelineDto) {
const pipelineId = await this.triggerService.triggerPipelineManually(
dto.pipelineDefId,
);
return {
pipelineId,
message: 'Pipeline triggered successfully',
};
}
@Get('pipelines')
@ApiOperation({ summary: 'List executed pipelines' })
@ApiQuery({ name: 'page', required: false, type: Number })
@ApiQuery({ name: 'limit', required: false, type: Number })
@ApiQuery({ name: 'status', required: false, type: String })
async listPipelines(
@Query('page') page: number = 1,
@Query('limit') limit: number = 20,
@Query('status') status?: string,
) {
const { items, total } = await this.pipelineEngine.listPipelines(
Number(page),
Number(limit),
status,
);
return {
items,
pagination: {
page: Number(page),
limit: Number(limit),
total,
totalPages: Math.ceil(total / Number(limit)),
},
};
}
@Get('pipelines/:id')
@ApiOperation({ summary: 'Get pipeline execution details' })
async getPipeline(@Param('id') id: string) {
const pipeline = await this.pipelineEngine.getPipelineStatus(id);
if (!pipeline) {
throw new NotFoundException('Pipeline not found');
}
return pipeline;
}
@Get('pipelines/:id/stages')
@ApiOperation({ summary: 'Get stages of a pipeline' })
async getPipelineStages(@Param('id') id: string) {
const pipeline = await this.pipelineEngine.getPipelineStatus(id);
if (!pipeline) {
throw new NotFoundException('Pipeline not found');
}
return pipeline.stage;
}
// ============ 모니터링 ============
@Get('schedules')
@ApiOperation({ summary: 'List registered schedules' })
async listSchedules(): Promise<ScheduledJob[]> {
return this.triggerService.getRegisteredSchedules();
}
@Post('triggers/reload')
@ApiOperation({ summary: 'Reload all triggers' })
async reloadTriggers() {
await this.triggerService.reloadTriggers();
return { message: 'Triggers reloaded successfully' };
}
@Get('rate-limits')
@ApiOperation({ summary: 'Get rate limit status for all domains' })
async getRateLimits() {
return this.rateLimiter.getAllLimits();
}
@Get('health')
@AllowPublicAccess()
@ApiOperation({ summary: 'Health check' })
async healthCheck() {
return {
status: 'ok',
timestamp: new Date().toISOString(),
};
}
}
+29
View File
@@ -0,0 +1,29 @@
import { Module } from '@nestjs/common';
import { ScheduleModule } from '@nestjs/schedule';
import { CrawlerController } from './crawler.controller';
import { CrawlerService } from './services/crawler.service';
import { RateLimiterService } from './services/rate-limiter.service';
import { RetryService } from './services/retry.service';
import { StageExecutorService } from './services/stage-executor.service';
import { PipelineEngineService } from './services/pipeline-engine.service';
import { TriggerService } from './services/trigger.service';
@Module({
imports: [ScheduleModule.forRoot()],
controllers: [CrawlerController],
providers: [
RateLimiterService,
CrawlerService,
RetryService,
StageExecutorService,
PipelineEngineService,
TriggerService,
],
exports: [
CrawlerService,
PipelineEngineService,
TriggerService,
RateLimiterService,
],
})
export class CrawlerModule {}
View File
Whitespace-only changes.
+103
View File
@@ -0,0 +1,103 @@
import {
IsString,
IsOptional,
IsEnum,
IsBoolean,
ValidateNested,
IsArray,
} from 'class-validator';
import { Type } from 'class-transformer';
import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
export class MetadataDto {
@ApiProperty({ description: '메타데이터 키' })
@IsString()
key: string;
@ApiProperty({ description: '메타데이터 값' })
value: string | any[];
@ApiPropertyOptional({
enum: ['string', 'number', 'boolean', 'array', 'json'],
default: 'string',
})
@IsOptional()
@IsEnum(['string', 'number', 'boolean', 'array', 'json'])
dataType?: 'string' | 'number' | 'boolean' | 'array' | 'json';
@ApiPropertyOptional({ default: false })
@IsOptional()
@IsBoolean()
isReserved?: boolean;
}
export class KafkaEventDto {
@ApiProperty({ description: 'Kafka/NATS 토픽명' })
@IsString()
topic: string;
@ApiPropertyOptional({ description: '메시지 키' })
@IsOptional()
@IsString()
key?: string;
}
export class PipelineTriggerDto {
@ApiProperty({ enum: ['always', 'kafka-event'] })
@IsEnum(['always', 'kafka-event'])
type: 'always' | 'kafka-event';
@ApiPropertyOptional({ type: KafkaEventDto })
@IsOptional()
@ValidateNested()
@Type(() => KafkaEventDto)
kafkaEvent?: KafkaEventDto;
}
export class ScheduleDto {
@ApiProperty({ description: 'Cron 표현식', example: '0 0 * * *' })
@IsString()
cron: string;
@ApiPropertyOptional({ description: '타임존', default: 'Asia/Seoul' })
@IsOptional()
@IsString()
timezone?: string;
}
export class EventSourceDto {
@ApiProperty({ description: '이벤트 소스 타입', example: 'web' })
@IsString()
type: string;
@ApiProperty({ description: '이벤트 소스 이름' })
@IsString()
name: string;
@ApiPropertyOptional({ type: [MetadataDto], description: '이벤트 소스 메타데이터' })
@IsOptional()
@IsArray()
@ValidateNested({ each: true })
@Type(() => MetadataDto)
metadatas?: MetadataDto[];
}
export class PaginationQueryDto {
@ApiPropertyOptional({ default: 1, description: '페이지 번호' })
@IsOptional()
page?: number = 1;
@ApiPropertyOptional({ default: 20, description: '페이지당 항목 수' })
@IsOptional()
limit?: number = 20;
}
export class PaginationResultDto<T> {
items: T[];
pagination: {
page: number;
limit: number;
total: number;
totalPages: number;
};
}
+39
View File
@@ -0,0 +1,39 @@
import {
IsString,
IsOptional,
IsArray,
ValidateNested,
} from 'class-validator';
import { Type } from 'class-transformer';
import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
import { MetadataDto, PipelineTriggerDto, ScheduleDto } from './common.dto';
export class CreatePipelinedefDto {
@ApiPropertyOptional({ default: 'v1' })
@IsOptional()
@IsString()
apiVersion?: string;
@ApiProperty({ type: PipelineTriggerDto })
@ValidateNested()
@Type(() => PipelineTriggerDto)
trigger: PipelineTriggerDto;
@ApiProperty({ type: [String], description: 'Stagedef ID 목록' })
@IsArray()
@IsString({ each: true })
stageIds: string[];
@ApiPropertyOptional({ type: ScheduleDto, description: '스케줄 설정 (선택)' })
@IsOptional()
@ValidateNested()
@Type(() => ScheduleDto)
schedule?: ScheduleDto;
@ApiPropertyOptional({ type: [MetadataDto] })
@IsOptional()
@IsArray()
@ValidateNested({ each: true })
@Type(() => MetadataDto)
metadatas?: MetadataDto[];
}
+48
View File
@@ -0,0 +1,48 @@
import {
IsString,
IsOptional,
IsArray,
ValidateNested,
IsObject,
} from 'class-validator';
import { Type } from 'class-transformer';
import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
import { MetadataDto, EventSourceDto } from './common.dto';
export class CreateStagedefDto {
@ApiPropertyOptional({ default: 'v1' })
@IsOptional()
@IsString()
apiVersion?: string;
@ApiProperty({ description: 'Stage 이름' })
@IsString()
name: string;
@ApiPropertyOptional({ description: '분류 체계' })
@IsOptional()
@IsString()
taxonomy?: string;
@ApiProperty({ type: [EventSourceDto], description: '이벤트 소스 목록' })
@IsArray()
@ValidateNested({ each: true })
@Type(() => EventSourceDto)
eventSource: EventSourceDto[];
@ApiPropertyOptional({ description: '데이터 스키마 (JSON Schema)' })
@IsOptional()
@IsObject()
schema?: any;
@ApiProperty({ description: '결과 발행할 NATS 토픽' })
@IsString()
topic: string;
@ApiPropertyOptional({ type: [MetadataDto] })
@IsOptional()
@IsArray()
@ValidateNested({ each: true })
@Type(() => MetadataDto)
metadatas?: MetadataDto[];
}
+5
View File
@@ -0,0 +1,5 @@
export * from './create-pipelinedef.dto';
export * from './update-pipelinedef.dto';
export * from './create-stagedef.dto';
export * from './trigger-pipeline.dto';
export * from './common.dto';
+21
View File
@@ -0,0 +1,21 @@
import { IsString, IsOptional, IsObject } from 'class-validator';
import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';
export class TriggerPipelineDto {
@ApiProperty({ description: 'Pipelinedef ID' })
@IsString()
pipelineDefId: string;
@ApiPropertyOptional({ description: '추가 변수 (선택)' })
@IsOptional()
@IsObject()
variables?: Record<string, any>;
}
export class TriggerPipelineResponseDto {
@ApiProperty({ description: '생성된 Pipeline ID' })
pipelineId: string;
@ApiProperty()
message: string;
}
@@ -0,0 +1,4 @@
import { PartialType } from '@nestjs/swagger';
import { CreatePipelinedefDto } from './create-pipelinedef.dto';
export class UpdatePipelinedefDto extends PartialType(CreatePipelinedefDto) {}
@@ -0,0 +1,35 @@
export interface CrawlResult {
success: boolean;
url: string;
statusCode?: number;
html?: string;
parsedData?: Record<string, any>;
error?: string;
timestamp: Date;
}
export interface ParserConfig {
selectors: Record<string, string>; // { title: 'h1.title', content: '.article-body' }
extractAttributes?: Record<string, string>; // { imageUrl: 'img.main@src' }
}
export interface RateLimitConfig {
maxTokens: number; // 최대 토큰 수
refillRate: number; // 초당 토큰 보충 속도
}
export interface ExecutionContext {
pipelineId: string;
pipelineDefId: string;
triggeredBy: 'api' | 'schedule' | 'nats';
triggeredAt: Date;
variables: Map<string, any>;
}
export interface StageExecutionResult {
stageIndex: number;
stageName: string;
status: 'completed' | 'failed';
crawlResults?: CrawlResult[];
error?: string;
}
+291
View File
@@ -0,0 +1,291 @@
import { Injectable, Logger, OnModuleDestroy } from '@nestjs/common';
import axios, { AxiosInstance, AxiosRequestConfig } from 'axios';
import * as cheerio from 'cheerio';
import { chromium, Browser, BrowserContext } from 'playwright';
import { MinioService } from '../../global/minio/minio.service';
import { RESTRequest, EventSource } from '../../global/database/schema';
import { CrawlResult, ParserConfig } from '../interfaces/crawler-result.interface';
import { RateLimiterService } from './rate-limiter.service';
@Injectable()
export class CrawlerService implements OnModuleDestroy {
private readonly logger = new Logger(CrawlerService.name);
private readonly httpClient: AxiosInstance;
private browser: Browser | null = null;
constructor(
private readonly minioService: MinioService,
private readonly rateLimiter: RateLimiterService,
) {
this.httpClient = axios.create({
timeout: 30000,
headers: {
'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
},
});
}
async onModuleDestroy() {
if (this.browser) {
await this.browser.close();
this.logger.log('Playwright browser closed');
}
}
/**
* Playwright 브라우저 인스턴스 초기화 (Lazy)
*/
private async getBrowser(): Promise<Browser> {
if (!this.browser) {
this.browser = await chromium.launch({
headless: true,
args: ['--no-sandbox', '--disable-setuid-sandbox'],
});
this.logger.log('Playwright browser launched');
}
return this.browser;
}
/**
* RESTRequest 스키마 기반 HTTP 요청 실행 (정적 페이지용)
*/
async executeRequest(request: RESTRequest): Promise<CrawlResult> {
const config = this.buildAxiosConfig(request);
const domain = new URL(request.url).hostname;
try {
// Rate limiting 적용
await this.rateLimiter.acquire(domain);
const response = await this.httpClient.request(config);
return {
success: true,
url: request.url,
statusCode: response.status,
html:
typeof response.data === 'string'
? response.data
: JSON.stringify(response.data),
timestamp: new Date(),
};
} catch (error) {
this.logger.error(`HTTP request failed: ${request.url}`, error);
return {
success: false,
url: request.url,
error: error instanceof Error ? error.message : 'Unknown error',
timestamp: new Date(),
};
}
}
/**
* Playwright로 동적 페이지 크롤링 (SPA용)
*/
async crawlDynamicPage(
url: string,
options?: {
waitSelector?: string;
waitTimeout?: number;
javascript?: boolean;
},
): Promise<CrawlResult> {
const domain = new URL(url).hostname;
try {
// Rate limiting 적용
await this.rateLimiter.acquire(domain);
const browser = await this.getBrowser();
const context: BrowserContext = await browser.newContext({
userAgent:
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
});
const page = await context.newPage();
try {
await page.goto(url, {
waitUntil: 'networkidle',
timeout: options?.waitTimeout ?? 30000,
});
// 특정 셀렉터가 나타날 때까지 대기
if (options?.waitSelector) {
await page.waitForSelector(options.waitSelector, {
timeout: options?.waitTimeout ?? 10000,
});
}
const html = await page.content();
return {
success: true,
url,
statusCode: 200,
html,
timestamp: new Date(),
};
} finally {
await context.close();
}
} catch (error) {
this.logger.error(`Dynamic page crawl failed: ${url}`, error);
return {
success: false,
url,
error: error instanceof Error ? error.message : 'Unknown error',
timestamp: new Date(),
};
}
}
/**
* HTML 파싱 및 데이터 추출
*/
parseHtml(html: string, config: ParserConfig): Record<string, any> {
const $ = cheerio.load(html);
const result: Record<string, any> = {};
// 셀렉터 기반 텍스트 추출
for (const [key, selector] of Object.entries(config.selectors)) {
const elements = $(selector);
result[key] =
elements.length > 1
? elements
.map((_, el) => $(el).text().trim())
.get()
: elements.first().text().trim();
}
// 속성 추출 (예: img@src)
if (config.extractAttributes) {
for (const [key, selectorAttr] of Object.entries(
config.extractAttributes,
)) {
const [selector, attr] = selectorAttr.split('@');
const elements = $(selector);
result[key] =
elements.length > 1
? elements.map((_, el) => $(el).attr(attr)).get()
: elements.first().attr(attr);
}
}
return result;
}
/**
* EventSource 기반 크롤링 실행
*/
async crawlFromEventSource(eventSource: EventSource): Promise<CrawlResult> {
const urlMeta = eventSource.metadatas.find((m) => m.key === 'url');
const selectorsMeta = eventSource.metadatas.find(
(m) => m.key === 'selectors',
);
const dynamicMeta = eventSource.metadatas.find((m) => m.key === 'dynamic');
const waitSelectorMeta = eventSource.metadatas.find(
(m) => m.key === 'waitSelector',
);
if (!urlMeta) {
return {
success: false,
url: '',
error: 'URL not found in EventSource metadatas',
timestamp: new Date(),
};
}
const url = urlMeta.value as string;
const isDynamic = dynamicMeta?.value === 'true' || String(dynamicMeta?.value) === 'true';
let crawlResult: CrawlResult;
if (isDynamic) {
// 동적 페이지 크롤링 (Playwright)
crawlResult = await this.crawlDynamicPage(url, {
waitSelector: waitSelectorMeta?.value as string | undefined,
});
} else {
// 정적 페이지 크롤링 (axios)
const request: RESTRequest = {
url,
method: 'GET',
};
crawlResult = await this.executeRequest(request);
}
// HTML 파싱
if (crawlResult.success && crawlResult.html && selectorsMeta) {
try {
const parserConfig: ParserConfig =
typeof selectorsMeta.value === 'string'
? JSON.parse(selectorsMeta.value)
: selectorsMeta.value;
crawlResult.parsedData = this.parseHtml(crawlResult.html, parserConfig);
} catch (error) {
this.logger.warn(`Failed to parse selectors config: ${error}`);
}
}
return crawlResult;
}
/**
* 크롤링 결과를 MinIO에 저장
*/
async saveResultToStorage(
result: CrawlResult,
stageId: string,
): Promise<string> {
const buffer = Buffer.from(JSON.stringify(result, null, 2));
const file = {
buffer,
originalname: `crawl-result-${stageId}-${Date.now()}.json`,
mimetype: 'application/json',
size: buffer.length,
} as Express.Multer.File;
return this.minioService.uploadFile(file);
}
/**
* axios 설정 빌드
*/
private buildAxiosConfig(request: RESTRequest): AxiosRequestConfig {
const config: AxiosRequestConfig = {
url: request.url,
method: request.method.toLowerCase() as
| 'get'
| 'post'
| 'put'
| 'delete'
| 'patch',
headers: request.headers || {},
params: request.queryParams,
data: request.body,
timeout: request.timeout || 30000,
};
// 인증 처리
if (request.auth) {
switch (request.auth.type) {
case 'BearerToken':
config.headers!['Authorization'] = `Bearer ${request.auth.token}`;
break;
case 'BasicAuth':
config.auth = {
username: request.auth.username!,
password: request.auth.password!,
};
break;
case 'APIKey':
config.headers!['X-API-Key'] = request.auth.token;
break;
}
}
return config;
}
}
@@ -0,0 +1,289 @@
import { Injectable, Logger } from '@nestjs/common';
import { ConnectionService } from '../../global/database/connection.service';
import { NatsService } from '../../global/nats/nats.service';
import { StageExecutorService } from './stage-executor.service';
import {
Pipelinedef,
Pipeline,
Stagedef,
Stage,
PipelineDocument,
} from '../../global/database/schema';
export type TriggerType = 'api' | 'schedule' | 'nats';
export interface PipelineExecutionSummary {
pipelineId: string;
pipelineDefId: string;
status: 'completed' | 'failed' | 'partial';
triggeredBy: TriggerType;
stageResults: {
stageName: string;
status: string;
error?: string;
}[];
startedAt: Date;
completedAt?: Date;
}
@Injectable()
export class PipelineEngineService {
private readonly logger = new Logger(PipelineEngineService.name);
constructor(
private readonly connectionService: ConnectionService,
private readonly natsService: NatsService,
private readonly stageExecutor: StageExecutorService,
) {}
/**
* Pipelinedef -> Pipeline 인스턴스 생성 및 실행
*/
async createAndExecutePipeline(
pipelineDefId: string,
triggeredBy: TriggerType,
): Promise<PipelineDocument> {
const startedAt = new Date();
// 1. Pipelinedef 조회
const pipelinedefModel =
await this.connectionService.getMoeumModels('pipelinedefs');
const pipelinedef = await pipelinedefModel.findById(pipelineDefId).exec();
if (!pipelinedef) {
throw new Error(`Pipelinedef not found: ${pipelineDefId}`);
}
// 2. Stagedef들 조회
const stagedefModel =
await this.connectionService.getMoeumModels('stagedefs');
const stagedefs = await stagedefModel
.find({ _id: { $in: pipelinedef.stage } })
.exec();
if (stagedefs.length === 0) {
throw new Error(`No stagedefs found for pipelinedef: ${pipelineDefId}`);
}
// 3. Stage 인스턴스들 생성
const stages: Stage[] = stagedefs.map((stagedef) => ({
apiVersion: stagedef.apiVersion,
name: stagedef.name,
schema: stagedef.schema,
taxonomy: stagedef.taxonomy,
executions: [],
status: 'pending' as const,
metadatas: [...stagedef.metadatas],
retryStrategy: {
limit: 3,
delay: 1000,
backoff: 2,
when: ['failed'],
notify: [],
},
}));
// 4. Pipeline 인스턴스 생성
const pipelineModel =
await this.connectionService.getMoeumModels('pipelines');
const pipeline = await pipelineModel.create({
apiVersion: pipelinedef.apiVersion,
name: `pipeline-${Date.now()}`,
stage: stages,
metadatas: [
...pipelinedef.metadatas,
{
key: 'triggeredBy',
value: triggeredBy,
dataType: 'string',
isReserved: true,
},
{
key: 'triggeredAt',
value: startedAt.toISOString(),
dataType: 'string',
isReserved: true,
},
{
key: 'pipelineDefId',
value: pipelineDefId,
dataType: 'string',
isReserved: true,
},
],
});
this.logger.log(`Pipeline created: ${pipeline._id}`);
// 5. 시작 이벤트 발행
await this.natsService.publish('crawler.pipeline.started', {
pipelineId: pipeline._id.toString(),
pipelineDefId,
triggeredBy,
timestamp: startedAt.toISOString(),
});
// 6. Stage들 병렬 실행 (비동기)
this.executeStagesInParallel(
pipeline._id.toString(),
stagedefs as Stagedef[],
triggeredBy,
);
return pipeline;
}
/**
* Stage들을 병렬로 실행
*/
private async executeStagesInParallel(
pipelineId: string,
stagedefs: Stagedef[],
triggeredBy: TriggerType,
): Promise<void> {
const pipelineModel =
await this.connectionService.getMoeumModels('pipelines');
try {
// Promise.allSettled로 모든 Stage 병렬 실행
const results = await Promise.allSettled(
stagedefs.map((stagedef, index) =>
this.stageExecutor.executeStage(pipelineId, index, stagedef),
),
);
// 결과 집계
const stageResults = results.map((r, i) => ({
stageName: stagedefs[i].name,
status: r.status === 'fulfilled' ? r.value.status : 'failed',
error:
r.status === 'rejected'
? r.reason?.message || 'Unknown error'
: undefined,
}));
const allSucceeded = results.every(
(r) => r.status === 'fulfilled' && r.value.status === 'completed',
);
const allFailed = results.every(
(r) =>
r.status === 'rejected' ||
(r.status === 'fulfilled' && r.value.status === 'failed'),
);
let pipelineStatus: 'completed' | 'failed' | 'partial';
if (allSucceeded) {
pipelineStatus = 'completed';
} else if (allFailed) {
pipelineStatus = 'failed';
} else {
pipelineStatus = 'partial';
}
const completedAt = new Date();
// Pipeline 상태 업데이트
await pipelineModel.findByIdAndUpdate(pipelineId, {
$push: {
metadatas: {
$each: [
{
key: 'completedAt',
value: completedAt.toISOString(),
dataType: 'string',
isReserved: true,
},
{
key: 'pipelineStatus',
value: pipelineStatus,
dataType: 'string',
isReserved: true,
},
],
},
},
});
// 완료 이벤트 발행
await this.natsService.publish('crawler.pipeline.completed', {
pipelineId,
status: pipelineStatus,
triggeredBy,
stageResults,
completedAt: completedAt.toISOString(),
});
this.logger.log(
`Pipeline ${pipelineId} completed with status: ${pipelineStatus}`,
);
} catch (error) {
this.logger.error(`Pipeline execution failed: ${pipelineId}`, error);
await this.natsService.publish('crawler.pipeline.failed', {
pipelineId,
error: error instanceof Error ? error.message : 'Unknown error',
timestamp: new Date().toISOString(),
});
}
}
/**
* Pipeline 실행 상태 조회
*/
async getPipelineStatus(pipelineId: string): Promise<Pipeline | null> {
const pipelineModel =
await this.connectionService.getMoeumModels('pipelines');
return pipelineModel.findById(pipelineId).exec();
}
/**
* Pipelinedef 목록 조회
*/
async listPipelinedefs(
page: number = 1,
limit: number = 20,
): Promise<{ items: Pipelinedef[]; total: number }> {
const pipelinedefModel =
await this.connectionService.getMoeumModels('pipelinedefs');
const skip = (page - 1) * limit;
const [items, total] = await Promise.all([
pipelinedefModel.find().skip(skip).limit(limit).exec(),
pipelinedefModel.countDocuments().exec(),
]);
return { items: items as Pipelinedef[], total };
}
/**
* 실행된 Pipeline 목록 조회
*/
async listPipelines(
page: number = 1,
limit: number = 20,
status?: string,
): Promise<{ items: Pipeline[]; total: number }> {
const pipelineModel =
await this.connectionService.getMoeumModels('pipelines');
const skip = (page - 1) * limit;
const filter: Record<string, any> = {};
if (status) {
filter['metadatas'] = {
$elemMatch: { key: 'pipelineStatus', value: status },
};
}
const [items, total] = await Promise.all([
pipelineModel
.find(filter)
.sort({ createdAt: -1 })
.skip(skip)
.limit(limit)
.exec(),
pipelineModel.countDocuments(filter).exec(),
]);
return { items: items as Pipeline[], total };
}
}
@@ -0,0 +1,138 @@
import { Injectable, Logger } from '@nestjs/common';
import { RateLimitConfig } from '../interfaces/crawler-result.interface';
interface TokenBucket {
tokens: number;
lastRefill: number;
config: RateLimitConfig;
}
@Injectable()
export class RateLimiterService {
private readonly logger = new Logger(RateLimiterService.name);
private readonly buckets: Map<string, TokenBucket> = new Map();
private readonly defaultConfig: RateLimitConfig = {
maxTokens: 10, // 최대 10개 토큰
refillRate: 2, // 초당 2개 토큰 보충
};
/**
* 도메인별 Rate Limit 설정
*/
setLimit(domain: string, config: RateLimitConfig): void {
const existing = this.buckets.get(domain);
this.buckets.set(domain, {
tokens: config.maxTokens,
lastRefill: Date.now(),
config,
});
this.logger.log(
`Rate limit set for ${domain}: ${config.maxTokens} tokens, ${config.refillRate} req/s`,
);
}
/**
* 설정된 Rate Limit 조회
*/
getLimit(domain: string): RateLimitConfig {
const bucket = this.buckets.get(domain);
return bucket?.config ?? this.defaultConfig;
}
/**
* 토큰 획득 (대기 필요시 대기)
*/
async acquire(domain: string): Promise<void> {
let bucket = this.buckets.get(domain);
// 버킷이 없으면 기본 설정으로 생성
if (!bucket) {
bucket = {
tokens: this.defaultConfig.maxTokens,
lastRefill: Date.now(),
config: this.defaultConfig,
};
this.buckets.set(domain, bucket);
}
// 토큰 보충
this.refill(bucket);
// 토큰이 없으면 대기
if (bucket.tokens <= 0) {
const waitTime = Math.ceil(1000 / bucket.config.refillRate);
this.logger.debug(`Rate limit hit for ${domain}, waiting ${waitTime}ms`);
await this.delay(waitTime);
this.refill(bucket);
}
// 토큰 소비
bucket.tokens--;
this.logger.debug(
`Token acquired for ${domain}, remaining: ${bucket.tokens}`,
);
}
/**
* 토큰 보충 (시간 경과에 따라)
*/
private refill(bucket: TokenBucket): void {
const now = Date.now();
const elapsed = (now - bucket.lastRefill) / 1000; // 초 단위
const tokensToAdd = elapsed * bucket.config.refillRate;
bucket.tokens = Math.min(
bucket.config.maxTokens,
bucket.tokens + tokensToAdd,
);
bucket.lastRefill = now;
}
/**
* 현재 사용 가능한 토큰 수 조회
*/
getAvailableTokens(domain: string): number {
const bucket = this.buckets.get(domain);
if (!bucket) {
return this.defaultConfig.maxTokens;
}
this.refill(bucket);
return Math.floor(bucket.tokens);
}
/**
* 모든 도메인의 Rate Limit 상태 조회
*/
getAllLimits(): Record<string, { config: RateLimitConfig; available: number }> {
const result: Record<string, { config: RateLimitConfig; available: number }> = {};
for (const [domain, bucket] of this.buckets) {
this.refill(bucket);
result[domain] = {
config: bucket.config,
available: Math.floor(bucket.tokens),
};
}
return result;
}
/**
* 특정 도메인의 Rate Limit 초기화
*/
resetLimit(domain: string): void {
const bucket = this.buckets.get(domain);
if (bucket) {
bucket.tokens = bucket.config.maxTokens;
bucket.lastRefill = Date.now();
this.logger.log(`Rate limit reset for ${domain}`);
}
}
/**
* 대기 함수
*/
private delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
}
+197
View File
@@ -0,0 +1,197 @@
import { Injectable, Logger } from '@nestjs/common';
import { NatsService } from '../../global/nats/nats.service';
import { RetryStrategy } from '../../global/database/schema';
export type RetryableOperation<T> = () => Promise<T>;
export interface RetryResult<T> {
success: boolean;
result?: T;
attempts: number;
errors: string[];
}
@Injectable()
export class RetryService {
private readonly logger = new Logger(RetryService.name);
constructor(private readonly natsService: NatsService) {}
/**
* 재시도 로직과 함께 작업 실행
*/
async executeWithRetry<T>(
operation: RetryableOperation<T>,
strategy: RetryStrategy,
): Promise<T> {
let lastError: Error | null = null;
let currentDelay = strategy.delay;
const errors: string[] = [];
const maxAttempts = strategy.limit + 1;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
this.logger.debug(`Attempt ${attempt}/${maxAttempts}`);
const result = await operation();
// 성공 시 반환
if (attempt > 1) {
this.logger.log(`Operation succeeded on attempt ${attempt}`);
}
return result;
} catch (error) {
lastError = error instanceof Error ? error : new Error(String(error));
errors.push(`Attempt ${attempt}: ${lastError.message}`);
this.logger.warn(`Attempt ${attempt} failed: ${lastError.message}`);
// 재시도 조건 확인
if (!this.shouldRetry(lastError, strategy.when)) {
this.logger.debug(
'Error does not match retry conditions, not retrying',
);
break;
}
// 마지막 시도가 아니면 대기
if (attempt < maxAttempts) {
this.logger.debug(`Waiting ${currentDelay}ms before next attempt`);
await this.delay(currentDelay);
currentDelay = Math.floor(currentDelay * strategy.backoff); // Exponential backoff
}
}
}
// 모든 재시도 실패 - 알림 발송
if (strategy.notify && strategy.notify.length > 0) {
await this.sendFailureNotification(strategy.notify, errors);
}
throw lastError || new Error('Operation failed after all retries');
}
/**
* 재시도 없이 결과를 래핑하여 반환
*/
async executeWithResult<T>(
operation: RetryableOperation<T>,
strategy: RetryStrategy,
): Promise<RetryResult<T>> {
const errors: string[] = [];
let currentDelay = strategy.delay;
const maxAttempts = strategy.limit + 1;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
const result = await operation();
return {
success: true,
result,
attempts: attempt,
errors,
};
} catch (error) {
const errorMessage =
error instanceof Error ? error.message : String(error);
errors.push(`Attempt ${attempt}: ${errorMessage}`);
if (!this.shouldRetry(error as Error, strategy.when)) {
break;
}
if (attempt < maxAttempts) {
await this.delay(currentDelay);
currentDelay = Math.floor(currentDelay * strategy.backoff);
}
}
}
// 모든 재시도 실패
if (strategy.notify && strategy.notify.length > 0) {
await this.sendFailureNotification(strategy.notify, errors);
}
return {
success: false,
attempts: errors.length,
errors,
};
}
/**
* 재시도 조건 확인
*/
private shouldRetry(error: Error, conditions: string[]): boolean {
// 모든 실패에 대해 재시도
if (conditions.includes('failed')) {
return true;
}
const errorMessage = error.message.toLowerCase();
// 특정 에러 타입에 대해서만 재시도
for (const condition of conditions) {
if (errorMessage.includes(condition.toLowerCase())) {
return true;
}
// HTTP 상태 코드 기반 조건 (예: '5xx', '429', '500')
if (condition.match(/^\d{3}$/) && errorMessage.includes(condition)) {
return true;
}
// 5xx 패턴 매칭
if (condition === '5xx' && errorMessage.match(/5\d{2}/)) {
return true;
}
// 4xx 패턴 매칭
if (condition === '4xx' && errorMessage.match(/4\d{2}/)) {
return true;
}
// 네트워크 에러
if (
condition === 'network' &&
(errorMessage.includes('econnrefused') ||
errorMessage.includes('enotfound') ||
errorMessage.includes('timeout') ||
errorMessage.includes('socket'))
) {
return true;
}
}
return false;
}
/**
* 실패 알림 발송
*/
private async sendFailureNotification(
notifyChannels: string[],
errors: string[],
): Promise<void> {
const notification = {
type: 'crawler.execution.failed',
timestamp: new Date().toISOString(),
errors,
message: `Operation failed after ${errors.length} attempts`,
};
for (const channel of notifyChannels) {
try {
await this.natsService.publish(`notifications.${channel}`, notification);
this.logger.log(`Failure notification sent to: ${channel}`);
} catch (error) {
this.logger.error(`Failed to send notification to ${channel}`, error);
}
}
}
/**
* 대기 함수
*/
private delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
}
@@ -0,0 +1,255 @@
import { Injectable, Logger } from '@nestjs/common';
import { ConnectionService } from '../../global/database/connection.service';
import { NatsService } from '../../global/nats/nats.service';
import { CrawlerService } from './crawler.service';
import { RetryService } from './retry.service';
import {
Stagedef,
Execution,
RetryStrategy,
} from '../../global/database/schema';
import {
CrawlResult,
StageExecutionResult,
} from '../interfaces/crawler-result.interface';
const DEFAULT_RETRY_STRATEGY: RetryStrategy = {
limit: 3,
delay: 1000,
backoff: 2,
when: ['failed', '5xx', 'network'],
notify: [],
};
@Injectable()
export class StageExecutorService {
private readonly logger = new Logger(StageExecutorService.name);
constructor(
private readonly connectionService: ConnectionService,
private readonly natsService: NatsService,
private readonly crawlerService: CrawlerService,
private readonly retryService: RetryService,
) {}
/**
* 단일 Stage 실행
*/
async executeStage(
pipelineId: string,
stageIndex: number,
stagedef: Stagedef,
): Promise<StageExecutionResult> {
const pipelineModel =
await this.connectionService.getMoeumModels('pipelines');
// Stage 상태를 'in-progress'로 변경
await pipelineModel.findByIdAndUpdate(pipelineId, {
$set: { [`stage.${stageIndex}.status`]: 'in-progress' },
});
await this.natsService.publish('crawler.stage.started', {
pipelineId,
stageIndex,
stageName: stagedef.name,
timestamp: new Date().toISOString(),
});
const executions: Execution[] = [];
const crawlResults: CrawlResult[] = [];
let finalStatus: 'completed' | 'failed' = 'completed';
try {
// EventSource별로 크롤링 실행
for (const eventSource of stagedef.eventSource) {
const execution = await this.executeEventSource(
pipelineId,
stageIndex,
eventSource,
stagedef,
);
executions.push(execution);
if (execution.status === 'failed') {
finalStatus = 'failed';
}
// 크롤링 결과 추출
const resultMeta = execution.metadatas.find(
(m) => m.key === 'crawlResult',
);
if (resultMeta) {
try {
const result = JSON.parse(resultMeta.value as string);
crawlResults.push(result);
} catch {
// JSON 파싱 실패 무시
}
}
}
// Stage 상태 및 실행 기록 업데이트
await pipelineModel.findByIdAndUpdate(pipelineId, {
$set: {
[`stage.${stageIndex}.status`]: finalStatus,
[`stage.${stageIndex}.executions`]: executions,
},
});
// 완료 이벤트 발행 (Stagedef의 topic으로)
await this.natsService.publish(stagedef.topic, {
pipelineId,
stageIndex,
stageName: stagedef.name,
status: finalStatus,
results: crawlResults.map((r) => ({
url: r.url,
success: r.success,
parsedData: r.parsedData,
})),
timestamp: new Date().toISOString(),
});
this.logger.log(
`Stage ${stagedef.name} completed with status: ${finalStatus}`,
);
} catch (error) {
finalStatus = 'failed';
this.logger.error(`Stage ${stagedef.name} failed`, error);
await pipelineModel.findByIdAndUpdate(pipelineId, {
$set: { [`stage.${stageIndex}.status`]: 'failed' },
$push: {
[`stage.${stageIndex}.executions`]: {
status: 'failed',
metadatas: [
{
key: 'error',
value: error instanceof Error ? error.message : 'Unknown error',
dataType: 'string',
isReserved: true,
},
],
createdAt: new Date(),
updatedAt: new Date(),
},
},
});
}
return {
stageIndex,
stageName: stagedef.name,
status: finalStatus,
crawlResults,
};
}
/**
* 단일 EventSource 실행 (재시도 로직 포함)
*/
private async executeEventSource(
pipelineId: string,
stageIndex: number,
eventSource: any,
stagedef: Stagedef,
): Promise<Execution> {
const startedAt = new Date();
const execution: Execution = {
status: 'in-progress',
metadatas: [
{
key: 'eventSourceName',
value: eventSource.name,
dataType: 'string',
isReserved: true,
},
{
key: 'eventSourceType',
value: eventSource.type,
dataType: 'string',
isReserved: true,
},
{
key: 'startedAt',
value: startedAt.toISOString(),
dataType: 'string',
isReserved: true,
},
],
createdAt: startedAt,
updatedAt: startedAt,
};
// RetryStrategy 추출 (Stagedef 메타데이터에서 또는 기본값)
const retryStrategyMeta = stagedef.metadatas.find(
(m) => m.key === 'retryStrategy',
);
let retryStrategy: RetryStrategy = DEFAULT_RETRY_STRATEGY;
if (retryStrategyMeta) {
try {
retryStrategy =
typeof retryStrategyMeta.value === 'string'
? JSON.parse(retryStrategyMeta.value)
: retryStrategyMeta.value;
} catch {
// 파싱 실패 시 기본값 사용
}
}
try {
// 재시도 로직과 함께 크롤링 실행
const crawlResult = await this.retryService.executeWithRetry(
() => this.crawlerService.crawlFromEventSource(eventSource),
retryStrategy,
);
execution.status = crawlResult.success ? 'completed' : 'failed';
execution.metadatas.push({
key: 'crawlResult',
value: JSON.stringify(crawlResult),
dataType: 'json',
isReserved: false,
});
// MinIO에 결과 저장
if (crawlResult.success) {
try {
const storageKey = await this.crawlerService.saveResultToStorage(
crawlResult,
`${pipelineId}-${stageIndex}`,
);
execution.metadatas.push({
key: 'storageKey',
value: storageKey,
dataType: 'string',
isReserved: false,
});
} catch (error) {
this.logger.warn(`Failed to save result to MinIO: ${error}`);
}
}
} catch (error) {
execution.status = 'failed';
execution.metadatas.push({
key: 'error',
value: error instanceof Error ? error.message : 'Unknown error',
dataType: 'string',
isReserved: true,
});
}
// 완료 시간 기록
const completedAt = new Date();
execution.metadatas.push({
key: 'completedAt',
value: completedAt.toISOString(),
dataType: 'string',
isReserved: true,
});
execution.updatedAt = completedAt;
return execution;
}
}
+327
View File
@@ -0,0 +1,327 @@
import {
Injectable,
Logger,
OnModuleInit,
OnModuleDestroy,
} from '@nestjs/common';
import { SchedulerRegistry } from '@nestjs/schedule';
import { CronJob } from 'cron';
import { NatsConnection, StringCodec, Subscription, connect } from 'nats';
import * as fs from 'fs/promises';
import { ConnectionService } from '../../global/database/connection.service';
import { SecretService } from '../../global/secret/secret.service';
import { PipelineEngineService } from './pipeline-engine.service';
import { Schedule } from '../../global/database/schema';
export interface ScheduledJob {
pipelineDefId: string;
jobName: string;
cronExpression: string;
timezone: string;
}
@Injectable()
export class TriggerService implements OnModuleInit, OnModuleDestroy {
private readonly logger = new Logger(TriggerService.name);
private scheduledJobs: Map<string, ScheduledJob> = new Map();
private natsSubscriptions: Subscription[] = [];
private natsConnection: NatsConnection | null = null;
private readonly stringCodec = StringCodec();
constructor(
private readonly schedulerRegistry: SchedulerRegistry,
private readonly connectionService: ConnectionService,
private readonly pipelineEngine: PipelineEngineService,
private readonly secretService: SecretService,
) {}
async onModuleInit() {
try {
// NATS 연결 초기화
await this.initializeNatsConnection();
// 기존 트리거 로드
await this.loadTriggers();
this.logger.log('TriggerService initialized');
} catch (error) {
this.logger.error('Failed to initialize TriggerService', error);
}
}
async onModuleDestroy() {
// 스케줄 정리
for (const [jobName] of this.scheduledJobs) {
try {
this.schedulerRegistry.deleteCronJob(jobName);
} catch {
// 무시
}
}
this.scheduledJobs.clear();
// NATS 구독 정리
for (const sub of this.natsSubscriptions) {
try {
await sub.drain();
} catch {
// 무시
}
}
this.natsSubscriptions = [];
// NATS 연결 종료
if (this.natsConnection) {
try {
await this.natsConnection.drain();
await this.natsConnection.close();
} catch {
// 무시
}
this.natsConnection = null;
}
this.logger.log('TriggerService destroyed');
}
/**
* NATS 연결 초기화
*/
private async initializeNatsConnection(): Promise<void> {
await this.secretService.ready();
const endpoint = await this.secretService.get('NATS_ENDPOINT');
const user = await this.secretService.get('NATS_USER');
const password = await this.secretService.get('NATS_PASS');
let tlsConfig: { ca?: string } | undefined;
try {
const cacert = (await fs.readFile('internal-service-ca.pem')).toString();
tlsConfig = { ca: cacert };
} catch {
this.logger.warn('TLS certificate not found, connecting without TLS');
}
this.natsConnection = await connect({
name: 'crawler-trigger-service',
servers: [endpoint],
tls: tlsConfig,
user,
pass: password,
maxReconnectAttempts: -1,
reconnectTimeWait: 500,
reconnect: true,
pingInterval: 20000,
timeout: 5000,
});
this.logger.log('NATS connection established for TriggerService');
}
/**
* 모든 트리거 로드 (스케줄 + 이벤트 구독)
*/
async loadTriggers(): Promise<void> {
const pipelinedefModel =
await this.connectionService.getMoeumModels('pipelinedefs');
const pipelinedefs = await pipelinedefModel.find().exec();
for (const pipelinedef of pipelinedefs) {
// 스케줄 트리거 확인 (메타데이터에서)
const scheduleMeta = pipelinedef.metadatas?.find(
(m) => m.key === 'schedule',
);
if (scheduleMeta) {
try {
const schedule: Schedule =
typeof scheduleMeta.value === 'string'
? JSON.parse(scheduleMeta.value)
: scheduleMeta.value;
await this.registerScheduleTrigger(
pipelinedef._id.toString(),
schedule,
);
} catch (error) {
this.logger.warn(
`Failed to parse schedule for pipelinedef ${pipelinedef._id}`,
);
}
}
// Kafka/NATS 이벤트 트리거
if (
pipelinedef.trigger?.type === 'kafka-event' &&
pipelinedef.trigger.kafkaEvent?.topic
) {
await this.registerNatsTrigger(
pipelinedef._id.toString(),
pipelinedef.trigger.kafkaEvent.topic,
);
}
}
this.logger.log(
`Loaded ${this.scheduledJobs.size} schedule triggers and ${this.natsSubscriptions.length} event triggers`,
);
}
/**
* 스케줄 트리거 등록
*/
async registerScheduleTrigger(
pipelineDefId: string,
schedule: Schedule,
): Promise<void> {
const jobName = `schedule-${pipelineDefId}`;
// 기존 작업이 있으면 제거
try {
if (this.schedulerRegistry.doesExist('cron', jobName)) {
this.schedulerRegistry.deleteCronJob(jobName);
this.scheduledJobs.delete(jobName);
}
} catch {
// 무시
}
const job = new CronJob(
schedule.cron,
async () => {
this.logger.log(`Scheduled trigger fired for pipeline: ${pipelineDefId}`);
try {
await this.pipelineEngine.createAndExecutePipeline(
pipelineDefId,
'schedule',
);
} catch (error) {
this.logger.error(
`Scheduled trigger failed: ${pipelineDefId}`,
error,
);
}
},
null,
true,
schedule.timezone || 'Asia/Seoul',
);
this.schedulerRegistry.addCronJob(jobName, job);
this.scheduledJobs.set(jobName, {
pipelineDefId,
jobName,
cronExpression: schedule.cron,
timezone: schedule.timezone || 'Asia/Seoul',
});
this.logger.log(
`Registered schedule trigger: ${jobName} (${schedule.cron})`,
);
}
/**
* NATS 이벤트 트리거 등록
*/
async registerNatsTrigger(
pipelineDefId: string,
topic: string,
): Promise<void> {
if (!this.natsConnection) {
this.logger.warn('NATS connection not available');
return;
}
const subscription = this.natsConnection.subscribe(topic, {
callback: async (err, msg) => {
if (err) {
this.logger.error(`NATS subscription error on ${topic}`, err);
return;
}
this.logger.log(`NATS trigger received on ${topic}`);
try {
// 메시지 페이로드 파싱 (필요시 사용)
let payload: any = {};
try {
payload = JSON.parse(this.stringCodec.decode(msg.data));
} catch {
// JSON 아닌 경우 무시
}
await this.pipelineEngine.createAndExecutePipeline(
pipelineDefId,
'nats',
);
} catch (error) {
this.logger.error(
`NATS trigger processing failed: ${pipelineDefId}`,
error,
);
}
},
});
this.natsSubscriptions.push(subscription);
this.logger.log(`Registered NATS trigger: ${topic} -> ${pipelineDefId}`);
}
/**
* API 기반 수동 트리거
*/
async triggerPipelineManually(pipelineDefId: string): Promise<string> {
const pipeline = await this.pipelineEngine.createAndExecutePipeline(
pipelineDefId,
'api',
);
return pipeline._id.toString();
}
/**
* 스케줄 제거
*/
async unregisterScheduleTrigger(pipelineDefId: string): Promise<void> {
const jobName = `schedule-${pipelineDefId}`;
try {
if (this.schedulerRegistry.doesExist('cron', jobName)) {
this.schedulerRegistry.deleteCronJob(jobName);
this.scheduledJobs.delete(jobName);
this.logger.log(`Unregistered schedule trigger: ${jobName}`);
}
} catch {
// 무시
}
}
/**
* 현재 등록된 스케줄 목록 조회
*/
getRegisteredSchedules(): ScheduledJob[] {
return Array.from(this.scheduledJobs.values());
}
/**
* 트리거 재로드
*/
async reloadTriggers(): Promise<void> {
// 기존 트리거 정리
for (const [jobName] of this.scheduledJobs) {
try {
this.schedulerRegistry.deleteCronJob(jobName);
} catch {
// 무시
}
}
this.scheduledJobs.clear();
for (const sub of this.natsSubscriptions) {
try {
await sub.drain();
} catch {
// 무시
}
}
this.natsSubscriptions = [];
// 다시 로드
await this.loadTriggers();
}
}
+8 -8
View File
@@ -1,7 +1,7 @@
import { Injectable, Scope, OnModuleInit } from '@nestjs/common'; import { Injectable, Scope, OnModuleInit } from '@nestjs/common';
import mongoose from 'mongoose'; import mongoose from 'mongoose';
import { SecretService } from '../secret/secret.service.js'; import { SecretService } from '../secret/secret.service.js';
import { moeumModelDefinitions, MoeumModelTypes } from './schema.js'; import { CrawlerModelDefinitions, CrawlerModelTypes } from './schema.js';
@Injectable({ scope: Scope.DEFAULT }) @Injectable({ scope: Scope.DEFAULT })
export class ConnectionService implements OnModuleInit { export class ConnectionService implements OnModuleInit {
@@ -31,19 +31,19 @@ export class ConnectionService implements OnModuleInit {
return this.connection; return this.connection;
} }
public async getMoeumModels<K extends keyof MoeumModelTypes>( public async getMoeumModels<K extends keyof CrawlerModelTypes>(
modelKey: K, modelKey: K,
): Promise<mongoose.Model<MoeumModelTypes[K]>> { ): Promise<mongoose.Model<CrawlerModelTypes[K]>> {
if (this.moeumModels) if (this.moeumModels)
return this.moeumModels.get(modelKey) as mongoose.Model< return this.moeumModels.get(modelKey) as mongoose.Model<
MoeumModelTypes[K] CrawlerModelTypes[K]
>; >;
const moeumModels = new Map<K, mongoose.Model<MoeumModelTypes[K]>>(); const moeumModels = new Map<K, mongoose.Model<CrawlerModelTypes[K]>>();
// const connection = await this.getTenantConnection(tenantId); // const connection = await this.getTenantConnection(tenantId);
const connection = await this.getConnection(); const connection = await this.getConnection();
moeumModelDefinitions.forEach((modelDefinition) => { CrawlerModelDefinitions.forEach((modelDefinition) => {
// 이미 생성된 경우 컴파일 스킵. // 이미 생성된 경우 컴파일 스킵.
if ( if (
Object.prototype.hasOwnProperty.call( Object.prototype.hasOwnProperty.call(
@@ -59,7 +59,7 @@ export class ConnectionService implements OnModuleInit {
} else { } else {
moeumModels.set( moeumModels.set(
modelDefinition.name as K, modelDefinition.name as K,
connection.model( connection.model<CrawlerModelTypes[K]>(
modelDefinition.name, modelDefinition.name,
modelDefinition.schema, modelDefinition.schema,
modelDefinition.schemaName, modelDefinition.schemaName,
@@ -69,7 +69,7 @@ export class ConnectionService implements OnModuleInit {
}); });
this.moeumModels = moeumModels; this.moeumModels = moeumModels;
return moeumModels.get(modelKey) as mongoose.Model<MoeumModelTypes[K]>; return moeumModels.get(modelKey) as mongoose.Model<CrawlerModelTypes[K]>;
} }
public async createSession( public async createSession(
File diff suppressed because it is too large. Load diff