Files
crawler/env-vars/dev.env
T
dahyeon a0d218a169 Implements a comprehensive web crawler microservice
Introduces a complete NestJS-based web crawler microservice. This release establishes a robust system for web data extraction through a flexible Pipeline/Stage architecture.

Key features include:
- Supports both static (axios, cheerio) and dynamic (Playwright) page crawling.
- Implements diverse triggering options: cron-based scheduling, NATS event-driven, and manual API calls.
- Enhances system resilience with domain-specific rate limiting and configurable exponential backoff retry mechanisms.
- Provides dedicated API endpoints and database schemas for managing pipelines, stages, and execution history.
- Ensures data persistence by saving crawl results to MinIO.

The `README.md` is entirely rewritten to provide comprehensive documentation on features, architecture, setup, API usage, and troubleshooting. Core dependencies for crawling, scheduling, and API validation are added, and database models are refactored to align with the new crawler system's data structures.
2025-12-21 14:29:25 +00:00

1 line
57 B
Bash

INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110