Implements a comprehensive web crawler microservice
Introduces a complete NestJS-based web crawler microservice. This release establishes a robust system for web data extraction through a flexible Pipeline/Stage architecture. Key features include: - Supports both static (axios, cheerio) and dynamic (Playwright) page crawling. - Implements diverse triggering options: cron-based scheduling, NATS event-driven, and manual API calls. - Enhances system resilience with domain-specific rate limiting and configurable exponential backoff retry mechanisms. - Provides dedicated API endpoints and database schemas for managing pipelines, stages, and execution history. - Ensures data persistence by saving crawl results to MinIO. The `README.md` is entirely rewritten to provide comprehensive documentation on features, architecture, setup, API usage, and troubleshooting. Core dependencies for crawling, scheduling, and API validation are added, and database models are refactored to align with the new crawler system's data structures.
This commit is contained in:
1 parent
60f8e249fa
commit
a0d218a169
26 files changed
+4367
-281
No files matched your search
+1
-1
@@ -1 +1 @@
|
||||
INFISICAL_PROJECT_ID=1ebbd0b9-7f76-40fb-91c3-9cf0f12a1510
|
||||
INFISICAL_PROJECT_ID=a33f45fb-7564-4e4e-9931-96201395d110
|
||||
Reference in new issue
Block a user