RAG & Data Ingestion Tools

Two open-source projects, CocoIndex and Firecrawl, provide streamlined ETL, scraping and formatting utilities for RAG and knowledge-base construction.
Key insights
CocoIndex simplifies data ingestion for RAG
CocoIndex is an open-source high-performance data conversion framework tailored for AI use cases. It provides end-to-end pipelines with ~100 lines of Python to define file reading, chunking, embedding generation and vector DB insertion. It supports local files, S3, Google Drive and targets like Postgres, Qdrant and LanceDB and includes components for PDF parsing, embedding, chunking and knowledge-graph building.
Firecrawl automates web-to-LLM-friendly content extraction
Firecrawl converts websites/pages into clean markdown or structured JSON for AI consumption. Features include single-page scrape, full-site crawl, structured extraction via prompts/schema, site mapping, change monitoring, PDF handling, SDKs for Python/JS, and integrations with LangChain/LlamaIndex. Firecrawl is open-source (AGPL-3.0) and supports self-hosting.