Description

This MCP server integrates web crawling capabilities from Crawl4AI with RAG (Retrieval Augmented Generation) functionality, enabling AI agents and coding assistants to crawl websites, store content in Supabase vector databases, and perform semantic searches over the crawled data. Developed by Cole Medin, it provides four essential tools: single page crawling, intelligent recursive website crawling with automatic URL type detection, source retrieval, and vector-based RAG queries with optional source filtering. The implementation is designed to be incorporated into Archon as a knowledge engine for AI coding assistants, with future plans to support different embedding models and local deployment options using Ollama.

Installation

git clone https://github.com/coleam00/mcp-crawl4ai-rag

Claude Desktop Configuration

Add this configuration to your Claude Desktop config file to enable this MCP server:

Config file location: ~/Library/Application Support/Claude/claude_desktop_config.json

Config file location: %APPDATA%\Claude\claude_desktop_config.json

Config file location: ~/.config/Claude/claude_desktop_config.json

claude_desktop_config.json

{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "node",
      "args": [
        "path/to/server.js"
      ]
    }
  }
}