searxNcrawl

Open-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key.

What is searxNcrawl?

Open-source MCP server that gives AI agents web search (SearXNG) and clean Markdown crawling of pages, whole sites and logged-in areas. Self-hosted, no API key. Firecrawl-compatible API planned.

Overview of searxNcrawl

What it is

searxNcrawl is an open-source MCP server and CLI toolkit that gives AI agents web search and clean Markdown crawling. It is built on Crawl4AI. Search queries configured providers, SearXNG and/or You.com, in parallel and merges the results. It runs on your own hardware, needs no API key with SearXNG and succeeds searxng-mcp.

Key features

  • Web search: queries your own SearXNG instance and/or the You.com API in parallel; results are merged and attributed to their providers, filtered by language and time range.
  • Page crawl: one or many URLs to Markdown or JSON, with repeated blocks removed and links optionally stripped.
  • Site crawl: whole sites with depth and page limits, tuned for documentation.
  • Authenticated crawls: capture a login session once with crawl-capture, then crawl behind the login.
  • STDIO and HTTP: plug into local MCP clients, or run an HTTP endpoint with opt-in Host/Origin checks and configurable CORS.
  • CLI and Python API: crawl, search and crawl-capture on the command line, or import it in Python.

Use cases

searxNcrawl is designed for AI agents that need to search the web and crawl pages into clean Markdown. It is suitable for self-hosted setups where no API key is desired for search, and for crawling documentation sites or pages behind a login. It can be used with MCP clients such as Claude Code, Codex, Cursor, VS Code, and Claude Desktop.

Getting started

You need uv and, for the search tool, at least one search provider: a SearXNG instance with JSON output enabled and/or a You.com API key (YDC_API_KEY). Crawling works without either.bashSEARXNG_URL=http://your-searxng:8888 uvx searxncrawlFor per-agent setup, see the repository README.

License and pricing

License: MIT. Pricing: open source (no cost).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/dasdigitalemomentumsearxncrawl.svg)](https://yunodb.com/tools/dasdigitalemomentumsearxncrawl?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/dasdigitalemomentumsearxncrawl?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/dasdigitalemomentumsearxncrawl.svg" alt="Featured on YunoDB" height="20"></a>