searcharvester

Self-hosted search + markdown harvester for AI agents. SearXNG (100+ engines) + FastAPI + trafilatura. Tavily-compatible /search plus /extract with size presets

What is searcharvester?

Self-hosted search + markdown harvester for AI agents. SearXNG (100+ engines) + FastAPI + trafilatura. Tavily-compatible /search plus /extract with size presets and pagination. One-command Docker Compose.

Overview of searcharvester

What it is

Searcharvester is a self-hosted software stack that combines search, content extraction, and deep research capabilities for AI agents. It integrates SearXNG (which provides access to over 100 search engines), FastAPI, and trafilatura to offer a Tavily-compatible /search endpoint and an /extract endpoint that converts URLs to clean markdown. The project is designed to be deployed with a single Docker Compose command.

Key features

  • Tavily-compatible /search endpoint using SearXNG (100+ engines), including image and video search- /extract endpoint that converts URLs to clean markdown via trafilatura, readability, and Defuddle, with size presets and pagination- /research endpoint that uses a deep research agent team to answer questions and produce markdown reports with citations
  • Web UI on port :9762 for starting jobs, selecting a model per agent role, following agents live, reading reports, and browsing sources
  • Search settings in the UI to configure SearXNG engines, proxies, and timeouts, applied with a SearXNG restart
  • No API keys for search, no quotas, fully self-hosted; the adapter image is available on GHCR

Use cases

Searcharvester is intended for AI agents that need self-hosted web search and content extraction without relying on external API quotas or keys. It can be used to gather information from multiple sources, produce structured markdown reports, and monitor agent research processes in real time through the provided web interface.

Getting started

git clone https://github.com/vakovalskii/searcharvester.gitcd searcharvestercp config.example.yaml config.yaml# Change server.secret_key (32+ chars)cat > .env

License and pricing

Searcharvester is open source under the AGPL-3.0 license. Pricing is not stated; it is self-hosted.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/vakovalskiisearcharvester.svg)](https://yunodb.com/tools/vakovalskiisearcharvester?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/vakovalskiisearcharvester?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/vakovalskiisearcharvester.svg" alt="Featured on YunoDB" height="20"></a>