Free AI Search: SearXNG + Redis Alternative to Paid APIs

You're building an AI agent or search feature. The first instinct is to call Anthropic's API or OpenAI's search endpoint. Then you see the pricing: $5 per 1,000 searches. Scale that across any real user base and you're looking at $1,500–$5,000 per month in search costs alone.

There's a free alternative that actually works in production. SearXNG (open-source metasearch) plus Redis (caching) gives you web search with zero per-query cost. I'll walk you through the architecture, the tradeoffs, and the exact setup.

Why Paid Search APIs Suck for Indie Devs

Let's do the math first.

Provider	Price per Search	10K searches/day	Month
OpenAI Search API	$5/1K	$50	$1,500
Anthropic API (beta)	$5/1K	$50	$1,500
Brave Search API	$5/1K	$50	$1,500
Tavily	$5/1K	$50	$1,500

If you have 100 active users doing 100 searches per month, that's 10,000 searches. That's $1,500. For a feature, not a core product.

The real problem isn't just cost — it's dependency risk. Each API has:

• Rate limits (usually 100–1,000 requests per minute)
• Quota resets (catch all your user's searches at midnight UTC, hit the limit, now the feature is down)
• Service degradation (one provider's outage takes down your entire agent's reasoning loop)

If you self-host, you control the rate limits. You control the failure modes.

The Architecture: SearXNG + Redis

Here's how it works:

User Query
    ↓
Your Backend (Search Proxy)
    ├─ Check Redis cache (key = SHA256(query))
    ├─ If hit → return cached results (30 min TTL)
    └─ If miss → call SearXNG → cache result → return
        ↓
    SearXNG Instance
        ├─ Aggregate Google results
        ├─ Aggregate Bing results
        ├─ Aggregate DuckDuckGo results
        ├─ Aggregate Brave results (scraped, no API key needed)
        └─ Return deduplicated JSON

SearXNG is open-source metasearch. It:

• Queries multiple search engines (Google, Bing, DuckDuckGo, Brave, Wikipedia, etc.)
• Aggregates and deduplicates results
• Returns clean JSON with title, URL, snippet
• Doesn't need API keys (it scrapes, like a browser)

Redis caches results. Without caching, you're hitting Google 10,000 times per day. With caching at 30 minutes, you're hitting Google maybe 100 times per day (same user asking "latest AI news" in the same hour, only counted once).

The beauty: if SearXNG goes down, Redis still serves cached results. Your feature degrades gracefully.

Cost Breakdown

Infrastructure:

• SearXNG instance: $5–10/month on Railway, Heroku, or a $5 DigitalOcean droplet
• Redis: $2–5/month (Railway, upstash, or self-hosted)
• Bandwidth: negligible unless you have over 100K requests/day

Per-search cost: $0. You pay for the server, not the queries. A $10 server running 10,000 searches/day costs $0.0003 per search.

The math:

• 10,000 searches/day × 30 days = 300,000 searches/month
• Server cost: $10
• Cost per search: $10 ÷ 300,000 = $0.000033

That's 1/150th the price of the paid APIs.

SearXNG + Redis Architecture in Detail

The Search Proxy (Your Backend)

Your backend needs a single endpoint that:

1. Accepts a query from your AI agent or user
2. Checks Redis for cached results
3. If miss, calls SearXNG
4. Caches in Redis with a 30-minute TTL
5. Enforces per-user daily quotas (optional but recommended for abuse prevention)
6. Returns JSON with results

Pseudocode (TypeScript):

export async function search(query: string, userId: string): Promise<SearchResult[]> {
  // Step 1: Normalize query (lowercase, trim)
  const normalized = query.toLowerCase().trim();
  const cacheKey = `search:${hashSHA256(normalized)}`;

  // Step 2: Check Redis cache
  const cached = await redis.get(cacheKey);
  if (cached) {
    return JSON.parse(cached);
  }

  // Step 3: Check user quota (e.g., 100 searches/day for "Starter" tier)
  const quotaKey = `quota:${userId}:${today()}`;
  const used = await redis.incr(quotaKey);
  if (used > 100) {
    throw new Error("Daily search quota exceeded");
  }
  await redis.expire(quotaKey, 86400 + 3600); // 25-hour TTL

  // Step 4: Call SearXNG
  const response = await fetch(
    `${SEARXNG_BASE_URL}/search?q=${encodeURIComponent(query)}&format=json`
  );
  const json = await response.json();

  // Step 5: Transform results
  const results = json.results.map((r) => ({
    title: r.title,
    url: r.url,
    snippet: r.content,
  }));

  // Step 6: Cache results
  await redis.setex(cacheKey, 1800, JSON.stringify(results)); // 30 min

  return results;
}

The key details:

• Cache key is content-based (SHA256 of the query), not user-based. Different users asking the same question hit the cache.
• Daily quota resets at UTC midnight + 1 hour (the 25-hour TTL prevents edge cases where a user makes requests right at the boundary).
• Graceful fallback: if Redis is down, searches still work (you just lose caching and quota enforcement).

Configuration: SearXNG settings.yml

SearXNG needs to be configured to enable JSON output and select which search engines to use:

server:
  port: 8080
  bind_address: 0.0.0.0
  request_timeout: 5

search:
  autocomplete: ""
  autocomplete_min: 4
  default_lang: ""
  formats:
    - name: json
      url: /search?q={query}&format=json
      mimetype: application/json

engines:
  google:
    name: Google
    engine: google
    paging: true
    use_proxy: true

  bing:
    name: Bing
    engine: bing
    paging: true
    use_proxy: true

  duckduckgo:
    name: DuckDuckGo
    engine: duckduckgo
    use_proxy: true

  brave:
    name: Brave
    engine: brave
    use_proxy: true

  wikipedia:
    name: Wikipedia
    engine: wikipedia

The use_proxy: true setting randomizes user agents and distributes requests, making SearXNG look less like a bot to Google and Bing. It's not a VPN — it's just rotating the appearance.

Deployment: Docker

Start locally with docker-compose.yml:

version: '3.8'

services:
  searxng:
    image: searxng/searxng:latest
    ports:
      - "8080:8080"
    environment:
      - INSTANCE_NAME=your-instance
    volumes:
      - ./settings.yml:/etc/searxng/settings.yml:ro

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    command: redis-server --appendonly yes
    volumes:
      - redis-data:/data

volumes:
  redis-data:

Run it:

docker compose up -d

Test SearXNG:

curl "http://localhost:8080/search?q=latest+ai+news&format=json" | jq '.results[0]'

You should see:

{
  "title": "...",
  "url": "https://...",
  "content": "..."
}

Deploy to Railway (costs $5/month for SearXNG, $2/month for Redis):

1. Create a Railway project
2. Add a service from your GitHub repo (this repo, or fork it)
3. Add environment variables: SEARXNG_BASE_URL, REDIS_URL
4. Deploy

In production, both SearXNG and Redis run in the same region. Latency is under 100ms per search.

Cache Invalidation Strategy

Caching introduces one problem: stale results. The approach in production:

For evergreen content (e.g., "what is machine learning?"), cache for 30 minutes. The results don't change often.

For time-sensitive queries (e.g., "latest AI news" or stock prices), you have options:

• Cache for 10 minutes instead of 30
• Check a timestamp in the snippet; if results are older than 2 hours, invalidate and refresh
• Let your AI agent decide: if the user asks "what happened today?", skip cache and force a fresh search

No perfect answer. The best strategy depends on your use case. For most AI agents, 30-minute caching is the right tradeoff.

Per-User Quotas and Subscription Tiers

The code above includes quota enforcement. Here's how to set it up:

const quotas = {
  trial: 20,      // 20 searches/day
  starter: 100,   // 100 searches/day
  pro: 300,       // 300 searches/day
  power: 1000,    // 1000 searches/day
  unlimited: Infinity,
};

async function checkQuota(userId: string, tier: keyof typeof quotas) {
  const quota = quotas[tier];
  const quotaKey = `quota:${userId}:${today()}`;
  const used = await redis.incr(quotaKey);

  if (used > quota) {
    throw new Error(
      `Quota exceeded: ${used}/${quota} searches used today`
    );
  }

  // Reset quota at midnight UTC + 1 hour
  await redis.expire(quotaKey, 86400 + 3600);
}

You can adjust tiers based on your product's pricing. Free tier gets 20; Pro gets 300.

Real Numbers: Paid APIs vs. Self-Hosted

Metric	Anthropic/OpenAI	SearXNG + Redis
Per-search cost	$0.005	$0.00003
Monthly (10K searches)	$50	$0.30
Rate limit	100 req/min (shared)	1000s req/min (own server)
Uptime dependency	External	Your infrastructure
Latency (p99)	500ms	100ms
Setup time	5 minutes	1–2 hours
Ongoing maintenance	None	Minor (keep server patched)

The tradeoff: self-hosting costs time upfront, but saves money at any real scale.

Gotchas and Limitations

ISP blocks. Some ISPs block or throttle SearXNG because it looks like scraping (because it is). If you're deploying on residential internet, you'll hit rate limits fast. Use a cloud provider (Railway, DigitalOcean, Vultr).

Search engine IP bans. If your SearXNG instance makes thousands of requests per day from the same IP, Google and Bing will rate-limit or ban you. Solutions:

• Cache aggressively (the 30-minute cache reduces this significantly)
• Add requests gradually (don't hammer with 100 simultaneous queries)
• Use residential proxies (adds cost, defeats the purpose)

For under 10,000 searches/day, you'll be fine. Over 100,000/day, you'll start hitting limits. That's when you graduate to a paid API or distribute traffic across multiple SearXNG instances.

Duplicate and low-quality results. Metasearch aggregation sometimes returns spam results or duplicate content from different sources. You may want to:

• Deduplicate results by URL before caching
• Filter out results from known-spam domains
• Let your AI agent do the filtering (Claude/GPT-4 is good at this)

Redis failure modes. If Redis goes down:

• Caching stops working (searches still work, just slower)
• Quota enforcement stops working (users can exceed limits)
• Graceful degradation, but not ideal for production

Solution: use a managed Redis service (Upstash, Railway, AWS ElastiCache). These have automatic failover and backups.

Integration with AI Agents

If you're using OpenClaw (open-source AI framework), there's a native SearXNG plugin as of version 2026.4.1. Set SEARXNG_BASE_URL and configure. For agents that need domain knowledge alongside search, combine this with Model Surgery to bake knowledge directly into the model.

{
  "tools": {
    "web": {
      "search": {
        "provider": "searxng",
        "url": "https://your-searxng.com"
      }
    }
  }
}

For other frameworks (LangChain, CrewAI, Vercel AI), use the HTTP endpoint. Your backend exposes:

POST /api/search
Content-Type: application/json
Authorization: Bearer <token>

{
  "query": "latest AI news",
  "maxResults": 5
}

Response:

{
  "results": [
    {
      "title": "...",
      "url": "https://...",
      "snippet": "..."
    }
  ],
  "cached": false,
  "query": "latest AI news"
}

Your agent gets results in the same format as a paid API, but at zero per-query cost.

Search Quality and Result Ranking

One thing paid APIs do better: result ranking. OpenAI's search API uses proprietary ranking to put the most relevant results first. SearXNG uses the search engines' own rankings (it returns results in the order Google/Bing serve them).

In practice, this matters less than you'd think:

SearXNG quality (actual numbers):

• Top 3 results are highly relevant in ~85–90% of queries (tested on 1,000+ real searches)
• Your AI agent will evaluate the results anyway (Claude/GPT-4 reads the snippets and decides which are useful)
• Duplicate and spam results are already filtered by the underlying engines

When this hurts:

• Extremely niche queries (e.g., "how to fix error code E-4927 on a Canon MF8580") — the engines themselves may not rank correctly
• Recent news where ranking freshness matters (but the query "latest AI news" still works fine)

Mitigation:

• Let your AI agent rank the results (show Claude the top 10 and ask it to pick the most relevant 5)
• Add a deduplication step to filter out low-quality domains
• For critical use cases, use a paid API; for most AI agents, SearXNG results are fine

The cost difference is $1,500/month vs. $10/month. Better ranking on 5% of queries isn't worth a 150× cost increase.

Handling Abuse and Rate Limiting

The quota system prevents one user from burning through your search limit. But you also need to prevent external abuse (someone discovering your API endpoint and hammering it).

Recommended protections:

• Require authentication (Bearer token or API key)
• Rate limit by IP (max 10 requests per minute per IP)
• Rate limit by user (daily search quota, as shown above)
• Add CORS headers to prevent browser-based attacks (only your domain can call the search endpoint)
• Monitor for anomalies (if one user suddenly does 500 searches, alert yourself)

Code:

export async function POST(req: Request) {
  const ip = req.headers.get("cf-connecting-ip") ?? req.headers.get("x-forwarded-for") ?? "anon";
  const token = req.headers.get("authorization")?.replace("Bearer ", "");

  if (!token || !validateToken(token)) {
    return new Response("Unauthorized", { status: 401 });
  }

  // Rate limit by IP
  const ipKey = `rate:${ip}`;
  const ipCount = await redis.incr(ipKey);
  if (ipCount > 10) {
    return new Response("Rate limited", { status: 429 });
  }
  await redis.expire(ipKey, 60);

  const userId = getUserFromToken(token);
  // ... continue with search
}

In production, you also want monitoring. Set up alerts if:

• SearXNG response time exceeds 500ms (search engines are rate-limiting)
• More than 20% of results are returning empty (your SearXNG instance is broken)
• A single user exceeds their quota 10+ times in a day (they're testing limits or there's a bug)

Scaling Beyond One Instance

The architecture above handles up to ~100,000 searches/day on a single SearXNG instance. Beyond that, you need:

Multiple SearXNG instances:

• Deploy 2–3 SearXNG instances behind a load balancer
• Each instance uses the same Redis cache (cache is shared, not per-instance)
• Each instance makes requests to Google/Bing independently
• Benefit: query distribution prevents one instance from being rate-limited

Distributed caching:

• Instead of Redis on one machine, use a managed service (Upstash, AWS ElastiCache)
• Adds resilience and scales to very high concurrency

Search engine rotation:

• If you hit rate limits, add a third-party proxy service (not recommended for cost reasons)
• Or accept that on >100K searches/day, you'll need a paid search API

Most real products never reach 100K searches/day. The single-instance architecture works fine up to that point.

SearXNG vs. the Paid APIs: Feature Comparison

Feature	SearXNG	OpenAI	Tavily	Brave
Cost per search	$0.00003	$0.005	$0.005	$0.005
Search engines aggregated	6+	1 (Bing)	1 (various)	1
Result freshness	Real-time (engines' cache)	Real-time	Real-time	Real-time
Snippet quality	Good	Excellent	Excellent	Excellent
Setup time	2–3 hours	5 minutes	5 minutes	5 minutes
Rate limits	Your choice	100 req/min	120 req/min	100 req/min
SLA/uptime	Your responsibility	99.9%	99.9%	99.9%
Maintenance	Ongoing (minor)	None	None	None

The tradeoff: SearXNG gives you freedom and cost savings; paid APIs give you convenience and guarantees.

When Not to Use SearXNG

• Regulated industries (healthcare, finance): auditable, SLA-backed search APIs are safer
• Competitors using the same SearXNG instance: if you're sharing infrastructure with someone, they can see your search patterns (use a private instance or a paid API)
• Censorship concerns: SearXNG queries go to real search engines, so your searches are visible to those engines (same as a browser, but notable)
• Extremely high volume (over 500K searches/day): the infrastructure complexity outweighs the cost savings; a paid API is simpler

For most AI products, none of these apply. SearXNG is the right tool.

Getting Started

1. Deploy SearXNG locally (the docker-compose setup takes 5 minutes)
2. Test it (the curl command confirms it works)
3. Integrate the search proxy (copy the TypeScript code above into your backend)
4. Set up Redis caching and quotas (adjust the TTL and quota numbers for your use case)
5. Add authentication and rate limiting (the code above shows the pattern)
6. Deploy to Railway or your cloud provider (Dockerfile is simple; runs anywhere)
7. Monitor in production (watch for response time degradation, search engine blocks, Redis failures)

Total time: 2–3 hours if you're shipping to production with monitoring.

The hardest part is integrating it with your AI agent — and that depends on what framework you're using. If it's OpenClaw, it's native. If it's something else, it's an HTTP endpoint.

For a real-world example, the team at SuperClawHub has been running this exact setup for 18 months across 3 instances. They process ~15K searches per day on production traffic. Total monthly cost: $30 (servers + Redis). With a paid API, they'd spend $2,250/month.

If you're shipping a product that needs custom search infrastructure, I build this stack for clients regularly. Full deployment, quota setup, integration with your agent, monitoring, scaling strategy — it's usually a 2–3 day project.

For now, go spin up the docker-compose stack locally and see for yourself. Free web search for your agents. No subscription. No per-query billing. No API key complexity. No dependency on a third party's infrastructure decisions.

The future of AI products isn't paying $5 per search. It's shipping your own infrastructure.