fast-crawl

Appearance

Fast-Crawl v1 API Documentation ​

Overview ​

The Fast-Crawl v1 API provides unified, high-performance search and web scraping capabilities via RESTful endpoints. It aggregates results from multiple search providers (Google, Bing) and supports content extraction from result URLs. All endpoints are versioned under /v1/.

Authentication ​

  • JWT: Pass a valid JWT token in the Authorization: Bearer <token> header.
  • API Key: Pass your API key in the x-api-key header.
  • CORS: All endpoints support CORS with configurable origins and methods.

Endpoints ​

1. Health Check ​

  • Route: GET /v1/health
  • Description: Returns API health status.
  • Response:
    json
    { "status": "ok", "uptime": 123.45 }

2. Search ​

  • Route: POST /v1/search

  • Description: Aggregates search results from multiple providers.

  • Request Body:

    FieldTypeRequiredDescription
    querystringYesSearch term
    providersstring[]NoList of providers (default: all)
    scrapebooleanNoScrape content from result URLs (default: false)
    countnumberNoNumber of results to return

    Example:

    json
    {
  "query": "openai api",
  "providers": ["google", "bing"],
  "scrape": false,
  "count": 10
}

  • Response:

    json
    {
  "query": "openai api",
  "providers": ["google", "bing"],
  "items": 10,
  "results": [
    {
      "title": "OpenAI API",
      "link": "https://platform.openai.com/docs/api-reference",
      "snippet": "The OpenAI API can be applied to virtually any task...",
      "rank": 1,
      "index": 0,
      "sources": ["google"],
      "scrapedContent": "..." // if scrape=true
    }
  ],
  "timeTakenMs": 123,
  "scraped": false
}

  • Errors: Returns structured error messages and appropriate HTTP status codes.

3. Scrape ​

  • Route: POST /v1/scrap

  • Description: Scrapes content from a given URL using Playwright.

  • Request Body:

    FieldTypeRequiredDescription
    urlstringYesURL to be scraped

    Example:

    json
    { "url": "https://example.com" }

  • Response:

    json
    {
  "url": "https://example.com",
  "content": "...scraped text..."
}

4. Results ​

  • Route: POST /v1/results

  • Description: Processes and merges search results, deduplicates, and ranks them.

  • Request Body:

    FieldTypeRequiredDescription
    resultsobject[]YesArray of search result items

    Example:

    json
    {
  "results": [
    { "title": "...", "link": "...", "snippet": "...", "sources": ["google"] }
  ]
}

  • Response:

    json
    {
  "items": 1,
  "results": [ ...deduplicated and ranked results... ]
}

Error Handling ​

  • All endpoints return structured error messages with context.
  • Search providers return empty results with error information on failure.
  • Logger captures errors for debugging.

Example Usage ​

Search Example (cURL) ​

bash
curl -X POST http://localhost:3000/v1/search \
  -H "Content-Type: application/json" \
  -H "x-api-key: <your-api-key>" \
  -d '{ "query": "openai api", "providers": ["google"], "count": 5 }'

See Also ​