fast-crawl

Appearance

Provider Plugin Authoring Guide ​

Fast-Crawl is modular—new search providers can be added as plugins. This guide explains how to author a provider plugin.

1. Provider Structure ​

  • Place your provider in src/v1/lib/ (e.g., src/v1/lib/searchDuckDuckGo.ts).
  • Export a function: async function search<Provider>(query, options): Promise<SearchResult[]>
  • Follow the interface in src/api/result.ts for result shape.

2. Example Skeleton ​

typescript
// src/v1/lib/searchDuckDuckGo.ts
import { SearchResult } from '../../api/result';

export async function searchDuckDuckGo(query: string, opts: ProviderOptions): Promise<SearchResult[]> {
  // Implement provider logic (API call, scraping, etc.)
  // Return results in the standard format
  return [];
}

3. Registering the Provider ​

  • Add your provider to the provider registry in the main search logic (see src/v1/routes/search.ts).
  • Update the API docs if exposing new options.

4. Testing ​

  • Place tests in src/v1/lib/searchDuckDuckGo.test.ts.
  • Use fixtures in src/test/fixtures/ for mock responses.

5. Error Handling ​

  • Return empty results with error info on failure (see other providers for pattern).

6. Best Practices ​

  • Respect rate limits and terms of service.
  • Use environment variables for API keys/secrets.
  • Document any provider-specific options.

For questions or to contribute a provider, open a PR or issue on GitHub.