Edit Source

Update the directory configuration.

Directory Configuration Guide

Configure a business directory so the scraper can discover companies. Follow each step to capture how the directory behaves, how to paginate it, and how to extract the fields from both list and detail pages.

BasicsName, key, base URL

ScraperAPIRendering, geo, proxy, throttling

Search URL & PaginationTemplate + paging strategy

List SelectorsGrab fields from the result cards

Detail PageOptional extra selectors

ExtrasJSON mappings and notes

NameKeyUnique identifierSource typeBase URLUse this source

Basic Information

Name: A human-readable name for the directory (e.g. "GelbeSeiten Germany", "Solar Installer Directory").

Base URL: The root domain you see in the address bar when visiting the homepage (e.g. https://www.gelbeseiten.de).

Use JS rendering (ScraperAPI render)Default country codeScraperAPI geo routing, e.g. deUse premium proxiesMax requests per minuteThrottle per directoryScraperAPI options JSONOptional. Example: { "device_type": "mobile" }

ScraperAPI Settings

Use JS Rendering: Toggle ON if results load via JavaScript frameworks (React, Angular, Vue). View Page Source - if you see empty HTML or missing results, enable JS rendering.

Default Country Code: ScraperAPI routing country (e.g. de) when the site behaves differently by region.

Use Premium Proxies: Turn on if the directory blocks basic scraping or heavily rate-limits.

Max Requests Per Minute: Limit fetch rate to prevent blocking.

Search URL templateUse placeholders {keyword}, {region}, {city}, {country}, {page}

Pagination modeMax pagesStops infinite loops

Search URL Template

Perform a manual search (keyword + region + city) and copy the URL. Replace dynamic parts with placeholders: {keyword}, {region}, {city}, {country}, {page}. Example:

https://www.gelbeseiten.de/suche/{keyword}/{region}?seite={page}

This tells the worker how to build search URLs for each Search Case. Great when the directory expects specific query parameters like what / where.

Pagination Configuration

PAGE_PARAM: URL changes with a query param (example: ?page=2, ?seite=3). Fill the Page Parameter Name.

LINK_SELECTOR: No page param; provide the CSS selector for the "Next" button (inspect the element, e.g. .pagination-next a).

AJAX_OFFSET: The directory loads more results via AJAX/infinite scroll. Provide the AJAX URL template with an {offset} or {page} placeholder, plus initial offset (start) and offset step (how much to increment each request).

NONE: Only one page of results. Set Max Pages to cap crawling (10-30 is typical).

List item selectorSelector for each company cardList name selectorInside the cardList profile URL selectorOptionalList website selectorOptional external linkList city selectorList region selectorList address selectorList phone selector

List Page Selectors

Inspect a search results page. Pick the outer selector that wraps one company card (e.g. .result-item, .mod-Treffer, .company-card).

Inside that card, capture:
- Company name selector.
- Profile URL selector (if the directory has detail pages).
- Website selector (direct external link if shown).
- City, Region, Address, Phone selectors as available.

Use browser Inspect to copy classes/IDs for each field.

Directory has company detail pages

Detail name selectorDetail website selectorDetail email selectorDetail phone selectorDetail address selectorDetail city selectorDetail region selectorDetail postal code selector

Detail Page Selectors (optional)

Enable this if clicking a result opens a company profile page. Inspect that page and copy selectors for name, website, email, phone, address, city, region, and postal code.

These selectors enrich the lead when the directory exposes more data on the detail page than on the list page.

Keyword mapping JSONOptional. Example: { "join": "+", "maxKeywords": 3 }Region mapping JSONOptional. Example: { "Bavaria": "Bayern" }Extra config JSONCookies, headers, special instructions for the worker

Extra Config & Summary

Use extra JSON for custom headers, cookies, additional placeholders, or worker instructions. Keyword and region mapping help normalize search terms if the directory expects different naming.

Summary: search manually, copy the URL and convert it to a template, set pagination, capture list selectors, optionally detail selectors, then save and test by running a Search Case on this Source.