Complete Guide: How to Scrape Sortlist Agency Directories Using Python

Evaluating digital service agencies manually across international markets is time-consuming and inefficient. Business development teams, market analysts, and partner managers frequently spend days browsing directory pages to collect company websites, team sizes, budget thresholds, and client ratings.

Sortlist organizes marketing, web development, branding, and design agencies into service and country listings. However, collecting this data at scale requires automated tools that respect platform structures and handle pagination cleanly.

This tutorial provides a complete technical walkthrough on how to scrape Sortlist agency directories programmatically using Python and the Sortlist Scraper Actor on Apify (reventadata/sortlist-scraper). You will learn how to configure service and country parameters, apply client-side filtering criteria, automate data harvesting, and export structured records for downstream analysis or CRM integration.

By following this guide, you will set up a reliable pipeline that extracts agency websites, location rosters, languages, headcount, client review counts, and budget ranges in minutes.

TL;DR

In this tutorial, you will build an automated Python workflow to extract public agency data from Sortlist:

1. Install apify-client~=3.0 and configure your API authentication token.
2. Define search parameters using official service slugs (web-application) and country codes (belgium-be).
3. Run the Sortlist Scraper Actor via the Apify API client.
4. Extract structured agency profiles containing websites, verified badges, client ratings, team sizes, and budgets.
5. Export clean datasets into JSON, CSV, or Excel formats for business intelligence and outreach workflows.

Prerequisites and setup

Before running the scraper, verify that your local development environment meets the following basic requirements:

1. Python 3.11 or newer: The official apify-client version 3.x series requires Python 3.11+.
2. Apify Account: Create an account on the Apify platform if you do not already have one. A free plan is available.
3. Apify API Token: Locate your personal API token in Apify Console under the API & Integrations settings tab.

Install the Apify Python client package:

pip install -U "apify-client~=3.0"

Set your token as an environment variable in your terminal session:

export APIFY_API_TOKEN="your_apify_token_here"

This ensures your script accesses credentials securely without hardcoding secret keys into your source code.

Understanding Sortlist Scraper

Sortlist Scraper is an Apify Actor engineered to browse public agency listings at /s/<service>/<country>. It extracts the structured data already embedded in the listing and returns clean records ready for analysis.

Behind the scenes:

• The Actor targets Next.js page data stored in <script id="__NEXT_DATA__">, extracting organicAgencies.data and organicAgencies.included parallel arrays.
• Page 1 of any listing delivers 20 agency profiles; subsequent pages deliver up to 100 agencies each until the listing concludes.
• Plain page requests succeeded in testing on Apify's platform (25 of 25 listing pages), meaning no proxy configuration is required by default.
• It reads listing pages only: it never fetches ?query= search URLs (disallowed by robots.txt) or agency profile pages (/agency/<slug>).
• Built-in deduplication ensures that agencies appearing across multiple service or country directories are saved only once per run.
• Rating figures, review tallies, headcount, and budget numbers are parsed directly into typed numeric fields rather than raw display text, making the dataset immediately usable for sorting and database imports.

Step-by-step walkthrough

Step 1: Initialize the client and specify inputs

To start scraping, import the required libraries and instantiate the ApifyClient. Next, construct your input configuration specifying target service categories and country designations:

import os
from apify_client import ApifyClient

# 1. Initialize ApifyClient with your API token
client = ApifyClient(os.environ["APIFY_API_TOKEN"])

# 2. Define the input: search web application agencies in Belgium
run_input = {
    "services": ["web-application"],
    "countries": ["belgium-be"],
    "maxItems": 20,
}

The services field accepts official Sortlist category slugs (such as web-application, branding, marketing, or software-development). The countries parameter requires country names with their two-letter code suffix (such as belgium-be, germany-de, france-fr, or switzerland-ch). Both slugs are validated before any network requests are sent.

Step 2: Configure advanced filtering options

No server-side filter for team sizes or office cities was confirmed on the listing pages, so Sortlist Scraper applies these filters after fetching the listing data:

• minRating: Minimum client rating out of 5 (e.g. 4.5). Unreviewed agencies lack ratings and are excluded when this is set.
• minReviews: Minimum total number of client reviews (e.g. 3).
• minTeamSize: Minimum team headcount reported by the agency (e.g. 10).
• languages: Array of accepted language codes (e.g. ["en", "nl", "fr"]). Regional variants like nl-BE match base codes automatically.
• locationContains: Case-insensitive and accent-insensitive substring filter for English office locations (e.g. "Antwerp" or "Brussels").

Step 3: Execute the scraper

Trigger the execution using client.actor("reventadata/sortlist-scraper").call(run_input=run_input):

# 3. Start the Actor run and wait for it to finish
print("Starting Sortlist extraction...")
run = client.actor("reventadata/sortlist-scraper").call(run_input=run_input)
if run is None or run.status != "SUCCEEDED":
    status = run.status if run else "unknown"
    raise SystemExit(f"Run did not succeed (status: {status}).")

# 4. Fetch the saved agencies from the dataset
items = client.dataset(run.default_dataset_id).list_items().items
print(f"Retrieved {len(items)} agencies.\n")

The script initiates the Actor run and polls execution progress until completion. If any listing URL returns an HTTP 404 on page 1, the Actor logs a warning and proceeds to the remaining directories.

Step 4: Examine the results

You can inspect the collected agency records directly in Python:

for item in items:
    print(item["name"])
    print(f"  Website: {item['website']}")
    if item["rating"] is not None:
        print(f"  Rating: {item['rating']} / 5 ({item['reviewsCount']} reviews)")
    else:
        print("  Rating: No reviews yet")
    print(f"  Team size: {item['teamSize']}")
    print(f"  Offices: {', '.join(item['locations'])}")
    print("-" * 40)

Here is a sample record, a sample from a capture dated 2026-10-21 from the Belgium Web Application listing parsed with the Actor's own parser (scrapedAt omitted):

[
  {
    "id": "fa2146d2-016b-4c81-9e2f-af6f1bb347ea",
    "url": "https://www.sortlist.com/agency/code-the-kiwi",
    "name": "Code The Kiwi",
    "tagline": "Maatwerk software web & mobile",
    "description": "Bij Code The Kiwi zijn we gespecialiseerd in het creëren van robuuste, schaalbare oplossingen die zijn afgestemd op uw bedrijfsbehoeften. Onze expertise strekt zich uit over de nieuwste technologieën en ontwikkel standaarden, zodat elke applicatie die we bouwen is geoptimaliseerd voor prestaties en gebruikerservaring.",
    "website": "https://codethekiwi.be",
    "logo": "https://sortlist-core-api.s3.eu-west-1.amazonaws.com/58arpu4ux2cu0n1hqvrgipw0n43d",
    "locations": [
      "Sint-Katelijne-Waver, Belgium"
    ],
    "languages": [
      "nl-BE",
      "en",
      "nl"
    ],
    "teamSize": 5,
    "registrationNumber": "1013800151",
    "registrationCountry": "BE",
    "rating": 5.0,
    "reviewsCount": 1,
    "recentRecommendedReviews": 0,
    "worksCount": 10,
    "serviceWorksCount": 10,
    "serviceBudgetMin": 3000,
    "serviceBudgetMax": 1000000,
    "budgetMin": 1000,
    "budgetMax": 1000000,
    "services": [
      {
        "name": "Web Application",
        "worksCount": 10
      },
      {
        "name": "Software Development",
        "worksCount": 10
      },
      {
        "name": "Artificial Intelligence",
        "worksCount": 3
      },
      {
        "name": "Mobile App",
        "worksCount": 10
      },
      {
        "name": "Product Management",
        "worksCount": 6
      },
      {
        "name": "Innovation",
        "worksCount": 7
      },
      {
        "name": "Ergonomy (UX/UI)",
        "worksCount": 10
      }
    ],
    "sectors": [
      "Others",
      "E-commerce",
      "Transportation",
      "Software & Computer Services",
      "Industrial Goods & Services",
      "Construction",
      "Energy & Oil",
      "Pharmaceuticals & Biotech"
    ],
    "awards": [
      {
        "name": "Microsoft Partner",
        "category": "",
        "wonAt": "2022-04-01"
      },
      {
        "name": "Google Partner",
        "category": "",
        "wonAt": "2022-04-01"
      },
      {
        "name": "Mollie partner",
        "category": "",
        "wonAt": "2022-11-01"
      }
    ],
    "topAwarded": false,
    "trustedPartner": false,
    "verified": true,
    "claimed": true,
    "sponsored": true,
    "updatedAt": "2026-09-30T13:32:07.483Z",
    "searchService": "Web Application",
    "search": "web-application/belgium-be"
  }
]

Step 5: Export dataset records

Datasets can be exported from Apify Console or via the API using formats such as CSV, JSON, XML, or Excel. This makes it easy to pipe records directly into CRM systems or business intelligence dashboards.

Working with the API

The Apify Python SDK allows you to orchestrate runs, stream data items, and handle execution lifecycles programmatically.

To integrate Sortlist Scraper into production scripts, initialize the client using your secure API token. The call() method handles execution synchronously, waiting for terminal status (SUCCEEDED, FAILED, or TIMED-OUT):

The call(run_input=run_input) method starts the run and blocks until completion, returning the Run object or None; this is what the code above demonstrates. For long-running extractions or distributed background jobs, client.actor("reventadata/sortlist-scraper").start(run_input=run_input) returns the Run object immediately, enabling asynchronous status polling.

When building automated data extraction pipelines, you should always inspect the returned run object to confirm completion before accessing dataset records. A successful run provides a default_dataset_id string referencing your output collection.

For large datasets, retrieve records using client.dataset(run.default_dataset_id).iterate_items(clean=True) on the dataset client rather than loading all items into memory at once with list_items(). The iterate_items(clean=True) method yields parsed dictionaries row by row across API pagination boundaries, keeping client memory usage minimal.

You can configure webhooks to notify external services upon completion. Configuring an ACTOR.RUN.SUCCEEDED event webhook provides a JSON payload containing defaultDatasetId, allowing downstream services like Make, Zapier, or custom microservices to ingest newly scraped records automatically.

If a run fails, the run log can be retrieved via client.log(run.id).get() to inspect worker traces and diagnose configuration issues. You can also specify timeout parameters on client operations to guarantee that long-running tasks never hang your local execution environment.

Advanced usage patterns

Sortlist Scraper supports sophisticated data extraction workflows:

1. Multi-Service Matrix Harvesting: You can pass multiple service slugs alongside multiple country codes. Setting services: ["web-application", "branding"] and countries: ["belgium-be", "germany-de"] systematically walks all four directory listings in a single run.
2. Scheduled Weekly Tracking: You can schedule automated runs in Apify Console using standard cron expressions (e.g. 0 8 * * 1 for every Monday at 8 AM). This allows you to track new market entrants and detect changes in agency positioning over time.
3. Headcount and Review Filtering: Combine minTeamSize: 10 and minReviews: 5 to keep agencies with a larger stated team and more client reviews.
4. Targeted City Extraction: Use locationContains: "Ghent" or locationContains: "Munich" to isolate agencies operating local branches within specific metropolitan hubs.
5. Deduplicated Multi-Directory Runs: When agencies offer both web application development and branding services, the scraper records them only once based on their unique agency slug, avoiding duplicate records in your CRM.
6. Cross-Border Expansion Analysis: Compare agency ecosystems across neighboring countries like Belgium, France, and Germany by running identical service slugs across each market to identify pricing trends, headcount distributions, and regional service density across European tech centers, evaluating market saturation and identifying under-served geographical regions.

Understanding your data

Each output record contains detailed public agency metrics extracted directly from the listing:

• rating: The client evaluation score out of 5, computed as reviews_rating_total / reviews_count * 5 and rounded to two decimal places. Agencies without reviews report null.
• teamSize: Headcount reported by the agency as an integer value, rather than a broad bracket range.
• serviceBudgetMin and serviceBudgetMax: Project budget boundaries specific to the searched service category. Values were shown as EUR on the Belgium, Germany and Switzerland listings checked; other markets were not checked.
• budgetMin and budgetMax: General minimum and maximum project engagement thresholds across all agency capabilities.
• locations: An array of English office locations. The primary headquarters office appears first in the list.
• languages: Array of language codes supported by agency staff. Regional codes like nl-BE are preserved alongside general codes.
• website: Direct link to the agency's external corporate website, providing the primary contact route.
• awards: Structured list of industry honors including award name, category, and date won.

Troubleshooting guide

When configuring and running your scraper, keep these troubleshooting guidelines in mind:

• HTTP 404 for unknown directory: Sortlist directory routes follow strict English naming conventions. Slugs like united-states-us or united-kingdom-gb do not exist under /s/. If a listing returns 404 on page 1, the Actor logs a warning and continues with the next listing.
• Invalid slug format ValueError: Service slugs must be lowercase alphanumeric strings with hyphens (e.g. web-application). Country slugs must end with a hyphenated two-letter code (e.g. belgium-be). Invalid inputs raise a ValueError immediately at startup.
• RuntimeError "No rows pushed": If your post-fetch filters (such as minRating or locationContains) eliminate every candidate agency across all queried directories, the Actor raises a RuntimeError to signal that zero rows were saved.
• Past-the-end pagination: The Actor automatically detects when the final page of a directory is reached (when next is absent from pagination metadata) and halts pagination cleanly before the site returns a 404 error.

Best practices

To maximize extraction efficiency and maintain high data quality:

• Start with small test runs: Test your configuration using a small maxItems value (such as 10 or 20) before executing full multi-country extractions.
• Verify slug availability: Manually check directory URLs in your browser (e.g. https://www.sortlist.com/s/web-application/belgium-be) to confirm the directory exists before initiating large matrix runs.
• Handle unreviewed agencies: Be mindful that setting a minRating filter drops all agencies with zero client reviews, which may exclude agencies that have no reviews yet.
• Respect platform pacing: The Actor automatically incorporates polite request delays (0.5 seconds between requests) to ensure responsible access without overwhelming platform infrastructure.

Pricing

For current pricing and details, see the Store page.

Conclusion and next steps

Extracting B2B agency data no longer requires manual copying from online directories. With Python and Sortlist Scraper, you can harvest agency profiles, contact websites, service budgets, client ratings, and team sizes in structured JSON format with a single automated run.

Try Sortlist Scraper yourself and let me know what you build!

About the author: reventadata builds production-grade web scrapers and data tools for platforms across Europe and worldwide.

Resources

• Sortlist Scraper on Apify Store
• Apify Python Client Documentation
• Apify Schedules Guide
• Apify Webhooks Documentation