FilterWise

Dealing with blocked or anti-bot protected sites

Some real estate portals use anti-bot technology to prevent automated scanning. This guide explains why sites block scrapers, which portals are commonly affected, and what alternatives you have.

Why Sites Block Scrapers

Real estate portals invest heavily in their listing data. They block automated access for several reasons:

  • Data protection: Preventing competitors from copying their listings
  • Server load: Automated traffic can strain their infrastructure
  • Business model: They want users to visit their site (and see their ads)
  • Fair use: Ensuring their platform remains responsive for human users

How Blocking Works

Sites use various techniques to detect and block automated access:

CAPTCHAs

The most visible protection. Sites like LeBonCoin present a CAPTCHA challenge (e.g., "click all images with traffic lights") that automated tools can't solve.

JavaScript Challenges

Some sites require complex JavaScript execution to render their content. If the scraper doesn't execute JavaScript (or doesn't execute it correctly), the page appears empty.

Rate Limiting

Sites track how many requests come from the same IP address. Too many requests in a short time trigger a block.

Browser Fingerprinting

Advanced sites check browser characteristics (screen size, fonts, plugins) to distinguish real browsers from automated tools.

Portal Compatibility by Country

Based on our experience, here are portals grouped by scanning compatibility. Note that FilterWise uses advanced scanning technology (including headless browser rendering) that works with many sites others can't access.

🇬🇧 United Kingdom

  • Rightmove ✅ — works reliably
  • Zoopla ✅ — works reliably
  • OnTheMarket ✅ — works reliably

🇫🇷 France

  • SeLoger ✅ — works reliably
  • Bien'ici ✅ — works with advanced rendering
  • PAP ✅ — works reliably
  • Logic-Immo ✅ — works reliably
  • LeBonCoin ❌ — aggressive anti-bot protection with CAPTCHAs. Use email alerts instead.

🇩🇪 Germany

  • ImmobilienScout24 ✅ — works reliably
  • Immowelt ✅ — works reliably

🇪🇸 Spain

  • Idealista ✅ — works reliably
  • Fotocasa ✅ — works reliably
  • Pisos.com ✅ — works reliably

🇵🇹 Portugal

  • Idealista ✅ — works reliably
  • Imovirtual ✅ — works reliably

🇮🇹 Italy

  • Idealista ✅ — works reliably
  • Immobiliare.it ✅ — works reliably
  • Casa.it ✅ — works reliably

🇧🇪 Belgium

  • Immoweb ✅ — works reliably

For a comprehensive list, see Supported portals.

What Happens When a Source is Blocked

When FilterWise encounters a block:

  1. The source status changes to "Blocked" in your dashboard
  2. No listings are extracted from that scan attempt
  3. A tooltip explains: "This site uses anti-bot protection. Try email alerts instead."
  4. The source remains in your list and will be retried on the next daily scan (blocks are sometimes temporary)

Alternatives When Scanning is Blocked

Alternative 1: Use the Email Filter (Recommended)

The most reliable alternative. Instead of scanning the portal directly:

  1. Set up email alerts on the blocked portal
  2. Redirect those alert emails to your FilterWise email address
  3. FilterWise processes the emails and analyzes each listing

Why this works: Email alerts come from the portal itself — there's no scanning involved. The portal sends you the listing data directly, and FilterWise extracts and analyzes it.

This is the recommended approach for LeBonCoin and other heavily protected sites.

Alternative 2: Use a Different Portal

Many listings appear on multiple portals simultaneously. If Portal A blocks scanning:

  1. Check if the same listings appear on Portal B
  2. Set up a scanning source for Portal B instead
  3. Use broader search criteria to compensate for any coverage gap

For example, in France, listings often appear on both SeLoger and LeBonCoin. If LeBonCoin blocks scanning, you can often find the same listings on SeLoger.

Alternative 3: Combine Both Methods

The most thorough approach:

  1. Use website monitoring for portals that allow it
  2. Use email forwarding for portals that block scanning
  3. FilterWise automatically deduplicates listings that appear in both sources

This gives you maximum coverage without missing listings from any portal.

Tips for Reducing Blocks

While FilterWise handles scanning automatically, these tips can help:

  1. Don't scan too frequently: Stick to the once-daily automatic scan rather than triggering manual scans constantly
  2. Use reasonable search scope: Broad searches that return hundreds of results are more likely to trigger rate limits
  3. Be patient with intermittent blocks: Some sites block temporarily (hours) then allow access again

Checking if a Site is Scan-Friendly

Before adding a website, do a quick test:

  1. Add the URL as a source
  2. Click Scan now
  3. Check the result:
    • Listings found > 0: The site works ✅
    • Listings found = 0, Status = Blocked: The site blocks scanning ❌
    • Listings found = 0, Status = Processed: The URL might be wrong (homepage, empty search, etc.)

Related articles: Supported portals · Adding websites · Choosing the right URLs · Redirecting emails to FilterWise