Web Scraping: UK Ethics and Practice for Journalists

What is web scraping?

Web scraping is the automated extraction of data from websites using code. Instead of copying and pasting data by hand, you write a script that visits a page, reads the HTML, and extracts the information you want. For journalists, common use cases include: downloading planning applications from a council website, collecting property listings over time, or archiving public social media content.

Always check first whether the data is available in a more convenient form: an API, a downloadable CSV, or an FOI request. Scraping is the option of last resort.

UK legal framework for scraping

Python scraping basics with BeautifulSoup

# Visit: https://example.com/robots.txt
# Look for Disallow rules that cover your target pages
# E.g.:
# User-agent: *
# Disallow: /planning-applications/  <- do not scrape this
# Allow: /open-data/                  <- this is fine

import requests
from bs4 import BeautifulSoup
import time

HEADERS = {
    'User-Agent': 'Mozilla/5.0 (journalist research; contact@example.com)'
}

def scrape_page(url):
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, 'html.parser')
    # Extract data here
    return soup.find_all('td', class_='planning-ref')

# Rate limit: wait 2 seconds between requests
for url in urls:
    data = scrape_page(url)
    time.sleep(2)   # be polite to the server

When scraping is appropriate

• 1Publicly accessible pages containing factual data with no API or downloadable format available.
• 2Archiving public pages that may be removed — for example, capturing council meeting records.
• 3Monitoring for changes over time — detecting when a government page is updated.
• 4Extracting data from HTML tables that cannot be downloaded as CSV.

Red flags

• ⚠The target URL requires a login you do not legitimately hold — stop immediately.
• ⚠The robots.txt explicitly disallows the pages you want to scrape.
• ⚠The data is personal data (names, contact details) — UK GDPR applies.
• ⚠Your scraper is sending hundreds of requests per minute — you risk crashing the server and potential legal action.
• ⚠The terms of service explicitly prohibit automated access.

Responsible scraping checklist

• I have checked whether an API or downloadable dataset already exists.
• I have checked the robots.txt file and am not scraping disallowed pages.
• I have read the site's terms of service.
• My scraper identifies itself in the User-Agent header.
• My scraper waits at least 1–2 seconds between requests.
• The data I am scraping does not include personal data, or I have a lawful basis for processing it.
• I have not bypassed any login, CAPTCHA, or access control.

Common mistakes

• ›Scraping at full speed with no delays — disrespectful and may get your IP banned.
• ›Not checking robots.txt before writing the scraper.
• ›Using a generic User-Agent that looks like a bot — identify yourself.
• ›Scraping personal data without considering UK GDPR.
• ›Not saving scraped data to disk immediately — if the script crashes you lose everything.

Related guides

Primary sources

• Copyright, Designs and Patents Act 1988 (legislation.gov.uk)
• UK GDPR — ICO guidance
• Computer Misuse Act 1990 (legislation.gov.uk)
• Beautiful Soup documentation
• Python requests library documentation

Frequently asked questions