Historical prices show what the market did. News can help explain what investors knew, when they knew it and why expectations changed.
For a stock-market prediction project, useful news data may include:
- Earnings announcements
- Product launches and recalls
- Management changes
- Mergers and acquisitions
- Regulatory decisions
- Lawsuits and investigations
- Analyst upgrades and downgrades
- Interest-rate and inflation news
- Supply-chain disruptions
- Industry-specific events
However, collecting news is not as simple as copying headlines into a CSV file. A reliable system must identify relevant articles, respect website rules, preserve accurate timestamps, remove duplicates and avoid using information that was unavailable at the time of prediction.
This tutorial builds a practical Python news-acquisition pipeline for an AI stock-prediction project. It uses RSS feeds for discovery, permitted webpage extraction for full article data, robots.txt checks, request throttling, structured metadata, deduplication and CSV/JSONL output.
The tutorial also explains when a news API is a better choice than web scraping and how to prepare the collected records for sentiment analysis or other natural-language-processing models.
This project is for educational and research purposes. Website terms, copyright rules, database rights and data-licensing requirements vary by source and jurisdiction. Check the applicable rules before collecting or redistributing content.
Quick Answer
The safest practical order for collecting financial news is:
- Use a licensed data feed or official API when one is available.
- Use the publisher’s RSS feed for article discovery.
- Scrape public HTML only when the site’s rules and terms permit it.
- Identify your crawler and limit its request rate.
- Store the original URL, source, publication time and collection time.
- Deduplicate syndicated and updated stories.
- Align each article with the first trading period in which it could have been known.
- Keep raw data unchanged and create cleaned data as a separate dataset.
A useful news record should look like this:
{
"source": "Example Financial News",
"url": "https://example.com/news/company-results",
"canonical_url": "https://example.com/news/company-results",
"title": "Example Company Reports Quarterly Results",
"published_at": "2026-08-05T08:30:00+00:00",
"collected_at": "2026-08-05T08:34:12+00:00",
"author": "News Desk",
"description": "The company reported its latest quarterly results.",
"text": "Full permitted article text or an authorised excerpt...",
"tickers": ["EXMPL"],
"content_hash": "..."
}The timestamp fields are especially important. If an article appeared after the market closed, a model predicting that same day’s closing price must not be allowed to use it.
Why News Data Is Useful in Stock-Market Prediction
Price and volume data describe market behaviour numerically. News adds information about events that may change future cash flows, risk or investor expectations.
Suppose a company’s share price falls by 8%. Historical price data can show:
- The size of the fall
- Trading volume
- Recent volatility
- Whether the price crossed a moving average
News data may reveal that the company:
- Missed its earnings forecast
- Lost a major customer
- Announced a product recall
- Received a regulatory penalty
An AI model can convert news text into features such as:
- Positive, neutral or negative sentiment
- Sentiment confidence
- Number of relevant articles
- Source diversity
- Novelty compared with earlier stories
- Event category
- Company or sector mentioned
- Time since publication
- Unexpectedness or severity
News does not automatically improve a prediction model. The benefit depends on the quality of the collection pipeline, the target being predicted and whether the information was genuinely available before the prediction time.
News APIs, RSS Feeds and HTML Scraping Compared
There are several ways to obtain news. They are not interchangeable.
| Method | Advantages | Limitations | Best use |
|---|---|---|---|
| Licensed financial feed | Consistent schema, strong coverage, clear usage rights | Often expensive | Production research and trading systems |
| News API | Structured JSON, easy queries, fewer broken selectors | Quotas, licensing limits, possible text truncation | Prototypes and article discovery |
| Publisher RSS feed | Lightweight, timestamped, easy to parse | Often contains only summaries | Discovery and monitoring |
| Permitted HTML scraping | Can extract page metadata and available text | Fragile, source-specific and legally sensitive | Research where permitted |
| Search-engine results | Broad discovery | Unstable, duplicate-heavy and often restricted | Manual research, not a primary dataset |
NewsAPI’s Everything endpoint, for example, is designed for article discovery and analysis using keyword, date, domain and language filters. GDELT also provides large-scale news datasets and live APIs. These options may be more suitable than maintaining many site-specific scrapers.
An API does not remove every problem. You must still check:
- Whether historical results are complete
- Whether full text or only a snippet is supplied
- How publication times are defined
- Whether the licence permits storage, modelling and redistribution
- Whether the provider corrects or deletes records later
- Whether the source list changes over time
For a learning project, RSS discovery plus conservative webpage extraction demonstrates the complete acquisition process without pretending that every news website can or should be crawled.
Legal, Ethical and Technical Rules
Before writing a scraper, create a source policy.
1. Read the Website’s Terms
Check the site’s terms of service, data policy and licensing information. A publicly viewable page is not automatically free to copy, store indefinitely or republish.
Do not bypass:
- Paywalls
- Authentication
- CAPTCHA challenges
- Access controls
- Rate limits
- Technical restrictions
If the website offers an API or licensed feed, use it when practical.
2. Check robots.txt
The Robots Exclusion Protocol is standardised in RFC 9309. A compliant crawler retrieves the site’s /robots.txt file and checks whether its user agent is permitted to request a particular URL.
robots.txt is not a complete legal permission system. An allowed path does not cancel copyright, contract or privacy obligations. A disallowed path should nevertheless be treated as a clear instruction not to crawl it.
3. Identify the Crawler
Use a descriptive user agent rather than pretending to be a normal browser:
StockNewsResearchBot/1.0 (+https://your-domain.example/bot; [email protected])The linked page can explain:
- Who operates the crawler
- Its research purpose
- Its normal request frequency
- How a publisher can contact you
- How to request removal
4. Crawl Slowly
A research crawler does not need to request the same server several times per second.
Use:
- A delay between requests
- Connection and read timeouts
- A limited retry count
- Exponential backoff for temporary failures
- Per-domain request scheduling
- Caching to avoid downloading unchanged pages
Respect HTTP 429 Too Many Requests and Retry-After responses.
5. Store Only What You Need
For many projects, a headline, permitted excerpt, source, URL and timestamp are enough. Storing and redistributing complete copyrighted articles can create additional legal and licensing risks.
Keep the original source URL so authorised users can return to the publisher.
Designing the Dataset Before Writing the Scraper
A crawler should write records into a predefined schema. Otherwise, every source produces a different collection of columns.
Use at least these fields:
| Field | Purpose |
source | Publisher or feed name |
url | URL discovered by the system |
canonical_url | Publisher’s preferred URL when available |
title | Article headline |
description | Summary or metadata description |
author | Author when supplied |
published_at | Claimed original publication time |
modified_at | Later modification time, if supplied |
collected_at | Time your system retrieved the page |
text | Permitted body text or excerpt |
language | Article language when known |
tickers | Companies matched by controlled rules |
content_hash | Hash used for deduplication |
status | Success, blocked, missing date or other state |
Do not overwrite published_at with the collection time. They answer different questions.
Also consider preserving the raw response metadata separately:
- HTTP status
ETagLast-Modified- Content type
- Retrieval duration
- Extractor version
- Feed URL
This information helps diagnose coverage gaps and reproduce the dataset later.
Project Structure
Create a folder with the following files:
stock-news-scraper/
├── scrape_news.py
├── requirements.txt
├── news_articles.csv
└── news_articles.jsonlThe output files are created automatically after the scraper runs.
Step 1: Install Python Packages
Create a virtual environment:
python -m venv .venvActivate it on Linux or macOS:
source .venv/bin/activateOn Windows PowerShell:
.venv\Scripts\Activate.ps1Create requirements.txt:
beautifulsoup4
feedparser
python-dateutil
requestsInstall the dependencies:
python -m pip install -r requirements.txtPin tested package versions before deploying a production pipeline. A lock file or fully pinned requirements file helps keep future runs reproducible.
Step 2: Configure Sources and Company Terms
The complete script below intentionally uses placeholder feeds. Replace them only with sources you are authorised to access.
RSS_SOURCES = [
{
"name": "Example Financial News",
"feed_url": "https://news.example.com/finance/rss.xml",
},
{
"name": "Example Exchange Announcements",
"feed_url": "https://exchange.example.com/announcements.xml",
},
]
COMPANIES = {
"AAPL": ["Apple", "Apple Inc."],
"MSFT": ["Microsoft", "Microsoft Corp."],
"NVDA": ["Nvidia", "NVIDIA Corporation"],
}Avoid using a ticker alone for ambiguous symbols. A short ticker such as CAT, IT or ON can occur frequently in normal English.
A stronger company dictionary includes:
- Official company name
- Common short name
- Unambiguous ticker forms such as
$AAPL - Former names
- Important subsidiaries
- Exchange name
- Country or industry context
Entity linking should be treated as its own data-quality problem, not as a simple substring search.
Step 3: Complete News-Scraping Script
Save the following as scrape_news.py:
from __future__ import annotations
import csv
import hashlib
import json
import re
import time
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from urllib.parse import parse_qsl, urlencode, urljoin, urlsplit, urlunsplit
from urllib.robotparser import RobotFileParser
import feedparser
import requests
from bs4 import BeautifulSoup
from dateutil import parser as date_parser
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
USER_AGENT = (
"StockNewsResearchBot/1.0 "
"(+https://your-domain.example/bot; [email protected])"
)
REQUEST_DELAY_SECONDS = 2.0
CONNECT_TIMEOUT_SECONDS = 5
READ_TIMEOUT_SECONDS = 20
MAX_ARTICLES_PER_FEED = 20
CSV_PATH = Path("news_articles.csv")
JSONL_PATH = Path("news_articles.jsonl")
RSS_SOURCES = [
{
"name": "Example Financial News",
"feed_url": "https://news.example.com/finance/rss.xml",
},
]
COMPANIES = {
"AAPL": ["Apple", "Apple Inc."],
"MSFT": ["Microsoft", "Microsoft Corp."],
"NVDA": ["Nvidia", "NVIDIA Corporation"],
}
TRACKING_PARAMETERS = {
"fbclid",
"gclid",
"mc_cid",
"mc_eid",
"ref",
"source",
}
@dataclass
class ArticleRecord:
source: str
url: str
canonical_url: str
title: str
description: str
author: str
published_at: str
modified_at: str
collected_at: str
text: str
language: str
tickers: list[str]
content_hash: str
status: str
def create_session() -> requests.Session:
retry = Retry(
total=3,
connect=3,
read=2,
status=3,
backoff_factor=1.0,
status_forcelist=(429, 500, 502, 503, 504),
allowed_methods=frozenset({"GET"}),
respect_retry_after_header=True,
)
adapter = HTTPAdapter(max_retries=retry)
session = requests.Session()
session.headers.update(
{
"User-Agent": USER_AGENT,
"Accept": (
"text/html,application/xhtml+xml,application/rss+xml,"
"application/xml;q=0.9,*/*;q=0.8"
),
}
)
session.mount("https://", adapter)
session.mount("http://", adapter)
return session
def utc_now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def normalise_space(value: str) -> str:
return re.sub(r"\s+", " ", value).strip()
def normalise_url(url: str) -> str:
parts = urlsplit(url)
filtered_query = []
for key, value in parse_qsl(parts.query, keep_blank_values=True):
lower_key = key.lower()
if lower_key.startswith("utm_"):
continue
if lower_key in TRACKING_PARAMETERS:
continue
filtered_query.append((key, value))
path = parts.path or "/"
if path != "/":
path = path.rstrip("/")
return urlunsplit(
(
parts.scheme.lower(),
parts.netloc.lower(),
path,
urlencode(filtered_query, doseq=True),
"",
)
)
def parse_datetime(value: Any) -> str:
if not value:
return ""
try:
parsed = date_parser.parse(str(value))
except (TypeError, ValueError, OverflowError):
return ""
if parsed.tzinfo is None:
return ""
return parsed.astimezone(timezone.utc).isoformat()
def matches_tickers(text: str) -> list[str]:
matched = []
for ticker, aliases in COMPANIES.items():
patterns = [rf"\${re.escape(ticker)}\b"]
patterns.extend(
rf"\b{re.escape(alias)}\b"
for alias in aliases
)
if any(re.search(pattern, text, re.IGNORECASE) for pattern in patterns):
matched.append(ticker)
return sorted(matched)
def get_robot_parser(
session: requests.Session,
url: str,
cache: dict[str, RobotFileParser | None],
) -> RobotFileParser | None:
parts = urlsplit(url)
origin = f"{parts.scheme}://{parts.netloc}"
if origin in cache:
return cache[origin]
robots_url = urljoin(origin, "/robots.txt")
parser = RobotFileParser()
parser.set_url(robots_url)
try:
response = session.get(
robots_url,
timeout=(CONNECT_TIMEOUT_SECONDS, READ_TIMEOUT_SECONDS),
)
response.raise_for_status()
parser.parse(response.text.splitlines())
cache[origin] = parser
except requests.RequestException as error:
print(f"Could not verify robots.txt for {origin}: {error}")
cache[origin] = None
return cache[origin]
def is_allowed(
session: requests.Session,
url: str,
cache: dict[str, RobotFileParser | None],
) -> bool:
parser = get_robot_parser(session, url, cache)
# Conservative policy: skip when robots.txt cannot be verified.
if parser is None:
return False
return parser.can_fetch(USER_AGENT, url)
def iter_json_ld(soup: BeautifulSoup):
for script in soup.select('script[type="application/ld+json"]'):
raw = script.string or script.get_text()
if not raw.strip():
continue
try:
value = json.loads(raw)
except json.JSONDecodeError:
continue
values = value if isinstance(value, list) else [value]
for item in values:
if not isinstance(item, dict):
continue
graph = item.get("@graph")
if isinstance(graph, list):
values.extend(
node for node in graph if isinstance(node, dict)
)
yield item
def find_news_json_ld(soup: BeautifulSoup) -> dict[str, Any]:
accepted_types = {
"Article",
"NewsArticle",
"ReportageNewsArticle",
}
for item in iter_json_ld(soup):
item_type = item.get("@type", "")
types = item_type if isinstance(item_type, list) else [item_type]
if any(value in accepted_types for value in types):
return item
return {}
def meta_content(soup: BeautifulSoup, *selectors: str) -> str:
for selector in selectors:
element = soup.select_one(selector)
if element and element.get("content"):
return normalise_space(str(element["content"]))
return ""
def author_from_json_ld(value: Any) -> str:
if isinstance(value, dict):
return normalise_space(str(value.get("name", "")))
if isinstance(value, list):
names = [author_from_json_ld(item) for item in value]
return ", ".join(name for name in names if name)
if isinstance(value, str):
return normalise_space(value)
return ""
def extract_article_text(soup: BeautifulSoup, json_ld: dict[str, Any]) -> str:
body = json_ld.get("articleBody")
if isinstance(body, str) and body.strip():
return normalise_space(body)
# Generic fallbacks work on some pages, but source-specific selectors
# are normally more accurate. Add them only for approved sources.
candidates = soup.select(
"article p, main article p, [itemprop='articleBody'] p"
)
paragraphs = []
for paragraph in candidates:
text = normalise_space(paragraph.get_text(" ", strip=True))
if len(text) >= 40:
paragraphs.append(text)
return "\n\n".join(paragraphs)
def make_hash(title: str, text: str) -> str:
material = normalise_space(f"{title}\n{text}").lower()
return hashlib.sha256(material.encode("utf-8")).hexdigest()
def extract_article(
session: requests.Session,
source_name: str,
url: str,
feed_title: str,
feed_summary: str,
feed_published: str,
) -> ArticleRecord:
response = session.get(
url,
timeout=(CONNECT_TIMEOUT_SECONDS, READ_TIMEOUT_SECONDS),
)
response.raise_for_status()
content_type = response.headers.get("Content-Type", "").lower()
if "text/html" not in content_type:
raise ValueError(f"Unsupported content type: {content_type}")
soup = BeautifulSoup(response.text, "html.parser")
json_ld = find_news_json_ld(soup)
title = normalise_space(
str(json_ld.get("headline", ""))
or meta_content(soup, 'meta[property="og:title"]')
or feed_title
)
description = normalise_space(
str(json_ld.get("description", ""))
or meta_content(
soup,
'meta[property="og:description"]',
'meta[name="description"]',
)
or feed_summary
)
canonical_element = soup.select_one('link[rel="canonical"]')
canonical_url = url
if canonical_element and canonical_element.get("href"):
canonical_url = urljoin(url, str(canonical_element["href"]))
canonical_url = normalise_url(canonical_url)
text = extract_article_text(soup, json_ld)
combined_text = f"{title}\n{description}\n{text}"
published_at = parse_datetime(
json_ld.get("datePublished")
or meta_content(
soup,
'meta[property="article:published_time"]',
)
or feed_published
)
modified_at = parse_datetime(
json_ld.get("dateModified")
or meta_content(
soup,
'meta[property="article:modified_time"]',
)
)
language = str(json_ld.get("inLanguage", ""))
if not language and soup.html:
language = str(soup.html.get("lang", ""))
return ArticleRecord(
source=source_name,
url=normalise_url(url),
canonical_url=canonical_url,
title=title,
description=description,
author=author_from_json_ld(json_ld.get("author")),
published_at=published_at,
modified_at=modified_at,
collected_at=utc_now_iso(),
text=text,
language=language,
tickers=matches_tickers(combined_text),
content_hash=make_hash(title, text),
status="ok" if published_at else "missing_published_time",
)
def load_existing_keys() -> tuple[set[str], set[str]]:
urls: set[str] = set()
hashes: set[str] = set()
if not JSONL_PATH.exists():
return urls, hashes
with JSONL_PATH.open("r", encoding="utf-8") as file:
for line in file:
try:
item = json.loads(line)
except json.JSONDecodeError:
continue
if item.get("canonical_url"):
urls.add(item["canonical_url"])
if item.get("content_hash"):
hashes.add(item["content_hash"])
return urls, hashes
def save_records(records: list[ArticleRecord]) -> None:
if not records:
return
with JSONL_PATH.open("a", encoding="utf-8") as file:
for record in records:
file.write(json.dumps(asdict(record), ensure_ascii=False) + "\n")
fieldnames = list(asdict(records[0]).keys())
write_header = not CSV_PATH.exists()
with CSV_PATH.open("a", encoding="utf-8", newline="") as file:
writer = csv.DictWriter(file, fieldnames=fieldnames)
if write_header:
writer.writeheader()
for record in records:
item = asdict(record)
item["tickers"] = json.dumps(item["tickers"])
writer.writerow(item)
def run() -> None:
session = create_session()
robot_cache: dict[str, RobotFileParser | None] = {}
seen_urls, seen_hashes = load_existing_keys()
new_records: list[ArticleRecord] = []
for source in RSS_SOURCES:
feed_url = source["feed_url"]
source_name = source["name"]
try:
feed_response = session.get(
feed_url,
timeout=(CONNECT_TIMEOUT_SECONDS, READ_TIMEOUT_SECONDS),
)
feed_response.raise_for_status()
except requests.RequestException as error:
print(f"Feed failed for {source_name}: {error}")
continue
feed = feedparser.parse(feed_response.content)
for entry in feed.entries[:MAX_ARTICLES_PER_FEED]:
url = normalise_url(str(entry.get("link", "")))
if not url or url in seen_urls:
continue
feed_title = normalise_space(str(entry.get("title", "")))
feed_summary = normalise_space(
BeautifulSoup(
str(entry.get("summary", "")),
"html.parser",
).get_text(" ", strip=True)
)
candidate_text = f"{feed_title}\n{feed_summary}"
if not matches_tickers(candidate_text):
continue
if not is_allowed(session, url, robot_cache):
print(f"Skipped by crawler policy: {url}")
continue
time.sleep(REQUEST_DELAY_SECONDS)
try:
record = extract_article(
session=session,
source_name=source_name,
url=url,
feed_title=feed_title,
feed_summary=feed_summary,
feed_published=str(
entry.get("published", "")
or entry.get("updated", "")
),
)
except (requests.RequestException, ValueError) as error:
print(f"Article failed: {url}: {error}")
continue
if record.canonical_url in seen_urls:
continue
if record.content_hash in seen_hashes:
continue
seen_urls.add(record.canonical_url)
seen_hashes.add(record.content_hash)
new_records.append(record)
print(f"Collected: {record.title}")
save_records(new_records)
print(f"Saved {len(new_records)} new article(s).")
if __name__ == "__main__":
run()Before running the script, change:
- The contact details in
USER_AGENT - The placeholder RSS feed
- The company and ticker dictionary
- The request delay if a source asks for a slower rate
- The extraction policy if full article text is not licensed for your use
Run it with:
python scrape_news.pyHow the Scraper Works
1. RSS Is Used for Discovery
The script does not crawl a website looking for every possible link. It reads an approved RSS feed and processes recent entries.
This reduces unnecessary requests and provides useful metadata such as:
- Headline
- Article URL
- Summary
- Publication or update time
The RSS result is still verified against the article page because feeds may contain abbreviated or differently formatted metadata.
2. Obvious Irrelevant Articles Are Filtered Early
The feed title and summary are checked against the controlled company dictionary before the article page is requested.
This saves bandwidth, but it can create false negatives. An article may discuss a company indirectly without naming it in the headline or summary.
For better recall, a production system might first collect all permitted feed metadata and perform entity recognition later.
3. robots.txt Is Checked Per Origin
The script uses Python’s RobotFileParser and caches the result for each origin. It adopts a conservative policy: if robots.txt cannot be verified, the page is skipped.
Different projects may use a different failure policy, but it should be documented and applied consistently.
4. Structured Metadata Is Preferred
Many publishers include Schema.org NewsArticle or Article data in a JSON-LD script. This may provide:
headlinedescriptiondatePublisheddateModifiedauthorarticleBodyinLanguage
Structured metadata is usually more stable than selecting CSS classes designed only for page layout.
The script then falls back to Open Graph metadata, normal meta descriptions and RSS values.
5. Generic Paragraph Extraction Is a Fallback
The selector:
article p, main article p, [itemprop='articleBody'] pworks on some pages but will not work perfectly everywhere. It may include captions, related-story text or subscription messages.
For an authorised source, add a tested source-specific extractor instead of building a single enormous selector for every website.
6. URLs and Content Are Deduplicated
The script removes common tracking parameters and fragments from URLs. It also creates a SHA-256 hash from the normalised title and body.
This catches:
- The same URL discovered twice
- Tracking variants of one URL
- Identical content published under different URLs
It does not catch heavily edited or syndicated versions of the same story. Near-duplicate detection is discussed later.
Using a News API Instead
If your use case and licence are supported, a news API can replace the discovery and extraction sections.
Here is a simple NewsAPI example:
import os
from datetime import date, timedelta
import requests
api_key = os.environ["NEWS_API_KEY"]
start_date = date.today() - timedelta(days=7)
response = requests.get(
"https://newsapi.org/v2/everything",
headers={"X-Api-Key": api_key},
params={
"q": 'Apple OR "Apple Inc" OR AAPL',
"from": start_date.isoformat(),
"language": "en",
"sortBy": "publishedAt",
"pageSize": 100,
},
timeout=(5, 20),
)
response.raise_for_status()
payload = response.json()
for article in payload.get("articles", []):
print(article["publishedAt"], article["title"])Set the key outside the source code:
export NEWS_API_KEY="your-key-here"Never commit an API key to Git.
The API response still requires:
- Pagination
- Date-window management
- Deduplication
- Query logging
- Rate-limit handling
- Licence compliance
- Ticker or company mapping
- Timestamp validation
- Coverage monitoring
Do not assume an API’s content field contains the complete article. Check the provider’s current documentation and plan.
Improving Deduplication
Financial news is frequently syndicated. Ten URLs may represent one original report rather than ten independent signals.
Basic deduplication should compare:
- Normalised canonical URL
- Exact content hash
- Normalised headline
- Publication time
- Source and author
For near duplicates, calculate similarity between titles or article embeddings.
Example using title tokens:
import re
def title_tokens(title):
return set(re.findall(r"[a-z0-9]+", title.lower()))
def jaccard_similarity(first_title, second_title):
first = title_tokens(first_title)
second = title_tokens(second_title)
if not first and not second:
return 1.0
if not first or not second:
return 0.0
return len(first & second) / len(first | second)A high similarity score is a review signal, not proof that two stories are identical.
Keep both concepts:
article_id: one collected pagestory_cluster_id: several pages describing the same underlying event
This allows the model to distinguish genuine source confirmation from simple republication.
The Most Important Finance Problem: Point-in-Time Correctness
Look-ahead bias occurs when the training dataset gives a model information that would not have been available at prediction time.
Imagine this sequence:
| Time | Event |
| Monday 4:00 PM | Market closes |
| Monday 4:15 PM | Company releases weak results |
| Monday 4:17 PM | News article is published |
| Tuesday 9:30 AM | Market reopens and price falls |
If the Monday after-hours article is assigned to Monday’s closing-price prediction, the dataset leaks future information.
For every record, preserve:
published_at: time claimed by the publishermodified_at: later edit timecollected_at: time observed by your collectoravailable_at: earliest defensible time the model could have received it
A conservative definition is:
available_at = max(published_at, first_observed_at)This prevents a crawler that runs on Tuesday from pretending it definitely possessed the article on Monday merely because the page now displays a Monday timestamp.
Aligning News with Trading Sessions
Do not group articles by calendar date alone.
The correct trading session depends on:
- Exchange timezone
- Market opening and closing time
- Weekends
- Public holidays
- Half-day trading sessions
- Pre-market and after-hours trading
- The model’s exact prediction cutoff
For a model that makes one prediction immediately before the regular market opens:
- News before the cutoff may belong to that session.
- News after the cutoff belongs to the next prediction window.
For a model predicting five-minute returns, publication timestamps may not be precise enough. You may need first-observed times and a realistic delay for acquisition, processing and model inference.
Cleaning the Text Without Destroying Evidence
Keep two layers:
Raw Layer
The raw layer should be append-only where practical. It contains the exact permitted data returned by the collector plus retrieval metadata.
Do not silently rewrite old raw records when cleaning rules change.
Processed Layer
The processed layer can contain:
- Lowercased text when appropriate
- Removed navigation fragments
- Normalised whitespace
- Sentence segmentation
- Language detection
- Company entities
- Sentiment score
- Event classification
- Embeddings
- Story clusters
- Trading-session assignment
Version the transformation code. A dataset should record which extractor, entity matcher and sentiment model produced each feature.
Avoid aggressive cleaning before using a transformer model. Punctuation, numbers, negation and casing can carry useful meaning.
For example:
Profit increased to $2.1 billion.should not become:
profit increased billionThe removed figure may be the most important part of the sentence.
Converting News into Model Features
After collection, aggregate article-level information into features for a defined time window.
Possible daily features include:
| Feature | Meaning |
article_count | Number of relevant articles available before cutoff |
unique_story_count | Number of deduplicated event clusters |
mean_sentiment | Average sentiment score |
min_sentiment | Most negative article score |
max_sentiment | Most positive article score |
negative_article_ratio | Share of articles classified as negative |
source_count | Number of distinct publishers |
news_volume_zscore | Abnormal news volume relative to history |
hours_since_latest_news | Recency of the latest relevant article |
earnings_news_count | Number of earnings-related stories |
regulatory_news_count | Number of regulatory stories |
Do not calculate a simple average before deduplicating. Twenty copied versions of one negative report should not automatically count as twenty independent negative events.
A weighted sentiment feature could be:
weighted sentiment = sum(sentiment × relevance × recency × source weight)
/ sum(relevance × recency × source weight)Every weighting rule should be learned or justified using training and validation data. Do not tune the rule on the final test period.
Matching Articles to the Correct Company
Company matching is harder than it looks.
Consider the word Apple. It may refer to:
- Apple Inc.
- The fruit
- Apple Records
- A different organisation with Apple in its name
A good entity-linking pipeline uses several signals:
- Full company name
- Ticker with exchange context
- Executives and products
- Industry terms
- Country
- Other companies mentioned
- Publisher section
- Named-entity recognition
Create a relevance score rather than accepting every keyword match.
Example rule:
+3 full company name in headline
+2 recognised product and company in body
+2 ticker written as $AAPL
+1 company name repeated in first paragraph
-3 ambiguous word used without business contextThen manually review samples near the threshold. False company assignments can corrupt sentiment features even when the scraper itself works perfectly.
Common Web-Scraping Problems
1. JavaScript-Rendered Pages
Some HTML responses contain very little article content because JavaScript loads the page after the browser starts.
Before introducing browser automation, look for:
- An official API
- RSS content
- JSON-LD metadata
- Server-rendered article HTML
- A licensed feed
Browser automation is slower and more resource-intensive. It also does not grant permission to access content.
2. Selector Changes
A class such as:
.article-body-v4-finalmay disappear during a redesign.
Prefer structured data and semantic elements, and maintain source-specific tests.
3. Missing or Ambiguous Dates
A page may show:
- Only a local time without timezone
- Only “two hours ago”
- A modified time instead of original publication time
- A date with no time
Do not invent precision. Flag the record for review or exclude it from time-sensitive experiments.
4. Corrected Articles
A publisher may change a headline or correct figures after publication.
Store version observations rather than replacing the earlier record without a trace. The model should see only the version that existed at the relevant time.
5. Paywalls and Login Pages
A successful HTTP status does not prove that article content was retrieved. The scraper may have captured a subscription prompt.
Add quality checks such as:
- Minimum text length
- Required headline
- Valid publication time
- Known boilerplate rejection
- Article-to-navigation text ratio
Never attempt to defeat a paywall or access control.
6. Soft Blocks
A server may return HTTP 200 with a block or challenge page.
Detect unexpected titles, content types and page templates. Stop requesting the source and investigate rather than increasing the request rate or disguising the crawler.
Testing the Acquisition Pipeline
A scraper is a data pipeline and needs tests.
Unit Tests
Test small functions using saved, authorised HTML fixtures:
- URL normalisation
- Date parsing
- Company matching
- JSON-LD extraction
- Content hashing
- Near-duplicate scoring
Source Contract Tests
For each approved source, periodically verify that:
- RSS is reachable
- Article links are valid
- The title is extracted
- The publication time includes a timezone
- Text length remains within an expected range
- The extractor does not collect navigation or legal notices as article text
Data-Quality Checks
Run checks after every batch:
assert all(record.url for record in records)
assert all(record.title for record in records)
assert all(record.collected_at for record in records)
assert len({record.content_hash for record in records}) == len(records)Also report:
- Articles discovered per source
- Articles collected per source
- Blocked or disallowed URLs
- Missing publication timestamps
- Extraction failures
- Duplicate rate
- Empty-text rate
- Articles per company
- Delay from publication to collection
A sudden drop from 500 articles per day to 10 is probably a pipeline problem, not a quiet financial-news day.
Scheduling the Scraper
For a Linux learning environment, a cron entry can run the script every 30 minutes:
*/30 * * * * cd /path/to/stock-news-scraper && /path/to/.venv/bin/python scrape_news.py >> scraper.log 2>&1Production systems should add:
- A scheduler lock to prevent overlapping runs
- Centralised logs
- Failure alerts
- Per-source backoff
- Idempotent writes
- Database transactions
- Secrets management
- Data-retention rules
- Source-level kill switches
Avoid running a frequent schedule merely because it is possible. Match the interval to the authorised access rate and the prediction horizon.
A daily model does not normally require scraping the same source every minute.
Recommended Production Architecture
A larger pipeline can separate the following stages:
- Discovery — receives article URLs from APIs, feeds or approved indexes.
- Policy check — applies source permissions, robots rules and request limits.
- Fetcher — retrieves permitted content and records HTTP metadata.
- Extractor — converts source documents into a common schema.
- Raw storage — preserves authorised original observations.
- Deduplicator — assigns canonical articles and story clusters.
- Entity linker — maps stories to companies, sectors and instruments.
- NLP processor — creates sentiment, event and embedding features.
- Point-in-time joiner — aligns features with valid market timestamps.
- Quality monitor — detects missing coverage and extraction drift.
Each stage should be repeatable and versioned. This makes it possible to improve sentiment analysis without recrawling every source or losing the original observation time.
Mistakes to Avoid
Scraping Everything First
Collecting millions of pages before defining the prediction target creates a large but poorly aligned dataset.
Start with:
- A small list of companies
- A defined prediction horizon
- A limited date range
- A few authorised sources
- A clear timestamp policy
Ignoring Source Changes
If the list of publishers changes over time, a model may learn differences in data coverage rather than genuine market behaviour.
Record source availability and collection failures.
Treating Every Mention as Relevant
An article that mentions a company in a related-links section should not necessarily affect its sentiment score.
Using the Current Page for Historical Backtests
The page viewed today may contain a corrected headline, updated text or changed publication label. It is not automatically a faithful copy of what existed on the historical date.
Randomly Splitting News Records
Near-duplicate stories can enter both the training and test sets. Split chronologically and, where possible, keep a complete story cluster in one split.
Evaluating Only Prediction Accuracy
A model can achieve a reasonable direction accuracy while still producing an unprofitable strategy after spreads, slippage, fees and delayed news processing.
Final Acquisition Checklist
Before using news in a stock-prediction model, confirm that:
- Each source has an approved access method.
- Terms and licences have been reviewed.
robots.txtis checked for HTML crawling.- The crawler identifies itself honestly.
- Requests use timeouts, retries and conservative rate limits.
- Paywalls and access controls are not bypassed.
- Raw and processed data are stored separately.
- Original, modified, collected and available times are distinguished.
- All timestamps are converted to a consistent timezone.
- Missing or ambiguous timestamps are flagged.
- Canonical URLs and tracking parameters are handled.
- Exact and near-duplicate stories are identified.
- Articles are linked to companies using more than ambiguous ticker strings.
- Collection gaps are monitored by source and date.
- News is aligned with the correct trading session and cutoff.
- Train, validation and test periods are separated chronologically.
- The data licence permits the intended storage, modelling and sharing.
Frequently Asked Questions
Is web scraping legal?
There is no single worldwide answer. It depends on the source, data, access method, contract, copyright, privacy rules and jurisdiction. robots.txt is important crawler guidance, but it is not a complete legal permission system. Obtain professional advice for commercial or high-risk use.
Should I scrape financial-news websites or use an API?
Use an authorised API or licensed feed when it provides the coverage you need. It normally produces more stable structured data and reduces scraper maintenance. HTML extraction is useful for permitted research sources that lack a suitable structured interface.
Can Beautiful Soup scrape JavaScript-rendered pages?
Beautiful Soup parses the HTML it receives; it does not run webpage JavaScript. Check for an official API, RSS feed, JSON-LD or server-rendered content before considering browser automation.
How often should a news scraper run?
The frequency should match the prediction horizon, source rules and permitted request rate. A daily model may need only hourly or daily collection. Intraday research requires more careful point-in-time timestamps and latency measurement.
Should I save the complete article text?
Only if your licence and use case permit it. For some projects, storing the headline, authorised summary, metadata, derived features and source URL is more appropriate.
What is the difference between published_at and collected_at?
published_at is the publication time claimed by the source. collected_at is when your system retrieved the record. Both are required to evaluate whether the information was truly available at prediction time.
Why are there duplicate financial-news stories?
Publishers syndicate wire reports, quote one another, update developing stories and create multiple URLs with tracking parameters. Deduplication should use canonical URLs, hashes and similarity or event clustering.
Can sentiment analysis predict stock prices?
Sentiment may be a useful feature, but it cannot guarantee accurate or profitable forecasts. The market reaction depends on expectations, surprise, source credibility, timing, liquidity and whether the information was already priced in.
Which sentiment model should I use?
Compare a finance-specific language model with simple baselines. Validate on later unseen data. A more complicated model is not automatically better, especially when labels are weak or the news-to-company mapping is inaccurate.
Can I use headlines without article bodies?
Yes. Headlines are cheaper to store and often contain the main event, but they can omit qualifications and context. Test headline-only, summary-only and authorised full-text features separately.
How do I avoid look-ahead bias?
Use the earliest defensible availability time, apply the correct exchange timezone and prediction cutoff, and split data chronologically. Do not use later edits or today’s version of a page as if it were the historical version.
What should I do when robots.txt cannot be retrieved?
Choose and document a conservative policy. The tutorial skips crawling when it cannot verify the rules. For a production system, pause that source and review the failure rather than repeatedly requesting pages.
What comes after data acquisition?
The next stages are text cleaning, entity linking, duplicate clustering, sentiment or event extraction, point-in-time aggregation and chronological model evaluation.
Is this financial advice?
No. This tutorial demonstrates data engineering and machine-learning concepts. It does not recommend buying, selling or holding any investment.
Final Result
We have designed a news-acquisition pipeline for a stock-market prediction project.
The completed approach:
- Prioritises licensed feeds, APIs and RSS over unnecessary HTML crawling.
- Checks source rules and
robots.txtbefore fetching permitted pages. - Uses an identifiable user agent, timeouts, throttling and limited retries.
- Extracts structured article metadata before falling back to generic HTML selectors.
- Stores source, URL, publication time, modification time and collection time.
- Matches articles to a controlled list of companies.
- Removes tracking URLs and exact content duplicates.
- Saves machine-readable JSONL and CSV outputs.
- Preserves the information needed for point-in-time backtesting.
- Prepares the records for entity linking, sentiment analysis and event classification.
The most important lesson is that news volume alone does not make a good dataset. A smaller collection with reliable source permissions, accurate timestamps, strong company matching and correct trading-session alignment is more valuable than a huge archive full of duplicates and future information.
The next stage is to transform the collected news into sentiment and event features, join those features to historical market data without leakage and compare the resulting model against price-only baselines.
Leave a Reply