SEO Title: Data Preprocessing for Stock Market Prediction Using AI – Part 2
Meta Description: Learn how to clean, transform and prepare historical stock-market data for an AI prediction model using Python, Pandas, NumPy and Scikit-learn.
Stock-market data cannot normally be fed directly into an artificial intelligence model.
Historical price files may contain missing trading days, duplicate records, incorrect data types and values with very different numerical ranges. Raw columns such as Open, High, Low, Close and Volume also provide limited information unless we transform them into features that help a model recognise market patterns.
In this second part of the stock-market prediction series, we will prepare historical stock data for machine learning.
We will cover:
- Loading historical stock prices
- Inspecting and cleaning the dataset
- Handling missing and duplicate values
- Creating daily returns
- Adding moving averages and technical features
- Creating a prediction target
- Preventing future-data leakage
- Splitting time-series data correctly
- Scaling the features
- Creating sequences for an LSTM model
- Saving the prepared dataset
The completed preprocessing pipeline can later be used with models such as:
- Linear regression
- Random forest
- XGBoost
- Artificial neural networks
- Long short-term memory networks, or LSTMs
Quick Answer
A basic stock-market preprocessing pipeline follows these steps:
Historical stock data
↓
Clean and sort the records
↓
Create technical features
↓
Create the prediction target
↓
Remove unavailable values
↓
Split data by date
↓
Fit the scaler on training data
↓
Transform the data into model-ready arrays
The most important rule is that the model must not receive information from the future.
For example, if we want to predict tomorrow’s closing price, today’s model input may contain today’s:
- Opening price
- Highest price
- Lowest price
- Closing price
- Trading volume
- Moving averages
- Volatility
- Momentum indicators
It must not contain tomorrow’s price or any indicator calculated using future records.
Why Data Preprocessing Matters
An AI model learns from the information it receives. If the input data is badly prepared, even an advanced model can produce unreliable results.
Consider the following raw data:
| Date | Open | High | Low | Close | Volume |
|---|---|---|---|---|---|
| 2024-05-20 | 189.33 | 191.92 | 189.01 | 191.04 | 44,320,000 |
| 2024-05-21 | 191.09 | 192.73 | 190.92 | 192.35 | 42,350,000 |
| 2024-05-22 | 192.27 | 192.82 | 190.27 | 190.90 | 34,610,000 |
A prediction model could use these values directly, but it would have difficulty identifying useful relationships such as:
- Whether the price increased from the previous day
- Whether the stock is above its recent average
- Whether recent prices are unusually volatile
- Whether trading activity is increasing
- Whether short-term momentum is positive or negative
Preprocessing converts the raw records into structured features that make these relationships easier for the model to learn.
What We Are Going to Predict
Before preparing the data, we must define the prediction objective.
A model could attempt to predict:
- Tomorrow’s closing price
- Tomorrow’s percentage return
- Whether the price will rise or fall
- The highest price during the next trading day
- Price movement over the next five trading days
- Whether the stock will outperform a market index
For this tutorial, we will prepare two possible targets:
Next_Close: tomorrow’s closing priceTarget_Up: whether tomorrow’s closing price is higher than today’s
Next_Close can be used for a regression model.
Target_Up can be used for a classification model.
Step 1: Install the Required Python Packages
Open a terminal and install the required packages:
pip install pandas numpy matplotlib scikit-learn yfinance joblib
The packages serve the following purposes:
| Package | Purpose |
|---|---|
| Pandas | Loading and manipulating tabular data |
| NumPy | Numerical calculations |
| Matplotlib | Plotting prices and processed features |
| Scikit-learn | Scaling data and preparing machine-learning input |
| yfinance | Downloading historical market data |
| Joblib | Saving the fitted scaler |
If you already have a CSV file containing historical prices, yfinance is optional.
Step 2: Import the Libraries
Create a Python file named preprocess_stock_data.py.
Add the following imports:
import joblib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler
If you want to download data directly:
import yfinance as yf
Step 3: Download Historical Stock Data
For this example, we will use Apple stock under the ticker symbol AAPL.
ticker = "AAPL"
data = yf.download(
ticker,
start="2018-01-01",
end="2024-05-26",
auto_adjust=False,
progress=False
)
print(data.head())
Depending on the library version and the selected ticker, the returned dataset may contain columns such as:
Date
Open
High
Low
Close
Adj Close
Volume
The date may be stored as the index rather than a normal column.
Convert it into a column:
data = data.reset_index()
Then inspect the first few records:
print(data.head())
Example output:
Date Open High Low Close Adj Close Volume
0 2018-01-02 42.540001 43.075001 42.314999 43.064999 40.568928 102223600
1 2018-01-03 43.132500 43.637501 42.990002 43.057499 40.561871 118071600
2 2018-01-04 43.134998 43.367500 43.020000 43.257500 40.750252 89738400
The exact values will depend on the selected date range and whether adjusted prices are used.
Loading Stock Data from a CSV File
If the data has already been downloaded, use:
data = pd.read_csv("AAPL_historical_data.csv")
Inspect it:
print(data.head())
print(data.columns)
print(data.shape)
If the date is stored in a column named Date, convert it into a proper date type:
data["Date"] = pd.to_datetime(
data["Date"],
errors="coerce"
)
Using errors="coerce" changes invalid dates into missing values. These can then be detected and removed during cleaning.
Step 4: Inspect the Dataset
Before changing the data, examine its structure.
print(data.info())
print(data.describe())
Check the number of missing values:
print(data.isnull().sum())
Check the number of duplicate rows:
print("Duplicate rows:", data.duplicated().sum())
Check the date range:
print("First date:", data["Date"].min())
print("Last date:", data["Date"].max())
This helps identify problems such as:
- Dates stored as text
- Prices stored as strings
- Missing closing prices
- Duplicate trading days
- Zero or negative prices
- Abnormally low or high trading volumes
Step 5: Select the Required Columns
For this tutorial, we need:
required_columns = [
"Date",
"Open",
"High",
"Low",
"Close",
"Volume"
]
data = data[required_columns].copy()
If you prefer adjusted closing prices, you can use Adj Close instead of Close.
Adjusted prices account for events such as stock splits and dividends. They may be more suitable when analysing long-term returns.
However, you should be consistent. Do not create some features from adjusted prices and others from unadjusted prices without understanding the difference.
Step 6: Convert Columns to the Correct Data Types
Make sure the date is recognised correctly:
data["Date"] = pd.to_datetime(
data["Date"],
errors="coerce"
)
Convert the numerical columns:
numeric_columns = [
"Open",
"High",
"Low",
"Close",
"Volume"
]
for column in numeric_columns:
data[column] = pd.to_numeric(
data[column],
errors="coerce"
)
Values that cannot be converted become NaN, which represents a missing value.
Check the result:
print(data.dtypes)
Expected output:
Date datetime64[ns]
Open float64
High float64
Low float64
Close float64
Volume int64
The exact numerical types may differ slightly.
Step 7: Sort the Records by Date
Time-series records must be arranged chronologically.
data = data.sort_values("Date")
data = data.reset_index(drop=True)
Do not assume that a CSV file is already sorted.
An incorrectly ordered dataset can produce invalid moving averages, returns and training sequences.
Step 8: Remove Duplicate Records
A dataset may contain the same trading day more than once.
Check duplicates based on the date:
duplicate_dates = data.duplicated(
subset=["Date"],
keep=False
)
print(data[duplicate_dates])
Remove duplicate dates:
data = data.drop_duplicates(
subset=["Date"],
keep="last"
)
Using keep="last" retains the last record for each duplicated date.
If the duplicate records contain conflicting prices, investigate where the data came from before deciding which record to keep.
Step 9: Handle Missing Values
Check missing values again:
print(data.isnull().sum())
Rows without a valid date or closing price should normally be removed:
data = data.dropna(
subset=["Date", "Close"]
)
For other price columns, one option is to use forward filling:
price_columns = [
"Open",
"High",
"Low",
"Close"
]
data[price_columns] = (
data[price_columns]
.ffill()
)
Forward filling replaces a missing value with the most recent available value.
However, do not automatically fill every missing record without checking why it is missing.
A safer approach for a small number of incomplete rows is:
data = data.dropna(
subset=[
"Open",
"High",
"Low",
"Close",
"Volume"
]
)
This tutorial will use complete rows:
data = data.dropna().copy()
Do Weekends Count as Missing Data?
Stock markets normally close on weekends and public holidays.
Therefore, the absence of Saturday and Sunday records does not automatically mean the dataset is incomplete.
Do not create artificial weekend prices unless your model has a specific reason to require calendar-day data.
For daily stock prediction, it is usually better to work with actual trading days.
Step 10: Validate the Price Records
Basic validation can identify obviously invalid rows.
invalid_rows = data[
(data["Open"] <= 0) |
(data["High"] <= 0) |
(data["Low"] <= 0) |
(data["Close"] <= 0) |
(data["Volume"] < 0)
]
print(invalid_rows)
We can also check whether the daily high and low are logically consistent:
invalid_price_ranges = data[
(data["High"] < data["Low"]) |
(data["High"] < data["Open"]) |
(data["High"] < data["Close"]) |
(data["Low"] > data["Open"]) |
(data["Low"] > data["Close"])
]
print(invalid_price_ranges)
If these records exist, confirm them against the original source before deleting or changing them.
To remove clearly invalid rows:
data = data[
(data["Open"] > 0) &
(data["High"] > 0) &
(data["Low"] > 0) &
(data["Close"] > 0) &
(data["Volume"] >= 0) &
(data["High"] >= data["Low"])
].copy()
Step 11: Create Daily Return Features
The absolute closing price tells us the stock’s value, but it does not directly show how much it moved.
Create a daily percentage return:
data["Daily_Return"] = (
data["Close"]
.pct_change()
)
The calculation is:
(Current closing price − Previous closing price)
-------------------------------------------------
Previous closing price
For example, if the closing price increases from $100 to $103:
Daily return = (103 − 100) ÷ 100
= 0.03
= 3%
Create a logarithmic return:
data["Log_Return"] = np.log(
data["Close"] /
data["Close"].shift(1)
)
Logarithmic returns are commonly used in financial analysis because returns across several periods can be added together more conveniently.
Step 12: Create Intraday Price Features
We can create features describing what happened within each trading day.
Opening-to-Closing Change
data["Open_Close_Change"] = (
data["Close"] - data["Open"]
) / data["Open"]
A positive value means the stock closed above its opening price.
Daily Trading Range
data["High_Low_Range"] = (
data["High"] - data["Low"]
) / data["Close"]
This represents the size of the day’s price range relative to the closing price.
Previous Closing Gap
data["Previous_Close"] = data["Close"].shift(1)
data["Opening_Gap"] = (
data["Open"] - data["Previous_Close"]
) / data["Previous_Close"]
A positive opening gap means the stock opened above the previous trading day’s closing price.
Step 13: Add Moving Averages
Moving averages smooth short-term price changes.
Create 5-day, 10-day, 20-day and 50-day simple moving averages:
data["SMA_5"] = (
data["Close"]
.rolling(window=5)
.mean()
)
data["SMA_10"] = (
data["Close"]
.rolling(window=10)
.mean()
)
data["SMA_20"] = (
data["Close"]
.rolling(window=20)
.mean()
)
data["SMA_50"] = (
data["Close"]
.rolling(window=50)
.mean()
)
The 5-day moving average represents approximately one trading week.
The 20-day moving average represents approximately one trading month.
The 50-day moving average provides a broader view of the price trend.
Create exponential moving averages:
data["EMA_12"] = (
data["Close"]
.ewm(span=12, adjust=False)
.mean()
)
data["EMA_26"] = (
data["Close"]
.ewm(span=26, adjust=False)
.mean()
)
Exponential moving averages give more weight to recent prices.
Step 14: Create Price-to-Moving-Average Ratios
A raw moving average may mostly repeat information already contained in the closing price.
Ratios can make the relationship clearer:
data["Close_to_SMA_5"] = (
data["Close"] / data["SMA_5"]
)
data["Close_to_SMA_20"] = (
data["Close"] / data["SMA_20"]
)
data["SMA_5_to_SMA_20"] = (
data["SMA_5"] / data["SMA_20"]
)
Interpretation:
- A value above
1means the first value is higher - A value below
1means the first value is lower - A value near
1means the values are similar
For example:
Close-to-SMA ratio = 1.04
This means the closing price is approximately 4% above its moving average.
Step 15: Add Rolling Volatility
Volatility describes how widely returns have fluctuated.
Create 5-day and 20-day rolling volatility:
data["Volatility_5"] = (
data["Daily_Return"]
.rolling(window=5)
.std()
)
data["Volatility_20"] = (
data["Daily_Return"]
.rolling(window=20)
.std()
)
A larger value indicates that recent daily returns have varied more widely.
The calculation only uses the current and previous records. It does not use future returns.
Step 16: Create Volume Features
Trading volume can provide useful information about market participation.
Create a 20-day average volume:
data["Volume_SMA_20"] = (
data["Volume"]
.rolling(window=20)
.mean()
)
Then compare current volume with the average:
data["Relative_Volume"] = (
data["Volume"] /
data["Volume_SMA_20"]
)
Example:
Relative volume = 1.50
This means the current volume is approximately 50% higher than its 20-day average.
Create the daily volume change:
data["Volume_Change"] = (
data["Volume"]
.pct_change()
)
Very large volume changes can create extreme values. These may need to be capped or transformed after inspecting their distribution.
Step 17: Add Momentum
Momentum measures how much the price has changed over a selected period.
data["Momentum_5"] = (
data["Close"] /
data["Close"].shift(5)
) - 1
data["Momentum_10"] = (
data["Close"] /
data["Close"].shift(10)
) - 1
A Momentum_5 value of 0.06 means the price has increased by approximately 6% over five trading days.
A value of -0.04 means it has decreased by approximately 4%.
Step 18: Calculate the Relative Strength Index
The Relative Strength Index, or RSI, compares recent gains with recent losses.
def calculate_rsi(series, period=14):
price_change = series.diff()
gains = price_change.clip(lower=0)
losses = -price_change.clip(upper=0)
average_gain = gains.ewm(
alpha=1 / period,
adjust=False,
min_periods=period
).mean()
average_loss = losses.ewm(
alpha=1 / period,
adjust=False,
min_periods=period
).mean()
relative_strength = (
average_gain / average_loss
)
rsi = 100 - (
100 / (1 + relative_strength)
)
return rsi
Apply it:
data["RSI_14"] = calculate_rsi(
data["Close"],
period=14
)
RSI is normally displayed on a scale from 0 to 100.
It should be treated as a numerical feature, not a guaranteed buy or sell signal.
Step 19: Add MACD
The Moving Average Convergence Divergence indicator uses the difference between two exponential moving averages.
data["MACD"] = (
data["EMA_12"] -
data["EMA_26"]
)
data["MACD_Signal"] = (
data["MACD"]
.ewm(span=9, adjust=False)
.mean()
)
data["MACD_Histogram"] = (
data["MACD"] -
data["MACD_Signal"]
)
The model can use these values as possible momentum and trend features.
Step 20: Add Bollinger Band Features
Bollinger Bands use a moving average and rolling standard deviation.
data["BB_Middle"] = (
data["Close"]
.rolling(window=20)
.mean()
)
data["BB_Std"] = (
data["Close"]
.rolling(window=20)
.std()
)
data["BB_Upper"] = (
data["BB_Middle"] +
2 * data["BB_Std"]
)
data["BB_Lower"] = (
data["BB_Middle"] -
2 * data["BB_Std"]
)
Create the price’s relative position within the bands:
band_width = (
data["BB_Upper"] -
data["BB_Lower"]
)
data["BB_Position"] = (
data["Close"] -
data["BB_Lower"]
) / band_width
Create a band-width feature:
data["BB_Width"] = (
band_width /
data["BB_Middle"]
)
Avoid dividing by zero:
data.replace(
[np.inf, -np.inf],
np.nan,
inplace=True
)
Step 21: Add Calendar Features
Markets may behave differently across days, months or reporting periods.
Create basic calendar features:
data["Day_Of_Week"] = (
data["Date"].dt.dayofweek
)
data["Month"] = (
data["Date"].dt.month
)
data["Quarter"] = (
data["Date"].dt.quarter
)
Day_Of_Week uses:
| Value | Day |
|---|---|
| 0 | Monday |
| 1 | Tuesday |
| 2 | Wednesday |
| 3 | Thursday |
| 4 | Friday |
These calendar features do not guarantee a useful pattern. Their value should be evaluated during model testing.
Step 22: Create the Prediction Target
Regression Target: Tomorrow’s Closing Price
data["Next_Close"] = (
data["Close"]
.shift(-1)
)
For each row, Next_Close contains the closing price from the following trading record.
Example:
| Date | Close | Next_Close |
|---|---|---|
| Monday | 100.00 | 102.00 |
| Tuesday | 102.00 | 101.50 |
| Wednesday | 101.50 | 104.00 |
The last record will not have a future closing price, so its target will be missing.
Classification Target: Will the Price Increase?
data["Target_Up"] = (
data["Next_Close"] >
data["Close"]
).astype(int)
The result is:
1if tomorrow’s closing price is higher0if tomorrow’s closing price is equal or lower
We should calculate Target_Up before removing the missing Next_Close row.
Predicting Tomorrow’s Return Instead
Another possible regression target is the next-day return:
data["Target_Return"] = (
data["Next_Close"] /
data["Close"]
) - 1
This may be easier to compare across stocks with different prices.
For example, a $5 price change means something very different for a $20 stock and a $500 stock. A percentage return provides a more consistent scale.
Step 23: Remove Rows Made Incomplete by Feature Creation
Rolling indicators create missing values at the beginning of the dataset.
For example, a 50-day moving average requires 50 trading records before it becomes available.
The shifted prediction target also creates a missing value at the end.
Replace infinite values and remove incomplete rows:
data = data.replace(
[np.inf, -np.inf],
np.nan
)
data = data.dropna().copy()
data = data.reset_index(drop=True)
Check the prepared dataset:
print(data.head())
print(data.tail())
print(data.shape)
print(data.isnull().sum())
Do not fill the missing target with zero. A zero future price or return would create false training examples.
Step 24: Select the Model Features
Choose the columns the model will receive.
feature_columns = [
"Open",
"High",
"Low",
"Close",
"Volume",
"Daily_Return",
"Log_Return",
"Open_Close_Change",
"High_Low_Range",
"Opening_Gap",
"SMA_5",
"SMA_10",
"SMA_20",
"SMA_50",
"EMA_12",
"EMA_26",
"Close_to_SMA_5",
"Close_to_SMA_20",
"SMA_5_to_SMA_20",
"Volatility_5",
"Volatility_20",
"Relative_Volume",
"Volume_Change",
"Momentum_5",
"Momentum_10",
"RSI_14",
"MACD",
"MACD_Signal",
"MACD_Histogram",
"BB_Position",
"BB_Width",
"Day_Of_Week",
"Month",
"Quarter"
]
Create the feature matrix and classification target:
X = data[feature_columns].copy()
y = data["Target_Up"].copy()
For closing-price regression, use:
y = data["Next_Close"].copy()
Step 25: Check the Target Distribution
For a direction-classification model, inspect how many examples belong to each class:
print(y.value_counts())
print(y.value_counts(normalize=True))
Example:
1 0.53
0 0.47
This means 53% of the records are upward days and 47% are non-upward days.
If one class is much larger than the other, accuracy alone can be misleading.
For example, if 70% of the samples are upward days, a model that always predicts “up” would achieve 70% accuracy without learning anything useful.
Later evaluation should include measures such as:
- Precision
- Recall
- F1 score
- Confusion matrix
- Balanced accuracy
- Strategy return after costs
Step 26: Split the Data Chronologically
Do not randomly shuffle stock-market data before splitting it.
A random split could place records from 2023 in the training set and records from 2020 in the test set. This does not represent a realistic prediction process.
Use the earlier records for training and the later records for testing.
split_index = int(
len(data) * 0.80
)
train_data = data.iloc[
:split_index
].copy()
test_data = data.iloc[
split_index:
].copy()
Check the periods:
print(
"Training period:",
train_data["Date"].min(),
"to",
train_data["Date"].max()
)
print(
"Testing period:",
test_data["Date"].min(),
"to",
test_data["Date"].max()
)
Create the training and test variables:
X_train = train_data[
feature_columns
].copy()
X_test = test_data[
feature_columns
].copy()
y_train = train_data[
"Target_Up"
].copy()
y_test = test_data[
"Target_Up"
].copy()
Step 27: Scale the Features Correctly
Stock-market columns use very different ranges.
For example:
Closing price: 50 to 200
RSI: 0 to 100
Daily return: -0.1 to 0.1
Trading volume: 20,000,000 to 200,000,000
Large values can dominate the training process for models that are sensitive to feature scale.
Use MinMaxScaler:
scaler = MinMaxScaler(
feature_range=(0, 1)
)
Fit it using only the training data:
X_train_scaled = scaler.fit_transform(
X_train
)
Transform the test data using the same fitted scaler:
X_test_scaled = scaler.transform(
X_test
)
This is important:
scaler.fit(X_train)
is correct.
This is incorrect:
scaler.fit(pd.concat([X_train, X_test]))
Fitting the scaler on the complete dataset allows information from the test period to affect the training transformation. This is a form of data leakage.
Why Scaling the Entire Dataset Is a Mistake
Suppose the highest stock price in the training period is $150, but the price reaches $220 during the test period.
If the scaler is fitted using the complete dataset, it learns about the future $220 value.
Although this may appear harmless, information from the test period has influenced the training preparation.
In a proper experiment:
- Fit all preprocessing operations on the training data.
- Apply the fitted operations to validation and test data.
- Never recalculate them using future observations.
Step 28: Convert the Scaled Data Back to DataFrames
NumPy arrays do not retain column names.
For easier inspection:
X_train_scaled = pd.DataFrame(
X_train_scaled,
columns=feature_columns,
index=X_train.index
)
X_test_scaled = pd.DataFrame(
X_test_scaled,
columns=feature_columns,
index=X_test.index
)
Check the results:
print(X_train_scaled.head())
print(X_train_scaled.min())
print(X_train_scaled.max())
Most training values should now fall between 0 and 1.
Some test values may fall below 0 or above 1 if the later market data moves outside the training range. This is possible and does not automatically mean scaling failed.
Step 29: Prepare LSTM Sequences
A standard machine-learning model normally receives one row at a time.
An LSTM model usually receives a sequence of consecutive records.
For example, if the sequence length is 30:
Previous 30 trading days → Predict the following day
Create a sequence function:
def create_sequences(
features,
targets,
sequence_length=30
):
X_sequences = []
y_sequences = []
feature_values = np.asarray(features)
target_values = np.asarray(targets)
for index in range(
sequence_length,
len(feature_values)
):
start_index = (
index - sequence_length
)
X_sequences.append(
feature_values[
start_index:index
]
)
y_sequences.append(
target_values[index]
)
return (
np.array(X_sequences),
np.array(y_sequences)
)
Create the training sequences:
sequence_length = 30
X_train_sequence, y_train_sequence = (
create_sequences(
X_train_scaled,
y_train,
sequence_length
)
)
Check their shapes:
print(
"X training shape:",
X_train_sequence.shape
)
print(
"y training shape:",
y_train_sequence.shape
)
Example:
X training shape: (1200, 30, 34)
y training shape: (1200,)
This means:
- 1,200 training examples
- 30 trading days per example
- 34 features per trading day
Creating Test Sequences Without Losing Context
The first test prediction may need the final 30 records from the training period.
If sequences are created only from X_test_scaled, the beginning of the test period will be discarded.
Combine the final training context with the test features:
test_context = pd.concat([
X_train_scaled.tail(
sequence_length
),
X_test_scaled
])
Prepare aligned targets:
test_targets_with_context = pd.concat([
y_train.tail(sequence_length),
y_test
])
Create sequences:
X_test_sequence, y_test_sequence = (
create_sequences(
test_context,
test_targets_with_context,
sequence_length
)
)
Check the result:
print(
"X test shape:",
X_test_sequence.shape
)
print(
"y test shape:",
y_test_sequence.shape
)
The number of test sequences should now match the number of test targets.
Step 30: Save the Prepared Data
Save the full feature dataset:
data.to_csv(
"AAPL_processed_data.csv",
index=False
)
Save the chronological training and test datasets:
train_data.to_csv(
"AAPL_train_data.csv",
index=False
)
test_data.to_csv(
"AAPL_test_data.csv",
index=False
)
Save the fitted scaler:
joblib.dump(
scaler,
"AAPL_feature_scaler.joblib"
)
Save the LSTM arrays:
np.save(
"X_train_sequence.npy",
X_train_sequence
)
np.save(
"y_train_sequence.npy",
y_train_sequence
)
np.save(
"X_test_sequence.npy",
X_test_sequence
)
np.save(
"y_test_sequence.npy",
y_test_sequence
)
The same scaler must be loaded when preparing new data for future predictions:
scaler = joblib.load(
"AAPL_feature_scaler.joblib"
)
Do not fit a new scaler every time one prediction is requested.
Complete Preprocessing Example
The following code combines the main steps into one script:
import joblib
import numpy as np
import pandas as pd
import yfinance as yf
from sklearn.preprocessing import MinMaxScaler
def calculate_rsi(series, period=14):
price_change = series.diff()
gains = price_change.clip(lower=0)
losses = -price_change.clip(upper=0)
average_gain = gains.ewm(
alpha=1 / period,
adjust=False,
min_periods=period
).mean()
average_loss = losses.ewm(
alpha=1 / period,
adjust=False,
min_periods=period
).mean()
relative_strength = (
average_gain / average_loss
)
return 100 - (
100 / (1 + relative_strength)
)
ticker = "AAPL"
data = yf.download(
ticker,
start="2018-01-01",
end="2024-05-26",
auto_adjust=False,
progress=False
)
data = data.reset_index()
if isinstance(
data.columns,
pd.MultiIndex
):
data.columns = (
data.columns
.get_level_values(0)
)
required_columns = [
"Date",
"Open",
"High",
"Low",
"Close",
"Volume"
]
data = data[
required_columns
].copy()
data["Date"] = pd.to_datetime(
data["Date"],
errors="coerce"
)
numeric_columns = [
"Open",
"High",
"Low",
"Close",
"Volume"
]
for column in numeric_columns:
data[column] = pd.to_numeric(
data[column],
errors="coerce"
)
data = data.drop_duplicates(
subset=["Date"],
keep="last"
)
data = data.sort_values("Date")
data = data.dropna()
data = data.reset_index(drop=True)
data = data[
(data["Open"] > 0) &
(data["High"] > 0) &
(data["Low"] > 0) &
(data["Close"] > 0) &
(data["Volume"] >= 0) &
(data["High"] >= data["Low"])
].copy()
data["Daily_Return"] = (
data["Close"].pct_change()
)
data["Log_Return"] = np.log(
data["Close"] /
data["Close"].shift(1)
)
data["Open_Close_Change"] = (
data["Close"] -
data["Open"]
) / data["Open"]
data["High_Low_Range"] = (
data["High"] -
data["Low"]
) / data["Close"]
data["Previous_Close"] = (
data["Close"].shift(1)
)
data["Opening_Gap"] = (
data["Open"] -
data["Previous_Close"]
) / data["Previous_Close"]
for period in [5, 10, 20, 50]:
data[f"SMA_{period}"] = (
data["Close"]
.rolling(period)
.mean()
)
data["EMA_12"] = (
data["Close"]
.ewm(span=12, adjust=False)
.mean()
)
data["EMA_26"] = (
data["Close"]
.ewm(span=26, adjust=False)
.mean()
)
data["Close_to_SMA_5"] = (
data["Close"] /
data["SMA_5"]
)
data["Close_to_SMA_20"] = (
data["Close"] /
data["SMA_20"]
)
data["SMA_5_to_SMA_20"] = (
data["SMA_5"] /
data["SMA_20"]
)
data["Volatility_5"] = (
data["Daily_Return"]
.rolling(5)
.std()
)
data["Volatility_20"] = (
data["Daily_Return"]
.rolling(20)
.std()
)
data["Volume_SMA_20"] = (
data["Volume"]
.rolling(20)
.mean()
)
data["Relative_Volume"] = (
data["Volume"] /
data["Volume_SMA_20"]
)
data["Volume_Change"] = (
data["Volume"]
.pct_change()
)
data["Momentum_5"] = (
data["Close"] /
data["Close"].shift(5)
) - 1
data["Momentum_10"] = (
data["Close"] /
data["Close"].shift(10)
) - 1
data["RSI_14"] = calculate_rsi(
data["Close"],
period=14
)
data["MACD"] = (
data["EMA_12"] -
data["EMA_26"]
)
data["MACD_Signal"] = (
data["MACD"]
.ewm(span=9, adjust=False)
.mean()
)
data["MACD_Histogram"] = (
data["MACD"] -
data["MACD_Signal"]
)
data["BB_Middle"] = (
data["Close"]
.rolling(20)
.mean()
)
data["BB_Std"] = (
data["Close"]
.rolling(20)
.std()
)
data["BB_Upper"] = (
data["BB_Middle"] +
2 * data["BB_Std"]
)
data["BB_Lower"] = (
data["BB_Middle"] -
2 * data["BB_Std"]
)
band_width = (
data["BB_Upper"] -
data["BB_Lower"]
)
data["BB_Position"] = (
data["Close"] -
data["BB_Lower"]
) / band_width
data["BB_Width"] = (
band_width /
data["BB_Middle"]
)
data["Day_Of_Week"] = (
data["Date"].dt.dayofweek
)
data["Month"] = (
data["Date"].dt.month
)
data["Quarter"] = (
data["Date"].dt.quarter
)
data["Next_Close"] = (
data["Close"].shift(-1)
)
data["Target_Return"] = (
data["Next_Close"] /
data["Close"]
) - 1
data["Target_Up"] = (
data["Next_Close"] >
data["Close"]
).astype(int)
data = data.replace(
[np.inf, -np.inf],
np.nan
)
data = data.dropna()
data = data.reset_index(drop=True)
feature_columns = [
"Open",
"High",
"Low",
"Close",
"Volume",
"Daily_Return",
"Log_Return",
"Open_Close_Change",
"High_Low_Range",
"Opening_Gap",
"SMA_5",
"SMA_10",
"SMA_20",
"SMA_50",
"EMA_12",
"EMA_26",
"Close_to_SMA_5",
"Close_to_SMA_20",
"SMA_5_to_SMA_20",
"Volatility_5",
"Volatility_20",
"Relative_Volume",
"Volume_Change",
"Momentum_5",
"Momentum_10",
"RSI_14",
"MACD",
"MACD_Signal",
"MACD_Histogram",
"BB_Position",
"BB_Width",
"Day_Of_Week",
"Month",
"Quarter"
]
split_index = int(
len(data) * 0.80
)
train_data = data.iloc[
:split_index
].copy()
test_data = data.iloc[
split_index:
].copy()
X_train = train_data[
feature_columns
]
X_test = test_data[
feature_columns
]
y_train = train_data[
"Target_Up"
]
y_test = test_data[
"Target_Up"
]
scaler = MinMaxScaler()
X_train_scaled = (
scaler.fit_transform(
X_train
)
)
X_test_scaled = (
scaler.transform(
X_test
)
)
data.to_csv(
f"{ticker}_processed_data.csv",
index=False
)
train_data.to_csv(
f"{ticker}_train_data.csv",
index=False
)
test_data.to_csv(
f"{ticker}_test_data.csv",
index=False
)
joblib.dump(
scaler,
f"{ticker}_feature_scaler.joblib"
)
np.save(
f"{ticker}_X_train.npy",
X_train_scaled
)
np.save(
f"{ticker}_X_test.npy",
X_test_scaled
)
np.save(
f"{ticker}_y_train.npy",
y_train.to_numpy()
)
np.save(
f"{ticker}_y_test.npy",
y_test.to_numpy()
)
print("Preprocessing completed.")
print("Total usable rows:", len(data))
print("Training rows:", len(train_data))
print("Testing rows:", len(test_data))
print("Number of features:", len(feature_columns))
Visualise the Prepared Data
Before training a model, plot some of the processed features.
Closing Price and Moving Averages
plt.figure(figsize=(14, 7))
plt.plot(
data["Date"],
data["Close"],
label="Closing Price",
linewidth=1
)
plt.plot(
data["Date"],
data["SMA_20"],
label="20-Day SMA"
)
plt.plot(
data["Date"],
data["SMA_50"],
label="50-Day SMA"
)
plt.title(
f"{ticker} Closing Price and Moving Averages"
)
plt.xlabel("Date")
plt.ylabel("Price")
plt.legend()
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()
Daily Returns
plt.figure(figsize=(14, 5))
plt.plot(
data["Date"],
data["Daily_Return"],
linewidth=0.8
)
plt.axhline(
0,
color="black",
linewidth=0.8
)
plt.title(
f"{ticker} Daily Returns"
)
plt.xlabel("Date")
plt.ylabel("Return")
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()
Visual inspection can reveal:
- Missing periods
- Unusual price jumps
- Stock splits
- Extremely large volume changes
- Features that appear incorrectly calculated
Preventing Data Leakage
Data leakage occurs when the training process receives information that would not have been available at prediction time.
It can make a model appear highly accurate during testing while failing on new market data.
Leakage Example 1: Using Tomorrow’s Price as a Feature
This is incorrect:
feature_columns = [
"Close",
"Next_Close"
]
Next_Close is the value we are trying to predict. It must only be used as the target.
Leakage Example 2: Centred Moving Average
This can use future observations:
data["Wrong_SMA"] = (
data["Close"]
.rolling(
window=5,
center=True
)
.mean()
)
Because center=True places the current date in the middle of the window, later prices may be included.
Use:
data["SMA_5"] = (
data["Close"]
.rolling(window=5)
.mean()
)
Leakage Example 3: Fitting the Scaler on All Data
Incorrect:
scaler.fit(data[feature_columns])
Correct:
scaler.fit(train_data[feature_columns])
Leakage Example 4: Randomly Shuffling Time-Series Records
This is normally inappropriate:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = (
train_test_split(
X,
y,
test_size=0.20,
shuffle=True
)
)
A chronological split better represents a real prediction scenario.
Leakage Example 5: Backfilling Missing Features
This can copy a future value backwards:
data = data.bfill()
For market prediction, backfilling can expose future information to earlier rows.
Dropping incomplete rows or using carefully justified past-only methods is safer.
Should Outliers Be Removed?
Large price movements are not always errors.
A sudden 10% move may be caused by:
- An earnings announcement
- A takeover proposal
- A regulatory decision
- A product announcement
- A broader market crash
- A stock split or data-adjustment problem
Do not remove a value simply because it is unusual.
First determine whether it represents:
- A genuine market event
- A stock split
- A currency or unit error
- A duplicated record
- A corrupted data point
Real market shocks may be important training information.
For extreme volume-change features, you may use clipping after determining suitable limits from the training data:
lower_limit = (
train_data["Volume_Change"]
.quantile(0.01)
)
upper_limit = (
train_data["Volume_Change"]
.quantile(0.99)
)
train_data["Volume_Change"] = (
train_data["Volume_Change"]
.clip(
lower=lower_limit,
upper=upper_limit
)
)
test_data["Volume_Change"] = (
test_data["Volume_Change"]
.clip(
lower=lower_limit,
upper=upper_limit
)
)
The clipping limits must be calculated from the training data only.
Feature Correlation
Check how strongly the numerical features are related:
correlation_matrix = (
data[feature_columns]
.corr()
)
print(
correlation_matrix["Close"]
.sort_values(
ascending=False
)
)
Columns such as Close, SMA 20, SMA 50 and EMA 26 may be highly correlated because they are all derived from the same price series.
High correlation does not automatically mean that a feature must be removed. However, keeping many nearly identical features can:
- Increase model complexity
- Make some models unstable
- Provide little additional information
- Increase the chance of overfitting
Feature selection should be evaluated using validation data, not based only on the training score.
Better Evaluation with Walk-Forward Validation
A single 80/20 split provides one view of performance.
Walk-forward validation tests the model across several later periods.
Example:
Train: 2018–2020 → Validate: 2021
Train: 2018–2021 → Validate: 2022
Train: 2018–2022 → Validate: 2023
Train: 2018–2023 → Test: 2024
Scikit-learn provides TimeSeriesSplit:
from sklearn.model_selection import TimeSeriesSplit
time_series_split = TimeSeriesSplit(
n_splits=5
)
X_all = data[feature_columns]
y_all = data["Target_Up"]
for fold, (
train_index,
validation_index
) in enumerate(
time_series_split.split(X_all),
start=1
):
X_fold_train = X_all.iloc[
train_index
]
X_fold_validation = X_all.iloc[
validation_index
]
y_fold_train = y_all.iloc[
train_index
]
y_fold_validation = y_all.iloc[
validation_index
]
fold_scaler = MinMaxScaler()
X_fold_train_scaled = (
fold_scaler.fit_transform(
X_fold_train
)
)
X_fold_validation_scaled = (
fold_scaler.transform(
X_fold_validation
)
)
print(
f"Fold {fold}:",
len(train_index),
"training records and",
len(validation_index),
"validation records"
)
Each fold uses only earlier records to predict a later period.
Common Data Preprocessing Mistakes
1. Using Too Little Historical Data
A neural network trained on only a few months of daily records may not have enough examples to learn a reliable pattern.
Daily market data provides roughly 250 trading records per year.
Five years of daily data may provide only around 1,250 observations before removing incomplete rows.
That is not a large dataset for a complex neural network.
2. Adding Too Many Indicators
Adding dozens of technical indicators does not guarantee better predictions.
Many indicators are calculated from the same price and volume data. This can create highly repetitive features.
Start with a smaller, explainable feature set and compare validation results.
3. Ignoring Trading Costs
A model may correctly predict small price movements but still lose money after:
- Brokerage fees
- Bid-ask spreads
- Slippage
- Taxes or exchange fees
- Currency-conversion costs
Prediction accuracy and trading profitability are not the same thing.
4. Training on Only One Market Condition
A model trained only during a strong bull market may perform poorly during:
- Bear markets
- Sideways markets
- High-interest-rate periods
- Financial crises
- Unusually volatile periods
The dataset should include different market conditions where possible.
5. Assuming a High Accuracy Means Future Profit
Financial markets change over time.
A pattern found in older data may disappear after:
- Market participants adapt
- Trading rules change
- Company conditions change
- Economic conditions shift
- The stock enters a different growth stage
A model must be tested on later unseen data.
Recommended Basic Feature Set
If you want to begin with a smaller dataset, use:
basic_features = [
"Daily_Return",
"Open_Close_Change",
"High_Low_Range",
"Close_to_SMA_5",
"Close_to_SMA_20",
"Volatility_5",
"Volatility_20",
"Relative_Volume",
"Momentum_5",
"Momentum_10",
"RSI_14",
"MACD_Histogram"
]
This set includes:
- Price movement
- Intraday range
- Short- and medium-term trend
- Volatility
- Volume
- Momentum
You can then compare it against the larger feature set.
Testing the Preprocessing Pipeline
Before training a model, run the following checks:
assert data["Date"].is_monotonic_increasing
assert not data[
feature_columns
].isnull().any().any()
assert not np.isinf(
data[feature_columns]
.to_numpy()
).any()
assert len(X_train) == len(y_train)
assert len(X_test) == len(y_test)
assert train_data["Date"].max() < (
test_data["Date"].min()
)
print(
"All preprocessing checks passed."
)
Also inspect the target:
print(
"Training target distribution:"
)
print(
y_train.value_counts(
normalize=True
)
)
print(
"Testing target distribution:"
)
print(
y_test.value_counts(
normalize=True
)
)
If these distributions are very different, market behaviour may have changed between the two periods.
That is useful information and should not automatically be “fixed” by shuffling the records.
Final Preprocessing Checklist
Before continuing to model training, confirm that:
- Dates use the correct data type
- Records are sorted from oldest to newest
- Duplicate trading dates have been handled
- Invalid price records have been investigated
- Missing features have been removed or handled safely
- Indicators use only present and previous data
- The prediction target is shifted correctly
- Target columns are excluded from the input features
- Training and test records are split chronologically
- The scaler is fitted only on training data
- LSTM sequences preserve chronological order
- Processed data and the fitted scaler have been saved
- No
NaNor infinite feature values remain
Final Result
We have transformed raw historical stock prices into a structured dataset suitable for machine learning.
The completed pipeline:
- Downloads or loads historical market data.
- Converts columns into appropriate data types.
- Sorts the records chronologically.
- Handles duplicates and missing values.
- Creates return, trend, volume, momentum and volatility features.
- Creates future-price and direction targets.
- removes rows without complete information.
- Splits the data without random shuffling.
- Fits a scaler using training records only.
- Creates sequential inputs for an LSTM model.
- Saves the prepared files for model training.
The next stage is to train a model using the processed features and compare its predictions against simple baselines.
A complicated neural network should not automatically be assumed to perform better. The model must be evaluated on unseen later data and compared against basic approaches such as:
- Predicting the same price as today
- Predicting the majority direction
- Logistic regression
- Linear regression
- Random forest
Stock-market prediction remains uncertain. Preprocessing cannot guarantee accurate forecasts, but it prevents avoidable errors and gives the model a cleaner, more realistic dataset.
Frequently Asked Questions
Why must stock data be split by date?
A prediction model is supposed to learn from the past and make predictions about the future. A chronological split recreates this process more realistically than a random split.
Should I predict the closing price or price direction?
Closing-price prediction is a regression problem. Direction prediction is a classification problem. Neither is automatically easier or more profitable. The best target depends on how the prediction will be used.
Why are the first rows missing after creating moving averages?
A 20-day moving average requires 20 historical records. The earliest rows do not have enough previous data, so their moving-average values are unavailable.
Should missing weekends be filled?
Normally, no. Weekends and market holidays are not trading days. Daily stock models usually work with actual trading records rather than artificial weekend prices.
Can I use adjusted closing prices?
Yes. Adjusted prices are useful for long-term analysis because they account for events such as stock splits and dividends. Use adjusted and unadjusted values consistently.
Which scaler is best for stock-market data?
MinMaxScaler is commonly used with neural networks and LSTMs. StandardScaler and robust scaling methods are also possible. The scaler must always be fitted using training data only.
Why can scaled test values exceed 1?
The test-period market may move beyond the minimum or maximum observed during training. MinMaxScaler uses the training range, so later values can legitimately fall below 0 or above 1.
How many previous days should an LSTM use?
Common starting points include 20, 30 or 60 trading days. The best sequence length should be determined using chronological validation rather than selecting the one with the best test result.
Can technical indicators guarantee profitable predictions?
No. Technical indicators are mathematical transformations of historical price and volume data. They may help a model identify patterns, but they do not guarantee future performance.
Is stock prediction accuracy the same as trading profitability?
No. A model must also overcome transaction costs, slippage, spreads and incorrect prediction sizes. A profitable strategy requires more than a high classification-accuracy score.
What comes after data preprocessing?
The next step is to train and evaluate several models using the prepared dataset. Start with a simple baseline before testing more complicated models such as neural networks or LSTMs.
Is this financial advice?
No. This tutorial demonstrates data preprocessing and machine-learning concepts. It does not recommend buying, selling or holding any particular investment.