SEO Title: Data Preprocessing for Stock Market Prediction Using AI – Part 2

Meta Description: Learn how to clean, transform and prepare historical stock-market data for an AI prediction model using Python, Pandas, NumPy and Scikit-learn.

Stock-market data cannot normally be fed directly into an artificial intelligence model.

Historical price files may contain missing trading days, duplicate records, incorrect data types and values with very different numerical ranges. Raw columns such as Open, High, Low, Close and Volume also provide limited information unless we transform them into features that help a model recognise market patterns.

In this second part of the stock-market prediction series, we will prepare historical stock data for machine learning.

We will cover:

Loading historical stock prices
Inspecting and cleaning the dataset
Handling missing and duplicate values
Creating daily returns
Adding moving averages and technical features
Creating a prediction target
Preventing future-data leakage
Splitting time-series data correctly
Scaling the features
Creating sequences for an LSTM model
Saving the prepared dataset

The completed preprocessing pipeline can later be used with models such as:

Linear regression
Random forest
XGBoost
Artificial neural networks
Long short-term memory networks, or LSTMs

Quick Answer

A basic stock-market preprocessing pipeline follows these steps:

Historical stock data
        ↓
Clean and sort the records
        ↓
Create technical features
        ↓
Create the prediction target
        ↓
Remove unavailable values
        ↓
Split data by date
        ↓
Fit the scaler on training data
        ↓
Transform the data into model-ready arrays

The most important rule is that the model must not receive information from the future.

For example, if we want to predict tomorrow’s closing price, today’s model input may contain today’s:

Opening price
Highest price
Lowest price
Closing price
Trading volume
Moving averages
Volatility
Momentum indicators

It must not contain tomorrow’s price or any indicator calculated using future records.

Why Data Preprocessing Matters

An AI model learns from the information it receives. If the input data is badly prepared, even an advanced model can produce unreliable results.

Consider the following raw data:

Date	Open	High	Low	Close	Volume
2024-05-20	189.33	191.92	189.01	191.04	44,320,000
2024-05-21	191.09	192.73	190.92	192.35	42,350,000
2024-05-22	192.27	192.82	190.27	190.90	34,610,000

A prediction model could use these values directly, but it would have difficulty identifying useful relationships such as:

Whether the price increased from the previous day
Whether the stock is above its recent average
Whether recent prices are unusually volatile
Whether trading activity is increasing
Whether short-term momentum is positive or negative

Preprocessing converts the raw records into structured features that make these relationships easier for the model to learn.

What We Are Going to Predict

Before preparing the data, we must define the prediction objective.

A model could attempt to predict:

Tomorrow’s closing price
Tomorrow’s percentage return
Whether the price will rise or fall
The highest price during the next trading day
Price movement over the next five trading days
Whether the stock will outperform a market index

For this tutorial, we will prepare two possible targets:

Next_Close: tomorrow’s closing price
Target_Up: whether tomorrow’s closing price is higher than today’s

Next_Close can be used for a regression model.

Target_Up can be used for a classification model.

Step 1: Install the Required Python Packages

Open a terminal and install the required packages:

pip install pandas numpy matplotlib scikit-learn yfinance joblib

The packages serve the following purposes:

Package	Purpose
Pandas	Loading and manipulating tabular data
NumPy	Numerical calculations
Matplotlib	Plotting prices and processed features
Scikit-learn	Scaling data and preparing machine-learning input
yfinance	Downloading historical market data
Joblib	Saving the fitted scaler

If you already have a CSV file containing historical prices, yfinance is optional.

Step 2: Import the Libraries

Create a Python file named preprocess_stock_data.py.

Add the following imports:

import joblib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.preprocessing import MinMaxScaler

If you want to download data directly:

import yfinance as yf

Step 3: Download Historical Stock Data

For this example, we will use Apple stock under the ticker symbol AAPL.

ticker = "AAPL"

data = yf.download(
    ticker,
    start="2018-01-01",
    end="2024-05-26",
    auto_adjust=False,
    progress=False
)

print(data.head())

Depending on the library version and the selected ticker, the returned dataset may contain columns such as:

Date
Open
High
Low
Close
Adj Close
Volume

The date may be stored as the index rather than a normal column.

Convert it into a column:

data = data.reset_index()

Then inspect the first few records:

print(data.head())

Example output:

        Date       Open       High        Low      Close  Adj Close    Volume
0 2018-01-02  42.540001  43.075001  42.314999  43.064999  40.568928  102223600
1 2018-01-03  43.132500  43.637501  42.990002  43.057499  40.561871  118071600
2 2018-01-04  43.134998  43.367500  43.020000  43.257500  40.750252   89738400

The exact values will depend on the selected date range and whether adjusted prices are used.

Loading Stock Data from a CSV File

If the data has already been downloaded, use:

data = pd.read_csv("AAPL_historical_data.csv")

Inspect it:

print(data.head())
print(data.columns)
print(data.shape)

If the date is stored in a column named Date, convert it into a proper date type:

data["Date"] = pd.to_datetime(
    data["Date"],
    errors="coerce"
)

Using errors="coerce" changes invalid dates into missing values. These can then be detected and removed during cleaning.

Step 4: Inspect the Dataset

Before changing the data, examine its structure.

print(data.info())
print(data.describe())

Check the number of missing values:

print(data.isnull().sum())

Check the number of duplicate rows:

print("Duplicate rows:", data.duplicated().sum())

Check the date range:

print("First date:", data["Date"].min())
print("Last date:", data["Date"].max())

This helps identify problems such as:

Dates stored as text
Prices stored as strings
Missing closing prices
Duplicate trading days
Zero or negative prices
Abnormally low or high trading volumes

Step 5: Select the Required Columns

For this tutorial, we need:

required_columns = [
    "Date",
    "Open",
    "High",
    "Low",
    "Close",
    "Volume"
]

data = data[required_columns].copy()

If you prefer adjusted closing prices, you can use Adj Close instead of Close.

Adjusted prices account for events such as stock splits and dividends. They may be more suitable when analysing long-term returns.

However, you should be consistent. Do not create some features from adjusted prices and others from unadjusted prices without understanding the difference.

Step 6: Convert Columns to the Correct Data Types

Make sure the date is recognised correctly:

data["Date"] = pd.to_datetime(
    data["Date"],
    errors="coerce"
)

Convert the numerical columns:

numeric_columns = [
    "Open",
    "High",
    "Low",
    "Close",
    "Volume"
]

for column in numeric_columns:
    data[column] = pd.to_numeric(
        data[column],
        errors="coerce"
    )

Values that cannot be converted become NaN, which represents a missing value.

Check the result:

print(data.dtypes)

Expected output:

Date      datetime64[ns]
Open             float64
High             float64
Low              float64
Close            float64
Volume             int64

The exact numerical types may differ slightly.

Step 7: Sort the Records by Date

Time-series records must be arranged chronologically.

data = data.sort_values("Date")
data = data.reset_index(drop=True)

Do not assume that a CSV file is already sorted.

An incorrectly ordered dataset can produce invalid moving averages, returns and training sequences.

Step 8: Remove Duplicate Records

A dataset may contain the same trading day more than once.

Check duplicates based on the date:

duplicate_dates = data.duplicated(
    subset=["Date"],
    keep=False
)

print(data[duplicate_dates])

Remove duplicate dates:

data = data.drop_duplicates(
    subset=["Date"],
    keep="last"
)

Using keep="last" retains the last record for each duplicated date.

If the duplicate records contain conflicting prices, investigate where the data came from before deciding which record to keep.

Step 9: Handle Missing Values

Check missing values again:

print(data.isnull().sum())

Rows without a valid date or closing price should normally be removed:

data = data.dropna(
    subset=["Date", "Close"]
)

For other price columns, one option is to use forward filling:

price_columns = [
    "Open",
    "High",
    "Low",
    "Close"
]

data[price_columns] = (
    data[price_columns]
    .ffill()
)

Forward filling replaces a missing value with the most recent available value.

However, do not automatically fill every missing record without checking why it is missing.

A safer approach for a small number of incomplete rows is:

data = data.dropna(
    subset=[
        "Open",
        "High",
        "Low",
        "Close",
        "Volume"
    ]
)

This tutorial will use complete rows:

data = data.dropna().copy()

Do Weekends Count as Missing Data?

Stock markets normally close on weekends and public holidays.

Therefore, the absence of Saturday and Sunday records does not automatically mean the dataset is incomplete.

Do not create artificial weekend prices unless your model has a specific reason to require calendar-day data.

For daily stock prediction, it is usually better to work with actual trading days.

Step 10: Validate the Price Records

Basic validation can identify obviously invalid rows.

invalid_rows = data[
    (data["Open"] <= 0) |
    (data["High"] <= 0) |
    (data["Low"] <= 0) |
    (data["Close"] <= 0) |
    (data["Volume"] < 0)
]

print(invalid_rows)

We can also check whether the daily high and low are logically consistent:

invalid_price_ranges = data[
    (data["High"] < data["Low"]) |
    (data["High"] < data["Open"]) |
    (data["High"] < data["Close"]) |
    (data["Low"] > data["Open"]) |
    (data["Low"] > data["Close"])
]

print(invalid_price_ranges)

If these records exist, confirm them against the original source before deleting or changing them.

To remove clearly invalid rows:

data = data[
    (data["Open"] > 0) &
    (data["High"] > 0) &
    (data["Low"] > 0) &
    (data["Close"] > 0) &
    (data["Volume"] >= 0) &
    (data["High"] >= data["Low"])
].copy()

Step 11: Create Daily Return Features

The absolute closing price tells us the stock’s value, but it does not directly show how much it moved.

Create a daily percentage return:

data["Daily_Return"] = (
    data["Close"]
    .pct_change()
)

The calculation is:

(Current closing price − Previous closing price)
-------------------------------------------------
              Previous closing price

For example, if the closing price increases from $100 to $103:

Daily return = (103 − 100) ÷ 100
             = 0.03
             = 3%

Create a logarithmic return:

data["Log_Return"] = np.log(
    data["Close"] /
    data["Close"].shift(1)
)

Logarithmic returns are commonly used in financial analysis because returns across several periods can be added together more conveniently.

Step 12: Create Intraday Price Features

We can create features describing what happened within each trading day.

Opening-to-Closing Change

data["Open_Close_Change"] = (
    data["Close"] - data["Open"]
) / data["Open"]

A positive value means the stock closed above its opening price.

Daily Trading Range

data["High_Low_Range"] = (
    data["High"] - data["Low"]
) / data["Close"]

This represents the size of the day’s price range relative to the closing price.

Previous Closing Gap

data["Previous_Close"] = data["Close"].shift(1)

data["Opening_Gap"] = (
    data["Open"] - data["Previous_Close"]
) / data["Previous_Close"]

A positive opening gap means the stock opened above the previous trading day’s closing price.

Step 13: Add Moving Averages

Moving averages smooth short-term price changes.

Create 5-day, 10-day, 20-day and 50-day simple moving averages:

data["SMA_5"] = (
    data["Close"]
    .rolling(window=5)
    .mean()
)

data["SMA_10"] = (
    data["Close"]
    .rolling(window=10)
    .mean()
)

data["SMA_20"] = (
    data["Close"]
    .rolling(window=20)
    .mean()
)

data["SMA_50"] = (
    data["Close"]
    .rolling(window=50)
    .mean()
)

The 5-day moving average represents approximately one trading week.

The 20-day moving average represents approximately one trading month.

The 50-day moving average provides a broader view of the price trend.

Create exponential moving averages:

data["EMA_12"] = (
    data["Close"]
    .ewm(span=12, adjust=False)
    .mean()
)

data["EMA_26"] = (
    data["Close"]
    .ewm(span=26, adjust=False)
    .mean()
)

Exponential moving averages give more weight to recent prices.

Step 14: Create Price-to-Moving-Average Ratios

A raw moving average may mostly repeat information already contained in the closing price.

Ratios can make the relationship clearer:

data["Close_to_SMA_5"] = (
    data["Close"] / data["SMA_5"]
)

data["Close_to_SMA_20"] = (
    data["Close"] / data["SMA_20"]
)

data["SMA_5_to_SMA_20"] = (
    data["SMA_5"] / data["SMA_20"]
)

Interpretation:

A value above 1 means the first value is higher
A value below 1 means the first value is lower
A value near 1 means the values are similar

For example:

Close-to-SMA ratio = 1.04

This means the closing price is approximately 4% above its moving average.

Step 15: Add Rolling Volatility

Volatility describes how widely returns have fluctuated.

Create 5-day and 20-day rolling volatility:

data["Volatility_5"] = (
    data["Daily_Return"]
    .rolling(window=5)
    .std()
)

data["Volatility_20"] = (
    data["Daily_Return"]
    .rolling(window=20)
    .std()
)

A larger value indicates that recent daily returns have varied more widely.

The calculation only uses the current and previous records. It does not use future returns.

Step 16: Create Volume Features

Trading volume can provide useful information about market participation.

Create a 20-day average volume:

data["Volume_SMA_20"] = (
    data["Volume"]
    .rolling(window=20)
    .mean()
)

Then compare current volume with the average:

data["Relative_Volume"] = (
    data["Volume"] /
    data["Volume_SMA_20"]
)

Example:

Relative volume = 1.50

This means the current volume is approximately 50% higher than its 20-day average.

Create the daily volume change:

data["Volume_Change"] = (
    data["Volume"]
    .pct_change()
)

Very large volume changes can create extreme values. These may need to be capped or transformed after inspecting their distribution.

Step 17: Add Momentum

Momentum measures how much the price has changed over a selected period.

data["Momentum_5"] = (
    data["Close"] /
    data["Close"].shift(5)
) - 1

data["Momentum_10"] = (
    data["Close"] /
    data["Close"].shift(10)
) - 1

A Momentum_5 value of 0.06 means the price has increased by approximately 6% over five trading days.

A value of -0.04 means it has decreased by approximately 4%.

Step 18: Calculate the Relative Strength Index

The Relative Strength Index, or RSI, compares recent gains with recent losses.

def calculate_rsi(series, period=14):
    price_change = series.diff()

    gains = price_change.clip(lower=0)
    losses = -price_change.clip(upper=0)

    average_gain = gains.ewm(
        alpha=1 / period,
        adjust=False,
        min_periods=period
    ).mean()

    average_loss = losses.ewm(
        alpha=1 / period,
        adjust=False,
        min_periods=period
    ).mean()

    relative_strength = (
        average_gain / average_loss
    )

    rsi = 100 - (
        100 / (1 + relative_strength)
    )

    return rsi

Apply it:

data["RSI_14"] = calculate_rsi(
    data["Close"],
    period=14
)

RSI is normally displayed on a scale from 0 to 100.

It should be treated as a numerical feature, not a guaranteed buy or sell signal.

Step 19: Add MACD

The Moving Average Convergence Divergence indicator uses the difference between two exponential moving averages.

data["MACD"] = (
    data["EMA_12"] -
    data["EMA_26"]
)

data["MACD_Signal"] = (
    data["MACD"]
    .ewm(span=9, adjust=False)
    .mean()
)

data["MACD_Histogram"] = (
    data["MACD"] -
    data["MACD_Signal"]
)

The model can use these values as possible momentum and trend features.

Step 20: Add Bollinger Band Features

Bollinger Bands use a moving average and rolling standard deviation.

data["BB_Middle"] = (
    data["Close"]
    .rolling(window=20)
    .mean()
)

data["BB_Std"] = (
    data["Close"]
    .rolling(window=20)
    .std()
)

data["BB_Upper"] = (
    data["BB_Middle"] +
    2 * data["BB_Std"]
)

data["BB_Lower"] = (
    data["BB_Middle"] -
    2 * data["BB_Std"]
)

Create the price’s relative position within the bands:

band_width = (
    data["BB_Upper"] -
    data["BB_Lower"]
)

data["BB_Position"] = (
    data["Close"] -
    data["BB_Lower"]
) / band_width

Create a band-width feature:

data["BB_Width"] = (
    band_width /
    data["BB_Middle"]
)

Avoid dividing by zero:

data.replace(
    [np.inf, -np.inf],
    np.nan,
    inplace=True
)

Step 21: Add Calendar Features

Markets may behave differently across days, months or reporting periods.

Create basic calendar features:

data["Day_Of_Week"] = (
    data["Date"].dt.dayofweek
)

data["Month"] = (
    data["Date"].dt.month
)

data["Quarter"] = (
    data["Date"].dt.quarter
)

Day_Of_Week uses:

Value	Day
0	Monday
1	Tuesday
2	Wednesday
3	Thursday
4	Friday

These calendar features do not guarantee a useful pattern. Their value should be evaluated during model testing.

Step 22: Create the Prediction Target

Regression Target: Tomorrow’s Closing Price

data["Next_Close"] = (
    data["Close"]
    .shift(-1)
)

For each row, Next_Close contains the closing price from the following trading record.

Example:

Date	Close	Next_Close
Monday	100.00	102.00
Tuesday	102.00	101.50
Wednesday	101.50	104.00

The last record will not have a future closing price, so its target will be missing.

Classification Target: Will the Price Increase?

data["Target_Up"] = (
    data["Next_Close"] >
    data["Close"]
).astype(int)

The result is:

1 if tomorrow’s closing price is higher
0 if tomorrow’s closing price is equal or lower

We should calculate Target_Up before removing the missing Next_Close row.

Predicting Tomorrow’s Return Instead

Another possible regression target is the next-day return:

data["Target_Return"] = (
    data["Next_Close"] /
    data["Close"]
) - 1

This may be easier to compare across stocks with different prices.

For example, a $5 price change means something very different for a $20 stock and a $500 stock. A percentage return provides a more consistent scale.

Step 23: Remove Rows Made Incomplete by Feature Creation

Rolling indicators create missing values at the beginning of the dataset.

For example, a 50-day moving average requires 50 trading records before it becomes available.

The shifted prediction target also creates a missing value at the end.

Replace infinite values and remove incomplete rows:

data = data.replace(
    [np.inf, -np.inf],
    np.nan
)

data = data.dropna().copy()
data = data.reset_index(drop=True)

Check the prepared dataset:

print(data.head())
print(data.tail())
print(data.shape)
print(data.isnull().sum())

Do not fill the missing target with zero. A zero future price or return would create false training examples.

Step 24: Select the Model Features

Choose the columns the model will receive.

feature_columns = [
    "Open",
    "High",
    "Low",
    "Close",
    "Volume",
    "Daily_Return",
    "Log_Return",
    "Open_Close_Change",
    "High_Low_Range",
    "Opening_Gap",
    "SMA_5",
    "SMA_10",
    "SMA_20",
    "SMA_50",
    "EMA_12",
    "EMA_26",
    "Close_to_SMA_5",
    "Close_to_SMA_20",
    "SMA_5_to_SMA_20",
    "Volatility_5",
    "Volatility_20",
    "Relative_Volume",
    "Volume_Change",
    "Momentum_5",
    "Momentum_10",
    "RSI_14",
    "MACD",
    "MACD_Signal",
    "MACD_Histogram",
    "BB_Position",
    "BB_Width",
    "Day_Of_Week",
    "Month",
    "Quarter"
]

Create the feature matrix and classification target:

X = data[feature_columns].copy()
y = data["Target_Up"].copy()

For closing-price regression, use:

y = data["Next_Close"].copy()

Step 25: Check the Target Distribution

For a direction-classification model, inspect how many examples belong to each class:

print(y.value_counts())
print(y.value_counts(normalize=True))

Example:

1    0.53
0    0.47

This means 53% of the records are upward days and 47% are non-upward days.

If one class is much larger than the other, accuracy alone can be misleading.

For example, if 70% of the samples are upward days, a model that always predicts “up” would achieve 70% accuracy without learning anything useful.

Later evaluation should include measures such as:

Precision
Recall
F1 score
Confusion matrix
Balanced accuracy
Strategy return after costs

Step 26: Split the Data Chronologically

Do not randomly shuffle stock-market data before splitting it.

A random split could place records from 2023 in the training set and records from 2020 in the test set. This does not represent a realistic prediction process.

Use the earlier records for training and the later records for testing.

split_index = int(
    len(data) * 0.80
)

train_data = data.iloc[
    :split_index
].copy()

test_data = data.iloc[
    split_index:
].copy()

Check the periods:

print(
    "Training period:",
    train_data["Date"].min(),
    "to",
    train_data["Date"].max()
)

print(
    "Testing period:",
    test_data["Date"].min(),
    "to",
    test_data["Date"].max()
)

Create the training and test variables:

X_train = train_data[
    feature_columns
].copy()

X_test = test_data[
    feature_columns
].copy()

y_train = train_data[
    "Target_Up"
].copy()

y_test = test_data[
    "Target_Up"
].copy()

Step 27: Scale the Features Correctly

Stock-market columns use very different ranges.

For example:

Closing price:       50 to 200
RSI:                  0 to 100
Daily return:       -0.1 to 0.1
Trading volume: 20,000,000 to 200,000,000

Large values can dominate the training process for models that are sensitive to feature scale.

Use MinMaxScaler:

scaler = MinMaxScaler(
    feature_range=(0, 1)
)

Fit it using only the training data:

X_train_scaled = scaler.fit_transform(
    X_train
)

Transform the test data using the same fitted scaler:

X_test_scaled = scaler.transform(
    X_test
)

This is important:

scaler.fit(X_train)

is correct.

This is incorrect:

scaler.fit(pd.concat([X_train, X_test]))

Fitting the scaler on the complete dataset allows information from the test period to affect the training transformation. This is a form of data leakage.

Why Scaling the Entire Dataset Is a Mistake

Suppose the highest stock price in the training period is $150, but the price reaches $220 during the test period.

If the scaler is fitted using the complete dataset, it learns about the future $220 value.

Although this may appear harmless, information from the test period has influenced the training preparation.

In a proper experiment:

Fit all preprocessing operations on the training data.
Apply the fitted operations to validation and test data.
Never recalculate them using future observations.

Step 28: Convert the Scaled Data Back to DataFrames

NumPy arrays do not retain column names.

For easier inspection:

X_train_scaled = pd.DataFrame(
    X_train_scaled,
    columns=feature_columns,
    index=X_train.index
)

X_test_scaled = pd.DataFrame(
    X_test_scaled,
    columns=feature_columns,
    index=X_test.index
)

Check the results:

print(X_train_scaled.head())
print(X_train_scaled.min())
print(X_train_scaled.max())

Most training values should now fall between 0 and 1.

Some test values may fall below 0 or above 1 if the later market data moves outside the training range. This is possible and does not automatically mean scaling failed.

Step 29: Prepare LSTM Sequences

A standard machine-learning model normally receives one row at a time.

An LSTM model usually receives a sequence of consecutive records.

For example, if the sequence length is 30:

Previous 30 trading days → Predict the following day

Create a sequence function:

def create_sequences(
    features,
    targets,
    sequence_length=30
):
    X_sequences = []
    y_sequences = []

    feature_values = np.asarray(features)
    target_values = np.asarray(targets)

    for index in range(
        sequence_length,
        len(feature_values)
    ):
        start_index = (
            index - sequence_length
        )

        X_sequences.append(
            feature_values[
                start_index:index
            ]
        )

        y_sequences.append(
            target_values[index]
        )

    return (
        np.array(X_sequences),
        np.array(y_sequences)
    )

Create the training sequences:

sequence_length = 30

X_train_sequence, y_train_sequence = (
    create_sequences(
        X_train_scaled,
        y_train,
        sequence_length
    )
)

Check their shapes:

print(
    "X training shape:",
    X_train_sequence.shape
)

print(
    "y training shape:",
    y_train_sequence.shape
)

Example:

X training shape: (1200, 30, 34)
y training shape: (1200,)

This means:

1,200 training examples
30 trading days per example
34 features per trading day

Creating Test Sequences Without Losing Context

The first test prediction may need the final 30 records from the training period.

If sequences are created only from X_test_scaled, the beginning of the test period will be discarded.

Combine the final training context with the test features:

test_context = pd.concat([
    X_train_scaled.tail(
        sequence_length
    ),
    X_test_scaled
])

Prepare aligned targets:

test_targets_with_context = pd.concat([
    y_train.tail(sequence_length),
    y_test
])

Create sequences:

X_test_sequence, y_test_sequence = (
    create_sequences(
        test_context,
        test_targets_with_context,
        sequence_length
    )
)

Check the result:

print(
    "X test shape:",
    X_test_sequence.shape
)

print(
    "y test shape:",
    y_test_sequence.shape
)

The number of test sequences should now match the number of test targets.

Step 30: Save the Prepared Data

Save the full feature dataset:

data.to_csv(
    "AAPL_processed_data.csv",
    index=False
)

Save the chronological training and test datasets:

train_data.to_csv(
    "AAPL_train_data.csv",
    index=False
)

test_data.to_csv(
    "AAPL_test_data.csv",
    index=False
)

Save the fitted scaler:

joblib.dump(
    scaler,
    "AAPL_feature_scaler.joblib"
)

Save the LSTM arrays:

np.save(
    "X_train_sequence.npy",
    X_train_sequence
)

np.save(
    "y_train_sequence.npy",
    y_train_sequence
)

np.save(
    "X_test_sequence.npy",
    X_test_sequence
)

np.save(
    "y_test_sequence.npy",
    y_test_sequence
)

The same scaler must be loaded when preparing new data for future predictions:

scaler = joblib.load(
    "AAPL_feature_scaler.joblib"
)

Do not fit a new scaler every time one prediction is requested.

Complete Preprocessing Example

The following code combines the main steps into one script:

import joblib
import numpy as np
import pandas as pd
import yfinance as yf

from sklearn.preprocessing import MinMaxScaler


def calculate_rsi(series, period=14):
    price_change = series.diff()

    gains = price_change.clip(lower=0)
    losses = -price_change.clip(upper=0)

    average_gain = gains.ewm(
        alpha=1 / period,
        adjust=False,
        min_periods=period
    ).mean()

    average_loss = losses.ewm(
        alpha=1 / period,
        adjust=False,
        min_periods=period
    ).mean()

    relative_strength = (
        average_gain / average_loss
    )

    return 100 - (
        100 / (1 + relative_strength)
    )


ticker = "AAPL"

data = yf.download(
    ticker,
    start="2018-01-01",
    end="2024-05-26",
    auto_adjust=False,
    progress=False
)

data = data.reset_index()

if isinstance(
    data.columns,
    pd.MultiIndex
):
    data.columns = (
        data.columns
        .get_level_values(0)
    )

required_columns = [
    "Date",
    "Open",
    "High",
    "Low",
    "Close",
    "Volume"
]

data = data[
    required_columns
].copy()

data["Date"] = pd.to_datetime(
    data["Date"],
    errors="coerce"
)

numeric_columns = [
    "Open",
    "High",
    "Low",
    "Close",
    "Volume"
]

for column in numeric_columns:
    data[column] = pd.to_numeric(
        data[column],
        errors="coerce"
    )

data = data.drop_duplicates(
    subset=["Date"],
    keep="last"
)

data = data.sort_values("Date")
data = data.dropna()
data = data.reset_index(drop=True)

data = data[
    (data["Open"] > 0) &
    (data["High"] > 0) &
    (data["Low"] > 0) &
    (data["Close"] > 0) &
    (data["Volume"] >= 0) &
    (data["High"] >= data["Low"])
].copy()

data["Daily_Return"] = (
    data["Close"].pct_change()
)

data["Log_Return"] = np.log(
    data["Close"] /
    data["Close"].shift(1)
)

data["Open_Close_Change"] = (
    data["Close"] -
    data["Open"]
) / data["Open"]

data["High_Low_Range"] = (
    data["High"] -
    data["Low"]
) / data["Close"]

data["Previous_Close"] = (
    data["Close"].shift(1)
)

data["Opening_Gap"] = (
    data["Open"] -
    data["Previous_Close"]
) / data["Previous_Close"]

for period in [5, 10, 20, 50]:
    data[f"SMA_{period}"] = (
        data["Close"]
        .rolling(period)
        .mean()
    )

data["EMA_12"] = (
    data["Close"]
    .ewm(span=12, adjust=False)
    .mean()
)

data["EMA_26"] = (
    data["Close"]
    .ewm(span=26, adjust=False)
    .mean()
)

data["Close_to_SMA_5"] = (
    data["Close"] /
    data["SMA_5"]
)

data["Close_to_SMA_20"] = (
    data["Close"] /
    data["SMA_20"]
)

data["SMA_5_to_SMA_20"] = (
    data["SMA_5"] /
    data["SMA_20"]
)

data["Volatility_5"] = (
    data["Daily_Return"]
    .rolling(5)
    .std()
)

data["Volatility_20"] = (
    data["Daily_Return"]
    .rolling(20)
    .std()
)

data["Volume_SMA_20"] = (
    data["Volume"]
    .rolling(20)
    .mean()
)

data["Relative_Volume"] = (
    data["Volume"] /
    data["Volume_SMA_20"]
)

data["Volume_Change"] = (
    data["Volume"]
    .pct_change()
)

data["Momentum_5"] = (
    data["Close"] /
    data["Close"].shift(5)
) - 1

data["Momentum_10"] = (
    data["Close"] /
    data["Close"].shift(10)
) - 1

data["RSI_14"] = calculate_rsi(
    data["Close"],
    period=14
)

data["MACD"] = (
    data["EMA_12"] -
    data["EMA_26"]
)

data["MACD_Signal"] = (
    data["MACD"]
    .ewm(span=9, adjust=False)
    .mean()
)

data["MACD_Histogram"] = (
    data["MACD"] -
    data["MACD_Signal"]
)

data["BB_Middle"] = (
    data["Close"]
    .rolling(20)
    .mean()
)

data["BB_Std"] = (
    data["Close"]
    .rolling(20)
    .std()
)

data["BB_Upper"] = (
    data["BB_Middle"] +
    2 * data["BB_Std"]
)

data["BB_Lower"] = (
    data["BB_Middle"] -
    2 * data["BB_Std"]
)

band_width = (
    data["BB_Upper"] -
    data["BB_Lower"]
)

data["BB_Position"] = (
    data["Close"] -
    data["BB_Lower"]
) / band_width

data["BB_Width"] = (
    band_width /
    data["BB_Middle"]
)

data["Day_Of_Week"] = (
    data["Date"].dt.dayofweek
)

data["Month"] = (
    data["Date"].dt.month
)

data["Quarter"] = (
    data["Date"].dt.quarter
)

data["Next_Close"] = (
    data["Close"].shift(-1)
)

data["Target_Return"] = (
    data["Next_Close"] /
    data["Close"]
) - 1

data["Target_Up"] = (
    data["Next_Close"] >
    data["Close"]
).astype(int)

data = data.replace(
    [np.inf, -np.inf],
    np.nan
)

data = data.dropna()
data = data.reset_index(drop=True)

feature_columns = [
    "Open",
    "High",
    "Low",
    "Close",
    "Volume",
    "Daily_Return",
    "Log_Return",
    "Open_Close_Change",
    "High_Low_Range",
    "Opening_Gap",
    "SMA_5",
    "SMA_10",
    "SMA_20",
    "SMA_50",
    "EMA_12",
    "EMA_26",
    "Close_to_SMA_5",
    "Close_to_SMA_20",
    "SMA_5_to_SMA_20",
    "Volatility_5",
    "Volatility_20",
    "Relative_Volume",
    "Volume_Change",
    "Momentum_5",
    "Momentum_10",
    "RSI_14",
    "MACD",
    "MACD_Signal",
    "MACD_Histogram",
    "BB_Position",
    "BB_Width",
    "Day_Of_Week",
    "Month",
    "Quarter"
]

split_index = int(
    len(data) * 0.80
)

train_data = data.iloc[
    :split_index
].copy()

test_data = data.iloc[
    split_index:
].copy()

X_train = train_data[
    feature_columns
]

X_test = test_data[
    feature_columns
]

y_train = train_data[
    "Target_Up"
]

y_test = test_data[
    "Target_Up"
]

scaler = MinMaxScaler()

X_train_scaled = (
    scaler.fit_transform(
        X_train
    )
)

X_test_scaled = (
    scaler.transform(
        X_test
    )
)

data.to_csv(
    f"{ticker}_processed_data.csv",
    index=False
)

train_data.to_csv(
    f"{ticker}_train_data.csv",
    index=False
)

test_data.to_csv(
    f"{ticker}_test_data.csv",
    index=False
)

joblib.dump(
    scaler,
    f"{ticker}_feature_scaler.joblib"
)

np.save(
    f"{ticker}_X_train.npy",
    X_train_scaled
)

np.save(
    f"{ticker}_X_test.npy",
    X_test_scaled
)

np.save(
    f"{ticker}_y_train.npy",
    y_train.to_numpy()
)

np.save(
    f"{ticker}_y_test.npy",
    y_test.to_numpy()
)

print("Preprocessing completed.")
print("Total usable rows:", len(data))
print("Training rows:", len(train_data))
print("Testing rows:", len(test_data))
print("Number of features:", len(feature_columns))

Visualise the Prepared Data

Before training a model, plot some of the processed features.

Closing Price and Moving Averages

plt.figure(figsize=(14, 7))

plt.plot(
    data["Date"],
    data["Close"],
    label="Closing Price",
    linewidth=1
)

plt.plot(
    data["Date"],
    data["SMA_20"],
    label="20-Day SMA"
)

plt.plot(
    data["Date"],
    data["SMA_50"],
    label="50-Day SMA"
)

plt.title(
    f"{ticker} Closing Price and Moving Averages"
)

plt.xlabel("Date")
plt.ylabel("Price")
plt.legend()
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()

Daily Returns

plt.figure(figsize=(14, 5))

plt.plot(
    data["Date"],
    data["Daily_Return"],
    linewidth=0.8
)

plt.axhline(
    0,
    color="black",
    linewidth=0.8
)

plt.title(
    f"{ticker} Daily Returns"
)

plt.xlabel("Date")
plt.ylabel("Return")
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()

Visual inspection can reveal:

Missing periods
Unusual price jumps
Stock splits
Extremely large volume changes
Features that appear incorrectly calculated

Preventing Data Leakage

Data leakage occurs when the training process receives information that would not have been available at prediction time.

It can make a model appear highly accurate during testing while failing on new market data.

Leakage Example 1: Using Tomorrow’s Price as a Feature

This is incorrect:

feature_columns = [
    "Close",
    "Next_Close"
]

Next_Close is the value we are trying to predict. It must only be used as the target.

Leakage Example 2: Centred Moving Average

This can use future observations:

data["Wrong_SMA"] = (
    data["Close"]
    .rolling(
        window=5,
        center=True
    )
    .mean()
)

Because center=True places the current date in the middle of the window, later prices may be included.

Use:

data["SMA_5"] = (
    data["Close"]
    .rolling(window=5)
    .mean()
)

Leakage Example 3: Fitting the Scaler on All Data

Incorrect:

scaler.fit(data[feature_columns])

Correct:

scaler.fit(train_data[feature_columns])

Leakage Example 4: Randomly Shuffling Time-Series Records

This is normally inappropriate:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = (
    train_test_split(
        X,
        y,
        test_size=0.20,
        shuffle=True
    )
)

A chronological split better represents a real prediction scenario.

Leakage Example 5: Backfilling Missing Features

This can copy a future value backwards:

data = data.bfill()

For market prediction, backfilling can expose future information to earlier rows.

Dropping incomplete rows or using carefully justified past-only methods is safer.

Should Outliers Be Removed?

Large price movements are not always errors.

A sudden 10% move may be caused by:

An earnings announcement
A takeover proposal
A regulatory decision
A product announcement
A broader market crash
A stock split or data-adjustment problem

Do not remove a value simply because it is unusual.

First determine whether it represents:

A genuine market event
A stock split
A currency or unit error
A duplicated record
A corrupted data point

Real market shocks may be important training information.

For extreme volume-change features, you may use clipping after determining suitable limits from the training data:

lower_limit = (
    train_data["Volume_Change"]
    .quantile(0.01)
)

upper_limit = (
    train_data["Volume_Change"]
    .quantile(0.99)
)

train_data["Volume_Change"] = (
    train_data["Volume_Change"]
    .clip(
        lower=lower_limit,
        upper=upper_limit
    )
)

test_data["Volume_Change"] = (
    test_data["Volume_Change"]
    .clip(
        lower=lower_limit,
        upper=upper_limit
    )
)

The clipping limits must be calculated from the training data only.

Feature Correlation

Check how strongly the numerical features are related:

correlation_matrix = (
    data[feature_columns]
    .corr()
)

print(
    correlation_matrix["Close"]
    .sort_values(
        ascending=False
    )
)

Columns such as Close, SMA 20, SMA 50 and EMA 26 may be highly correlated because they are all derived from the same price series.

High correlation does not automatically mean that a feature must be removed. However, keeping many nearly identical features can:

Increase model complexity
Make some models unstable
Provide little additional information
Increase the chance of overfitting

Feature selection should be evaluated using validation data, not based only on the training score.

Better Evaluation with Walk-Forward Validation

A single 80/20 split provides one view of performance.

Walk-forward validation tests the model across several later periods.

Example:

Train: 2018–2020 → Validate: 2021
Train: 2018–2021 → Validate: 2022
Train: 2018–2022 → Validate: 2023
Train: 2018–2023 → Test: 2024

Scikit-learn provides TimeSeriesSplit:

from sklearn.model_selection import TimeSeriesSplit

time_series_split = TimeSeriesSplit(
    n_splits=5
)

X_all = data[feature_columns]
y_all = data["Target_Up"]

for fold, (
    train_index,
    validation_index
) in enumerate(
    time_series_split.split(X_all),
    start=1
):
    X_fold_train = X_all.iloc[
        train_index
    ]

    X_fold_validation = X_all.iloc[
        validation_index
    ]

    y_fold_train = y_all.iloc[
        train_index
    ]

    y_fold_validation = y_all.iloc[
        validation_index
    ]

    fold_scaler = MinMaxScaler()

    X_fold_train_scaled = (
        fold_scaler.fit_transform(
            X_fold_train
        )
    )

    X_fold_validation_scaled = (
        fold_scaler.transform(
            X_fold_validation
        )
    )

    print(
        f"Fold {fold}:",
        len(train_index),
        "training records and",
        len(validation_index),
        "validation records"
    )

Each fold uses only earlier records to predict a later period.

Common Data Preprocessing Mistakes

1. Using Too Little Historical Data

A neural network trained on only a few months of daily records may not have enough examples to learn a reliable pattern.

Daily market data provides roughly 250 trading records per year.

Five years of daily data may provide only around 1,250 observations before removing incomplete rows.

That is not a large dataset for a complex neural network.

2. Adding Too Many Indicators

Adding dozens of technical indicators does not guarantee better predictions.

Many indicators are calculated from the same price and volume data. This can create highly repetitive features.

Start with a smaller, explainable feature set and compare validation results.

3. Ignoring Trading Costs

A model may correctly predict small price movements but still lose money after:

Brokerage fees
Bid-ask spreads
Slippage
Taxes or exchange fees
Currency-conversion costs

Prediction accuracy and trading profitability are not the same thing.

4. Training on Only One Market Condition

A model trained only during a strong bull market may perform poorly during:

Bear markets
Sideways markets
High-interest-rate periods
Financial crises
Unusually volatile periods

The dataset should include different market conditions where possible.

5. Assuming a High Accuracy Means Future Profit

Financial markets change over time.

A pattern found in older data may disappear after:

Market participants adapt
Trading rules change
Company conditions change
Economic conditions shift
The stock enters a different growth stage

A model must be tested on later unseen data.

Recommended Basic Feature Set

If you want to begin with a smaller dataset, use:

basic_features = [
    "Daily_Return",
    "Open_Close_Change",
    "High_Low_Range",
    "Close_to_SMA_5",
    "Close_to_SMA_20",
    "Volatility_5",
    "Volatility_20",
    "Relative_Volume",
    "Momentum_5",
    "Momentum_10",
    "RSI_14",
    "MACD_Histogram"
]

This set includes:

Price movement
Intraday range
Short- and medium-term trend
Volatility
Volume
Momentum

You can then compare it against the larger feature set.

Testing the Preprocessing Pipeline

Before training a model, run the following checks:

assert data["Date"].is_monotonic_increasing

assert not data[
    feature_columns
].isnull().any().any()

assert not np.isinf(
    data[feature_columns]
    .to_numpy()
).any()

assert len(X_train) == len(y_train)
assert len(X_test) == len(y_test)

assert train_data["Date"].max() < (
    test_data["Date"].min()
)

print(
    "All preprocessing checks passed."
)

Also inspect the target:

print(
    "Training target distribution:"
)

print(
    y_train.value_counts(
        normalize=True
    )
)

print(
    "Testing target distribution:"
)

print(
    y_test.value_counts(
        normalize=True
    )
)

If these distributions are very different, market behaviour may have changed between the two periods.

That is useful information and should not automatically be “fixed” by shuffling the records.

Final Preprocessing Checklist

Before continuing to model training, confirm that:

Dates use the correct data type
Records are sorted from oldest to newest
Duplicate trading dates have been handled
Invalid price records have been investigated
Missing features have been removed or handled safely
Indicators use only present and previous data
The prediction target is shifted correctly
Target columns are excluded from the input features
Training and test records are split chronologically
The scaler is fitted only on training data
LSTM sequences preserve chronological order
Processed data and the fitted scaler have been saved
No NaN or infinite feature values remain

Final Result

We have transformed raw historical stock prices into a structured dataset suitable for machine learning.

The completed pipeline:

Downloads or loads historical market data.
Converts columns into appropriate data types.
Sorts the records chronologically.
Handles duplicates and missing values.
Creates return, trend, volume, momentum and volatility features.
Creates future-price and direction targets.
removes rows without complete information.
Splits the data without random shuffling.
Fits a scaler using training records only.
Creates sequential inputs for an LSTM model.
Saves the prepared files for model training.

The next stage is to train a model using the processed features and compare its predictions against simple baselines.

A complicated neural network should not automatically be assumed to perform better. The model must be evaluated on unseen later data and compared against basic approaches such as:

Predicting the same price as today
Predicting the majority direction
Logistic regression
Linear regression
Random forest

Stock-market prediction remains uncertain. Preprocessing cannot guarantee accurate forecasts, but it prevents avoidable errors and gives the model a cleaner, more realistic dataset.

Frequently Asked Questions

Why must stock data be split by date?

A prediction model is supposed to learn from the past and make predictions about the future. A chronological split recreates this process more realistically than a random split.

Should I predict the closing price or price direction?

Closing-price prediction is a regression problem. Direction prediction is a classification problem. Neither is automatically easier or more profitable. The best target depends on how the prediction will be used.

Why are the first rows missing after creating moving averages?

A 20-day moving average requires 20 historical records. The earliest rows do not have enough previous data, so their moving-average values are unavailable.

Should missing weekends be filled?

Normally, no. Weekends and market holidays are not trading days. Daily stock models usually work with actual trading records rather than artificial weekend prices.

Can I use adjusted closing prices?

Yes. Adjusted prices are useful for long-term analysis because they account for events such as stock splits and dividends. Use adjusted and unadjusted values consistently.

Which scaler is best for stock-market data?

MinMaxScaler is commonly used with neural networks and LSTMs. StandardScaler and robust scaling methods are also possible. The scaler must always be fitted using training data only.

Why can scaled test values exceed 1?

The test-period market may move beyond the minimum or maximum observed during training. MinMaxScaler uses the training range, so later values can legitimately fall below 0 or above 1.

How many previous days should an LSTM use?

Common starting points include 20, 30 or 60 trading days. The best sequence length should be determined using chronological validation rather than selecting the one with the best test result.

Can technical indicators guarantee profitable predictions?

No. Technical indicators are mathematical transformations of historical price and volume data. They may help a model identify patterns, but they do not guarantee future performance.

Is stock prediction accuracy the same as trading profitability?

No. A model must also overcome transaction costs, slippage, spreads and incorrect prediction sizes. A profitable strategy requires more than a high classification-accuracy score.

What comes after data preprocessing?

The next step is to train and evaluate several models using the prepared dataset. Start with a simple baseline before testing more complicated models such as neural networks or LSTMs.

Is this financial advice?

No. This tutorial demonstrates data preprocessing and machine-learning concepts. It does not recommend buying, selling or holding any particular investment.

Part 2: Data Preprocessing for Stock Market Prediction Using AI

Quick Answer

Why Data Preprocessing Matters

What We Are Going to Predict

Step 1: Install the Required Python Packages

Step 2: Import the Libraries

Step 3: Download Historical Stock Data

Loading Stock Data from a CSV File

Step 4: Inspect the Dataset

Step 5: Select the Required Columns

Step 6: Convert Columns to the Correct Data Types

Step 7: Sort the Records by Date

Step 8: Remove Duplicate Records

Step 9: Handle Missing Values

Do Weekends Count as Missing Data?

Step 10: Validate the Price Records

Step 11: Create Daily Return Features

Step 12: Create Intraday Price Features

Opening-to-Closing Change

Daily Trading Range

Previous Closing Gap

Step 13: Add Moving Averages

Step 14: Create Price-to-Moving-Average Ratios

Step 15: Add Rolling Volatility

Step 16: Create Volume Features

Step 17: Add Momentum

Step 18: Calculate the Relative Strength Index

Step 19: Add MACD

Step 20: Add Bollinger Band Features

Step 21: Add Calendar Features

Step 22: Create the Prediction Target

Regression Target: Tomorrow’s Closing Price

Classification Target: Will the Price Increase?

Predicting Tomorrow’s Return Instead

Step 23: Remove Rows Made Incomplete by Feature Creation

Step 24: Select the Model Features

Step 25: Check the Target Distribution

Step 26: Split the Data Chronologically

Step 27: Scale the Features Correctly

Why Scaling the Entire Dataset Is a Mistake

Step 28: Convert the Scaled Data Back to DataFrames

Step 29: Prepare LSTM Sequences

Creating Test Sequences Without Losing Context

Step 30: Save the Prepared Data

Complete Preprocessing Example

Visualise the Prepared Data

Closing Price and Moving Averages

Daily Returns

Preventing Data Leakage

Leakage Example 1: Using Tomorrow’s Price as a Feature

Leakage Example 2: Centred Moving Average

Leakage Example 3: Fitting the Scaler on All Data

Leakage Example 4: Randomly Shuffling Time-Series Records

Leakage Example 5: Backfilling Missing Features

Should Outliers Be Removed?

Feature Correlation

Better Evaluation with Walk-Forward Validation

Common Data Preprocessing Mistakes

1. Using Too Little Historical Data

2. Adding Too Many Indicators

3. Ignoring Trading Costs

4. Training on Only One Market Condition

5. Assuming a High Accuracy Means Future Profit

Recommended Basic Feature Set

Testing the Preprocessing Pipeline

Final Preprocessing Checklist

Final Result

Frequently Asked Questions

Why must stock data be split by date?

Should I predict the closing price or price direction?

Why are the first rows missing after creating moving averages?

Should missing weekends be filled?

Can I use adjusted closing prices?

Which scaler is best for stock-market data?

Why can scaled test values exceed 1?

How many previous days should an LSTM use?

Can technical indicators guarantee profitable predictions?

Is stock prediction accuracy the same as trading profitability?

What comes after data preprocessing?

Is this financial advice?