Blog

Best Japanese Second-Hand Stores to Visit in 2026: Hard Off, Book Off, 2nd Street, Komehyo and More

Japan is one of the best countries for second-hand shopping.

Unlike a typical flea market, many Japanese reuse chains organise used products almost like normal retail stores. Depending on the shop, you can find:

Cameras
Game consoles
Laptops
Audio equipment
Watches
Designer bags
Japanese clothing
Sneakers
Anime figures
Pokémon merchandise
Manga
Musical instruments
Household goods
Vintage items

The biggest attraction is not necessarily that everything is cheap.

It is the possibility of finding a specific discontinued model, Japan-only item or unusually well-kept product for considerably less than buying new.

For Malaysian travellers, second-hand shopping can be particularly worthwhile for:

Cameras and lenses
Japanese fashion
Watches
Retro gaming
Anime collectibles
Musical instruments
Luxury bags
Small electronics

However, second-hand purchases require more checking than normal retail purchases because warranty, condition and return policies vary considerably.

Exchange Rate Used

¥100 = RM3.00

Therefore:

¥500 ≈ RM15
¥1,000 ≈ RM30
¥3,000 ≈ RM90
¥5,000 ≈ RM150
¥10,000 ≈ RM300
¥20,000 ≈ RM600
¥50,000 ≈ RM1,500
¥100,000 ≈ RM3,000

Quick Answer

These are some of the best Japanese second-hand chains to visit:

Store	Best For
Hard Off	Electronics, cameras, audio, gaming and musical instruments
Off House	Clothing, household goods, bags and general lifestyle products
Hobby Off	Figures, toys, trading cards and models
Book Off	Books, manga, games, CDs and smaller used goods
Book Off Super Bazaar	Large one-stop second-hand shopping
2nd Street	Used clothing, bags, shoes and general lifestyle products
Komehyo	Luxury watches, bags, jewellery and designer goods
RAGTAG	Designer and premium second-hand fashion
Kindal	Japanese and international designer clothing
Mandarake	Manga, anime, figures and rare collectibles
Surugaya	Retro games, figures, anime merchandise and hobby products

Hard Off Group currently operates more than 1,000 stores across its various reuse formats, while 2nd Street says it operates more than 900 Japanese stores and covers fashion as well as furniture, appliances and lifestyle goods.

Which Second-Hand Store Should You Visit?

If your time is limited, choose according to what you want to buy.

Electronics and Cameras

Go to:

Hard Off
Book Off Super Bazaar
2nd Street larger branches
Specialist camera stores

Clothing

Go to:

2nd Street
RAGTAG
Kindal
Off House

Luxury Watches and Bags

Go to:

Komehyo
RAGTAG
Established specialist resale stores

Anime and Figures

Go to:

Mandarake
Surugaya
Hobby Off

Retro Games

Go to:

Surugaya
Hard Off
Book Off
Mandarake

Cheap General Treasure Hunting

Go to:

Hard Off
Off House
Book Off Super Bazaar

1. Hard Off

Despite its unusual English name, Hard Off is one of Japan’s most useful second-hand chains for electronics and technical products.

Its main categories include:

Audio
Televisions
Computers
Cameras
Smartphones and tablets
Games
Musical instruments
Watches
Tools
Records
DVDs and CDs

Hard Off officially lists products such as televisions, computers, computer accessories, audio equipment, digital cameras, game consoles and musical instruments among its principal categories.

Best Things to Buy at Hard Off

For Malaysian travellers, the most interesting categories are:

Cameras
Camera lenses
Headphones
Amplifiers
Portable audio
Retro consoles
Game controllers
Watches
Guitars
Guitar effects pedals
Computer accessories

Large branches can be much more interesting than small city-centre locations.

For example, Hard Off’s Akihabara branch currently lists cameras, computers, games, watches, musical instruments, smartphones, audio equipment and a dedicated junk section among its categories.

Hard Off’s Famous Junk Section

One of the most interesting parts of Hard Off is usually labelled:

ジャンク

or:

JUNK

This does not simply mean cosmetically scratched.

Hard Off officially explains that its junk category may contain products that do not operate correctly or no longer have normal product value. They are intended for repair, parts or experimentation. Junk items have no product warranty and cannot be returned or exchanged.

That distinction is extremely important.

Example of a Hard Off Junk Purchase

You might see:

Used camera body:

¥15,000 (RM450)

Junk camera body:

¥3,000 (RM90)

The ¥3,000 camera might:

Not power on
Have a shutter problem
Have no battery
Have corrosion
Have an unknown fault
Simply be untested

Sometimes enthusiasts find excellent bargains.

But if you cannot repair electronics, treat the junk section as:

Parts only.

Do not assume:

“It probably works.”

Hard Off explicitly states that even written test observations on junk products do not constitute a guarantee of operation.

Should Malaysians Buy Hard Off Junk?

Only when:

You understand the product.
You can test it yourself.
You need spare parts.
The price is low enough to accept complete failure.
You are comfortable with zero warranty.

Avoid buying an expensive junk camera, laptop or game console merely because the price appears attractive.

2. Off House

Off House is part of the same Hard Off group but focuses more on household and lifestyle products.

Depending on the branch, you may find:

Clothing
Shoes
Bags
Kitchen products
Furniture
Sports equipment
Camping equipment
Household items
Small appliances
Accessories

Off House is useful if your travelling companion is not interested in electronics but still wants to browse second-hand bargains.

Hard Off Group also runs multiple specialised formats, including Off House, Hobby Off, Mode Off and Garage Off.

What Is Worth Buying at Off House?

Good choices include:

Japanese jackets
Bags
Outdoor clothing
Kitchen tools
Small camping products
Japanese-brand clothing
Shoes in excellent condition

Poor choices for tourists include:

Large furniture
Large appliances
Heavy cookware
Bulky household storage

Remember that something can be extremely cheap but still be a terrible travel purchase because of baggage weight.

3. Hobby Off

Hobby Off specialises in toys and collectibles.

Its official categories include:

Game hardware
Games
Trading cards
Miniatures
Figures
Railway models
Model cars
Plastic models
Radio-controlled products
Character merchandise

Best For

Pokémon cards
Anime figures
Gundam
Model kits
Tomica
Railway models
Retro toys
Character merchandise

Prices range dramatically.

You may find a small used figure for:

¥500 = RM15

while a rare collectible can cost:

¥50,000 = RM1,500

or much more.

Check Figure Condition Carefully

Before buying, inspect:

Missing accessories
Broken parts
Yellowing
Paint transfer
Box condition
Dust
Smoke smell
Sun fading

Collectors place considerable value on boxes.

If you intend to resell the item later, box condition may matter almost as much as the figure itself.

4. Book Off

Book Off originally became famous for used books, manga and media.

Today, larger branches may also carry:

Games
Consoles
CDs
Blu-rays
Trading cards
Smartphones
Electronics
Watches
Fashion
Luxury products

Book Off’s tourist information describes some locations as one-stop stores offering branded products, watches, fashion, golf items, CDs and games alongside books.

Best Things to Buy at Book Off

Manga

Japanese-language manga can be extremely inexpensive.

Common older volumes may cost only a few hundred yen.

A ¥220 manga volume costs approximately:

RM6.60

However, manga is heavy.

Ten books can easily add several kilograms to your luggage.

Japanese Books

Book Off is excellent for:

Japanese-learning materials
Children’s books
Art books
Manga
Game guides
Hobby books
Photography books

If you are studying Japanese, used children’s books can provide much better value than buying new learning material.

Used Games

Book Off may sell:

Switch games
PlayStation games
Nintendo DS games
3DS games
Retro cartridges

Check:

Language
Region compatibility
Download-code usage
Included accessories

A used game containing a one-time download code may have already had the code redeemed.

Book Off Tax-Free Shopping

Book Off currently provides tax-free shopping at participating tourist-oriented stores.

Its official tourist guidance says eligible tax-free purchases currently require at least ¥5,000 before tax, together with the visitor’s passport, although some products are excluded.

Tax-free availability depends on the branch, so look for the tax-free symbol before shopping.

5. Book Off Super Bazaar

If you only have time for one general second-hand shop, Book Off Super Bazaar can be an excellent choice.

These larger locations may contain several categories under one roof.

Depending on the branch, you may find:

Books
Manga
Games
Electronics
Clothing
Sneakers
Sports products
Watches
Bags
Toys
Trading cards
Musical instruments

Several current Super Bazaar branches also advertise tax-free shopping for eligible visitors.

Super Bazaar vs Normal Book Off

Category	Normal Book Off	Super Bazaar
Books	Excellent	Excellent
Manga	Excellent	Excellent
Games	Good	Very good
Clothing	Limited	Much better
Sports	Limited	Better
Luxury bags	Depends	Better
Electronics	Depends	Better
Toys	Good	Very good
Time needed	30–60 min	1–3 hours

If you enjoy treasure hunting, a Super Bazaar can easily consume half a day.

6. 2nd Street

2nd Street is one of Japan’s strongest chains for second-hand clothing.

It currently says it operates more than 900 Japanese stores and primarily handles:

Clothing
Bags
Shoes
Furniture
Electronics
Lifestyle goods

Best For

Japanese streetwear
Uniqlo
GU
Workwear
Outdoor brands
Sneakers
Designer clothing
Bags
Jackets

2nd Street Store Types Matter

Not every branch is identical.

Some branches focus more heavily on fashion, while larger “total reuse” stores can carry:

Furniture
Appliances
Musical instruments
Outdoor products
Children’s products

For example, a current total-reuse branch describes its range as clothing, fashion accessories, furniture, appliances, musical instruments, outdoor equipment and children’s goods.

Meanwhile, urban stores such as Harajuku can be more fashion-focused.

What Can RM100 Buy at 2nd Street?

RM100 is approximately ¥3,333.

Depending on brand and condition, you may find:

Several basic T-shirts
One Japanese-brand shirt
Used Uniqlo outerwear
Used sneakers
Small bag

But designer pieces can still cost thousands of ringgit.

Second-hand does not necessarily mean cheap.

Used Clothing Inspection Checklist

Before buying, check:

Collar

Look for:

Yellowing
Makeup
Sweat marks

Underarms

Check for:

Discolouration
Odour
Fabric weakness

Cuffs

Look for:

Fraying
Stains
Stretching

Zips

Open and close them several times.

Buttons

Check for:

Missing buttons
Loose stitching
Non-original replacements

Pockets

Check inside every pocket.

Fabric

Look for:

Holes
Pilling
Moth damage
Sun fading

2nd Street Used Electronics

Some 2nd Street branches also sell used electrical products.

The chain currently provides a return/refund framework for qualifying products: purchases with defects can generally be returned within seven days, while qualifying used electrical products priced at ¥1,000 before tax or more may receive a refund guarantee of up to six months. Batteries and other consumables are excluded.

That sounds reassuring, but there is a major issue for tourists:

If a device fails after you return to Malaysia, taking it back to the Japanese purchase branch may be impractical.

So test everything before leaving Japan.

7. Komehyo

Komehyo is very different from rummaging through a ¥500 clothing rack.

It is better described as a premium reuse retailer.

Komehyo operates as a licensed second-hand dealer and sells both used and new goods through its official platform. Its categories prominently include designer bags and watches.

Best For

Rolex
Grand Seiko
Cartier
Omega
Chanel
Louis Vuitton
Hermès
Jewellery
Designer accessories

Why Buy Luxury Second-Hand in Japan?

A reputable Japanese resale retailer can be attractive because you may get:

A discontinued reference
Older designs
Better pricing than new
Condition information
Store receipt
Established dealer reputation

However, second-hand luxury should still be inspected carefully.

Do not assume:

“Japan means counterfeit products do not exist.”

Buy from an established specialist rather than a random seller when spending several thousand ringgit.

Example Luxury Budget

Suppose a used watch costs:

¥200,000

Approximate Malaysian value:

RM6,000

A second watch costs:

¥500,000

Approximate value:

RM15,000

At these values, inspect:

Reference
Serial
Bracelet
Service history
Box
Warranty papers
Authenticity
Return conditions

Saving RM1,000 is irrelevant if the watch immediately needs an RM3,000 service.

8. RAGTAG

RAGTAG specialises in curated used fashion.

The company describes itself as a used-fashion select shop carrying everything from popular casual labels to luxury and designer brands. It was founded in 1985 and continues to operate stores across major Japanese cities.

Best For

Japanese designer fashion
Streetwear
Luxury clothing
Shoes
Bags
Premium casual brands

You may find brands such as:

Comme des Garçons
Issey Miyake
Yohji Yamamoto
Sacai
Undercover
BEAMS
United Arrows
International luxury labels

RAGTAG vs 2nd Street

Category	RAGTAG	2nd Street
Budget basics	Limited	Better
Designer clothing	Excellent	Good
Japanese fashion	Excellent	Excellent
Luxury	Strong	Depends on branch
Treasure hunting	Curated	Broader
Lowest prices	Less likely	More likely
Store organisation	More boutique-like	Varies

RAGTAG is better when you already know what designers you like.

2nd Street is better when you simply want to browse.

9. Kindal

Kindal is another specialist used-fashion chain.

Its official site describes the business as a used select shop specialising in the purchase and sale of branded second-hand clothing.

Its store network includes locations in Tokyo, Osaka, Kyoto and other regions, and a new Aoyama store opened in February 2026.

Best For

Japanese designer clothing
Streetwear
Luxury labels
Archive pieces
Sneakers
Bags

Kindal is particularly useful for shoppers who already search by:

Brand
Model
Collection
Season

Used Designer Clothing Tip

Search the label online while inside the store.

For example:

A jacket costs:

¥18,000 = RM540

Before buying, check:

Original retail price
Year
Material
Whether it is a men’s or women’s cut
Current Malaysian resale price
Whether alterations were made

A designer label does not automatically mean a used piece is good value.

10. Mandarake

Mandarake is one of Japan’s best-known destinations for collectible Japanese pop culture.

Its inventory spans categories such as:

Manga
Anime
Games
Toys
Tokusatsu
Dolls
Vintage toys
American comics
Character merchandise

Its online listings also clearly note that second-hand items may have minor damage.

Best For

Old manga
Rare figures
Vintage toys
Sailor Moon
Gundam
Dragon Ball
Pokémon
Godzilla
Tokusatsu
Retro anime
Doujinshi
Collector goods

Mandarake Is Not Necessarily Cheap

For common items, other stores may be cheaper.

Mandarake is particularly strong when you want something:

Rare
Old
Discontinued
Collectible
Difficult to identify elsewhere

A rare figure may cost more at Mandarake because the shop already understands its collector value.

That is different from finding an overlooked bargain at a suburban Hard Off.

11. Surugaya

Surugaya is another major destination for Japanese hobby and gaming products.

Its official store describes a large selection spanning games, books, DVDs, CDs, trading cards and figures, from current products to retro games.

Certain branches can be enormous.

Surugaya’s flagship operation in Shizuoka promotes approximately 10 million anime and hobby-related items across its facilities, with dedicated game and hobby sections.

Best Things to Buy at Surugaya

Retro games
Consoles
Figures
Pokémon goods
Nintendo merchandise
Anime goods
Model kits
Trading cards
Character products
CDs
Game guides

Its Kyoto Teramachi branch, for example, currently advertises games, figures, model kits, Chiikawa, Hololive, Pokémon and Nintendo-related merchandise.

Surugaya vs Mandarake

Category	Surugaya	Mandarake
Retro games	Excellent	Very good
Anime figures	Excellent	Excellent
Manga	Good	Excellent
Very old collectibles	Good	Excellent
Modern character goods	Excellent	Good
Huge searchable inventory	Excellent	Excellent
Collector expertise	Strong	Very strong

Visit both if anime and gaming are a major part of your trip.

Best Second-Hand Stores by Product

Product	Best Starting Point
Cameras	Hard Off
Camera lenses	Hard Off / specialist camera store
Retro game consoles	Surugaya / Hard Off
Game software	Surugaya / Book Off
Manga	Book Off
Anime figures	Mandarake / Surugaya
Pokémon merchandise	Surugaya / Hobby Off
Cheap clothing	2nd Street / Off House
Designer clothing	RAGTAG / Kindal
Luxury bags	Komehyo
Used watches	Komehyo
Musical instruments	Hard Off
Audio equipment	Hard Off

The Suburban Store Advantage

One useful second-hand shopping strategy is to avoid focusing only on:

Shibuya
Shinjuku
Harajuku
Akihabara

Central tourist locations are convenient, but suburban branches can sometimes provide:

Larger floor space
Broader household inventory
More ordinary local items
Less competition from tourists

The disadvantage is transport time.

Do not spend ¥2,000 in train fares and three hours travelling just to save ¥1,000 on a jacket.

Example RM100 Second-Hand Haul

RM100 is approximately ¥3,333.

Possible haul:

Item	JPY	Approx. RM
Used manga	¥220	RM6.60
Used T-shirt	¥800	RM24
Retro game	¥700	RM21
Small figure	¥800	RM24
Used accessory	¥700	RM21
Total	¥3,220	RM96.60

Example RM300 Haul

RM300 is approximately ¥10,000.

Item	JPY	Approx. RM
Japanese-brand jacket	¥3,500	RM105
Used game	¥1,500	RM45
Figure	¥2,000	RM60
Camera accessory	¥2,000	RM60
Manga	¥800	RM24
Total	¥9,800	RM294

Example RM1,000 Haul

RM1,000 is approximately ¥33,333.

Possible purchases include:

Used camera lens
Premium Japanese jacket
Several games
Designer sneakers
Vintage watch

Example:

Item	JPY	Approx. RM
Used lens	¥20,000	RM600
Japanese jacket	¥6,000	RM180
Retro games	¥4,000	RM120
Collectible	¥3,000	RM90
Total	¥33,000	RM990

Used Camera Checklist

For a used camera, inspect:

Sensor
LCD
EVF
Autofocus
Shutter
Buttons
Control dials
Memory-card slot
USB port
Battery
Language setting

Ask whether:

Battery is included
Charger is included
Shutter count is known
Warranty is provided

Used Lens Checklist

Use a phone flashlight to inspect the glass.

Look for:

Fungus
Haze
Scratches
Heavy dust
Oil
Separation

Test:

Autofocus
Manual focus
Zoom
Stabilisation
Aperture

Malaysia’s humid climate makes lens fungus particularly undesirable.

Avoid importing a fungus-contaminated lens into your existing camera collection.

Used Laptop Checklist

Check:

Battery health
Keyboard language
Charger voltage
SSD capacity
USB ports
Wi-Fi
Display
Dead pixels
Account locks
Operating-system licence

A Japanese laptop may have a JIS keyboard rather than the US layout commonly used in Malaysia.

Try typing before buying.

Used Smartphone Checklist

Check:

SIM lock
IMEI
Battery health
Screen burn-in
Cameras
Face ID or fingerprint reader
Charging
Speaker
Microphone
Japanese cellular bands
Account lock

Do not buy a smartphone that is still linked to someone’s:

Apple ID
Google account
Manufacturer account

Used Gaming Console Checklist

Check:

Region restrictions
Language
Power supply
Controller drift
HDMI output
Cartridge reader
Disc drive
Wi-Fi
Storage
Battery

Older handheld batteries may need replacement even when the console itself is in excellent condition.

Japan Voltage Still Matters for Used Electronics

Used electronics can have the same voltage problem as new electronics.

Japan uses 100V.

If an appliance says:

AC100V

do not connect it directly to Malaysia’s roughly 230V supply.

Small electronics using a charger marked:

100–240V, 50/60Hz

are much easier to own in Malaysia.

Used rice cookers, amplifiers, kitchen appliances and older audio equipment should be checked particularly carefully.

Used Watch Checklist

Check:

Reference number
Serial number
Accuracy
Crown
Date function
Bracelet
Spare links
Crystal
Bezel
Water resistance
Service history
Box and papers

For an automatic watch, ask whether the store has measured its daily rate.

For expensive watches, consider whether a service may soon be required.

Used Bag Checklist

Check:

Corners
Handle wear
Strap
Interior stains
Zip
Odour
Peeling
Hardware
Serial or authenticity markings

Leather items can look excellent under shop lighting but reveal scratches in daylight.

Use your phone flashlight.

Used Sneaker Checklist

Inspect:

Sole separation
Hardened foam
Heel wear
Insole
Odour
Cracking
Midsole yellowing
Stitching

A shoe can be unworn but still deteriorate with age.

Old foam and glue may fail soon after purchase.

Condition Labels You May See

Useful Japanese words include:

Japanese	Meaning
中古	Used
未使用	Unused
未開封	Unopened
美品	Very good condition
使用感あり	Signs of use
キズ	Scratch
汚れ	Dirt or stain
色あせ	Fading
欠品	Missing part
動作確認済み	Operation checked
動作未確認	Operation not checked
ジャンク	Junk / as-is
保証なし	No warranty
箱なし	No box
付属品	Accessories
現状品	Sold as currently presented

The most dangerous phrase for beginners is:

動作未確認

It means the operation has not been checked.

Do not mentally translate that as:

“Probably fine.”

Returns Can Be Difficult for Tourists

Store policies differ.

2nd Street currently provides short-term returns for qualifying defective purchases and longer refund guarantees on certain electrical products, while Hard Off’s junk goods specifically have no returns or warranty.

Even when you technically have a warranty, it may require bringing the product back to Japan.

Therefore:

Test your purchase before flying home.

Tax-Free Shopping at Second-Hand Stores

Second-hand goods can still qualify for tax-free shopping when purchased from a participating tax-free retailer and when the relevant requirements are met.

Examples:

Book Off currently operates tax-free participating stores.
Hard Off says some of its branches currently support tax-free purchases, with conditions varying by branch. Its tax-free store list was updated in July 2026.
Certain 2nd Street branches explicitly advertise tax-free service.

Do not assume every branch participates.

Look for:

TAX FREE

or ask:

免税できますか？

Menzei dekimasu ka?

Meaning:

Can I buy this tax-free?

Important Tax-Free Change From November 1, 2026

Japan changes its tourist tax-free system on November 1, 2026.

Under the new refund method:

You pay the tax-inclusive price at the store.
You must take the qualifying goods out of Japan within 90 days.
Customs confirms at departure that you are carrying them.
The retailer or its appointed operator refunds the consumption-tax-equivalent amount after confirmation.

The revised system also removes the old separation between general goods and consumables and uses a minimum purchase threshold of ¥5,000 before tax without that category distinction.

For travellers visiting before November 1, the existing system remains in effect.

Tax-Free Example

Suppose you buy a used camera for:

¥55,000 including 10% consumption tax

Approximately:

RM1,650

The underlying pre-tax amount is:

¥50,000

Approximately:

RM1,500

Potential tax component:

¥5,000

Approximately:

RM150

Whether the entire RM150 is effectively saved depends on the applicable system and retailer procedure.

Do not buy something solely to reach the tax-free threshold.

Second-Hand vs New: When Used Is Better

Used is particularly attractive for:

Cameras

Camera bodies depreciate quickly.

Lenses

Good lenses can remain useful for many years.

Japanese Fashion

A ¥20,000 shirt may appear used for ¥5,000.

Games

There is often little reason to buy an ordinary physical game new if a complete used copy is much cheaper.

Mechanical Watches

Used can provide access to discontinued references.

Collectibles

The product may no longer be manufactured.

When Buying New Is Better

Consider buying new when:

Battery health matters.
Warranty is important.
Hygiene matters.
Price difference is small.
Replacement parts are difficult.
You cannot test the product.
The used product is very old.
Software support is ending.

Example:

New headphones:

¥18,000 = RM540

Used:

¥16,000 = RM480

Saving:

RM60

For only RM60, the new headphones may provide:

Fresh battery
Full accessories
Better hygiene
Warranty

The used option is not automatically better.

Used Electronics: Calculate Battery Cost

Suppose:

Used camera:

¥40,000 = RM1,200

New battery:

¥8,000 = RM240

Actual effective cost:

RM1,440

If a new camera with warranty costs RM1,550, the used bargain becomes much less attractive.

Apply the same logic to:

Laptops
Phones
Handheld consoles
Headphones
Smartwatches

What Malaysians Should Avoid Buying Second-Hand

Be cautious with:

Very old lithium batteries
Unsealed personal grooming products
100V-only appliances
Ancient SSDs
Old hard drives
Heavily used power banks
Damaged helmets
Child safety seats
Old running shoes
Cracked electrical cables

Some products are inexpensive second-hand because much of their useful life has already been consumed.

Best Second-Hand Shopping Areas

Tokyo

Strong areas include:

Akihabara — electronics, games, anime
Shinjuku — cameras, fashion and general shopping
Harajuku — fashion
Shimokitazawa — vintage clothing
Koenji — vintage clothing
Nakano — collectibles

Osaka

Look around:

Nipponbashi Den Den Town
Shinsaibashi
Amerikamura
Umeda

Kyoto

Teramachi is particularly convenient for a mixture of:

Games
Anime
Fashion
Hobby products

Surugaya currently operates a multi-floor Teramachi location near Kyoto Kawaramachi.

Do Not Ignore Smaller Cities

Second-hand shopping can be excellent outside Tokyo and Osaka.

You may find:

Larger stores
Lower competition
Easier parking
More ordinary Japanese household items

If your itinerary already includes:

Nagoya
Fukuoka
Hiroshima
Sapporo
Sendai
Shizuoka

check whether a large reuse shop is close to your hotel.

Do not travel there solely because TikTok says one branch is “the cheapest in Japan.”

Inventory changes constantly.

How Long Should You Spend?

Hard Off

Allow:

30–90 minutes

Enthusiasts may need much longer.

Book Off

Allow:

30–60 minutes

Book Off Super Bazaar

Allow:

1–3 hours

2nd Street

Allow:

30–90 minutes

Komehyo

Allow:

30–90 minutes

Mandarake

Allow:

1–3 hours

Surugaya

Allow:

1–3 hours

Do not place several huge second-hand stores into the same sightseeing day unless shopping is the main purpose.

Practical Shopping Strategy

Before Your Trip

Create a shortlist.

Example:

Item	Malaysia Price	Maximum Japan Price
Camera lens	RM1,500	RM1,200
Jacket	RM500	RM300
Retro game	RM250	RM150
Watch	RM2,000	RM1,600

This prevents buying something simply because the Japanese price tag looks low.

While Shopping

For purchases above around ¥10,000 (RM300):

Photograph the model number.
Search the Malaysian price.
Inspect the condition.
Confirm accessories.
Check warranty.
Check voltage or region.
Ask about returns.
Check tax-free availability.
Consider luggage weight.
Only then buy.

Luggage Considerations

Second-hand shopping can quickly create baggage problems.

Heavy purchases include:

Books
Manga
Records
Amplifiers
Consoles
Cameras
Model kits
Shoes

Example

Twenty manga volumes at roughly 180 g each:

3.6 kg

Add:

Two pairs of shoes: 2 kg
Camera equipment: 2 kg
Figures: 1 kg

Total:

8.6 kg

A cheap ¥110 manga volume becomes much less cheap if you need to purchase additional airline baggage.

Keep Packaging for Electronics

For electronics, retain:

Receipt
Product label
Warranty
Charger information
Serial number

You can discard some bulky packaging later if necessary, but do not throw everything away immediately.

Common Mistakes Malaysians Make

Thinking Everything Is Cheap

Popular vintage and collectible products can be expensive.

Buying Junk Without Understanding It

Hard Off junk has no warranty or returns.

Ignoring Battery Age

Battery replacement may erase the saving.

Forgetting Japan’s 100V Electricity

Used appliances can be just as incompatible as new ones.

Buying Clothing Without Inspecting the Underarms

Shop lighting can hide stains.

Buying Shoes Only Because They Look New

Old glue and foam can deteriorate even without heavy use.

Buying a Camera Without Checking Language

Some Japanese domestic electronics may have Japanese-only interfaces.

Forgetting Accessories

A missing proprietary charger can be expensive.

Assuming Second-Hand Means Environmentally Friendly and Therefore Worth Buying

It is still wasteful if you never use it.

Spending Hours Saving RM20

Your holiday time has value.

Frequently Asked Questions

Is second-hand shopping in Japan worth it?

Yes.

It is particularly worthwhile for:

Cameras
Clothing
Watches
Games
Anime collectibles
Musical instruments

But compare condition and Malaysia pricing.

Is Hard Off cheap?

It can be.

Hard Off is particularly good for treasure hunting rather than guaranteed lowest prices.

The junk section may be extremely cheap, but those products have no warranty and may not work.

What is the difference between Hard Off and Book Off?

Hard Off focuses more heavily on:

Electronics
Cameras
Audio
Instruments
Computers

Book Off is stronger for:

Books
Manga
Games
Media

Large Book Off Super Bazaar stores expand into many other product categories.

What is the difference between Hard Off and Off House?

Hard Off focuses more on technical products.

Off House focuses more heavily on:

Clothing
Household products
Bags
Furniture
Lifestyle items

Is 2nd Street good for clothes?

Yes.

It is one of the best general chains for used fashion and currently has more than 900 Japanese stores.

Is 2nd Street cheap?

It depends on the item.

Basic clothing may be inexpensive, while rare streetwear and designer products can still command high resale prices.

Where should I buy second-hand designer clothes?

Start with:

RAGTAG
Kindal
2nd Street urban fashion branches

RAGTAG and Kindal specialise more strongly in branded and designer resale.

Where should I buy second-hand luxury watches?

Established specialists such as Komehyo are a better starting point than an unknown shop.

Komehyo operates as a licensed used-goods dealer and maintains dedicated watch and luxury-goods categories.

Where should I buy anime figures?

Try:

Mandarake
Surugaya
Hobby Off

Where should I buy retro games?

Start with:

Surugaya
Hard Off
Book Off
Mandarake

Can second-hand items be tax-free?

Yes, at participating tax-free shops when the purchase meets the applicable requirements.

Book Off, some Hard Off branches and selected 2nd Street locations currently advertise tax-free service.

Can I return a used product?

It depends.

Some products and stores provide return or refund protection, while Hard Off junk specifically does not.

Read the receipt before leaving.

Should I buy a second-hand camera?

Yes, if:

The saving is meaningful.
The camera works correctly.
The language is usable.
Battery condition is acceptable.
The store has a good reputation.

Japan is particularly strong for used photography equipment.

Should I buy second-hand headphones?

Usually only when the discount is substantial.

Consider:

Battery degradation
Hygiene
Ear pad wear
Warranty

How much should I budget?

Shopping Level	JPY	Approx. RM
Casual browsing	¥2,000–5,000	RM60–150
Clothing and games	¥5,000–15,000	RM150–450
Camera or fashion purchase	¥15,000–50,000	RM450–1,500
Premium watch or bag	¥50,000–200,000	RM1,500–6,000
Luxury resale shopping	¥200,000+	RM6,000+

Final Verdict

Japanese second-hand stores are worth including in a Japan itinerary even if you do not normally buy used products.

For Malaysians, the strongest options are:

Hard Off for cameras, electronics, audio and gaming
Off House for inexpensive household and fashion products
Hobby Off for figures and toys
Book Off for manga, books and games
Book Off Super Bazaar for large one-stop treasure hunting
2nd Street for used fashion
Komehyo for luxury watches and bags
RAGTAG for designer clothing
Kindal for Japanese and international fashion
Mandarake for rare anime and collectibles
Surugaya for games, anime and hobby products

For most travellers, a second-hand budget of RM100–500 is enough for an enjoyable shopping session.

Camera, watch and luxury buyers may spend substantially more, but those purchases should be treated like serious retail transactions rather than souvenirs.

Always check:

Condition
Model number
Accessories
Battery health
Voltage
Language
Region compatibility
Warranty
Return policy
Malaysia price
Luggage weight

And remember one rule above all when entering Hard Off:

If the label says “JUNK”, assume it does not work unless you can personally prove otherwise.

That single rule can turn Japan’s second-hand stores from an expensive mistake into one of the most interesting and valuable shopping experiences of your trip.

August 12, 2026

Best Japanese Electronics to Buy in Japan in 2026: A Malaysian Traveller’s Guide

Japan is famous for electronics, but buying electronics in Japan is very different from buying snacks, cosmetics or stationery.

A product may look like a bargain but become difficult or even unsafe to use in Malaysia because of:

Japan’s 100V electricity supply
Japanese-only menus
Region restrictions
Domestic-only warranty
Different accessories
Replacement-part availability
Smartphone app restrictions
Different wireless or account regions
Malaysian prices being cheaper during local promotions

For Malaysian travellers, the best electronics to buy in Japan are generally portable devices that support 100–240V charging, do not depend heavily on Japanese services and offer a clear price or model advantage over Malaysia.

Good examples include:

Cameras and lenses
Headphones and earbuds
Portable audio equipment
Electric shavers with worldwide-voltage support
Dual-voltage hair-care devices
Selected gaming accessories
Compact electronics accessories
Japan-exclusive devices that you have researched beforehand

Large Japanese domestic appliances are usually much less attractive.

Japan uses 100V AC, with 50Hz in eastern Japan and 60Hz in western Japan. Malaysia uses roughly 230V, meaning a Japanese 100V-only appliance should not simply be plugged into a Malaysian socket.

Exchange Rate Used

¥100 = RM3.00

Therefore:

¥1,000 ≈ RM30
¥5,000 ≈ RM150
¥10,000 ≈ RM300
¥20,000 ≈ RM600
¥50,000 ≈ RM1,500
¥100,000 ≈ RM3,000
¥200,000 ≈ RM6,000

Prices below are practical budget ranges rather than guaranteed store prices.

Quick Answer

The Japanese electronics most worth considering are:

Electronics	Worth Buying?	Typical Budget
Cameras	⭐⭐⭐⭐⭐	RM1,500–15,000+
Camera lenses	⭐⭐⭐⭐⭐	RM600–10,000+
Used cameras	⭐⭐⭐⭐⭐	RM500–10,000+
Sony headphones	⭐⭐⭐⭐	RM300–1,800
Wireless earbuds	⭐⭐⭐⭐	RM200–1,500
Panasonic shavers	⭐⭐⭐⭐	RM300–2,000
Dual-voltage hair dryers	⭐⭐⭐⭐	RM300–1,500
Electric toothbrushes	⭐⭐⭐	RM150–1,000
Nintendo accessories	⭐⭐⭐	RM100–1,500
Nintendo Switch 2	⚠️ Check version carefully	RM1,800–2,100+
iPhone	⭐⭐⭐	RM3,000–8,000+
MacBook or iPad	⭐⭐⭐	RM2,000–12,000+
Portable audio players	⭐⭐⭐⭐	RM500–6,000+
Small USB-C electronics	⭐⭐⭐⭐	RM30–600
Rice cookers	⚠️ Usually not recommended	RM300–3,000+
Japanese-only hair appliances	❌ Usually avoid	—
100V-only kitchen appliances	❌ Usually avoid	—
Large televisions	❌ Generally impractical	—

For most Malaysian travellers, a practical electronics-shopping budget is around RM500–2,000 unless you are specifically buying a camera, computer or premium audio product.

The Most Important Rule: Check the Voltage

Japan’s standard electricity supply is 100V. Malaysia operates at a much higher voltage.

This matters enormously when buying electrical appliances.

Safe Type: 100–240V

A product labelled:

INPUT: 100–240V, 50/60Hz

is generally designed for international voltage ranges.

You would normally only need the correct plug or cable.

Typical products often using universal-voltage power adapters include:

Laptop chargers
Smartphone chargers
Camera chargers
USB-C chargers
Some electric shavers
Some hair-care devices

However, always read the exact model’s label.

Dangerous Type: AC100V Only

If the product says:

AC100V

do not connect it directly to a Malaysian 230V socket.

Doing so can:

Damage the device
Blow an internal fuse
Overheat components
Damage heating elements
Create a fire or electrical hazard

This warning is particularly important for:

Rice cookers
Hair dryers
Straighteners
Kettles
Toasters
Hot plates
Takoyaki machines
Electric pots
Vacuum cleaners
Kitchen appliances

A simple travel plug adapter does not convert voltage.

Plug Adapter vs Voltage Transformer

These are completely different products.

Plug Adapter

Changes the physical plug shape.

Example:

Japanese two-flat-pin plug → Malaysian three-pin socket.

It does not change 230V to 100V.

Transformer

Changes electrical voltage.

For example:

Malaysia 230V → Japan 100V.

However, transformers can be:

Heavy
Expensive
Noisy
Inefficient
Unsuitable for very high-wattage appliances

Buying a 100V-only ¥8,000 appliance for RM240 makes little sense if you then need a large transformer costing hundreds of ringgit.

For most travellers, simply avoid 100V-only appliances.

1. Japanese Cameras

Cameras are one of the strongest electronics categories to shop for in Japan.

Major Japanese camera brands include:

Sony
Canon
Nikon
Fujifilm
OM System
Panasonic
Ricoh
Sigma

Japan offers:

New models
Used equipment
Older discontinued cameras
Large lens selections
Specialist camera stores
Accessories
Collectible film cameras

Typical Budget

Camera Type	Approx. Budget
Used compact camera	RM500–2,000
Premium compact camera	RM2,000–7,000
Entry mirrorless	RM2,000–4,500
Mid-range mirrorless	RM4,000–8,000
Full-frame body	RM5,000–15,000+
Professional body	RM10,000–25,000+

Why Cameras Can Be Worth Buying

Japan has a particularly strong second-hand camera market.

You can often inspect:

Exact cosmetic condition
Shutter count where available
Lens glass
Accessories
Box
Battery
Charger
Warranty condition

Stores specialising in used photography equipment often grade their stock.

This makes Japan more attractive than buying an unknown used camera from an individual seller.

Camera Warning: Japanese Menus

Some Japanese domestic models may have:

Japanese-only menus
Different language options
Domestic firmware
Japan-only warranty

Do not assume that every Sony, Panasonic or Canon camera bought in Japan can be switched to English.

Before buying:

Turn on the camera.
Enter the settings menu.
Find language settings.
Confirm English is actually listed.
Do not rely only on the salesperson saying “international model.”

Camera Buying Checklist

Check:

Sensor condition
Dead pixels
Autofocus
Buttons
Control wheels
USB port
HDMI port
Memory-card slot
Battery health
LCD
EVF
Stabilisation
Flash
Hot shoe
Shutter count
Language selection

For an expensive used camera, test it with your own memory card when permitted.

2. Camera Lenses

Lenses may be an even better Japan purchase than camera bodies.

Popular Japanese brands include:

Canon
Nikon
Sony
Fujifilm
Sigma
Tamron
Panasonic
OM System

Typical Budget

Lens Type	Approx. Budget
Used prime lens	RM300–1,500
Standard prime	RM600–2,000
Standard zoom	RM1,000–4,000
Professional zoom	RM3,000–10,000+
Telephoto	RM2,000–15,000+

Used Lens Inspection

Look through the lens under strong light.

Check for:

Fungus
Haze
Scratches
Dust
Oil on aperture blades
Damaged filter thread
Decentring
Loose zoom barrel
Autofocus noise

A small amount of internal dust is common and often harmless.

Fungus is much more serious.

Malaysia’s humid climate can also make existing fungal contamination worse.

3. Used Film Cameras

Japan is an excellent place for older:

Nikon
Canon
Olympus
Pentax
Minolta
Contax
Leica
Fujifilm cameras

But vintage does not mean reliable.

Check

Light seals
Shutter speeds
Meter
Film advance
Rewind
Battery compartment
Corrosion
Lens fungus
Rangefinder alignment

A ¥10,000 film camera costs approximately RM300, but repairs could easily exceed the purchase price.

Buy from a reputable shop rather than assuming every vintage camera is a bargain.

4. Sony Noise-Cancelling Headphones

Sony headphones are a practical Japanese electronics purchase because they:

Charge over USB
Are easy to carry
Usually work internationally
Do not involve Japan’s 100V appliance problem when charged from an appropriate USB power source

Sony’s current premium line includes the WH-1000XM6, and Sony was still actively promoting the WH-1000XM6 and WF-1000XM6 generation in 2026.

Typical Budget

Product	Approx. Budget
Basic Sony headphones	RM150–500
Mid-range ANC	RM500–1,000
WH-1000X series	RM1,000–1,800
Premium earbuds	RM700–1,500

Is Sony Cheaper in Japan?

Sometimes.

Compare:

Japan street price
Tax-free price
Sony Malaysia promotions
Shopee or Lazada authorised-store promotions
Malaysian warranty
Credit-card conversion fee

For headphones costing RM1,500, saving RM50 is probably not enough to sacrifice easy local warranty support.

A RM300–500 difference is much more meaningful.

5. Japanese Wireless Earbuds

Other Japanese brands worth checking include:

Sony
Audio-Technica
Technics
Final
JVC

Typical Budget

RM200–1,500

Premium specialist earphones can cost much more.

Why They Are Easy to Buy

Earbuds generally avoid most voltage issues because the charging case uses USB power.

Check:

Bluetooth compatibility
App availability
English language
Replacement ear tips
Warranty
Battery replacement options

6. Audio-Technica Headphones

Audio-Technica is a Japanese brand with products ranging from affordable headphones to professional studio equipment.

Categories include:

Wired headphones
Wireless headphones
Turntable equipment
Microphones
Gaming headsets
Earphones

Typical Budget

Type	Approx. Budget
Entry wired headphone	RM150–400
Studio headphone	RM300–1,000
Wireless headphone	RM300–1,500
Premium audiophile model	RM1,500–6,000+

Japan may offer a wider selection of specialist models than Malaysian consumer-electronics stores.

7. Portable Audio Players

Japan remains a good market for dedicated music players and audiophile equipment.

You may find products from:

Sony Walkman
Astell&Kern
FiiO
HiBy
Shanling
Japanese specialist audio brands

Typical Budget

RM500–8,000+

Before buying, check:

English interface
Streaming-app support
Google Play compatibility
Storage
microSD support
Battery condition
Headphone output
Bluetooth codecs

A ¥150,000 music player is approximately RM4,500.

At this price, compare carefully with Malaysian specialist audio shops.

8. Panasonic Electric Shavers

Electric shavers are one of the more practical Japanese appliances to consider.

Panasonic’s Lamdash range continues to include high-end multi-blade models, including its current five-blade Lamdash Pro products.

Typical Budget

Shaver Type	Approx. Budget
Basic rechargeable	RM150–400
Three-blade	RM300–700
Five-blade	RM600–1,500
Premium cleaning-station model	RM1,000–2,000

Important Shaver Check

Do not assume every shaver has universal voltage.

Look for:

100–240V

Also check replacement blade availability.

Panasonic publishes compatibility information for replacement shaver blades, including its 2026 models.

A cheap Japanese shaver is poor value if you cannot easily replace:

Outer foil
Inner blades
Cleaning cartridges

after returning home.

9. Japanese Hair Dryers

Japanese hair dryers are popular, especially Panasonic Nanoe models.

However, hair dryers are also one of the easiest electronics purchases to get wrong.

Japan’s standard models may be designed for 100V only.

Never assume a ¥30,000 hair dryer can be used in Malaysia.

Look Specifically for Worldwide-Voltage Models

Panasonic sells selected hair dryers designed for international use.

For example, Panasonic states that its EH-NA9F can operate across 100–120V and 200–240V ranges with voltage selection.

This is exactly the type of specification Malaysians should look for.

Typical Budget

RM300–1,500

Hair Dryer Checklist

Check the printed label for:

100–240V support
50/60Hz
Maximum wattage
Voltage switch
Plug type
Malaysia warranty

Do not trust a salesperson who says:

“It is okay overseas.”

Read the specifications yourself.

10. Japanese Hair Straighteners

The same voltage warning applies to straighteners.

Some Panasonic products support overseas voltage ranges. For example, Panasonic lists selected straighteners with AC100–120V / 220–240V support.

Typical Budget

RM150–1,000

Check:

Voltage
Plate temperature
Heat-up time
Automatic shut-off
Plug
Warranty

11. Electric Toothbrushes

Japanese electronics retailers carry toothbrushes from:

Panasonic
Philips Sonicare
Oral-B

Typical Budget

RM150–1,000

Potential advantages include:

Compact Japanese models
Replacement-head bundles
Travel cases
Premium models

But check replacement-head availability in Malaysia.

If the Japanese model uses a unique head unavailable locally, long-term ownership becomes inconvenient.

12. iPhone

Buying an iPhone in Japan may sometimes make sense, but compare carefully.

Apple Japan currently sells the iPhone 17 generation and states that iPhones bought directly from Apple Japan without a carrier contract are SIM-free and can be used with compatible networks.

Potential Advantages

Japan pricing
SIM-free purchase from Apple
Stock availability
Tax or exchange-rate advantage in some situations

Potential Disadvantages

Malaysia may run better promotions.
Warranty handling may be less convenient.
Cellular model differences need checking.
Trade-in values differ.
Malaysian instalment promotions may be better.

A current Malaysian authorised reseller lists the iPhone 17 from around RM3,999, so compare the exact storage capacity before buying in Japan.

Should Malaysians Buy an iPhone in Japan?

Use this formula:

Japan purchase price
− tax saving if applicable

card conversion cost
warranty inconvenience
possible accessory cost

versus:

Malaysia retail price
− local voucher
− bank discount
− trade-in
− instalment promotion

Do not compare Japanese retail price only against Malaysia’s official list price.

13. iPad and MacBook

Apple laptops and tablets are generally easier to use internationally than 100V-only appliances because their power adapters are designed around portable international use.

However, still check:

Keyboard layout
Charger plug
Exact model
Memory
SSD capacity
AppleCare eligibility
Malaysian price

Japanese Keyboard Warning

A Japanese MacBook may be sold with a Japanese JIS keyboard.

The key layout differs from the common English US layout used by many Malaysians.

Look at:

Space bar size
Enter key
Extra Japanese input keys
Symbol positions

Do not buy an expensive laptop before typing on the keyboard.

14. Nintendo Switch 2 — Major Warning for Malaysians

This is one of the most important 2026 electronics warnings.

Nintendo sells a Nintendo Switch 2 Japanese-Language System (Japan Only).

As of August 2026, Nintendo lists the Japanese-language domestic unit at ¥59,980, approximately:

RM1,799

Nintendo explicitly states that this version:

Is intended for Japan
Uses Japanese as the system language
Can only link to Nintendo Accounts whose country/region is set to Japan
Uses the Japanese-region Nintendo eShop

This version is not the one most Malaysian tourists should buy.

Nintendo Switch 2 Multi-Language Version

Nintendo also lists a multi-language model at:

¥69,980

Approximately:

RM2,099

It supports 16 system languages, but Nintendo states that the Japanese multi-language version is sold through the Nintendo Store.

Even the Multi-Language Switch 2 Has an Overseas Warranty Warning

Nintendo states that when either Japanese Switch 2 version is used overseas, failures or problems arising during overseas use are outside its Japanese warranty/repair coverage. Nintendo also does not guarantee overseas-region physical software with these Japanese systems.

Recommendation for Malaysians

Unless the Japanese saving is substantial:

Buy the Malaysian or appropriate international-region Switch 2.

The cheaper Japan-only unit is not worth saving several hundred ringgit if you need:

English menus
Malaysian Nintendo Account
International convenience
Easy warranty support

15. Nintendo Accessories

Accessories are a much safer Japan purchase.

Consider:

Joy-Con 2
Pro Controller
Carrying cases
Charging accessories
Character accessories
Amiibo
Limited designs

Typical Budget

RM50–600

Check that:

The accessory is compatible with your console version.
Any wireless features work normally.
The product is not region-account dependent.

16. Retro Gaming Hardware

Japan has a large second-hand gaming market.

You may find:

Game Boy
Game Boy Advance
Nintendo DS
Nintendo 3DS
PSP
PS Vita
Retro consoles
Controllers
Cartridges

Typical Budget

From RM100 for common accessories to several thousand ringgit for rare systems.

Retro Console Checklist

Check:

Screen yellowing
Dead pixels
Battery swelling
Buttons
Analogue sticks
Charging
Cartridge slot
Disc drive
Region restrictions
Language
Replacement battery availability

An old rechargeable battery should be considered a potential replacement item, not a permanent component.

17. PlayStation Accessories

Accessories are usually safer purchases than Japanese domestic consoles.

Look for:

Controllers
Charging docks
Headsets
Limited-edition accessories

Check:

Malaysia pricing
Warranty
Compatibility
Colour availability

Large consoles consume significant luggage space and may offer little saving once Malaysian promotions are considered.

18. USB-C Chargers

USB-C chargers are practical electronics to buy in Japan when they clearly state:

100–240V

Consider:

GaN chargers
Multi-port chargers
USB-C PD chargers
Compact travel chargers

Typical Budget

RM50–400

Check:

Total wattage
Wattage per port
USB Power Delivery
PPS support if needed
Certification
Plug configuration

Avoid unknown ultra-cheap chargers for expensive devices.

19. Power Banks

Power banks may be useful, but airline lithium-battery rules matter.

Before buying, check:

Capacity in mAh
Watt-hours
USB-C input
USB-C output
PD wattage
Battery certification
Airline restrictions

Typical Budget

RM50–400

Power banks generally belong in cabin baggage rather than checked luggage.

Check your airline’s latest lithium-battery rules before travel.

20. Japanese Camera Batteries

Official spare camera batteries can sometimes be worth buying.

Typical Budget

RM100–500

Avoid suspiciously cheap batteries.

Lithium batteries are safety-critical components.

Buy from:

Camera shop
Manufacturer store
Reputable electronics retailer

21. SD Cards and Memory Cards

Memory cards are widely available but are not automatically cheaper in Japan.

Check:

Capacity
UHS rating
V30/V60/V90
Write speed
Warranty
Authenticity

Typical Budget

RM30–1,000+

For high-bitrate 4K or 8K recording, buying an unsuitable slow card can cause recording failure.

22. Rice Cookers

Japanese rice cookers are famous.

Brands include:

Zojirushi
Tiger
Panasonic
Toshiba
Mitsubishi

Japan offers extremely sophisticated models with features such as:

Pressure cooking
Induction heating
Multiple rice textures
Clay-pot-inspired cooking
Premium inner pots

However, for Malaysian tourists:

Japanese domestic rice cookers are generally NOT recommended unless explicitly designed for 220–240V export use.

Why Japanese Rice Cookers Are Problematic

Most domestic models are intended for Japan’s 100V electricity supply. Japan’s official tourism guidance confirms the country operates at 100V.

A rice cooker may use:

700W
1,000W
1,200W
More

A transformer powerful enough to handle such continuous loads is:

Large
Heavy
Expensive

It removes most of the benefit of buying the cooker cheaply in Japan.

Better Option

Look specifically for products labelled:

Overseas model
220–240V
230V
100–240V

Electronics retailers in tourist districts may carry export models.

Expect export models to cost more than domestic models.

23. Japanese Kettles and Toasters

The same problem applies to:

Electric kettles
Toasters
Hot plates
Sandwich makers
Takoyaki grills

These are high-wattage heating devices and are often designed for 100V domestic use.

Recommendation

Usually skip them.

Buy a Malaysian 230V model instead.

24. Japanese Vacuum Cleaners

Japanese domestic vacuum cleaners can be:

Compact
Lightweight
Well designed

But many are 100V-only.

Replacement parts may also be difficult to obtain in Malaysia.

For most tourists:

Not worth buying.

The exception is a portable rechargeable vacuum using a universal-voltage USB or charging adapter and having easily available replacement batteries.

25. Japanese Beauty Devices

Japan sells many electronic beauty products, including:

Facial devices
Heated eyelash tools
Hair devices
Facial steamers
Electric cleansing tools
Scalp massagers

Typical Budget

RM100–3,000+

Before buying:

Check voltage.
Check English instructions.
Check replacement consumables.
Check warranty.
Check whether the device has meaningful evidence for its claims.
Avoid buying purely because the packaging says “No.1”.

26. Japanese Translation Devices

Japan sells standalone translation devices designed for travellers.

These may look useful, but smartphones can already perform many translation functions.

Before buying, check whether the device requires:

Subscription
Japanese SIM
Wi-Fi
Cloud service
Annual payment

A RM1,000 translator may be unnecessary if your phone performs the same job.

27. Electronic Dictionaries

Electronic dictionaries remain available in Japan, particularly for students.

Brands have historically included:

Casio Ex-word
Sharp Brain

Who Might Benefit

Japanese-language learners
Students
Users wanting offline dictionaries

Who Should Skip Them

Most casual tourists.

A smartphone provides much greater flexibility.

28. Japanese Alarm Clocks

Japanese clocks from brands such as:

Seiko
Citizen
Rhythm

can be attractive and affordable.

Typical Budget

RM30–500

However, check radio-controlled models.

A clock designed to receive Japanese terrestrial time signals may not automatically synchronise in Malaysia.

Choose a conventional quartz clock unless you understand the radio feature.

29. Smart Home Devices

Be cautious with Japanese domestic smart-home products.

They may depend on:

Japanese server region
Japanese app store
Japanese phone number
Japan-only Wi-Fi services
Japanese language
Japan-specific radio standards

Before buying, confirm:

App works in Malaysia
Region can be Malaysia
Cloud service is accessible
Wi-Fi frequencies are supported
Product remains functional without Japanese services

30. Used Electronics

Japan is particularly good for used electronics.

Stores may carry:

Cameras
Laptops
Smartphones
Headphones
Tablets
Gaming devices
Audio equipment

Used products may be labelled with condition grades.

Used Electronics Checklist

Battery

Check battery health whenever possible.

Display

Look for:

Dead pixels
Burn-in
Yellowing
Pressure marks

Ports

Test:

USB
HDMI
Headphone jack
Charging port

Wireless

Check:

Wi-Fi
Bluetooth
Cellular where relevant

Account Lock

Ensure devices are not:

iCloud locked
Google locked
Manufacturer-account locked

Accessories

Confirm whether the price includes:

Charger
Cable
Battery
Box
Manual

Language

Check the menu before purchasing.

Best Electronics Stores in Japan

Bic Camera

Good for:

Cameras
Computers
Audio
Gaming
Appliances
Travel electronics

Yodobashi Camera

Good for:

Huge product selection
Cameras
Audio
Computers
Gaming
Home electronics

Yamada Denki

Good for:

Appliances
Computers
Household electronics

Edion

Good for:

Appliances
Computers
Cameras
Regional shopping outside Tokyo

Joshin

Good for:

Electronics
Gaming
Hobby products

Sofmap

Good for:

Computers
Gaming
Used electronics
Cameras

Map Camera

Good for:

Cameras
Lenses
Used photography equipment

Hard Off

Good for:

Used electronics
Retro audio
Cameras
Gaming equipment

The exact best store depends more on the product than the chain.

Don Quijote vs Electronics Stores

Don Quijote sells electronics, but specialist retailers are usually better for expensive devices.

Category	Better Choice
Camera	Bic / Yodobashi / specialist camera store
Lens	Camera specialist
Laptop	Bic / Yodobashi / Apple
Headphones	Electronics or audio specialist
Gaming console	Electronics or official store
Cheap accessories	Don Quijote can be fine
Emergency charger	Don Quijote
Travel plug	Don Quijote / electronics store
Beauty appliances	Electronics store

For a ¥200,000 camera, knowledgeable staff and warranty clarity matter more than saving ¥1,000.

Japan-Only Model Checklist

Before buying any JDM electronic device, ask these questions:

Does it support English?
Does it support 220–240V?
Does the charger support 100–240V?
Does it require a Japanese account?
Does it require a Japanese app?
Is there a Malaysia-compatible warranty?
Are replacement parts available in Malaysia?
Can the firmware be updated outside Japan?
Does Bluetooth or Wi-Fi work normally?
Is the product cheaper than Malaysia after promotions?

If several answers are uncertain, skip it.

Example RM500 Electronics Budget

RM500 is approximately ¥16,667.

Possible haul:

Product	JPY	Approx. RM
Compact USB-C charger	¥4,000	RM120
Quality cable	¥2,000	RM60
SD card	¥3,000	RM90
Camera battery	¥5,000	RM150
Small electronic accessory	¥2,000	RM60
Total	¥16,000	RM480

Example RM1,000 Electronics Budget

RM1,000 is approximately ¥33,333.

Possible purchases include:

Wireless headphones
Electric shaver
Dual-voltage hair appliance
Used compact camera
Audio equipment

Example:

Product	JPY	Approx. RM
Shaver	¥20,000	RM600
Replacement foil	¥5,000	RM150
USB-C charger	¥4,000	RM120
Accessories	¥4,000	RM120
Total	¥33,000	RM990

Example RM2,000 Electronics Budget

RM2,000 is approximately ¥66,667.

Possible options include:

Premium headphones
Used camera
Premium shaver
Nintendo Switch 2
Entry mirrorless camera

Remember that the Japan-only Nintendo Switch 2 currently costs ¥59,980, roughly RM1,799, but its Japanese-only system and account restrictions make it unsuitable for many Malaysians.

Example RM5,000 Electronics Budget

RM5,000 is approximately ¥166,667.

Possible purchases:

Mirrorless camera
Premium lens
Apple product
Premium portable audio player
Multiple audio devices

At this price level, compare Malaysian prices online before paying.

A 5% price difference on RM5,000 equals:

RM250

A 10% difference equals:

RM500

That is enough to justify proper comparison.

Tax-Free Electronics Shopping Before November 1, 2026

Eligible visitors can currently use Japan’s existing tax-free procedure at participating retailers.

Exact procedures vary by store.

Typically you should carry:

Original passport
Payment method
Purchased goods
Receipt

Do not assume that every displayed price is already tax-free.

Major Tax-Free Change on November 1, 2026

Japan will change its tourist tax-free system to a refund method from November 1, 2026. Under the new system, tax-free shops initially sell goods at the tax-inclusive price; export is then confirmed and the consumption-tax-equivalent amount is refunded through the applicable process.

This is especially relevant for expensive electronics.

A traveller purchasing a ¥220,000 camera after November 1 should not assume the store will simply remove tax at checkout in the same way as before.

Keep the goods and documentation available for departure procedures.

Understand Japan’s 10% Consumption Tax Properly

Suppose the tax-inclusive electronics price is:

¥110,000

This equals approximately:

RM3,300

The pre-tax price is:

¥100,000

Approximately:

RM3,000

The tax portion is:

¥10,000

Approximately:

RM300

You do not calculate the tax-free price simply by subtracting 10% from ¥110,000.

Warranty: Often More Important Than Tax-Free Savings

Imagine:

Malaysia authorised price:

RM2,000

Japan effective price:

RM1,850

Saving:

RM150

If the Japan model has:

Japan-only warranty
No Malaysian parts
Japanese-only menu

the RM150 saving may not be worthwhile.

For inexpensive headphones, you may accept the risk.

For a RM10,000 camera, warranty differences become much more important.

Bringing Electronics Back to Malaysia

Keep expensive electronics:

Receipt
Box where practical
Serial number
Warranty document
Tax documentation

Royal Malaysian Customs states that travellers carrying dutiable or taxable goods should use the appropriate declaration process, while the Green Lane is for travellers carrying goods not subject to duty or tax.

Do not assume that removing a product from its box automatically removes customs obligations.

For expensive cameras, laptops or multiple new devices, check current Malaysian Customs traveller rules before travelling home.

Lithium Battery Packing

Electronics containing lithium batteries require special attention during flights.

Examples include:

Power banks
Spare camera batteries
Laptop batteries
Drone batteries
Gaming-device batteries

Airline rules differ according to:

Battery capacity
Installed vs spare battery
Cabin vs checked baggage

As a general travel practice, power banks and spare lithium batteries normally belong in cabin baggage rather than checked luggage.

Always verify your airline’s current policy.

Electronics Luggage Strategy

Carry Expensive Devices in Cabin Baggage

Prefer cabin baggage for:

Camera
Lens
Laptop
Tablet
Smartphone
Headphones

Do Not Check Fragile Electronics

Checked luggage can experience:

Impact
Crushing
Rain
Temperature changes
Theft

Keep Batteries Protected

Cover exposed terminals.

Use original cases where appropriate.

Photograph Serial Numbers

Before flying home, photograph:

Product
Serial number
Receipt

This can help with warranty or travel-insurance claims.

Products Most Malaysians Should NOT Buy

100V-Only Rice Cookers

Too inconvenient unless explicitly designed for Malaysian voltage.

100V Hair Dryers

Potentially unsafe on Malaysia’s supply.

Domestic Electric Kettles

High-power appliance requiring a large transformer.

Domestic Toasters

Same problem.

Large TVs

Problems include:

Transport
Damage
Warranty
Tuner compatibility
Voltage
Customs
Oversize baggage

Japanese-Only Smart Appliances

Avoid unless you have confirmed app and region support.

Japan-Only Nintendo Switch 2

The cheaper ¥59,980 domestic model is Japanese-language and Japan-account restricted.

Best Electronics to Buy Under RM300

RM300 is approximately ¥10,000.

Good options include:

USB-C charger
Charging cable
SD card
Camera accessory
Wired earphones
Basic headphones
Small grooming device
Gaming accessories
Retro gaming accessories

Best Electronics Under RM500

Good options include:

Better headphones
Entry electric shaver
Portable audio accessories
Used compact camera
Dual-voltage hair appliance on sale
Official gaming controller

Best Electronics Under RM1,000

Good options include:

Premium earbuds
Mid-range headphones
Better Panasonic shaver
Used lens
Used camera
Premium grooming devices

Best Electronics Under RM2,000

Good options include:

Sony premium headphones plus accessories
Premium Panasonic shaver
Used mirrorless camera
Quality lens
Portable music player
Nintendo console if the version is suitable

Best Electronics Under RM5,000

Good options include:

Mirrorless camera
Premium camera lens
iPhone
iPad
Premium headphones plus camera equipment
Audiophile music player

At this price, do not purchase without comparing Malaysia.

Common Mistakes Malaysians Make

Assuming All Japanese Electronics Are High Quality

Japan sells cheap electronics too.

Brand and model still matter.

Ignoring Voltage

This is the biggest mistake.

Thinking a Plug Adapter Converts Voltage

It does not.

Buying Domestic Appliances

Many are designed only for 100V.

Buying Nintendo Switch 2 Because It Looks Cheap

The ¥59,980 Japanese domestic Switch 2 is Japanese-language and Japan-account restricted.

Forgetting Warranty

Saving RM100 is not worthwhile if a RM5,000 device has no convenient Malaysian support.

Ignoring Japanese Menus

Test the actual device.

Assuming Electronics Stores Are Always Cheaper

Malaysia may have:

Shopee vouchers
Credit-card promotions
Instalments
Trade-in offers
Free gifts

Buying Too Much Because It Is Tax-Free

Tax-free does not make an unnecessary purchase good value.

Throwing Away Receipts

Keep them for customs and warranty.

Frequently Asked Questions

Are electronics cheaper in Japan than Malaysia?

Sometimes.

Japan can offer particularly good value for:

Used cameras
Camera lenses
Japanese audio equipment
Japan-exclusive models
Selected grooming products

But smartphones, computers and mainstream headphones can be similarly priced or cheaper in Malaysia during promotions.

What is the best electronic item to buy in Japan?

For most travellers:

Camera lens
Used camera
Sony or Japanese-brand headphones
Electric shaver
Dual-voltage hair-care product

These categories avoid many of the problems associated with large 100V domestic appliances.

Can Japanese electronics work in Malaysia?

Portable electronics often can.

Look for:

100–240V, 50/60Hz

A product marked only:

AC100V

should not be connected directly to Malaysia’s approximately 230V supply. Japan officially uses 100V.

Can I use a travel adapter?

A travel adapter only changes the plug shape.

It does not change voltage.

Can I use a transformer?

Technically yes for appropriately rated equipment.

However, high-wattage transformers can be large and expensive.

Buying a proper Malaysian-voltage appliance is usually better.

Are Japanese rice cookers worth buying?

Not usually for Malaysian travellers.

Most Japanese domestic models are 100V.

Buy an export model specifically rated for 220–240V if you really want a Japanese rice cooker.

Are Panasonic hair dryers safe in Malaysia?

Only models designed for the relevant voltage.

Panasonic offers selected worldwide-voltage models; for example, it identifies the EH-NA9F as supporting 100–120V and 200–240V.

Do not assume all Nanoe dryers have this capability.

Can I buy a Nintendo Switch 2 in Japan?

Yes, but be very careful.

The standard Japanese domestic unit is Japanese-only and requires a Nintendo Account set to Japan.

A more expensive multi-language model exists, but Nintendo also warns about overseas warranty limitations for Japanese Switch 2 hardware.

For most Malaysians, buying the appropriate Malaysian or international model is safer.

Is an iPhone bought in Japan SIM-free?

Apple Japan states that iPhones bought directly from Apple without a carrier contract are SIM-free.

Still compare:

Exact model
Storage
Malaysian price
Cellular compatibility
Warranty
Local promotions

Should I buy a MacBook in Japan?

Only after checking the keyboard.

Japanese retail units may use JIS keyboard layouts that differ from the standard US-English layout many Malaysians are familiar with.

Are Sony headphones cheaper in Japan?

They can be, especially during promotions.

Compare the exact model with Malaysian authorised retailers.

Sony’s current premium headphone generation includes the WH-1000XM6 and WF-1000XM6.

Where should I buy electronics?

Good options include:

Bic Camera
Yodobashi Camera
Yamada Denki
Edion
Joshin
Sofmap
Specialist camera shops
Official brand stores

For expensive electronics, specialist stores generally provide better product advice than general discount stores.

Is second-hand electronics shopping safe?

It can be.

Japan has a strong used-electronics market, but inspect:

Battery
Screen
ports
account locks
accessories
warranty
language

Buy expensive used equipment from reputable shops.

How much should I budget?

Shopping Level	Approx. RM
Small accessories	RM100–300
Headphones or grooming device	RM300–1,000
Premium portable electronics	RM1,000–2,500
Camera or laptop	RM2,500–8,000
Premium camera gear	RM5,000–15,000+
Serious electronics haul	RM10,000+

Final Verdict

Japan is still an excellent place for electronics shopping in 2026, but Malaysians should be much more selective than they are when buying snacks, cosmetics or stationery.

The best purchases are usually:

Cameras
Camera lenses
Used photography equipment
Sony headphones
Japanese audio products
Panasonic electric shavers
Dual-voltage hair-care products
Compact USB-C electronics
Selected gaming accessories

The products to approach very carefully are:

Rice cookers
Hair dryers
Straighteners
Kettles
Toasters
Vacuum cleaners
Smart-home equipment
Japanese domestic gaming consoles

The single most important check is the voltage label.

If you see:

100–240V

the product is potentially suitable for Malaysian use.

If you see:

AC100V only

do not simply plug it into a Malaysian socket.

Also check:

English language
Region restrictions
Warranty
Replacement parts
Malaysian price
App compatibility
Battery rules
Tax-free procedure

The Nintendo Switch 2 is a good example of why research matters. Japan’s cheaper ¥59,980 (RM1,799) model looks attractive until you realise it is Japanese-language, Japan-account restricted and intended for domestic use.

For most Malaysian travellers, spending around RM500–2,000 on carefully selected portable Japanese electronics is reasonable.

Spend RM5,000 or more only when you have already checked the exact model against Malaysia pricing and confirmed that warranty, voltage and regional compatibility will not become problems after you return home.

August 11, 2026

Building a Smart Search Engine with PHP: A Simplified Approach

A website search box looks simple, but a useful search feature must do more than check whether two strings are identical.

Visitors may type several words, use only part of a word, make a spelling mistake or search with a related term that does not appear in the page title. Good search results should also place the most relevant pages first instead of sorting everything by date.

This tutorial explains how to build a practical search engine using PHP 8, PDO and MySQL FULLTEXT search. It does not require Elasticsearch, OpenSearch, Solr or an artificial-intelligence API.

By the end, the project will support:

Full-text searching across titles, excerpts and article content
Relevance scoring
Extra weight for title matches
Prefix matching such as migrat*
Category filters
Pagination
Synonym expansion
Safe result highlighting
A fallback for short or uncommon searches
“Did you mean?” spelling suggestions
Search analytics without storing visitors’ personal information

The completed example is suitable for a blog, documentation website, tutorial library or small product catalogue.

Quick Answer

The simplest reliable approach is to let MySQL find and rank possible results with a FULLTEXT index, then use PHP to improve the query and display the results.

The core SQL looks like this:

SELECT
    id,
    title,
    slug,
    excerpt,
    MATCH(title, excerpt, body)
        AGAINST(:query IN NATURAL LANGUAGE MODE) AS score
FROM articles
WHERE status = 'published'
  AND MATCH(title, excerpt, body)
      AGAINST(:boolean_query IN BOOLEAN MODE)
ORDER BY score DESC
LIMIT 10;

MySQL’s natural-language mode returns a relevance value for each row. Boolean mode adds useful operators, including required words, excluded words, phrases and prefix wildcards. According to the MySQL full-text documentation, an InnoDB table needs a matching FULLTEXT index for the columns used in MATCH().

PHP can then:

Normalise the visitor’s query.
Convert its words into a safe Boolean query.
Add related words.
Apply filters and pagination.
Escape and highlight the displayed results.

This gives a useful search experience without adding a separate search server.

What Makes a Search Engine “Smart”?

For this tutorial, “smart” does not mean that the search engine understands language like a chatbot. It means the system handles normal visitor behaviour sensibly.

Visitor behaviour	Search feature
Searches `php migration`	Requires both important words
Types `migrat`	Prefix search can find `migration` and `migrations`
Searches `js`	A synonym can also search for `JavaScript`
Misspells `databse`	The page can suggest `database`
Searches within Tutorials	A category filter limits the results
Finds many results	Pagination divides them into manageable pages
Searches an exact title	Title weighting places it near the top
Searches a two-letter term	A controlled `LIKE` fallback can still help

A search engine becomes useful through several small improvements. It does not need to begin with machine learning.

Why Not Use Only SQL LIKE?

A beginner may start with this query:

SELECT *
FROM articles
WHERE title LIKE '%php%'
   OR excerpt LIKE '%php%'
   OR body LIKE '%php%';

This works on a small table, but it has important limitations:

A leading wildcard such as %php% normally prevents efficient use of a standard B-tree index.
Every matching row is treated equally.
Searching several words becomes awkward.
Common words can create noisy results.
It does not provide full-text relevance scoring.
Scanning a large body column becomes expensive as the table grows.

LIKE is still useful as a controlled fallback for short or unusual terms. It should not be the main search method for a growing content library.

How the Project Works

The request moves through four stages:

PHP validates the request. It limits the query length, category and page number.
PHP builds a search expression. It removes Boolean operators supplied by the visitor, keeps useful words and adds optional synonyms.
MySQL finds candidates. A FULLTEXT index filters the table and calculates relevance.
PHP presents the results. It escapes all database content, highlights matching words and creates pagination links.

The database remains the source of truth. There is no separate search index to synchronise.

Requirements

The example uses:

PHP 8.1 or later
The PDO MySQL extension
MySQL 8.0 or later
An InnoDB database
UTF-8 using utf8mb4

The code also uses the mbstring extension for safer Unicode string handling.

Check the installed PHP version:

php -v

Check whether PDO MySQL and mbstring are enabled:

php -m | grep -E 'pdo_mysql|mbstring'

On Debian or Ubuntu, the packages are commonly installed with:

sudo apt install php-mysql php-mbstring

Package names may differ if several PHP versions are installed.

Project Structure

Create a folder containing these files:

smart-search/
├── config.php
├── functions.php
├── search.php
└── setup.sql

setup.sql creates the sample data. config.php opens the database connection. functions.php contains reusable search helpers. search.php processes the request and displays the interface.

Step 1: Create the Database

Create setup.sql:

CREATE DATABASE IF NOT EXISTS smart_search
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_0900_ai_ci;

USE smart_search;

CREATE TABLE categories (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    slug VARCHAR(100) NOT NULL UNIQUE
) ENGINE=InnoDB;

CREATE TABLE articles (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    category_id BIGINT UNSIGNED NOT NULL,
    title VARCHAR(255) NOT NULL,
    slug VARCHAR(255) NOT NULL UNIQUE,
    excerpt TEXT NOT NULL,
    body LONGTEXT NOT NULL,
    status ENUM('draft', 'published') NOT NULL DEFAULT 'draft',
    published_at DATETIME NULL,
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
        ON UPDATE CURRENT_TIMESTAMP,

    CONSTRAINT fk_articles_category
        FOREIGN KEY (category_id) REFERENCES categories(id),

    INDEX idx_articles_listing (status, category_id, published_at),
    FULLTEXT INDEX ft_articles_title (title),
    FULLTEXT INDEX ft_articles_all (title, excerpt, body)
) ENGINE=InnoDB;

CREATE TABLE search_terms (
    term VARCHAR(100) PRIMARY KEY,
    popularity INT UNSIGNED NOT NULL DEFAULT 1
) ENGINE=InnoDB;

CREATE TABLE search_queries (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    query VARCHAR(200) NOT NULL,
    results_count INT UNSIGNED NOT NULL,
    searched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_search_queries_date (searched_at),
    INDEX idx_search_queries_results (results_count)
) ENGINE=InnoDB;

INSERT INTO categories (name, slug) VALUES
    ('PHP', 'php'),
    ('Databases', 'databases'),
    ('Security', 'security');

INSERT INTO articles
    (category_id, title, slug, excerpt, body, status, published_at)
VALUES
    (
        1,
        'PHP Form Validation for Beginners',
        'php-form-validation-for-beginners',
        'Learn how to validate and safely display data submitted through a PHP form.',
        'This tutorial covers required fields, email validation, error messages and output escaping with PHP.',
        'published',
        '2026-01-10 09:00:00'
    ),
    (
        2,
        'Understanding MySQL Indexes',
        'understanding-mysql-indexes',
        'A practical introduction to database indexes, query plans and performance.',
        'Learn when MySQL can use an index, how composite indexes work and how to inspect a query with EXPLAIN.',
        'published',
        '2026-02-15 09:00:00'
    ),
    (
        3,
        'Preventing SQL Injection with PDO',
        'preventing-sql-injection-with-pdo',
        'Use prepared statements and parameter binding to protect PHP database queries.',
        'This security guide explains SQL injection, PDO prepared statements and safe query construction.',
        'published',
        '2026-03-20 09:00:00'
    ),
    (
        1,
        'Building a REST API with PHP',
        'building-a-rest-api-with-php',
        'Create a small JSON API using modern PHP and a MySQL database.',
        'The project includes routing, validation, prepared statements, JSON responses and error handling.',
        'published',
        '2026-04-05 09:00:00'
    );

INSERT INTO search_terms (term, popularity) VALUES
    ('php', 100),
    ('mysql', 90),
    ('database', 80),
    ('validation', 70),
    ('security', 65),
    ('injection', 60),
    ('indexes', 50),
    ('prepared', 45),
    ('statements', 40),
    ('api', 35);

Two full-text indexes are created deliberately:

ft_articles_title lets the ranking formula give title matches extra weight.
ft_articles_all searches the title, excerpt and body together.

The columns in a MATCH() expression must correspond to a suitable FULLTEXT index. Do not change one list without changing the related index and queries.

Import the file:

mysql -u root -p < setup.sql

Use a restricted application account in production rather than connecting the website as the MySQL root user.

Step 2: Create a Safe PDO Connection

Create config.php:

<?php

declare(strict_types=1);

$databaseHost = getenv('SEARCH_DB_HOST') ?: '127.0.0.1';
$databaseName = getenv('SEARCH_DB_NAME') ?: 'smart_search';
$databaseUser = getenv('SEARCH_DB_USER') ?: 'search_user';
$databasePass = getenv('SEARCH_DB_PASS') ?: '';

$dsn = sprintf(
    'mysql:host=%s;dbname=%s;charset=utf8mb4',
    $databaseHost,
    $databaseName
);

$pdo = new PDO($dsn, $databaseUser, $databasePass, [
    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
    PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
    PDO::ATTR_EMULATE_PREPARES => false,
    PDO::ATTR_STRINGIFY_FETCHES => false,
]);

Keep credentials in environment variables. Do not commit a real password to Git or place it in a public web directory.

PDO placeholders represent data values, not SQL identifiers. Category IDs, query text, limits and offsets can be bound as values. A visitor must never be allowed to supply a table name, column name or raw ORDER BY expression.

The PHP PDO manual explains that parameters should be represented by named or question-mark placeholders and that a unique marker should normally be used for each value in a prepared statement.

Step 3: Build the Search Helper Functions

Create functions.php:

<?php

declare(strict_types=1);

function escapeHtml(string $value): string
{
    return htmlspecialchars(
        $value,
        ENT_QUOTES | ENT_SUBSTITUTE,
        'UTF-8'
    );
}

function normaliseQuery(string $query): string
{
    $query = mb_strtolower(trim($query), 'UTF-8');

    // Remove MySQL Boolean operators supplied by the visitor.
    $query = preg_replace('/[^\p{L}\p{N}\s-]+/u', ' ', $query) ?? '';
    $query = preg_replace('/\s+/u', ' ', $query) ?? '';

    return mb_substr(trim($query), 0, 200, 'UTF-8');
}

function queryTerms(string $query): array
{
    if ($query === '') {
        return [];
    }

    $parts = preg_split('/\s+/u', $query) ?: [];
    $terms = [];

    foreach ($parts as $part) {
        $part = trim($part, '-');

        if (mb_strlen($part, 'UTF-8') < 2) {
            continue;
        }

        $terms[] = mb_substr($part, 0, 40, 'UTF-8');

        if (count($terms) === 8) {
            break;
        }
    }

    return array_values(array_unique($terms));
}

function buildBooleanQuery(array $terms): string
{
    $equivalents = [
        'js' => ['javascript'],
        'javascript' => ['js'],
        'db' => ['database'],
        'database' => ['db'],
        'secure' => ['security'],
        'security' => ['secure'],
    ];

    $relatedTerms = [
        'mysql' => ['database', 'sql'],
        'php' => ['backend'],
    ];

    $parts = [];

    foreach ($terms as $term) {
        $alternatives = [$term . '*'];

        foreach ($equivalents[$term] ?? [] as $equivalent) {
            $alternatives[] = $equivalent . '*';
        }

        // Require the original word or an approved equivalent.
        $parts[] = count($alternatives) === 1
            ? '+' . $alternatives[0]
            : '+(' . implode(' ', $alternatives) . ')';

        // Related words are optional and can improve a result's score.
        foreach ($relatedTerms[$term] ?? [] as $relatedTerm) {
            $parts[] = $relatedTerm . '*';
        }
    }

    return implode(' ', array_unique($parts));
}

function highlightTerms(string $text, array $terms): string
{
    if ($terms === []) {
        return escapeHtml($text);
    }

    usort(
        $terms,
        static fn (string $a, string $b): int =>
            mb_strlen($b, 'UTF-8') <=> mb_strlen($a, 'UTF-8')
    );

    $escapedTerms = array_map(
        static fn (string $term): string => preg_quote($term, '/'),
        $terms
    );

    $pattern = '/(' . implode('|', $escapedTerms) . ')/iu';
    $parts = preg_split(
        $pattern,
        $text,
        -1,
        PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY
    );

    if ($parts === false) {
        return escapeHtml($text);
    }

    $output = '';

    foreach ($parts as $part) {
        if (preg_match($pattern, $part) === 1) {
            $output .= '<mark>' . escapeHtml($part) . '</mark>';
        } else {
            $output .= escapeHtml($part);
        }
    }

    return $output;
}

function pageUrl(string $query, int $categoryId, int $page): string
{
    return '?' . http_build_query([
        'q' => $query,
        'category' => $categoryId ?: null,
        'page' => $page,
    ]);
}

Why remove Boolean operators?

MySQL Boolean search assigns meaning to characters such as +, -, *, ", parentheses and @. Passing raw visitor input into Boolean mode would let visitors change the structure of the search expression or produce invalid queries.

The application therefore keeps only letters, numbers, whitespace and ordinary hyphens. It creates the operators itself after validation.

Why limit the query?

Search endpoints are public and easy to automate. Limiting the input to 200 characters and eight useful terms prevents needlessly expensive expressions. A production site should also apply a reasonable request-rate limit at the application, reverse-proxy or CDN layer.

Why append an asterisk?

In MySQL Boolean full-text search, the trailing * is a prefix operator. A search for migrat* can match migrate, migration and migrations.

The MySQL manual also notes that a wildcarded term is not removed merely because it is shorter than the configured minimum token length or is a stopword. This makes prefix mode helpful, but it should still be tested with the vocabulary used by the real website.

Step 4: Build the Complete Search Page

Create search.php:

<?php

declare(strict_types=1);

require __DIR__ . '/config.php';
require __DIR__ . '/functions.php';

$query = normaliseQuery((string) ($_GET['q'] ?? ''));
$categoryId = filter_input(
    INPUT_GET,
    'category',
    FILTER_VALIDATE_INT,
    ['options' => ['min_range' => 1]]
) ?: 0;
$page = filter_input(
    INPUT_GET,
    'page',
    FILTER_VALIDATE_INT,
    ['options' => ['min_range' => 1]]
) ?: 1;

$perPage = 10;
$results = [];
$total = 0;
$usedFallback = false;
$terms = queryTerms($query);
$canSearch = mb_strlen($query, 'UTF-8') >= 2;

$categoryStatement = $pdo->query(
    'SELECT id, name FROM categories ORDER BY name'
);
$categories = $categoryStatement->fetchAll();

if ($canSearch && $terms !== []) {
    $booleanQuery = buildBooleanQuery($terms);
    $offset = ($page - 1) * $perPage;

    $categoryWhere = $categoryId > 0
        ? ' AND a.category_id = :category_id'
        : '';

    $countSql = <<<SQL
        SELECT COUNT(*)
        FROM articles AS a
        WHERE a.status = 'published'
          AND a.published_at <= NOW()
          AND MATCH(a.title, a.excerpt, a.body)
              AGAINST(:count_boolean IN BOOLEAN MODE)
        {$categoryWhere}
    SQL;

    $countStatement = $pdo->prepare($countSql);
    $countStatement->bindValue(':count_boolean', $booleanQuery);

    if ($categoryId > 0) {
        $countStatement->bindValue(
            ':category_id',
            $categoryId,
            PDO::PARAM_INT
        );
    }

    $countStatement->execute();
    $total = (int) $countStatement->fetchColumn();

    if ($total > 0) {
        $searchSql = <<<SQL
            SELECT
                a.id,
                a.title,
                a.slug,
                a.excerpt,
                a.published_at,
                c.name AS category_name,
                (
                    MATCH(a.title)
                        AGAINST(:rank_title IN NATURAL LANGUAGE MODE) * 4
                    + MATCH(a.title, a.excerpt, a.body)
                        AGAINST(:rank_all IN NATURAL LANGUAGE MODE)
                    + MATCH(a.title, a.excerpt, a.body)
                        AGAINST(:rank_boolean IN BOOLEAN MODE) * 0.5
                    + CASE WHEN a.title = :exact_title THEN 8 ELSE 0 END
                    + CASE WHEN a.title LIKE :title_prefix THEN 4 ELSE 0 END
                ) AS score
            FROM articles AS a
            INNER JOIN categories AS c ON c.id = a.category_id
            WHERE a.status = 'published'
              AND a.published_at <= NOW()
              AND MATCH(a.title, a.excerpt, a.body)
                  AGAINST(:filter_boolean IN BOOLEAN MODE)
              {$categoryWhere}
            ORDER BY score DESC, a.published_at DESC
            LIMIT :limit OFFSET :offset
        SQL;

        $statement = $pdo->prepare($searchSql);
        $statement->bindValue(':rank_title', $query);
        $statement->bindValue(':rank_all', $query);
        $statement->bindValue(':rank_boolean', $booleanQuery);
        $statement->bindValue(':exact_title', $query);
        $statement->bindValue(':title_prefix', $query . '%');
        $statement->bindValue(':filter_boolean', $booleanQuery);

        if ($categoryId > 0) {
            $statement->bindValue(
                ':category_id',
                $categoryId,
                PDO::PARAM_INT
            );
        }

        $statement->bindValue(':limit', $perPage, PDO::PARAM_INT);
        $statement->bindValue(':offset', $offset, PDO::PARAM_INT);
        $statement->execute();
        $results = $statement->fetchAll();
    }
}

// Controlled fallback for short terms or words absent from the FULLTEXT index.
if ($canSearch && $results === [] && $page === 1) {
    $usedFallback = true;
    $fallbackCategory = $categoryId > 0
        ? ' AND a.category_id = :fallback_category'
        : '';

    $fallbackSql = <<<SQL
        SELECT
            a.id,
            a.title,
            a.slug,
            a.excerpt,
            a.published_at,
            c.name AS category_name,
            CASE
                WHEN a.title = :fallback_exact THEN 10
                WHEN a.title LIKE :fallback_prefix THEN 5
                ELSE 1
            END AS score
        FROM articles AS a
        INNER JOIN categories AS c ON c.id = a.category_id
        WHERE a.status = 'published'
          AND a.published_at <= NOW()
          AND (
              a.title LIKE :fallback_title
              OR a.excerpt LIKE :fallback_excerpt
          )
          {$fallbackCategory}
        ORDER BY score DESC, a.published_at DESC
        LIMIT 10
    SQL;

    $fallback = $pdo->prepare($fallbackSql);
    $fallback->bindValue(':fallback_exact', $query);
    $fallback->bindValue(':fallback_prefix', $query . '%');
    $fallback->bindValue(':fallback_title', '%' . $query . '%');
    $fallback->bindValue(':fallback_excerpt', '%' . $query . '%');

    if ($categoryId > 0) {
        $fallback->bindValue(
            ':fallback_category',
            $categoryId,
            PDO::PARAM_INT
        );
    }

    $fallback->execute();
    $results = $fallback->fetchAll();
    $total = count($results);
}

$totalPages = $usedFallback
    ? 1
    : max(1, (int) ceil($total / $perPage));

if ($canSearch && $page === 1) {
    $log = $pdo->prepare(
        'INSERT INTO search_queries (query, results_count)
         VALUES (:query, :results_count)'
    );
    $log->bindValue(':query', $query);
    $log->bindValue(':results_count', $total, PDO::PARAM_INT);
    $log->execute();
}
?>
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Smart PHP Search</title>
    <style>
        * { box-sizing: border-box; }
        body {
            margin: 0;
            color: #1f2937;
            background: #f8fafc;
            font-family: Arial, sans-serif;
            line-height: 1.6;
        }
        main {
            width: min(900px, calc(100% - 32px));
            margin: 48px auto;
        }
        form, article {
            padding: 24px;
            margin-bottom: 18px;
            background: #fff;
            border: 1px solid #e5e7eb;
            border-radius: 12px;
        }
        .search-row {
            display: grid;
            grid-template-columns: 1fr 180px auto;
            gap: 10px;
        }
        input, select, button {
            min-height: 44px;
            padding: 10px 12px;
            font: inherit;
        }
        button {
            color: #fff;
            background: #2563eb;
            border: 0;
            border-radius: 6px;
            cursor: pointer;
        }
        h1 { line-height: 1.2; }
        h2 { margin: 0 0 8px; line-height: 1.3; }
        h2 a { color: #1d4ed8; text-decoration: none; }
        .meta { color: #64748b; font-size: 0.9rem; }
        mark { background: #fef08a; padding: 0 2px; }
        .pagination { display: flex; flex-wrap: wrap; gap: 8px; }
        .pagination a, .pagination strong {
            padding: 7px 11px;
            background: #fff;
            border: 1px solid #cbd5e1;
            border-radius: 6px;
            text-decoration: none;
        }
        @media (max-width: 650px) {
            .search-row { grid-template-columns: 1fr; }
        }
    </style>
</head>
<body>
<main>
    <h1>Search the Article Library</h1>

    <form method="get" action="">
        <label for="q">What are you looking for?</label>
        <div class="search-row">
            <input
                id="q"
                name="q"
                type="search"
                value="<?= escapeHtml($query) ?>"
                minlength="2"
                maxlength="200"
                required
            >

            <select name="category" aria-label="Category">
                <option value="0">All categories</option>
                <?php foreach ($categories as $category): ?>
                    <option
                        value="<?= (int) $category['id'] ?>"
                        <?= $categoryId === (int) $category['id']
                            ? 'selected'
                            : '' ?>
                    >
                        <?= escapeHtml($category['name']) ?>
                    </option>
                <?php endforeach; ?>
            </select>

            <button type="submit">Search</button>
        </div>
    </form>

    <?php if ($query !== ''): ?>
        <p>
            <?= number_format($total) ?> result<?= $total === 1 ? '' : 's' ?>
            for <strong><?= escapeHtml($query) ?></strong>
        </p>

        <?php if ($usedFallback && $results !== []): ?>
            <p>Showing broader title and excerpt matches.</p>
        <?php endif; ?>

        <?php if ($results === []): ?>
            <article>
                <h2>No results found</h2>
                <p>Try fewer words, remove the category filter or check the spelling.</p>
            </article>
        <?php endif; ?>

        <?php foreach ($results as $result): ?>
            <article>
                <h2>
                    <a href="/articles/<?= escapeHtml($result['slug']) ?>">
                        <?= highlightTerms($result['title'], $terms) ?>
                    </a>
                </h2>

                <p class="meta">
                    <?= escapeHtml($result['category_name']) ?> ·
                    <?= escapeHtml(date(
                        'j M Y',
                        strtotime($result['published_at'])
                    )) ?>
                </p>

                <p><?= highlightTerms($result['excerpt'], $terms) ?></p>
            </article>
        <?php endforeach; ?>

        <?php if ($totalPages > 1): ?>
            <nav class="pagination" aria-label="Search-result pages">
                <?php for ($number = 1; $number <= $totalPages; $number++): ?>
                    <?php if ($number === $page): ?>
                        <strong aria-current="page"><?= $number ?></strong>
                    <?php else: ?>
                        <a href="<?= escapeHtml(pageUrl(
                            $query,
                            $categoryId,
                            $number
                        )) ?>">
                            <?= $number ?>
                        </a>
                    <?php endif; ?>
                <?php endfor; ?>
            </nav>
        <?php endif; ?>
    <?php endif; ?>
</main>
</body>
</html>

The same named placeholder is not reused for different positions. This keeps the query compatible with native PDO prepared statements.

The LIMIT and OFFSET values are bound explicitly as integers. They are calculated by the application rather than accepted as raw SQL text.

Step 5: Run the Project

Set the database credentials in the terminal:

export SEARCH_DB_HOST='127.0.0.1'
export SEARCH_DB_NAME='smart_search'
export SEARCH_DB_USER='search_user'
export SEARCH_DB_PASS='replace-with-a-strong-password'

Start PHP’s development server from the project folder:

php -S 127.0.0.1:8000

Open:

http://127.0.0.1:8000/search.php

Try these searches:

Search	Expected behaviour
`php`	Finds the PHP tutorials
`php form`	Requires both prefixes and ranks the form tutorial highly
`mysql index`	Finds the database indexing article
`inject`	Prefix matching finds `injection`
`db`	Treats `database` as an approved equivalent
`pdo`	Finds the security tutorial
`api` with PHP filter	Limits the results to the PHP category

The built-in server is for local development. Use Apache, Nginx or another properly configured production server for a public site.

Understanding the Ranking Formula

The result score combines five signals:

MATCH(title) AGAINST(:query) * 4
+ MATCH(title, excerpt, body) AGAINST(:query)
+ MATCH(title, excerpt, body) AGAINST(:boolean_query) * 0.5
+ exact_title_bonus
+ title_prefix_bonus

Signal	Purpose
Title relevance × 4	A title match is normally more intentional than a body mention
Overall relevance	Measures the query against all searchable content
Boolean relevance × 0.5	Gives a smaller boost to approved equivalent and related terms
Exact title + 8	Rewards a query that equals the complete title
Title prefix + 4	Rewards titles beginning with the complete query

MySQL describes its InnoDB full-text relevance as a variation of TF-IDF: words contribute more when they occur in a document but are less common across the complete collection. The score is useful for ordering results, but its raw numeric value should not be shown as a percentage. A score of 3.2 does not mean “32% relevant.”

Ranking weights are business rules. Test them with real queries and adjust them gradually. Do not change several weights simultaneously, because that makes improvements difficult to measure.

Adding “Did You Mean?” Suggestions

Full-text prefix matching helps with incomplete words, but it does not automatically correct a misspelling such as databse.

PHP’s levenshtein() calculates the minimum insertions, replacements and deletions required to transform one string into another. It is useful for comparing a short query term with a small, controlled vocabulary.

Add this function to functions.php:

function spellingSuggestion(PDO $pdo, string $term): ?string
{
    if (mb_strlen($term, 'UTF-8') < 4 || strlen($term) > 40) {
        return null;
    }

    $statement = $pdo->query(
        'SELECT term
         FROM search_terms
         ORDER BY popularity DESC
         LIMIT 500'
    );

    $bestTerm = null;
    $bestDistance = PHP_INT_MAX;

    foreach ($statement->fetchAll(PDO::FETCH_COLUMN) as $candidate) {
        $distance = levenshtein($term, (string) $candidate);

        if ($distance < $bestDistance) {
            $bestDistance = $distance;
            $bestTerm = (string) $candidate;
        }
    }

    $maximumDistance = mb_strlen($term, 'UTF-8') <= 6 ? 1 : 2;

    if ($bestTerm === $term || $bestDistance > $maximumDistance) {
        return null;
    }

    return $bestTerm;
}

When there are no results and the query contains one word:

$suggestion = null;

if ($results === [] && count($terms) === 1) {
    $suggestion = spellingSuggestion($pdo, $terms[0]);
}

Display it as a link:

<?php if ($suggestion !== null): ?>
    <p>
        Did you mean
        <a href="<?= escapeHtml(pageUrl($suggestion, $categoryId, 1)) ?>">
            <?= escapeHtml($suggestion) ?>
        </a>?
    </p>
<?php endif; ?>

Do not compare every query with every word in a large database. Levenshtein comparison has computational cost. Use a limited vocabulary based on popular tags, approved keywords and successful searches.

For multilingual text, PHP’s standard levenshtein() compares bytes rather than providing language-aware linguistic correction. A dedicated search engine becomes more appropriate when high-quality fuzzy matching across several languages is essential.

Improving Synonyms

The tutorial keeps synonyms in a PHP array so the behaviour is easy to understand. A real website may move them to a table:

CREATE TABLE search_synonyms (
    term VARCHAR(100) NOT NULL,
    synonym VARCHAR(100) NOT NULL,
    PRIMARY KEY (term, synonym)
) ENGINE=InnoDB;

Possible pairs include:

Visitor term	Optional related term
`js`	`javascript`
`db`	`database`
`air con`	`air conditioner`
`vacuum mop`	`wet dry vacuum`
`yen converter`	`currency converter`

True equivalents can form a required alternatives group. For example, +(js* javascript*) means that a result must contain either form. Broader related terms should remain optional. If a visitor searches mysql, requiring every result to contain database as well could incorrectly remove a relevant page.

Review synonym pairs manually. Two words that are related are not always interchangeable.

Safe Highlighting

Search highlighting creates an easy place to introduce cross-site scripting.

This is unsafe:

echo str_ireplace($query, '<mark>' . $query . '</mark>', $result['title']);

It mixes HTML markup with unescaped visitor input and database content.

The highlightTerms() helper splits the original text, escapes every piece with htmlspecialchars() and introduces only the application’s own <mark> tags. PHP documents that htmlspecialchars() converts characters such as &, quotes, < and > into HTML entities.

Output escaping is still required even if the database content was entered through an administrator-only form. Stored data should not be assumed safe for every output context.

Common Full-Text Search Problems

1. A Word Produces No Results

The word may:

Be shorter than the configured minimum token size
Appear in the stopword list
Be absent when the index was built
Use a language that needs different tokenisation
Be searched against columns that do not match the index

Check the InnoDB token size:

SHOW VARIABLES LIKE 'innodb_ft_min_token_size';

MySQL documents that changing the minimum or maximum word length requires rebuilding the affected full-text indexes. Do not change the server setting only to solve one unusual query; test the effect on index size and relevance first.

2. MATCH Columns Do Not Match the Index

This query:

MATCH(title, body) AGAINST('php')

cannot simply use an index created for a different column combination:

FULLTEXT(title, excerpt, body)

Create the index that matches the MATCH() list or make the query use the indexed list.

3. Boolean Results Are Not Properly Sorted

Boolean mode does not promise automatic descending relevance order. Calculate a score and use an explicit ORDER BY score DESC.

4. Every Word Is Required

The sample uses +word* for each original term. This gives precise results but may be too strict for a large query.

One alternative is:

Require the first two meaningful terms.
Make later terms optional.
Remove common filler words in PHP.
Retry using optional terms only when the strict search returns nothing.

Make fallback behaviour visible to the visitor instead of silently returning unrelated content.

5. Pagination Becomes Slow on Deep Pages

LIMIT 10 OFFSET 100000 makes MySQL find and skip many earlier rows. Standard offset pagination is acceptable for ordinary website search, where users rarely browse hundreds of pages.

For very deep result sets, use cursor-based pagination with a stable combination such as score, publication date and ID. Because calculated floating-point scores can tie or change as content changes, cursor search needs more careful design than a normal article list.

6. Search Finds Draft Content

Always apply visibility rules inside the SQL query:

WHERE status = 'published'
  AND published_at <= NOW()

Do not fetch hidden rows and remove them later in PHP. The database query should never return content that the visitor is not authorised to see.

Measuring Search Quality

The search_queries table records the normalised query, result count and time. It deliberately does not store an IP address, user-agent string or visitor identity.

Useful weekly reports include:

Searches Returning No Results

SELECT query, COUNT(*) AS searches
FROM search_queries
WHERE results_count = 0
  AND searched_at >= NOW() - INTERVAL 30 DAY
GROUP BY query
ORDER BY searches DESC
LIMIT 50;

These queries can reveal:

Missing articles
Missing product names
Common misspellings
Needed synonyms
Terms excluded by full-text settings

Most Popular Searches

SELECT query, COUNT(*) AS searches
FROM search_queries
WHERE searched_at >= NOW() - INTERVAL 30 DAY
GROUP BY query
ORDER BY searches DESC
LIMIT 50;

Popular searches can inform navigation, internal links and future content.

Better Success Metrics

A non-zero result count does not prove that the results were useful. If privacy requirements allow, measure aggregate events such as:

Search-result click-through rate
Percentage of searches followed by another query
Queries that lead to a conversion or completed task
Position of the result that visitors select

Avoid collecting more personal data than the site genuinely needs. Define a retention period for raw search logs and aggregate older records.

Performance Checklist

Use this checklist before publishing:

Confirm that the table uses InnoDB.
Create a FULLTEXT index matching every MATCH() column list.
Keep the normal status, category and publication-date index.
Limit query length and term count.
Limit results per page.
Bind values through PDO prepared statements.
Avoid searching a large body column with %term% except as a limited fallback.
Use EXPLAIN to inspect the main query.
Apply server-side rate limiting.
Cache repeated anonymous searches if the content changes infrequently.
Re-test results after changing token sizes, stopwords or ranking weights.

Inspect the query plan with:

EXPLAIN
SELECT id, title
FROM articles
WHERE MATCH(title, excerpt, body)
      AGAINST('+php* +security*' IN BOOLEAN MODE);

Measure with realistic data. A table containing four sample articles cannot reveal how the query will behave with 100,000 articles.

Security Checklist

A search page accepts public input, so treat it like any other exposed endpoint.

Use PDO placeholders for every visitor-supplied value.
Construct SQL identifiers and sorting rules from a fixed application allowlist.
Remove Boolean operators from raw query text.
Escape query text, titles, excerpts, categories, slugs and URLs for their output context.
Set a maximum query length and maximum term count.
Bind LIMIT and OFFSET as integers.
Do not display database exception messages in production.
Apply rate limits and request-size limits.
Search only rows the visitor is allowed to access.
Use a restricted database user.
Keep database credentials outside the repository and web root.
Add sensible HTTP security headers.

Prepared statements protect bound values from SQL injection. They do not make dynamically concatenated column names, directions or SQL fragments safe.

When MySQL Search Is Enough

MySQL full-text search is a good starting choice when:

The content already lives in MySQL.
The site has a modest or medium-sized collection.
Search traffic is not extreme.
Basic relevance, filters and prefix matching are sufficient.
The team wants minimal infrastructure.
Immediate consistency is valuable.

It is especially suitable for a blog or internal knowledge base that has outgrown LIKE but does not yet need a separate search cluster.

When to Use a Dedicated Search Engine

Consider a dedicated engine when the project requires several of these features:

High-quality typo tolerance on every query
Fast autocomplete and search-as-you-type
Complex faceted navigation
Language-specific stemming and tokenisation
Geographical or vector search
Millions of records with heavy search traffic
Advanced relevance tuning and experimentation
Search across several databases and services
Independent scaling of indexing and querying

The trade-off is operational complexity. A separate engine creates another copy of searchable data, so the application must handle indexing, updates, deletions, retries, monitoring and recovery from synchronisation failures.

Start with the smallest system that meets the actual requirements. Build search analytics early so a future migration is based on evidence rather than guesswork.

Testing the Search Engine

Test more than happy-path queries.

Test	Expected outcome
Empty query	No database search is executed
One-character query	Rejected by the form or ignored by term parsing
Query longer than 200 characters	Truncated server-side and limited client-side
`php' OR 1=1 --`	Treated as search text, not SQL
`<script>alert(1)</script>`	Displayed only as escaped text
`++++php****`	Normalised before the Boolean expression is built
Invalid category ID	Treated as no category or returns no matching category
Page `-5`	Replaced with page 1
Page beyond the final page	Returns no rows without exposing an error
Draft article containing the query	Never shown
Future scheduled article	Never shown before `published_at`
Two identical relevance scores	Newer publication date breaks the tie
Misspelled popular term	Offers a cautious suggestion

Also create a small relevance test set:

Query: php validation
Expected top result: PHP Form Validation for Beginners

Query: sql injection
Expected top result: Preventing SQL Injection with PDO

Query: mysql index
Expected top result: Understanding MySQL Indexes

Run these checks whenever ranking logic, content structure, stopwords or indexes change.

Frequently Asked Questions

Can PHP build a search engine without Elasticsearch?

Yes. PHP and MySQL full-text search can provide useful relevance ranking, Boolean operators, prefix matching, filters and pagination for many websites. A dedicated search platform becomes useful when the requirements exceed what the database can deliver comfortably.

Is FULLTEXT faster than LIKE?

For word-based searches across a growing text collection, a suitable FULLTEXT index is normally much more appropriate than scanning large columns with LIKE '%term%'. Actual performance depends on the data, query and server, so verify it with EXPLAIN and realistic load tests.

Does MySQL FULLTEXT support partial words?

Boolean mode supports a trailing * prefix operator. For example, migrat* can match words beginning with migrat. It is prefix matching, not arbitrary substring matching.

Can visitors use quotes and minus signs?

MySQL supports advanced Boolean operators, but a public application should not pass them through automatically. The tutorial strips them and creates a controlled expression. If advanced search syntax is a product requirement, parse and validate each supported operator explicitly.

Why use both natural-language and Boolean modes?

Boolean mode filters the candidates and enables required prefixes. Natural-language mode supplies an intuitive relevance contribution for ranking those candidates. The application then adds its own title bonuses.

Can I search HTML stored in the body column?

It is better to index searchable plain text. HTML tags, menus, shortcodes and repeated template text add noise. Store or generate a clean text version when the source content contains heavy markup.

How can I search WordPress posts?

The same principles apply, but avoid altering WordPress core tables casually. A WordPress plugin can use the query APIs, maintain its own search index table, or integrate with an established search plugin. Measure the existing search first before replacing it.

Will Levenshtein correct every spelling mistake?

No. It measures edit distance, not meaning or pronunciation. Restrict suggestions to a curated vocabulary and a small maximum distance. Do not automatically redirect every misspelling.

Should the search query be stored?

Aggregate query logs are valuable for discovering missing content and poor results. Store only what is necessary, avoid unnecessary personal identifiers, limit access and define a retention period.

Can this work for Malay-language content?

It can search Malay words stored with utf8mb4, but relevance, stopwords, prefixes and synonyms should be tested using real Malay queries. A dedicated engine may be better if the site needs language-aware stemming, typo tolerance or mixed-language tokenisation.

Does a higher relevance score mean the result is correct?

No. It means the row scored higher under the current formula. Search quality must be checked with expected-result tests and real visitor behaviour.

Final Summary

A smart website search does not need to begin with a large search cluster or an AI model.

The simplified PHP approach is:

Store clean searchable content in MySQL.
Add full-text indexes for the title and complete document.
Normalise and limit every visitor query.
Build a controlled Boolean prefix expression.
Use MySQL to filter and score the candidates.
Give title and exact-query matches extra weight.
Add filters, pagination, synonyms and a controlled fallback.
Escape every displayed value and highlight matches safely.
Use a limited vocabulary for spelling suggestions.
Measure zero-result queries and improve the system using evidence.

This design is simple enough for a beginner to understand, but it includes the security and relevance rules needed for a real project. Begin with the working version, add realistic content, record expected top results and tune one ranking rule at a time.

August 7, 2026

Advanced Use Cases: Conditional Migrations in Laravel

Laravel migrations are normally deterministic: every environment receives the same migration files, Laravel runs each pending migration once, and the migrations table records what has completed.

Real production systems are not always that tidy.

You may need to introduce a table only when an optional module is enabled, repair databases created before migrations were adopted, use different SQL for PostgreSQL and MySQL, migrate hundreds of tenant databases, or delay a constraint until a large backfill has finished.

These situations lead developers towards conditional migrations.

Conditional migrations can be useful, but they can also create a more serious problem than the one they solve: two environments running the same application version with different and undocumented schemas.

This guide explains the advanced Laravel patterns that make conditional migrations safer, where Laravel 13’s shouldRun() method fits, and when a condition should be replaced by a staged deployment instead.

The examples use current Laravel 13 syntax. Most schema-guard and connection patterns also work in earlier Laravel versions, but first-class migration skipping and some schema inspection methods depend on the framework version installed in your project.

Quick Answer

A conditional migration is a migration whose behaviour depends on a known condition such as:

Whether a feature or module is enabled
Whether a legacy table or column already exists
Which database driver is active
Which database connection is being migrated
Whether a prerequisite migration or backfill has completed
Which tenant database is currently selected

Use the narrowest mechanism that matches the requirement:

Requirement	Recommended Laravel mechanism
Do not run this pending migration yet	Define `shouldRun(): bool`
Add a column only if it is missing	`Schema::whenTableDoesntHaveColumn()` or an explicit `Schema::hasColumn()` guard
Repair known legacy schema differences	Inspect the schema, validate the state, then reconcile it
Use vendor-specific SQL	Check `DB::connection()->getDriverName()` and keep each branch tested
Modify a non-default database	Set the migration `$connection` or use `Schema::connection()`
Migrate separate tenant databases	Run the tenant migration path once per tenant connection
Change a large table safely	Use expand, backfill, verify and contract across several deployments
Wait for data cleanup before adding a constraint	Fail clearly when the prerequisite is unmet; do not silently skip
Prevent two deployment nodes migrating together	Run `php artisan migrate --isolated --force` with a shared lock-capable cache

The most important rule is:

A migration condition should represent stable deployment state, not random request-time behaviour.

Do not base schema creation on the current user, a percentage rollout, a temporary API response, the time of day, or whichever application server happens to run the command.

What Is a Conditional Migration?

A normal migration always performs the same change:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::table('users', function (Blueprint $table) {
            $table->string('timezone')->nullable();
        });
    }

    public function down(): void
    {
        Schema::table('users', function (Blueprint $table) {
            $table->dropColumn('timezone');
        });
    }
};

A conditional migration checks something before deciding whether or how to make the change:

public function up(): void
{
    if (! Schema::hasColumn('users', 'timezone')) {
        Schema::table('users', function (Blueprint $table) {
            $table->string('timezone')->nullable();
        });
    }
}

The second example looks safer because it will not fail when the column already exists. However, it changes the meaning of the migration.

If the column exists with the wrong type, length, collation or nullability, the guard silently accepts it. The application may then fail later even though Laravel reports that every migration has run.

Conditional logic should therefore answer two questions:

Is this an expected alternative state?
If it exists, is it compatible with what the application requires?

Merely checking that an object exists is not always enough.

Three Different Meanings of “Skip”

Many migration bugs happen because three different behaviours are all described as “skipping the migration.”

1. Skip the Migration Before `up()` Runs

Current Laravel migrations may define shouldRun():

public function shouldRun(): bool
{
    return (bool) config('features.audit_log_schema');
}

When the method returns false, Laravel skips that migration. It is not treated like a completed schema change, so it can be considered again during a later migration run when the condition becomes true.

Laravel documents shouldRun() under Skipping Migrations.

This behaviour is suitable when the migration genuinely should remain pending.

2. Enter `up()` but Return Without Changing Anything

An older and still useful pattern is a guard inside up():

public function up(): void
{
    if (Schema::hasColumn('users', 'timezone')) {
        return;
    }

    Schema::table('users', function (Blueprint $table) {
        $table->string('timezone')->nullable();
    });
}

Laravel called the migration successfully, so it records the migration as completed even when the method returned before issuing DDL.

This is useful for a reconciliation migration: an imported database may already contain the required column, and the migration exists to bring every database to one recorded baseline.

It is not appropriate when you want Laravel to try again later.

3. Run the Migration but Choose a Different Branch

Sometimes every environment must complete the migration, but the implementation differs:

$driver = DB::connection($this->getConnection())->getDriverName();

match ($driver) {
    'pgsql' => $this->createPostgresIndex(),
    'mysql', 'mariadb' => $this->createMySqlIndex(),
    default => throw new RuntimeException(
        "Unsupported database driver: {$driver}"
    ),
};

This is not really a skipped migration. It is a portable migration with explicit database-specific implementations.

The distinction matters when diagnosing migrate:status, rolling back, adding later migrations and comparing environments.

Use Case 1: Feature-Gated Migrations with `shouldRun()`

Suppose an audit-log module is present in the codebase but will be enabled during a later release window.

Create a stable configuration value:

// config/features.php

return [
    'audit_log_schema' => env('AUDIT_LOG_SCHEMA_ENABLED', false),
];

The migration can define shouldRun():

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function shouldRun(): bool
    {
        return (bool) config('features.audit_log_schema');
    }

    public function up(): void
    {
        Schema::create('audit_events', function (Blueprint $table) {
            $table->id();
            $table->foreignId('user_id')->nullable()->index();
            $table->string('event');
            $table->string('auditable_type')->nullable();
            $table->unsignedBigInteger('auditable_id')->nullable();
            $table->json('metadata')->nullable();
            $table->ipAddress('ip_address')->nullable();
            $table->timestamps();

            $table->index(
                ['auditable_type', 'auditable_id'],
                'audit_events_auditable_index'
            );
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('audit_events');
    }
};

When This Is Appropriate

shouldRun() is useful when:

A module is deployed before it is activated.
A customer-specific installation includes an optional component.
A phased infrastructure change requires a schema migration at a controlled time.
A Laravel package publishes migrations that only apply when its integration is enabled.

The Flag Must Be Stable and Global

A database schema normally applies to the complete application, not to one HTTP request or one user.

Avoid this type of condition:

public function shouldRun(): bool
{
    return Feature::active('new-checkout');
}

if new-checkout is a percentage-based or user-scoped feature flag. The result could depend on the current scope, cached resolution or machine running the migration.

A schema gate should be one of the following:

A deployment configuration deliberately set for the whole environment
A globally scoped, persistently stored feature state
A release manifest or module registry shared by every deployment node
An explicit command option processed by a controlled migration runner

Do not use a random rollout calculation for DDL.

Configuration Cache Matters

Production Laravel applications commonly cache configuration. Code should read the condition through config(), not call env() directly inside the migration.

After changing the environment variable, rebuild the application’s configuration cache before expecting shouldRun() to see the new value:

php artisan config:cache
php artisan migrate --isolated --force

Also verify that every deployment node has the same configuration. A migration should not run on one node merely because its release directory contains a different cached configuration file.

Use Case 2: Add a Column Only When It Is Missing

Laravel’s schema builder can inspect tables, columns and indexes.

Current schema builder methods include:

Method	Purpose
`Schema::hasTable('users')`	Check whether a table exists
`Schema::hasColumn('users', 'timezone')`	Check one column
`Schema::hasColumns('users', ['timezone', 'locale'])`	Check several columns
`Schema::hasIndex('users', ['email'], 'unique')`	Check an index by columns and optional type
`Schema::getColumnType('users', 'timezone')`	Read a column type
`Schema::getColumns('users')`	Inspect column metadata
`Schema::getIndexes('users')`	Inspect index metadata
`Schema::getForeignKeys('posts')`	Inspect foreign-key metadata

Laravel 13’s API also exposes conditional helpers such as whenTableHasColumn, whenTableDoesntHaveColumn, whenTableHasIndex and whenTableDoesntHaveIndex. See the current Schema Builder API.

Concise Conditional Helper

public function up(): void
{
    Schema::whenTableDoesntHaveColumn(
        'users',
        'timezone',
        function (Blueprint $table) {
            $table->string('timezone', 64)->nullable();
        }
    );
}

This is concise and expressive when existence is the only compatibility requirement.

Explicit Guard with a Clear Failure

For advanced migrations, an explicit check is often easier to audit:

public function up(): void
{
    if (! Schema::hasTable('users')) {
        throw new RuntimeException(
            'Cannot add users.timezone because the users table is missing.'
        );
    }

    if (Schema::hasColumn('users', 'timezone')) {
        $type = Schema::getColumnType('users', 'timezone');

        if ($type !== 'varchar') {
            throw new RuntimeException(
                "Expected users.timezone to be varchar; found {$type}."
            );
        }

        return;
    }

    Schema::table('users', function (Blueprint $table) {
        $table->string('timezone', 64)->nullable();
    });
}

The exact type names returned by schema inspection can vary by database platform and framework version. Test type assertions against every supported driver instead of assuming MySQL and PostgreSQL return identical strings.

Guard the Rollback Too

If the up() method tolerates an already-existing column, the down() method requires careful thought.

This rollback is dangerous:

public function down(): void
{
    Schema::table('users', function (Blueprint $table) {
        $table->dropColumn('timezone');
    });
}

The migration may have found a column created manually years earlier. Rolling back would delete a column the migration did not create.

For a legacy reconciliation migration, an intentionally irreversible rollback may be safer:

public function down(): void
{
    throw new RuntimeException(
        'This reconciliation migration cannot be reversed safely.'
    );
}

Another option is to create a separate, explicit removal migration after confirming ownership and taking a backup.

Do not pretend every production schema change has a safe automatic reverse operation.

Use Case 3: Repair Inconsistent Legacy Databases

Conditional migrations are often introduced when an application existed before its schema was fully managed by Laravel.

For example:

Production has a manually created customer_code column.
Staging has the same column but with a different length.
New developer databases do not have the column.
One customer’s self-hosted installation uses a different index name.

The wrong approach is to add if (! Schema::hasColumn(...)) everywhere until migrations stop failing. That hides drift instead of repairing it.

Use a reconciliation migration with an explicit state table:

Observed state	Action
Table missing	Stop; a prerequisite or database selection is wrong
Column missing	Add the expected column
Column exists and is compatible	Record migration as complete without changing it
Column exists but is incompatible	Stop with a diagnostic error or convert it through a reviewed branch
Unexpected duplicate index	Stop and inspect before deleting anything

Example:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
use RuntimeException;

return new class extends Migration
{
    public function up(): void
    {
        if (! Schema::hasTable('customers')) {
            throw new RuntimeException(
                'Expected the customers table before reconciling customer_code.'
            );
        }

        if (! Schema::hasColumn('customers', 'customer_code')) {
            Schema::table('customers', function (Blueprint $table) {
                $table->string('customer_code', 32)->nullable();
            });

            return;
        }

        $column = collect(Schema::getColumns('customers'))
            ->firstWhere('name', 'customer_code');

        if ($column === null) {
            throw new RuntimeException(
                'customer_code exists but its metadata could not be inspected.'
            );
        }

        if (($column['nullable'] ?? null) !== true) {
            throw new RuntimeException(
                'customers.customer_code must be nullable before this release.'
            );
        }
    }

    public function down(): void
    {
        throw new RuntimeException(
            'Legacy schema reconciliation is intentionally irreversible.'
        );
    }
};

This migration does not silently rewrite an unknown column. It makes a known missing change, accepts a known compatible state and fails on an unexpected state.

That failure is useful. It prevents the deployment from claiming success while the database remains incompatible.

Use Case 4: Database-Driver-Specific Migrations

Laravel’s schema builder is database-agnostic for common operations, but not every database offers the same features or DDL behaviour.

Common differences include:

PostgreSQL partial indexes
PostgreSQL CONCURRENTLY
MySQL generated columns and online DDL options
SQL Server filtered or online indexes
SQLite limitations during table alteration
Different JSON, enum, collation and full-text behaviour

Detect the Active Driver

use Illuminate\Support\Facades\DB;

$driver = DB::connection($this->getConnection())->getDriverName();

Use an explicit match statement and fail on unsupported databases:

public $withinTransaction = false;

public function up(): void
{
    $connection = DB::connection($this->getConnection());
    $driver = $connection->getDriverName();

    match ($driver) {
        'pgsql' => $connection->statement(
            'CREATE INDEX CONCURRENTLY IF NOT EXISTS '
            .'orders_open_created_at_index '
            .'ON orders (created_at) '
            .'WHERE completed_at IS NULL'
        ),

        'mysql', 'mariadb' => Schema::table(
            'orders',
            function (Blueprint $table) {
                if (! Schema::hasIndex(
                    'orders',
                    ['completed_at', 'created_at']
                )) {
                    $table->index(
                        ['completed_at', 'created_at'],
                        'orders_completed_created_index'
                    );
                }
            }
        ),

        default => throw new RuntimeException(
            "Unsupported database driver: {$driver}"
        ),
    };
}

Laravel’s base migration class exposes $withinTransaction, which controls whether Laravel wraps a migration in a transaction when the database grammar supports transactional schema changes. Its current API is documented in Illuminate\Database\Migrations\Migration.

PostgreSQL does not allow CREATE INDEX CONCURRENTLY inside a normal transaction, which is why that migration opts out. Once transaction wrapping is disabled, partial failure becomes more important: inspect the resulting index before retrying.

Laravel 13 Online Index Creation

Laravel 13 can express online index creation for PostgreSQL and SQL Server:

Schema::table('orders', function (Blueprint $table) {
    $table->index('created_at')->online();
});

For PostgreSQL, Laravel emits CONCURRENTLY; for SQL Server, it emits the platform’s online option. The current behaviour and supported databases are documented under Online Index Creation.

Do not add online() to a cross-database migration without testing each supported connection. MySQL online DDL has different syntax and operational restrictions.

Prefer Capability Checks Over Environment Names

This is fragile:

if (app()->environment('production')) {
    // PostgreSQL SQL
}

Production may change database vendors, and staging should normally reproduce production behaviour.

Check the capability that actually matters—the driver, server version, existing index or installed extension—and fail clearly when it is unsupported.

Use Case 5: Non-Default Database Connections

A migration can target a named connection by setting $connection:

return new class extends Migration
{
    protected $connection = 'analytics';

    public function up(): void
    {
        Schema::connection('analytics')->create(
            'daily_metrics',
            function (Blueprint $table) {
                $table->date('metric_date');
                $table->string('metric_name');
                $table->decimal('metric_value', 18, 4);

                $table->primary(['metric_date', 'metric_name']);
            }
        );
    }

    public function down(): void
    {
        Schema::connection('analytics')->dropIfExists('daily_metrics');
    }
};

Laravel documents both the migration $connection property and Schema::connection() in its migration documentation.

Avoid Cross-Database Atomicity Assumptions

A migration that updates two different database connections is not automatically one atomic transaction:

DB::connection('mysql')->table('users')->update(...);
DB::connection('analytics')->table('identities')->insert(...);

The first operation may commit while the second fails. Normal Laravel transactions do not create a distributed transaction across unrelated connections.

Prefer:

One migration path per owned database.
A resumable synchronization command or job.
A checkpoint table recording copied ranges.
Reconciliation queries comparing both sides.
An explicit cutover after verification.

Conditional connection logic should choose the correct database; it should not disguise a distributed data migration as one atomic schema change.

Use Case 6: Optional Modules and Package Migrations

A reusable application may contain optional modules such as:

Audit logging
Billing
Inventory
SAML authentication
Search analytics
Customer-specific integrations

There are two clean patterns.

Pattern A: Separate Migration Paths

Store module migrations separately:

database/migrations/
database/migrations/modules/billing/
database/migrations/modules/audit/

Run only the selected module path:

php artisan migrate \
    --path=database/migrations/modules/audit \
    --force

This is explicit in the deployment pipeline and works well when installations intentionally have different products.

Pattern B: Published Package Migrations

Laravel packages can publish migration files with publishesMigrations():

public function boot(): void
{
    $this->publishesMigrations([
        __DIR__.'/../database/migrations' => database_path('migrations'),
    ]);
}

Laravel updates published migration timestamps so they are ordered correctly. See the official Package Development documentation.

Once a package migration has been published and run, do not change that old migration in a later package version. Create a new migration just as you would in the host application.

Do Not Let Module Conditions Become Invisible

If two installations intentionally use different module schemas, record that fact outside the migration code:

Keep a module registry table.
Store the enabled module set in deployment configuration.
Include it in support diagnostics.
Test upgrades from every supported module combination.
Ensure backups and restore procedures include module-owned tables.

Without an inventory, support engineers cannot tell whether a missing table is intentional or broken.

Use Case 7: Multi-Tenant Database Migrations

Multi-tenancy requires two very different designs.

Shared Database, Shared Schema

If every tenant shares the same tables and rows are separated by tenant_id, the schema is global.

Do not conditionally add a column for only one tenant. Adding a column changes the table for every tenant even if only one tenant’s code uses it.

Use runtime feature flags, tenant configuration or a separate extension table instead.

Separate Database Per Tenant

If every tenant has a separate database, each database needs its own migration history.

A controlled Artisan command can select each database and run a dedicated tenant migration path:

<?php

namespace App\Console\Commands;

use App\Models\Tenant;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Artisan;
use Illuminate\Support\Facades\Config;
use Illuminate\Support\Facades\DB;

class MigrateTenants extends Command
{
    protected $signature = 'tenants:migrate
        {--tenant=* : Tenant IDs to migrate}
        {--force : Run without a production confirmation}';

    protected $description = 'Run pending migrations for tenant databases';

    public function handle(): int
    {
        $query = Tenant::query()->where('status', 'active');

        if ($ids = $this->option('tenant')) {
            $query->whereIn('id', $ids);
        }

        $failed = [];

        $query->orderBy('id')->each(function (Tenant $tenant) use (&$failed) {
            Config::set(
                'database.connections.tenant.database',
                $tenant->database_name
            );

            DB::purge('tenant');

            $exitCode = Artisan::call('migrate', [
                '--database' => 'tenant',
                '--path' => 'database/migrations/tenant',
                '--force' => (bool) $this->option('force'),
            ]);

            if ($exitCode !== self::SUCCESS) {
                $failed[] = $tenant->id;
            }

            $this->output->write(Artisan::output());
        });

        DB::purge('tenant');

        if ($failed !== []) {
            $this->error(
                'Failed tenant IDs: '.implode(', ', $failed)
            );

            return self::FAILURE;
        }

        return self::SUCCESS;
    }
}

This is a simplified example. A production tenant migrator should also provide:

A distributed lock so two operators cannot migrate the same tenant together
An allowlist or trusted registry for database names and connection details
Per-tenant start, finish, duration and error logs
Retry policy that does not repeat already successful migrations
A canary group before the complete tenant fleet
Batching so hundreds of tenants do not overload the database server
A report of pending migrations by tenant
A way to pause on excessive failure rate
Backups or snapshots suitable for the hosting model

Do not place one large foreach loop inside a schema migration and switch connections invisibly. Tenant orchestration is an operational command; individual tenant migrations should remain normal and auditable.

Use Case 8: Conditional Indexes and Constraints

Indexes and constraints often require stronger checks than columns.

Add an Index Only When Missing

public function up(): void
{
    Schema::whenTableDoesntHaveIndex(
        'orders',
        ['account_id', 'created_at'],
        function (Blueprint $table) {
            $table->index(
                ['account_id', 'created_at'],
                'orders_account_created_index'
            );
        }
    );
}

An explicit version is helpful when index names differ across legacy databases:

if (! Schema::hasIndex('orders', ['account_id', 'created_at'])) {
    Schema::table('orders', function (Blueprint $table) {
        $table->index(
            ['account_id', 'created_at'],
            'orders_account_created_index'
        );
    });
}

Check the columns and uniqueness type, not only a conventional name. Two indexes can have different names but provide the same column ordering.

Do Not Silently Skip a Required Constraint

Suppose orders.account_id must become non-null after a backfill.

This is unsafe:

if (! DB::table('orders')->whereNull('account_id')->exists()) {
    // Add the constraint.
}

If bad rows remain, the migration does nothing but can still be recorded as complete.

Fail clearly instead:

public function up(): void
{
    $missing = DB::table('orders')
        ->whereNull('account_id')
        ->count();

    if ($missing > 0) {
        throw new RuntimeException(
            "Cannot require orders.account_id; {$missing} rows are null."
        );
    }

    Schema::table('orders', function (Blueprint $table) {
        $table->unsignedBigInteger('account_id')
            ->nullable(false)
            ->change();
    });
}

This is a precondition, not an optional feature. A failed precondition should stop the release and explain what must be completed.

Use Case 9: Large Data Backfills

Developers sometimes make a migration conditional on table size:

if (DB::table('orders')->count() < 100000) {
    // Backfill here.
}

This produces the opposite of a controlled deployment. Small databases receive the change while large, important databases silently do not.

For large changes, separate schema evolution from data movement.

Phase 1: Expand the Schema

Add a nullable column that old code can ignore:

Schema::table('orders', function (Blueprint $table) {
    $table->string('status_v2', 32)->nullable()->index();
});

Deploy code that can read the old representation and dual-write both representations.

Phase 2: Backfill in Resumable Batches

Use an Artisan command or queue jobs rather than one long schema migration:

DB::table('orders')
    ->whereNull('status_v2')
    ->orderBy('id')
    ->chunkById(1000, function ($orders) {
        foreach ($orders as $order) {
            DB::table('orders')
                ->where('id', $order->id)
                ->whereNull('status_v2')
                ->update([
                    'status_v2' => match ($order->legacy_status) {
                        0 => 'pending',
                        1 => 'paid',
                        2 => 'cancelled',
                        default => 'unknown',
                    },
                ]);
        }
    });

Laravel recommends chunkById() when records are being updated during iteration because ordinary chunk offsets can produce inconsistent results. See the Query Builder documentation.

The extra whereNull() on the update makes the operation restartable and avoids overwriting a newer dual-written value.

For higher throughput, perform set-based updates where the database can express the transformation safely. The row-by-row example prioritises readability.

Phase 3: Verify

Check at least:

Remaining null or invalid values
Counts grouped by old and new state
Records changed after the backfill began
Application error rate
Query latency and database locks
A sample of business-critical records

Phase 4: Switch Reads

Deploy code that treats status_v2 as authoritative. Keep the old field temporarily if rollback requires it.

Phase 5: Contract

In a later release, add the non-null constraint and eventually remove the old field.

The condition belongs in release orchestration: “Has the backfill and verification completed?” It should not be an invisible if that produces different final schemas.

Avoid Eloquent Models Inside Old Migrations

This may work today:

Order::query()->whereNull('status_v2')->each(...);

It may fail next year when a new developer creates a database from scratch.

The current Order model could then contain:

Global scopes that expect columns not yet created
Attribute casts incompatible with old data
Observers that dispatch jobs or external calls
Relationships depending on later tables
Events and accessors that did not exist when the migration was written

Prefer the query builder and explicit table names in migrations:

DB::table('orders')->whereNull('status_v2')->update(...);

For a large or operationally sensitive data change, put the backfill in a versioned command or job. Keep the schema migration focused on DDL and precondition checks.

Environment-Conditional Migrations: Usually a Warning Sign

This pattern is common:

if (app()->environment('production')) {
    Schema::create('reporting_snapshots', ...);
}

It creates a production-only schema that staging and automated tests cannot reproduce.

Problems include:

Bugs appear only during the production migration.
Developers cannot create a faithful local database.
migrate:fresh does not produce the same structure everywhere.
Later migrations may assume the table exists and fail outside production.
A renamed environment such as prod or production-blue can change behaviour.

Better options are:

Use the same schema everywhere and disable behaviour with a runtime flag.
Put infrastructure-specific migrations in an explicit path.
Target a named database connection.
Test production-specific database capabilities in a matching staging database.
Keep local substitutes for external warehouses or analytics systems.

An environment condition is defensible only when schema divergence is an intentional product requirement and is documented, inventoried and tested.

Conditions That Should Never Control a Migration

Avoid conditions based on:

The Current User

Artisan migration commands do not have a meaningful authenticated web user, and database schema is normally global.

Random Percentage Rollouts

A 10% feature flag must not create a table with a 10% probability.

Current Traffic or Load Alone

If load is too high, pause the release through orchestration. Do not let the migration report success without performing the required change.

A Remote API That Can Temporarily Fail

A timeout should not determine the permanent schema. Resolve remote configuration before the deployment and store the decision locally.

The Presence of Application Data Without a Defined State Model

“Run only when there are users” creates different schemas for new and established databases. Define a precondition or backfill phase instead.

Hostname or Pod Name

Choose one migration runner explicitly. Do not make schema behaviour depend on an ephemeral server name.

Safe Production Deployment Pattern

Conditional migrations still require a controlled deployment process.

1. Review the Generated SQL

Laravel can display the SQL without applying it:

php artisan migrate --pretend

--pretend is useful for review, but it is not proof that the migration will complete within the required time or locking budget.

2. Check Migration Status

php artisan migrate:status

Record the pending migrations in the deployment evidence.

3. Back Up and Test Restore

A backup is only useful if the team can restore it within the application’s recovery target. For destructive or long-running DDL, consider a database snapshot, replica or point-in-time recovery position.

4. Deploy Backward-Compatible Code

Old and new application instances may overlap during a rolling or zero-downtime deployment. The schema change must be safe for both versions.

Use the expand-and-contract sequence:

Add compatible schema.
Deploy code that supports both states.
Backfill and verify.
Switch reads and writes.
Remove old schema in a later release.

5. Run One Migration Process

For multi-server deployments:

php artisan migrate --isolated --force

Laravel’s --isolated option obtains an atomic lock through the configured cache driver. Other migration commands that cannot obtain the lock do not run, but Laravel documents that they still exit successfully. See Isolating Migration Execution.

This produces two operational requirements:

Deployment nodes must use the same lock-capable cache store.
A successful shell exit does not prove that a particular node ran the migrations.

Prefer one designated migration job, and verify migrate:status afterwards.

6. Monitor Database and Application Health

Watch:

Lock waits and blocked queries
Replication lag
CPU, I/O and temporary space
Migration duration
Error rate and failed jobs
Query latency
Business-level correctness metrics

7. Restart Long-Lived Workers

Laravel queue workers and Octane processes keep booted application state in memory. Restart or reload them according to the deployment platform after compatible schema and code are active.

Rollback Versus Roll-Forward

down() is valuable during development, but a production rollback is not always equivalent to restoring the previous system.

Consider a migration that adds a column, deploys code that writes data into it, and then runs for two hours. Dropping that column during rollback destroys the new data.

Before deployment, classify the change:

Change	Typical recovery strategy
Add nullable column	Roll back application code; keep harmless column
Add index	Drop index if safe and necessary
Rename column	Prefer compatibility phase; do not depend on immediate rollback
Backfill transformed data	Keep original field until verification period ends
Add non-null constraint	Remove constraint if old code still writes nulls
Drop column or table	Restore from backup or roll forward; automatic rollback may be impossible
Cross-database data move	Reconcile and resume rather than assume atomic rollback

For many production failures, a roll-forward migration is safer than attempting to run a destructive down() method under pressure.

Conditional down() methods must not hide uncertainty. If the migration cannot know whether it owns an existing object, fail with a clear message and require an explicit recovery procedure.

Testing Conditional Migrations

A conditional migration needs more tests than a normal migration because it has more possible starting states.

Test the Empty Database Path

A new database must reach the expected final schema:

php artisan migrate:fresh --env=testing

Never run migrate:fresh against a database containing valuable data. It drops all tables before recreating them.

Test Every Supported Starting State

For a guarded column migration, create tests for:

Table exists and column is missing.
Table and compatible column already exist.
Table is missing unexpectedly.
Column exists with an incompatible definition.
Migration is retried after partial non-transactional DDL.
Rollback is attempted after each successful path.

Test Both Flag Values

For shouldRun():

Run with the stable flag disabled and confirm no object is created.
Enable the flag and run migrations again.
Confirm the object is created exactly once.
Run migrations once more and confirm there is no further change.

Test the Real Database Engines

SQLite is useful for fast tests but does not reproduce all MySQL, MariaDB, PostgreSQL or SQL Server DDL behaviour.

If production uses PostgreSQL and the migration has a PostgreSQL branch, run it against the supported PostgreSQL version in CI or a pre-production environment.

Verify the Result, Not Only the Exit Code

Assertions should inspect:

Tables and columns
Column type, nullability and default
Index columns, order and uniqueness
Foreign keys and actions
Expected row counts or reconciliation totals
Pending migration status

An exit code of zero says that Laravel did not report an error. It does not prove the schema matches the application’s contract.

Common Mistakes

1. Wrapping Every Migration in `hasTable()`

if (! Schema::hasTable('orders')) {
    Schema::create('orders', ...);
}

For a normal application-created table, “already exists” may indicate a duplicate migration, wrong database, manual change or naming collision. Automatically accepting it can mask the root cause.

Use this guard only for a documented alternative state.

2. Checking Only a Column Name

A price column can exist as an integer in one database and a decimal in another. Inspect the attributes that matter to the application.

3. Using `shouldRun()` for a Required Prerequisite

If a constraint is required for the release, do not leave it pending silently. Fail with a diagnostic message when the backfill is incomplete.

4. Using a User-Scoped Feature Flag

Schemas are normally environment-wide. Use a global deployment decision.

5. Performing Millions of Updates in `up()`

Long transactions, locks, timeouts and unresumable failures make this dangerous. Use a resumable backfill command.

6. Editing a Migration That Already Ran

Laravel will not rerun it merely because its file changed. Create a new migration.

7. Assuming `--isolated` Means This Node Migrated

Another node may own the lock, while this command exits successfully without running. Use one migration runner and verify status.

8. Assuming Every DDL Operation Rolls Back

Transaction support differs by database and operation. Test failure halfway through the migration.

9. Catching Every Exception and Continuing

This is especially dangerous:

try {
    Schema::table(...);
} catch (Throwable $exception) {
    // Ignore it.
}

Laravel may record the migration as successful while the schema change failed. Catch only errors you can identify, validate and recover from; otherwise let the deployment stop.

10. Making Production the First Real Test

Database-specific branches and online DDL must be exercised against production-like engine versions and realistic table sizes.

A Practical Decision Framework

Before adding an if statement to a migration, ask these questions.

Question 1: Should Every Installation Have the Same Final Schema?

Yes: prefer a deterministic migration. Use conditions only to reconcile known starting states.
No: document the supported schema variants and how the module or tenant model selects them.

Question 2: Should Laravel Try Again Later?

Yes: use shouldRun() with a stable global condition.
No: enter up(), validate the current state, reconcile it and let Laravel record completion.

Question 3: Is the Condition Optional or Required?

Optional feature: skipping may be valid.
Required prerequisite: fail clearly rather than skip.

Question 4: Is the Difference About Capability?

Different database driver or version: branch on the capability and test every branch.
Different environment name: reconsider the design.

Question 5: Does the Change Move Large Amounts of Data?

Yes: separate the schema migration from a resumable backfill.
No: a small deterministic data update may be acceptable, preferably through the query builder.

Question 6: Can Old and New Code Run Together?

No: redesign the migration as expand and contract or use a planned maintenance window.
Yes: define how long compatibility must remain.

Question 7: What Happens Halfway Through?

Know whether DDL is transactional, which objects may already exist on retry, how to verify the state and whether recovery is roll-forward or rollback.

Production Checklist

Before approving a conditional migration, confirm that:

The condition represents a documented, expected state.
The condition is stable for the complete migration run.
Every deployment node sees the same global configuration.
shouldRun() is used only when the migration should remain pending.
An up() guard is used intentionally when a no-op should still be recorded.
Existing objects are checked for compatibility, not only existence.
Unsupported database drivers fail clearly.
Every database-specific branch is tested on the real engine.
Large backfills are resumable and separate from DDL.
Required prerequisites stop the migration instead of silently skipping.
The migration does not depend on current Eloquent models or observers.
Old and new application versions can coexist with the expanded schema.
The down() method will not delete an object the migration did not create.
The team has a backup, restore plan and recovery decision.
php artisan migrate --pretend has been reviewed.
php artisan migrate:status is checked before and after deployment.
Only one controlled runner executes production migrations.
--isolated uses a shared cache driver that supports atomic locks.
Monitoring covers locks, replication lag, errors and business correctness.
Long-lived queue and application workers are restarted or reloaded.
The final schema is verified directly after the command succeeds.

Frequently Asked Questions

What is a conditional migration in Laravel?

It is a migration whose execution or implementation depends on a condition, such as an enabled module, an existing schema object, the active database driver, a selected connection or a completed prerequisite.

How do I skip a migration conditionally in Laravel 13?

Define a public shouldRun(): bool method on the migration. Laravel calls it before running the migration. Return true when the migration should execute and false when it should be skipped.

What is the difference between `shouldRun()` and returning early from `up()`?

shouldRun() prevents the migration from running and leaves it available for a later attempt. Returning early from up() completes the migration call without schema work, so Laravel records it as run. Use the first for deferred execution and the second for deliberate reconciliation.

Should every migration use `Schema::hasTable()`?

No. For normal deterministic migrations, an unexpected existing table should usually stop the process so the cause can be investigated. Existence guards are appropriate only for documented legacy or optional states.

Can I check whether a column exists before adding it?

Yes. Use Schema::hasColumn() or Schema::whenTableDoesntHaveColumn(). For legacy databases, also validate important attributes such as type, nullability, default and indexes.

Can a migration use a feature flag?

Yes, but the flag must represent a stable global deployment decision. Do not use a user-scoped, random or percentage-based flag to control database schema.

Should I use `app()->environment()` inside a migration?

Usually no. It creates schema differences that automated tests and staging may not reproduce. Prefer a named connection, explicit migration path, capability check or runtime feature flag.

How do I run different SQL for MySQL and PostgreSQL?

Use DB::connection()->getDriverName(), branch explicitly, test both paths and throw an exception for unsupported drivers. Use Laravel’s schema builder where it expresses the required operation correctly.

Can conditional migrations make a migration idempotent?

They can make specific operations safe to retry, but a few hasColumn() checks do not prove the complete migration is idempotent. You must account for indexes, constraints, data changes and partial DDL failure.

Should I put a large data backfill inside a migration?

Normally no. Add compatible schema first, backfill through a resumable command or jobs, verify the result, switch application reads and then add constraints or remove old fields in later migrations.

Should a migration silently wait when data is not ready?

If the schema change is required for the release, no. Throw a clear exception stating which prerequisite remains incomplete. Silent no-ops can be recorded as completed and leave the application unprotected.

How do I migrate separate tenant databases?

Use a controlled orchestration command that selects each tenant connection and runs a dedicated migration path. Keep a migration history per tenant database, log results and use locks to prevent concurrent runs.

Is `php artisan migrate --isolated` enough for multi-server deployment?

It prevents concurrent migration execution when every node shares a cache store that supports atomic locks. However, a node that does not obtain the lock can still exit successfully. Use one designated migration job and verify pending status afterwards.

Are migration rollbacks always safe?

No. Dropping a newly used column can destroy data, and some database DDL is not transactional. For additive changes, rolling application code back while keeping the column is often safer. Plan the recovery method before deploying.

Can I edit an old migration to add a new condition?

Do not edit a migration that has already been shared or deployed. Existing databases will not rerun it. Create a new reconciliation or corrective migration.

Final Summary

Conditional migrations are most valuable when they make an expected schema difference explicit.

Laravel provides several useful mechanisms:

shouldRun() for a migration that should remain pending until a stable condition becomes true
hasTable(), hasColumn(), hasIndex() and metadata inspection for known legacy states
Conditional schema helpers for concise, guarded changes
Named connections for separate databases
Driver detection for database-specific SQL
$withinTransaction for operations whose transaction requirements differ
--isolated to prevent concurrent production migration runners

The framework cannot decide whether a condition represents a legitimate variant or hidden schema drift. That remains an architectural decision.

Use conditions to select between states you deliberately support. Fail when an unexpected state would violate the application’s database contract. For large or breaking changes, avoid a clever one-file migration and use a staged sequence: expand the schema, deploy compatible code, backfill, verify, switch traffic and contract later.

The safest conditional migration is not the one with the most defensive if statements. It is the one whose starting states, final state, retry behaviour and recovery path are all known before production runs it.

August 6, 2026

Web Scraping for News Data Acquisition in Stock Market Prediction AI

Historical prices show what the market did. News can help explain what investors knew, when they knew it and why expectations changed.

For a stock-market prediction project, useful news data may include:

Earnings announcements
Product launches and recalls
Management changes
Mergers and acquisitions
Regulatory decisions
Lawsuits and investigations
Analyst upgrades and downgrades
Interest-rate and inflation news
Supply-chain disruptions
Industry-specific events

However, collecting news is not as simple as copying headlines into a CSV file. A reliable system must identify relevant articles, respect website rules, preserve accurate timestamps, remove duplicates and avoid using information that was unavailable at the time of prediction.

This tutorial builds a practical Python news-acquisition pipeline for an AI stock-prediction project. It uses RSS feeds for discovery, permitted webpage extraction for full article data, robots.txt checks, request throttling, structured metadata, deduplication and CSV/JSONL output.

The tutorial also explains when a news API is a better choice than web scraping and how to prepare the collected records for sentiment analysis or other natural-language-processing models.

This project is for educational and research purposes. Website terms, copyright rules, database rights and data-licensing requirements vary by source and jurisdiction. Check the applicable rules before collecting or redistributing content.

Quick Answer

The safest practical order for collecting financial news is:

Use a licensed data feed or official API when one is available.
Use the publisher’s RSS feed for article discovery.
Scrape public HTML only when the site’s rules and terms permit it.
Identify your crawler and limit its request rate.
Store the original URL, source, publication time and collection time.
Deduplicate syndicated and updated stories.
Align each article with the first trading period in which it could have been known.
Keep raw data unchanged and create cleaned data as a separate dataset.

A useful news record should look like this:

{
  "source": "Example Financial News",
  "url": "https://example.com/news/company-results",
  "canonical_url": "https://example.com/news/company-results",
  "title": "Example Company Reports Quarterly Results",
  "published_at": "2026-08-05T08:30:00+00:00",
  "collected_at": "2026-08-05T08:34:12+00:00",
  "author": "News Desk",
  "description": "The company reported its latest quarterly results.",
  "text": "Full permitted article text or an authorised excerpt...",
  "tickers": ["EXMPL"],
  "content_hash": "..."
}

The timestamp fields are especially important. If an article appeared after the market closed, a model predicting that same day’s closing price must not be allowed to use it.

Why News Data Is Useful in Stock-Market Prediction

Price and volume data describe market behaviour numerically. News adds information about events that may change future cash flows, risk or investor expectations.

Suppose a company’s share price falls by 8%. Historical price data can show:

The size of the fall
Trading volume
Recent volatility
Whether the price crossed a moving average

News data may reveal that the company:

Missed its earnings forecast
Lost a major customer
Announced a product recall
Received a regulatory penalty

An AI model can convert news text into features such as:

Positive, neutral or negative sentiment
Sentiment confidence
Number of relevant articles
Source diversity
Novelty compared with earlier stories
Event category
Company or sector mentioned
Time since publication
Unexpectedness or severity

News does not automatically improve a prediction model. The benefit depends on the quality of the collection pipeline, the target being predicted and whether the information was genuinely available before the prediction time.

News APIs, RSS Feeds and HTML Scraping Compared

There are several ways to obtain news. They are not interchangeable.

Method	Advantages	Limitations	Best use
Licensed financial feed	Consistent schema, strong coverage, clear usage rights	Often expensive	Production research and trading systems
News API	Structured JSON, easy queries, fewer broken selectors	Quotas, licensing limits, possible text truncation	Prototypes and article discovery
Publisher RSS feed	Lightweight, timestamped, easy to parse	Often contains only summaries	Discovery and monitoring
Permitted HTML scraping	Can extract page metadata and available text	Fragile, source-specific and legally sensitive	Research where permitted
Search-engine results	Broad discovery	Unstable, duplicate-heavy and often restricted	Manual research, not a primary dataset

NewsAPI’s Everything endpoint, for example, is designed for article discovery and analysis using keyword, date, domain and language filters. GDELT also provides large-scale news datasets and live APIs. These options may be more suitable than maintaining many site-specific scrapers.

An API does not remove every problem. You must still check:

Whether historical results are complete
Whether full text or only a snippet is supplied
How publication times are defined
Whether the licence permits storage, modelling and redistribution
Whether the provider corrects or deletes records later
Whether the source list changes over time

For a learning project, RSS discovery plus conservative webpage extraction demonstrates the complete acquisition process without pretending that every news website can or should be crawled.

Legal, Ethical and Technical Rules

Before writing a scraper, create a source policy.

1. Read the Website’s Terms

Check the site’s terms of service, data policy and licensing information. A publicly viewable page is not automatically free to copy, store indefinitely or republish.

Do not bypass:

Paywalls
Authentication
CAPTCHA challenges
Access controls
Rate limits
Technical restrictions

If the website offers an API or licensed feed, use it when practical.

2. Check robots.txt

The Robots Exclusion Protocol is standardised in RFC 9309. A compliant crawler retrieves the site’s /robots.txt file and checks whether its user agent is permitted to request a particular URL.

robots.txt is not a complete legal permission system. An allowed path does not cancel copyright, contract or privacy obligations. A disallowed path should nevertheless be treated as a clear instruction not to crawl it.

3. Identify the Crawler

Use a descriptive user agent rather than pretending to be a normal browser:

StockNewsResearchBot/1.0 (+https://your-domain.example/bot; [email protected])

The linked page can explain:

Who operates the crawler
Its research purpose
Its normal request frequency
How a publisher can contact you
How to request removal

4. Crawl Slowly

A research crawler does not need to request the same server several times per second.

Use:

A delay between requests
Connection and read timeouts
A limited retry count
Exponential backoff for temporary failures
Per-domain request scheduling
Caching to avoid downloading unchanged pages

Respect HTTP 429 Too Many Requests and Retry-After responses.

5. Store Only What You Need

For many projects, a headline, permitted excerpt, source, URL and timestamp are enough. Storing and redistributing complete copyrighted articles can create additional legal and licensing risks.

Keep the original source URL so authorised users can return to the publisher.

Designing the Dataset Before Writing the Scraper

A crawler should write records into a predefined schema. Otherwise, every source produces a different collection of columns.

Use at least these fields:

Field	Purpose
`source`	Publisher or feed name
`url`	URL discovered by the system
`canonical_url`	Publisher’s preferred URL when available
`title`	Article headline
`description`	Summary or metadata description
`author`	Author when supplied
`published_at`	Claimed original publication time
`modified_at`	Later modification time, if supplied
`collected_at`	Time your system retrieved the page
`text`	Permitted body text or excerpt
`language`	Article language when known
`tickers`	Companies matched by controlled rules
`content_hash`	Hash used for deduplication
`status`	Success, blocked, missing date or other state

Do not overwrite published_at with the collection time. They answer different questions.

Also consider preserving the raw response metadata separately:

HTTP status
ETag
Last-Modified
Content type
Retrieval duration
Extractor version
Feed URL

This information helps diagnose coverage gaps and reproduce the dataset later.

Project Structure

Create a folder with the following files:

stock-news-scraper/
├── scrape_news.py
├── requirements.txt
├── news_articles.csv
└── news_articles.jsonl

The output files are created automatically after the scraper runs.

Step 1: Install Python Packages

Create a virtual environment:

python -m venv .venv

Activate it on Linux or macOS:

source .venv/bin/activate

On Windows PowerShell:

.venv\Scripts\Activate.ps1

Create requirements.txt:

beautifulsoup4
feedparser
python-dateutil
requests

Install the dependencies:

python -m pip install -r requirements.txt

Pin tested package versions before deploying a production pipeline. A lock file or fully pinned requirements file helps keep future runs reproducible.

Step 2: Configure Sources and Company Terms

The complete script below intentionally uses placeholder feeds. Replace them only with sources you are authorised to access.

RSS_SOURCES = [
    {
        "name": "Example Financial News",
        "feed_url": "https://news.example.com/finance/rss.xml",
    },
    {
        "name": "Example Exchange Announcements",
        "feed_url": "https://exchange.example.com/announcements.xml",
    },
]

COMPANIES = {
    "AAPL": ["Apple", "Apple Inc."],
    "MSFT": ["Microsoft", "Microsoft Corp."],
    "NVDA": ["Nvidia", "NVIDIA Corporation"],
}

Avoid using a ticker alone for ambiguous symbols. A short ticker such as CAT, IT or ON can occur frequently in normal English.

A stronger company dictionary includes:

Official company name
Common short name
Unambiguous ticker forms such as $AAPL
Former names
Important subsidiaries
Exchange name
Country or industry context

Entity linking should be treated as its own data-quality problem, not as a simple substring search.

Step 3: Complete News-Scraping Script

Save the following as scrape_news.py:

from __future__ import annotations

import csv
import hashlib
import json
import re
import time
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from urllib.parse import parse_qsl, urlencode, urljoin, urlsplit, urlunsplit
from urllib.robotparser import RobotFileParser

import feedparser
import requests
from bs4 import BeautifulSoup
from dateutil import parser as date_parser
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


USER_AGENT = (
    "StockNewsResearchBot/1.0 "
    "(+https://your-domain.example/bot; [email protected])"
)

REQUEST_DELAY_SECONDS = 2.0
CONNECT_TIMEOUT_SECONDS = 5
READ_TIMEOUT_SECONDS = 20
MAX_ARTICLES_PER_FEED = 20

CSV_PATH = Path("news_articles.csv")
JSONL_PATH = Path("news_articles.jsonl")

RSS_SOURCES = [
    {
        "name": "Example Financial News",
        "feed_url": "https://news.example.com/finance/rss.xml",
    },
]

COMPANIES = {
    "AAPL": ["Apple", "Apple Inc."],
    "MSFT": ["Microsoft", "Microsoft Corp."],
    "NVDA": ["Nvidia", "NVIDIA Corporation"],
}

TRACKING_PARAMETERS = {
    "fbclid",
    "gclid",
    "mc_cid",
    "mc_eid",
    "ref",
    "source",
}


@dataclass
class ArticleRecord:
    source: str
    url: str
    canonical_url: str
    title: str
    description: str
    author: str
    published_at: str
    modified_at: str
    collected_at: str
    text: str
    language: str
    tickers: list[str]
    content_hash: str
    status: str


def create_session() -> requests.Session:
    retry = Retry(
        total=3,
        connect=3,
        read=2,
        status=3,
        backoff_factor=1.0,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET"}),
        respect_retry_after_header=True,
    )

    adapter = HTTPAdapter(max_retries=retry)
    session = requests.Session()
    session.headers.update(
        {
            "User-Agent": USER_AGENT,
            "Accept": (
                "text/html,application/xhtml+xml,application/rss+xml,"
                "application/xml;q=0.9,*/*;q=0.8"
            ),
        }
    )
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session


def utc_now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()


def normalise_space(value: str) -> str:
    return re.sub(r"\s+", " ", value).strip()


def normalise_url(url: str) -> str:
    parts = urlsplit(url)

    filtered_query = []
    for key, value in parse_qsl(parts.query, keep_blank_values=True):
        lower_key = key.lower()
        if lower_key.startswith("utm_"):
            continue
        if lower_key in TRACKING_PARAMETERS:
            continue
        filtered_query.append((key, value))

    path = parts.path or "/"
    if path != "/":
        path = path.rstrip("/")

    return urlunsplit(
        (
            parts.scheme.lower(),
            parts.netloc.lower(),
            path,
            urlencode(filtered_query, doseq=True),
            "",
        )
    )


def parse_datetime(value: Any) -> str:
    if not value:
        return ""

    try:
        parsed = date_parser.parse(str(value))
    except (TypeError, ValueError, OverflowError):
        return ""

    if parsed.tzinfo is None:
        return ""

    return parsed.astimezone(timezone.utc).isoformat()


def matches_tickers(text: str) -> list[str]:
    matched = []

    for ticker, aliases in COMPANIES.items():
        patterns = [rf"\${re.escape(ticker)}\b"]
        patterns.extend(
            rf"\b{re.escape(alias)}\b"
            for alias in aliases
        )

        if any(re.search(pattern, text, re.IGNORECASE) for pattern in patterns):
            matched.append(ticker)

    return sorted(matched)


def get_robot_parser(
    session: requests.Session,
    url: str,
    cache: dict[str, RobotFileParser | None],
) -> RobotFileParser | None:
    parts = urlsplit(url)
    origin = f"{parts.scheme}://{parts.netloc}"

    if origin in cache:
        return cache[origin]

    robots_url = urljoin(origin, "/robots.txt")
    parser = RobotFileParser()
    parser.set_url(robots_url)

    try:
        response = session.get(
            robots_url,
            timeout=(CONNECT_TIMEOUT_SECONDS, READ_TIMEOUT_SECONDS),
        )
        response.raise_for_status()
        parser.parse(response.text.splitlines())
        cache[origin] = parser
    except requests.RequestException as error:
        print(f"Could not verify robots.txt for {origin}: {error}")
        cache[origin] = None

    return cache[origin]


def is_allowed(
    session: requests.Session,
    url: str,
    cache: dict[str, RobotFileParser | None],
) -> bool:
    parser = get_robot_parser(session, url, cache)

    # Conservative policy: skip when robots.txt cannot be verified.
    if parser is None:
        return False

    return parser.can_fetch(USER_AGENT, url)


def iter_json_ld(soup: BeautifulSoup):
    for script in soup.select('script[type="application/ld+json"]'):
        raw = script.string or script.get_text()
        if not raw.strip():
            continue

        try:
            value = json.loads(raw)
        except json.JSONDecodeError:
            continue

        values = value if isinstance(value, list) else [value]

        for item in values:
            if not isinstance(item, dict):
                continue

            graph = item.get("@graph")
            if isinstance(graph, list):
                values.extend(
                    node for node in graph if isinstance(node, dict)
                )

            yield item


def find_news_json_ld(soup: BeautifulSoup) -> dict[str, Any]:
    accepted_types = {
        "Article",
        "NewsArticle",
        "ReportageNewsArticle",
    }

    for item in iter_json_ld(soup):
        item_type = item.get("@type", "")
        types = item_type if isinstance(item_type, list) else [item_type]

        if any(value in accepted_types for value in types):
            return item

    return {}


def meta_content(soup: BeautifulSoup, *selectors: str) -> str:
    for selector in selectors:
        element = soup.select_one(selector)
        if element and element.get("content"):
            return normalise_space(str(element["content"]))
    return ""


def author_from_json_ld(value: Any) -> str:
    if isinstance(value, dict):
        return normalise_space(str(value.get("name", "")))

    if isinstance(value, list):
        names = [author_from_json_ld(item) for item in value]
        return ", ".join(name for name in names if name)

    if isinstance(value, str):
        return normalise_space(value)

    return ""


def extract_article_text(soup: BeautifulSoup, json_ld: dict[str, Any]) -> str:
    body = json_ld.get("articleBody")
    if isinstance(body, str) and body.strip():
        return normalise_space(body)

    # Generic fallbacks work on some pages, but source-specific selectors
    # are normally more accurate. Add them only for approved sources.
    candidates = soup.select(
        "article p, main article p, [itemprop='articleBody'] p"
    )

    paragraphs = []
    for paragraph in candidates:
        text = normalise_space(paragraph.get_text(" ", strip=True))
        if len(text) >= 40:
            paragraphs.append(text)

    return "\n\n".join(paragraphs)


def make_hash(title: str, text: str) -> str:
    material = normalise_space(f"{title}\n{text}").lower()
    return hashlib.sha256(material.encode("utf-8")).hexdigest()


def extract_article(
    session: requests.Session,
    source_name: str,
    url: str,
    feed_title: str,
    feed_summary: str,
    feed_published: str,
) -> ArticleRecord:
    response = session.get(
        url,
        timeout=(CONNECT_TIMEOUT_SECONDS, READ_TIMEOUT_SECONDS),
    )
    response.raise_for_status()

    content_type = response.headers.get("Content-Type", "").lower()
    if "text/html" not in content_type:
        raise ValueError(f"Unsupported content type: {content_type}")

    soup = BeautifulSoup(response.text, "html.parser")
    json_ld = find_news_json_ld(soup)

    title = normalise_space(
        str(json_ld.get("headline", ""))
        or meta_content(soup, 'meta[property="og:title"]')
        or feed_title
    )

    description = normalise_space(
        str(json_ld.get("description", ""))
        or meta_content(
            soup,
            'meta[property="og:description"]',
            'meta[name="description"]',
        )
        or feed_summary
    )

    canonical_element = soup.select_one('link[rel="canonical"]')
    canonical_url = url
    if canonical_element and canonical_element.get("href"):
        canonical_url = urljoin(url, str(canonical_element["href"]))

    canonical_url = normalise_url(canonical_url)
    text = extract_article_text(soup, json_ld)
    combined_text = f"{title}\n{description}\n{text}"

    published_at = parse_datetime(
        json_ld.get("datePublished")
        or meta_content(
            soup,
            'meta[property="article:published_time"]',
        )
        or feed_published
    )

    modified_at = parse_datetime(
        json_ld.get("dateModified")
        or meta_content(
            soup,
            'meta[property="article:modified_time"]',
        )
    )

    language = str(json_ld.get("inLanguage", ""))
    if not language and soup.html:
        language = str(soup.html.get("lang", ""))

    return ArticleRecord(
        source=source_name,
        url=normalise_url(url),
        canonical_url=canonical_url,
        title=title,
        description=description,
        author=author_from_json_ld(json_ld.get("author")),
        published_at=published_at,
        modified_at=modified_at,
        collected_at=utc_now_iso(),
        text=text,
        language=language,
        tickers=matches_tickers(combined_text),
        content_hash=make_hash(title, text),
        status="ok" if published_at else "missing_published_time",
    )


def load_existing_keys() -> tuple[set[str], set[str]]:
    urls: set[str] = set()
    hashes: set[str] = set()

    if not JSONL_PATH.exists():
        return urls, hashes

    with JSONL_PATH.open("r", encoding="utf-8") as file:
        for line in file:
            try:
                item = json.loads(line)
            except json.JSONDecodeError:
                continue

            if item.get("canonical_url"):
                urls.add(item["canonical_url"])
            if item.get("content_hash"):
                hashes.add(item["content_hash"])

    return urls, hashes


def save_records(records: list[ArticleRecord]) -> None:
    if not records:
        return

    with JSONL_PATH.open("a", encoding="utf-8") as file:
        for record in records:
            file.write(json.dumps(asdict(record), ensure_ascii=False) + "\n")

    fieldnames = list(asdict(records[0]).keys())
    write_header = not CSV_PATH.exists()

    with CSV_PATH.open("a", encoding="utf-8", newline="") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        if write_header:
            writer.writeheader()

        for record in records:
            item = asdict(record)
            item["tickers"] = json.dumps(item["tickers"])
            writer.writerow(item)


def run() -> None:
    session = create_session()
    robot_cache: dict[str, RobotFileParser | None] = {}
    seen_urls, seen_hashes = load_existing_keys()
    new_records: list[ArticleRecord] = []

    for source in RSS_SOURCES:
        feed_url = source["feed_url"]
        source_name = source["name"]

        try:
            feed_response = session.get(
                feed_url,
                timeout=(CONNECT_TIMEOUT_SECONDS, READ_TIMEOUT_SECONDS),
            )
            feed_response.raise_for_status()
        except requests.RequestException as error:
            print(f"Feed failed for {source_name}: {error}")
            continue

        feed = feedparser.parse(feed_response.content)

        for entry in feed.entries[:MAX_ARTICLES_PER_FEED]:
            url = normalise_url(str(entry.get("link", "")))
            if not url or url in seen_urls:
                continue

            feed_title = normalise_space(str(entry.get("title", "")))
            feed_summary = normalise_space(
                BeautifulSoup(
                    str(entry.get("summary", "")),
                    "html.parser",
                ).get_text(" ", strip=True)
            )

            candidate_text = f"{feed_title}\n{feed_summary}"
            if not matches_tickers(candidate_text):
                continue

            if not is_allowed(session, url, robot_cache):
                print(f"Skipped by crawler policy: {url}")
                continue

            time.sleep(REQUEST_DELAY_SECONDS)

            try:
                record = extract_article(
                    session=session,
                    source_name=source_name,
                    url=url,
                    feed_title=feed_title,
                    feed_summary=feed_summary,
                    feed_published=str(
                        entry.get("published", "")
                        or entry.get("updated", "")
                    ),
                )
            except (requests.RequestException, ValueError) as error:
                print(f"Article failed: {url}: {error}")
                continue

            if record.canonical_url in seen_urls:
                continue
            if record.content_hash in seen_hashes:
                continue

            seen_urls.add(record.canonical_url)
            seen_hashes.add(record.content_hash)
            new_records.append(record)
            print(f"Collected: {record.title}")

    save_records(new_records)
    print(f"Saved {len(new_records)} new article(s).")


if __name__ == "__main__":
    run()

Before running the script, change:

The contact details in USER_AGENT
The placeholder RSS feed
The company and ticker dictionary
The request delay if a source asks for a slower rate
The extraction policy if full article text is not licensed for your use

Run it with:

python scrape_news.py

How the Scraper Works

1. RSS Is Used for Discovery

The script does not crawl a website looking for every possible link. It reads an approved RSS feed and processes recent entries.

This reduces unnecessary requests and provides useful metadata such as:

Headline
Article URL
Summary
Publication or update time

The RSS result is still verified against the article page because feeds may contain abbreviated or differently formatted metadata.

2. Obvious Irrelevant Articles Are Filtered Early

The feed title and summary are checked against the controlled company dictionary before the article page is requested.

This saves bandwidth, but it can create false negatives. An article may discuss a company indirectly without naming it in the headline or summary.

For better recall, a production system might first collect all permitted feed metadata and perform entity recognition later.

3. robots.txt Is Checked Per Origin

The script uses Python’s RobotFileParser and caches the result for each origin. It adopts a conservative policy: if robots.txt cannot be verified, the page is skipped.

Different projects may use a different failure policy, but it should be documented and applied consistently.

4. Structured Metadata Is Preferred

Many publishers include Schema.org NewsArticle or Article data in a JSON-LD script. This may provide:

headline
description
datePublished
dateModified
author
articleBody
inLanguage

Structured metadata is usually more stable than selecting CSS classes designed only for page layout.

The script then falls back to Open Graph metadata, normal meta descriptions and RSS values.

5. Generic Paragraph Extraction Is a Fallback

The selector:

article p, main article p, [itemprop='articleBody'] p

works on some pages but will not work perfectly everywhere. It may include captions, related-story text or subscription messages.

For an authorised source, add a tested source-specific extractor instead of building a single enormous selector for every website.

6. URLs and Content Are Deduplicated

The script removes common tracking parameters and fragments from URLs. It also creates a SHA-256 hash from the normalised title and body.

This catches:

The same URL discovered twice
Tracking variants of one URL
Identical content published under different URLs

It does not catch heavily edited or syndicated versions of the same story. Near-duplicate detection is discussed later.

Using a News API Instead

If your use case and licence are supported, a news API can replace the discovery and extraction sections.

Here is a simple NewsAPI example:

import os
from datetime import date, timedelta

import requests


api_key = os.environ["NEWS_API_KEY"]
start_date = date.today() - timedelta(days=7)

response = requests.get(
    "https://newsapi.org/v2/everything",
    headers={"X-Api-Key": api_key},
    params={
        "q": 'Apple OR "Apple Inc" OR AAPL',
        "from": start_date.isoformat(),
        "language": "en",
        "sortBy": "publishedAt",
        "pageSize": 100,
    },
    timeout=(5, 20),
)

response.raise_for_status()
payload = response.json()

for article in payload.get("articles", []):
    print(article["publishedAt"], article["title"])

Set the key outside the source code:

export NEWS_API_KEY="your-key-here"

Never commit an API key to Git.

The API response still requires:

Pagination
Date-window management
Deduplication
Query logging
Rate-limit handling
Licence compliance
Ticker or company mapping
Timestamp validation
Coverage monitoring

Do not assume an API’s content field contains the complete article. Check the provider’s current documentation and plan.

Improving Deduplication

Financial news is frequently syndicated. Ten URLs may represent one original report rather than ten independent signals.

Basic deduplication should compare:

Normalised canonical URL
Exact content hash
Normalised headline
Publication time
Source and author

For near duplicates, calculate similarity between titles or article embeddings.

Example using title tokens:

import re


def title_tokens(title):
    return set(re.findall(r"[a-z0-9]+", title.lower()))


def jaccard_similarity(first_title, second_title):
    first = title_tokens(first_title)
    second = title_tokens(second_title)

    if not first and not second:
        return 1.0
    if not first or not second:
        return 0.0

    return len(first & second) / len(first | second)

A high similarity score is a review signal, not proof that two stories are identical.

Keep both concepts:

article_id: one collected page
story_cluster_id: several pages describing the same underlying event

This allows the model to distinguish genuine source confirmation from simple republication.

The Most Important Finance Problem: Point-in-Time Correctness

Look-ahead bias occurs when the training dataset gives a model information that would not have been available at prediction time.

Imagine this sequence:

Time	Event
Monday 4:00 PM	Market closes
Monday 4:15 PM	Company releases weak results
Monday 4:17 PM	News article is published
Tuesday 9:30 AM	Market reopens and price falls

If the Monday after-hours article is assigned to Monday’s closing-price prediction, the dataset leaks future information.

For every record, preserve:

published_at: time claimed by the publisher
modified_at: later edit time
collected_at: time observed by your collector
available_at: earliest defensible time the model could have received it

A conservative definition is:

available_at = max(published_at, first_observed_at)

This prevents a crawler that runs on Tuesday from pretending it definitely possessed the article on Monday merely because the page now displays a Monday timestamp.

Aligning News with Trading Sessions

Do not group articles by calendar date alone.

The correct trading session depends on:

Exchange timezone
Market opening and closing time
Weekends
Public holidays
Half-day trading sessions
Pre-market and after-hours trading
The model’s exact prediction cutoff

For a model that makes one prediction immediately before the regular market opens:

News before the cutoff may belong to that session.
News after the cutoff belongs to the next prediction window.

For a model predicting five-minute returns, publication timestamps may not be precise enough. You may need first-observed times and a realistic delay for acquisition, processing and model inference.

Cleaning the Text Without Destroying Evidence

Keep two layers:

Raw Layer

The raw layer should be append-only where practical. It contains the exact permitted data returned by the collector plus retrieval metadata.

Do not silently rewrite old raw records when cleaning rules change.

Processed Layer

The processed layer can contain:

Lowercased text when appropriate
Removed navigation fragments
Normalised whitespace
Sentence segmentation
Language detection
Company entities
Sentiment score
Event classification
Embeddings
Story clusters
Trading-session assignment

Version the transformation code. A dataset should record which extractor, entity matcher and sentiment model produced each feature.

Avoid aggressive cleaning before using a transformer model. Punctuation, numbers, negation and casing can carry useful meaning.

For example:

Profit increased to $2.1 billion.

should not become:

profit increased billion

The removed figure may be the most important part of the sentence.

Converting News into Model Features

After collection, aggregate article-level information into features for a defined time window.

Possible daily features include:

Feature	Meaning
`article_count`	Number of relevant articles available before cutoff
`unique_story_count`	Number of deduplicated event clusters
`mean_sentiment`	Average sentiment score
`min_sentiment`	Most negative article score
`max_sentiment`	Most positive article score
`negative_article_ratio`	Share of articles classified as negative
`source_count`	Number of distinct publishers
`news_volume_zscore`	Abnormal news volume relative to history
`hours_since_latest_news`	Recency of the latest relevant article
`earnings_news_count`	Number of earnings-related stories
`regulatory_news_count`	Number of regulatory stories

Do not calculate a simple average before deduplicating. Twenty copied versions of one negative report should not automatically count as twenty independent negative events.

A weighted sentiment feature could be:

weighted sentiment = sum(sentiment × relevance × recency × source weight)
                     / sum(relevance × recency × source weight)

Every weighting rule should be learned or justified using training and validation data. Do not tune the rule on the final test period.

Matching Articles to the Correct Company

Company matching is harder than it looks.

Consider the word Apple. It may refer to:

Apple Inc.
The fruit
Apple Records
A different organisation with Apple in its name

A good entity-linking pipeline uses several signals:

Full company name
Ticker with exchange context
Executives and products
Industry terms
Country
Other companies mentioned
Publisher section
Named-entity recognition

Create a relevance score rather than accepting every keyword match.

Example rule:

+3  full company name in headline
+2  recognised product and company in body
+2  ticker written as $AAPL
+1  company name repeated in first paragraph
-3  ambiguous word used without business context

Then manually review samples near the threshold. False company assignments can corrupt sentiment features even when the scraper itself works perfectly.

Common Web-Scraping Problems

1. JavaScript-Rendered Pages

Some HTML responses contain very little article content because JavaScript loads the page after the browser starts.

Before introducing browser automation, look for:

An official API
RSS content
JSON-LD metadata
Server-rendered article HTML
A licensed feed

Browser automation is slower and more resource-intensive. It also does not grant permission to access content.

2. Selector Changes

A class such as:

.article-body-v4-final

may disappear during a redesign.

Prefer structured data and semantic elements, and maintain source-specific tests.

3. Missing or Ambiguous Dates

A page may show:

Only a local time without timezone
Only “two hours ago”
A modified time instead of original publication time
A date with no time

Do not invent precision. Flag the record for review or exclude it from time-sensitive experiments.

4. Corrected Articles

A publisher may change a headline or correct figures after publication.

Store version observations rather than replacing the earlier record without a trace. The model should see only the version that existed at the relevant time.

5. Paywalls and Login Pages

A successful HTTP status does not prove that article content was retrieved. The scraper may have captured a subscription prompt.

Add quality checks such as:

Minimum text length
Required headline
Valid publication time
Known boilerplate rejection
Article-to-navigation text ratio

Never attempt to defeat a paywall or access control.

6. Soft Blocks

A server may return HTTP 200 with a block or challenge page.

Detect unexpected titles, content types and page templates. Stop requesting the source and investigate rather than increasing the request rate or disguising the crawler.

Testing the Acquisition Pipeline

A scraper is a data pipeline and needs tests.

Unit Tests

Test small functions using saved, authorised HTML fixtures:

URL normalisation
Date parsing
Company matching
JSON-LD extraction
Content hashing
Near-duplicate scoring

Source Contract Tests

For each approved source, periodically verify that:

RSS is reachable
Article links are valid
The title is extracted
The publication time includes a timezone
Text length remains within an expected range
The extractor does not collect navigation or legal notices as article text

Data-Quality Checks

Run checks after every batch:

assert all(record.url for record in records)
assert all(record.title for record in records)
assert all(record.collected_at for record in records)
assert len({record.content_hash for record in records}) == len(records)

Also report:

Articles discovered per source
Articles collected per source
Blocked or disallowed URLs
Missing publication timestamps
Extraction failures
Duplicate rate
Empty-text rate
Articles per company
Delay from publication to collection

A sudden drop from 500 articles per day to 10 is probably a pipeline problem, not a quiet financial-news day.

Scheduling the Scraper

For a Linux learning environment, a cron entry can run the script every 30 minutes:

*/30 * * * * cd /path/to/stock-news-scraper && /path/to/.venv/bin/python scrape_news.py >> scraper.log 2>&1

Production systems should add:

A scheduler lock to prevent overlapping runs
Centralised logs
Failure alerts
Per-source backoff
Idempotent writes
Database transactions
Secrets management
Data-retention rules
Source-level kill switches

Avoid running a frequent schedule merely because it is possible. Match the interval to the authorised access rate and the prediction horizon.

A daily model does not normally require scraping the same source every minute.

Recommended Production Architecture

A larger pipeline can separate the following stages:

Discovery — receives article URLs from APIs, feeds or approved indexes.
Policy check — applies source permissions, robots rules and request limits.
Fetcher — retrieves permitted content and records HTTP metadata.
Extractor — converts source documents into a common schema.
Raw storage — preserves authorised original observations.
Deduplicator — assigns canonical articles and story clusters.
Entity linker — maps stories to companies, sectors and instruments.
NLP processor — creates sentiment, event and embedding features.
Point-in-time joiner — aligns features with valid market timestamps.
Quality monitor — detects missing coverage and extraction drift.

Each stage should be repeatable and versioned. This makes it possible to improve sentiment analysis without recrawling every source or losing the original observation time.

Mistakes to Avoid

Scraping Everything First

Collecting millions of pages before defining the prediction target creates a large but poorly aligned dataset.

Start with:

A small list of companies
A defined prediction horizon
A limited date range
A few authorised sources
A clear timestamp policy

Ignoring Source Changes

If the list of publishers changes over time, a model may learn differences in data coverage rather than genuine market behaviour.

Record source availability and collection failures.

Treating Every Mention as Relevant

An article that mentions a company in a related-links section should not necessarily affect its sentiment score.

Using the Current Page for Historical Backtests

The page viewed today may contain a corrected headline, updated text or changed publication label. It is not automatically a faithful copy of what existed on the historical date.

Randomly Splitting News Records

Near-duplicate stories can enter both the training and test sets. Split chronologically and, where possible, keep a complete story cluster in one split.

Evaluating Only Prediction Accuracy

A model can achieve a reasonable direction accuracy while still producing an unprofitable strategy after spreads, slippage, fees and delayed news processing.

Final Acquisition Checklist

Before using news in a stock-prediction model, confirm that:

Each source has an approved access method.
Terms and licences have been reviewed.
robots.txt is checked for HTML crawling.
The crawler identifies itself honestly.
Requests use timeouts, retries and conservative rate limits.
Paywalls and access controls are not bypassed.
Raw and processed data are stored separately.
Original, modified, collected and available times are distinguished.
All timestamps are converted to a consistent timezone.
Missing or ambiguous timestamps are flagged.
Canonical URLs and tracking parameters are handled.
Exact and near-duplicate stories are identified.
Articles are linked to companies using more than ambiguous ticker strings.
Collection gaps are monitored by source and date.
News is aligned with the correct trading session and cutoff.
Train, validation and test periods are separated chronologically.
The data licence permits the intended storage, modelling and sharing.

Frequently Asked Questions

Is web scraping legal?

There is no single worldwide answer. It depends on the source, data, access method, contract, copyright, privacy rules and jurisdiction. robots.txt is important crawler guidance, but it is not a complete legal permission system. Obtain professional advice for commercial or high-risk use.

Should I scrape financial-news websites or use an API?

Use an authorised API or licensed feed when it provides the coverage you need. It normally produces more stable structured data and reduces scraper maintenance. HTML extraction is useful for permitted research sources that lack a suitable structured interface.

Can Beautiful Soup scrape JavaScript-rendered pages?

Beautiful Soup parses the HTML it receives; it does not run webpage JavaScript. Check for an official API, RSS feed, JSON-LD or server-rendered content before considering browser automation.

How often should a news scraper run?

The frequency should match the prediction horizon, source rules and permitted request rate. A daily model may need only hourly or daily collection. Intraday research requires more careful point-in-time timestamps and latency measurement.

Should I save the complete article text?

Only if your licence and use case permit it. For some projects, storing the headline, authorised summary, metadata, derived features and source URL is more appropriate.

What is the difference between published_at and collected_at?

published_at is the publication time claimed by the source. collected_at is when your system retrieved the record. Both are required to evaluate whether the information was truly available at prediction time.

Why are there duplicate financial-news stories?

Publishers syndicate wire reports, quote one another, update developing stories and create multiple URLs with tracking parameters. Deduplication should use canonical URLs, hashes and similarity or event clustering.

Can sentiment analysis predict stock prices?

Sentiment may be a useful feature, but it cannot guarantee accurate or profitable forecasts. The market reaction depends on expectations, surprise, source credibility, timing, liquidity and whether the information was already priced in.

Which sentiment model should I use?

Compare a finance-specific language model with simple baselines. Validate on later unseen data. A more complicated model is not automatically better, especially when labels are weak or the news-to-company mapping is inaccurate.

Can I use headlines without article bodies?

Yes. Headlines are cheaper to store and often contain the main event, but they can omit qualifications and context. Test headline-only, summary-only and authorised full-text features separately.

How do I avoid look-ahead bias?

Use the earliest defensible availability time, apply the correct exchange timezone and prediction cutoff, and split data chronologically. Do not use later edits or today’s version of a page as if it were the historical version.

What should I do when robots.txt cannot be retrieved?

Choose and document a conservative policy. The tutorial skips crawling when it cannot verify the rules. For a production system, pause that source and review the failure rather than repeatedly requesting pages.

What comes after data acquisition?

The next stages are text cleaning, entity linking, duplicate clustering, sentiment or event extraction, point-in-time aggregation and chronological model evaluation.

Is this financial advice?

No. This tutorial demonstrates data engineering and machine-learning concepts. It does not recommend buying, selling or holding any investment.

Final Result

We have designed a news-acquisition pipeline for a stock-market prediction project.

The completed approach:

Prioritises licensed feeds, APIs and RSS over unnecessary HTML crawling.
Checks source rules and robots.txt before fetching permitted pages.
Uses an identifiable user agent, timeouts, throttling and limited retries.
Extracts structured article metadata before falling back to generic HTML selectors.
Stores source, URL, publication time, modification time and collection time.
Matches articles to a controlled list of companies.
Removes tracking URLs and exact content duplicates.
Saves machine-readable JSONL and CSV outputs.
Preserves the information needed for point-in-time backtesting.
Prepares the records for entity linking, sentiment analysis and event classification.

The most important lesson is that news volume alone does not make a good dataset. A smaller collection with reliable source permissions, accurate timestamps, strong company matching and correct trading-session alignment is more valuable than a huge archive full of duplicates and future information.

The next stage is to transform the collected news into sentiment and event features, join those features to historical market data without leakage and compare the resulting model against price-only baselines.

August 6, 2026

Pokémon UNITE: Catching ’Em All in a New Way — Review and Beginner Guide

Pokémon UNITE takes a familiar idea—choosing, training and evolving Pokémon—and places it inside a fast team strategy game.

It is not a traditional Pokémon adventure. There is no large world to explore, no Pokédex to complete and no turn-based battle system. Instead, two teams select Pokémon and compete to score the most points before the timer ends.

Each standard match is short enough to finish in about ten minutes, but it still includes many of the things Pokémon fans recognise: wild Pokémon, evolution, signature moves, held items and powerful Legendary Pokémon.

This review and guide explains what Pokémon UNITE is, how its battles work, whether it is still worth playing, how the free-to-start system affects progress and what beginners should do during their first matches.

Quick Answer

Pokémon UNITE is a free-to-start 5-on-5 multiplayer online battle arena game for Nintendo Switch and compatible mobile devices.

Players defeat wild Pokémon, gain levels, choose new moves, collect Aeos energy and deposit that energy into the opposing team’s goal zones. The team with the most points when time expires wins.

Pokémon UNITE Review Score

Category	Score
Core gameplay	8.5/10
Accessibility for beginners	8/10
Pokémon presentation	8.5/10
Teamwork and strategy	8.5/10
Solo-player experience	7/10
Progression and monetisation	6.5/10
Overall	8/10

Final verdict: Pokémon UNITE is easy to start, difficult to master and especially enjoyable with friends. Its ten-minute format makes the MOBA genre more approachable, although matchmaking, team dependence and multiple currencies can sometimes make the experience frustrating.

The game remains free to download. Optional purchases can speed up some unlocks or buy cosmetics, but beginners can learn the game and build a useful roster without paying.

Pokémon UNITE at a Glance

Detail	Information
Genre	Multiplayer online battle arena (MOBA) / team strategy
Standard format	5 players against 5 players
Standard match length	Approximately 10 minutes
Platforms	Nintendo Switch, iOS and Android
Price	Free-to-start with optional in-game purchases
Cross-platform play	Yes, between Switch and mobile
Cross-progression	Supported when the correct account is linked
Main objective	Collect Aeos energy and score more points than the opposing team
Main modes	Casual, ranked, quick, custom and Solo Mode, with rotating special modes
Internet required	Yes

The official game listing confirms that Pokémon UNITE supports cross-platform battles and allows progress to be synchronised between Switch and mobile through a linked Nintendo Account or Pokémon Trainer Club account. An internet connection is required to play. See the official Pokémon UNITE app listing for the current platform information.

What Is Pokémon UNITE?

Pokémon UNITE is Pokémon’s version of a MOBA.

In a traditional MOBA, each player controls one character with a specific role. Teams fight, gain experience, protect parts of the map and compete for objectives. Pokémon UNITE simplifies this formula and replaces towers and conventional heroes with Pokémon, goal zones and Aeos energy.

The basic match looks like this:

Ten players are divided into two teams.
Each player chooses one Pokémon.
The Pokémon begins the match at a low level.
Players defeat wild Pokémon and opposing players to gain experience.
Pokémon learn stronger moves and may temporarily evolve.
Defeated wild and opposing Pokémon drop Aeos energy.
Players carry that energy to enemy goal zones and score it.
The team with the highest score at the end wins.

The official overview describes the same central loop: defeat Pokémon, collect the Aeos energy they drop, deposit it into opposing goal zones and work with teammates to defend your own goals.

Unlike many longer MOBAs, a standard UNITE match has a fixed time limit. You do not need to destroy an enemy base or wait for a team to surrender. The clock creates a clear ending, which makes the game easier to fit into a short break.

Is This Really “Catching ’Em All”?

Not in the traditional sense.

You do not walk through grass, throw Poké Balls or store captured Pokémon in boxes. Wild Pokémon inside a match are temporary sources of experience and Aeos energy.

The collection system instead revolves around Unite licenses. A license allows you to select that Pokémon in eligible battles. You gradually expand your playable roster through rewards, events, the License Journey and the in-game shop.

The game also offers temporary rental licenses, limited free rotations and practice modes. These allow you to try some Pokémon before permanently unlocking them.

Therefore, the title “catching ’em all in a new way” refers to building a roster of playable Pokémon, not completing a conventional Pokédex.

This change works surprisingly well because every Pokémon is more than a collectible picture. Unlocking a new license gives you a different playstyle to learn. A Defender such as Slowbro has a completely different job from a mobile Speedster, a ranged Attacker or a healing Supporter.

How a Pokémon UNITE Match Works

1. Choose a Pokémon and Battle Position

Before the match begins, every player selects a Pokémon. A team cannot normally field two copies of the same Pokémon, so you may need a second or third choice if another player picks your favourite first.

You should also declare where you intend to play:

Top path
Bottom path
Central area

Declaring a position reduces confusion. If two players both assume they are taking the central area, they may compete for the same wild Pokémon and leave a path without enough help.

A balanced team usually includes a mixture of damage, durability, mobility and support. Five fragile Attackers may deal considerable damage but can collapse when the opposing team starts a coordinated fight.

2. Defeat Wild Pokémon

At the beginning of a match, your Pokémon is relatively weak. Defeating nearby wild Pokémon provides experience and Aeos energy.

Experience matters because it unlocks:

Better statistics
New moves
Move upgrades
Evolution for Pokémon that evolve
The Pokémon’s Unite Move

Do not immediately run towards the opposing team while ignoring nearby wild Pokémon. A one-level advantage can decide an early fight, while falling behind can make it difficult to contest objectives later.

3. Learn and Upgrade Moves

Pokémon generally begin with basic moves and choose stronger ones as they level up. The same Pokémon can often choose between two possible moves at each upgrade point.

This creates different builds. One move combination may provide safer ranged damage, while another may favour close combat, mobility or crowd control.

Beginners should use the recommended move path for their first few matches. After learning the Pokémon’s basic rhythm, test the alternative moves in practice or Solo Mode.

4. Collect Aeos Energy

Wild Pokémon and knocked-out opponents can drop Aeos energy. Walk over the energy to collect it.

The amount you can carry is limited and generally increases as your Pokémon gains levels. Carrying a large amount creates both an opportunity and a risk:

You can score more points if you reach an enemy goal.
Scoring takes longer when you carry more energy.
If you are knocked out, you may drop part of what you carry.

This means that repeatedly saving energy for one enormous score is not always the best choice. Several safe scores can be more valuable than carrying a full load into a losing fight.

5. Score in Enemy Goal Zones

Stand inside an opposing goal zone and hold the scoring control. The scoring action can be interrupted, so check for nearby enemies first.

Scoring is easier when:

Teammates are nearby
Opponents have been knocked out or forced to retreat
The goal zone is undefended
You carry only a small amount of energy
Your team has obtained a scoring-related objective effect
An ally helps increase the scoring speed

Do not dive into a defended goal merely because you are carrying many points. Being knocked out may give the opponent experience and prevent you from helping at the next team objective.

6. Contest Major Objectives

Large wild Pokémon appear during a match and provide valuable team-wide advantages. Their identities and effects can vary by map, game mode and season.

In 2026, ranked play can rotate between versions of Theia Sky Ruins featuring Rayquaza, Groudon or Kyogre. The Pokémon UNITE team has said that these versions require different tactics and selections. The current details are explained in the official February 2026 update notice.

The evergreen rule is simple:

Learn what the current objective provides, arrive before it appears and avoid starting it blindly when several opponents are still able to steal the final hit.

An objective is a team decision, not a private duel. Sometimes the correct play is to attack it. Sometimes it is better to hide nearby, defeat the opponents first or defend it while protecting an existing lead.

7. Play the Final Stretch Carefully

The final two minutes are the most dangerous part of a standard match. Scores are normally worth double, and the central Legendary objective may give a powerful advantage.

Many apparently safe matches are lost here because a leading team becomes impatient.

If your team is ahead:

Do not automatically rush the central objective.
Watch the minimap and defend entrances.
Keep your Unite Move ready if possible.
Prevent opponents from scoring large amounts.
Attack the objective only when the situation is favourable.

If your team is behind:

Group instead of entering one at a time.
Look for an opponent carrying many points.
Contest the central objective together.
Use quick chat to coordinate.
After winning the objective or team fight, score immediately.

The final stretch is designed to keep both teams involved until the end. It creates exciting comebacks, but it can also feel punishing when one mistake reverses eight minutes of good play.

Understanding the Five Pokémon Roles

Pokémon UNITE separates its roster into five battle types. The official roster describes their general strengths.

Role	Main job	Typical strengths	Main beginner mistake
Attacker	Deal sustained or burst damage	Ranged damage, strong moves	Standing too close to danger
Speedster	Move quickly and eliminate vulnerable targets	Mobility, burst damage, scoring	Entering fights without an escape
All-Rounder	Fight for extended periods	Balanced damage and durability	Assuming “balanced” means unkillable
Defender	Protect allies and disrupt enemies	High endurance, crowd control	Chasing kills instead of protecting the team
Supporter	Heal, shield or control fights	Team utility, recovery, status effects	Playing alone where support abilities have little value

Attacker

Attackers can deal heavy damage but usually have lower endurance. They are often easiest to understand mechanically: stay at a safe distance and attack whatever threatens the team.

However, an Attacker is not automatically the easiest role. Good positioning is essential. If you move ahead of the Defender, an opposing Speedster may knock you out before you can react.

Speedster

Speedsters are mobile Pokémon designed to enter quickly, deal burst damage and escape. They are frequently suited to the central area, where they can level efficiently and move to either path.

The role is attractive but unforgiving. A Speedster that enters at the wrong time may disappear immediately. Beginners should learn when to wait rather than treating every visible opponent as a target.

All-Rounder

All-Rounders combine offence and endurance. They can stay in fights longer than most Attackers and may become powerful after reaching important levels.

They are useful for players who enjoy direct combat, but each All-Rounder still has unique strengths. Some require careful move combinations; others need time to become strong.

Defender

Defenders protect teammates, absorb pressure and prevent opponents from moving freely. They may block routes, stun enemies, push them away or hold an area while allies deal damage.

The scoreboard may not always make their contribution obvious. A Defender that stops three opponents from reaching a goal can be more useful than a player chasing one unnecessary knockout.

Supporter

Supporters improve the performance of the complete team through healing, shields, movement effects or crowd control.

This role is strongest when teammates cooperate. Solo queue can occasionally be frustrating because a Supporter cannot force allies to remain nearby or act on an opening.

For a complete beginner, Defender, a durable All-Rounder or a straightforward ranged Attacker is normally easier than a mechanically demanding Speedster or highly team-dependent Supporter.

Paths and the Central Area

The main battlefield has a top path, bottom path and central area.

Top and Bottom Paths

Path players usually work in pairs. Their early responsibilities are to:

Defeat nearby wild Pokémon
Share experience sensibly
Protect their own goal zone
Look for safe opportunities to score
Avoid giving the opposing team easy knockouts
Rotate towards important objectives

Do not take every wild Pokémon from your partner, especially if your partner needs one more level to evolve or unlock a key move. A team gains more from two useful players than from one over-levelled player and one under-levelled teammate.

Central Area

The central player normally clears the wild Pokémon in the middle, gains levels quickly and then helps one of the paths.

This role carries responsibility. The central player should not remain hidden in the middle for the whole match. The faster experience is intended to create a temporary advantage that can help a path win a fight or secure neutral wild Pokémon.

Path players should also avoid taking the first central-area clear unless the team has agreed to a different strategy. Removing that experience can delay the central player’s move upgrades.

Held Items, Battle Items and Boost Emblems

Held Items

A Pokémon can equip up to three held items. These provide passive statistical bonuses or special effects.

Held items can support different goals, such as:

Increasing damage
Improving durability
Strengthening basic attacks
Providing shields or recovery
Supporting allies
Rewarding successful scoring

Held items can be upgraded outside battle. Because upgrade materials are limited early on, beginners should avoid raising every item equally.

Start with a small group of broadly useful items that match the Pokémon you actually play. Check the current recommended sets inside the game, because item strength and popular builds can change after balance updates.

The official overview confirms that players may equip up to three held items, as well as a separate battle item.

Battle Items

A battle item is an active tool with a cooldown. It may provide movement, healing, protection, scoring help or another temporary effect.

Use battle items deliberately. Do not press one merely because it is available. A mobility item saved for escaping an enemy Unite Move may be more valuable than using it to arrive at a minor wild Pokémon a second earlier.

Boost Emblems

Boost emblems provide additional loadout customisation outside battle. Their colour combinations and statistics can become complicated, but beginners do not need a perfect emblem collection before entering normal matches.

Use a recommended emblem loadout first. Learn positioning, farming and objectives before spending too much time chasing tiny statistical improvements.

Good decisions normally matter more than a slightly more efficient emblem page.

Unite Moves

A Unite Move is a Pokémon’s most powerful ability. It becomes available after reaching the required level and recharges after use.

Good situations for a Unite Move include:

A major objective fight
Protecting several teammates
Securing multiple knockouts
Escaping with a large amount of Aeos energy
Stopping a large final-stretch score
Creating a safe opportunity for the team to score

Poor situations include using it against one nearly defeated opponent shortly before a decisive objective appears.

The exact recharge time differs between Pokémon and can be influenced by battle actions or items. Rather than memorising one universal timer, watch the charge indicator and plan ahead.

As the final two minutes approach, ask one question: Will I have my Unite Move ready for the most important fight?

Best Beginner Strategy: What to Do During a Match

Opening: 10:00 to Around 8:00

Follow the path or central-area position you declared.
Defeat wild Pokémon efficiently.
Avoid stealing your partner’s entire share of experience.
Score small amounts when it is safe.
Do not chase opponents deep into their side of the map.
Watch for the first contested wild Pokémon and early rotations.

Your goal is not to become the match hero immediately. It is to reach important move and evolution levels without giving the opposing team free experience.

Middle: Around 8:00 to 5:00

Watch objective timers.
Move before an objective appears, not after the fight is already lost.
Use the minimap to check whether another path needs help.
Continue collecting experience between fights.
Score when opponents are visible elsewhere.
Avoid carrying maximum energy for too long.

This is where many beginners stop farming and begin wandering from fight to fight. Keep gaining levels. A knockout is only useful when it leads to experience, scoring, an objective or map control.

Late Middle: Around 5:00 to 2:00

Avoid unnecessary solo fights.
Save a Unite Move for the final stretch when practical.
Break or protect goal zones according to the score situation.
Continue watching central and side objectives.
Group more frequently with teammates.

Repeatedly being knocked out now may leave you under-levelled for the decisive fight.

Final Stretch: 2:00 to 0:00

Check whether your team is probably winning or losing.
Group near the final objective.
Do not attack it without knowing where opponents are.
Defeat or zone opponents before committing.
After gaining the advantage, score quickly.
Return to defend after scoring.

Do not celebrate before the timer reaches zero. One unattended player carrying 50 energy can score 100 points during the final stretch.

Ten Beginner Mistakes to Avoid

1. Treating Every Match Like a Knockout Competition

Knockouts help, but points decide the match. A player can have many knockouts and still lose by ignoring goals and objectives.

2. Ignoring the Minimap

The minimap shows teammates, visible opponents, goals and objective activity. Look at it every few seconds, especially before crossing into enemy territory.

3. Fighting on the Opponent’s Goal Zone

Enemy goal zones can help opponents recover and make them harder to defeat. Do not take an equal fight on terrain that favours them unless your team has a clear reason.

4. Chasing Too Far

A low-health opponent can be bait. Chasing across the map may separate you from the team, place you near respawning enemies and make you miss an objective.

5. Arriving Late to Objectives

Moving only after an objective appears often means arriving after the opposing team has established position. Watch the timer and rotate early.

6. Starting the Final Objective While Ahead

If your team is winning, attacking the objective may give the losing team an opportunity to steal it. Defending the area and forcing opponents to approach can be safer.

7. Scoring While Teammates Are Fighting Nearby

Sometimes a large score is correct. At other times, your absence turns an even team fight into a four-against-five loss. Read the situation before leaving.

8. Using the Same Build on Every Pokémon

An item that helps a physical basic attacker may offer little value to a special-attacking Supporter. Match the loadout to the Pokémon’s mechanics and role.

9. Refusing to Change Roles

You may prefer Attackers, but a team with four Attackers often needs durability or support. Learning one Pokémon from at least two roles makes team selection easier.

10. Blaming Teammates Instead of Reviewing Decisions

You cannot control matchmaking. You can control positioning, timing, map awareness, communication and whether you enter an unwinnable fight.

After a loss, identify one decision you could improve. That creates progress even when the complete team played poorly.

Solo Mode: A Better Place to Learn

New players who feel nervous about immediate player-versus-player matches can use Solo Mode. Introduced in 2026, this mode provides CPU stages, rewards and several types of challenges.

It is useful for:

Learning controls
Testing unfamiliar Pokémon
Practising move combinations
Understanding scoring
Trying held items
Learning the basic flow without pressure from teammates

Solo Mode should not be treated as perfect preparation for ranked play. Human opponents are less predictable, punish mistakes differently and coordinate in ways that CPU players may not.

Use Solo Mode to become comfortable, then move into casual battles before entering ranked matches.

Casual, Ranked and Quick Battles

Casual Battles

Casual battles use the main rules without placing the same emphasis on rank progression. They are suitable for learning new Pokémon and testing builds.

Players still want to win, so casual does not mean that objectives and teamwork should be ignored.

Ranked Battles

Ranked matches add competitive progression. Winning increases progress while losing can reduce it, depending on the current ranked system and performance rules.

Enter ranked play when you can:

Use at least two or three Pokémon confidently
Fill more than one role
Understand the current final objective
Watch the minimap consistently
Avoid unnecessary fights
Keep a useful loadout ready

Do not choose a Pokémon for the first time in ranked mode.

Quick Battles

Quick battles take place on smaller maps with shorter timers and modified rules. They are useful when you have little time or want a faster, less conventional match.

Quick-battle habits do not always transfer directly to standard ranked play, because the maps, team sizes and objectives may differ.

Rotating and Special Modes

Pokémon UNITE regularly introduces event modes and unusual rule sets. These can include cooperative battles, sports-like minigames or maps with unique mechanics.

They provide variety and are often a good place to earn event rewards, but availability can change. Check the current in-game event page instead of relying on an old schedule.

Is Pokémon UNITE Pay-to-Win?

The fairest answer is: money can accelerate access and buy cosmetics, while player skill, teamwork and decision-making still determine most matches—but the progression system has attracted reasonable criticism.

The game contains several currencies and reward systems. Depending on the current version, these may be used for:

Unlocking Unite licenses
Buying Holowear
Purchasing Trainer clothing
Upgrading held items
Unlocking battle-pass rewards
Participating in limited events

Real money can make some unlocks faster. However, spending money does not teach map awareness, positioning, move timing or objective control.

The most important beginner rule is not to buy everything immediately.

Instead:

Use free rotations and practice modes.
Choose two or three Pokémon you genuinely enjoy.
Complete beginner missions and the License Journey.
Invest upgrade resources in a small selection of useful held items.
Treat Holowear as cosmetic spending, not a gameplay requirement.
Set a fixed spending limit before opening the shop.

Parents should review purchase controls because the game uses optional purchases, event offers and cosmetic rewards that can encourage repeated spending.

What Pokémon UNITE Does Well

1. It Makes the MOBA Genre Easier to Approach

The game removes much of the intimidating complexity found in older MOBAs. There is no large in-match shop filled with recipes, and standard matches have a predictable ten-minute limit.

The basic goal—collect energy and score—is immediately understandable.

2. Pokémon Have Strong Identities

Pokémon do not feel like identical characters wearing different skins. Their movement, attacks, evolutions and Unite Moves reflect their personalities and familiar abilities.

Seeing a small first-stage Pokémon evolve during the match provides a satisfying sense of growth even though levels reset in the next battle.

3. Matches Stay Meaningful Until the End

The final stretch creates genuine tension. A losing team has a reason to continue, while a leading team cannot become careless.

This comeback system can be frustrating, but it prevents many matches from becoming hopeless halfway through.

4. Cross-Platform Play Is Convenient

Switch and mobile players can compete together. Cross-progression also allows a linked account to continue on different devices.

The Switch version offers physical controls and a television display. The mobile version is convenient and allows interface customisation around touch controls.

5. Team Composition Matters

The five-role system creates meaningful cooperation without requiring new players to memorise hundreds of items. A well-timed Defender or Supporter can change a fight as decisively as a high-damage Attacker.

6. The Roster and Modes Continue to Grow

New Pokémon, balance adjustments, events and map variations keep the game from remaining static. The official site continues to list new playable Pokémon and game updates in 2026.

Where Pokémon UNITE Can Be Frustrating

1. Solo Queue Is Unpredictable

One player may choose the wrong path, ignore objectives or repeatedly enter fights alone. Because teamwork matters, another player’s decisions can directly affect your result.

2. The Final Two Minutes Can Feel Too Powerful

A team that played well for most of the match can lose after one failed final objective fight. The comeback potential creates excitement but can also make the earlier portion feel less important than it should.

3. The Menus and Currencies Are Busy

New players may encounter missions, events, currencies, exchanges, item upgrades, passes and notification markers at the same time. The battle itself is easier to understand than the reward interface.

4. Unlocking a Large Roster Takes Time

Free players can build a useful roster, but collecting every license is a long-term project. New Pokémon may also be easier to obtain through specific events or paid currency during parts of their release period.

5. Balance Changes Can Make Old Guides Obsolete

Move damage, cooldowns, items, maps and objectives can change. A tier list or exact build may become outdated quickly.

This is why beginners should learn principles—positioning, farming, grouping and objective timing—rather than copying one “best build” forever.

Nintendo Switch or Mobile: Which Is Better?

Feature	Nintendo Switch	Mobile
Controls	Physical buttons and analogue stick	Touchscreen controls
Screen	TV or handheld display	Phone or tablet
Portability	Good	Excellent
Interface aiming	Comfortable after learning controls	Direct touch can feel quick and precise
Communication	Quick chat; text entry can be slower	Touch interface can make menus easier
Best for	Console players and longer sessions	Convenient play anywhere

Neither platform provides an automatic strategic advantage. Use the one whose controls feel more natural.

If you plan to use both, link the correct account carefully at the beginning. Account-linking mistakes can create separate progress profiles and may be difficult to reverse.

Recommended First-Hour Plan

Follow this sequence instead of entering ranked play immediately:

Complete every basic tutorial.
Open the settings and review targeting and camera controls.
Try several roles in practice or Solo Mode.
Select one durable Pokémon and one ranged Pokémon to learn.
Read every move description.
Use a recommended held-item set.
Play casual matches until scoring and objective timing feel natural.
Practise checking the minimap every few seconds.
Learn the current ranked map and final Legendary objective.
Enter ranked only after you can use more than one Pokémon comfortably.

This will not make you an expert in one hour, but it prevents the most common early mistakes.

Beginner Settings Worth Checking

The exact menu wording may differ between Switch and mobile, but review the following:

Advanced attack controls: Separate targeting for opposing players and wild Pokémon can reduce mistakes.
Targeting priority: Choose whether the game prioritises low-health targets, nearby targets or another rule.
Lock-on icons: Useful when several opponents are grouped together.
Move aiming: Adjust sensitivity and aim-assist behaviour.
Camera controls: Make it easy to inspect objectives and distant fights.
Move-learning selection: Decide whether upgrades should be manual or automatically selected after a delay.
Performance settings: Stable frame rate is more valuable than maximum visual quality on a weaker phone.
Quick-chat messages: Equip useful calls such as gathering at an objective, defending a goal or retreating.

Test settings in practice mode. Changing several controls during a real match can distract you from the team.

Is Pokémon UNITE Suitable for Children?

Pokémon UNITE is colourful and does not present realistic violence, but it is an online competitive game.

Parents should consider:

Matches involve other online players.
Voice chat and other communication features may be available subject to account, age and system restrictions.
Optional in-game purchases are present.
Limited-time rewards can encourage frequent play.
Competitive losses may be frustrating for younger players.
A stable internet connection is required.

Useful family rules include:

Require permission before any purchase.
Protect the platform account with a password or purchase PIN.
Disable or restrict voice communication where appropriate.
Agree on a match limit instead of an unclear time limit, because leaving midway disadvantages the complete team.
Explain that poor behaviour from another player should be muted or reported, not answered.

Solo Mode and CPU battles are good starting points for a child who wants to learn without pressure from other players.

Frequently Asked Questions

Is Pokémon UNITE free?

Yes. It is a free-to-start game with optional in-game purchases. You can download it and play without buying the game upfront.

Do I need Nintendo Switch Online?

Free-to-play Nintendo Switch games generally do not require a paid Nintendo Switch Online membership for online play. Check the current Nintendo store page and regional account terms in case platform policies change.

Can mobile and Switch players play together?

Yes. Pokémon UNITE supports cross-platform battles between Nintendo Switch and compatible mobile devices.

Can I use the same progress on Switch and mobile?

Yes, when the account is linked correctly through a supported Nintendo Account or Pokémon Trainer Club account. Follow the linking instructions before creating separate progress on both platforms.

How long is a Pokémon UNITE match?

A standard match lasts about ten minutes. Quick battles are shorter and special modes may use different timers.

Do Pokémon levels carry into the next match?

No. Match levels reset. Every battle begins with the Pokémon at a low level so both teams can develop during that match.

Do Pokémon evolve permanently?

No. Evolution during a Unite Battle is temporary. The Pokémon returns to its normal pre-battle state after the match.

Can I catch wild Pokémon?

You defeat wild Pokémon for experience and Aeos energy, but you do not permanently capture them. Playable Pokémon are unlocked through Unite licenses.

What is the easiest role for a beginner?

A durable Defender, a straightforward All-Rounder or a simple ranged Attacker is normally a good starting point. The best choice is one whose move descriptions and battle distance you understand.

Should beginners play the central area?

They can, but the position carries responsibility. The central player must clear efficiently, help paths and arrive at objectives. A path position may be easier while learning the map.

When should I start ranked matches?

Start when you understand scoring, can use several Pokémon, know the current map objectives and can follow a declared path without becoming lost.

Is the game pay-to-win?

Purchases can speed up some unlocks and buy cosmetics, but money cannot replace mechanical skill, positioning, teamwork or map awareness. The upgrade and currency systems can still create progression concerns, so free players should invest resources carefully.

What happens if there is a tie?

Under the established standard rule, the team that reached the tied final score first wins. Rules can vary in special modes.

Is voice chat required?

No. Quick-chat signals and minimap awareness are enough for ordinary play. Voice communication can help organised teams but should be used with appropriate privacy and parental controls.

Which Pokémon is currently the strongest?

There is no permanent answer. Balance patches, map rotations and team compositions change the metagame. Choose a Pokémon you can play consistently and check recent in-game recommendations or official update notes.

Is Pokémon UNITE still worth playing in 2026?

Yes, particularly for players who want short team battles, enjoy Pokémon or want a more approachable MOBA. The growing roster, Solo Mode, License Journey and rotating Theia Sky Ruins objectives give new players several ways to begin.

Final Verdict

Pokémon UNITE succeeds because it keeps the excitement of Pokémon growth while removing the slower structure of a traditional role-playing game.

A match begins simply: choose a Pokémon, defeat wild opponents and score energy. Under that simple surface are positioning, team composition, move timing, level advantages, objective control and final-stretch decisions.

The game is at its best when five players understand their roles and move together. It is at its worst when matchmaking produces a disorganised team or when one final objective reverses an otherwise controlled match.

The progression screens and currencies could be clearer, and collecting the complete roster requires patience. However, the actual battles remain fast, responsive and recognisably Pokémon.

Pokémon UNITE is recommended for:

Pokémon fans who want real-time team battles
Players curious about MOBAs but intimidated by longer games
Friends looking for a free cross-platform game
Players who enjoy learning several roles and strategies
Anyone who prefers fixed ten-minute matches

It may not suit:

Players looking for a traditional Pokémon adventure
People who dislike team-dependent competitive games
Players who want to unlock every character immediately
Anyone who cannot maintain a stable internet connection

The best way to judge it is to download it, complete the tutorials and play several Solo Mode or casual matches. There is no upfront purchase, and the core battle system becomes clear quickly.

Overall score: 8/10.

Pokémon UNITE does not let you catch them all in the usual way. It lets you build a roster, learn what makes each Pokémon valuable and discover how five very different partners can work as one team.

August 6, 2026

Best Practices for Migrations in Microservices: A Zero-Downtime Guide

Migrating a microservices system is rarely a single database command or one deployment.

A real migration may change several things at the same time:

Which service owns a business capability
Which database stores the data
How APIs communicate
Which events are published and consumed
How existing records are transformed
How traffic moves from old code to new code
How the team detects errors and recovers

The difficult part is not creating the new table, endpoint or service. The difficult part is keeping the system correct while old and new versions run together.

This guide explains the main best practices for planning and performing microservice migrations with minimal downtime and a practical recovery path.

Quick Answer

The safest microservice migrations are small, backward-compatible and observable.

Use this general sequence:

Discover the real dependencies and define the migration boundary.
Expand the system by adding compatible schemas, APIs or events.
Synchronise old and new paths while existing data is copied.
Verify data, behaviour, performance and security continuously.
Shift traffic gradually using flags, routing rules or canary releases.
Stabilise the new path while retaining a tested recovery option.
Contract by removing obsolete fields, endpoints, topics and code only after all consumers have moved.

This is often called expand and contract or parallel change. The important idea is that a breaking change is divided into several compatible releases instead of being performed all at once. Martin Fowler describes the three broad stages as expand, migrate and contract in his explanation of Parallel Change.

Risk	Safer practice
Big-bang replacement	Incremental strangler migration
Renaming or deleting a column immediately	Add, backfill, switch and remove later
Copying millions of rows in one transaction	Small resumable batches
Writing to a database and broker separately	Transactional outbox or CDC
Assuming messages arrive once	Idempotent consumers and deduplication
Changing an API response without checking clients	Backward-compatible changes and contract tests
Deploying all traffic to new code	Canary, percentage rollout or routing by tenant
Treating rollback as “deploy the old image”	Design data-aware rollback or roll-forward
Declaring success after deployment	Reconcile data and monitor business metrics

What Does “Migration” Mean in Microservices?

The word migration can describe several different operations.

1. Service Migration

A business capability is moved from a monolith or an existing service into a new microservice.

Example:

Monolith order module
        ↓
New Order Service

2. Database Schema Migration

The structure of a service-owned database changes.

Examples include:

Adding a column
Creating an index
Splitting one table into several tables
Changing a data type
Adding a constraint
Removing an obsolete table

3. Data Migration

Existing records are copied, transformed or reassigned to a different service.

Example:

monolith.customers
        ↓
customer_service.customers

4. API Migration

Consumers move from an old HTTP, gRPC or GraphQL contract to a new one.

5. Event Migration

Producers and consumers move to a new event name, payload, topic or schema version.

6. Infrastructure Migration

The service moves to a new cluster, region, cloud account, runtime or deployment platform.

These migrations often overlap. Extracting an Order Service, for example, can involve a new database, a historical backfill, a new API, new events, traffic routing and changes to several consumers.

That is why the migration should be managed as a sequence of controlled state transitions rather than one deployment ticket.

Why Microservice Migrations Are Difficult

In a monolith, code and database changes can sometimes be released together. In a microservices environment, independent deployment creates a mixed-version period:

Old application instances may still be serving traffic.
New instances may already use the new schema.
Some consumers may understand a new event field while others do not.
A mobile application may continue using an old API for months.
Delayed messages may contain an older event format.
Backfill jobs may still be copying historical records.
Cached data may represent the old ownership model.

The system must remain correct throughout this period.

A migration can therefore fail even when every individual component appears healthy. Typical failure modes include:

Lost writes during data copying
Duplicate messages during retries
Old consumers rejecting a new payload
Long database locks causing request timeouts
Partial data movement between services
Two databases disagreeing about the same entity
Rollback code being unable to understand data written by the new version
A silent business error that produces no infrastructure alert

The objective is not merely “the deployment completed.” The objective is that the new system is correct, recoverable and measurably better or at least equivalent.

1. Define the Boundary and Success Criteria First

Do not start with migration scripts. Start with the reason and the boundary.

Write a short migration definition containing:

The business capability being moved
The present owner and future owner
The source and destination data stores
All known API consumers
All event producers and consumers
Expected traffic and data volume
Availability and latency targets
Acceptable data lag during transition
Recovery point objective and recovery time objective
Compliance, audit and retention requirements
Clear completion and abort conditions

Example success criteria:

- 100% of order-creation traffic reaches Order Service.
- Reconciliation mismatch stays below 0.01% for seven days.
- No unexplained missing or duplicate orders exist.
- p95 creation latency remains below 400 ms.
- Error rate remains below 0.5%.
- Every known consumer has stopped reading the legacy order tables.
- The rollback or roll-forward runbook has been tested.

“Move orders to a microservice” is not a testable success criterion. Precise thresholds create an objective go/no-go decision.

2. Discover Dependencies Before Changing Ownership

Documentation is useful, but the real dependency graph may be larger than the documented one.

Inspect:

Source-code references
Database queries and stored procedures
Foreign keys, views and triggers
Scheduled jobs and ETL pipelines
Reports and business-intelligence tools
API gateway and reverse-proxy routes
Event topics and consumer groups
Cache keys
Search indexes
Data exports and partner integrations
Support and administrative tools
Manual operational procedures

Combine static discovery with runtime evidence:

Distributed traces
Database query logs
API access logs
Broker consumer information
Network telemetry
Audit logs

For every dependency, record:

Item	Example
Owner	Billing team
Dependency	Reads `orders.status` directly
Interface	Shared PostgreSQL table
Criticality	Required for invoice generation
Migration action	Replace with Order API or event projection
Deadline	Before legacy table becomes read-only
Verification	Compare invoice count and total value

Hidden database readers are especially dangerous. A service cannot truly own its data while unrelated services continue to query its tables directly.

3. Give Each Service Clear Data Ownership

The usual target is not necessarily one physical database server per service. The important rule is logical ownership:

One service controls the schema.
Other services do not write its tables.
Other services obtain information through published contracts.
Schema changes are made by the owning team.

Depending on scale and risk, services may use:

Separate database servers
Separate database instances
Separate databases on the same server
Separate schemas with strictly controlled permissions

The last option can be an intermediate step, but it requires access controls that prevent accidental cross-service queries.

Avoid a Distributed Monolith

Moving code into several repositories while retaining one freely shared database does not create independent services. It can produce the operational cost of microservices without the isolation benefits.

Common warning signs are:

Service A joins Service B’s tables.
Several services run migrations against the same schema.
A column cannot be changed without coordinating many teams.
Business rules are enforced by undocumented cross-service database triggers.
One database outage stops every service.

During an incremental migration, temporary sharing may be unavoidable. Make it explicit, time-limited and measurable. Track every temporary dependency to removal.

4. Prefer Incremental Migration Over a Big-Bang Rewrite

A full rewrite creates a long period in which the new system receives little production feedback. It also concentrates data, code and operational risk into one cutover.

The strangler fig pattern replaces capabilities gradually. A routing layer sends selected requests to the new service while the rest continue to use the legacy system. AWS describes this as a way to reduce transformation risk and business disruption during monolith-to-microservice migration in its Strangler Fig guidance.

A typical progression is:

Stage 1: All traffic → Legacy system
Stage 2: Selected operation or tenant → New service
Stage 3: Most traffic → New service, fallback retained
Stage 4: All traffic → New service
Stage 5: Legacy capability removed

Choose a first migration slice that has:

A clear business boundary
Manageable data volume
Few synchronous dependencies
Measurable outcomes
Real value even if later migrations pause

Do not automatically start with the smallest module. A technically small module with ten hidden consumers may be riskier than a larger but well-isolated capability.

5. Use Expand-and-Contract Database Changes

A destructive schema change should normally span multiple releases.

Suppose customers.full_name must become display_name.

The unsafe approach is:

ALTER TABLE customers
RENAME COLUMN full_name TO display_name;

Old application instances may still query full_name and fail immediately.

Use the following phases instead.

Phase A: Expand

Add the new structure without removing the old one:

ALTER TABLE customers
ADD COLUMN display_name VARCHAR(200) NULL;

Deploy code that can tolerate both schemas. Depending on the application and database, it may temporarily write both fields or use a compatibility layer.

Phase B: Backfill

Copy existing values in small batches:

UPDATE customers
SET display_name = full_name
WHERE id > :last_id
  AND id <= :batch_end_id
  AND display_name IS NULL;

Make the operation resumable. Store progress, limit the batch size and monitor database load.

Phase C: Switch Reads

Deploy code that reads display_name. Keep the old field available while older instances and consumers are retired.

Phase D: Verify

Check for missing or conflicting values:

SELECT COUNT(*) AS missing_values
FROM customers
WHERE display_name IS NULL;

If the two columns are expected to match during transition:

SELECT COUNT(*) AS mismatched_values
FROM customers
WHERE display_name IS DISTINCT FROM full_name;

Use the equivalent null-safe comparison for the selected database engine.

Phase E: Contract

Only after all code versions and consumers have moved should the old column be removed:

ALTER TABLE customers
DROP COLUMN full_name;

The contract stage may be days or weeks later. That delay is intentional.

6. Keep Schema Changes Backward-Compatible During Rolling Deployments

During a rolling deployment, old and new instances coexist. Both versions must work with the current schema.

Usually safer changes include:

Adding a nullable column
Adding a table that old code ignores
Adding an optional response or event field
Adding a non-unique index using the database’s online or concurrent mechanism

Potentially breaking changes include:

Dropping or renaming a column
Changing a field’s meaning
Changing a data type in place
Adding NOT NULL before data is populated
Replacing a default while old code depends on it
Tightening a constraint before invalid historical records are repaired

GitLab’s zero-downtime migration guidance highlights a subtle example: older processes can retain stale schema information and depend on a database default while a post-deployment migration removes it. The later insert can then fail. This illustrates why compatibility must be evaluated against every live application version, not just the new one. See Avoiding downtime in migrations.

Adding a Required Column Safely

Do not add a mandatory field and expect every existing row and old instance to satisfy it immediately.

Use this sequence:

Add the column as nullable or with a compatible temporary default.
Deploy writers that populate it.
Backfill historical rows.
Verify that no invalid rows remain.
Add the constraint using an engine-appropriate low-lock method.
Remove a temporary default only after old writers are gone.

Creating Indexes Carefully

Index creation on a large busy table can block writes or consume significant I/O and CPU.

PostgreSQL supports:

CREATE INDEX CONCURRENTLY idx_orders_customer_id
ON orders (customer_id);

The PostgreSQL documentation states that CREATE INDEX CONCURRENTLY can create an index without locking out writes, although it takes more work and has restrictions. Check the exact behaviour and failure recovery for the database version in use. See the official CREATE INDEX documentation.

Do not copy PostgreSQL syntax into MySQL, SQL Server or another engine. Each database has different online DDL capabilities and locking behaviour.

7. Separate Schema Migration from Large Data Backfills

A schema migration should normally be short and predictable. Updating millions of records inside the deployment migration can cause:

Long locks
Transaction-log growth
Replica lag
Increased I/O
Request latency
Deployment timeout
Difficult recovery after partial failure

Use a background migration for large data changes.

A good backfill worker should be:

Batched: processes a limited number of records at a time
Resumable: continues from a checkpoint after interruption
Idempotent: safely processes a record more than once
Throttled: slows down when database health degrades
Observable: exposes progress, failures and estimated completion
Auditable: records the migration version and transformation rule
Cancellable: can stop without corrupting partially processed data

GitLab recommends batched background migrations when data migrations exceed its normal migration time limits. Its batched background migration documentation is a useful production example.

Example Backfill Loop

repeat:
    claim the next ID range
    update no more than 1,000 rows
    commit
    record the checkpoint
    measure latency and replica lag
    pause if safety limits are exceeded
until no rows remain

Prefer a stable indexed cursor such as the primary key over deep OFFSET pagination. Offset queries become increasingly expensive and can behave unpredictably while rows are inserted or deleted.

Do not assume one batch size is correct for every environment. Tune it using production-like data and real monitoring.

8. Run Migrations as a Controlled Deployment Step

Avoid allowing every application replica to run migrations automatically at startup. If 20 pods start together, they may compete for locks, repeat work or leave the deployment in an unclear state.

A safer pattern is one controlled migration job that:

Acquires an advisory or migration lock.
Checks the expected current schema version.
Runs a bounded migration.
Records success or failure.
Releases the lock.
Allows the application rollout to proceed only when appropriate.

In Kubernetes, a Job represents a one-off task that runs to completion, according to the official Kubernetes Job documentation. Teams commonly use a dedicated Job or an equivalent release phase for migrations.

An init container can also run setup before an application container, but attaching a database migration to every pod startup can create concurrency and rollout problems. Choose the mechanism based on deployment guarantees, not convenience.

Additional controls should include:

One migration owner per service
Explicit timeouts
Lock timeouts
Statement timeouts
Preflight checks
An append-only migration history
Checksums or immutability for released migration files
Alerts on failed or unusually slow migrations

Never silently edit a migration that has already run in production. Add a new corrective migration.

9. Move Data with Snapshot, Change Capture and Reconciliation

Copying a table while the application is writing to it creates a moving target.

A robust live-data migration often uses four stages.

Stage 1: Initial Snapshot

Copy the historical records from the source to the destination.

Preserve stable identifiers where possible. Record the snapshot boundary, such as a log sequence number, timestamp or change-stream offset.

Stage 2: Capture Ongoing Changes

While the snapshot runs, capture inserts, updates and deletes that occur in the source.

Possible methods include:

Change data capture from the database log
A transactional outbox
Existing domain events, if their completeness has been proven
A temporary compatibility layer

Stage 3: Catch Up and Reconcile

Apply captured changes until lag is acceptably low. Then compare source and destination.

Useful checks include:

Total row count
Count by tenant, date or status
Sum of important monetary fields
Minimum and maximum identifiers
Null and constraint violations
Hashes of canonical record representations
Random and risk-based samples
Business outcomes, such as number and value of completed orders

A single total row count is not enough. Two datasets can have the same number of rows and different contents.

Stage 4: Cut Over Ownership

Move reads and writes to the destination gradually. Continue monitoring changes and discrepancies during a defined confidence period.

The cutover plan must state which system is authoritative at every stage. Avoid a period in which teams cannot answer, “If the two values disagree, which one wins?”

10. Avoid Unsafe Dual Writes

Consider this code:

save order in database
publish OrderCreated event

Two failures are possible:

The database commit succeeds but event publication fails.
The event is published but the database transaction later fails.

Retrying the whole operation can also create duplicates.

This is the dual-write problem. The AWS transactional outbox guidance explains how separate database and message operations can leave distributed systems inconsistent.

Transactional Outbox Pattern

Write the business record and an outbox record in the same local database transaction:

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    status,
    total_amount
) VALUES (
    :id,
    :customer_id,
    'created',
    :total_amount
);

INSERT INTO outbox_events (
    event_id,
    aggregate_type,
    aggregate_id,
    event_type,
    payload,
    created_at
) VALUES (
    :event_id,
    'order',
    :id,
    'OrderCreated',
    :payload,
    CURRENT_TIMESTAMP
);

COMMIT;

A separate relay or CDC connector publishes committed outbox rows to the broker. Debezium provides an Outbox Event Router specifically for this pattern.

The outbox improves reliability, but consumers must still expect duplicates. The relay may publish an event and fail before recording completion.

Idempotent Consumer

Give every event a stable ID. The consumer records processed IDs in the same transaction as its local state change:

BEGIN;

INSERT INTO processed_messages (
    consumer_name,
    event_id,
    processed_at
) VALUES (
    'billing-service',
    :event_id,
    CURRENT_TIMESTAMP
)
ON CONFLICT DO NOTHING;

-- Continue only if the insert affected one row.
-- Apply the business update here.

COMMIT;

The exact SQL differs by database. The important property is that deduplication and the local business update succeed or fail together.

11. Replace Cross-Service Transactions with Explicit Workflows

Once each service owns its database, a single ACID transaction normally cannot cover the complete business process.

For example:

Create order → Reserve stock → Authorise payment → Arrange shipment

Use a saga when several local transactions form one distributed business workflow.

A saga defines:

The forward steps
The responsible service for each step
Retry rules
Timeouts
Idempotency behaviour
Compensating actions
The point after which compensation is no longer appropriate
The final failure states visible to users and operators

Example compensation:

If payment fails after stock is reserved:
    release the reservation
    mark the order as payment_failed

Compensation is a business action, not a database rollback. Refunding a payment is a new auditable transaction; it does not erase the original charge.

The Azure Saga pattern guidance notes that retryable saga operations should be idempotent so a temporary failure does not create repeated side effects.

Use choreography for small, understandable event flows. Consider orchestration when the process has many participants, deadlines, complex compensation or a strong need for central visibility. Neither style removes the need for service autonomy, tracing and failure handling.

12. Evolve APIs Without Breaking Consumers

Service providers and consumers are rarely deployed at exactly the same time.

Prefer additive changes:

Add an optional field.
Add a new endpoint or operation.
Accept both old and new input representations temporarily.
Preserve existing status codes and field meanings.
Use tolerant readers where appropriate.

Treat these as breaking changes unless proven otherwise:

Removing or renaming a field
Changing a field from optional to required
Changing units, timezone or semantic meaning
Returning a different error code
Changing empty data from [] to null
Changing pagination behaviour
Reusing a field for a different purpose

Version Only When Necessary

API versioning can help manage unavoidable breaking changes, but it is not a replacement for compatibility discipline.

When a new version is necessary:

Publish a deprecation policy.
Identify every consumer.
Measure usage of the old version.
Provide a migration guide.
Set a realistic removal date.
Alert owners who still use the old contract.
Keep both versions until the exit criteria are met.

Use Consumer-Driven Contract Tests

Contract tests verify the behaviour that each consumer actually depends on. They can reject a provider deployment that breaks a known consumer before production.

Pact describes itself as a code-first consumer-driven contract testing tool and generates contracts from consumer tests. See the official Pact introduction.

Contract tests complement—not replace—unit, integration, security, performance and end-to-end tests.

13. Treat Event Schemas as Long-Lived Contracts

An event can remain in a topic, retry queue, archive or replay system long after the producer deployment.

Every event should have a clear envelope, for example:

{
  "event_id": "01JXYZ...",
  "event_type": "OrderCreated",
  "schema_version": 2,
  "occurred_at": "2026-08-06T08:30:00Z",
  "correlation_id": "req-12345",
  "producer": "order-service",
  "aggregate_id": "ord-9081",
  "payload": {
    "customer_id": "cus-42",
    "currency": "MYR",
    "total_amount": "129.90"
  }
}

Best practices include:

Use stable event IDs.
Include an event occurrence time, not only publication time.
Preserve correlation and causation information.
Add optional fields instead of changing existing meanings.
Never reuse an event name for different semantics.
Do not renumber or reuse removed Protobuf field numbers.
Validate compatibility in CI.
Test old consumers against new producer schemas.
Retain readers for historical formats when replay is required.

Confluent’s schema evolution documentation distinguishes backward, forward and full compatibility. Select the compatibility rule based on deployment and replay requirements rather than relying blindly on a registry default.

Events Should Describe Facts

Prefer:

OrderCancelled

over an ambiguous database-shaped event such as:

OrdersTableRowUpdated

A domain event should express what happened. Exposing internal table structure couples consumers to the producer’s storage model and makes future migrations harder.

14. Make Retries Safe

Retries are normal in distributed systems. Timeouts do not prove that an operation failed; the remote service may have completed it before the response was lost.

For commands that cause side effects, accept an idempotency key:

POST /payments
Idempotency-Key: checkout-874-payment-1

The receiving service stores the key and the result. A retry returns the original result or safely resumes the operation instead of creating another payment.

The same principle applies to:

Backfill batches
Event consumers
Saga steps
File imports
Webhook handlers
Traffic replay

Azure’s microservice assessment guidance describes deriving and storing idempotency keys so retried work can be detected and skipped safely. See Microservices assessment and readiness.

Do not claim “exactly once” merely because the broker offers an exactly-once feature. End-to-end behaviour also includes databases, external APIs and application side effects. Design the business operation itself to tolerate repetition.

15. Control Cutover with Flags and Routing

A migration should have a control plane that allows operators to change behaviour without an emergency rebuild.

Useful controls include:

Feature flag by tenant or user cohort
Percentage-based traffic routing
Route by endpoint
Route by region
Read-source selection
Write-path selection
Event-consumer enablement
Shadow traffic
Emergency kill switch

A sensible rollout might be:

Internal users → 1% → 5% → 25% → 50% → 100%

Promotion should depend on measurements, not a fixed timer alone.

At each stage, compare:

Error rate
Latency percentiles
Saturation
Data mismatch rate
Event lag
Duplicate rate
Business conversion or completion rate
Support incidents

Shadow Traffic

Shadowing sends a copy of production requests to the new service but does not use its response for the user.

It is useful for checking:

Compatibility
Performance
Result differences
Unexpected input shapes

Shadow requests must not create real side effects. Disable writes or direct them to an isolated destination. Remove or protect personal and secret data according to the organisation’s security requirements.

16. Build Observability Before Cutover

Do not wait for a migration failure to decide what should be measured.

Technical Metrics

Request rate, error rate and duration
Database CPU, I/O, locks and connection usage
Query latency
Replica lag
Queue depth and consumer lag
Retry and dead-letter counts
Backfill throughput and remaining records
CDC lag
Cache hit rate
Resource saturation

Migration Metrics

Percentage of traffic on the new path
Records copied
Records remaining
Source-destination mismatches
Writes handled by each path
Old API or event-schema usage
Number of unknown consumers
Compensation rate

Business Metrics

Orders created
Payments completed
Inventory reserved
Invoices generated
Total monetary value
User completion rate

A service can return HTTP 200 while calculating the wrong total. Business invariants often detect migration defects faster than infrastructure metrics.

Use consistent correlation IDs across HTTP calls, events, logs and saga steps. Distributed traces help reveal which path processed a request, but traces should be combined with metrics, logs and reconciliation reports.

17. Design Rollback Before Deployment

“Redeploy the previous image” is not a complete rollback plan.

After a migration begins, the new version may have:

Written data the old version cannot understand
Published events old consumers cannot parse
Moved ownership to another database
Triggered external side effects
Applied an irreversible schema change

For each phase, decide whether the response is:

Traffic rollback: route users back to the old path
Code rollback: deploy the previous compatible version
Data restore: restore a verified backup or point-in-time copy
Compensation: perform a business reversal
Roll-forward: fix the new path while preserving already committed state

Many production migrations are safer to roll forward after data has changed. The runbook should state the point of no return and the approved response after it.

A Useful Rollback Matrix

Phase	Failure	Response
Expand schema	Migration lock timeout	Stop migration; application remains on old schema
Backfill	High replica lag	Pause worker and resume from checkpoint later
Shadow traffic	Result mismatch	Keep responses non-authoritative; investigate
5% cutover	Error rate exceeds limit	Route traffic back to legacy path
New writes authoritative	Consumer defect	Fix or roll forward; replay retained events
Old column removed	Old code requested	Restore only if recovery plan supports it; otherwise roll forward

Test the recovery path in a production-like environment. An untested rollback plan is an assumption.

18. Secure the Migration Path

Temporary migration components often receive broad access and may be forgotten after cutover.

Apply normal production security controls:

Least-privilege source and destination credentials
Separate identities for schema migration, backfill and application runtime
Encrypted connections and storage
Secret rotation
Audit logging
Network restrictions
Masked or synthetic non-production data
Access expiry for temporary tools
Validation and safe parsing of imported records
Approval for destructive operations

Do not place credentials in migration scripts, container images or command history.

If personal data moves to a new service, update:

Data inventory
Retention and deletion processes
Data-subject request workflows
Encryption-key ownership
Backup and restore scope
Regional residency controls
Audit evidence

Migration completion includes removing temporary accounts, firewall rules, topics, buckets and data copies.

19. Test the Migration as a State Machine

Testing only the final architecture misses the riskiest part: intermediate states.

Test at least these states:

Old code + old schema
Old code + expanded schema
Old and new code + expanded schema
New code + partially backfilled data
New code + completed backfill
New code + contracted schema

Important test categories include:

Migration Tests

Upgrade from a realistic previous production version
Re-run an already completed migration
Interrupt and resume a backfill
Apply the migration to production-sized data
Verify lock and duration limits
Verify downgrade or roll-forward behaviour

Contract Tests

Old consumer against new provider
New consumer against old provider during the transition
Old event reader against new producer schema
Replay historical events through the new consumer

Failure Tests

Database unavailable during a batch
Broker publish timeout
Duplicate event delivery
Out-of-order event delivery
Consumer crash after business commit but before acknowledgement
CDC pause and recovery
Network partition during cutover
Partial destination outage

Reconciliation Tests

Intentional missing row is detected
Intentional field mismatch is detected
Duplicate record is detected
Monetary totals use correct precision and currency
Deleted records or tombstones are handled correctly

Use anonymised production-like volume and data distribution. A migration that completes in seconds on 1,000 test rows may behave very differently on 500 million production rows.

20. Create a Migration Runbook

The runbook should be executable by an engineer who did not design the migration.

Include:

Purpose and scope
Owners and communication channel
Architecture before and after
Dependency inventory
Preconditions
Backup and restore verification
Exact deployment sequence
Flags and routing controls
Validation queries
Dashboards and alert links
Go/no-go thresholds
Pause, abort and roll-forward instructions
Escalation contacts
Expected duration for each stage
Cleanup tasks
Evidence to retain

Conduct a rehearsal. Record actual timings and revise the runbook instead of assuming the first plan is accurate.

For a high-risk migration, assign explicit roles:

Migration lead
Database operator
Service owner
Observability lead
Business validator
Incident commander if thresholds are breached
Communications owner

This reduces confusion during a time-sensitive cutover.

Complete Example: Extracting an Order Service

Assume a monolith owns order creation, and the goal is a new Order Service with its own database.

Step 1: Discover

The team finds these dependencies:

Checkout creates orders.
Billing reads order totals.
Warehouse reads new orders from a scheduled query.
Customer support searches the monolith database.
Finance exports completed orders nightly.

The warehouse job and support search were not present in the original architecture diagram.

Step 2: Establish the Contract

Create:

POST /orders
GET /orders/{id}
OrderCreated
OrderStatusChanged

Add contract tests for checkout and billing. Design event fields as backward-compatible schemas.

Step 3: Prepare the Destination

Create the Order Service database and schema. Set service-specific credentials. Add dashboards, alerts and audit logs before sending production traffic.

Step 4: Backfill History

Copy orders in stable ID batches. Record each checkpoint and the source change-stream position.

For every batch:

Insert or update idempotently.
Compare count and monetary total.
Log invalid records separately.
Stop automatically if error or database-load thresholds are exceeded.

Step 5: Capture New Changes

Use CDC or a transactional outbox to transfer changes that occur after the snapshot boundary.

Measure:

Change lag
Failed records
Duplicate deliveries
Source-destination mismatch

Step 6: Shadow Reads

The production response still comes from the monolith, but the application also queries Order Service asynchronously and compares canonical results.

Differences are categorised instead of merely counted:

Expected formatting difference
Timing or eventual-consistency difference
Missing record
Incorrect status
Incorrect financial value
Unknown

Step 7: Move Consumers

Move billing, warehouse, support and finance to the published API or event projections. Deny new direct database integrations.

Monitor legacy access logs until no known consumer reads the old tables.

Step 8: Shift Reads

Move internal users first, followed by 1%, 5%, 25%, 50% and 100% of eligible traffic. Hold each stage until the defined error, latency and mismatch thresholds pass.

Step 9: Shift Writes

Make Order Service the authoritative writer. Ensure commands are idempotent and events use the transactional outbox.

Do not allow both systems to accept unrelated authoritative writes. If a transitional write path is needed, define one owner and one replication direction.

Step 10: Stabilise

Run at 100% while retaining the ability to route reads or commands according to the approved recovery plan. Continue reconciliation for the agreed confidence period.

Step 11: Contract and Clean Up

After exit criteria pass:

Remove legacy writes.
Revoke old database access.
Disable temporary CDC or replication if no longer required.
Archive or remove obsolete tables according to retention policy.
Remove old flags, routes and code.
Update architecture and operational documentation.
Record final evidence and lessons learned.

The migration is complete only after cleanup. Permanent compatibility code and unused data pipelines become future failure points.

Common Microservice Migration Mistakes

1. Performing the Schema and Code Change in One Release

This assumes every instance and consumer changes simultaneously. Rolling deployments and independent consumers make that assumption unsafe.

Use compatible stages.

2. Moving Code Without Moving Ownership

If the new service still depends on direct writes to the monolith’s tables, the old coupling remains.

Define a real system of record and remove cross-service writes.

3. Running a Huge Backfill During Deployment

Long data updates make deployment duration unpredictable and can overload the database.

Use a separate resumable background process.

4. Trusting Dual Writes Without Failure Analysis

Two writes to two systems are not atomic. A success response from one does not guarantee the other succeeded.

Use outbox, CDC or another design with explicit consistency guarantees.

5. Assuming Messages Are Delivered Once and in Order

Retries, rebalances and network failures can produce duplicates or reordering.

Use stable IDs, idempotency, version checks and ordering only where the business requires it.

6. Testing Only with Empty Databases

Empty-database tests do not reveal lock duration, invalid historical data, volume, skew or backfill behaviour.

Test upgrades using realistic data.

7. Monitoring Only CPU and HTTP Errors

Infrastructure can be healthy while orders, invoices or balances are wrong.

Monitor business invariants and reconcile data.

8. Keeping the Old Path Forever

Temporary routes, flags and dual-read code create permanent complexity when no removal date exists.

Set exit criteria and a cleanup owner before the migration begins.

9. Deleting Old Data Too Early

Early deletion eliminates recovery and comparison options.

Retain data according to a documented recovery, compliance and cost decision—not an improvised cleanup.

10. Calling a Backup a Rollback Plan

A backup is useful only if it is complete, recent, restorable and compatible with the intended recovery point.

Test restoration and measure how long it takes.

Migration Readiness Checklist

Planning

Business reason and scope are documented.
Current and future system of record are identified.
API, event, database, batch and reporting dependencies are inventoried.
Success, pause and abort thresholds are measurable.
Data volume and expected migration duration are estimated.
Security, privacy and retention requirements are reviewed.

Design

Changes are backward-compatible during mixed-version operation.
Expand, migrate and contract releases are separated.
Large backfills are batched, resumable and idempotent.
The dual-write problem is addressed.
Consumers tolerate duplicates and required reordering scenarios.
API and event contracts have a compatibility strategy.
The authoritative owner is clear at every phase.

Testing

Migration scripts run against production-like data volume.
Intermediate schema and application combinations are tested.
Contract tests cover known consumers.
Historical events can be replayed where required.
Backfill interruption and resumption are tested.
Reconciliation detects injected errors.
Rollback or roll-forward procedures are rehearsed.

Deployment

Migration execution is controlled and locked.
Backups and restoration are verified.
Flags or routing controls are ready.
Dashboards and alerts exist before cutover.
Correlation IDs connect requests, events and logs.
Owners and escalation paths are available.

Completion

Traffic is fully on the intended path.
Reconciliation meets the agreed threshold.
Old consumers and direct database readers are gone.
The confidence period has completed.
Obsolete schemas, routes, events and flags are removed.
Temporary credentials and infrastructure are revoked.
Documentation and incident lessons are updated.

Frequently Asked Questions

What is the safest migration strategy for microservices?

For most live systems, the safest general approach is an incremental migration using expand-and-contract changes, gradual traffic shifting, continuous reconciliation and explicit recovery controls. The exact tools depend on the database, broker and deployment platform.

What is an expand-and-contract migration?

It divides a breaking change into compatible stages. First, expand the system by adding the new field, endpoint or schema. Next, migrate consumers and data. Finally, contract the system by removing the obsolete interface after it is no longer used.

Can microservices share one database?

They can share physical infrastructure, especially during transition, but each service should have clearly controlled schema and data ownership. Other services should not directly write the owner’s tables. Separate credentials and permissions help enforce the boundary.

Should database migrations run when the application starts?

Small systems sometimes use this approach, but it becomes risky when many replicas start simultaneously. A controlled, single migration job with locking, timeouts and recorded status is usually easier to reason about.

How can a large table be migrated without downtime?

Add compatible destination structures, copy data in resumable batches, capture ongoing changes, reconcile source and destination, shift traffic gradually, then remove the old structure only after the new path is stable.

Is dual writing to old and new databases safe?

Not automatically. One write can succeed while the other fails. If temporary duplication is required, define the authoritative source, failure behaviour, repair process and reconciliation. Prefer patterns such as transactional outbox and CDC where they fit.

What is the difference between CDC and a transactional outbox?

CDC captures committed changes from a database log. A transactional outbox deliberately stores domain-event records in an outbox table within the same transaction as the business update; a relay or CDC tool then publishes them. Raw CDC exposes storage-level changes, while an outbox can publish intentional domain contracts.

How do you prevent duplicate event processing?

Give each event a stable ID and make consumers idempotent. Record the processed ID atomically with the local business change, or design the state transition so repeating it has no additional effect.

Should every breaking API change create a new version?

Not every change is breaking. Prefer additive, backward-compatible evolution. Create a new version when compatibility cannot reasonably be preserved, then publish a deprecation plan and measure old-version usage until all consumers migrate.

How should event schemas be versioned?

Choose explicit backward, forward or full compatibility based on producer and consumer rollout order and replay requirements. Enforce the rule in CI or a schema registry, retain stable field meanings and test older consumers with newer schemas.

What should be monitored during cutover?

Monitor technical signals such as errors, latency, locks and event lag; migration signals such as traffic percentage, backfill progress and mismatch rate; and business signals such as completed orders, payments and monetary totals.

Is rollback always possible?

No. After new data formats, external actions or destructive changes occur, deploying the old code may be unsafe. Define rollback and roll-forward options for each phase and identify the point after which roll-forward is the approved response.

When is a migration finished?

It is finished when the new owner handles the intended traffic, data has been reconciled, old consumers have moved, the confidence period has passed and temporary code, access, infrastructure and old schemas have been safely removed.

Final Summary

Reliable microservice migration is a compatibility and data-correctness problem, not merely a deployment task.

The most important practices are:

Discover real dependencies before changing ownership.
Define one authoritative owner at every stage.
Migrate incrementally instead of using a big-bang rewrite.
Use expand-and-contract changes for databases, APIs and events.
Keep large backfills separate, batched and resumable.
Use transactional outbox or CDC to avoid unsafe dual writes.
Make commands, consumers and migration jobs idempotent.
Reconcile data using business invariants, not only row counts.
Shift traffic gradually with measurable gates.
Design recovery before the first production change.
Remove temporary migration paths after a confidence period.

The ideal migration may look slower on a project plan because it uses several releases. In production, those smaller reversible steps usually reduce risk, reveal problems earlier and allow normal product development to continue while the architecture evolves.

August 6, 2026

PHP Form Validation for Kids and Beginners — Part 10

Web forms allow visitors to send information to a PHP program.

However, we should never assume that submitted information is complete, correct or safe to display.

A visitor might:

Leave a required box empty
Enter letters where a number is expected
Type an invalid email address
Select a value that our form does not allow
Enter a very long value by mistake
Change the form using browser developer tools
Send the request without using our webpage at all

This is why PHP programs need form validation.

In Part 10 of this PHP tutorial for kids and complete beginners, you will learn:

What form validation means
Why HTML validation is not enough by itself
How to check whether a form was submitted
How to safely read values from $_POST
How to check required fields
How to validate names, ages, email addresses and choices
How to show useful error messages
How to keep old values inside the form
How to escape information before displaying it
How to build a complete Kids’ Club registration form
How to organise repeated validation code with functions

The examples use beginner-friendly PHP 8 syntax and do not require a database.

Quick Answer

PHP form validation checks whether submitted information follows the rules of our program.

Here is a small example:

<?php

$name = "";
$nameError = "";

if ($_SERVER["REQUEST_METHOD"] === "POST") {
    $name = trim($_POST["name"] ?? "");

    if ($name === "") {
        $nameError = "Please enter your name.";
    }
}

?>

<form method="post">
    <label for="name">Name</label>

    <input
        type="text"
        id="name"
        name="name"
        value="<?php echo htmlspecialchars($name); ?>"
    >

    <p><?php echo $nameError; ?></p>

    <button type="submit">Submit</button>
</form>

The program:

Waits for a POST request.
Reads the submitted name.
Removes unnecessary spaces with trim().
Checks whether the name is empty.
Displays an error if the visitor did not enter a name.
Safely places the old value back inside the form.

We will improve every part of this example during the tutorial.

What Is Form Validation?

Form validation means checking submitted information before using it.

Imagine that we are creating a registration form with these rules:

Field	Rule
Name	Required and no longer than 50 characters
Email	Required and must look like an email address
Age	Must be a whole number from 7 to 17
Favourite colour	Must be one of the available choices
Agreement	Must be selected

If the submitted data follows all the rules, it is valid.

If one or more rules are broken, the program should not continue as though everything is correct. It should explain the problem and allow the visitor to fix it.

Good validation should answer three questions:

Is the value present?
Is it the correct type or format?
Is it acceptable for this particular program?

An age of 500 is made from numbers, but it is not acceptable for a children’s club. Format checking alone is therefore not enough.

Client-Side and Server-Side Validation

HTML can perform some validation inside the browser.

<input
    type="email"
    name="email"
    required
>

The required attribute asks the browser to prevent an empty submission. The email type asks the browser to check the basic email format.

This is called client-side validation because it happens in the visitor’s browser.

It is useful because the visitor receives quick feedback. However, it must not be our only validation.

A visitor or another program can:

Remove the HTML attributes
Change the page using developer tools
Turn off browser validation
Send a request directly to the PHP file
Submit extra values that never appeared in the original form

PHP validation runs on the server. This is called server-side validation.

Validation type	Runs where?	Main purpose
HTML or JavaScript	Visitor’s browser	Fast and friendly feedback
PHP	Web server	Final trusted decision

Use both when possible, but always let PHP make the final decision.

A Basic HTML Form

Create a new file named:

registration.php

Start with this form:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Kids' Club Registration</title>
</head>
<body>

    <h1>Kids' Club Registration</h1>

    <form method="post">
        <label for="name">Name</label>
        <input type="text" id="name" name="name">

        <label for="email">Email</label>
        <input type="email" id="email" name="email">

        <label for="age">Age</label>
        <input type="number" id="age" name="age">

        <button type="submit">Register</button>
    </form>

</body>
</html>

The form uses:

method="post"

When the button is pressed, the browser sends the values to PHP in a POST request.

Because the form has no action attribute, it submits to the same page.

Checking Whether the Form Was Submitted

The page is normally requested with GET when we first open it.

The form sends a POST request when it is submitted.

PHP provides the request method through:

$_SERVER["REQUEST_METHOD"]

We can check it like this:

<?php

if ($_SERVER["REQUEST_METHOD"] === "POST") {
    echo "The form was submitted.";
}

The strict comparison operator === checks both value and type.

For this form, our validation code should run only after a POST request. Without the if statement, the page could display errors before the visitor has entered anything.

Safely Reading Values from POST

A submitted text field can be read from $_POST:

$name = $_POST["name"];

However, this assumes that name definitely exists.

Someone can submit a request without that field. PHP may then show an “Undefined array key” warning.

Use the null coalescing operator instead:

$name = $_POST["name"] ?? "";

This means:

Use $_POST["name"] if it exists.
Otherwise, use an empty string.

For a simple beginner form, we can then remove spaces from the beginning and end:

$name = trim($_POST["name"] ?? "");

If a visitor enters:

   Aina

trim() changes it to:

Aina

A More Defensive Version

A request can be changed so that name is sent as an array instead of text. Passing an array into trim() causes a type error.

We can guard against that:

$nameInput = $_POST["name"] ?? "";

if (!is_string($nameInput)) {
    $nameInput = "";
}

$name = trim($nameInput);

This is slightly longer, but it prevents malformed input from crashing the validation code.

Later, we will place this repeated work inside a function.

Checking a Required Field

Create variables before processing the form:

<?php

$name = "";
$nameError = "";

if ($_SERVER["REQUEST_METHOD"] === "POST") {
    $nameInput = $_POST["name"] ?? "";

    if (!is_string($nameInput)) {
        $nameInput = "";
    }

    $name = trim($nameInput);

    if ($name === "") {
        $nameError = "Please enter your name.";
    }
}

We compare the cleaned name with an empty string:

if ($name === "")

If it is empty, we store an error message.

We can display the message beneath the input:

<label for="name">Name</label>

<input type="text" id="name" name="name">

<?php if ($nameError !== ""): ?>
    <p><?php echo $nameError; ?></p>
<?php endif; ?>

The paragraph appears only when an error exists.

Checking the Length of a Name

A required-field check prevents an empty value, but someone could still submit thousands of characters.

We can add a maximum length:

if ($name === "") {
    $nameError = "Please enter your name.";
} elseif (strlen($name) > 50) {
    $nameError = "Your name must be 50 characters or fewer.";
}

The elseif runs only when the first condition is false.

The browser can also receive the same limit:

<input
    type="text"
    id="name"
    name="name"
    maxlength="50"
    required
>

Remember that the HTML rule improves the experience, while PHP still enforces the trusted rule.

For multilingual production websites, developers often use mb_strlen() so that multibyte characters are counted properly. That function requires PHP’s mbstring extension. We use strlen() here to keep the first project simple.

Validating an Email Address

Checking whether an email contains @ is not enough.

This is too weak:

if (!str_contains($email, "@")) {
    $emailError = "Invalid email.";
}

PHP provides filter_var() with FILTER_VALIDATE_EMAIL:

if ($email === "") {
    $emailError = "Please enter an email address.";
} elseif (filter_var($email, FILTER_VALIDATE_EMAIL) === false) {
    $emailError = "Please enter a valid email address.";
}

Examples:

Value	Basic result
`[email protected]`	Valid format
`aina.example.com`	Invalid format
`aina@`	Invalid format
Empty string	Required-field error

Email validation checks the format. It does not prove that:

The mailbox exists
The address belongs to the visitor
The visitor can receive messages

A real registration system normally sends a verification link or code to confirm ownership.

Validating a Whole Number and Range

Our club accepts ages from 7 to 17.

We want to reject:

An empty value
Words such as twelve
Decimals such as 12.5
Ages lower than 7
Ages greater than 17

PHP can validate the number and range together:

$ageValue = filter_var(
    $age,
    FILTER_VALIDATE_INT,
    [
        "options" => [
            "min_range" => 7,
            "max_range" => 17
        ]
    ]
);

if ($age === "") {
    $ageError = "Please enter your age.";
} elseif ($ageValue === false) {
    $ageError = "Age must be a whole number from 7 to 17.";
}

The strict check is important:

$ageValue === false

Some valid filters can return values that PHP might otherwise treat as false. A strict comparison makes our intention clear.

The HTML input should also describe the range:

<input
    type="number"
    id="age"
    name="age"
    min="7"
    max="17"
    step="1"
    required
>

Again, the PHP check remains necessary.

Validating a Select Menu

Suppose the form offers three favourite colours:

<select id="colour" name="colour">
    <option value="">Choose a colour</option>
    <option value="red">Red</option>
    <option value="blue">Blue</option>
    <option value="green">Green</option>
</select>

It may appear impossible to submit another value, but a request can be edited.

Create a list of allowed values:

$allowedColours = [
    "red",
    "blue",
    "green"
];

Then validate the submitted value:

if (!in_array($colour, $allowedColours, true)) {
    $colourError = "Please choose one of the available colours.";
}

The final argument true tells in_array() to use strict comparison.

This method is sometimes called an allowlist. Instead of trying to imagine every bad value, we accept only the values our program understands.

Validating a Checkbox

Checkboxes behave differently from text inputs.

If a checkbox is not selected, its name is normally absent from $_POST.

HTML:

<input
    type="checkbox"
    id="agree"
    name="agree"
    value="yes"
>

<label for="agree">
    I agree to follow the club rules.
</label>

PHP:

$agreed = (
    isset($_POST["agree"])
    && $_POST["agree"] === "yes"
);

if (!$agreed) {
    $agreeError = "You must agree to the club rules.";
}

We do not merely check whether any value was submitted. We check for the exact allowed value yes.

Validation, Sanitisation and Escaping Are Different

These words are related, but they do not mean the same thing.

Job	Question	Example
Validation	Is this value acceptable?	Is age an integer from 7 to 17?
Sanitisation	Should the value be transformed or cleaned?	Remove surrounding spaces with `trim()`
Escaping	How can this value be safely placed into a particular output?	Convert special HTML characters before displaying text

Consider this name:

<script>alert('Hello')</script>

If we directly place it into HTML, the browser may treat it as code.

Do not output untrusted text like this:

echo $_POST["name"];

Escape it for HTML:

echo htmlspecialchars(
    $name,
    ENT_QUOTES | ENT_SUBSTITUTE,
    "UTF-8"
);

The browser then displays the characters as text instead of interpreting them as an HTML tag.

It is helpful to create a short function:

function e(string $value): string
{
    return htmlspecialchars(
        $value,
        ENT_QUOTES | ENT_SUBSTITUTE,
        "UTF-8"
    );
}

Now we can write:

<?php echo e($name); ?>

Escaping depends on where the value is going. HTML text, HTML attributes, URLs, JavaScript, SQL and terminal commands have different rules. htmlspecialchars() is for HTML output; it is not a database security function.

Creating a Sticky Form

A sticky form keeps valid submitted values after an error.

Without a sticky form, a visitor who makes one mistake may need to type everything again.

For a text input:

<input
    type="text"
    id="name"
    name="name"
    value="<?php echo e($name); ?>"
>

For a select menu:

<option
    value="blue"
    <?php echo $colour === "blue" ? "selected" : ""; ?>
>
    Blue
</option>

For a checkbox:

<input
    type="checkbox"
    id="agree"
    name="agree"
    value="yes"
    <?php echo $agreed ? "checked" : ""; ?>
>

Do not place raw submitted text into an HTML value attribute. Always escape it first.

Passwords are a common exception: password boxes are normally cleared rather than refilled after an error.

Storing Errors in an Array

Separate variables such as $nameError and $emailError work, but a larger form becomes easier to manage with an array.

$errors = [];

Add an error using the field name as the key:

if ($name === "") {
    $errors["name"] = "Please enter your name.";
}

Display one field’s error:

<?php if (isset($errors["name"])): ?>
    <p class="error">
        <?php echo e($errors["name"]); ?>
    </p>
<?php endif; ?>

Check whether the entire form is valid:

if ($errors === []) {
    $success = true;
}

An empty error array means that no validation rule failed.

Reusing Code with a Helper Function

Several text fields need the same safe reading steps.

We can create a function:

function postText(string $key): string
{
    $value = $_POST[$key] ?? "";

    if (!is_string($value)) {
        return "";
    }

    return trim($value);
}

Now we can read values with:

$name = postText("name");
$email = postText("email");
$age = postText("age");
$colour = postText("colour");

This makes the main validation section shorter and gives every text field consistent basic handling.

The function does not decide whether a value is valid. It only returns a trimmed string or an empty string. Each field still has its own rules.

Complete Project: Kids’ Club Registration Form

The following project combines all the lessons into one file.

Save it as:

registration.php

<?php

function e(string $value): string
{
    return htmlspecialchars(
        $value,
        ENT_QUOTES | ENT_SUBSTITUTE,
        "UTF-8"
    );
}

function postText(string $key): string
{
    $value = $_POST[$key] ?? "";

    if (!is_string($value)) {
        return "";
    }

    return trim($value);
}

$name = "";
$email = "";
$age = "";
$colour = "";
$agreed = false;

$errors = [];
$success = false;

$allowedColours = [
    "red",
    "blue",
    "green"
];

if ($_SERVER["REQUEST_METHOD"] === "POST") {
    $name = postText("name");
    $email = postText("email");
    $age = postText("age");
    $colour = postText("colour");

    $agreed = (
        isset($_POST["agree"])
        && $_POST["agree"] === "yes"
    );

    if ($name === "") {
        $errors["name"] = "Please enter your name.";
    } elseif (strlen($name) > 50) {
        $errors["name"] = (
            "Your name must be 50 characters or fewer."
        );
    }

    if ($email === "") {
        $errors["email"] = (
            "Please enter an email address."
        );
    } elseif (
        filter_var(
            $email,
            FILTER_VALIDATE_EMAIL
        ) === false
    ) {
        $errors["email"] = (
            "Please enter a valid email address."
        );
    }

    if ($age === "") {
        $errors["age"] = "Please enter your age.";
    } else {
        $validAge = filter_var(
            $age,
            FILTER_VALIDATE_INT,
            [
                "options" => [
                    "min_range" => 7,
                    "max_range" => 17
                ]
            ]
        );

        if ($validAge === false) {
            $errors["age"] = (
                "Age must be a whole number from 7 to 17."
            );
        }
    }

    if (!in_array($colour, $allowedColours, true)) {
        $errors["colour"] = (
            "Please choose an available colour."
        );
    }

    if (!$agreed) {
        $errors["agree"] = (
            "You must agree to follow the club rules."
        );
    }

    if ($errors === []) {
        $success = true;
    }
}

?>
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Kids' Club Registration</title>

    <style>
        * {
            box-sizing: border-box;
        }

        body {
            margin: 0;
            padding: 30px 16px;
            background: #f2f6ff;
            color: #1f2937;
            font-family: Arial, sans-serif;
        }

        .card {
            width: 100%;
            max-width: 620px;
            margin: 0 auto;
            padding: 28px;
            border-radius: 16px;
            background: #ffffff;
            box-shadow: 0 10px 30px rgba(0, 0, 0, 0.08);
        }

        h1 {
            margin-top: 0;
        }

        .field {
            margin-bottom: 20px;
        }

        label,
        legend {
            display: block;
            margin-bottom: 7px;
            font-weight: bold;
        }

        input[type="text"],
        input[type="email"],
        input[type="number"],
        select {
            width: 100%;
            padding: 11px;
            border: 1px solid #9ca3af;
            border-radius: 8px;
            font: inherit;
        }

        fieldset {
            padding: 0;
            border: 0;
        }

        .checkbox-row {
            display: flex;
            gap: 9px;
            align-items: flex-start;
        }

        .checkbox-row label {
            margin: 0;
            font-weight: normal;
        }

        .error {
            margin: 7px 0 0;
            color: #b91c1c;
            font-size: 0.92rem;
        }

        .error-summary {
            margin-bottom: 22px;
            padding: 14px;
            border: 1px solid #f87171;
            border-radius: 8px;
            background: #fef2f2;
        }

        .success {
            padding: 18px;
            border: 1px solid #34d399;
            border-radius: 8px;
            background: #ecfdf5;
        }

        button {
            padding: 11px 18px;
            border: 0;
            border-radius: 8px;
            background: #2563eb;
            color: white;
            font: inherit;
            font-weight: bold;
            cursor: pointer;
        }

        button:hover {
            background: #1d4ed8;
        }
    </style>
</head>
<body>

<main class="card">
    <h1>Kids' Club Registration</h1>

    <?php if ($success): ?>
        <div class="success">
            <h2>Registration received!</h2>

            <p>
                Welcome, <?php echo e($name); ?>.
                Your form passed all the validation checks.
            </p>

            <p>
                A real website could now save the information
                or send a confirmation message.
            </p>
        </div>
    <?php else: ?>

        <?php if ($errors !== []): ?>
            <div class="error-summary">
                <strong>Please correct the form.</strong>

                <ul>
                    <?php foreach ($errors as $error): ?>
                        <li><?php echo e($error); ?></li>
                    <?php endforeach; ?>
                </ul>
            </div>
        <?php endif; ?>

        <form method="post" novalidate>
            <div class="field">
                <label for="name">Name</label>

                <input
                    type="text"
                    id="name"
                    name="name"
                    maxlength="50"
                    value="<?php echo e($name); ?>"
                    required
                >

                <?php if (isset($errors["name"])): ?>
                    <p class="error">
                        <?php echo e($errors["name"]); ?>
                    </p>
                <?php endif; ?>
            </div>

            <div class="field">
                <label for="email">Email address</label>

                <input
                    type="email"
                    id="email"
                    name="email"
                    value="<?php echo e($email); ?>"
                    required
                >

                <?php if (isset($errors["email"])): ?>
                    <p class="error">
                        <?php echo e($errors["email"]); ?>
                    </p>
                <?php endif; ?>
            </div>

            <div class="field">
                <label for="age">Age</label>

                <input
                    type="number"
                    id="age"
                    name="age"
                    min="7"
                    max="17"
                    step="1"
                    value="<?php echo e($age); ?>"
                    required
                >

                <?php if (isset($errors["age"])): ?>
                    <p class="error">
                        <?php echo e($errors["age"]); ?>
                    </p>
                <?php endif; ?>
            </div>

            <div class="field">
                <label for="colour">Favourite colour</label>

                <select id="colour" name="colour" required>
                    <option value="">Choose a colour</option>

                    <option
                        value="red"
                        <?php echo $colour === "red" ? "selected" : ""; ?>
                    >
                        Red
                    </option>

                    <option
                        value="blue"
                        <?php echo $colour === "blue" ? "selected" : ""; ?>
                    >
                        Blue
                    </option>

                    <option
                        value="green"
                        <?php echo $colour === "green" ? "selected" : ""; ?>
                    >
                        Green
                    </option>
                </select>

                <?php if (isset($errors["colour"])): ?>
                    <p class="error">
                        <?php echo e($errors["colour"]); ?>
                    </p>
                <?php endif; ?>
            </div>

            <fieldset class="field">
                <legend>Club rules</legend>

                <div class="checkbox-row">
                    <input
                        type="checkbox"
                        id="agree"
                        name="agree"
                        value="yes"
                        <?php echo $agreed ? "checked" : ""; ?>
                    >

                    <label for="agree">
                        I agree to be kind and follow the club rules.
                    </label>
                </div>

                <?php if (isset($errors["agree"])): ?>
                    <p class="error">
                        <?php echo e($errors["agree"]); ?>
                    </p>
                <?php endif; ?>
            </fieldset>

            <button type="submit">Register</button>
        </form>
    <?php endif; ?>
</main>

</body>
</html>

Why Does the Form Use novalidate?

The project contains:

<form method="post" novalidate>

novalidate temporarily disables the browser’s automatic validation. This makes it easier to test and see our PHP error messages during the lesson.

After you understand the PHP checks, remove novalidate:

<form method="post">

The final form will then use both browser validation and PHP validation.

How the Complete Project Works

1. Helper Functions Are Created

The e() function escapes text for HTML:

function e(string $value): string

The postText() function safely reads a submitted string:

function postText(string $key): string

2. Default Values Are Prepared

Before the form is submitted, the field values are empty and success is false:

$name = "";
$errors = [];
$success = false;

This also prevents undefined-variable warnings in the HTML section.

3. PHP Waits for POST

if ($_SERVER["REQUEST_METHOD"] === "POST")

The validation runs only after submission.

4. Every Field Is Checked

Each field has rules that match its purpose:

Name: required and maximum length
Email: required and valid format
Age: required integer in a permitted range
Colour: must appear in the allowlist
Agreement: exact checkbox value required

5. Success Requires No Errors

if ($errors === []) {
    $success = true;
}

The program does not accept the form if even one error remains.

6. The Page Shows One of Two Views

If $success is true, the confirmation is shown.

Otherwise, the form is shown with its previous values and any error messages.

How to Run the Project

Open a terminal in the folder containing registration.php.

Run PHP’s built-in development server:

php -S localhost:8000

Open this address in your browser:

http://localhost:8000/registration.php

Stop the development server by pressing:

Ctrl + C

The built-in server is useful for learning and local development. It is not intended to be a public production web server.

Testing the Validation

Do not test only the happy path. Try values that should fail.

Test	Expected result
Submit everything empty	Required-field errors appear
Name contains spaces only	Name error appears after `trim()`
Name is longer than 50 characters	Length error appears
Email is `hello`	Email-format error appears
Age is `6`	Range error appears
Age is `18`	Range error appears
Age is `12.5`	Whole-number error appears
No colour chosen	Colour error appears
Agreement not selected	Agreement error appears
One field is wrong	Other valid values remain in the form
Name contains `<b>Aina</b>`	Tags appear as text, not formatted HTML
Every field is valid	Success message appears

You can also use browser developer tools to change a colour option to an unexpected value. PHP should reject it because it is not in $allowedColours.

Testing invalid input is part of programming. It helps prove that the program follows its rules when users make mistakes or requests are modified.

Common PHP Form Validation Mistakes

1. Trusting required by Itself

This improves browser validation:

<input name="name" required>

It does not replace PHP validation. A request can bypass the HTML page.

2. Reading Missing Keys Directly

Risky:

$name = $_POST["name"];

Safer:

$name = $_POST["name"] ?? "";

For defensive code, also confirm that the value is a string.

3. Showing Submitted Data Without Escaping

Unsafe:

echo $_POST["name"];

Safer for HTML output:

echo htmlspecialchars(
    $name,
    ENT_QUOTES | ENT_SUBSTITUTE,
    "UTF-8"
);

4. Checking Only the Data Type

This checks whether age is an integer:

filter_var($age, FILTER_VALIDATE_INT)

The program must still check whether the integer is in the allowed range.

5. Trusting Select and Radio Values

A menu is not a security boundary. Validate its value against an allowlist on the server.

6. Using empty() Without Understanding It

empty() treats several different values as empty, including the string "0".

For required text, an explicit check is often clearer:

if ($value === "")

This is especially important if 0 might be a valid answer.

7. Removing Characters Instead of Defining Rules

Silently deleting unexpected characters may change a person’s real name or other important data.

Whenever possible:

Trim unnecessary surrounding spaces.
Validate the value according to clear rules.
Show an error if the value is unacceptable.
Escape it for the output context when displayed.

8. Confusing Email Format with Email Ownership

FILTER_VALIDATE_EMAIL checks the format. Verification normally requires sending a link or code to the address.

9. Saving Data Before All Checks Pass

Do not write to a file or database and then discover that another field is invalid.

The usual order is:

Read the request.
Validate all fields.
If errors exist, show the form again.
If there are no errors, perform the intended action.

10. Believing Validation Solves Every Security Problem

Validation is one part of secure form handling.

A production form that changes data may also need:

CSRF protection
Authentication and authorisation
Safe database queries using prepared statements
Rate limiting
Secure file-upload rules
Password hashing
HTTPS
Server-side logging without exposing private information

We will meet several of these ideas in later lessons.

Validation Rules Should Match the Project

There is no universal rule that every name must contain only letters.

Real names can contain:

Spaces
Hyphens
Apostrophes
Accented letters
Characters from many writing systems

A rule such as this can reject genuine names:

if (!preg_match("/^[a-zA-Z]+$/", $name)) {
    // This is too restrictive for many real names.
}

For a beginner registration form, requiring a non-empty name and a reasonable maximum length is often more suitable.

Validation is a program-design decision. Ask what the application truly needs rather than copying a strict rule from an unrelated tutorial.

Practice Exercises

Try these tasks before reading the answers.

Exercise 1: Username

Create a username field with these rules:

Required
At least 3 characters
No more than 20 characters

Exercise 2: Lucky Number

Create a lucky-number field that accepts only whole numbers from 1 to 100.

Exercise 3: Favourite Animal

Create a select menu containing:

Cat
Dog
Rabbit

Validate the result with an allowlist.

Exercise 4: Sticky Nickname

Create a nickname input. If another field has an error, keep the submitted nickname inside the form and escape it safely.

Exercise 5: Error List

Store three errors inside an array and use foreach to display them as an HTML unordered list.

Exercise 6: Add a Hobby

Add a hobby field to the complete project. It should be optional but no longer than 100 characters.

Exercise Answers

Answer 1

$username = postText("username");

if ($username === "") {
    $errors["username"] = "Please enter a username.";
} elseif (strlen($username) < 3) {
    $errors["username"] = (
        "The username must contain at least 3 characters."
    );
} elseif (strlen($username) > 20) {
    $errors["username"] = (
        "The username must contain no more than 20 characters."
    );
}

Answer 2

$luckyNumber = postText("lucky_number");

$validLuckyNumber = filter_var(
    $luckyNumber,
    FILTER_VALIDATE_INT,
    [
        "options" => [
            "min_range" => 1,
            "max_range" => 100
        ]
    ]
);

if ($validLuckyNumber === false) {
    $errors["lucky_number"] = (
        "Enter a whole number from 1 to 100."
    );
}

Answer 3

$animal = postText("animal");

$allowedAnimals = [
    "cat",
    "dog",
    "rabbit"
];

if (!in_array($animal, $allowedAnimals, true)) {
    $errors["animal"] = "Please choose a valid animal.";
}

Answer 4

<input
    type="text"
    id="nickname"
    name="nickname"
    value="<?php echo e($nickname); ?>"
>

The value must be escaped before it is placed inside the HTML attribute.

Answer 5

$errors = [
    "Please enter your name.",
    "Please enter a valid email address.",
    "Please choose a colour."
];

<ul>
    <?php foreach ($errors as $error): ?>
        <li><?php echo e($error); ?></li>
    <?php endforeach; ?>
</ul>

Answer 6

$hobby = postText("hobby");

if (strlen($hobby) > 100) {
    $errors["hobby"] = (
        "Your hobby must be 100 characters or fewer."
    );
}

There is no required-field check because an empty hobby is allowed.

PHP Form Validation Cheat Sheet

Check for POST

if ($_SERVER["REQUEST_METHOD"] === "POST") {
    // Validate submitted values.
}

Read a Value with a Default

$name = $_POST["name"] ?? "";

Trim a String Safely

$value = $_POST["field"] ?? "";

if (!is_string($value)) {
    $value = "";
}

$value = trim($value);

Check Required Text

if ($value === "") {
    $errors["field"] = "This field is required.";
}

Check Maximum Length

if (strlen($value) > 50) {
    $errors["field"] = "Use 50 characters or fewer.";
}

Validate an Email

if (filter_var($email, FILTER_VALIDATE_EMAIL) === false) {
    $errors["email"] = "Enter a valid email address.";
}

Validate an Integer Range

$number = filter_var(
    $value,
    FILTER_VALIDATE_INT,
    [
        "options" => [
            "min_range" => 1,
            "max_range" => 100
        ]
    ]
);

if ($number === false) {
    $errors["number"] = "Enter a whole number from 1 to 100.";
}

Validate an Allowed Choice

if (!in_array($choice, $allowedChoices, true)) {
    $errors["choice"] = "Choose an available option.";
}

Escape HTML Output

echo htmlspecialchars(
    $value,
    ENT_QUOTES | ENT_SUBSTITUTE,
    "UTF-8"
);

Check Whether Validation Passed

if ($errors === []) {
    echo "The form is valid.";
}

Frequently Asked Questions

What is PHP form validation?

PHP form validation checks submitted values on the server before the program accepts or uses them. It can check required fields, formats, lengths, ranges and allowed choices.

Why is HTML required not enough?

HTML validation happens in the browser and can be changed or bypassed. PHP must repeat the important checks on the server.

What does trim() do?

trim() removes whitespace from the beginning and end of a string. It is useful when a visitor accidentally enters extra spaces.

What does the double question mark mean?

The null coalescing operator ?? provides a default value when an array key does not exist or contains null.

$name = $_POST["name"] ?? "";

Should I use GET or POST for a registration form?

Use POST for a form that submits registration information or causes a change. GET is more suitable for retrieval operations such as searches and filters whose parameters can appear in the URL.

POST values not appearing in the address bar does not automatically make them encrypted. Use HTTPS for information travelling between the browser and server.

Does filter_var() clean all dangerous input?

No. filter_var() performs the filter you specifically request. For example, FILTER_VALIDATE_EMAIL checks email format. It does not automatically make the value safe for HTML, SQL or every other use.

Why use htmlspecialchars()?

It converts special characters so the browser treats them as text in HTML output rather than interpreting them as markup. Use it when displaying untrusted text in HTML.

Is validation the same as escaping?

No. Validation decides whether a value follows the program’s rules. Escaping safely represents a value in a particular output context. A valid name must still be escaped before being inserted into HTML.

What is a sticky form?

A sticky form keeps previously submitted values after validation fails. It prevents visitors from having to re-enter every correct field.

Why validate a select menu?

A visitor can modify HTML or send a request directly. PHP should accept only values that the server recognises.

Should error messages explain the exact problem?

For normal validation errors, yes. “Age must be a whole number from 7 to 17” is more useful than “Invalid input.” Security-sensitive systems may use more general messages for information such as login failures.

Can I save the successful form to a database now?

You can after learning safe database access. Use PDO prepared statements, validate the data first, and never build SQL by joining raw submitted values into a query.

What is CSRF protection?

Cross-site request forgery protection helps prove that a state-changing form was intentionally submitted from your application. It normally uses a random token stored in a server-side session. It becomes important when forms create, update or delete real data.

Why does the example not save anything?

Part 10 focuses on validation. Keeping the first project independent of a database makes it easier to see the complete request, validation and response process.

Final Summary

Form validation protects the rules and reliability of a PHP application.

In this tutorial, we learned:

The browser can help with validation, but PHP makes the final decision.
$_SERVER["REQUEST_METHOD"] tells us how the page was requested.
The ?? operator prevents warnings when a submitted key is missing.
is_string() can reject malformed array input before string functions are used.
trim() removes unnecessary surrounding spaces.
Required fields can be checked against an empty string.
Length rules prevent unexpectedly large values.
filter_var() can validate email addresses and integers.
Number fields often need both a type check and a range check.
Select, radio and checkbox values must also be checked on the server.
An allowlist accepts only choices our program understands.
Validation, sanitisation and output escaping have different purposes.
htmlspecialchars() safely represents untrusted text in HTML.
Sticky forms improve the experience after an error.
An error array makes larger forms easier to organise.
A form should continue only when every important check passes.

The best way to learn validation is to test both correct and incorrect values. Try missing fields, wrong formats, unexpected choices and very long text, then confirm that the program responds clearly without crashing.

In Part 11, we can learn how PHP sessions and cookies remember information between different page requests. We can then use a session to add a CSRF token and create safer multi-page projects.

August 6, 2026

AI-Powered Image Captioning System with PHP: Complete Beginner Tutorial

Image captioning allows a computer to examine a picture and produce a written description of what it sees.

Instead of manually writing captions for every uploaded image, we can build a PHP application that sends the image to an AI vision model and receives a useful caption in return.

In this tutorial, we will build a complete AI-powered image captioning system using PHP, HTML, CSS, JavaScript and the OpenAI Responses API.

The finished application will allow a user to:

Select a JPG, PNG or WebP image
Preview the image before submitting it
Choose a caption style
Upload the image securely to PHP
Send the image to an AI model
Display the generated caption
Copy the caption with one click
Handle invalid files and API errors safely

The project uses one PHP file, so it is suitable for beginners and can run on XAMPP, Laragon, MAMP or a normal PHP web server.

Important: This project uses a paid API. API usage is separate from a ChatGPT subscription. Check the current model availability and pricing before deploying it publicly.

Quick Answer

The basic process is:

The browser sends an image to PHP.
PHP checks the file size and MIME type.
PHP converts the image into a Base64 data URL.
PHP sends the image and caption instructions to a vision-capable AI model.
The model returns text describing the image.
PHP safely displays the caption in the browser.

The central API request looks like this:

$payload = [
    "model" => "gpt-5-nano",
    "input" => [
        [
            "role" => "user",
            "content" => [
                [
                    "type" => "input_text",
                    "text" => $prompt
                ],
                [
                    "type" => "input_image",
                    "image_url" => $imageDataUrl,
                    "detail" => "auto"
                ]
            ]
        ]
    ],
    "max_output_tokens" => 300,
    "store" => false
];

OpenAI’s current image-input guide confirms that the Responses API accepts images as fully qualified URLs, file IDs or Base64 data URLs. The model used in this tutorial accepts image input and returns text. See the official images and vision guide and GPT-5 nano model page.

What Is AI Image Captioning?

AI image captioning combines computer vision and natural-language generation.

Computer vision helps the model examine visual information such as:

Objects
Colours
People
Animals
Buildings
Text visible in the image
Actions
Relationships between objects
The general setting

The language-generation part turns those observations into readable text.

For example, an uploaded picture might produce:

A brown dog runs across a grassy field while carrying a red ball.

The same image could receive different captions depending on the instruction:

Caption type	Possible result
Short caption	A dog carrying a red ball in a field.
Social-media caption	Chasing sunshine, fresh air and one very important red ball.
Alt text	Brown dog running on grass with a red ball in its mouth.
Detailed description	A medium-sized brown dog runs from left to right across a green field while holding a red rubber ball.

This is why the prompt matters. We are not only sending an image; we are also telling the model what kind of description to produce.

Where Can Image Captioning Be Used?

An image-captioning system can be used in:

Blogging platforms
Product catalogues
Social-media tools
Photo-management applications
Content-management systems
Accessibility workflows
News and media websites
Real-estate listings
Travel websites
Classroom projects
Internal document systems

For example, a travel blogger could upload a photograph of Fushimi Inari Shrine and ask the application to suggest a short caption. An online shop could request a draft product description. A content editor could generate a first version of image alt text and then review it before publishing.

AI output should be treated as a draft. A model can overlook an object, misread visible text or describe something incorrectly. Human review remains important, especially for accessibility, journalism, medical images, identity-related claims and product information.

How the PHP Application Works

Stage	What happens
1. Select	The user chooses an image in the browser.
2. Preview	JavaScript displays a local preview.
3. Upload	The browser sends the form using `multipart/form-data`.
4. Validate	PHP checks the upload error, size and actual MIME type.
5. Encode	PHP converts the image bytes into a Base64 data URL.
6. Request	PHP sends the prompt and image to the Responses API.
7. Extract	PHP finds the returned `output_text`.
8. Display	The caption is escaped and shown to the user.

The API key stays on the server. It is never placed inside JavaScript or sent to the user’s browser.

Project Requirements

You will need:

PHP 8.1 or newer
The PHP cURL extension
The PHP Fileinfo extension
A web server or PHP’s built-in development server
An OpenAI API key
An API account with billing or available credits
A code editor such as Visual Studio Code

Check your PHP version:

php -v

Check whether cURL and Fileinfo are enabled:

php -m

Look for:

curl
fileinfo

If you are using XAMPP or Laragon, these extensions are commonly included, although cURL may need to be enabled in php.ini.

Step 1: Create the Project Folder

Create a folder named:

php-image-captioner

Inside it, create:

index.php

The project begins with only one file:

php-image-captioner/
└── index.php

Keeping the first version in one file makes the request flow easier to understand. Later, the CSS, JavaScript and API code can be separated into their own files or classes.

Step 2: Create and Protect the API Key

Create an API key from your API account.

Do not write the key directly into index.php, commit it to Git or expose it in browser-side JavaScript.

Use an environment variable named:

OPENAI_API_KEY

Linux or macOS

For the current terminal session:

export OPENAI_API_KEY="your_api_key_here"

Then start the PHP server from the same terminal.

Windows PowerShell

For the current PowerShell session:

$env:OPENAI_API_KEY="your_api_key_here"

Apache

Production servers should inject the secret through their deployment system, hosting control panel, secret manager or server environment. Avoid placing secrets in a publicly accessible .env file.

PHP will read the value with:

$apiKey = getenv("OPENAI_API_KEY");

If an API key has ever appeared in a public repository or webpage, revoke it and create a new one. Removing the visible text from the latest commit is not enough because the secret may remain in Git history.

Step 3: Add the Complete PHP Application

Copy the following code into index.php:

<?php

declare(strict_types=1);

session_start();

const MAX_IMAGE_SIZE = 5 * 1024 * 1024;

$allowedMimeTypes = [
    "image/jpeg",
    "image/png",
    "image/webp"
];

$captionStyles = [
    "short" => [
        "label" => "Short caption",
        "prompt" => "Write one clear factual sentence describing this image. "
            . "Keep it below 25 words. Do not start with 'This image shows'."
    ],
    "social" => [
        "label" => "Social-media caption",
        "prompt" => "Write an engaging but truthful social-media caption for this image. "
            . "Use no more than 35 words and at most two suitable emojis. "
            . "Do not invent names, places or events that cannot be confirmed visually."
    ],
    "alt" => [
        "label" => "Accessible alt text",
        "prompt" => "Write concise alt text for this image. Describe the most important "
            . "visible content and function in no more than 125 characters. "
            . "Do not begin with 'image of' or 'picture of'. "
            . "Do not guess identity or sensitive personal attributes."
    ],
    "detailed" => [
        "label" => "Detailed description",
        "prompt" => "Describe this image accurately in two or three sentences. "
            . "Mention the main subjects, visible actions, setting and important details. "
            . "Clearly express uncertainty instead of guessing."
    ]
];

$caption = null;
$error = null;
$selectedStyle = $_POST["caption_style"] ?? "short";

if (
    !is_string($selectedStyle)
    || !isset($captionStyles[$selectedStyle])
) {
    $selectedStyle = "short";
}

if (empty($_SESSION["csrf_token"])) {
    $_SESSION["csrf_token"] = bin2hex(random_bytes(32));
}

function extractOutputText(array $response): ?string
{
    $parts = [];

    foreach ($response["output"] ?? [] as $outputItem) {
        foreach ($outputItem["content"] ?? [] as $contentItem) {
            if (
                ($contentItem["type"] ?? "") === "output_text"
                && isset($contentItem["text"])
            ) {
                $parts[] = trim((string) $contentItem["text"]);
            }
        }
    }

    $text = trim(implode("\n", array_filter($parts)));

    return $text !== "" ? $text : null;
}

if ($_SERVER["REQUEST_METHOD"] === "POST") {
    $submittedToken = $_POST["csrf_token"] ?? "";

    if (
        !is_string($submittedToken)
        || !hash_equals($_SESSION["csrf_token"], $submittedToken)
    ) {
        $error = "The form session has expired. Refresh the page and try again.";
    } elseif (!isset($_FILES["image"])) {
        $error = "Please choose an image.";
    } elseif ($_FILES["image"]["error"] !== UPLOAD_ERR_OK) {
        $error = "The image could not be uploaded. Please try another file.";
    } elseif ($_FILES["image"]["size"] > MAX_IMAGE_SIZE) {
        $error = "The image is too large. The maximum size is 5 MB.";
    } else {
        $temporaryPath = $_FILES["image"]["tmp_name"];
        $fileInfo = new finfo(FILEINFO_MIME_TYPE);
        $mimeType = $fileInfo->file($temporaryPath);

        if (!in_array($mimeType, $allowedMimeTypes, true)) {
            $error = "Only JPG, PNG and WebP images are allowed.";
        } else {
            $imageBytes = file_get_contents($temporaryPath);

            if ($imageBytes === false) {
                $error = "PHP could not read the uploaded image.";
            } else {
                $apiKey = getenv("OPENAI_API_KEY");

                if (!$apiKey) {
                    $error = "The server is missing its API configuration.";
                } else {
                    $imageDataUrl = "data:"
                        . $mimeType
                        . ";base64,"
                        . base64_encode($imageBytes);

                    $payload = [
                        "model" => "gpt-5-nano",
                        "input" => [
                            [
                                "role" => "user",
                                "content" => [
                                    [
                                        "type" => "input_text",
                                        "text" => $captionStyles[$selectedStyle]["prompt"]
                                    ],
                                    [
                                        "type" => "input_image",
                                        "image_url" => $imageDataUrl,
                                        "detail" => "auto"
                                    ]
                                ]
                            ]
                        ],
                        "max_output_tokens" => 300,
                        "store" => false
                    ];

                    $curl = curl_init("https://api.openai.com/v1/responses");

                    curl_setopt_array($curl, [
                        CURLOPT_POST => true,
                        CURLOPT_RETURNTRANSFER => true,
                        CURLOPT_CONNECTTIMEOUT => 10,
                        CURLOPT_TIMEOUT => 60,
                        CURLOPT_HTTPHEADER => [
                            "Authorization: Bearer " . $apiKey,
                            "Content-Type: application/json"
                        ],
                        CURLOPT_POSTFIELDS => json_encode(
                            $payload,
                            JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR
                        )
                    ]);

                    $responseBody = curl_exec($curl);
                    $curlError = curl_error($curl);
                    $statusCode = (int) curl_getinfo(
                        $curl,
                        CURLINFO_HTTP_CODE
                    );

                    curl_close($curl);

                    if ($responseBody === false) {
                        $error = "The caption service could not be reached: "
                            . $curlError;
                    } else {
                        try {
                            $responseData = json_decode(
                                $responseBody,
                                true,
                                512,
                                JSON_THROW_ON_ERROR
                            );

                            if ($statusCode < 200 || $statusCode >= 300) {
                                $apiMessage = $responseData["error"]["message"]
                                    ?? "The API rejected the request.";

                                $error = "Caption generation failed: "
                                    . $apiMessage;
                            } else {
                                $caption = extractOutputText($responseData);

                                if ($caption === null) {
                                    $error = "The API returned no caption. Please try again.";
                                }
                            }
                        } catch (JsonException $exception) {
                            $error = "The caption service returned an invalid response.";
                        }
                    }
                }
            }
        }
    }
}

?>
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>AI Image Caption Generator</title>

    <style>
        :root {
            color-scheme: light;
            --background: #f4f7fb;
            --card: #ffffff;
            --text: #172033;
            --muted: #667085;
            --primary: #5b4cf0;
            --primary-dark: #4338ca;
            --border: #dce2eb;
            --success: #ecfdf3;
            --success-border: #86efac;
            --danger: #fff1f2;
            --danger-border: #fda4af;
        }

        * {
            box-sizing: border-box;
        }

        body {
            margin: 0;
            min-height: 100vh;
            padding: 40px 18px;
            font-family: Arial, Helvetica, sans-serif;
            color: var(--text);
            background: linear-gradient(135deg, #eef2ff, var(--background));
        }

        .app {
            width: min(760px, 100%);
            margin: 0 auto;
            padding: 32px;
            border: 1px solid rgba(255, 255, 255, 0.8);
            border-radius: 22px;
            background: var(--card);
            box-shadow: 0 20px 60px rgba(23, 32, 51, 0.12);
        }

        h1 {
            margin: 0 0 10px;
            font-size: clamp(1.8rem, 5vw, 2.5rem);
        }

        .intro {
            margin: 0 0 28px;
            color: var(--muted);
            line-height: 1.6;
        }

        label {
            display: block;
            margin-bottom: 8px;
            font-weight: 700;
        }

        input[type="file"],
        select {
            width: 100%;
            margin-bottom: 20px;
            padding: 12px;
            border: 1px solid var(--border);
            border-radius: 10px;
            background: #fff;
            font: inherit;
        }

        input[type="file"]::file-selector-button {
            margin-right: 12px;
            padding: 9px 14px;
            border: 0;
            border-radius: 8px;
            color: #fff;
            background: var(--primary);
            cursor: pointer;
        }

        .preview {
            display: none;
            width: 100%;
            max-height: 380px;
            margin: 0 0 22px;
            border-radius: 14px;
            object-fit: contain;
            background: #f8fafc;
        }

        .generate-button,
        .copy-button {
            border: 0;
            border-radius: 10px;
            color: #fff;
            background: var(--primary);
            font: inherit;
            font-weight: 700;
            cursor: pointer;
        }

        .generate-button {
            width: 100%;
            padding: 14px 18px;
        }

        .generate-button:hover,
        .copy-button:hover {
            background: var(--primary-dark);
        }

        .generate-button:disabled {
            cursor: wait;
            opacity: 0.7;
        }

        .message,
        .result {
            margin-top: 24px;
            padding: 18px;
            border: 1px solid;
            border-radius: 12px;
            line-height: 1.6;
        }

        .message {
            border-color: var(--danger-border);
            background: var(--danger);
        }

        .result {
            border-color: var(--success-border);
            background: var(--success);
        }

        .result h2 {
            margin: 0 0 8px;
            font-size: 1.1rem;
        }

        .caption-text {
            margin: 0 0 14px;
            white-space: pre-wrap;
        }

        .copy-button {
            padding: 9px 14px;
        }

        .help {
            margin-top: -12px;
            margin-bottom: 20px;
            color: var(--muted);
            font-size: 0.9rem;
        }

        @media (max-width: 600px) {
            body {
                padding: 18px 12px;
            }

            .app {
                padding: 22px;
                border-radius: 16px;
            }
        }
    </style>
</head>
<body>

<main class="app">
    <h1>AI Image Caption Generator</h1>

    <p class="intro">
        Upload an image and choose how you want the AI to describe it.
    </p>

    <form method="post" enctype="multipart/form-data" id="caption-form">
        <input
            type="hidden"
            name="csrf_token"
            value="<?php echo htmlspecialchars(
                $_SESSION["csrf_token"],
                ENT_QUOTES,
                "UTF-8"
            ); ?>"
        >

        <label for="image">Choose an image</label>
        <input
            type="file"
            name="image"
            id="image"
            accept="image/jpeg,image/png,image/webp"
            required
        >

        <p class="help">JPG, PNG or WebP. Maximum file size: 5 MB.</p>

        <img id="preview" class="preview" alt="Selected image preview">

        <label for="caption_style">Caption style</label>
        <select name="caption_style" id="caption_style">
            <?php foreach ($captionStyles as $value => $style): ?>
                <option
                    value="<?php echo htmlspecialchars($value); ?>"
                    <?php echo $selectedStyle === $value ? "selected" : ""; ?>
                >
                    <?php echo htmlspecialchars($style["label"]); ?>
                </option>
            <?php endforeach; ?>
        </select>

        <button type="submit" class="generate-button" id="generate-button">
            Generate caption
        </button>
    </form>

    <?php if ($error !== null): ?>
        <div class="message" role="alert">
            <?php echo htmlspecialchars($error, ENT_QUOTES, "UTF-8"); ?>
        </div>
    <?php endif; ?>

    <?php if ($caption !== null): ?>
        <section class="result" aria-live="polite">
            <h2>Generated caption</h2>

            <p class="caption-text" id="caption-text"><?php
                echo htmlspecialchars($caption, ENT_QUOTES, "UTF-8");
            ?></p>

            <button type="button" class="copy-button" id="copy-button">
                Copy caption
            </button>
        </section>
    <?php endif; ?>
</main>

<script>
    const imageInput = document.getElementById("image");
    const preview = document.getElementById("preview");
    const form = document.getElementById("caption-form");
    const generateButton = document.getElementById("generate-button");
    const copyButton = document.getElementById("copy-button");

    imageInput.addEventListener("change", () => {
        const file = imageInput.files[0];

        if (!file) {
            preview.removeAttribute("src");
            preview.style.display = "none";
            return;
        }

        preview.src = URL.createObjectURL(file);
        preview.style.display = "block";
    });

    form.addEventListener("submit", () => {
        generateButton.disabled = true;
        generateButton.textContent = "Generating caption...";
    });

    if (copyButton) {
        copyButton.addEventListener("click", async () => {
            const captionText = document
                .getElementById("caption-text")
                .innerText;

            try {
                await navigator.clipboard.writeText(captionText);
                copyButton.textContent = "Copied!";

                setTimeout(() => {
                    copyButton.textContent = "Copy caption";
                }, 1500);
            } catch (error) {
                copyButton.textContent = "Copy failed";
            }
        });
    }
</script>

</body>
</html>

Step 4: Run the Project

Open a terminal inside the project folder.

Make sure the OPENAI_API_KEY environment variable is available, then run:

php -S localhost:8000

Open this address in your browser:

http://localhost:8000

Select an image, choose a caption style and click Generate caption.

The first request may take several seconds because the image must be uploaded, encoded, processed and returned.

Understanding the Important PHP Code

The full example is long because it includes validation, error handling, styling and JavaScript. The essential PHP sections are easier to understand separately.

1. Start a Session

session_start();

The session stores the CSRF token used to confirm that the submitted form originated from the application.

2. Limit the File Size

const MAX_IMAGE_SIZE = 5 * 1024 * 1024;

This creates a five-megabyte application limit.

PHP and the web server can have their own upload limits. If PHP rejects a large upload before the script runs, inspect these php.ini settings:

upload_max_filesize = 6M
post_max_size = 7M

post_max_size should be slightly larger than upload_max_filesize because the full form request contains more than the image bytes.

Restart the web server after modifying php.ini.

3. Check the Actual MIME Type

$fileInfo = new finfo(FILEINFO_MIME_TYPE);
$mimeType = $fileInfo->file($temporaryPath);

Do not rely only on:

$_FILES["image"]["type"]

That value comes from the browser and can be misleading. Fileinfo examines the uploaded file on the server.

Checking the filename extension alone is also insufficient. A dangerous or unrelated file can be renamed to end with .jpg.

4. Convert the Image to Base64

$imageDataUrl = "data:"
    . $mimeType
    . ";base64,"
    . base64_encode($imageBytes);

This creates a data URL similar to:

data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ...

Base64 makes it possible to include the image inside the JSON request. It increases the data size, so the tutorial limits uploads to 5 MB.

For larger production workflows, uploading once and referencing a file ID can be more efficient than repeatedly embedding the same image.

5. Send Text and Image Input Together

"content" => [
    [
        "type" => "input_text",
        "text" => $captionStyles[$selectedStyle]["prompt"]
    ],
    [
        "type" => "input_image",
        "image_url" => $imageDataUrl,
        "detail" => "auto"
    ]
]

The model receives two related inputs:

The written instruction
The image to examine

The detail setting controls how the image is processed. auto lets the API select an appropriate level. A low-detail setting may reduce input usage for simple images, while high detail may be more suitable when small objects or text matter. Confirm supported values and current image-token behaviour in the official image-input documentation.

6. Keep the Key on the Server

"Authorization: Bearer " . $apiKey

The authorization header is created by PHP. The browser never sees the API key.

Never make a direct API request from public frontend JavaScript using a permanent secret key. Anyone who can inspect the page or network request could steal it and use your account.

7. Decode the Response

$responseData = json_decode(
    $responseBody,
    true,
    512,
    JSON_THROW_ON_ERROR
);

The JSON_THROW_ON_ERROR option causes PHP to throw a JsonException if the response is invalid JSON. The application catches the exception and shows a controlled error.

8. Extract `output_text`

The raw Responses API result contains an output array. Each output item can contain one or more content items.

The helper function searches them:

function extractOutputText(array $response): ?string
{
    $parts = [];

    foreach ($response["output"] ?? [] as $outputItem) {
        foreach ($outputItem["content"] ?? [] as $contentItem) {
            if (
                ($contentItem["type"] ?? "") === "output_text"
                && isset($contentItem["text"])
            ) {
                $parts[] = trim((string) $contentItem["text"]);
            }
        }
    }

    $text = trim(implode("\n", array_filter($parts)));

    return $text !== "" ? $text : null;
}

This is safer than assuming the caption is always at one fixed numeric position.

9. Escape the Caption Before Displaying It

echo htmlspecialchars(
    $caption,
    ENT_QUOTES,
    "UTF-8"
);

AI output should be treated as untrusted text. htmlspecialchars() prevents returned HTML-like content from becoming executable webpage markup.

Why Use Different Caption Styles?

One description does not suit every job.

Short Caption

Useful beneath photographs in articles or galleries.

Prompt goal:

Write one clear factual sentence below 25 words.

Social-Media Caption

Allows slightly more personality while still instructing the model not to invent unsupported facts.

The model should not claim that a picture was taken in Tokyo merely because the street looks Japanese. It should not invent an event name, person’s identity or exact date.

Accessible Alt Text

Alt text communicates an image’s important content or function to people who cannot see it.

Good alt text depends on context. The same image may need different alt text on different pages. A decorative image may need empty alt text instead of an AI description.

Therefore, AI can suggest alt text, but the website author should decide whether the result is appropriate.

Detailed Description

Useful when the user needs more context than a short caption provides. It can mention the setting, positions, visible actions and important objects.

Choosing a Model

This tutorial uses:

"model" => "gpt-5-nano"

According to the current official model page, GPT-5 nano accepts text and image input and produces text output. It is positioned as the fastest, cheapest GPT-5 model, making it a practical starting point for a small captioning demo.

If its captions are not accurate enough for your images, replace the model ID with another vision-capable model available to your API project. A stronger model can improve difficult scene understanding, small-text recognition and instruction following, but it will usually cost more and may respond more slowly.

Model names, availability and pricing can change. Check the official OpenAI model guide before launching a production system.

Do not silently change models in a live application. Test the new model using a fixed set of representative images and compare:

Factual accuracy
Missing important details
Invented details
Caption length
Latency
Cost per request
Performance on text-heavy images
Performance on people, products and unusual scenes

Improving Caption Accuracy

Give a Specific Instruction

Weak prompt:

Describe this.

Better prompt:

Write one factual caption below 25 words. Mention the main subject and visible action. Do not guess the place, identity or event.

Specific requirements help the model understand what the application needs.

Tell the Model What Not to Guess

Image models can make plausible but unsupported assumptions.

Useful restrictions include:

Do not identify unknown people.
Do not infer ethnicity, religion, health or other sensitive traits.
Do not invent a location.
Do not assume the occasion.
Do not claim uncertain text is readable.
Express uncertainty when an object is unclear.

Preserve Enough Image Quality

Very small, blurred or heavily compressed images are difficult to understand. If important text occupies only a tiny portion of the image, crop the relevant area or use a suitable detail setting.

Test with Real Images

Do not test only with clear stock photography. Include examples that match the intended application:

Dark images
Crowded scenes
Screenshots
Products on similar backgrounds
Images containing text
Portrait and landscape orientations
Mobile photographs
Illustrations
Partly obstructed subjects

Security Considerations

A demonstration running on a private computer is different from a public upload service. Before allowing unknown visitors to use the application, strengthen it.

1. Add Rate Limiting

Without rate limiting, one visitor or bot could send many requests and consume the API budget.

Possible limits include:

Requests per IP address
Requests per signed-in account
Daily account allowance
Maximum concurrent requests
Monthly spending alerts
A hard application budget

A session-based counter alone is not strong protection because users can start new sessions.

2. Require Authentication

If the tool is intended for staff or registered users, require login before permitting API requests.

3. Validate on the Server

The HTML accept attribute improves the file picker, but it is not a security control. An attacker can send a custom HTTP request.

Always enforce the file type and size in PHP.

4. Do Not Trust Original Filenames

The tutorial does not save the file permanently. If you add storage, generate a random server-side filename instead of using the uploaded name.

Store uploads outside the public web root where possible. If they must be public, configure the upload directory so PHP scripts cannot execute there.

5. Protect Secrets

Keep the API key in a protected server environment.
Never log the complete authorization header.
Never return the key in an error response.
Use different keys for development and production.
Revoke exposed keys immediately.
Restrict access to deployment secrets.

6. Do Not Display Raw AI Output as HTML

Use htmlspecialchars() for normal text output.

If a future version intentionally supports Markdown, process it with a well-maintained parser configured to block dangerous HTML and URLs.

7. Consider Image Privacy

Users may upload photographs containing faces, addresses, identity cards, vehicle plates, private documents or location information.

Explain clearly:

What is uploaded
Which external service processes it
Whether the application stores it
How long logs are kept
Who can access the generated caption

Avoid keeping uploaded images unless the application actually needs them.

8. Moderate Public Content Where Necessary

A public tool may receive disturbing, illegal or abusive content. Establish an acceptable-use policy, reporting process and moderation controls suitable for the audience and jurisdiction.

9. Avoid Revealing Detailed Errors to Public Users

During development, the API’s error message helps with debugging. In production, log technical details privately and show the visitor a simpler message.

For example:

error_log("Caption API error: " . $apiMessage);
$error = "We could not generate a caption. Please try again later.";

Do not log Base64 image data or sensitive image content unnecessarily.

Common Errors and Solutions

Error: `Call to undefined function curl_init()`

The PHP cURL extension is unavailable or disabled.

Enable cURL in the active php.ini, then restart PHP or the web server.

Find the loaded configuration file with:

php --ini

Remember that command-line PHP and web-server PHP can load different configuration files.

Error: The Server Is Missing Its API Configuration

PHP cannot read OPENAI_API_KEY.

Confirm that the environment variable exists in the same environment that runs PHP. Setting it in one terminal does not automatically make it available to Apache, PHP-FPM or another terminal.

Error: HTTP 401

Common causes include:

Missing API key
Incorrect API key
Revoked key
Extra spaces or quote characters
The key not reaching the PHP process

Do not print the full key while debugging. At most, check whether the environment variable exists.

Error: HTTP 429

This commonly indicates a rate-limit or quota problem.

Check the API account’s usage, billing state, project limits and rate limits. Add retry logic with backoff for temporary rate limits, but do not endlessly retry quota failures.

Error: The Uploaded File Is Rejected

Check:

The actual MIME type
The five-megabyte application limit
upload_max_filesize
post_max_size
Reverse-proxy request limits
Web-server request limits

Error: PHP Times Out

The example gives cURL a 60-second timeout. A slow connection, large image or busy service can exceed a lower server timeout.

For a production application, consider a background job and status endpoint rather than keeping a web request open for a long time.

Error: No Caption Was Returned

Log the response structure in a protected development environment. The result may have been incomplete, refused or returned in an unexpected form.

Do not assume that every successful HTTP response contains usable caption text.

Error: Base64 Request Is Too Large

Base64 representation is larger than the original binary file. Reduce the upload limit, resize the image before sending it or use a file-upload workflow appropriate for the API.

Optional Improvement: Resize Images Before Sending

Large camera photographs can consume more bandwidth and memory than a caption requires.

A production application can resize images using GD or Imagick before encoding them. Preserve the aspect ratio and avoid reducing the image so much that important objects or text disappear.

For example, a general caption may not require a 6000-pixel-wide photograph. A maximum dimension of 1600 or 2000 pixels may be sufficient for many cases, but the correct value should be tested against your own image set.

Resizing can:

Reduce upload time
Reduce PHP memory usage
Reduce request size
Standardise image dimensions
Potentially lower processing cost

Do not overwrite the user’s original unless that is an explicit product feature.

Optional Improvement: Return Structured JSON

A later version may need more than one caption field.

For example:

{
    "short_caption": "A child flies a red kite on a beach.",
    "alt_text": "Child flying a red kite beside the sea.",
    "keywords": ["child", "kite", "beach", "sea"],
    "needs_review": false
}

Structured output is useful when the result must be saved to a database or sent to another system.

The application should validate every returned field before using it. JSON produced by a model should not be trusted merely because it looks structured.

Optional Improvement: Save Caption History

You can add a database table containing:

Column	Purpose
`id`	Unique record ID
`user_id`	Owner of the request
`image_path`	Stored image location, if retained
`caption_style`	Selected output type
`caption`	Generated text
`model`	Model used
`status`	Pending, completed or failed
`created_at`	Request time

Do not save Base64 strings in a normal database column without a strong reason. Object storage or a protected filesystem is usually more suitable for images, while the database stores a reference.

Add retention and deletion rules instead of keeping every uploaded image forever.

Optional Improvement: Use a Queue

The single-page version waits for the API before returning the webpage.

That is acceptable for a small tutorial. A busy production application can instead:

Validate the upload.
Save a pending job.
Return a job ID.
Process the image in a worker.
Save the caption.
Let the browser check the job status.

A queue prevents slow AI requests from occupying all available web workers. It also makes controlled retries easier.

A queue is not automatically necessary for a private tool or low-traffic site. Add it when request volume, latency and reliability justify the extra complexity.

Testing Checklist

Test the application with:

A normal JPG photograph
A PNG screenshot
A WebP image
An image just below 5 MB
An image above 5 MB
A text file renamed to .jpg
An empty form submission
An invalid API key
A missing API key
A very dark image
A crowded image
An image containing small text
An image containing a person
A portrait-oriented phone image
A caption containing quotation marks and symbols

Also confirm that:

The API key does not appear in the HTML source.
The API key does not appear in browser developer tools.
Returned text is escaped.
The submit button disables while waiting.
The mobile layout remains usable.
Failed requests do not expose stack traces or server paths in production.

Frequently Asked Questions

Does this project train an image-captioning model?

No. It uses an existing multimodal model through an API. Training a vision-and-language model from the beginning requires a large labelled dataset, substantial computing power and considerably more machine-learning knowledge.

Can PHP understand images by itself?

PHP can upload, validate, resize and store image files, but the caption is generated by the external AI model. PHP manages the web application and API communication.

Do I need Composer?

No. This version uses PHP’s built-in cURL extension, so it does not require a Composer package.

A larger project may use an HTTP client such as Guzzle or a maintained SDK to organise requests more cleanly.

Is the OpenAI API included with ChatGPT Plus?

No. ChatGPT subscriptions and API usage are billed separately. Configure API billing and limits for the API project used by the application.

Why use the Responses API?

It accepts multimodal input, including text and images, in a single request and is the current API pattern demonstrated in OpenAI’s image-input documentation.

Why is the image converted to Base64?

The image exists temporarily on the PHP server and may not have a public URL. A Base64 data URL allows PHP to send its bytes inside the JSON request.

Can I send an image URL instead?

Yes, if the URL is publicly reachable and meets the API’s requirements. Do not let users supply arbitrary URLs without considering server-side request forgery, private-network access and untrusted content.

Can I use this system to generate alt text?

It can generate a draft, but good alt text depends on the purpose and context of the image. A human editor should review it. Decorative images may require empty alt text rather than a description.

Can the AI identify a person in a photograph?

Do not build the captioning workflow around guessing or confirming identity. The tutorial’s prompts tell the model not to invent names or sensitive personal attributes.

Will the caption always be correct?

No. Vision models can miss objects, misunderstand relationships, invent details or misread text. Review important captions before publishing them.

Can I use the application on shared hosting?

Possibly. The host must support a suitable PHP version, cURL, Fileinfo, outbound HTTPS requests, environment-based secrets and request sizes large enough for the encoded image.

Some shared hosts block outbound API requests or provide no safe way to configure secrets. Check with the hosting provider.

Does the project save uploaded images?

No. It reads PHP’s temporary upload and sends it to the API. The application does not move it into permanent local storage.

How much does each caption cost?

Cost depends on the selected model, image size and detail, prompt size and output length. Model prices can change, so use the official pricing and model pages rather than hard-coding an old estimate into the application.

Should I use a queue?

For a simple private tool, probably not. For a public or busy site, a queue can improve reliability and prevent slow API calls from tying up all PHP workers.

Can I add several languages?

Yes. Add a language selector and include the chosen language in the prompt.

For example:

Write the caption in Bahasa Melayu.

Validate the selected language against a server-side allow-list instead of inserting unrestricted form text into important instructions.

Final Result

We have built a complete AI-powered image-captioning application with PHP.

The finished system:

Accepts JPG, PNG and WebP uploads.
Previews the selected image in the browser.
Validates file size and MIME type on the server.
Protects the form with a CSRF token.
Keeps the API key out of frontend code.
Converts the image into a Base64 data URL.
Sends text and image input to a vision-capable model.
Supports short, social, alt-text and detailed captions.
Extracts text from the Responses API result.
Escapes the caption before displaying it.
Provides a copy button and clear error messages.

The single-file version is a strong learning project, but a public system should also add authentication, durable rate limiting, spending controls, private logging, privacy disclosures and monitoring.

Most importantly, remember that an AI caption is a suggestion rather than guaranteed truth. Review the result before using it for accessibility, publishing or business data.

August 5, 2026

Securing Your Laravel API: Common Vulnerabilities and Solutions

Laravel provides secure building blocks for authentication, validation, database access and authorization. However, a Laravel API is not automatically secure simply because it uses the framework.

Most serious API vulnerabilities are caused by application logic: an endpoint checks whether a user is logged in but does not check whether that user owns the requested record, a controller accepts fields it should never accept, or a public endpoint has no request limit.

This guide explains the most common Laravel API security problems, shows vulnerable code, and replaces it with safer Laravel code.

The examples use modern Laravel 12 and Laravel 13 conventions. The same security principles also apply to older supported Laravel applications, although some file locations and installation commands may differ.

Quick Answer

To secure a Laravel API:

Authenticate protected routes with Laravel Sanctum or Passport.
Authorize every action on every protected resource.
Never trust a record ID supplied by the client.
Validate requests with Form Request classes.
Update models using validated, explicitly allowed fields.
Avoid raw SQL containing request data.
Rate-limit login, password-reset, search, upload and expensive endpoints.
Return only the fields the client needs.
Restrict uploaded file type, size and storage location.
Prevent server-side request forgery when your API fetches URLs.
Keep APP_DEBUG=false and protect the .env file in production.
Log security events without logging passwords, tokens or sensitive personal data.
Use HTTPS and keep Laravel, PHP and Composer packages supported and patched.
Add automated tests that attempt unauthorized access—not only successful requests.

The most important rule is this:

Authentication proves who the caller is. Authorization decides what that caller is allowed to do.

An endpoint needs both when it works with private or restricted data.

Common Laravel API Vulnerabilities

Vulnerability	Common Laravel mistake	Safer solution
Broken object-level authorization	`Post::findOrFail($id)` after checking only that the user is logged in	Policies, gates or ownership-scoped queries
Broken function-level authorization	Any authenticated user can call an admin route	Role or permission checks enforced on the server
Broken authentication	Permanent tokens, no revocation, no login throttling	Sanctum or Passport, short lifetimes where appropriate, revocation and throttling
Mass assignment	`$model->update($request->all())`	Form Requests, `$request->validated()` and `$fillable`
SQL injection	Concatenating request data into raw SQL	Query Builder bindings and allowlists for column names
Excessive data exposure	Returning complete Eloquent models	API Resources and explicit fields
Unrestricted resource consumption	No limits on requests, pagination, uploads or exports	Rate limits and hard server-side limits
Unsafe file upload	Trusting extensions or storing files publicly	Content-based validation, generated names and private storage
SSRF	Fetching any URL submitted by a user	Host allowlists, blocked private IPs, timeouts and redirect controls
CSRF or CORS mistakes	Disabling protections or allowing every origin with credentials	Correct Sanctum SPA configuration and narrow CORS rules
Sensitive-data exposure	Debug mode, secrets in logs or committed `.env` files	Production configuration, secret management and log redaction
Outdated dependencies	Ignoring security advisories	Supported versions, `composer audit` and controlled updates

These risks overlap with the OWASP API Security Top 10, including broken object-level authorization, broken authentication, unrestricted resource consumption, server-side request forgery and security misconfiguration.

1. Use Proper API Authentication

Authentication identifies the user or system making a request.

For a first-party single-page application, mobile application or simple token-based API, Laravel generally recommends Sanctum. Passport is appropriate when the application genuinely requires OAuth 2 features. See Laravel’s authentication guidance before choosing between them.

In a modern Laravel application, API support can be installed with:

php artisan install:api

Protect private routes with auth:sanctum:

<?php

use App\Http\Controllers\PostController;
use Illuminate\Support\Facades\Route;

Route::middleware('auth:sanctum')->group(function () {
    Route::get('/posts', [PostController::class, 'index']);
    Route::post('/posts', [PostController::class, 'store']);
    Route::patch('/posts/{post}', [PostController::class, 'update']);
    Route::delete('/posts/{post}', [PostController::class, 'destroy']);
});

A route outside this group is public unless another middleware protects it.

Do Not Create Your Own Plain-Text Token System

Avoid adding an api_token column to the users table and comparing raw permanent tokens manually. A home-made implementation may omit hashing, expiry, abilities, rotation, revocation and secure lookup behaviour.

Sanctum stores a hash of each personal access token in the database. The plain-text token is returned only when it is created, so it must be shown to the user once and then handled like a password.

$token = $user->createToken(
    'mobile-app',
    ['posts:read', 'posts:write']
)->plainTextToken;

Do not write $token to logs or analytics.

Limit Token Abilities

Do not give every integration unrestricted access.

Route::post('/posts', [PostController::class, 'store'])
    ->middleware([
        'auth:sanctum',
        'abilities:posts:write',
    ]);

Abilities reduce what a token is intended to do. They do not replace model authorization. A token with posts:write must still be prevented from editing another user’s post.

Revoke Tokens

Provide a way to revoke the current token:

public function logout(Request $request): JsonResponse
{
    $request->user()->currentAccessToken()?->delete();

    return response()->json([
        'message' => 'Token revoked.',
    ]);
}

You may also revoke all tokens after a password change, account compromise or administrative security action:

$user->tokens()->delete();

Decide whether tokens need an expiry time based on the sensitivity of the application. Long-lived integration tokens may be necessary, but they should have narrow abilities, visible last-used information, rotation procedures and immediate revocation support.

2. Prevent Broken Object-Level Authorization

Broken object-level authorization, often called BOLA or IDOR, is one of the most common API vulnerabilities.

Consider this endpoint:

GET /api/invoices/8421

If a logged-in user changes 8421 to 8422, can they see another customer’s invoice?

Vulnerable Example

public function show(int $id): JsonResponse
{
    $invoice = Invoice::findOrFail($id);

    return response()->json($invoice);
}

Placing this controller behind auth:sanctum only proves that the caller is authenticated. It does not prove that the invoice belongs to the caller.

UUIDs do not solve this problem. A UUID may be harder to guess than an integer, but leaked, logged or shared UUIDs still require authorization checks.

Solution A: Scope the Query to the Authenticated User

public function show(Request $request, int $id): InvoiceResource
{
    $invoice = $request->user()
        ->invoices()
        ->findOrFail($id);

    return new InvoiceResource($invoice);
}

The query searches only within the authenticated user’s invoices. A record belonging to another user is not returned.

This approach is particularly useful for strictly owned records.

Solution B: Use a Laravel Policy

Generate a policy:

php artisan make:policy InvoicePolicy --model=Invoice

Define the authorization rule:

<?php

namespace App\Policies;

use App\Models\Invoice;
use App\Models\User;

class InvoicePolicy
{
    public function view(User $user, Invoice $invoice): bool
    {
        return $user->id === $invoice->user_id;
    }

    public function update(User $user, Invoice $invoice): bool
    {
        return $user->id === $invoice->user_id
            && $invoice->status === 'draft';
    }
}

Authorize the action in the controller:

public function show(Invoice $invoice): InvoiceResource
{
    Gate::authorize('view', $invoice);

    return new InvoiceResource($invoice);
}

Laravel policies group authorization rules around a model, while gates are useful for actions not tied to one model. Laravel explains both approaches in its authorization documentation.

Check Every Operation

Authorization is required for more than show.

Check all relevant operations:

Listing records
Viewing one record
Creating records under a parent resource
Updating records
Deleting or restoring records
Downloading attachments
Exporting data
Viewing comments or activity logs
Changing status
Assigning a record to another user

A secure show method does not compensate for an insecure update, download or export method.

3. Prevent Broken Function-Level Authorization

Object-level authorization asks, “Can this user access this record?”

Function-level authorization asks, “Can this user perform this type of action at all?”

Vulnerable Example

Route::delete('/admin/users/{user}', function (User $user) {
    $user->delete();

    return response()->noContent();
})->middleware('auth:sanctum');

Every authenticated user can call this route.

Hiding the delete button in the frontend is not security. Attackers can call the API directly.

Safer Example

Use a policy, gate or trusted permission package and enforce the result on the server:

Gate::define('delete-user', function (User $currentUser, User $targetUser) {
    return $currentUser->is_admin
        && $currentUser->id !== $targetUser->id;
});

Then authorize the action:

public function destroy(User $user): Response
{
    Gate::authorize('delete-user', $user);

    $user->delete();

    return response()->noContent();
}

For larger applications, use policies or a well-maintained role-and-permission system so the rules remain consistent and testable.

4. Prevent Mass-Assignment Vulnerabilities

Mass assignment happens when an array is used to create or update several model attributes at once.

It is convenient, but dangerous when the array contains fields the user should not control.

Vulnerable Example

public function update(Request $request, User $user): JsonResponse
{
    $user->update($request->all());

    return response()->json($user);
}

An attacker may submit:

{
    "name": "Normal Name",
    "is_admin": true,
    "account_balance": 100000
}

If the model accepts those attributes, the user may change protected values.

Use `$fillable`

class User extends Authenticatable
{
    protected $fillable = [
        'name',
        'timezone',
    ];
}

Do not use this in a sensitive model without very careful control:

protected $guarded = [];

An empty $guarded array makes all attributes mass assignable.

Use a Form Request and Validated Data

Create a request class:

php artisan make:request UpdateProfileRequest

<?php

namespace App\Http\Requests;

use Illuminate\Foundation\Http\FormRequest;

class UpdateProfileRequest extends FormRequest
{
    public function authorize(): bool
    {
        return true;
    }

    public function rules(): array
    {
        return [
            'name' => ['required', 'string', 'max:100'],
            'timezone' => ['required', 'timezone'],
        ];
    }
}

Then update only validated fields:

public function update(UpdateProfileRequest $request): UserResource
{
    $user = $request->user();
    $user->update($request->validated());

    return new UserResource($user->refresh());
}

For additional clarity, select fields explicitly:

$user->update(
    $request->safe()->only([
        'name',
        'timezone',
    ])
);

Treat $fillable, validation and authorization as separate layers:

$fillable controls which model attributes may be mass assigned.
Validation controls acceptable request structure and values.
Authorization controls whether the current user may perform the action.

One layer does not replace the others.

5. Validate Every External Input

Do not validate only forms. Validate all external data, including:

JSON request bodies
Query-string filters
Route parameters
Uploaded files
Webhook payloads
Third-party API responses before important use
Import files
Sorting and pagination parameters

Example Form Request

<?php

namespace App\Http\Requests;

use Illuminate\Foundation\Http\FormRequest;
use Illuminate\Validation\Rule;

class StoreOrderRequest extends FormRequest
{
    public function authorize(): bool
    {
        return $this->user() !== null;
    }

    public function rules(): array
    {
        return [
            'product_id' => [
                'required',
                'integer',
                Rule::exists('products', 'id')
                    ->where('is_active', true),
            ],
            'quantity' => ['required', 'integer', 'min:1', 'max:20'],
            'delivery_note' => ['nullable', 'string', 'max:500'],
        ];
    }
}

Validation prevents malformed values, but business rules must still be checked. For example, a valid product_id does not prove that the customer may order a restricted product.

Set Maximum Limits

Never accept an unlimited per_page value:

$validated = $request->validate([
    'per_page' => ['sometimes', 'integer', 'min:1', 'max:100'],
]);

$perPage = $validated['per_page'] ?? 20;

$orders = Order::query()->paginate($perPage);

The same principle applies to date ranges, batch sizes, search lengths, export rows, nested JSON arrays and file sizes.

6. Prevent SQL Injection

Laravel’s Query Builder and Eloquent use parameter binding for normal values. Problems appear when developers concatenate request data into raw SQL or let users choose raw column names and expressions.

Vulnerable Raw Query

$email = $request->input('email');

$users = DB::select(
    "SELECT * FROM users WHERE email = '$email'"
);

Use Bindings

$users = DB::select(
    'SELECT * FROM users WHERE email = ?',
    [$request->string('email')->toString()]
);

Better still, use Eloquent or Query Builder when possible:

$user = User::query()
    ->where('email', $request->string('email'))
    ->first();

Laravel warns that raw expressions are inserted into queries as strings and may introduce SQL injection. When a raw expression is necessary, use bindings rather than concatenation. See the official Query Builder documentation.

Allowlist Sort Columns

Database bindings protect values, not arbitrary identifiers such as column names.

Do not do this:

$users = User::orderBy(
    $request->input('sort'),
    $request->input('direction')
)->get();

Use explicit allowlists:

$validated = $request->validate([
    'sort' => ['sometimes', 'in:name,created_at'],
    'direction' => ['sometimes', 'in:asc,desc'],
]);

$sort = $validated['sort'] ?? 'created_at';
$direction = $validated['direction'] ?? 'desc';

$users = User::query()
    ->orderBy($sort, $direction)
    ->paginate(20);

Use the same technique for selectable report fields, aggregate functions, table names and search operators.

7. Add Rate Limits and Resource Limits

Rate limiting reduces brute-force attacks, scraping, accidental loops and denial-of-service pressure.

Laravel includes a rate-limiting abstraction backed by the application’s cache. For distributed production servers, a shared store such as Redis prevents each application server from maintaining an independent counter.

Define a Named API Limiter

<?php

namespace App\Providers;

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\RateLimiter;
use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    public function boot(): void
    {
        RateLimiter::for('api-standard', function (Request $request) {
            $key = $request->user()?->id ?: $request->ip();

            return Limit::perMinute(60)->by($key);
        });
    }
}

Apply it to routes:

Route::middleware([
    'auth:sanctum',
    'throttle:api-standard',
])->group(function () {
    Route::apiResource('orders', OrderController::class);
});

Laravel documents named limiters and route middleware in its routing documentation.

Sensitive Endpoints Need Separate Limits

Do not use one generous limit for everything. Apply stricter limits to:

Login attempts
Registration
Password-reset requests
One-time-password verification
Email or SMS sending
Search endpoints
File uploads
PDF or report generation
AI or third-party paid API calls
Data exports
Webhook retries

Rate limiting is only one control. Also set:

Maximum request-body size at the web server
Maximum upload size
Maximum pagination size
Database query timeouts where appropriate
HTTP client connection and response timeouts
Queue job timeouts and retry limits
Maximum batch-operation size
Maximum export date range

Without hard limits, one accepted request may still consume excessive memory, CPU, database time or third-party credits.

8. Return Only the Data the Client Needs

An API can be properly authenticated and still leak data by returning complete models.

Risky Example

return response()->json(User::findOrFail($id));

The response may include internal fields that the current client does not need, such as:

Internal status flags
Administrative notes
Provider identifiers
Verification timestamps
Billing references
Soft-delete timestamps
Security-related metadata

Laravel normally hides the password and remember token on the default User model, but custom models and future columns may not be protected automatically.

Use an API Resource

Create a resource:

php artisan make:resource UserResource

<?php

namespace App\Http\Resources;

use Illuminate\Http\Request;
use Illuminate\Http\Resources\Json\JsonResource;

class UserResource extends JsonResource
{
    public function toArray(Request $request): array
    {
        return [
            'id' => $this->id,
            'name' => $this->name,
            'avatar_url' => $this->avatar_url,
            'created_at' => $this->created_at?->toIso8601String(),
        ];
    }
}

Return the resource:

return new UserResource($user);

API Resources create an explicit response contract. They are clearer and safer than exposing every current and future database column.

Also review nested relationships. Returning a safe user object with an unrestricted orders, payments or notes relationship may still leak private data.

9. Secure File Uploads

File upload endpoints require more than checking the filename extension.

Laravel can validate a file’s content-derived MIME type using the fluent File rule.

use Illuminate\Validation\Rules\File;

$validated = $request->validate([
    'document' => [
        'required',
        File::types(['pdf', 'jpg', 'jpeg', 'png'])
            ->max('5mb'),
    ],
]);

Laravel’s validation documentation notes that SVG files are not allowed by the normal image rule by default because SVG can introduce cross-site scripting risks. Do not enable SVG uploads unless the application has a safe sanitisation and serving strategy. See Laravel file validation.

Store with a Generated Name

$path = $validated['document']->store(
    'private/documents',
    'local'
);

Laravel generates the stored filename. Do not use an untrusted original filename as the filesystem path.

Keep Private Files Private

Do not store identity documents, invoices, medical files or private attachments on a directly public disk.

Serve them through an authorized controller or use short-lived signed storage URLs after authorization.

public function download(Document $document): BinaryFileResponse
{
    Gate::authorize('view', $document);

    return response()->download(
        Storage::disk('local')->path($document->path),
        $document->original_name
    );
}

Additional controls may include malware scanning, image re-encoding, archive rejection, decompression limits and separate storage domains. Never execute uploaded files.

10. Prevent Server-Side Request Forgery

Server-side request forgery, or SSRF, occurs when an attacker causes your server to request an unintended address.

A vulnerable endpoint may accept a URL for an image importer, webhook tester or page preview:

$response = Http::get($request->input('url'));

An attacker may try to reach:

localhost
Internal admin panels
Private network services
Cloud metadata endpoints
Services protected from the public internet

The safest approach is to avoid arbitrary destinations and allow only known hosts.

$validated = $request->validate([
    'url' => ['required', 'url:https'],
]);

$url = $validated['url'];
$host = strtolower((string) parse_url($url, PHP_URL_HOST));

$allowedHosts = [
    'images.example.com',
    'cdn.example.com',
];

abort_unless(
    in_array($host, $allowedHosts, true),
    422,
    'The URL host is not allowed.'
);

$response = Http::connectTimeout(3)
    ->timeout(8)
    ->withoutRedirecting()
    ->get($url);

If arbitrary public URLs are a genuine requirement, the protection must be stronger:

Permit only https where possible.
Resolve the hostname and reject loopback, private, link-local, multicast and reserved IP ranges.
Re-check every redirect destination.
Protect against DNS rebinding.
Restrict outbound traffic at the network layer.
Set connection, response and size limits.
Do not forward the user’s authorization headers or cookies.

URL validation alone does not prove that a destination is safe.

11. Configure CSRF and CORS Correctly

CSRF and CORS solve different problems.

CSRF protection prevents another website from causing a browser to perform an unwanted authenticated action.
CORS controls which browser origins may read or send permitted cross-origin requests.

CORS is not authentication. Command-line tools, mobile apps and malicious servers are not stopped by a browser’s CORS policy.

Bearer-Token APIs

A stateless API using a token in the Authorization header is normally not authenticated automatically by browser cookies. Traditional CSRF risk is therefore different from cookie-authenticated routes.

The token must still be protected against theft, logging and insecure client storage.

Sanctum SPA Authentication

For a first-party SPA using Sanctum’s cookie-based authentication, follow Laravel’s stateful SPA configuration and CSRF-cookie flow. Do not disable CSRF verification simply to make requests work.

The frontend typically first requests:

/sanctum/csrf-cookie

It then sends the login or authenticated request with credentials configured correctly. Refer to the current Laravel Sanctum documentation because the required middleware and client configuration depend on the Laravel version and frontend setup.

Keep CORS Narrow

If credentials are permitted, list trusted origins explicitly. Do not combine credentialed requests with a broad or reflected origin policy.

Review:

Allowed origins
Allowed methods
Allowed headers
Credential support
Preflight caching
Development origins accidentally left in production

Only publish the CORS configuration file if the defaults need to be changed:

php artisan config:publish cors

12. Protect Secrets and Production Configuration

Disable Debug Mode in Production

Production should use:

APP_ENV=production
APP_DEBUG=false

Laravel’s configuration documentation warns that production debug mode can expose sensitive configuration values.

Do not return stack traces, SQL queries, internal file paths or exception details to API clients. Send a stable error structure and keep full diagnostic details in protected server logs.

Protect the `.env` File

Never commit .env to Git. Ensure the web server’s document root points to Laravel’s public directory—not the project root.

Secrets include:

Database passwords
APP_KEY
API tokens
OAuth client secrets
Payment-provider keys
Mail credentials
Cloud storage credentials
Webhook signing secrets

Use environment variables or a secret-management service. Give each environment separate credentials and the minimum required privileges.

Do not casually rotate APP_KEY on an existing application. Data encrypted with the old key may become unreadable unless a planned key-rotation procedure and previous-key support are in place.

Cache Production Configuration

During deployment, run:

php artisan config:cache
php artisan route:cache
php artisan view:cache

Access environment values through configuration files and config() rather than calling env() throughout application code.

13. Do Not Leak Secrets Through Logs

Logs are essential for investigating attacks, but they can become another sensitive database.

Do not log:

Passwords
Full bearer tokens
Session cookies
CSRF tokens
Credit-card data
Complete identity documents
Password-reset links
Private webhook signatures
Full request bodies containing personal data

Risky code:

Log::info('Login request', $request->all());

Safer code:

Log::warning('Failed login attempt', [
    'email_hash' => hash('sha256', strtolower($request->input('email'))),
    'ip' => $request->ip(),
    'user_agent' => $request->userAgent(),
]);

Even hashed identifiers may be personal data depending on context and applicable law. Collect only what the security and support teams genuinely need, restrict log access and configure retention.

Useful security events include:

Repeated failed authentication
Token creation and revocation
Password and email changes
Two-factor authentication changes
Administrative actions
Permission changes
Large exports
Rejected webhook signatures
Unusual rate-limit activity

Do not show internal security reasons to an attacker. For example, a login response can remain generic while the server records whether the account was missing, disabled or supplied an incorrect password.

14. Verify Webhook Signatures

A webhook endpoint is public because an external provider must reach it. Public does not mean unverified.

Do not trust a webhook merely because it contains a plausible order ID or payment status.

Verify the signature using the provider’s official SDK or documented algorithm, using the raw request body when required.

General structure:

public function handle(Request $request): Response
{
    $payload = $request->getContent();
    $signature = $request->header('X-Provider-Signature');

    abort_unless(
        $this->signatureIsValid($payload, $signature),
        401
    );

    // Process the verified event idempotently.

    return response()->noContent();
}

Also:

Reject missing or invalid signatures.
Enforce timestamp tolerance when supported.
Store provider event IDs to prevent duplicate processing.
Make processing idempotent.
Queue slow work after verification.
Rate-limit or network-restrict endpoints when compatible with the provider.
Keep signing secrets separate for development and production.

Never invent a generic signature procedure when a provider supplies an official one.

15. Use HTTPS and Secure Infrastructure

API credentials and private data must be encrypted in transit with HTTPS.

Redirect HTTP to HTTPS at the load balancer or web server and renew certificates automatically. Configure Laravel’s trusted proxy settings correctly so URL generation and secure-cookie behaviour recognize the original HTTPS connection.

Infrastructure controls should include:

A web root pointing only to public
No directory listing
Restricted file permissions
A database not exposed publicly
Separate database users with limited privileges
Firewall rules permitting only required ports
Protected Redis, queue and monitoring services
Encrypted backups with tested restoration
Outbound network restrictions for sensitive environments
Security headers appropriate to the API and any browser frontend

Laravel security cannot compensate for a publicly exposed database, Redis server or administration panel.

16. Keep Dependencies Supported and Patched

Check the application for known Composer advisories:

composer audit

If the project includes frontend dependencies, also review:

npm audit

An audit warning does not mean that every suggested automatic update is safe. Review:

Which direct or transitive package is affected.
Whether the vulnerable code path is used.
Which fixed version is available.
Whether the update contains breaking changes.
Whether tests and a staging deployment pass.

Keep these components on supported versions:

PHP
Laravel framework
Composer
Authentication packages
Queue and cache clients
Web server and operating system
JavaScript dependencies used by the frontend

Remove packages the application no longer uses. Every unnecessary dependency increases maintenance and attack surface.

17. Avoid User Enumeration

Authentication and account-recovery endpoints can reveal whether an email address exists.

Risky responses:

{ "message": "No account uses this email." }

and:

{ "message": "The password is incorrect." }

These allow attackers to build a list of registered users.

For login, return a generic response:

{ "message": "The provided credentials are invalid." }

For password reset, return the same public response whether or not the account exists:

{
    "message": "If an account matches that email, reset instructions will be sent."
}

Use rate limiting as well. Avoid creating obvious response-time differences between existing and nonexistent accounts.

18. Test Security Failures

Many test suites verify only that authorized users receive 200 OK. Security tests must also prove that unauthorized requests fail.

Test Cross-User Record Access

use App\Models\Invoice;
use App\Models\User;
use Laravel\Sanctum\Sanctum;

it('prevents a user from viewing another users invoice', function () {
    $owner = User::factory()->create();
    $attacker = User::factory()->create();

    $invoice = Invoice::factory()
        ->for($owner)
        ->create();

    Sanctum::actingAs($attacker);

    $this->getJson("/api/invoices/{$invoice->id}")
        ->assertForbidden();
});

If the controller intentionally scopes the query to the user’s relationship, expect 404 Not Found instead:

->assertNotFound();

Both approaches can be valid. Apply one behaviour consistently and do not leak extra record information unintentionally.

Test Mass Assignment

it('does not allow a profile update to grant admin access', function () {
    $user = User::factory()->create([
        'is_admin' => false,
    ]);

    Sanctum::actingAs($user);

    $this->patchJson('/api/profile', [
        'name' => 'Updated Name',
        'timezone' => 'Asia/Kuala_Lumpur',
        'is_admin' => true,
    ])->assertSuccessful();

    expect($user->refresh()->is_admin)->toBeFalse();
});

Depending on the validation contract, it may be better to reject unknown fields with 422 Unprocessable Content rather than ignore them. Whichever behaviour you choose, test it.

Additional Security Tests

Add tests for:

Requests without authentication
Expired or revoked tokens
Tokens missing required abilities
Normal users calling admin endpoints
Cross-tenant access
Invalid webhook signatures
Replay of the same webhook event
Excessive pagination values
Disallowed sort columns
Oversized and disallowed file uploads
Rate-limit responses
Sensitive fields missing from API resources
Error responses not containing stack traces

A test for every discovered authorization bug prevents the same vulnerability from returning later.

A Secure Laravel API Controller Example

The following controller combines authentication through route middleware, authorization, validation, safe assignment and an API Resource.

<?php

namespace App\Http\Controllers;

use App\Http\Requests\UpdateInvoiceRequest;
use App\Http\Resources\InvoiceResource;
use App\Models\Invoice;
use Illuminate\Support\Facades\Gate;

class InvoiceController extends Controller
{
    public function show(Invoice $invoice): InvoiceResource
    {
        Gate::authorize('view', $invoice);

        return new InvoiceResource($invoice);
    }

    public function update(
        UpdateInvoiceRequest $request,
        Invoice $invoice
    ): InvoiceResource {
        Gate::authorize('update', $invoice);

        $invoice->update(
            $request->safe()->only([
                'billing_name',
                'billing_address',
                'notes',
            ])
        );

        return new InvoiceResource($invoice->refresh());
    }
}

Routes:

Route::middleware([
    'auth:sanctum',
    'throttle:api-standard',
])->group(function () {
    Route::get('/invoices/{invoice}', [
        InvoiceController::class,
        'show',
    ]);

    Route::patch('/invoices/{invoice}', [
        InvoiceController::class,
        'update',
    ]);
});

This structure does not make the entire application automatically secure, but it keeps important controls visible and testable.

Laravel API Security Checklist

Use this checklist before releasing an API.

Authentication

Private routes require the correct authentication middleware.
Sanctum or Passport is used instead of a home-made token system.
Tokens can be revoked.
Token abilities are limited where useful.
Login and password-reset endpoints are rate-limited.
Sensitive accounts support multi-factor authentication where appropriate.

Authorization

Every endpoint that accepts a record ID checks access to that record.
Index endpoints return only permitted records.
Download and export endpoints are authorized.
Admin functions check roles or permissions on the server.
Cross-tenant access has dedicated automated tests.

Input and Database

Form Requests validate bodies, query strings and files.
Pagination, array and batch sizes have maximum values.
Models use carefully reviewed $fillable attributes.
Controllers update from validated allowlisted fields.
Request data is never concatenated into raw SQL.
Sort columns and query operators use allowlists.

Responses and Files

API Resources expose only required fields.
Sensitive relationships are not returned accidentally.
Private uploads are not directly public.
Uploaded type and size are validated.
Stored filenames are generated by the application.
Downloads perform authorization before returning a file.

Configuration and Operations

APP_ENV=production is set in production.
APP_DEBUG=false is set in production.
The web root points to Laravel’s public directory.
The .env file is not committed or publicly readable.
HTTPS is enforced.
Logs exclude passwords, tokens and unnecessary personal data.
Backups are encrypted and restoration is tested.
Laravel, PHP and dependencies are supported and patched.
composer audit is reviewed during the release process.
Monitoring alerts on repeated authentication failures and unusual errors.

Common Laravel API Security Mistakes

“The Route Uses `auth:sanctum`, So It Is Secure”

auth:sanctum verifies identity. It does not automatically decide which invoices, projects, files or users that identity may access.

“We Use UUIDs, So IDs Cannot Be Guessed”

UUIDs reduce simple enumeration but do not replace authorization. IDs leak through browser history, logs, screenshots, emails, analytics and related API responses.

“CORS Blocks Attackers”

CORS is enforced by browsers. It does not prevent direct requests from scripts, servers or mobile tools.

“Validation Prevents Mass Assignment”

Only if the application updates from the validated data. Calling $request->all() after validation can reintroduce unwanted fields.

“Eloquent Prevents Every SQL Injection”

Normal value binding is safe, but unsafe raw expressions and user-controlled column names can still create injection risks.

“The Frontend Hides the Admin Button”

Frontend visibility is not authorization. The backend must reject unauthorized calls.

“A Successful Request Test Is Enough”

Security depends on rejected requests. Test anonymous users, other users, lower roles, revoked tokens and malformed data.

Frequently Asked Questions

Is Laravel secure by default?

Laravel provides secure tools and sensible defaults, including password hashing, query parameter binding, validation, CSRF protection and authentication middleware. Application-specific authorization, data exposure, rate limits, token policy and server configuration remain the developer’s responsibility.

Should I use Laravel Sanctum or Passport?

Use Sanctum for most first-party SPAs, mobile applications and simple token APIs. Use Passport when the application specifically requires OAuth 2 flows and capabilities. Do not choose Passport merely because it sounds more secure; unnecessary complexity can create configuration mistakes.

Is `auth:sanctum` enough to protect an API route?

No. It authenticates the caller. You must still authorize access to records and actions using scoped queries, gates, policies, roles or permissions.

Does Laravel prevent SQL injection automatically?

Eloquent and Query Builder bind normal values, which protects common queries. Raw expressions, string concatenation and user-controlled identifiers can still be dangerous. Use bindings and allowlists.

Should an API return 403 or 404 for another user’s record?

Both approaches are used. A policy commonly returns 403 Forbidden; an ownership-scoped query commonly returns 404 Not Found. Returning 404 may reveal less about whether the record exists. Choose intentionally, keep behaviour consistent and test it.

Are UUIDs safer than numeric IDs?

They are harder to enumerate, but they are not an authorization control. Every object still requires an access check.

Should I disable CSRF protection for my API?

Do not disable it blindly. Pure bearer-token endpoints and cookie-authenticated SPA endpoints have different requirements. Sanctum’s stateful SPA authentication relies on correct cookies, domains, credentials and CSRF handling.

How should API tokens be stored in a frontend?

For first-party browser SPAs, Laravel Sanctum’s secure, HTTP-only cookie approach is normally preferable to placing long-lived tokens in browser storage. Native mobile applications should use the platform’s secure credential storage. Never expose tokens in URLs, logs or analytics.

How often should dependencies be updated?

Review security advisories continuously or during every release pipeline, apply critical fixes promptly and schedule regular supported-version updates. Test changes in staging rather than ignoring updates until the framework is no longer supported.

What is the most important Laravel API security test?

For applications containing private records, create two users and prove that one user cannot view, update, delete, download or export the other user’s data. Repeat the test across every tenant and role boundary.

Final Summary

Laravel offers strong security features, but developers must apply them consistently.

The most important improvements are:

Authenticate protected API routes.
Authorize every protected record and action.
Scope queries to the current user or tenant.
Validate all external input.
Update models only with validated, allowlisted fields.
Avoid unsafe raw SQL and user-controlled identifiers.
Apply request and resource limits.
Return explicit API Resources instead of full models.
Secure uploads and outbound URL requests.
Keep production debug mode disabled.
Protect secrets and remove them from logs.
Verify webhook signatures.
Use HTTPS and supported dependencies.
Test that forbidden requests fail.

Do not begin by adding complicated security packages. First make the application’s trust boundaries clear: who is calling, which record they are requesting, which action they want to perform, which fields they may change, and how much work one request is allowed to trigger.

When those checks are explicit, centralized and covered by tests, a Laravel API becomes much harder to misuse.

August 5, 2026