Day 9: Implementing Pagination and Filtering in Your API

In this tutorial, you’ll learn how to implement pagination and filtering in your Express API. Pagination allows users to retrieve data in small chunks instead of fetching everything at once, while filtering enables users to query data based on specific criteria.

What You Will Learn Today:

1. Implementing pagination for large datasets
2. Adding filtering to query specific data
3. Combining pagination and filtering for optimized data retrieval
4. Testing pagination and filtering using Postman

Step 1: Setting Up Pagination

Pagination allows users to request data in smaller “pages” rather than retrieving the entire dataset in one go. This is especially useful for large datasets like lists of users, products, or posts.

Modifying the Route to Implement Pagination

Let’s implement pagination for the GET /api/users route so users can fetch the data page by page.

1. Open controllers/userController.js and modify the getAllUsers controller to include pagination:

// Controller to get all users with pagination
exports.getAllUsers = async (req, res) => {
  const { page = 1, limit = 10 } = req.query; // Get page and limit from query parameters

  try {
    const users = await User.find()
      .limit(limit * 1) // Convert limit to a number
      .skip((page - 1) * limit) // Calculate the offset for pagination
      .exec();

    // Get total number of users for pagination info
    const count = await User.countDocuments();

    res.json({
      users,
      totalPages: Math.ceil(count / limit),
      currentPage: page,
    });
  } catch (error) {
    res.status(500).json({ message: 'Error fetching users', error });
  }
};

Explanation:

• page and limit: These are query parameters provided by the user to specify which page to retrieve and how many items per page. Defaults are page = 1 and limit = 10.
• User.find().limit(limit).skip(): This retrieves a limited number of users (defined by limit) and skips over records to start at the correct offset for the current page.
• countDocuments(): This counts the total number of user documents in the database, allowing us to calculate the total number of pages.

Sample API Request

To get the first page of users, you would send a request like this:

GET http://localhost:5000/api/users?page=1&limit=10

This request will return the first 10 users and information about the total number of pages.

Step 2: Adding Filtering to the API

In addition to pagination, we can add filtering capabilities so that users can retrieve data based on specific criteria, such as filtering users by their role (admin, user), age, or any other field.

Modifying the Route to Implement Filtering

1. Modify controllers/userController.js to allow filtering based on user roles:

// Controller to get all users with pagination and filtering
exports.getAllUsers = async (req, res) => {
  const { page = 1, limit = 10, role } = req.query; // Add 'role' to query parameters

  let filter = {};
  if (role) {
    filter.role = role; // Apply filter if role is provided
  }

  try {
    const users = await User.find(filter) // Apply the filter to the query
      .limit(limit * 1)
      .skip((page - 1) * limit)
      .exec();

    const count = await User.countDocuments(filter); // Apply the filter to the count query

    res.json({
      users,
      totalPages: Math.ceil(count / limit),
      currentPage: page,
    });
  } catch (error) {
    res.status(500).json({ message: 'Error fetching users', error });
  }
};

Explanation:

• filter: This object contains the criteria to filter the users by. In this case, it filters users by the role field if a role query parameter is provided.
• User.find(filter): The filter is applied to the query, allowing the API to return only the users that match the specified role.
• User.countDocuments(filter): This also applies the filter when counting the total number of documents, so pagination works correctly with filtered results.

Sample API Request with Filtering

To get the first page of users who are admin:

GET http://localhost:5000/api/users?page=1&limit=10&role=admin

This will return only the admin users and paginate the results.

Step 3: Combining Pagination and Filtering

Pagination and filtering work well together to provide users with the ability to retrieve only the data they need, efficiently. You can combine pagination and filtering to enable users to request specific subsets of data.

1. Let’s extend our API to allow filtering by multiple fields, such as role and name.

// Controller to get all users with pagination and multiple filters
exports.getAllUsers = async (req, res) => {
  const { page = 1, limit = 10, role, name } = req.query; // Add 'name' to query parameters

  let filter = {};
  if (role) {
    filter.role = role;
  }
  if (name) {
    filter.name = new RegExp(name, 'i'); // Use a regular expression to match names (case-insensitive)
  }

  try {
    const users = await User.find(filter)
      .limit(limit * 1)
      .skip((page - 1) * limit)
      .exec();

    const count = await User.countDocuments(filter);

    res.json({
      users,
      totalPages: Math.ceil(count / limit),
      currentPage: page,
    });
  } catch (error) {
    res.status(500).json({ message: 'Error fetching users', error });
  }
};

Explanation:

• name: We added the ability to filter by name. Using a regular expression allows for partial matching (e.g., searching for Joh will return users with names like “John”).
• Multiple filters: The API now supports filtering by multiple fields, allowing users to refine their searches further.

Sample API Request with Multiple Filters

To get the first page of users who are admin and whose name contains “John”:

GET http://localhost:5000/api/users?page=1&limit=10&role=admin&name=John

This request will return only the admin users whose name contains “John” and paginate the results.

Step 4: Testing Pagination and Filtering with Postman

Let’s test the pagination and filtering functionality using Postman.

1. Testing Pagination

1. Open Postman and create a GET request to http://localhost:5000/api/users?page=1&limit=5.
2. You should receive a response with the first 5 users and information about the total number of pages.

2. Testing Filtering

1. Create a GET request to http://localhost:5000/api/users?role=admin.
2. You should receive a response containing only users with the role of admin.

3. Testing Combined Pagination and Filtering

1. Create a GET request to http://localhost:5000/api/users?page=1&limit=5&role=user&name=Jane.
2. You should receive a paginated response containing users with the role of user and whose name contains “Jane”.

Step 5: Recap and Summary

In this tutorial, you learned how to implement pagination and filtering in your Express API to optimize data retrieval. Here’s a quick summary of what you’ve done:

• Added pagination to your API, allowing users to retrieve data in small chunks (pages).
• Added filtering capabilities to allow users to query data based on specific criteria (e.g., by role or name).
• Combined pagination and filtering to provide users with refined and efficient data retrieval options.
• Tested the pagination and filtering functionality using Postman.

With pagination and filtering implemented, your API is now more efficient and can handle large datasets without overloading the client or server.

Next Up: Day 10 – Optimizing and Deploying Your API

In Day 10, the final part of this series, we’ll focus on optimizing and deploying your API to production. You’ll learn how to fine-tune performance and deploy your API using services like Heroku or AWS.

Stay tuned for more advanced features tomorrow!