Day 9: Monitoring Cloud Applications with Prometheus and Grafana

Overview:
In cloud environments, monitoring is essential to ensure your applications are running smoothly and to quickly identify any performance bottlenecks or issues. Prometheus is an open-source monitoring and alerting toolkit that helps you collect and query metrics, while Grafana provides a beautiful, customizable interface for visualizing those metrics.

In this tutorial, we will walk you through setting up Prometheus and Grafana to monitor a cloud application running on Docker. By the end, you’ll have a monitoring stack that visualizes the metrics of your cloud application.

Table of Contents

Prerequisites:

Docker installed.
Basic knowledge of cloud applications and containerization.
A simple application (we’ll use Node.js for this tutorial).

Step 1: Setting Up a Simple Web Application

We’ll start by creating a simple Node.js web application and exposing some metrics that Prometheus can scrape.

Create a project directory:

mkdir monitoring-app
cd monitoring-app

Initialize a Node.js project:

npm init -y

Install Express and Prometheus Client:

npm install express prom-client

Create a server.js file:

touch server.js

Add the following code to server.js:

const express = require('express');
const client = require('prom-client'); // Prometheus client library

const app = express();
const PORT = process.env.PORT || 3000;

// Create a default registry and a counter
const register = new client.Registry();
const httpRequestCounter = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
});

register.registerMetric(httpRequestCounter);
register.setDefaultLabels({ app: 'monitoring-app' });

client.collectDefaultMetrics({ register });

// Simulate HTTP requests and update the counter
app.get('/', (req, res) => {
  httpRequestCounter.inc();
  res.send('Hello from the monitored app!');
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

This simple Express application exposes an endpoint /metrics, where Prometheus can scrape the metrics. We’re using prom-client to track basic application metrics, such as HTTP requests.

Step 2: Dockerizing the Application

We’ll run the Node.js application in a Docker container for consistency across environments.

Create a Dockerfile:

touch Dockerfile

Add the following content to the Dockerfile:

# Use official Node.js image as the base image
FROM node:18-alpine

# Set working directory in the container
WORKDIR /usr/src/app

# Copy package.json and install dependencies
COPY package*.json ./
RUN npm install

# Copy the application code
COPY . .

# Expose port 3000 for the app
EXPOSE 3000

# Run the application
CMD ["node", "server.js"]

Build the Docker image:

docker build -t monitoring-app .

Run the Docker container:

docker run -p 3000:3000 monitoring-app

The application should now be running at http://localhost:3000, and Prometheus can scrape metrics from http://localhost:3000/metrics.

Step 3: Setting Up Prometheus

Now that we have an application exposing metrics, we need to set up Prometheus to collect them.

Create a directory for Prometheus:

mkdir prometheus
cd prometheus

Create a Prometheus configuration file:

touch prometheus.yml

Add the following content to prometheus.yml:

global:
  scrape_interval: 5s  # How often to scrape metrics

scrape_configs:
  - job_name: 'monitoring-app'
    static_configs:
      - targets: ['host.docker.internal:3000']  # Use 'localhost:3000' for non-Docker environments

This configuration tells Prometheus to scrape metrics from the /metrics endpoint every 5 seconds.

Create a Dockerfile for Prometheus:

touch Dockerfile

Add the following content to the Dockerfile:

FROM prom/prometheus

# Add the Prometheus config file
COPY prometheus.yml /etc/prometheus/prometheus.yml

Build and run the Prometheus Docker container:

docker build -t prometheus .
docker run -p 9090:9090 prometheus

Prometheus should now be running at http://localhost:9090. You can visit the Prometheus web UI and explore the metrics collected.

Step 4: Setting Up Grafana

Grafana will help us visualize the data Prometheus collects from our application.

Run the Grafana Docker container:

docker run -d -p 3001:3000 grafana/grafana

Grafana should now be accessible at http://localhost:3001.

Log in to Grafana:

Username: admin
Password: admin (you can change this upon login).

Add Prometheus as a Data Source:

Go to Configuration -> Data Sources -> Add data source.
Select Prometheus.
Set the URL to http://host.docker.internal:9090 (or http://localhost:9090 if not using Docker).
Click Save & Test.

Step 5: Creating Dashboards in Grafana

Now that Grafana is set up, let’s create a dashboard to visualize the metrics.

Create a new dashboard:

Go to Dashboards -> New Dashboard.
Click Add New Panel.

Query Prometheus for Metrics:

In the query editor, type http_requests_total.
You should see the total number of HTTP requests that your application has handled.

Visualize the Data:

Select the type of visualization you want (e.g., graph, bar chart).
Click Apply to add the panel to your dashboard.

Save the Dashboard:

Click Save Dashboard, give it a name, and save it.

Now you have a real-time monitoring dashboard that shows key metrics from your application.

Step 6: Setting Up Alerts in Prometheus (Optional)

Prometheus allows you to set up alerts based on certain thresholds, such as high CPU usage or increased response times.

Modify the prometheus.yml file to add alerting rules:

rule_files:
  - "alert.rules"

scrape_configs:
  - job_name: 'monitoring-app'
    static_configs:
      - targets: ['host.docker.internal:3000']

Create an alert.rules file:

touch alert.rules

Add an alert rule for high request rates:

groups:
  - name: example_alert
    rules:
    - alert: HighRequestRate
      expr: rate(http_requests_total[1m]) > 5
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "High request rate detected"

This rule will trigger an alert if the request rate exceeds 5 requests per second for more than 1 minute.

Restart the Prometheus container to apply the changes.

Step 7: Scaling the Setup

As your application grows, you can scale the monitoring setup to include multiple services, applications, and environments. You can also integrate with cloud-based metrics services like AWS CloudWatch for hybrid monitoring.

7.1 Multi-Application Monitoring

You can easily extend this setup to monitor multiple applications by adding more scrape_configs to the prometheus.yml file.

scrape_configs:
  - job_name: 'app1'
    static_configs:
      - targets: ['host.docker.internal:3000']

  - job_name: 'app2'
    static_configs:
      - targets: ['host.docker.internal:4000']

7.2 Cloud-Based Metrics

You can combine Prometheus and Grafana with cloud-based services (like AWS CloudWatch or Google Cloud Monitoring) to create a comprehensive monitoring stack that works across on-premise and cloud environments.

Conclusion

In this tutorial, you set up a complete monitoring stack using Prometheus and Grafana to monitor a cloud application running in Docker. You configured Prometheus to scrape application metrics, set up Grafana to visualize those metrics, and created custom dashboards to track performance.

Prometheus and Grafana provide powerful tools for monitoring cloud applications, offering flexibility, scalability, and real-time insights. You can extend this setup to monitor more complex cloud environments, set up advanced alerts, and integrate with other cloud-native tools.

Feel free to copy the code and configuration to set up monitoring for your own applications!

Post Views: 276