Prometheus and Grafana Part 2
January 22, 2019

How To Visualize Prometheus Data with Grafana

Analytics

Prometheus and Grafana are growing in popularity. Here, we break down how to visualize Prometheus data with Grafana. And we give you some Grafana Prometheus dashboard examples.

What Is Prometheus and Grafana?

Prometheus is an open source monitoring and alerting toolkit. Grafana is an open source visualization and analytics software.

How to Use Prometheus and Grafana

Here's you can use Prometheus and Grafana to create strong and visually appealing dashboards. 

Although Prometheus can graph data, Grafana provides a full framework for sharing dashboards, creating advanced queries and graphs, and allowing for sharing and reuse of those dashboards.

So that you can go from this:

Prometheus can graph data

To this:

Grafana provides a full framework

Revisiting our architecture diagram from part 1, recall that Grafana can directly query Prometheus to render the data it needs for its dashboard. Prometheus is a native data store for Grafana, so configuration is very simple. Grafana itself persists data about the dashboards it has saved, but no data is actually held in Grafana. This makes it a very lightweight and portable solution. So, let’s get into it!

First we'll get Grafana up and running — and connected to Prometheus. 

1. Find a Server For Grafana

First, find a suitable server for Grafana, preferably a Linux-based server, and download the appropriate distribution and unarchive it. The default config is good enough to get you started, so just execute the server:

execute the server

 

Note that to do this, we just run the grafana-server script from within the /bin directory of the Grafana release.

2. Make the Port Accessible on the Server

You’ll see that Grafana starts up on port 3000, so make sure that port is accessible on the server and open it up in a web browser by browsing to http://[hostname]:3000. Login with the default credentials (username ‘admin’ and password ‘admin’) and you’ll be greeted with the initialization screen:

initialization screen

3. Add a Data Source

We need to add a data source, which will be our Prometheus server on port 9090 by default. Click “Add Data Source”, select Prometheus, and add the relevant details:

Add Data Source screen
Add Data Source screen

Of course, you’ll replace localhost with the hostname of your Prometheus server. If you need anything else, like authentication for instance, those details can be set up below:

Add Data Source screen

The default values should be fine for what we’ve worked on so far in this blog.

4. Save and Test Prometheus on Grafana

Click Save and Test, and ensure that the connectivity is successful:

Click Save and Test

That’s it! Grafana is now capable of querying metrics from Prometheus!

Next, we'll build our panel. 

5. Choose the Prometheus Data to Query

Now that we have Grafana gathering metrics, let’s choose some data that we want to query. The easiest way to do this is to find the exact query coordinates using Prometheus, and copy them as a query into Grafana.

6. Open Prometheus

Let’s start with System CPU Load. If we open Prometheus (by default, http://[hostname]:9090, and start typing “cpu” into our Prometheus query, we’ll see that we get some metric labels back:

metric labels

We can select one of those metrics and click Execute to find the full coordinates:

click Execute to find the full coordinates

Since there may be multiple metrics, say for instance, Enqueue Time across multiple queues in ActiveMQ, getting the specific coordinate from Prometheus is essential in making sure that we are monitoring the right data. This is better illustrated by looking at a metric like EnqueueCount:

looking at a metric like EnqueueCount

Notice that we have multiple points being gathered across differing “destinationName” objects.

[Related Blog: Prometheus Monitoring]

 

How to Create Grafana Prometheus Dashboard (Examples)

Here are some examples of a Grafana Prometheus dashboard. 

7. Create a Grafana Prometheus Dashboard

Once we have the right metric coordinates captured, it’s time to create our first Dashboard in Grafana. Click the Grafana Logo to get the side toolbar, and then click “+” followed by “Dashboard”:

Grafana side toolbar

This will create a new dashboard and add our first panel:

a new dashboard and add our first panel

8. Configure Your Dashboard For the First Panel

Grafana is arranged into independent panels that can have widgets added to them. These panels can be resized and dragged around the screen. Let’s create a graph of our SystemCPULoad metric. Click “Graph,” and a blank default Graph will be created:

a graph of our SystemCPULoad metric

Click the Panel Title to get a toolbar, and then click “Edit”:

Panel Title to get a toolbar

This is where we can configure all the details for our graph:

configure all the details

There’s a lot going on in there. For now, copy the SystemCPULoad query that we pulled from Prometheus earlier into the query line:

 copy the SystemCPULoad query

Upon clicking off of the query, you’ll see that Grafana immediately executes the query and starts rendering graph data:

 copy the SystemCPULoad query

Move to the General tab, and let’s give our Graph a suitable name:

give our Graph a suitable name

Now let’s label our Y axis and change our number format to Percent(0.0-1.0). Click on the Axes tab and make those updates:

Click on the Axes tab

Since we aren’t monitoring multiple metrics in this Graph, we don’t need a Legend, so let’s turn it off via the Legend tab:

turn it off via the Legend tab

 That’s it! Click the X to the upper right, and we should see the following finished product as your Grafana Prometheus dashboard example:

see the finished product

Note that we can resize this graph by using the indicated handle on the lower right of the graph.

Next, we'll show you how to add more metrics.

9. Configure Your Dashboard to Monitor More Prometheus Metrics

1) Great start! But we’ll want to monitor more metrics. Let’s add another Panel for our Enqueue Count metrics. Click the Add Panel button:

add another Panel

This will bring up our familiar new Panel interface. Choose “Singlestat,” then drag the System CPU Load graph to position it over the new Panel by clicking the title of the Panel and dragging the graph:

Choose “Singlestat,”

Just like in our Graph creation, click the “Panel Title” and edit it, and this time enter the EnqueueCount metric that we grabbed from Prometheus:

click the “Panel Title” and Edit it

You’ll notice that when you click off, this metric may not match with the value you saw in Prometheus. That’s because the default stat type for a Single Statistic is an Average Value rather than an absolute value. Click the Options tab and choose “Current” for “Stat”:

Click the Options tab

Click “General” and give it a title, and we’re good to go!

Click “General” and give it a title

Continue adding and resizing panels and metrics until you’re happy with the final results, then click the Gear Icon to give your dashboard a name:

click the Gear Icon to give your dashboard a name

 

click the Gear Icon to give your dashboard a name

Finally, click the “Save” icon to save your finished product:

click the “Save” icon to save your finished product

10. Enhance Your Prometheus Grafana Dashboard

There’s a lot you can do to further enhance your dashboard. You can drill down into a period of time by clicking and dragging on a section of the graph:

drill down into a period of time

Which will update your time range:

update your time range

And you can click the time range to get a drop-down with various presets, or set a range there:

click the time range

Get familiar with the various options that are available to you and keep a lookout for a future blog post that will detail more advanced layout and query functionality within Grafana. And there we have it! We now have Prometheus gathering metrics from a Java application, and Grafana querying Prometheus and displaying that data!

Get More Out of Your Grafana Prometheus Dashboard

Stay tuned for part 3 of this series, where we will learn how to make use of Prometheus’s AlertManager functionality to set threshold and alerts for our Java application data. We’re here to help you And in the meantime, OpenLogic architects are available to assist you with this and other popular open source solutions! Contact us today.

TALK TO AN EXPERT