Monitoring Java Applications with Prometheus and Grafana: Part 1
Gathering Java metrics inside of Prometheus
Prometheus is a powerful and popular open source time series database, with a focus on storing and exposing metrics and statistics. It was originally conceived as a way to monitor containers running within Kubernetes but it has quickly evolved into a flexible, enterprise-ready monitoring solution.
In order to take full advantage of Prometheus, and its visualization counterpart Grafana, we’ll need to be able to gather metrics that aren’t generated within a Kubernetes clustering environment. To understand how Prometheus can gather these metrics, let’s take a quick look at Prometheus’ architecture, with respect to monitoring Java applications:
Note from the diagram that Prometheus can gather metrics in several different ways. Applications such as batch jobs and ephemeral workloads can push metrics into Prometheus using its Pushgateway. Prometheus can discover new metrics running inside of supported environments like Kubernetes by using its Service Discovery mechanisms. It can also pull metrics from Prometheus exporters, which is what we’ll be doing. Alerts can be triggered via the AlertManager, which can interact with downstream systems like PagerDuty. And visualization engines like Grafana can query data out of Prometheus and display it, which are concepts we will visit in the next blog.
In this blog series, we’ll use a special type of Prometheus exporter called a JMX (Java Management Extensions) Exporter. The exporter exposes a Java application’s JMX objects in a way that is friendly for Prometheus to consume.
JMX is a Java-native specification that allows developers to expose a Java application’s metrics, statistics, and basic operations in a standard way.
An overview of JMX can be found here.
For this post, just understand that a Java application can create JMX objects (called mBeans or Management Beans) that can store data or give access to application operations. Some of these objects are standard to the JVM, while others are specific to a particular application.
For instance, here’s a common JMX introspection tool called JConsole. This tool ships with most variants of the Java JDK. Here, it is browsing objects in a running instance of ActiveMQ:
Well-constructed Java applications will expose lots of relevant and helpful metrics, meant to be consumed by monitoring solutions like Prometheus. Most enterprise-class monitoring solutions will support gathering JMX metrics, as it is considered a standard and practical way to gather data about running Java applications. JMX can also provide useful insight into the health of the JVM, by inspecting the mBeans inside of java.lang:
So, as you can see, there’s plenty of delicious monitoring data just waiting to be consumed!
How to bring monitoring data into Prometheus
Enough theory -- let’s get right into how to bring this data into Prometheus!
Goal 1: Configuring the JMX Exporter
1) Download the Java JMX Exporter jar. There are two distributions available. I find that the native JMX Java Agent is the easiest to work with, but there is also a “standalone” HTTP JMX Exporter available. We’ll use the Java Agent in this post. Head here and click the “Download Jar” link:
2) Place the downloaded .jar file somewhere accessible to the Java application. If you want to reuse the agent for multiple Java applications running on the same instance, you can place it in a common area. In my case, I’ve just dropped it in the bin directory of ActiveMQ, but it really doesn’t matter where you place it. For this demo (on a test machine – don’t do this with root in real life), I’ve just dropped it into its own directory called /root/prometheus-jmx:
3) Configure the config.yml file for your application. In our case, we’ll use a very basic configuration that will expose all metrics, but, this file can be used to provide fine-grained control over which mBeans are exported, as well as configure HTTPS:
4) Save the config.yml file somewhere convenient, I’ve just put it in the same directory as the .jar as you can see above 5) Now you’ll need to add a JVM parameter to the Java application itself, requesting that it load the agent on application startup. Normally, a Java application will have a configuration file for specifying JVM parameters. In the case of ActiveMQ, that is the /bin/env file. Add the following parameter:
For instance, in ActiveMQ’s env file, I’ve updated the variable that will be passed into the app as a JVM parameter:
6) That’s it! Fire up your Java application as normal, and verify that you can see metrics being exposed in a simple text format when you visit http://[HOSTNAME]:[HTTP_PORT]/metrics:
Goal 2: Consuming metrics with Prometheus
1) If you haven’t already downloaded Prometheus, do so and extract it. 2) We’ll need to create a new config file (or add new tasks to an existing one). We’ll create a file called prom-amq.yml and place it in Prometheus’s root directory (though you can place yours anywhere):
3) Here’s a sample configuration file:
4) Under our global settings, we set a global scrape_interval (a global amount of time between Prometheus gathering data from this source), and we configure a label called “monitor” named “activemq”. Under “scrape_configs”, we create a job_name called “activemq”, override the global “scrape_interval” for this job to 5 seconds, and tell Prometheus to hit “[hostname]:8080” for the data, which will correspond to the hostname and port that you configured in the JMX Exporter config. 5) Launch Prometheus, passing in the configuration file as a parameter, by running the following command:
Goal 3: Validate data gathering
1) At this point, you should have a functional Prometheus instance gathering data from the JMX Exporter. Open Prometheus in a web browser and click Status -> Targets, and validate that the State is Up:
2) Query some data and ensure that it matches what you see in the JMX Exporter Output. Click the Prometheus section to return to the main screen, and use the pull-down next to the “Execute” button to select a metric and view its contents:
Although Prometheus is great at grabbing and maintaining this data, its data visualization capabilities are quite limited compared to a solution like Grafana. So, read Part 2 of this blog series, where I’ll show you how to query this data from Grafana and start building sophisticated dashboards and visualization.
In the meantime, OpenLogic architects are available to assist you with this and other popular open source solutions! Contact us today.