Prometheus monitoring can be tricky. In this blog, we share tips to make it easier. Read along or jump to the section that interests you most:
Prometheus is a monitoring tool. Prometheus is a powerful and popular open source time series tool and database. It stores and exposes metrics and statistics. It was originally used to monitor containers running within Kubernetes. But it has quickly evolved into a flexible, enterprise-ready monitoring solution.
Here's how to export your data from Prometheus.
First, gather metrics that aren’t generated within a Kubernetes clustering environment.
Let’s take a quick look at Prometheus’ architecture. Here, we're looking at Java application performance monitoring (APM):
Prometheus can gather metrics in several different ways. Batch jobs and ephemeral workload applications can push metrics into Prometheus. This is done using its Pushgateway. Prometheus can also discover new metrics running inside of supported environments like Kubernetes. This is done by using its Service Discovery mechanisms.
It can also pull metrics from Prometheus exporters, which is what we’ll be doing. Alerts can be triggered via the AlertManager. This can interact with downstream systems like PagerDuty. And visualization engines like Grafana can query data out of Prometheus and display it.
Prometheus Exporters are used to export data metrics from Prometheus. You'll need to download and install a Prometheus exporter in order to pull metrics.
There are different types of Prometheus exporters you can use. Here, we'll focus on JMX Exporter.
JMX Exporter is a Java-native specification. It allows developers to expose a Java application’s metrics, statistics, and basic operations in a standard way from Prometheus.
Here, we’ll use a special type of Prometheus exporter called a JMX (Java Management Extensions) Exporter. The Prometheus JMX exporter exposes a Java application’s JMX objects in a way that is friendly for Prometheus to consume.
A Java application can create JMX objects (called mBeans or Management Beans). These can store data or give access to application operations. Some of these objects are standard to the JVM. Others are specific to a particular application.
For instance, here’s a common JMX introspection tool called JConsole. This tool ships with most variants of the Java JDK.
Here, it is browsing objects in a running instance of ActiveMQ:
Well-constructed Java applications will expose lots of relevant and helpful metrics. These are meant to be consumed by monitoring solutions like Prometheus.
Most enterprise-class monitoring solutions will support gathering JMX metrics. This is considered a standard and practical way to gather data about running Java applications. JMX can also provide useful insight into the health of the JVM, by inspecting the mBeans inside of java.lang:
So, as you can see, there’s plenty of delicious monitoring data just waiting to be consumed!
There are two distributions available. I find that the native JMX Java Agent is the easiest to work with, but there is also a “standalone” HTTP JMX Exporter available. We’ll use the Java Agent in this post. Head here and click the “Download Jar” link:
Place the downloaded .jar file somewhere accessible to the Java application. If you want to reuse the agent for multiple Java applications running on the same instance, you can place it in a common area.
In my case, I’ve just dropped it in the bin directory of ActiveMQ, but it really doesn’t matter where you place it. For this demo (on a test machine – don’t do this with root in real life), I’ve just dropped it into its own directory called /root/prometheus-jmx:
In our case, we’ll use a very basic configuration that will expose all metrics. But this file can be used to provide fine-grained control over which mBeans are exported. It can also configure HTTPS:
[[email protected] prometheus-jmx]# cat config.yml
Save the config.yml file somewhere convenient, I’ve just put it in the same directory as the .jar as you can see above.
Now you’ll need to add a JVM parameter to the Java application itself, requesting that it load the agent on application startup. Normally, a Java application will have a configuration file for specifying JVM parameters. In the case of ActiveMQ, that is the /bin/env file. Add the following parameter:
For instance, in ActiveMQ’s env file, I’ve updated the variable that will be passed into the app as a JVM parameter:
ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS_MEMORY -javaagent:/root/prometheus-jmx/jmx_prometheus_javaagent-0.3.1.jar=8080:/root/prometheus-jmx/config.yml -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=$ACTIVEMQ_CONF/login.config"
That’s it! Fire up your Java application as normal, and verify that you can see metrics being exposed in a simple text format when you visit http://[HOSTNAME]:[HTTP_PORT]/metrics:
Here's how Prometheus monitoring works. Watch the webinar recording below or keep reading for a recap.
If you haven’t already downloaded Prometheus, do so and extract it.
We’ll need to create a new config file (or add new tasks to an existing one). We’ll create a file called prom-amq.yml and place it in Prometheus’s root directory (though you can place yours anywhere):
Here’s a sample configuration file:
- job_name: 'activemq'
- targets: [localhost:8080']
Seta global scrape_interval (a global amount of time between Prometheus gathering data from this source) in global settings. Configure a label called “monitor” named “activemq”.
Under “scrape_configs”, create a job_name called “activemq”, override the global “scrape_interval” for this job to 5 seconds. Tell Prometheus to hit “[hostname]:8080” for the data. This will correspond to the hostname and port that you configured in the JMX Exporter config.
Launch Prometheus, passing in the configuration file as a parameter, by running the following command:
At this point, you should have a functional Prometheus instance gathering data from the JMX Exporter. Open Prometheus in a web browser and click Status -> Targets, and validate that the State is Up:
Query some data and ensure that it matches what you see in the JMX Exporter Output. Click the Prometheus section to return to the main screen, and use the pull-down next to the “Execute” button to select a metric and view its contents:
Prometheus monitoring is incredibly useful for Java applications. But it can be tricky to get the most out of Prometheus applications without expert help. Luckily, the open source experts at OpenLogic can help.
Our experts are skilled in Prometheus, Grafana, and many other analytics solutions. Find out how our open source experts can help you get the most out of your data analytics solutions. Get in touch today.
Talk to a Prometheus Expert
Chief Architect, Perforce Software
Justin has over 20 years of experience working in various software roles. He is an outspoken free software evangelist, delivering enterprise solutions, technical leadership, and community education on databases, architectures, and integration projects.