Your application just slipped into unconsciousness, and now any request you make just hangs. To make matters worse, your logs don’t show anything useful. So how can you check your application's vital signs when it drops on you like this?
All your production applications should be under critical care with monitoring systems that continually look at key statistics and life signs. Thus, if an application becomes unresponsive, you just need to glance at the monitor on your office wall to see the statistics for your environment.But what if somehow you find yourself in a position where you have a critical problem with an application but don’t have any monitoring infrastructure? How can you quickly check the following vital signs?
Ideally, you should never be in this position with production systems. There should always be some type of monitoring infrastructure in place to help you respond to these types of problems. Only development—and sometimes functional testing QA environments—should be routinely setup without monitoring in place. Monitoring is an essential part of production environments as well as very useful feature to have in load testing environments.
Having said that, if you're experiencing application problems and don’t have monitoring in place, you’d undoubtedly prefer solutions to lectures about best practices that haven’t been followed. So, let's cut to the chase and show one possible solution.
In most situations you should have a monitoring system setup to look at all the information covered below. This article is written for the rare situation in which you do not having a monitoring system.
If your JBoss installation is mostly unaltered from its defaults, it is likely you still have the JMX Console, Tomcat status and Admin Console enabled, and you can use them to get answers to key questions that will help you check your application’s vital signs. If you have them enabled, good for you. But if you leave them on in a production system, don't forget to at least secure them—the mechanisms presented here should be at least secured if not disabled in production systems.Note that there are many reasons why an application might hang, and also that there are many other things you can check (e.g. EJB pools) that are not covered here. This article is not intended to be a detailed course in JBoss and application brain surgery, but rather just a brief paramedic’s manual covering the most common signs and causes of unresponsive behavior. In other words, this article is intended to help you determine what’s wrong, not how to cure it - that might be a topic for subsequent articles.
To access the JMX Console go to http://localhost:8080/jmx-console, where localhost is your server's name. Tomcat status is available at http://localhost:8080/status?full=true. And (if you're using a JBoss version that includes it) the admin console is at http://localhost:8080/admin-console.
Note that there are differences between JBoss 4 and JBoss 5.1. We will first talk about methods that work in both JBoss 4 and JBoss 5, and then present a somewhat easier method that works only in JBoss 5.1.
Investigating data source usage is useful when you’re unsure if you have enough data sources available for application. If you ran out of the available data source connections, your system is unresponsive because your queries are stuck.
To find statistics about data source usage, you need first to find the name of your data source. Let’s suppose that it’s named MyDataSource.Go to the JMX console, and find the section called jboss.jca. You should see MBean there with the object name:name=MyDataSource,service=ManagedConnectionPoolNote: There are multiple similarly named entries in the JMX console, so make sure that you use right one in the right section for the right data source!Click on that link, and the following arguments are especially useful:
Checking connector thread usage is useful when you want to find out how many of the available request threads are used. If all connector threads are blocked, your incoming requests are waiting on the current requests to proceed before their processing can be started.
To find statistics about connector usage, the best way is to look at http://localhost:8080/status?full=true. This will give you the following statistics:
There are many other useful things available here, including a list of all currently-deployed applications and per-servlet statistics.
Be sure that you are looking at the statistics for the right connector for your setup - AJP connector if you are using AJP (e.g. setup with Apache HTTPD, mod_jk, and JBoss) or HTTP connector if your HTTP requests go directly to JBoss. The default configuration has AJP connector listening on port 8009 and HTTP connector on port 8080.The picture below shows statistics for the HTTP connector on a recently-started server with only one user:
Server information is useful when your questions are:
This line of questioning is helpful to collect basic version information for the software you are using, as well as data about what is your server is currently doing.
You probably know the versions and configuration if you were the person who installed the system, and this information is always available in the JBoss boot.log file. You can also double-check this information with only browser access by finding the following three entries in the jboss.system section of the JMX Console.These three beans provide answers to all of the questions listed above:
If your CPU usage is stuck at 100% all the time, it might be an indication of multiple problems such as pooling when you shouldn’t, memory leaks, and CPU-intensive code.
A thread dump can help you detect pooling or infinite loop types of problems. Do multiple thread dumps and compare the results to determine if you’re still stuck in the same place on some threads. If these threads have high CPU usage as well, they’re doubly suspicious. Both of these can be checked in the SystemInfo bean, as discussed in the previous section.Memory leaks are slightly more complex. For additional details on fixing memory leaks please see the tutorial, How to Fix Memory Leaks in Java.
All of the techniques covered so far apply to both JBoss 4 and JBoss 5, but the JMX Console is somewhat intimidating and may provide the impression of a "big, difficult to navigate pile of MBeans." JBoss 5.1 comes to the rescue here with an admin console that delivers a more user-friendly way of obtaining important information.
You can access the JBoss 5.1 admin console at http://localhost:8080/admin-console. All of the previously-mentioned information is readily available within the admin console. Look at the tree on the left side.At the root of the tree you can get information about the OS version.To get information about the JBoss version and configuration, click on the server in the left side tree view (e.g., JBoss AS 5, which is the default entry in the tree).Information about the data source is located under JBossAS Servers/YourServer/Resources/Datasources in the tree on the left side. Find your data source, and then look at both the Summary and Metrics panels on the right side. The example above shows the DefaultDS data source, but your data source is probably different. Be sure to look at statistics for your own data source—not, for example, of DefaultDS!To find connector statistics, look in the tree on the left side under JBossAS Servers/YourServer/Resources/JBoss Web/Connectors.
If you prefer to work from a command line, you can get the information covered above by using the twiddle.sh command line utility located in the $JBOSS_HOME/bin. The names of the MBeans that you should use in twiddle.sh are the same as the names of the MBeans you are using in JMX Console.
Sometimes, but not always. A lot depends on the situation. What to do in each situation involves some degree of “it depends” and may be the subject of future articles.
What if you don't have monitoring installed, can't get monitoring information from any application, and your consoles and twiddle.sh don't work? Or if all of the vital signs covered above look normal but you are still stuck with a comatose application? Well, in this case there's very little that JBoss paramedic techniques can do to help, and you might need to perform a little exploratory surgery to get more information.
You have couple of options here:
The JBoss Group: "JBoss 4.0 - The Official Guide", Sams Publishing, April 30, 2005, ISBN 978-0672326486
Javid Jamae, Peter Johnson: "JBoss in Action: Configuring the JBoss Application Server", Manning Publications, January 28, 2009, ISBN 978-1933988023http://www.jboss.org/community/wiki/SecureTheJmxConsolehttp://www.jboss.org/community/wiki/AccessControlForJMXConsole
Allowed tags: <a> link, <b> bold, <i> italics