Micro Focus Diagnostics – Tips and Tricks

Table of Contents

Micro Focus Diagnostics – Tips and Tricks – Mar 2021

1. Diagnostics issues with integration to BSM

The relation between the diagnostics server and BSM is lost, and the integration stops working at random. 

Diagnostics Server often deregisters itself with HP BSM after effectively applying it to Business Service Manager (BSM).

The following error is recorded in the Diagnostics server.log: 

SEVERE   registrar           : Attempt to set RegistrarData.lanID to null!
java.lang.NullPointerException: Attempt to set RegistrarData.lanID to null!
…
WARNING  diag_server         : AdminLogin referrer ('http://BSM_SERVER:80') does not appear to be us (DIAGNOSTIC_SERVER_MACHINE) or the currently registered BSM ('10.121.6.3').

The following steps can be implemented  to resolve the issue: 

  • The following workaround must be used in order to resolve this issue: 
  1. Access the commander’s machine. 
  2. Open: MercuryDiagnostics\Server\etc\webserver.properties
  3. This line should be added to the properties file: bac.registration.check.referer=false
  4. Reboot the commander
  5. Check to see if the issue has been resolved.

2. The Diagnostics .NET Agent installation fails when the optional Probe Aggregator Service is selected

What is Diagnostics .NET Agent? 

The Diagnostics.NET Agent is a program that allows you to perform diagnostics on your server. It is mounted on the server that hosts the application you’d like to monitor. The.NET AppDomains you want to control are automatically discovered and instrumented during agent installation and setup.

Process invocations, collection points, and the start and end of business and server transactions are all captured by the handler. Many of Software’s Diagnostics products, such as LoadRunner, Performance Center, and BSM, are compatible with the.NET Agent.

If the installation of the.NET Agent may be hampered by security restrictions then you can follow the below solution

  1. When attempting to install the Diagnostics.NET Agent on a Windows Server platform with the optional Probe Aggregator Service selected, the installation fails. 
  2. In the report, the following error is mentioned: “ProbeAggregator/log/ProbeAggregator/JSW” file:
  3. FATAL  | wrapper  | 2014/04/18 16:54:03 | OpenSCManager failed – Access is denied. (0x5)
  4. The following error may also be found in the “/setup/SetupResults” file has:
  5. Running command: C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\bin\ProbeAggregator-windows-x86-64.exe -i “C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\etc\ProbeAggregatorJSW.conf”
  6. Error message: . Error number: 1
  7. Probe Aggregator setup failed.
  8. Removing temporary setup files folder: C:\MercuryDiagnostics\.NET Probe\setup\pasetupfile

This problem may be induced by the installation platform’s security restrictions.

The following steps can be implemented  to resolve the issue: 

  • When installing the.NET Agent with the Probe Aggregator Service, 
  1. Make sure the user installing it is a complete administrator. 
  2. User Account Control (UAC) is turned off. 
  3. That the installer is started by right-clicking and selecting “Run as administrator…”

3. How to set up SSL on a Diagnostics server using a certificate from CA authority?

What is an SSL Certificate? 

SSL-Secure Sockets Layer is a security protocol that is widely used on e-commerce sites and pages where users must submit personal or credit card details. An SSL certificate is a form of digital certificate that enables an encrypted link and provides authentication for a website.

These certificates inform the client that the web service host demonstrated domain ownership to the certificate authority when the certificate was issued. 

  • Instructions to set up HTTPS with CA certificate
  • How to set up SSL on a Diagnostics server using a certificate from CA authority?

How keystore and certificates work in Java?

  • The keystore file is Java’s registry for certificates and keys. Keytool, which comes bundled with every JRE, is a native tool for managing keystore data (Java Runtime Environment). 
  • A keystore contains all imported (trusted) certificates for outbound HTTPS connections, as well as their public keys, but it also contains the SERVER certificate and its specific private key, which is used to serve incoming HTTPS connections. 
  • When using keytool to create a keystore, you have two choices for the SERVER certificate: 
  1. Create a file containing a private key and a self-signed certificate from it.
  2. Create a file containing a private key and a certificate request file (.csr) containing that key, so that a CA authority can generate a certificate based on that key. For SSL to operate in the SERVER, the certificate must be imported to the keystore file containing the matching private key after it is generated by the CA authority. 
  • The Diagnostics Install & Configuration Guide has instructions for option #1. 
  • Follow these generic keytool instructions to enforce option #2:
  1. Generate keystore:
<diagnostics_server_install_dir>/_jvm/bin/keytool -genkey –keystore <diagnostics_server_install_dir>/etc/keystore -storepass <password> -alias SERVER -keyalg RSA -keypass <password> -dname "CN=<diagnostics_server_hostname>, OU=Diagnostics, O=Hewlett-Packard, L=Palo Alto, S=CA, C=USA" -validity 3650
  1. Generate Certificate request file:
keytool -certreq -alias SERVER -file <diagnostics_server_install_dir>/etc/diag.csr -keystore
<diagnostics_server_install_dir>/etc/keystore -storepass <password>

3. Request that the CA authority generate the certificate from the diag.csr file and send it back in.cer format. 

  1. Import the cert.cer certificate file from the CA authority into the keystore on the server:
keytool -import -trustcacerts -alias SERVER -file cert.cer –keystore <diagnostics_server_install_dir>/etc/keystore
  1. For the private key (in the keystore) and public key (in the certificate) to match, the certificate file must be imported in the correct keystore file from which the certificate request file was created; otherwise, SSL will not function. 
  2. To use the new keystore register, restart the Diagnostics server. 
  3. If your CA authority created the certificate with a different private key, you’ll need to generate the keystore with the matching key using different instructions, as shown next.

Generating keystore file from existing certificate and private key files

  1. To make this function, ask your CA to give you a certificate and private key file in any format supported by openssl binaries, such as private.key and cert.cer in PEM format. 
  2. Once you have the files, use openssl to convert them to pkcs12 format (you can request this format from the CA authority to avoid having to use openssl): 
  • openssl pkcs12 -export -in cert.cer -inkey private.key -certfile cert.cer -name “SERVER” -out keystore.p12
  1. When prompted, type the desired password for the new.p12 file (which may be the same as the password for the keystore file). 
  2. If you don’t have access to openssl binaries, you can use the following online tools
  3. After you’ve built the.p12 file, use keytool to build a new keystore with the following key:
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype pkcs12 -destkeystore <diagnostics_server_install_dir>/etc/keystore
  1. When prompted by keytool, type the desired password for the new keystore register. 
  2. Following the instructions to obfuscate passwords and set up SSL options in the keystore
  3. <diagnostics_server_install_dir>/etc/security.properties file as shown in Install & Configuration Guide:
# passwords for the keyStore and key
keyStorePassword=<password used by keytool when keystore was created>
keyPassword=<private key password used in openssl command> 
  1. Restart Diagnostics server to pick up the new keystore file.

4. Error SEVERE archive: Unable to create archive after upgrade Diagnostic 9.23 Server

How to upgrade Diagnostics Server? 

You can use the update wizard to upgrade Diagnostics Server components, or you can use the silent upgrade protocol to upgrade the components and deploy the silent upgrade to several computers.

When you run the installation wizard on a device that already has a previous version of the product, the wizard switches to update mode for the installed component or components and allows you to install components that are not on the same machine.

If you get the SEVERE archive error: Unable to build archive following upgrade Server Diagnostics 9.23

  • Diagnostic Server was down and hit with the following error after upgrading the binary from 9.23 to 9.24: 
  • SEVERE archive : Unable to create archive com.sleepycat.je.EnvironmentFailureException (JE 6.0.11) JE 4.1 BIN
  • Eltas were found in the recovery interval. Before upgrading to JE 5.0, the following utility must be run using JE 4.1 (4.1.20 or later): DbPreUpgrade_4_1 . See the change log. UNEXPECTED_STATE: Unexpected internal state, may have side effects.

The error occurs because the symbol table is not up to date with the Diagnostic Server update.

The following steps can be implemented  to resolve the issue: 

  1. Please read the updated document and execute the following command:
<diagnostics_server_install_dir>\bin\sleepycat_upgrade.cmd
  1. This will bring the symbol table up to date with the Diagnostic Server and resolve the issue.

5. Error: “…initializing com[email protected]8b0e7414 java.lang.IncompatibleClassChangeError” reported by the Diagnostics Java Agent probing a WebSphere Application Server

What is WebSphere Application Server? 

WebSphere Application Server (WAS) is a web application server that runs on IBM’s WebSphere platform. It is, more precisely, a software platform and middleware for hosting Java-based web applications.

It is the centerpiece of IBM’s WebSphere software suite. It was initially founded by Donald F. Ferguson, who later became CTO of Software for Dell. In 1998, the first version was released. 

This issue may be caused by a problem with the IBM J9 VM JRE, which would result in certain metrics for affected probes not being displayed. To fix this issue, a number of alternative measures are suggested.

The following error may be identified when the HP Diagnostics Java Agent is configured to probe a Java application running on the IBM J9 VM JRE:

2015-09-22 13:54:24,558 SEVERE capture.metrics [Metrics Collection] Error initializing com[email protected]8b0e7414

java.lang.IncompatibleClassChangeError
  • When using the Java Agent 9.23 IP2 in a WebSphere Application Server running on the IBM J9 VM JRE 1.7.0 64bit on a Linux 2.6 64 bit platform, for example, this error was observed. It’s been used on other networks as well. 
  • Certain metrics, such as JDBC-related metrics, may not be available for the affected probe when this issue arises. The following are some examples of available metrics:

This error may be caused by a problem with the IBM J9 VM JRE (specific FixPacks).

The following steps can be implemented  to resolve the issue: 

  • This issue may be resolved by:
  1. Upgrading to HP Diagnostics 9.24 (or later) or,
  2. Upgrading the IBM J9 VM JRE 1.7 to the latest available fix pack (equivalent to or later than IBM JRE 1.7 SR6 FixPack 1).
  • If none of the above upgrade options is feasible (or effective), the following modification to a probe JAR file is required:
  1. Take a backup copy of the JAR file “probe-jmx-was6.jar” delivered with the Java Agent in the folder/directory “<Java_Agent_Install>\JavaAgent\DiagnosticsAgent\lib” (e.g. “C:\MercuryDiagnostics\JavaAgent\DiagnosticsAgent\lib”),
  2. Unzip the JAR file into a temporary location and locate the folder/directory named “com\mercury\diagnostics\capture\metrics\jmx”,
  3. Remove all classes from this folder/directory except:
  1. WebSphere6JMXCollector.class
  2. WebSphere6MetricGroup.class
  • Zip up the remaining files to recreate the JAR file “probe-jmx-was6.jar”,
  • Replace the “probe-jmx-was6.jar” JAR file folder/directory “<Java_Agent_Install>\JavaAgent\DiagnosticsAgent\lib” with the modified file (a sample JAR file is probe-jmx-was6.jar),
  • Restart the Java application.

6. HPE Diagnostics User Interface freezes or hangs

After launching the HPE Diagnostics User Interface (UI) by selecting “Open Diagnostics” (or “Open in This Window”) and entering a username and password, performing an action such as displaying data for all Java Probes for long periods of time (days or weeks) can cause the UI to freeze and prevent further actions. This problem can arise in a larger Diagnostics environment with a large number of Mediators and probes. 

Once the Diagnostics UI has frozen, there is no way to restore it, and the browser tab that is showing it must be locked. The UI can then be opened in a new browser tab (or new browser instance), but the freezing behavior is likely to return.

  • One potential cause is that the Java applet that runs the Diagnostics UI has insufficient memory, causing the applet process to run out of memory and terminate while the UI is performing the user action. The UI remains visible in the browser after this phase ends, but it is no longer available. 
  • Open the Windows Task Manager and look for the process called “jp2launcher.exe” to confirm that the Java applet process is ending. While the Diagnostics UI is being initialized (after the username and password have been validated), this loop will appear: 
  • When the Diagnostics UI freezes, and the “jp2launcher.exe” is no longer available, the process may have run out of memory.

The following steps can be implemented  to resolve the issue: 

  • Increasing the heap (memory) available to the Java applet is a workaround for this problem. This can be accomplished by making the following changes to the Java runtime parameters: 
  1. To find the “Java (32-bit)” Control Panel entry, open the Windows Control Panel and check for “Java.” To open the Java Control Panel, click on the entry: 

Note: If there is no “Java (32-bit)” Control Panel entry this may be due to there being multiple Java versions installed on the client platform. 

Locate the Java 32-bit installed in a folder under “C:\Program Files (x86)” (typically “C:\Program Files (x86)\Java\jre7\bin”) and execute the file “javacpl.exe” using right-click “Run as administrator”.

  1. To open the “Java Runtime Environment Settings” dialog, click the “Java” tab and then the “View” button. 
  2. The default heap size is usually 256MB, but this can vary depending on the device. Before the process terminates, the actual memory size of the process can be seen in the Windows Task Manager. Increase the heap size and test to see if the UI freeze persists; if so, increase the heap size as required. The following example configures a 700MB heap, but up to 1GB (“-Xmx1G”) can be used (memory is restricted in this 32-bit process):

7. Error: “HttpWebServer port 35000 appears to be in use” when starting the Diagnostics .NET Agent

  1. On the port range 35000 to 35100,.NET Agents fail to start and are unable to communicate. 
  2. The agent (probe) fails to start as planned after a successful installation of a Diagnostics.NET Agent, and the following error message is written in the agent log file: 
  • HttpWebServer The port 35000 of the HttpWebServer appears to be in operation. 
  1. According to the agent configuration in the file at: 
  • This message is repeated for all ports in the range 35000 to 35100.
  • C:\MercuryDiagnostics\.NET Probe\etc\probe_config.xml
  1. As a result, no data for this probe is recorded in the Diagnostics UI, and it does not appear in the Diagnostics Health View. The following alert message appears in the Diagnostics UI: 
  • The probe is not currently active or is incorrectly configured.
  1. Navigating to:

http://<probe_host>:35000 results on no response.

  1. The netstat order, on the other hand, indicates that ports 35000 to 35100 are open for use.

The following steps can be implemented  to resolve the issue: 

  1. For the ports in the.NET Agent Profiler’s port set, a “urlacl” must be listed. To fix the issue, follow these steps:
  • As an administrator, log in to the Windows platform where the.NET Agent is running and/or use “Run as Administrator” to open a Windows PowerShell script. 
  • Run the following command (specific to the Windows platform):

cscript “C:\MercuryDiagnostics\.NET Probe\bin\AddHttpURLPrefixes.js” (for Windows Server 2008)

cscript “C:\MercuryDiagnostics\.NET Probe\bin\AddHttpURLPrefixesW2k3.js” (for Windows Server 2003)

  • Using the command “iisreset,” restart the probed program. 
  1. If the Windows platform where the.NET Agent is mounted has several network cards, edit the configuration file at:
  • <.NET Agent Install>\etc\probe_config.xml (default is “C:\MercuryDiagnostics\.NET Probe\etc\probe_config.xml”)
  • and ensure that the a value is specified for the “host_name” in the following setting:
  • <diagnosticsserver url=”http://<mediator_server>:2006/commander” registered_hostname=”<host_name>” />
  1. Also edit the configuration file at: 
  • <.NET Agent Install>\etc\metrics.config (default is “C:\MercuryDiagnostics\.NET Probe\etc\metrics.config”)
  • and ensure that the a value is specified for the “host_name” in the following setting:
  • metrics.agent.registered_hostname = <host_name>

If an alternative port range has been specified for the .Net Agent Profiler then follow these steps:

  • Make a backup copy of the JavaScript file for the Windows platform in use, 
  • Edit the JavaScript file for the Windows platform in use and change the lines below (the updated port range in this example is 38100 to 38200):

                    var minPort = 38100;

                   var maxPort = 38200;

  • Save the JavaScript  file and run it as described above.

8. Configuring the Diagnostics .NET Agent and Probe Aggregator to use non-default ports


Why is Probe Aggregator important for .NET Agent? 

When dealing with a large number of probes from.NET agents, it is suggested to use Probe Aggregator. Because of its functions like reducing network communication between the probe and the server, as well as the server’s load and also the Probe Aggregator improves scalability.

The use of the Probe Aggregator was used to evaluate and enforce load testing and power management. There should be no more than 50 probes connected when.Net agents are connected directly to a Diagnostics Mediator without using Probe Aggregation.

  • The ports used by the Diagnostics.NET Agent and Probe Aggregator components on Windows servers when they are built using the default (out-of-the-box) configuration are:
  1. The .NET Agent connects to Probe Aggregator on port 45000 (using HTTP),
  2. The .NET Agent connects to Probe Aggregator on port 2626 (using TCP),
  3. The .NET Agent Profiler listens (for HTTP) on the lowest available port in the range 35000 to 35100,
  4. The Probe Aggregator listens (for HTTP) on port 45000,
  5. The Probe Aggregator listens (for TCP) on port 2626,
  6. The Probe Aggregator, which runs in a JVM, is also probed with a Java Agent. The Java Agent Profiler listens (for HTTP) on port 35000.

The following steps can be implemented  to resolve the issue: 

  • To configure the.NET Agent and Probe Aggregator to use alternate ports, edit the.NET Agent configuration files as follows:
  1. probe_config.xml (“C:\MercuryDiagnostics\.NET Probe\etc\probe_config.xml”)
  2. metrics.config (C:\MercuryDiagnostics\.NET Probe\etc\metrics.config)
  • which are located in the folder “<.NET Agent Install>\etc” (default is “C:\MercuryDiagnostics\.NET Probe\etc”) and the following Probe Aggregator configuration files:
  1. probeaggregator.properties (“C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\etc\probeaggregator.properties”)
  2. webserver.properties (C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\etc\webserver.properties”)
  • which are located in the folder “<.NET Agent Install>\ProbeAggregator\etc” (default is “C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\etc”) and the following Probe Aggregator probe (Java Agent) configuration file:
  1. webserver.properties (C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\probe\etc\webserver.properties”)

2. which is located in the folder “<.NET Agent Install>\ProbeAggregator\probe\etc” (default is “C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\probe\etc”)

  • Before making any changes, make a backup copy of these configuration files. 
  • The following alternative ports and port ranges are configured in the example below:
  1. The Probe Aggregator will listen (for HTTP) on port 38000 (default is 45000),
  2. The Probe Aggregator will listen (for TCP) on port 38626 (default is 2626),
  3. The .NET Agent Profiler listens (for HTTP) on the lowest available port in the range 38100 to 38200 (default is 35000 to 35100),
  4. The Probe Aggregator probe (Java Agent) listens (for HTTP) on port 38100 (default is 35000).
  • Edit the file to reconfigure the.NET Agent to use these alternate ports. “probe_config.xml” and modify the following settings:

<diagnosticsserver url=”http://127.0.0.1:38000/commander” />

<mediator ssl=”false” host=”127.0.0.1″ port=”38626″ metrichost=”127.0.0.1″ metricport=”38000″ />

<webserver start=”38100″ end=”38200″ />

  • Edit the file “metrics.config” and modify the following settings:

metrics.server.uri = http://127.0.0.1:38000/metricdata/

  • Edit the file “probeaggregator.properties” and modify the following settings:

mediator.port.number=38626

event.port=38626

  • Edit the file “webserver.properties” in the folder “<.NET Agent Install>\ProbeAggregator\etc” and modify the following setting:

jetty.port = 38000

  • Edit the file “webserver.properties” in the folder “<.NET Agent Install>\ProbeAggregator\probe\etc” and modify the following setting:

jetty.port = 38100

  • After you’ve edited and saved the above configuration files, follow these steps:
  • In Windows “Services” restart the “HPE Diagnostics Probe Aggregator” service,
  • In Windows “Services” restart the “HPE Diagnostics Metrics Agent” service,
  • Open a Windows PowerShell (using “Run as Administrator”) command window and enter the command “iisreset”.:
  • To confirm that these port changes are active, review the log files for the .NET Agent at:

<.NET Agent Install>\log (default is “C:\MercuryDiagnostics\.NET Probe\log”)

and the Probe Aggregator at:

<.NET Agent Install>\ProbeAggregator\log (default is “C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\log”)

and the Probe Aggregator probe (Java Agent) at:

<.NET Agent Install>\ProbeAggregator\probe\log (default is “C:\MercuryDiagnostics\.NET Probe\ProbeAggregator\probe\log”)

9. .Net agent causing memory and CPU overhead during load test.

What is CPU Overhead? 

The amount of work that a computer’s central processing unit can do and the percentage of that power that is used by individual computing tasks are both measured by Central Processing Unit (CPU) overhead. Supporting the functions of the computer’s operating system is one of its activities.

It also includes Internal components, such as hard drives and graphics processing units with providing connective features too, such as networking support, and lastly external components, such as the computer’s display. Processor power is provided for applications by the need for CPU overhead. 

  • When a load test is running, .net agent monitoring will trigger some overhead on the application. 
  • The.net agent comes with the recommended settings for controlling a production application right out of the box. However, some of these control metrics may add to the complexity of load test scenarios.

The following steps can be implemented  to resolve the issue: 

  • To reduce the over head trun off the following settings:
  1. Stacktracesampling :
 <stacktracesampling enabled="false" maxactivethreads="100" suspendthread="true" outboundcalls="false" rate="100" tardymethodlatency="150"/>
  1. RUM probe:
 <rum enabled="false" ...
  1. Threads Monitoring: 
<process name="ASP.NET" monitorthreads="false" enablealldomains="false">

10. After upgrading, history data lost, custom view and application lost

What is the use of Custom View in Diagnostics 9.40?

You can use a custom view to save unique display and print settings (such as hidden rows and columns, cell selections, filter settings, and window settings) for a worksheet so that you can easily add these settings to that worksheet when needed. A custom view can also provide a special print area.

You can create several custom views per worksheet, but each custom view can only be applied to the worksheet that was active at the time the custom view was developed. You can uninstall a custom view if you no longer need it.

If the History data, custom views, and the application were all lost after upgrading to 9.40.

The following steps can be implemented to resolve the issue: 

  • The problem arose because the storage and archive folders were not copied back to the new folder after the upgrade. 
  • We recommend that you take the following measures in your environment to resolve the problem: 
  • Until updating, make a complete backup. 
  • Stop using the diagnostic facility. 
  • Make a backup of the Diagnostic Server 9.40 root folder ( in case we need to recovery)
  • Replace the “Server\archive” folder and “Server\storage\permissions\server-xxx\Default Client\_enterprise\_server.properties” file from backup of version 9.30
  • Start service