Micro Focus Application Performance Management – Tips And Tricks

Table of Contents

Application Performance Management – Tips And Tricks – May 2021

1. Not able to create new views in APM

Users are not able to create new views in APM. There’s no sign of any error but APM is not saving the contents of the view. Actually the error is this: URM transaction validation error: The resource key: {{…},Topology_TQL} is referenced from resource: {{integration_tqls_bundle},Topology_RESOURCE_BUNDLE} but isn’t exist in the URM!!

Solution

Here are the steps to resolve this error:

Step 1: Remove Corrupted TQL.

Step 2: Open JMX console http://localhost:21212/ – > UCMDB:service=URM Services -> listResourceTypes -> listResourceTypes <invoke> Topology_RESOURCE_BUNDLE.

Under Resources of type: Topology_RESOURCE_BUNDLE

Step 3: Click the below Resource IDs and delete the corrupted entry from the XML.

SLM

Service_Health

UCMDB

template_based_view

Here are the steps to remove from these bundles

Error: URM transaction validation error: The resource key: {{…},Topology_TQL} is referenced from resource: {{integration_tqls_bundle},Topology_RESOURCE_BUNDLE} but isn’t exist in the URM!

To remove this, you can go through the following steps:

Step 1: On DPS go to: https://localhost:8443/jmxconsole/HtmlAdaptor?action=inspectMBean&name=UCMDB:service=Grouping Services

SLM

Service_Health

UCMDB

template_based_view

Step 2: Go to fix bundle in JPM and provide the bundle Name of “integration_tqls_bundle”

Step 3: Go to command removes any non-existing resources from the bundle so should fix this issue.

After this we no longer saw the name in the UI and the issue was corrected.

2. APM cannot get data from RUM in reports

We are having an issue regarding data from Real User Monitor. Currently we are using APM 9.51.151, Build 164 on Windows 2012 R2. Every time we try to get a report, we are confronted with the error “There was a problem retrieving data from the Real User Monitor engines. “

Logging into the RUM Engine GUI and updating configuration without issue and checking the RTSM-Integration user password without issue can help to get into APM. We are using the RUM Engine as Version: 9.50.10 Build: 162 on Windows 2012 R2. I’m uploading the APM server Log Grabber file.

Solution

You can go through the following steps to solve this issue:

Step 1: Stop RUM service and go to the following on the Engine machine:

productDir\EJBContainer\server\mercury\conf

Step 2: Edit the “users.xml” file and replace the “encrypted Password” field with “password” and type your new password.

Step 3: After that save the changes and restart RUM Engine.

Step 4: Open Engine JMX console – http://:8180/jmx-console.

Step 5: Type username/password you just changed and access and remember it.

Step 6: Open BSM UI – BSM – Admin – End User Management – Setting Tab – Real User monitor Setting Tab – click on the relevant engine.

Step 7: Mark “Override default connection settings” and type username/password from step 2.

Step 8: Save and try to open reports again.

Step 9: After executing the steps below we need to restart APM first and then RUM engine.

Step 10: Resync the configuration :Tools > Monitoring Configuration Information > Sync All.

3. APM 9.40 / 9.5x – how to enable DEBUG logging for HornetQ

Users want to enable DEBUG logging for HornetQ APM 9.40 / 9.5x? Before HornetQ DEBBUG logging in BSM 9.26,we used to modify the file <HPBSM>\HornetQ\config/<typical/distributed>\logging properties and change the log level to the level required.

But now this is not the way to operate it in APM. Also the folders are changed with APM 9.40 until now the file can be found at <HPBSM>\HornetQ\config\logging.properties (no more <typical/distributed>). However enabling debug logging in logging.properties followed by a HornetQ restart didn’t do anything good. No file HornetQ. Log is created (and written to) in <HPBSM>\HornetQ\logs.

 Solution

There is another way to enable DEBUG logging. You just need to follow the commands written below.

Create a backup copy of <HPBSM>\HornetQ\bin\run. Bat in run. Bat change the line

old

set JVM_ARGS=%CLUSTER_PROPS% -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods %TOPAZ_PARAMS% -Dhornetq.config.dir=%CONFIG_DIR% -Ddata.dir=%HORNETQ_DATA% -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djava.util.logging.config.file=%CONFIG_DIR%\logging.properties -Djava.library.path=.

new

set JVM_ARGS=%CLUSTER_PROPS% -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods %TOPAZ_PARAMS% -Dhornetq.config.dir=%CONFIG_DIR% -Ddata.dir=%HORNETQ_DATA% -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djava.util.logging.config.file=%CONFIG_DIR%\logging.properties -Dlog4j.configuration=file:%CONFIG_DIR%\log4j.properties -Djava.library.path=.

and save the file.

This tells HornetQ to use the log4j logging facility.

– create a backup copy of <HPBSM>\HornetQ\config\logging.properties

from logging.properties delete all lines and add the two lines (all between the ***)

***

handlers=org.hornetq.integration.logging.Log4jLoggerHandler

.level=ALL

***

create the file <HPBSM>\HornetQ\config\log4j.properties and add (all between the ***)

***

log.file.path=log/${log.folder.path.output}

loglevel=DEBUG

def.file.max.size=2000KB

def.files.backup.count=10

msg.layout=%d [%t] (%F:%L) %-5p – %m%n

log4j.appender.STDOUT=org.apache.log4j.RollingFileAppender

log4j.appender.STDOUT.File=${topaz.home}/${log.file.path}/hornetq.log

log4j.appender.STDOUT.MaxFileSize=${def.file.max.size}

log4j.appender.STDOUT.MaxBackupIndex=${def.files.backup.count}

log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout

log4j.appender.STDOUT.layout.ConversionPattern=%d [%t] %-5p %c{1}.%M(%L) – %m  %n

log4j.rootCategory=DEBUG, STDOUT

log4j.category.org.hornetq=DEBUG

log4j.category.org.hornetq.core.management=DEBUG

***

save the file

restart HornetQ or (recommended) APM

After HornetQ starts you’ll find the file <HPBSM>\log\hornetq.log with (for example)

..

2020-10-20 14:02:15,763 [main] DEBUG BasicProperties.debug(14) – loading file:C:\HPBSM\conf\legacy.properties  

..

2020-10-20 14:02:19,426 [Thread-2] INFO  JmsOperations.<init>(57) – createConnection [user=hpbsm;pwd=1Z1Z8753EED2646B66B5FCB407744C468B59ZA3EB341CF503D4869FA2AC866B38CE29EA7E24B4ED33B1DB02E3C474597F247D;connectId=SettingsManager_AppClassLoader_1984697014]  

2020-10-20 14:02:19,597 [Thread-2] DEBUG InternalLoggerFactory.debug(71) – Using SLF4J as the default logging framework  

2020-10-20 14:02:19,603 [Thread-2] DEBUG PlatformDependent.debug(71) – Platform: Windows  

2020-10-20 14:02:19,617 [Thread-2] DEBUG PlatformDependent.debug(76) – Java version: 8  

2020-10-20 14:02:19,618 [Thread-2] DEBUG PlatformDependent.debug(76) – -Dio.netty.noUnsafe: false  

..

2020-10-20 14:02:20,008 [Thread-2] DEBUG NioEventLoop.debug(76) – -Dio.netty.selectorAutoRebuildThreshold: 512  

2020-10-20 14:02:20,116 [Thread-2] DEBUG ThreadLocalRandom.debug(71) – -Dio.netty.initialSeedUniquifier: 0x962d62767179790d  

2020-10-20 14:02:20,696 [main] DEBUG XmlBeanDefinitionReader.doLoadBeanDefinitions(395) – Loaded 1 bean definitions from file [C:\HPBSM\conf\spring\analytics-spring.xml]  

..

2020-10-20 14:02:21,800 [main] DEBUG XmlBeanDefinitionReader.doLoadBeanDefinitions(395) – Loaded 1 bean definitions from file [C:\HPBSM\conf\ServiceManager\common\spring\jvm-statistics-spring.xml]  

2020-10-20 14:02:21,801 [main] DEBUG GenericApplicationContext.prepareRefresh(594) – Refreshing [email protected]0a903b  

2020-10-20 14:02:22,210 [main] DEBUG DefaultListableBeanFactory.getSingleton(213) – Creating shared instance of singleton bean ‘org.springframework.context.annotation.internalConfigurationAnnotationProcessor’  

..

2020-10-20 14:03:02,089 [main] DEBUG NetUtil.debug(76) – Loopback address: /0:0:0:0:0:0:0:1  

2020-10-20 14:03:04,527 [Thread-2] INFO  JmsOperations.<init>(57) – createConnection [user=hpbsm;pwd=1Z1Z8753EED2646B66B5FCB407744C468B59ZA3EB341CF503D4869FA2AC866B38CE29EA7E24B4ED33B1DB02E3C474597F247D;connectId=SettingsManager_AppClassLoader_1984697014]  

2020-10-20 14:03:04,786 [Thread-2] INFO  JmsOperations.<init>(54) – Tue Oct 20 14:03:04 CEST 2020   —- ConnectionManager start —-  

..

2020-10-20 14:53:04,822 [ConnectionManagerBackgroundTrd] INFO  BackgroundThread.run(182) – Background thread started. connections will be checked every 300000 ms  

2020-10-20 14:53:04,822 [ConnectionManagerBackgroundTrd] DEBUG BackgroundThread.reconnect(82) – start reconnecting connections  

2020-10-20 14:53:04,822 [ConnectionManagerBackgroundTrd] INFO  BackgroundThread.reconnect(111) – Tracker reconnect summary: 1 connection(s) were checked, 1 were already connected, 0 were reconnected, 0 failed to reconnect.  

2020-10-20 14:53:04,822 [ConnectionManagerBackgroundTrd] INFO  BackgroundThread.run(186) – Background thread finished  

..

4. BPM 9.51 and later on Linux – PostInstall step deletes /opt/HP after completing the installation

While installing BPM 9.51 and Linux, the PostInstall step deletes /opt/HP.

Solution

After the completion of all steps in installing BPM 9.51 and Linux, the PostInstall step deletes /opt/HP, it will execute the PostInstall step, which – among others – executes postinstall_bpm.sh.

The script postinstall_bpm.sh contains

..

HPBPM_HP=/opt/HP

FOLDERS_LIST=”$HPBPM_HP”

for FOLDER in ${FOLDERS_LIST}

do

if [ -d $FOLDER ]; then

echo “Deleting $FOLDER”

rm -rf $FOLDER

fi

done

..

which means that it deletes /opt/HP with all files underneath, which might still have products installed into it, namely SiteScope or BSM. Users should install BPM 9.51 and newer on Linux ONLY if there is nothing else installed under “/opt/HP”, or as my first application.

The issue has been reported to R&D via CR OCTCR19G1143237, the issue will be fixed in a future BPM release. A hotfix is available from support for BPM 9.53 (can be used with BPM 9.51 and 9.52 as well) as hotfix 15035, but it has to be applied during the installation of BPM, and cannot fix the issue afterward.

5. APM 9.51 – message Server is NOT READY

Users need to know that the message Server is not ready in APM 9.51. If you have noticed, it’s an indication that JBoss is not starting. The status page returns the “Server is not ready” message. Processes are not loading. The wrapper.log file from the <HPBSM>\log\supervisor folder contains this error: “Error: Password file read access must be restricted: c:\HPBSM/JRE64/lib/management/jmxremote.password”

The root cause of this problem is that the Windows Management Instrumentation command-line (WMIC) utility works incorrectly with different regional settings. As a result, it impacts the assignment mechanism of ownership and permissions (files jmxremote.access and jmxremote.password). 

Solution

To fix the issue, follow the below written steps:

Step 1. Disable APM.

Step 2. Navigate to <APM root directory>\JRE\lib\management.

Step 3. Right-click jmxremote.password and select Properties.

Step 4. Click the Security tab.

Step 5. Click Edit.

Step 6. Click Add and add the Administrators group.

Step 7. Allow Read and Write Permissions for the Administrators group.

Step 8. Repeat steps 2 – 7 for the jmxremote.access file.

Step 9. Navigate to <HPE APM root directory>\JRE64\lib\management.

Step 10. Repeat steps 3 – 8.

Step 11. Enable APM.

6. APM 9.51 HornetQ Bus Web Console doesn’t work

Users are facing the same issue on both Linux and Windows. The console fails with java.lang.NoClassDefFoundError: org/apache/commons/Lang/text/StrBuilder, also issue with special characters in the Oracle database password and the usage of Oracle Advanced Security (OAS). 

APM 9.51 Linux, 3 x GW, 2 x DPS, running on RHES release 7.8 (Maipo)

LB Apache in front of the GWs

The HornetQ Bus Web Console doesn’t work. One can start the server fine via <HPBSM>/tools/busWebConsole/bin/Start’s but when then accessing the console via

http://localhost:9000

the browser returns

HTTP Error: 500

Problem accessing /login. Reason:

class.java.lang.NoClassDefFoundError

At the same time one can see at the console where the server was started

java.lang.NoClassDefFoundError: org/apache/commons/lang/text/StrBuilder

at org.apache.velocity.runtime.parser.ParserTokenManager.<init>(ParserTokenManager.java:5161)

at org.apache.velocity.runtime.parser.Parser.<init>(Parser.java:3497)

..

Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.text.StrBuilder

at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

This happens on both APM 9.51 on Linux and on Windows, it works fine with APM 9.50 on Windows and LLinu.x. There is a known issue reported under OCTIM19G786572 UPGRADE HornetQ console not working in 9.51

Solution

This can be resolved the following way:

there is one file in <HPBSM>\tools\busWebConsole\lib directory that is causing HornetQ console to not work in 9.51, commons-lang-2.1.jar

in 9.50 we are using commons-lang-2.4.jar

After deleting 

commons-lang-2.1.jar

and replacing it with 

commons-lang-2.4.jar (taken from an APM 9.40 or APM 9.50 installation, it also can be downloaded here)

The HornetQ console worked as expected after this (APM on Windows with MS SQL as database).

In the case of this customer / environment (APM on Linux with Oracle as database)

one now can open the GUI, however – one cannot login.

The customer tried with the APM admin user / password, even tried admin / admin, nothing worked.

The tool is not documented anywhere (yet), the only information found is from a BSM 9.26 training slide which shows

..

use BSM admin credentials for accessing the HornetQ web console

..

As per R&D the HornetQ Web Console login takes the password of the APM admin credentials, so this should work.

Enabled DEBUG logging by updating the file

<HPBSM>\conf\core\Tools\log4j\bus-tools\console.properties

change

loglevel=<whatever>

to

loglevel=DEBUG

saved the file

started the hosting process on the DPS

try to login to http://<DPS>:9000

<HPBSM>\log\bus-tools\console.all.log shows

2020-06-22 14:37:37,032 DEBUG – Error java.sql.SQLException: Oracle Error ORA-12650

at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)

at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)

..

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:413)

..

at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)

..

at com.hpe.apm.bus.DatabaseManager.getHornetQUserName(DatabaseManager.java:71)

at com.hpe.apm.bus.HornetQCredentials.getUser(HornetQCredentials.java:10)

..

ORA-12650: No Common Encryption Or Data Integrity Algorithm When Using SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=sha256

Well, this is a known issue with this customer:

using APM 9.40 the customer database team increased the security requirements regarding data transport for Oracle and thus only accept this encryption:

ddoracle.EncryptionLevel=required

ddoracle.EncryptionTypes=AES256

ddoracle.DataIntegrityTypes=SHA256

ddoracle.DataIntegrityLevel=required

In APM 9.40 it was not supported by default, because the JDBC driver used was too old. 

Therefore the customer got a driver update via

QCIM1I134265 – APM 9.40 – does APM support SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER/ .._CLIENT = (SHA256) / SQLNET.ALLOWED_LOGON_VERSION_SERVER=12a

In APM 9.51 this is supported out of the box, but it seems that the HornetQ Web Console code was not updated to support this transport encryption,

it comes with its own libraries under <BSM>/tools/busWebConsole/lib

It took quite some iterations, but finally R&D was able to get it working, as the customer uses

special characters in the database password, which is not supported by all odbc libraries

customer uses Oracle Advanced Security (OAS) with the following settings in <HPBSM>/conf/jdbc.drivers.properties:

ddoracle.EncryptionLevel=required

ddoracle.EncryptionTypes=AES256

ddoracle.DataIntegrityTypes=SHA256

ddoracle.DataIntegrityLevel=required

which is not supported by all odbc libraries

In the end R&D decided to change the HornetQ Web Console code to use the same library main APM code uses, mcoracle.jar. R&D created a customer-specific hotfix, available via  rdapps.swinfra.net > hotfix > # > hotfix > 14839. If one needs this hotfix, please open a case with support and request the hotfix via OCTCR19G1058228 APM 9.51 – HornetQ Bus Web Console doesn’t work, same issue on Windows and Linux

7. BPM 9.53 on Windows 10 – undetected installation issue causes UFT scripts to fail

The UFT script failed because of an undetected installation issue on BPM 9.53 on Windows 10. This document describes an issue with the installation of the BPM 9.53 on Windows 10. BPM has been installed for UFT monitoring purposes and the integration with UFT works fine.

UFT can record and replay scripts, BPM can play the scripts as well, but some issues have been detected when BPM runs on Windows 10. The issues have only been observed on Windows 10, all work fine and as expected when BPM is running on Windows 2016.

Solution

The issue is found. Running a script manually from BPM Admin shows the script as running (one can see the application running, following the script), but afterwards BPM complains “no data received”, the transactions show question marks instead of success or failure.

Running script from schedule from APM shows the same script running (as above), but BPM complains: “failed to start”. This however is incorrect, the script was started and executed successfully. Issue was solved with knowledge document KM03034995 for BPM 9.40

(softwaresupport.softwaregrp.com > doc KM03034995) where a similar issue happened as well occasionally.

Files were copied from the BPM folder to the LG folder:

stop BPM

create backup copies of

<LGSA_HOME>\dat\mdrv.dat

<LGSA_HOME>\bin\bpm.dll

copy the following files 

<BPM_HOME>\dat\mdrv.dat -> <LGSA_HOME>\dat\mdrv.dat

<BPM_HOME>\bin\bpm.dll  -> <LGSA_HOME>\bin\bpm.dll

start BPM

Now the scripts are executed as expected and report data

8. HTTP guide and additional information

Users need to know about information on how to set gateway URLs for the users to be HTTPs, also need to know the impact and all the changes needed. Currently we have APM 12.5 with a 1 Gateway,1 DPS, 1 SiteScope, and several BPMs.

Solution

The communication process for APM is known as hardening and you may find the details on our following links.

• Introduction to Hardening docs.microfocus.com > itom > Application Performance Management:9.51 > Hardening

• Hardening Workflow docs.microfocus.com > itom > Application Performance Management:9.51 > Hardening Work flow

• General Security Recommendations docs.microfocus.com > itom > Application Performance Management:9.51 > General Security

• Using TLS in APM docs.microfocus.com > itom > Application Performance Management:9.51 > TLS

• Configure Secure Access to APM Front End docs.microfocus.com > itom > Application Performance Management:9.51 > Secureaccess

• Using Basic Authentication in APM docs.microfocus.com > itom > Application Performance Management:9.51 > BasicAuth

9. Record script in VuGEN 12.60 but fails on replay in BPM 9.51

Failure in replaying the record script on BPM. The script replays in VuGen and also in BPM when launched outside of BPM with no problem. While monitoring a site that uses 2FA, we found that a one time code is provided which allows us to by-pass 2FA. The failure in replaying is because the one-time code popup begins again causing the script to fail.

Solution

The solution to this is provided in following steps:

Step 1: Stop the BPM running as a process and start it as a service.

Step 2: Create a new instance for the BPM attached to APM and configure it to “Run as a Specific User”.

Step 3: Provide the user’s name and password for BPM server with Administrator privileges.

Step 4: Then attach one of the Availity scripts to it and monitor it.

10. RUM Engine status is in Critical state (Server Health Check up) 10.2.247.47

Error is found in RUM while showing the retrieval status. While connecting to the Application Performance Management server, the Configuration type ENGINE_SETTINGS shows this error.

Solution

Here are the errors shown by the configmanager.log from the rum engine machine.

(BSMConfigurationRetriever.java:125) ERROR – error occurred while retrieving configuration for data type [ENGINE_SETTINGS] com.mercury.rum.engine.configmanager.configretriever.DataTypeRetrieverException: Register Engine Failed

Caused by: com.mercury.rum.engine.configmanager.tmc.TMCHttpRequestInvokerException: TMC Request [registerHost] failed on IOEXception: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateExpiredException: Not After: Thu Dec 05 13:03:02 IST 2019

Since we saw that the APM is configured in SSL on port 443, it seems that the APM certificate has expired and you need to import the issued APM certificate in the RUM Engine KeyStore \JRE\lib\security\cacerts. Please refer to the below link and find the related section: “Importing the APM/BSM Certificate into the RUM Engine”

community.microfocus.com > t5 > Application Delivery Management > Using KeyStore Explorer to exchange trusted certificates between > ba-p > 224621

Please work with your security team to validate the above as well and contact the support team for assistance.