Micro Focus Server Automation – Tips and Tricks

Table of Contents

Micro Focus Server Automation – Tips and Tricks – Mar 2021

1. Procedure to reinstall agent on a satellite after recent tool corruption due to cert issues

There have been observations made on multiple problems that occurred on the core and satellite after the launch of the recent tool.  Even after repeated attempts to solve the complication on the satellites, they are not fully functional yet.

The agent showed up and the scripts were able to be run but the OGFS didn’t work and one of them had no connectivity at all with the comm test. Attempts on copying place certs from previous SR have not solved the problem completely as these satellites are required to be in a good state or use OGFS for attempting a recent.

These troubles faced can be solved by the steps given below.

1. If there are CORD patches installed, the users are requested to rollback them.

/opsware_installer/patch_opsware.sh –verbose

If it is a patched system, the following message is displayed:

Welcome to the Opsware Installer. It appears that you
have previously completed installation of this patch on
this system.
Press ‘r’ to remove this patch.
Press ‘s’ to show patch contents.
Press ‘q’ to quit.
Selection:

Enter r at the prompt to remove the patch.

2. The uninstall script must be invoked.

/opsware_installer/uninstall_opsware.sh -r
The user can decide where the response file is saved during the installation, but normally it can be found at /usr/tmp directory.

3. Deactivate and delete the server by going to SA web GUI, once the uninstallation is complete.

4. Verify and then force remove any SA RPMs (if satellite id in Linux):

rpm -e –nodeps `rpm -qa
grep ˆOPSW`

5. Check if SA processes are running and stop them if they are.

ps -ef
grep ops[w]
awk '{print $2}'
xargs kill -9

6. Wipe all SA files or directories present.

Do not manually craft any files. Copy and paste this multi-line command.

rm -rf /opt/opsware \
/etc/opt/opsware \
/var/opt/opsware \
/var/log/opsware \
/etc/init.d/opsware* \
/etc/rc?.d/*opsware*

7. Restart the cores.

2. Solving the issue with SLES for SAP that shows unknown OS

Users have reported to be facing trouble with a server that has OS “SLES 15 for SAP” and shows that the OS is unknown in Java clients.

The fix for the issue is,

class SuSEPlatform(linux_platform.LinuxPlatform):
hostname_file = '/etc/HOSTNAME'
examine_cmds = (
(
"%s -q sles-release --queryformat '%%{VERSION}'" % rpm_handler.RpmHandler.cli, 'SLES'),
(
"%s -q suse-release --queryformat '%%{VERSION}'" % rpm_handler.RpmHandler.cli, 'SUSE'),
(
"awk '/VERSION/ { print $1 }' /etc/SuSE-release", 'SLES', ['8.0'], 'SUSE'))

There are 3 ways to solve the problem faced like said above,

1.  rpm -q sles-release –queryformat. Which for SAP I that was already mentioned was missing and it isn’t considered as an option as it will break dependencies for SAP release.

2.  rpm -q suse-release –queryformat will also not be used here.

3.  In SLES /etc/SuSE-release is not present as it was deprecated. 

For the fix, /etc/SuSE-release file must be created with the information mentioned below.

==
'mid': '173750001',
'os_flavor': None,
'os_spversion': None,
'os_version': 'Linux SLES-15-X86_64',
'reboot_required': 0,
'serial_num': 'CZ3950Y0G3',
'server_location': [],

3. Solution for the script errors by the SA agent

It has been reported that an SA agent repeatedly gives error messages.

To solve this problem,

The problem arises mainly because the environment variable “COMSPEC” is not set in the agent process. There are chances of it missing system-wide.

Below given is a small scale reproduction where it was possible to remove the COMSPEC environment manually within a cmd.exe shell.

class SuSEPlatform(linux_platform.LinuxPlatform):
hostname_file = '/etc/HOSTNAME'
examine_cmds = (
(
"%s -q sles-release --queryformat '%%{VERSION}'" % rpm_handler.RpmHandler.cli, 'SLES'),
(
"%s -q suse-release --queryformat '%%{VERSION}'" % rpm_handler.RpmHandler.cli, 'SUSE'),
(
"awk '/VERSION/ { print $1 }' /etc/SuSE-release", 'SLES', ['8.0'], 'SUSE'))C:\Windows\system32>echo %COMSPEC%C:\Windows\system32\cmd.exeC:\Windows\system32>"%PROGRAMFILES%\Opsware\agent\lcpython15\python.exe" -c "from coglib import lcos;o,i,e=lcos.popen3('dir \\');print o.read()"Volume in drive C has no label.Volume Serial Number is B275-61FADirectory of C:\08/22/2013 08:52 AM
PerfLogs01/24/2020 02:25 PM
Program Files08/22/2013 08:39 AM
Program Files (x86)10/11/2016 04:09 PM
Users01/24/2020 04:24 PM
Windows0 File(s) 0 bytes5 Dir(s) 32,823,705,600 bytes freeC:\Windows\system32>set COMSPEC=C:\Windows\system32>"%PROGRAMFILES%\Opsware\agent\lcpython15\python.exe" -c "from coglib import lcos;o,i,e=lcos.popen3('dir \\');print o.read()"Traceback (most recent call last):File "", line 1, inFile ".\lcos.py", line 924, in popen3File ".\lcos.py", line 634, in __init__pywintypes.error: (203, 'CreateProcess', 'The system could not find the environment option that was entered.')

The fix is done by adding the value or changing the value. To get, right-click on ‘my computer’ or ‘This PC’ in the windows explorer. Go to the ‘system properties’ dialog, then the ‘Advanced’ tab, and then click on the (environment variables) button.

4. Fix for Windows 2016 being reported as an unknown OS

Users have complained that even after successfully installing the platform installer for windows 2016 on SA 10.21, windows servers are reporting an error that says “unknown” OS when checking the HPSA UI/ device properties.

The solution for the problem is,

Both versions of the map configuration files were not updated with the new version of the windows.

/etc/opt/opsware/spin/os_version_map.conf
/opt/opsware/spin/etc/os_version_map.conf

On opening, these two files will see the other windows but not the one with 2016, even though the platform installation was successful, it fails to change the two files.

The platform installer ID must be checked in the NGUI, and make sure that the installer has the ID 170402. After that, the OS version needs to be added in both conf files (. *Windows. *Server 2016. *x64. *: 170402), example:

[[email protected] bkt]# cat /etc/opt/opsware/spin/os_version_map.conf
This file contains a mapping from regular expressions to platform_ids.
The regular expressions are matched IN ORDER against the os_version
submitted during hardware registration. First match wins.
.*Windows .*Server 2008.*R2.*.Itanium : 100200
.*Windows .*Server 2008.*R2.* : 170092
.*Windows .*Server 2003.*x64.* : 60100
.*Windows NT 6\.1.* : 170092
.*Windows NT 6\.2.* : 170099
.*Windows .*Server 2008.*x64.* : 170076
.*Windows .*Server 2012.*R2.*x64.* : 95000
.*Windows .*Server 2012.*x64.* : 170099
.*Windows .*Server 2016.*x64.* : 170402
.*Windows 8\.1.*x64.* : 170301
.*Windows 7.*x64.* : 170101
.*Windows 10.*x64.* : 170302
.*Windows 7.* : 170100
.*Hyper-V .*Server 2012.*R2.*x64.* : 170103
.*Windows .*Server 2003.* : 10007
.*Windows .*Server 2008.* : 160076
VMware ESX (Server )?3\.0.* : 10056
VMware ESX (Server )?3\.5.* : 20001
VMware ESX (Server )?3i.* : 30076
VMware ESXi (Server )?3\.5.* : 30076
VMware ESX 4\.1.* : 20096
VMware ESX 4\.0.* : 10096
VMware ESX e\.x\.p.* : 10096
VMware ESXi 5\.0.* : 140086
VMware ESXi 4\.1.* : 130086
VMware ESXi 4\.0.* : 130076
VMware ESXi e\.x\.p.* : 130076

Edit and add the two lines in two os_maps on ALL cores/slices in your environment.

On finishing, run the bs_hardware on unknown OS servers, the server will be reported as windows 2016.

5. How to unlock DB users

The DB user was found to be locked. The problem can appear when the user tries to login with an incorrect password for several times.

The fix for the issue is,

Execute the following commands.

su – oracle
sqlplus “/as sysdba”
SQL> select username, expiry_date, account_status from dba_users;

After getting the output of this SQL, check which account is locked.

The locked account can be unlocked by the following query:

SQL> alter user <LOCKED_USERNAME> account unlock;

6. Performing the platform installer deployment

The steps for the instalment of the platform installer deployment are given below.

The platform installer for new OSs is available to get from the link

1. upload to the SA core server through SFTP (WinSCP, FileZilla, etc..)

2. SSH into the server core present above.

3. (As Root) /opt/opsware/bin/apxtool import <$FULL_PATH_NAME_OF_FILE>
Example:
[root]/home/$user# /opt/opsware/bin/apxtool import /home/$user/PlatformInstaller-Windows2012R2-50.0.46817.0.0.zip
APX with the unique name ‘com.hp.sa.platform.installer.Windows_2012_R2’ does not exist.
Register it into the system? Y/N Y
Info: Successfully registered APX ‘Windows 2012 R2 Platform Installer’ (5120001) in folder ‘/Opsware/Tools/Managed Platforms Support’. 

4. Login into the SA core server from above through the desktop client.

5. Proceed to the Administration column in platform support and the user will notice Windows 2012 R2.

6. Check the state column. If it’s not installed then right click and select Run.

7. After the installation, verify if the state is set to installed. If it is, right click and delete the platform installer.

7. Solution for the transaction errors “ActionStatus.ABORT_Only” is not in a valid state

A user has reported that it was able to login into HPSA but it was not able to install software. The software that was tried to be installed was a windows software.

They also had mesh issues and had to restart SA across the entire mesh. The error remains even after multiple attempts. The job failed to get a JOBID and doesn’t show up in logs and sessions.

The error that appeared was, 

Transaction TransactionImple < ac, BasicAction: 0:ffff0ad68a34:4eebfd0a:5e989733:899b6d status: ActionStatus.ABORT_ONLY > is not in a valid state to be invoking cache operations on.

The solution for the issue faced is,

Command to resolve the problem with the users: 1-line command must be run in each core separately.

env LD_LIBRARY_PATH=/opt/opsware/lib PYTHONPATH=/opt/opsware/pylibs2:/opt/opsware/spin /opt/opsware/bin/python2 -c 'from opsware_common import conf;conf.__conf__.load("/etc/opt/opsware/spin/spin.args");import spinconf,truthdb,spinobj;spinconf.initLocal();spinconf.loadConfFromTruth();db=truthdb.DB();cur=db.getCursor();cur.execute("update aaa.role ar_c set role_name=substr(:1

role_id

:2

role_name,1,4000) where namespace=:3 and role_name not like :4 and role_hash like :5 and role_name not like :6

(select role_name from aaa.role ar, aaa.role_child arc where ar.role_id=arc.role_id and arc.child_role_id=ar_c.role_id)",("__orphaned_","_","OPE","__orphaned_%","{ope}%","%

"));print "Mismatched child OPE aaa.roles rows updated: %d" % cur.cur.cur.rowcount;db.commit()'

Note: The command is same but now it includes a “commit” statement to really update the records in the database.

As precaution it is advised to run the query on all the cores.

/opt/opsware/support/bin/sql -a “select ar_p.role_id, ar_p.rolespace, ar_p.namespace, ar_p.role_name, ar_p.role_hash, ar_c.role_id, ar_c.namespace, ar_c.namespace, ar_c.role_name, ar_c.role_hash from aaa.role ar_p, aaa.role_child arc, aaa.role ar_c where ar_p.role_id=arc.role_id and arc.child_role_id=ar_c.role_id and ar_c.namespace=’OPE’ and ar_c.role_name not like ‘%’

ar_p.role_name” 

If the output is obtained, then the 1-liner in the environment must be run to fix the accounts that are impacted there as well.

Once executed in all cores, it will be possible to remediate again.

8. How to set SQLNET.ALLOWED_LOGON_VERSION_SERVER to 11 or lower

For adding the “SQLNET.ALLOWED_LOGON_VERSION_SERVER” parameter to the “sqlnet.ora” of the oracle database, it must be set to 11 or lower. If the value is increased to 12 or higher then the process will fail.

The error message that will be displayed is that the account is locked.

WARNING: an error was detected while running an external

command or script.  The output follows:

Baseline ADM: '/opt/opsware/da/webapps/arm/WEB-INF/baselineData.sh  -targets'
22 Jul 2020 01:08:26,099 INFO  BaselineOptions - Using options: saServer='localhost' saPort='1026' saUser='detuser' useSSL='false' dmaServer='localhost' dmaProtocol='http' dmaPort=7080 noDeviceGroupMirror='false' hasTargets='true' xmlFile='/opt/opsware/da/webapps/arm/WEB-INF/classes/daData.xml'
22 Jul 2020 01:08:30,380 INFO  DAConfiguration - Read configuration information from '/etc/opt/opsware/da/da.conf'
22 Jul 2020 01:08:30,417 INFO  DAConfiguration - Read configuration information from '/etc/opt/opsware/da/da_custom.conf'
22 Jul 2020 01:08:30,418 INFO  DAConfiguration - Read config property 'truth.tnsdir': '/var/opt/oracle'
22 Jul 2020 01:08:30,418 INFO  DAConfiguration - Read config property 'truth.port': 'XXXX'
22 Jul 2020 01:08:30,419 INFO  DAConfiguration - Read config property 'truth.servicename': 'truth'
22 Jul 2020 01:08:30,419 INFO  DAConfiguration - Read config property 'truth.host': 'XXXXXXXXXXXXX'
22 Jul 2020 01:08:30,419 INFO  DAConfiguration - Read config property 'truth.sid': 'truth'
22 Jul 2020 01:08:30,829 INFO  HibernateConnectionUrl - OracleDatabaseSource url='jdbc:oracle:thin:@truth'
22 Jul 2020 01:08:30,830 INFO  HibernateConnectionUrl - Using hibernate connection url jdbc:oracle:thin:@truth
22 Jul 2020 01:09:19,182 WARN  BasicResourcePool$AcquireTask - [email protected]e5 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
java.sql.SQLException: ORA-28000: the account is locked

During the installation, the ADM component will be attempting to log into the database 30 times. In the above example that is given, the ‘TWIST’ schema user had implemented a policy setup that would automatically lock the account after 10 failed login attempts. As a result of that, the final error indicated that the account was locked.

The cause for the issue was found by the use of an older version of the ojdbc6.jar to connect to the database. 

A login failure message will be displayed if the value of the “SQLNET.ALLOWED_LOGON_VERSION_SERVER” parameter of “sqlnet.ora” is set to 12 or higher.

In previous versions of the oracle, the mixup of the login protocols will show one of the errors.

ORA-28040: No matching authentication protocol error

ORA-03134: Connections to this server version are no longer supported

To resolve the issue, follow the steps given below.

1. change the “SQLNET.ALLOWED_LOGON_VERSION_SERVER” parameter in “sqlnet.ora” to 11 or lower.

2. Restart the database.

3. The 3rd step results because of the initially created while “SQLNET.ALLOWED_LOGON_VERSION_SERVER” is set to “12” or higher, then the password will be stored without the “10G” version.

The versions of the password can be checked by the following query.

# /opt/opsware/support/bin/sql “select password_versions from dba_users where username=’TWIST'”

Query #1 on Facility_id 2 (SA1060B):

PASSWORD_VERSIONS

—————–

11G 12C

After the “SQLNET.ALLOWED_LOGON_VERSION_SERVER” is set back to “11” or lower, the database is restarted, and then the schema user TWIST password is reset, the output of the above query should will be the following:

# /opt/opsware/support/bin/sql “select password_versions from dba_users where username=’TWIST'”

Query #1 on Facility_id 2 (SA1060B):

PASSWORD_VERSIONS

—————–

10G 11G 12C

9. How to install the server extension for registered software can not be installed

There are reports reading that the server extension for registered software cannot be installed on some hosts.

The solution for this issue is as follows.

2020-01-08 21:18:44:027 2204 1708 COMAPI WARNING: ISusInternal::GetUpdateMetadata2 failed, hr=8007000E

Considering that the OS here is Windows 7, the required patch for the fix that will rectify the complication can be obtained from the link.

After completing the installation, the following commands must be entered in an elevated command prompt to make it work error free.

Net stop wuauserv
Sc config wuauserv type= own
Net start wuauserv

10. Solution for the unresponsiveness of server automation 10.60

Various complaints have been made regarding the unresponsive state of the server automation 10.60.  The few users who were able to log in were unable to perform any tasks. Even after restarting the application a number of times the problem still exists.

These steps provided below can fix the problem encountered.

1. Stop SA -/etc/init.d/opsware-sas stop

2. Rebuild the fuse adapter.

as root user: /opt/opsware/adapter/Linux/src/reinstall

There will be messages displayed like 

Creating new FUSE build log /var/log/opsware/adapter/adapter-build-2020-02-19T21:50:58.954778168Z
Building FUSE kernel module…
Building OGFS extension module…
Build of ogfs extension module succeeded.
Building adapter…
Rebuild succeeded.

If the rebuild fails to succeed then send the adapter file as shown above.

3. The user is then instructed to restart SA and test again.

The procedure for the making the restart,

service opsware-sas stop on slice 1.
service opsware-sas stop on slice 2.
service opsware-sas stop on Core (Slice 0).

service opsware-sas restart on Core (Slice 0).
service opsware-sas restart on Slice 1.
service opsware-sas restart on Slice 2.

Upon investigating the root cause of the problem encountered, the following was found.

From the spin logs of the 093 slice the output there is:

[19/Feb/2020 20:11:22 +0000] ERROR “spin.permissions” – – – – “{‘msg’: ‘DENIED’, ‘timestamp’: ’19/Feb/2020 201121′, ‘line’: 859, ‘hostname’: ‘USMAGLKFNG093’, ‘method’: ‘checkObjectPerms’, ‘module’: ‘spinmethods.py’}”
[19/Feb/2020 21:57:55 +0000] INFO “Adding new unit kernel-2.6.32-754.27.1.el6.x86_64.rpmRPM to recommended_patches table” – – – – “”
The hpsa_fuse adapter on the 093 servers cannot be accessed; that was the main record gathered by these logs.