Micro Focus Server Automation – Tips and Tricks

Table of Contents

Micro Focus Server Automation – Tips and Tricks – Apr 2021

Server Automation (SA): Upgrading SA core to RH7.6/RH7.7 fails

Have you tried to upgrade the OS of a Server Automation (SA) with a core from RH7.x to RH7.6/RH7.7? It fails in the evaluation phase with an error about the ksh rpm.

An error is seen while attempting to perform a minor OS upgrade from Redhat 7.x (7.0 thru 7.5) to Redhat 7.6 or 7.7. Here is the error when performing the “yum upgrade” command. 

Package: OPSWoracle_rdbms_part3-12.1.0.2.0-6.x86_64 (installed)

Requires: /usr/bin/ksh

Removing: ksh-20120801-137.el7.x86_64 (@rhel-7-server-rpms

Not found

Updated By: ksh-20120801-139.el7.x86_64 (rhel-7-server-rpms)

Not found

The upgrade then aborts.  

Note #1:  The ksh package version being removed (ksh-20120801-137) will vary depending on the version of Redhat 7.x already loaded.  

Note #2:  Only minor OS upgrades are allowed on SA core servers.  Major OS upgrades (for example, RH 6.x to RH7.x) are not permitted.  

This problem arises due to the change in the linkage used by the later versions of Redhat for the ksh binary.Previous versions of Redhat provided a link between /bin/ksh and /usr/bin/ksh.  

To address the problem, one of two workarounds can be employed. 

Method 1: Instruct yum to skip the ksh rpm upgrade during the upgrade of the OS.Instead of “yum upgrade”, use “yum -x ksh upgrade”

Method 2: Download the latest ksh rpm and upgrade ksh PRIOR to issuing the yum command to upgrade the OS. Download the latest ksh rpm and store it in /tmp (for example ksh-20120801-139.el7.x86_64.rpm). Upgrade the ksh rpm with the command “rpm -U /tmp/ksh-20120801-139.el7.x86_64.rpm” continue to upgrade the OS using the command “yum upgrade”

WINDOWS 2016 showing as UNKNOWN system in HPSA

Users for Windows 2016 servers are newly provisioned and are showing UNKNOWN within HPSA. The latest 10.23 HPSA patches have been applied. This matter is critical because of this we cannot deploy Windows 2016 servers.

Regarding the problem, SA 10.23 by default does not support Windows 2016, so just put in the Windows 2016 Platform Installer in order for SA to be able to detect and handle Windows 2016.

Use the provided reference: Platform Installer – Windows 2016 50.0.73174.0.1

Signature verification failure Session

You can run all probes tool records.

Running WAY:1018:wayrpc test: FAILURE

Check state machine: SUCCESS

Check session tables: SUCCESS

Check lock-down status: SUCCESS

Check signature failures: FAILURE (details below)

————————–

Signature verification failures: Session:350430100

————————–

Solution

Actually, there are zombie sessions but not in this case. The session was gone after waybot restart.

Add RH8 content to redhat_import

Do you know how to add the proper content labels to enable downloading of RH 8 patches using redhat_import?RH 8 agent support has been released for SA and new entries are necessary to enable redhat_import to download patches from Redhat.

The “os” labels are needed for patches, the “appstream” labels are an example of how to add other streams available from RH content_labels:

 rhel-8-for-x86_64-baseos-rpms{8}

rhel-8-for-x86_64-appstream-rpms{8}

[rhel-8-for-x86_64-baseos-rpms{8}]

platform=Red Hat Enterprise Linux 8 X86_64

[rhel-8-for-x86_64-appstream-rpms{8}]

platform=Red Hat Enterprise Linux 8 X86_64

Server Automation (SA): MpC utility fails due to user authentication

There is an error while attempting to use the marketplace-connector.sh utility in order to download and import content from the MarketPlace website. The error is “Download failed due to error in user authentication” is seen.

Server Automation (SA) uses the script marketplace-connector.sh in order to contact the MarketPlace website and download content from the website. When attempting to run the utility, the following error is seen:

Determining content status information for stream <MpC Stream>

Finished determining content status information for stream <MpC Stream>

Download failed due to an error in user authentication.

Where <MpC Stream> is one of the content screams available from the MarketPlace website.

This error can be seen when first attempting to use the MarketPlace connector or after using it successfully in the past.  

The marketplace-connector.sh script needs to be configured to use the MarketPlace/MySupport user set up for the user. This error is seen because the user id configured in the marketplace-connector.sh utility does not exist on the MarketPlace website, exists but has the incorrect password, or the user id is in an outdated (email-based) form.

To address this issue, first determine the user id being used by the marketplace-connector.sh utility. 

1. In the server where the marketplace-connector.sh utility is being run, go to the directory where the utility is installed and issue the command./marketplace-connector.sh read-config.In the chart of options/values shown, the MpC user id being used will appear as the value for the “username” option.  

2. Verify that the username value is NOT in the form of an email address. For example, “[email protected]”.

Email-based usernames were obsoleted in favor of “unified” or one-word usernames. For example “johndoe” or “john_doe_user”.

If an email-based username is seen, proceed to the MarketPlace website and create a new account.  If necessary, engage the Microfocus account team for assistance doing this.  

3. If a valid, “one-word” username is being used, verify that it can be used to log into the MarketPlace website. If this login works, go to the “Data Center Automation” section of the website, select the “Security and Compliance for Server Automation” section, and click on the “Download” button next to the MpC content stream that was being attempted when the error was seen.  This will ensure that the account associated with this userid has sufficient permissions to pull this content.  

4. Re-enter/correct the username and password fields being used by the marketplace-connector.sh utility by going to the directory where the utility is installed and issuing the command ./marketplace-connector.sh write-config –username=<one-word-username>.You will be prompted for the password for the account.. enter the one used when logging into the MarketPlace website in the step above.

5. Re -run the “./marketplace-connector.sh” command in order to attempt to download and import the MpC content stream.

If this error is still seen, open a Micro Focus Support case (if there is not already one opened) and provide the output from the command./marketplace-connector.sh read-config and the marketplace-connector.log logfile (located in the directory where the utility is installed).

Server Automation (SA): Steps to prep SA for porting Python 2 code to Python 3

What are the steps that can be taken to prepare current Python 2 code bases in existing Server Automation (SA) environments to function in future SA versions that use Python 3?

Existing versions of Server Automation (SA versions 2018.08 and older) currently using resolution Python 2. However, Python 2 is approaching the end of its support life. This document is an overview of the preparation needed for the next versions of SA that will run on Python 3.  This preparation must take place prior to upgrading any existing SA versions (2018.08 and older) to future versions of SA.  

First of all, existing SA versions (2018.08 and older) will not be getting updated to Python 3. The steps are mentioned below. 

We can only add support for running Python 2/3 compatible code and allow users to begin converting their existing Python codebase to something that will work on future versions of SA that will be running Python 3. 

Section 1 – How to get SA ready for porting python 2 code to python 3

To enable the python 2 & 3 compatibility on your cores/satellites/agents you need to install the below packages:

SA 2018.08:

SRVA_00278 – server automation 2018.08.004 rollup

SRVA_00266 – server automation 2018.08.002 Agents

SRVA_00267 – server automation 2018.08.002 HPSAPython

SA 10.60

SRVA_00280 – server automation 10.60.014 rollup

SRVA_00281 – server automation 10.60.014 Agents

SRVA_00263 – server automation 10.60.012 HPSApython

SA10.51

SRVA_00271 – server automation 10.51.011 rollup

SRVA_00272 – server automation 10.51.011 Agents

SRVA_00273 – server automation 10.51.011 HPSAPython

SA10.23

SRVA_00274 – server automation 10.23.015 rollup

SRVA_00275 – server automation 10.23.015 Agents

SRVA_00276 – server automation 10.23.015 HPSAPython

Note #1: At the time of this document’s creation, the packages listed above were the latest for each version of SA listed.  If newer versions of the “rollup” or “Agent” package type are available, they can be used instead.  However, the “HPSAPython” package should remain

the same, and can be installed with the newer “rollup” and “Agent” packages.

Note #2: SA10.2x versions prior to 10.23 will have to upgrade to 10.23 before being able to make use of this document.  Likewise, SA 10.50 environments will need to upgrade to SA 10.51 prior to being able to use this document.  SA versions prior to 10.2x CANNOT make use of this new Python 3 functionality and will need to be upgraded to at least 10.23.  

To install the server automation rollup follow the installation instructions from the patch.sh.txt. The agents have the installation instructions in the AGENT_README.txt and the HPSAPython package can be installed following the instructions from the README.txt.

Section 2 – How to port your python scripts

You have a choice between two tools in porting your code automatically: Futurize and Modernize. Which tool you choose will depend on how much like Python 3 you want your code to be. 

Futurize does its best to make Python 3 idioms and practices exist in Python 2, e.g. backporting the bytes type from Python 3 so that you have semantic parity between the major versions of Python. 

Modernize, on the other hand, is more conservative and targets a Python 2/3 subset of Python, directly relying on ‘six’ to help provide compatibility. 

As Python 3 is the future, it might be best to consider Futurize to begin adjusting to any new practices that Python 3 introduces which you are not accustomed to yet. Regardless of which tool you choose, they will update your code to run under Python 3 while staying compatible with the version of Python 2 you started with.

Depending on how conservative you want to be, you may want to run the tool over your test suite first and visually inspect the diff to make sure the transformation is accurate. After you have transformed your test suite and verified that all the tests still pass as expected, then you can transform your application code knowing that any tests which fail is a translation failure.

Unfortunately, the tools can’t automate everything to make your code work under Python 3 and so there are a handful of things you will need to update manually to get full Python 3 support (which of these steps are necessary vary between the tools). 

A detailed description that should be followed for this process can be found in the official documentation. For each tool there is an official documentation that must be consulted and followed.

The needed libraries are delivered and available in the python delivered with SA. The versions of the libraries are future-0.17.1 for future and six-1.11.0 for modernize. These versions have to be used during the code porting. 

We strongly recommend doing the code porting in test environments first. Neither of the two solutions offer a direct or clean code porting process, there will be issues you have to solve manually.

Note: New content should be developed with Python 3 compatibility in mind.

Section 3 – References

Futurize  Modernize    Python 2&3 official documentation

Can’t install HPSA agent: the system cannot find install_tool_x64.exe

Do you know that we cannot install HPSA agents? If you try to do so there is an error. The error is listed below.

[01/Aug/2019 12:39:48] [TRACE] RunCommand(‘install_tool_x64.exe –zap –loglevel=trace’)

[01/Aug/2019 12:39:48] [ERROR] RunCommand() – Popen failed : ‘install_tool_x64.exe –zap –loglevel=trace’ : Error : (2) : ‘The system cannot find the file specified.

[01/Aug/2019 12:39:48] [ERROR] zap_agent: FAIL

[01/Aug/2019 12:39:48] [ERROR] RunCommand() – Popen failed : ‘install_tool_x64.exe –unpack=”C:\Users\X18303~1\AppData\Local\Temp\3\~5504-1.WRK\opsware-agent.exe”,”C:\Program Files\Opsware\agent” –install=”C:\Program Files\Opsware\agent”,”C:\Users\X18303~1\AppData\Local\Temp\3\~5504-1.WRK” –loglevel=trace’ : Error : (2) : ‘The system cannot find the file specified.

[01/Aug/2019 12:39:48] [TRACE] install_tool: FAIL

[01/Aug/2019 12:39:48] [ERROR] Opsware agent installation failed.

The server was showing the following problem. ComSpec=C:\Windows\system32\cmd.exe;C:\Windows\SysWOW64\cmd.exe

And on our test servers it was set like this:

ComSpec=C:\Windows\system32\cmd.exe

Open the CMD as administrator on the Windows server with issues and run the next command:

echo %COMSPEC%

If multiple paths, involve your windows admin team and let them change the value in the registry files. Once done you can confirm only C:\windows\system32\cmd.exe is in use and you shouldn’t have the problem.

Server Automation (SA): “Timed Out” message seen when installing latest Windows patches

When using Server Automation to install the latest Microsoft Windows monthly patches, several servers return a “Timed out” message during the installation.

When running a Server Automation (SA) job to install the latest Microsoft Windows cumulative patch, many of the servers the job is run against return a Failed result with the error “Timed Out”.  Checking the servers, the error is reported against, many of them actually show that the patch(s) were installed correctly.  

When running this type of job, SA will first make sure that it can talk to the managed server (that the SA agent on the managed server is responding to) and then proceed to download the necessary patch files.  If both of these steps are successful, then the SA agent is told to have its server begin the patch installation.  

The SA job by default waits up to one hour for the SA managed server to complete this installation. If after one hour the SA job has not heard back from the managed server that the patch installation is complete, it will mark the server as having “Failed” the patch installation, and indicate it has “Timed Out”.  

However, the installation of the process on the managed server may just be taking longer than expected to install the patch.  The act of marking this server as “Timed Out” in the SA job does not cancel/stop the installation process on the managed server from continuing on. 

If the patch installation is successful, the managed server will return its results back to the SA job.. However, since the SA job has already marked the server as “Failed” it does not undo this status as there could be additional post-installation steps in the SA job that have not been performed.

In this way, the SA job will show the server as “Failed – Timed Out” and yet the patch installation was successfully completed.  

It is possible to change the default amount of time the SA job will wait for the managed servers to complete the actions it’s been instructed to complete (like installing patches, running a script, etc).  To change this default value, pull up the SA java gui, select “Administration”, then “System Configuration” and then-“Configuration Parameters”.

In the right-hand pane, click on “Command Engine (way)”.   This will show all of the “way” parameters that can be modified.  Look through the available parameters (or filter them via the search field in the top right portion of the window pane) and locate the “way.remediate.package_alarm_timeout” parameter. 

By default, this parameter will have a value of “3600” seconds (or 1 hour).  You can increase this value and then save it.  Changing this parameter will not impact SA jobs that are currently running, but any new SA jobs will use this value.

Note:  Take care when changing this parameter. if a very large value is used (say several hours), then the entire SA job could potentially be delayed in completing for that amount of time if even ONE managed server encounters a delay in performing the action requested.  Changing this parameter in small steps is recommended.  

Server Automation (SA): uln_import command fails with “expected string of buffer”

While using the Server Automation (SA) tool “uln_import” the error “TypeError: expected string or buffer” is encountered when attempting to register the SA core with the Oracle Linux web server.

Server Automation (SA) can download and import into its database OEL content.  It does this by using a utility on the SA core server called “uln_import”, which connects to the Oracle Linux website that provides OEL content to users/accounts that have purchased an account with Oracle to access this content.  

The initial registration of the SA core server to the Oracle Linux website is performed by defining variables in the /etc/opt/opsware/patch_importer/uln_import.conf file, and then running the command “/opt/opsware/patch_importer/bin/uln_import –verbose –show_conf”.  

During the running of the above command to register the SA server on the Oracle website, the following error is seen: <date/time> patch_importer_versioned 807 ERROR Unexpected error: failed to register user: XML-RPC Server Error: TypeError: expected string or buffer.

This error is seen both on the session screen where the command is being run from and in the logfile for the utility 

(/var/log/opsware/patch_importer/uln_importer.log)

The problem is seen because the uln_import code cannot properly process the variables defined in the configuration file for the utility (/etc/opt/opsware/patch_importer/uln_import.conf).  

Care should be taken when editing the /etc/opt/opsware/patch_importer/uln_import.conf file to ensure extra whitespaces or control characters are not accidentally added into the file.  A good recommendation would be to save/backup the existing uln_import.conf file, and then copy the /etc/opt/opsware/patch_importer/uln_import.conf-sample to the uln_import.conf file (overwriting it) to ensure there is no corruption in the file. 

Then change each of the required parameters for your environment in the uln_import.conf file and try to again register the SA server with the Oracle website using the uln_import command.  

If that still does not address the issue, bring in a fresh copy of the config file (from uln_import.conf-sample) and then, one by one, add in each of the parameters into the conf file, rerunning the uln_import command after each addition.

While the connection may not work because some of the parameters are missing, the “expected string or buffer” error should not show up UNLESS the variable changed in the config file is the cause of the issue.  

NOTE:  Whitespace characters/spaces CANNOT precede any of the variable names.  

[email protected]” (without the quotes) is acceptable

[email protected]E.COM” (without the quotes) is not.

Server Automation (SA): “Failed to register user” error seen when running uln_import

When using the Server Automation (SA) utility uln_import to register the SA core system with Oracle, the error “Failed to register user” is encountered.

Server Automation (SA) can download and import into its database OEL content.  It does this by using a utility on the SA core server called “uln_import”, which connects to the Oracle Linux website that provides OEL content to users/accounts that have purchased an account with Oracle to access this content.  

The initial registration of the SA core server to the Oracle Linux website is performed by defining variables in the /etc/opt/opsware/patch_importer/uln_import.conf file, and then running the command “/opt/opsware/patch_importer/bin/uln_import –verbose –show_conf”.

While running this command, the following error is encountered and the command aborts.ULNError: failed to register user: XML-RPC Server Error: MaxRetryError: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) 2019-09-07 20:19:58,679 patch_importer_versioned 807 ERROR Unexpected error: failed to register user: XML-RPC Server Error: MaxRetryError: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

This error is seen both on the console session where the uln_import command is being run and in the uln_importer.log file found in the /var/log/opsware/patch_importer directory.

The uln_import utility makes use of configuration variables defined in the file /etc/opt/opsware/patch_importer/uln_import.conf. Three of these variables (username, password, csi) are used when registering the SA sy

stem with the Oracle Linux website that provides the OEL content.  The error is seen when this information is not being accepted by the Oracle website. 

Assuming the values for these three variables were added correctly, verify that they can be used to log directly into the Oracle Linux website by pointing a web browser to the URL http://linux.oracle.com, and signing in using this information.  That will verify that the account information is valid.

If the values work logging into the Oracle website manually, another potential cause of the issue may be due to the certificates that are provided by default for the uln_import utility to use.  There are some cases where it may be necessary to import the certificate for the Oracle Linux website into the certificate file ca-bundle.crt.  This is particularly true if a proxy is NOT defined in the uln_import.conf file.  

To import the Oracle Linux certificate into the SA servers ca-bundle.crt file, do follow the below written steps.

Step 1: Make a backup of the existing certificate file  cp /opt/opsware/openssl/certs/ca-bundle.crt /opt/opsware/openssl/certs/ca-bundle.crt_backup.

Step 2: Get the latest certificate from Oracle 

Step 3: Open the downloaded file (ULN-CA-CERT.sha2) and look at the end of the file to locate the last certificate. 

Step 4: Copy this certificate (starting with and including the “BEGIN CERTIFICATE” line to and including the “END CERTIFICATE” line).  

Step 5: Open /edit the /opt/opsware/openssl/certs/ca-bundle.crt and add this certificate to the end of the file.

Step 6: Save the file. 

Step 7: Retry with the uln_import command and attempt to register with the Oracle website.

If neither of these solutions work, open a Support ticket and provide the /var/log/opsware/patch_importer/uln_importer.log and 

the /etc/opt/opsware/patch_importer/uln_import.conf files.