Micro Focus Server Automation – Tips and Tricks

Table of Contents

Micro Focus Server Automation – Tips and Tricks – Feb 2021

1. Server Automation (SA): Upgrading SA core to RH7.6/RH7.7 fails error

The users may face the error about the ksh rpm while attempting to upgrade the OS of a Server Automation (SA) core system from RH7.x to RH7.6/RH7.7 that fails the evaluation phase. While attempting to perform a minor OS upgrade from Redhat 7.x (7.0 thru 7.5) to Redhat 7.6 or 7.7, the error was detected while performing “yum upgrade” command. The error is stated below

Error: Package: OPSWoracle_rdbms_part3-12.1.0.2.0-6.x86_64 (installed)
   Requires: /usr/bin/ksh
   Removing: ksh-20120801-137.el7.x86_64 (@rhel-7-server-rpms)
     Not found
   Updated By: ksh-20120801-139.el7.x86_64 (rhel-7-server-rpms)
     Not found

The ksh package version being removed (ksh-20120801-137) will vary depending on the version of Redhat 7.x already loaded. Also only minor OS upgrades are allowed on SA core servers.

The problem occurs due to the change in the linkage being used by the later versions of Redhat for the ksh binary. The previous versions of Redhat provided a link between /bin/ksh and /usr/bin/ksh. 

The problem can be fixed by following the steps

  1. Command yum to skip the ksh rpm upgrade during the upgrade of the OS.
  2. Instead of “yum upgrade”, use “yum -x ksh upgrade”
  3. Download the latest ksh rpm and upgrade ksh PRIOR to issuing the yum command to upgrade the OS
  4. Download the latest  ksh rpm and store it in /tmp (for example ksh-20120801-139.el7.x86_64.rpm)
  5. With the command upgrade the ksh “rpm rpm -U /tmp/ksh-20120801-139.el7.x86_64.rpm”.
  6. Continue the upgrade of OS using the command “yum upgrade”.

2. Server Automation (SA): uln_import command fails with “expected string of buffer” Error

While using the Server Automation (SA) tool the user may face the error “uln_import” the error “TypeError: expected string or buffer” is encountered when attempting to register the SA core with the Oracle linux web server. Server Automation (SA) does this by using the utility on the SA core server called “uln_import”.

It downloads and imports into its database OEL content. The uln_import connects the Oracle Linux website that provides OEL content to the users that have purchased an account with Oracle to access this content. The initial registration of the SA core server to the Oracle Linux website is performed by defining variables in the /etc/opt/opsware/patch_importer/uln_import.conf file, and then running the command

"/opt/opsware/patch_importer/bin/uln_import --verbose --show_conf". 

While running the above command for registering SA server on the Oracle website the user may face the following error: 

<date/time> patch_importer_versioned 807 ERROR Unexpected error: failed to register user: XML-RPC Server Error: TypeError: expected string or buffer.

This error is seen both on the session screen where the command is being run from and in the logfile for the utility.

/var/log/opsware/patch_importer/uln_importer.log

The problem occurs due to the uln_import code cannot properly process the variables defined in the configuration file for the utility.

/etc/opt/opsware/patch_importer/uln_import.conf

The problem can be fixed by following the steps given below:

I.        An excessive care must be taken while editing the /etc/opt/opsware/patch_importer/uln_import.conf file to ensure extra whitespaces or control characters are not accidentally added into the file.

 II.        A better recommendation would be to backup the existing uln_import.conf file, and then copy the/etc/opt/opsware/patch_importer/uln_import.conf-sample to the uln_import.conf file (overwriting it) to ensure there is no corruption in the file.

III.        Later change the required parameters for your environment in the uln_import.conf file and try to again register the SA server with the Oracle website using the uln_import command.

IV.        If still the issue is not addressed then bring a fresh copy of config file (from uln_import.conf-sample) and then, one by one, add each of the parameters into the conf file, rerunning the uln_import command after each addition.

 V.        The connection may not work because some of the parameters are missing, the “expected string or buffer error should not show up UNLESS the variable changed in the config file is the cause of the issue. 

VI.        Whitespace characters cannot precede any of the variable names.

[email protected]” (without the quotes) is acceptable

[email protected]” (without the quotes) is not.

3. Platform Support – SuSE 15 Error

On Server Automation 10.23 the user tried to install Platform Support for SuSE 15, but it was unsuccessful. After the platform was upgraded and the user reached 10.60 but the platform installer for SuSE 15 is still the same as it was for SA 10.23.

The version of the platform support says that it’s not compatible with SA 10.60. Various other platforms like Java GUI, apxtool, were used as alternatives but the result was either not successful or rolled back. New platform support is required to be installed to manage SuSE 15. If there is a way to force remove the old content or at least replace it with the new then that could be a solution to the problem.

The solution to the problem is given within the following steps:

          I.        Download the latest platform support from the MF website.

          II.        The SA won’t let the user uninstall the older version from Platform Support but try installing the newer version overwriting the oldest.

        III.        It would work fine over the course of time

4. Agent uses sha1WithRSAEncryption Signature Algorithm in Certificate Error

When a third party does the security finding against the system a called out problem occurs with version 75.0.807460 that the signature algorithm used in the SSL Certificate returned is sha1 instead of sha256 or sha512. The command used to show this was openssl s_client -connect: 1002 2>/dev/null openssl x509 -noout –textIs. There could be other secure signature methods or it could require any updated agent code.

The solution for the issue is given below:

    I.        To enable SHA256 or SHA512 in your agent, the user needs to recertificate the entire core. 

 II.        During the recertification, the agent on the managed servers will be updated with the same algorithm.

The agents cannot run a higher crypta than what the actual core is running. Everyone must use the same version.

5. Server Automation (SA): “Failed to register user” error seen when running uln_import Error

When using the Server Automation (SA) utility uln_import to register the SA core system with Oracle, the user may face the error “Failed to register user”. Server Automation downloads and imports databases into OLE content.

It uses a utility on the SA core server called “uln_import” which connects the Oracle Linux website that provides OEL content to users that have purchased an account with Oracle to get access to the content. This registration of SA server to Oracle website is performed by defining variables in the /etc/opt/opsware/patch_importer/uln_import.conf file.

After run the command “/opt/opsware/patch_importer/bin/uln_import –verbose –show_conf”.  While running the command the user observes the following error and the command aborts.

ULNError: failed to register user: XML-RPC Server Error: MaxRetryError: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
2019-09-07 20:19:58,679 patch_importer_versioned 807 ERROR Unexpected error: failed to register user: XML-RPC Server Error: MaxRetryError: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590).

The error is encountered in both the console session where the uln_import command is being run and in the uln_importer.log file found in the /var/log/opsware/patch_importer directory.

The cause of the issue is the uln_import utility which makes use of configuration variables defined in the file /etc/opt/opsware/patch_importer/uln_import.conf. The three variables username, password, csi are used while registering the SA system with the Oracle Linux website that provides the OLE content. The error is encountered when the information is not being accepted by the Oracle website.

To fix the issue following steps are required

        I.        With an assumption that the value of the three variables i.e. username, password, and csi  are added correctly verify that they can be used to log in directly into the Oracle Linux website by pointing a web browser to the URL http://linux.oracle.com, and signing in using this information.

   II.        This will verify that the account information is valid.

   III.        If the values work logging into the Oracle website manually other potential cause of the issue  must be the certificate  that are provided by default for the uln_import utility to use

   IV.        There are some cases in which it is necessary to import the certificate for the Oracle Linux website into the certificate file ca-bundle.crt. This is particularly true if a proxy is not defined in the uln_import.conf file.

  V.        Make a backup of the existing certificatecp /opt/opsware/openssl/certs/ca-bundle.crt /opt/opsware/openssl/certs/ca-bundle.crt_backup  

   VI.        From Oracle get the latest certificate Wget https://linux-update.oracle.com/rpms/ULN-CA-CERT.sha2

VII.        Open the downloaded file (ULN-CA-CERT.sha2) and look at the end of the file to locate the last certificate. Copy the certificate which starts from “BEGIN CERTIFICATE” line to and including the “END CERTIFICATE” line.

VIII.        Open/edit the /opt/opsware/openssl/certs/ca-bundle.crt and add this certificate to the end of the file.

   IX.        Save the file

  X.        Retry the uln_import command and attempt to register with the Oracle website.

   XI.        If neither of these solutions work, open a Support ticket and provide the /var/log/opsware/patch_importer/uln_importer.log and the /etc/opt/opsware/patch_importer/uln_import.conf files.  

6. Server Automation (SA): “Timed Out” message seen when installing latest Windows patches Error

When using the Server Automation (SA) to install the latest Microsoft Windows monthly patches, the user may face an issue with several servers that popup with a message “Timed out” during installation. The job of the installation is run against and the return of failed error with the message Timed out. Checking the servers the error is reported again since many of them actually show that the patch was installed correctly.

The SA makes sure to contact the managed server firstly while running this kind of job and then will proceed with the downloading of the patch file if the SA agent is responding on the managed server. If both the steps are successful then the SA server begins the patch installation.

The SA job is to by default complete the patch installation on the managed server in an hour. If after an hour the SA is not heard back about the installation from the server then the task is complete if not then it will mark the server with Failed or Timed out the message. However, the installation of the process on the managed server may just be taking longer than expected to install the patch.

When the act of marking or failing is done by the managed server it doesn’t stop or cancel the installation process on the managed server from continuing on. If the patch installation is successful, the managed server will return its results back to the SA job.

But since the server is marked with Failed it doesn’t undo the status and there are additional post-installation steps in the SA job that have not been performed. In this way, the SA job will show the server as “Failed – Timed Out” and yet the patch installation was successfully completed. 

The issue could be fixed by following the below steps

    I.        If possible to change the default amount of time the SA managed server will wait to complete the actions it’s been instructed to complete.

 II.        To change the default value pull up the SA java gui, select “Administration”, then “System Configuration” and then-“Configuration Parameters”.

III.        In the right-side panel click on “Command Engine (way)”

IV.        This will show all the “way” parameters that are able to modify.

 V.        Look for the available parameters or filter them via the search field in the top right portion of the window panel and locate the “way.remediate.package_alarm_timeout” parameter.

VI.        By default this parameter has a value of “3600” seconds or 1 hour.

VII.        Increase the value and save it.

VIII.        Changing this parameter will not impact SA jobs that are currently running, but any new SA jobs will use this value.  

IX.        A proper care must be taken when changing the parameters if a higher value is used then the entire SA job would be delayed in completing for that amount of time if even one managed server encounters a delay in performing the action requested.

 X.        Changing this parameter in small steps is recommended.

7. Server Automation (SA): Steps to prep SA for porting Python 2 code to Python 3 Error

The steps that can be considered while preparing current Python 2 codebases in existing Server Automation (SA) environments to function in future SA versions that use Python 3. The existing version of SA i.e. 2018.08 and older uses Python 2.  However, Python 2 is nearly at the end of its support life. Here is the overview for the preparation needed for the next version of SA that will run on Python 3. The preparation must be done prior to upgrading any existing SA versions 2018.08 and

Older to future versions of SA.

The solution to prepare Python 3 is given below:

Being clear the existing version 2018.08 will not get updated to Python 3 the steps mentioned here will only add support for running Python 2/3 compatible code while allowing the user to begin converting their existing Python codebase to something that will support and work on the future version of SA and run on Python 3.

Ø How to get SA ready for porting python 2 code to python 3

    I.        To enable the python 2 & 3 compatibilities on the SA cores/satellites/agents the user need to install the below packages:

SA 2018.08:

• SRVA_00278 – server automation 2018.08.004 rollup

• SRVA_00266 – server automation 2018.08.002 Agents

• SRVA_00267 – server automation 2018.08.002 HPSAPython

SA 10.60

• SRVA_00280 – server automation 10.60.014 rollup

• SRVA_00281 – server automation 10.60.014 Agents

• SRVA_00263 – server automation 10.60.012 HPSApython

SA10.51

• SRVA_00271 – server automation 10.51.011 rollup

• SRVA_00272 – server automation 10.51.011 Agents

• SRVA_00273 – server automation 10.51.011 HPSAPython

SA10.23

• SRVA_00274 – server automation 10.23.015 rollup

• SRVA_00275 – server automation 10.23.015 Agents

• SRVA_00276 – server automation 10.23.015 HPSAPython

 II.        Always look for the newer version of the “rollup” or “Agent” package type since various versions are available till the time instead of the above listed.

III.        The “HPSAPython” package remains the same and can be installed with the newer “rollup” and “Agent” packages.

IV.        SA10.2x versions prior to 10.23 will have to upgrade to 10.23 before being able to make use of the document.

 V.        SA 10.50 environments will need to upgrade to SA 10.51 prior to being able to use the document.

VI.        SA versions prior to 10.2x CANNOT make use of this new Python 3 functionality which will need to be upgraded to at least 10.23.

VII.        To install a server automatically rolling up the user must follow the instruction of installation from the patch.sh.txt. 

VIII.        For agents the installation instructions are provided in the AGENT_README.txt and the HPSAPython package and that can be installed following the instructions from the README.txt.

How to port your python scripts

The user has a choice between two tools in porting the code automatically that is Futurize and Modernize. The choice depends upon the like of Python 3 in the code. Futurize is best in making Python 3 idioms and practices exist in Python 2 for example: backporting the bytes type from Python 3 so the user can have semantic parity between the major versions of Python.

Modernize, on the other hand, is conservative and targets a Python 2/3 subset of Python directly relying on ‘six’ to help provide compatibility.

As Python 3 is the future, Futurize seems to be the better choice since it can adjust to any new practice that Python 3 introduces which the users are not accustomed to yet. Regardless of which tool the user chooses, they will update the code to run under Python 3 while staying compatible with the version of Python 2 that the user began with.

Depending on how conservative the user wants to be, they may want to run the tool over their test suite first and visually inspect the difficulties to make sure the transformation is accurate. After transforming the test suite and verifying that all the tests still pass as expected, the user can transform their application code knowing that any tests which fail are a translation failure.

Unfortunately, the tools can’t automate everything to make the code work under Python 3 and so there are a handful of things that will need to update manually to get full Python 3 support. The necessary steps vary between the tools.

The detailed description which is needed to be followed during the process can be found in the official documentation. For every tool, there is the official documentation that must be consulted and followed.

The needed libraries are delivered and available in the python delivered with SA. The versions of the libraries are future-0.17.1 for futurize and six-1.11.0 for modernize. These versions are to be used while the code is ported. A strong recommendation is to perform the porting in the test environment first.

Neither of the two solutions offer a direct or clean code porting process, there will be issues that must be solved manually.  New content should be developed with Python 3 compatibility in mind.

 For References check out the below links:

Futurize – http://python-future.org/automatic_conversion.html

Modernize – https://python-modernize.readthedocs.io/en/latest/index.html

8. Server Automation (SA): MpC utility fails due to user authentication Error

When the user attempts to use the marketplace-connector.sh utility in order to download and import content from the MarketPlace website, an error “Download failed due to error in user authentication” is encountered.

The Server Automation (SA) uses the marketplace scripts of connector.sh in order to contact the Marketplace website to download the content from the website. While attempting to run the utility the following error is observed:  

Determining content status information for stream <MpC Stream>

 Finished determining content status information for stream <MpC Stream>

Download failed due to an error in user authentication.

<MpC Stream> is one of the content screams available from the MarketPlace website.  

The error can be seen while attempting to use the Marketplace connector for the first time or after using it in the past successfully.

The cause of the issue is the marketplace-connector.sh script needs to be configured to use the Marketplace user set up for the customer. The error occurs due to the user id configured in the marketplace-connector.sh utility does not exist on the MarketPlace website. It can also occur when the id exists but the password is incorrect or the id is outdated.

The solution to fix the issue is explained in the following steps:

  1. Firstly, determine the user id used by the marketplace-connector.sh utility.
  2. On the server where the marketplace-connector.sh utility is being run search the directory where the utility is installed and issue the command ./marketplace-connector.sh read-config
  3. In the chart of options shown, the MpC user id being used will appear as the value for the “username” option.
  4. Secondly, verify the username value and it should not be in the form of an email address. For example “[email protected]”.
  5. Email-based usernames were obsoleted in favour of “unified” or one-word usernames. For example “johndoe” or “john_doe_user”.
  6. If there is an email-based user id proceed to the MarketPlace website at https://marketplace.microfocus.com/itom/content/marketplace-connector-mpc and create a new account.
  7. If necessary, the user can engage the Micro focus account team for assistance doing that.
  8. Thirdly, if a valid, “one-word” username is being used, verify that it can be used to log into the MarketPlace website.
  9. If the one-word username login works go on to the “Data Center Automation” section of the website, select the “Security and Compliance for Server Automation” section then click on “Download” button next to the MpC content stream that was being attempted when the error was seen.
  10. This will ensure that the user id credited to the account has sufficient permissions to pull this content.
  11. Fourthly, , re-enter/correct the username and password fields being used by the marketplace-connector.sh utility by going to the directory where the utility is installed and issuing the command ./marketplace-connector.sh write-config –username=<one-word-username>
  12. This will prompt for the password for the account, enter the one used when logging into the MarketPlace website in the step above.
  13. Finally, rerun the “./marketplace-connector.sh” command in order to attempt to download and import the MpC content stream.
  14. If the error still occurs, open a Micro Focus Support case if there is not already one opened and provide the output from the command ./marketplace-connector.sh read-config and the marketplace-connector.log log file which is located in the directory where the utility is installed.

9. Opsware-sas: Cannot start “twist” – dependency check failed Error

While verifying communication with “vault” the user waits for approx. 5 minutes and then the error Operation timed out – FAILURE: Exception: Error establishing a tunnel to Vault:[email protected] through opswgw at localhost:8080: 503opsware-sas: Cannot start “twist” – dependency check failed to perform “start” operation on Opsware SAS components appears.

To solve the issue the following steps could be considered:

    I.        To reproduce it:

1- /etc/init.d/opsware-sas stop vaultdaemon

2- /etc/init.d/opsware-sas stop twist

3- /etc/init.d/opsware-sas start twist. Please check the commands below:

[[email protected] ~]# /etc/init.d/opsware-sas start twist
Verify twist dependencies...
Verify hostname "spin" is resolvable: SUCCESS
Verify hostname "twist" is resolvable: SUCCESS
Verify hostname "spin" is listening on port 1004: SUCCESS
Verify communication with "spin":
SUCCESS
Verify hostname "localhost" is listening on port 1007: SUCCESS
Verify communication with "spin":
SUCCESS
Verifying en_US.UTF-8 locale is available: SUCCESS
Verify hostname "way" is listening on port 1018: SUCCESS
Verify communication with "vault": (Waiting approx. 5 minutes)
Operation timed out - FAILURE: Exception: Error establishing a tunnel to Vault:[email protected] through opswgw at localhost:8080: 503
opsware-sas: Cannot start "twist" - dependency check failed
Failed to perform "start" operation on Opsware SAS components.

 II.        To solve the issue:

1- /etc/init.d/opsware-sas start vaultdaemon

2- /etc/init.d/opsware-sas start twist. Please check the outputs below:

[[email protected] ~]# /etc/init.d/opsware-sas start vaultdaemon
Verify vaultdaemon dependencies...
Verify hostname "16.40.174.132" is resolvable: SUCCESS
Verify hostname "16.40.174.132" is listening on port 1521: SUCCESS
Verify vaultdaemon can connect to Truth database: [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
[main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
starting the vault in verify mode
SUCCESS
Starting vaultdaemon ...
Starting /etc/opt/opsware/startup//vaultdaemon
Successfully performed "start" operation on Opsware SAS components.
[[email protected] ~]# /etc/init.d/opsware-sas start twist
Verify twist dependencies...
Verify hostname "spin" is resolvable: SUCCESS
Verify hostname "twist" is resolvable: SUCCESS
Verify hostname "spin" is listening on port 1004: SUCCESS
Verify communication with "spin": SUCCESS
Verify hostname "localhost" is listening on port 1007: SUCCESS
Verify communication with "spin": SUCCESS
Verifying en_US.UTF-8 locale is available: SUCCESS
Verify hostname "way" is listening on port 1018: SUCCESS
Verify communication with "vault": SUCCESS
Starting twist ...
Starting twist:
Starting Web Services Data Access Engine.
nohup: redirecting stderr to stdout
Successfully performed "start" operation on Opsware SAS components.

10. OSBP ‘Set Media Source’ does not work when booting with Windows PE 10 Error

When the user tries to run the OS buildplan on a ProLiant BL460c Gen9 server in UEFI mode after booting over PXE with WinPE 10.0 the buildplan shows the following error at step “Set Media Source”. Step 7 of 19 which is Run Server Script ‘Set Media Source’, Failed to Set Media Source. The specified server cannot perform the requested operation.Set Media Source failed with exit code 120.

If the server is changed to BIOS mode while choosing WinPE 4.0 after PXE boot, the OS plan runs without any error. But if the user shifts to WinPE 10.0 the error appears.

Set media source step configuration:

Script: /Opsware/Tools/OS Provisioning/OS Build Plan Steps/Set Media Source

Script Type: Python

Parameters: “smb://@[email protected]/osmedia”

To solve the problem following steps could be of help

  • After “net use” failed to work and the user found out that Windows 10 and also WinPE does not allow “anonymous” connections by default.
  • The user needs to provide a username and password to make it work.
  • It’s fine if the user doesn’t exist.
  • Changing the Set Media Source String from “smb://@[email protected]/osmedia” to “smb://user:[email protected]@[email protected]/osmedia” does the trick and the error disappears while the system starts working.