Test Automation

Table of contents

Related Topics

Overview of Steps involved with Automation Process

Prerequisites for Installation

General

Agent Specific

Installing the Test Automation Package

Post Install Considerations

Default Settings

Creating an Answer File

Executing Test Cases

Viewing Test Results

Viewing Execution Logs

Troubleshooting Tips

General

Agent Specific

System Requirements

Review the System Requirements for the Agent for which you want to execute the tests.

Installing 32 bit Microsoft Visual C++ 2008 Redistributable package

Provides  information on how to install the package

 

 

Overview of Steps involved with Automation Process

The following is a overview of the various steps involved with successfully using the Test Automation Process:

  1. Before installing, check the System Requirements and Pre-Requisites for the iDataAgent & Automation package.  Perform the installation by following the step-by-step install procedure.
  2. Check and perform any post-install configurations needed before executing the Test cases.
  3. Create a Answers File (all the questions to be answered according to the test environment). 
  4. Save and make this Answer file the "default".
  5. Execute the tests using Test Automation GUI
  6. Analyze the Test execution results.  Check the logs for any obvious errors.

Prerequisites for Installation

Before installing and using the Test Automation Package, ensure that the client computer satisfies the following prerequisites:

General

Agent Specific

Agent

Prerequisites

File Archiver for Windows Automation currently supports only executing tests on Local Instance only.
Oracle iDataAgent
  • For executing Oracle Automation test cases, the Oracle Client needs to be installed with Admin options on a Windows 32-bit machine.
  • Install oracle client with administrator privileges and create a file tsnames.ora at the following location:

    Oracle_home\network\admin

  • When you are using the test automation package for the Oracle iDataAgent on the client that has Oracle 10g database, the 10g cx_Oracle module will be installed automatically.
  • If you are using the Test Automation Package for the client that has Oracle 11g database. uninstall the 10g cx_Oracle module and install the 11g cx_Oracle module
Windows File System iDataAgent
  • Client computer needs to have multiple volumes to enable mount point creation for the tests.
  • To utilize data archiving capability, the Driver for File System Data Archiver needs have been to be installed.

 

Exchange Mailbox iDataAgent and Exchange Mailbox Archiver iDataAgent
  • The iDataAgent & Automation Package have to be installed directly on Exchange Server.
  • Automation Package should be installed on a path that doesn't contain any spaces.
  • Duplicate Exchange Databases shouldn’t be used for automation. The Database name given as input for automation should be unique in the Exchange Server.
  • Automation will fail if PowerShell SCOPE is not set correctly in Exchange Management Shell. Make sure Exchange Management Shell has the correct DOMAIN name set for the SCOPE.
  • In order to set the scope to correct domain, set a GLOBAL variable ‘AdminSessionADSettings.DefaultScope’ as follows:
    1. Go to Exchange server installation directory and open bin folder (like C:\Program Files\Microsoft\Exchange Server\bin)
    2. Find the file name “Exchange.ps1”
    3. Find the block (like ######## Exchange Variables ####### ) in that Exchange.ps1 file
    4. In that block if the variable $AdminSessionADSettings.DefaultScope exists then set the value of it to your machine’s domain name like $AdminSessionADSettings.DefaultScope = “medusa.loc”
    5. Be sure that the variable $AdminSessionADSettings.DefaultScope must occur after the occurrence of the variable $AdminSessionADSettings.ViewEntireForest(if this exist)
    6. If that variable $AdminSessionADSettings.DefaultScope does not exist then first find another variable called $AdminSessionADSettings.ViewEntireForest and delete it and then add the variable $AdminSessionADSettings.DefaultScope = <machine domain name>.

      For Example:  $AdminSessionADSettings.DefaultScope = “medusa.loc”)

    7. Reboot the machine after the above steps
    8. If problem still persists, make sure the DNS Entries are correct. If required, try removing all the DNS entries and have only PRIMARY DNS address
  • You must configure the Storage Policy manually and give as input to the test cases
  • Following registry keys should be created in the machine where CommServe is installed. You must restart the Services after creating these registry keys:

    nAllowHugeMessageSize, DWORD, under CommServe key, set to 1

    QCMD_SUPPORT_MODIFY_SUBCLIENT, DWORD, under EventManager, set to 1

  • Exchange Database iDataAgent installation is required on any instance for Test case 8720
  • Some test cases will try to create storage groups and databases. The Exchange Server should be able to accommodate them. The total number of storage groups present on the Exchange Server and total number of databases that are mounted should be less than the maximum limit permissible by the Exchange Server.
  • Ensure that the Active Directory Snap-In for PowerShell is installed on the machine for Test case 30099.
  • Set the limits for Exchange database where the subclient content mailboxes are created using the Exchange Management Console.  Once you set the limits, the generation of messages by automation script is not disrupted due to exceeding the warning limits.
Lotus Notes iDataAgent
  • Domino Admin client must be installed on a Windows client machine.
  • Domino Admin client should be connected to the Domino server.
  • User logged into Admin client should be the Domino Administrator.
  • User has to create a subclient, add content to it and specify the subclient name in answer file.
MySQL iDataAgent
  • MySQL Server should be configured with following user grant permission for both Windows / Linux Clients:

    GRANT ALL PRIVILEGES ON *.* TO '<dba_username>'@'<automation_controller_box_ip>' IDENTIFIED BY '<password>'

  • Since Instance (New MySQL Server) creation is not supported for MySQL iDataAgent from command line, it has to be configured first manually in GUI.
  • On the registry of Commserve computer, under the HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\<Instance>\EventManager section, create a DWORD type key QCMD_SUPPORT_MODIFY_SUBCLIENT with value set to 1.
SQL Server iDataAgent
  • The default subclient of the instance (s) being used must be configured to use a valid storage policy.
  • The date and time on the client computer must be in synchronized with CommServe computer.
Unix File System iDataAgent
  • The test automation package needs to be installed on a windows computer and all the UNIX AGENT test cases need to be executed from the windows computer.

Installing the Test Automation Package

1. Run Setup.exe from the Software Installation Disc (DVD 11) on a client computer that satisfies the prerequisites.  Select the desired Language option. Click Next.
2. Select to Install on the OS/Platform detected.
3. Select the option to 'Accept' the License Agreement. Click Next.
4. Select the option to 'Create a New Instance'. Click Next.
5. Expand Internal Use Tools | Automation Tools and select Test Automation.

Click Next.

NOTES:

  • Make sure to install Test Automation Package to a new instance of its own (NOT on Instance001).
  • Do not enter a CommServe Name if prompted. The Test Automation Package is installed in Decoupled mode and hence will not need the CS name.
6. If this computer and the CommServe is separated by a firewall, select the option 'Configure Firewall Services' and then click Next.  

For firewall options and configuration instructions, see Firewall Configuration and continue with the installation.

If firewall configuration is not required, click Next.

7. Do not enter a CommServe Name here.  Click Next.

Click Yes in the next message.

NOTES:

  • The Test Automation Package is installed in Decoupled mode and hence will not need the CS name.
 
   
8. Enter the "Client Name" and "Client Interface Name" here.  Click Next.
9. Enter the appropriate port numbers for "CVD" and "EVMGRC" here.  Click Next.
10. If "Windows Firewall" has been configured in the setup, select Add programs to the Windows Firewall Exclusion List, to add CommCell programs and services to the Windows Firewall Exclusion List.

Else, click Next.

11. Enter or browse to select the install destination folder and click Next.
12.
  • Click Browse.
  • Select the location to save the Test Automation Settings file (Answers File).
  • Click Next.

NOTES:
You can save the Settings file to a location different from Simpana Install location.

13. Click Next.
14. Click OK.


 

 

 

15. Click Reboot Now to reboot the client.

Post-Install Considerations

Default Settings

The Test Automation Package uses several default settings (examples below) in the file AutomationPackage\ScriptsLibrary\AutomationConstants.py file.  Before executing any Test cases, the values for the various constants used for test execution needs to be modified in this file to suit the test environment.

Creating An Answer File

Before you begin executing test cases, an Answer File needs to be created using the Test Automation interface. The following sections describe how to create answers file:

Creating aN Answer File

1. Run the TestAutomation.exe from <Automation Install Location>\AutomationPackage\IniHelper\<Win32 or WinX64>\  
2. Select File | New | Custom Test Set Answer File.
3.
  • Enter the name of the answer file in Answer File name.
  • Enter the e-mail address of the contact person in the Enter E-Mail IDs.
  • Enter the location to save the answer file or click Browse to select a location for the answer file.
  • Click OK.

 

4. Select Actions | Add Custom Test Set.
5.
  • Enter the name of the new test set in Enter New Test Set Name.
  • Click Next to proceed to the next step.
6.
  • Enter the host name of the test computer in Enter Host Name.
  • Enter the user name to login to the test computer in Enter User Name.
  • Enter the user password in Enter Password.
  • Enter the fully qualified domain name of the test computer in Enter Domain.
  • Enter the host name of the test computer in Enter Host Name.
  • Click Next to proceed to the next step.
7.
  • Select the desired Agent to test from Select DataAgent Type.
  • Click Next to proceed to the next step.
8.
  • The test cases applicable for the selected DataAgent is displayed in the left pane. Select the desired test cases. to execute and click on > to add it to list of Selected Test Cases.
  • Click Finish to complete to the next step.
  • Click OK.
9.
  • Select the Answer File that you have created and select Actions | Make Default option to make the newly created Answer File as the default answer file.

    Verify the registry to make sure that the answer file you have saved is the default answer file.

  • For the selected test cases, enter answers for all Test Cases within this Test Set on the right pane.
  • After answering all the Test cases in the test set, select File | Save.

 

Executing Test Cases

Pre-requisites Before Executing Test Cases

General

Agent Specific

Agent / Feature

Pre-requisites

File Archiver for Windows iDataAgent
  • Local Instance needs to be configured for the client.
  •  
  • An Archiverset must be configured to use a valid storage policy.
  • Ensure that on the windows computer, a value of ‘0’ is set for the registry key NtfsDisableLastAccessUpdate at location SYSTEM\CurrentControlSet\Control\FileSystem\
Oracle iDataAgent
  • From the computer where Oracle Test Automation is launching, ensure that there is no duplicate entry of the same connect string in the Tnsnames.ora.
  • Disable flashback Recovery area from the Oracle DB. If Flashback recovery is enabled, Rman may complain about particular archive logs being missing, in which case the user have to manually crosscheck archivelogs by connecting the Rman.
  • Ensure that the Oracle Service name and oracle connect string are the same.  If they are different, user has to manually update the Tnsnames.ora file on the controller from where Oracle Automation is being launched.
  • Ensure that Time zone on the Client computer and Commserver computer are the same.  Redirect and Partial restore test cases may fail if different.
Lotus Notes iDataAgent
  • You must create database inside Domino directory with at least one document in it.  qcreate support is not available for Lotus Notes iDataAgent for databases to be created automatically inside Domino directory.
Unix File System iDataAgent
  • A SMB/CIFS share needs to be created on UNIX Client Computer which can be accessible from the windows computer, where Automation package is installed and test cases will be executed. The data will be created in the SMB/CIFS share.
  • For executing Add-On Archiver Test cases on Unix File System iDataAgent, ensure that on the windows computer:
    • the DMFSWIN Driver for Data Archiver is installed.
    • A value of ‘0’ is set for the registry key NtfsDisableLastAccessUpdate at location SYSTEM\CurrentControlSet\Control\FileSystem\
Windows File System iDataAgent
  • The default sub client of the default Backupset must be configured to use a valid storage policy.
  • For executing Add-On Archiver Test cases on Windows File System iDataAgent, ensure that on the windows computer:
    • the DMFSWIN Driver for Data Archiver is installed.
    • A value of ‘0’ is set for the registry key NtfsDisableLastAccessUpdate at location SYSTEM\CurrentControlSet\Control\FileSystem\
  • To execute test cases verifying the archiving feature,  the archiving capability at the agent level must be turned on.
MS SQL iDataAgent
  • The default sub client of the instance (s) being used must be configured to use a valid storage policy
Media Management Feature
  • There are no prerequisites before executing these test cases.

Sybase iDataAgent
  • Ensure the checkbox 'Enable Auto Discovery' on the General tab of Sybase Server Properties dialog is checked if the Sybase Instance was created manually. Failure to ensure this may result in Automation jobs to fail with an error. If the Instance was created through Automation, the option is automatically checked.

 

Executing Test cases

Test cases can be executed using the Test Automation GUI by following the procedure listed below and the results of such execution can further be viewed using the Test Automation GUI.

 

1. Run the TestAutomation.exe with Administrative privileges from, <Automation Install Location>\AutomationPackage\IniHelper\<Win32 or WinX64>\

 

 
2. Select File | Open | browse your computer to the location of the Custom Test Set Answer File created.
  • Select the Answers File.
  • Click Open.

The selected Answer File is now shown in the Test Automation GUI

3. Highlight the selected Answers File.  Right click and select Run Test Sets option.
4. The test execution will now begin and will be shown in the Test Automation GUI. 

Once the test execution completes, the status will be displayed accordingly in the Test Automation GUI and additionally an alert e-mail (for the e-mail id configured during Custom Test creation) will also be generated containing the results of the test execution.

 

Viewing Results of Test Case execution

1. To view the results of Custom Test set currently being executed, right click at the Custom Test set level and select the option RUN HISTORY.

Alternatively, this report is also accessible via Actions | Show Test Report option.

2. From the dropdown, select the desired Test execution date/time and click Display Report button to view the results of selected execution in a new browser window.

 

Viewing Test Execution Logs

1. In order to help 'debug' any test case execution failure, a log file is automatically created each time a Test case is executed.  This log file so created contains all the action or events that occurred during the each Test case execution, including the possible failure reasons.

This log file, which will be of the format <testcaseid>.log (same name as the Test case executed) and will be created under a DATE STAMP folder at location; <Automation Install Location>\AutomationPackage\logs\<DATE STAMP>

 

 

 

 

 

Trouble shooting TIPS

General

  • Check if the answers file location is set correctly in the registry key 'INIFILE' at location; HKLM\SOFTWARE\CommVault Systems\<Instancexxx>\Automation\
  • Check the answers file to see if all the required 'answers' has been filled in correctly.
  • Confirm the values used for constants in AutomationConstants.py file correctly reflects various entities in the Test environment.
  • If there are issues with successfully completing the jobs, check the JPR, associated event messages and relevant Agent logs as well as Automation Test case logs, for any obvious errors.  For each instance of Test case execution, the Automation Test case log files are available at the location; <Automation Install Location>\AutomationPackage\Logs\<Execution Time Stamp folder>

 

Agent Specific

Agent

Test Cases

Exchange Mailbox iDataAgent and Exchange Mailbox Archiver
  • When test cases are run first time on a fresh set for up it has been noticed that the Exchange EXE sometimes crashes at the initial attempts. But after few retries this issue gets resolved on its own
  • Querying the properties in the mailboxes gets delayed. This is to ensure that the created and sent messages have all got delivered and all other expected changes to the message properties have got completed by the time we query the mailbox. By default this delay time is set to 90 seconds. To increase or decrease this time, change the value of ‘EXCHANGE_EXE_DELAY_TIME_BEFORE_QUERYING’ in AutomationConstants.py. Sometimes test cases may fail at validation part as the above time delay is not sufficient for the changes to take effect in your Exchange Server. Retry the test cases after increasing the time delay in such situations.
  • If CommCell Console or CommServe time is changed and if Simpana services are not restarted after this time change, the validation fails occur due to errors in clock skew time difference between CommServe and CommCell.
Lotus Notes iDataAgent Notespeek [Lotus Notes Utility] can be used to check if any document stub is causing restore failure of any particular document.

Lndbtool script [Simpana utility] be used to check for DBIID of a database and if transaction logs are backed up.