Troubleshooting Restore - Oracle iDataAgent

Backup Restore  

The following section provides information on troubleshooting restores.

Browse Failures

Point in time Table Browse Failures When you have encryption enabled for the client, point in time table browse operation fails with the following error message:

Pass-phrase protection is on for client [80], but pass-phrase was not specified.

Make sure that the pass phrase is exported to the MediaAgent when encryption is enabled for the client.

  1. From the CommCell Browser, right-click the client and select Properties.
  2. Click the Encryption tab.
  3. Click Via Pass-Phrase.
  4. Click Export.
  5. In the Destination Computer box, select the MediaAgent.
  6. In the Pass-Phrase box, type the pass-phrase used for encryption.
  7. In the Re-enter Pass-Phrase box, re-type the pass-phrase to confirm.
  8. Click Export.
  9. Click OK.

restore Failures

Table Restore Failures Make sure that the Oracle Services are running as Local System.
Database Restore Failures After performing an Oracle restore operation from the CommCell Console where options were selected for Redirect, Rename and Recover at the same time, you must click the Refresh button on the Subclient Properties (Content) tab or run a backup after the restore operation has completed before proceeding with another restore. This is necessary to ensure that the CommCell Console recognizes the changes that were made to the Oracle database and control file, so that it reflects the current structure of the database to be restored, otherwise the restore will fail.
Unable to create Duplicate Database
  • When you are creating a duplicate database or using an auxiliary instance for a table restore, make sure that one of the databases use sys as the connect string.
  • If a duplicate database restore fails with error PLS-00553: character set name is not recognized; then make sure that the  character sets are the same between the location from where you are running RMAN, and the location of the target database. As this is an Oracle related issue, please contact Oracle support for more information.
Increase in sbtio.log file size

Sometimes, jobs fail due to increase in the size of sbtio.log file in the $UDUMP directory.

To resolve this, set the size limit for thesbtio.log file using the sMAXORASBTIOLOGFILESIZE registry key. Once the specified size limit is reached, the sbtio.log file gets pruned automatically.

Table Level Restore Intermittent Failures The  table level restore operation may fail intermittently due to an error in the Oracle's DataPump utility and the following error message will be displayed:

UDE-00008: operation generated ORACLE error 31623

ORA-31623: a job is not attached to this session via the specified handle

In such cases, set the sNODATAPUMPEXPORT registry key to Y on the client and re-submit the job.

Control File Restores Failures Ensure that the DBID is assigned for the instance. Make sure that the DBID value for the database you are restoring is automatically displayed in Instance Properties.
Command Line Restore Failures Verify the availability of the required resource then rerun the RMAN command line operation
  Sometimes, the third party command line jobs may hang when you perform large backups and restores.
This happens since ClOraControlAgent updates the job manager for every 100MB data transfer and this causes the thread failure for large backups/ restores after transferring some of the data. The following exception will be seen in the clOraControlAgent.log:

5710030 304 02/22 03:47:23 608119 OraAgentBase::NotifyCommServeJobContinue() - m_jobObject->setUnCompBytesToAdd(105119744) ...
5710030 304 02/22 03:47:24 608119 CvThread::start_func() - Unhandled exception.
5710030 405 02/22 03:47:37 608119 ClOraControlAgent::OnClientTimeout() - Got timed out while waiting for msg from client 0
 

You can set sBYTESDIFFMBS registry key <value> in MBs in OracleAgent/.properties.
This will update the job manager at every <value> in MBs specified in the key.

Unable to create a Standby Database Standby database fails with the following error message:

temporary file TEMP01.DBF conflicts with file used by target database

Make sure that the Standby Role Initialization parameter, DB_FILE_NAME_CONVERT, is set to add all the temp datafiles from the primary database location to the standby database location, as follows:

DB_FILE_NAME_CONVERT='<primary_database_temp_datafile_old_location>','

<standby_database_temp_datafile_new_location>

restore error on linux client when switch database mode is enabled

When restoring Oracle database on Linux clients, if the Switch database mode for restore option is selected to keep database in correct mode during restore, the database may not restart after switching the database mode. Also, the restore operation may fail with the following error message.

RMAN Script execution failed with error [RMAN-04014: startup failed: ORA-27137: unable to allocate large pages to create a shared memory segment]. Please check the Logs for more details.

This issue occurs if the oracle user has a higher ulimit configuration than the root user. To resolve this issue, apply the ulimit value of Oracle user for the restore using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client>, and then click Properties.
  3. Click the Registry Key Settings tab.
  4. Click Add.
  5. In the Name box, type OracleUser.
  6. In the Location box, select or type OracleAgent from the list.
  7. In the Type box, select Value.
  8. In the Value box, type the Oracle user name (eg., oracle) and then click OK.
  9. Click OK.
  10. Restart Calypso Services on the client.

Recovering Data Associated with Deleted Clients and Storage Policies

The following procedure describes the steps involved in recovering data associated with the following entities:

Before you Begin

This procedure can be performed when the following are available:

Recovering Deleted Data

  1. Locate the latest Disaster Recovery Backup which contains the information on the entity (Storage Policy, Client, Agent, Backup Set or Instance) that you are trying to restore.
  2. On a standby computer, install the CommServe software. For more information on installing the CommServe, see CommServe Deployment.
  3. Restore the CommServe database using the CommServe Disaster Recovery Tool from the Disaster Recovery Backup described in Step 1. (See Restore a Disaster Recovery Backup for step-by-step instructions.)
  4. Verify and ensure that the Bull Calypso Client Event Manager Bull Calypso Communications Service (EvMgrS) is running.
  5. If you did not have a CommCell Migration license available in the CommServe when the disaster recovery backup was performed, apply the IP Address Change license and the CommCell Migration license on the standby CommServe. See Activate Licenses for step-by-step instructions.
  6. Export the data associated with the affected clients from the standby CommServe as described in Export Data from the Source CommCell.
    When you start the Command Line Interface to capture data, use the name of the standby CommServe in the -commcell argument.
  7. Import the exported data to the main CommServe as described in Import Data on the Destination CommCell.

    This will bring back the entity in the CommServe database and the entity will now be visible in the CommCell Browser. (Press F5 to refresh the CommCell Browser if the entity is not displayed after a successful merge.)

  8. If you have additional data that was backed up after the disaster recovery backup and before the deletion of the entity, use the procedure described in Import Metadata from a Tape or Optical Media to obtain the necessary information.
  9. You can now browse and restore the data from the appropriate entity.

    As a precaution, mark media (tape and optical media) associated with the source CommCell as READ ONLY before performing a data recovery operation in the destination CommCell.

Optimizing Memory Allocation for Table Restores

When restoring large tables, the restore operation may fail if there is insufficient memory allocation for creating the auxiliary instance.

Use the following steps to optimize the memory allocation for the auxiliary instance:

Allocating Memory for Auxiliary Instance

By default, 16MB pool size is allocated for the auxiliary instance. Use the following steps to increate this size limit:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client>, and then click Properties.
  3. Click the Registry Key Settings tab.
  4. Click Add.
  5. In the Name box, type sLARGEPOOLSIZE.
  6. In the Location box, select iDataAgent.
  7. In the Type box, select REG_SZ.

    On Unix Client, select Value.

  8. In the Value box, type <Value>.

    For example, 32M.

  9. Click OK.

Allocating Memory for Oracle Streams

By default, the system allocates 48 MB for the Oracle streams. You can modify this value using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click <Client> and then click Properties.
  3. Click the Registry Key Settings tab.
  4. Click Add.
  5. In the Name box, type sSTREAMSPOOLSIZE.
  6. In the Location box, select iDataAgent.
  7. In the Type box, select REG_SZ.

    On Unix Client, select Value.

  8. In the Value box, type <Value>.

    For example, 96M.

  9. Click OK.

Setting the UNDO Retention Period

Whenever a transaction is committed, the old undo information, is retained by default for a period of 1800 secs. You can modify this value, using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client>, and then click Properties.
  3. Click the Registry Key Settings tab.
  4. Click Add.
  5. In the Name box, type sUNDORETENTIONSIZE.
  6. In the Location box, select iDataAgent.
  7. In the Type box, select REG_SZ.

    On Unix Client, select Value.

  8. In the Value box, type <Value>.
  9. Click OK.

Viewing RMAN Errors

CommCell Console Errors

Point-in-Time Recovery

When you recover a database to a point in time, the RMAN command ALTER DATABASE OPEN RESETLOGS is executed which will reset the SCN (System Change Number) and time stamp on every object of the database (i.e., datafiles and control files). Also, only the archived redo logs that match the RESETLOGS SCN and timestamp value will be applied to the database, thus recovering the database to a time that is not current. This is a very useful operation if the point-in-time to which you are trying to recover is certain and known, but can be counterproductive if you are guessing at the point-in-time.

If you are not sure about the point-in-time for the recovery, it is recommended to restore the data and the control files to a point in time without recovery. This method will allow you to restore the database to a state that you can make the determination whether or not you have achieved the correct point-in-time, without invoking the"ALTER DATABASE OPEN RESETLOGS" statement that would reset SCNs and time stamps on the database objects.

After determining the correct point-in-time through this method, you can recover the database to the point in time to reset your Oracle database to the desired incarnation.

Sample scripts are provided below for your Oracle database administrator to use as reference for developing custom scripts that you can run from the RMAN command line, to perform special operations apart from the CommCell Console.

Sample Script for Resetting a Database After RESETLOGS

The following example resets a database after performing an incomplete media recovery:

run {
allocate channel dev1 type disk;
set until logseq 1234 thread 1;
restore database skip tablespace readonly;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

reset database;

Sample Script for Resetting the Database to an Old Incarnation

The following command makes an old incarnation of database PROD1 current again:

# obtain primary key of old incarnation

list incarnation of database prod1; List of Database Incarnations
DB Key
------
Inc Key
-------
DB Name
-------
DB ID
-----
CUR
---
Reset SCN
---------
Reset Time
----------
1 2 PROD1 1224038686 NO 1 02-JUL-98
1 582 PROD1 1224038686 YES 59727 10-JUL-98

shutdown immediate;

# reset database to old incarnation

reset database to incarnation 2;

# recover it
run {
allocate channel dev1 type disk;
restore controlfile;
startup mount;
restore database;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

Completed with One or More Errors

Restore jobs from Oracle iDataAgent will be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

Restore Completed with Warnings

Restore jobs from Oracle for Oracle iDataAgent will be displayed as "Completed with Warnings" in the Job History in the following case:

Oracle Errors

If you receive an Oracle error during an Oracle restore operation, we recommend that you follow procedures published by Oracle Corporation on resolving the specific error. We also advise you to consult with your on-site Oracle database administrator, as needed.