Troubleshooting - SAP for Oracle Backup

Backup Restore  

The following section provides information on some of the troubleshooting scenarios related to backup:

Backup operation fails if the index cannot be recovered

Symptom

Backup operation fails with an error message, if the index file present cannot be recovered.

Resolution

As a workaround to this issue, create the key CreateNewIndexForSapOracle before running a backup at the location mentioned below:

Now run the backup operation. After the backup operation is complete, you might need to delete the key CreateNewIndexForSapOracle from its location (if it did not delete automatically).

Backup Failure

From CommCell Console due to Invalid User Permissions Make sure to use the following user accounts when creating SAP for Oracle instances:

For Unix clients:

<SID_name>adm

For Windows clients:

<client_name>/<SID_name>adm
From CommCell Console when data and log storage policies use different Media Agents During backups, if the data and log storage policies use different Media Agents, the backup fails with the following error:

UPDATEINDEX:Failed to ReOpen IndexCacheDir

To avoid this, ensure that the data and log storage polices use the same Media Agent.

If you want to use storage policies with different Media Agents, then it is recommended to create separate subclients for data and log backups.
On AIX when the CIO option is turned on Make sure to mount the data volumes of JFS2 with "mount -o cio" option.

Online backup of oracle database not working with JFS2 and Oracle 10g:

As per SAP Note- 948294, Starting with Oracle 10g, the advanced I/O capabilities of JFS2 are used by default from the Oracle database server. If file system type JFS2 is detected by the database server, all Oracle data files residing on this file system are accessed using the CIO (Concurrent IO) option. CIO bypasses the file system cache from the AIX operating system, which results in better IO performance for the Oracle database. If a data file is opened in CIO mode by Oracle, all access to this data file via the regular file system cache is prohibited by the AIX OS. Tools like cp, dd, cpio, tar, dbv etc., do not open Oracle data files with the CIO option. Therefore these tools receive an error from the AIX operating system when trying to access the Oracle data files if the database is open.

You need to mount all JFS2 file systems containing Oracle data files (sap data directories) with the mount -o cio option. This enforces a bypass of the file systen cache for all programs.

Oracle redo log and control files are part of the Oracle data files. If these files are on JFS2 also (origlogA, mirrlogA, origlogB, mirrlogB), use the mount -o cio option accordingly.

Note that the access to the Oracle data files by all other programs which do not support CIO directly will probably be significantly slowed down. This is because caching, read ahead etc. is no longer possible.

To lessen performance penalty during database backup use the dd command with a big blocksize parameter.

Example from file initSID.sap:

tape_copy_cmd = dd

disk_copy_cmd = dd

dd_flags = "bs=1024k"

dd_in_flags = "bs=1024k"

compress_cmd = "dd bs=1024k if=$ | ( compress -c > $ ) 2>&1"

For restore:

uncompress_cmd = "uncompress -c $ | dd bs=1024k of=$ 2>&1"

For verify:

uncompress_cmd = "uncompress -c $ > $"

For "compress-only" (option "-k only"):

compress_cmd = "dd bs=1024k if=$ | compress -c > $

As an alternative, use a backup utility (BACKINT) which supports CIO.

Another alternative is the use of Oracle RMAN for online backup. The use of Oracle RMAN can be controlled by the BR*Tools.

With Oracle RMAN as backup utility, the mount -option -cio for JFS2 is not required when mounting file systems holding Oracle data files.

Do not mount JFS2 file systems holding $ORACLE_HOME, logfiles etc. with the -o cio option! Header Data
Shared Memory Error Issue:

The backup failed because the shared memory on the HP-UX PA-RISC client has not been configured per operational guidelines.

Resolution:

Add the DisableIPC_GLOBAL file in the /apps/simpana/Base directory on the client where the backup failed.

  1. Stop the Calypso software.
  2. Create an empty file called DisableIPC_GLOBAL in the /apps/simpana/Base directory. From the command line, enter the following:

    touch /apps/simpana/Base/DisableIPC_Global

  3. Restart the Calypso software.

Command Line Errors

Prior to running backups from the RMAN command line for SAP for Oracle iDataAgent, ensure that the SBT_LIBRARY path and environment variables for CvClientName and CvInstanceName are defined in the RMAN script.

For example, on a Solaris client, provide the path as given below:

util_par_file = ?/dbs/init@.utl rman_parms="BLKSIZE=1048576,SBT_LIBRARY=/opt/calypso/Base64/libobk.so,ENV=(CvClientName=sunsign,CvInstanceName=Instance001)" rman_channels=1

where Cvclientname and CvInstancename are the names of the Client and Instance (e.g., Instance001) on which the iDataAgent is installed.

The SBT_LIBRARY for the various platforms are listed below:

Platform SBT_LIBRARY
AIX with 64 bit Oracle <Client Agent Install Path>/Base/libobk.a(shr.o)
HP UX PA RISC 64 bit Oracle, <Client Agent Install Path>/Base64/libobk.sl
Solaris with 64 bit Oracle <Client Agent Install Path>/Base64/libobk.so
Linux on System Z with 64 bit Oracle <Client Agent Install Path>/Base64/libobk.so
All Other Unix platforms <Client Agent Install Path>/Base/libobk.so
SBT_LIBRARY is not applicable for Windows platform.

Pending State for SAP for Oracle Backups

You may receive an error message indicating that the SAP for Oracle backup has gone to a pending state after a given percentage of the job has completed. This error has occurred because the SAPSECULIB (libsapsecu.so) library cannot be found or the permissions are not correct. To correct the problem, complete the following steps:

  1. Ensure that the permissions for the br* executables are correct.
  2. Ensure that the SHLIB_PATH and DIR_LIBRARY environment variables are set correctly.
  3. On Unix systems, ensure that the DIR_LIBRARY environment variable is set on the directory that contains the libsapsecu library; for example:

    DIR_LIBRARY = /usr/sap/<SID>/SYS/exe/run

    This is especially important if you use an external scheduler (for example, cron, at, or BACKINT scheduler) to start the BR*Tools. Also, ensure that DIR_LIBRARY is set for both user ora<sid> and <sid>adm.

  4. Ensure that the libsapsecu.so library is accessible.
  5. Ensure that the SAPSYSTEMNAME environment variable is set correctly.
  6. Ensure that the contents of the database (i.e., SAP tables and SAP owner) are accessible. This is especially important if you use an external scheduler (for example, cron, at, or BACKINT scheduler) to start BR*Tools. In such a case, the SAPSYSTEMNAME variable must be set in the environment of this scheduler (for example, in an environment file); in this case, it must be set within the backup profile of Calypso (be sure to consult your backup vendor).
  7. You can verify this setup by appending "-TRC11" to the brbackup command line that is called by Calypso.  The created trace file will contain all the active environment variables at runtime of the brbackup command.

SAP for Oracle RMAN Backup Failure

When using RMAN utility on Solaris client Ensure that you set the following parameter on the client computer:

crle -64 -c /var/ld/64/ld.config -l/opt/calypso/Base64:/lib/64:/usr/lib/64
When using RMAN_UTIL after the Logs have been Reset Use the util_file interface instead to backup all the Archive logs after the logs have been reset. However, subsequent log backups can be performed using the RMAN_util interface.

Oracle Errors

If you receive an Oracle error during a SAP for Oracle backup operation, we recommend that you follow procedures published by Oracle Corporation on resolving the specific error. We also advise you to consult with your on-site Oracle database administrator, as needed.

Completed with One or More Errors

Backup jobs from SAP for Oracle iDataAgent will be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

Backups failing with Intermediate Index Cache selected in MediaAgent Properties

The following error may occur if Enable Intermediate Index Cache option is selected at the MediaAgent Properties level while running a backup using SAP for Oracle iDataAgent:

5584 1584 04/19 13:45:43 3618 UPDATEINDEX: Failed to ReOpen IndexCacheDir. ErrCode=0
5584 1584 04/19 13:45:43 3618 UPDATEINDEX: Failed to ReOpen IndexCacheDir. ErrCode=1

Disable the Intermediate Index Cache option in the MediaAgent Properties window Catalog tab.