Getting Started | Advanced | Troubleshooting | FAQ | Support |
Spreadsheet for Calculating Retention
Log Retention Rules for Databases
Manage Retention of Individual Jobs
Override Pre-Defined Retention Rules for a Backup Job
Data Aging of Specific Data (Granular Data Aging)
Aging Data in Select Storage Policy Copies
Aging Data in Select Libraries
Data Aging in the CommServe Time Zone
Data Aging in the Storage Policy Time Zone
Data Aging in the Client Computer's Time Zone
Data Aging and Storage Policy Copies
Data Aging and Media Recycling
Data Aging and Deconfigured Clients
Data Aging of Disaster Recovery Data
Data Aging of Audit Trail Data
Data Aging of Job History Data
Data Aging for Deduplicated Data
Data Aging for SnapProtect Snapshots
Effects of Other Operations on Data Aging
Enable / Disable Data Aging Jobs in a Storage Policy
Suspend / Resume Data Aging Jobs
Enabling Restartability for Data Aging Jobs
Data Retention Forecast and Compliance Report
Understanding the data retention and aging process is important for managing storage in your CommCell.
Basic retention is defined as follows:
A retention cycle starts with a full (or synthetic full) backup followed by all subsequent incremental, differential, or transactional log backups that are dependent upon the full backup.
When retention cycles are defined, the number of cycles must be exceeded before the data is eligible for aging.
Retention days are the minimum number of days that all data protection jobs must be kept.
When retention days are defined, the number of days must be exceeded before the data is eligible for aging.
When a combination of days and cycles are used, the retention days must be exceeded for all jobs in the cycle before the data is eligible for aging.
The following example illustrates how days and cycles work together:
The primary copy of a storage policy is set to 2 cycles and 14 days.
All clients pointing to the storage policy has a schedule of one full backup every Monday, followed by an incremental backup on subsequent days of the week.
In the above example, the primary copy will always have a minimum of two weeks worth of data at any given point.
Note, however, that the actual aging will occur only when the cycle is completed in the third week.
Extended retention rules can be used in the following circumstances:
In all other cases, it is recommended that the Auxiliary Copy feature be used for extended storage as it actually creates another physical copy of the data, thereby reducing the risk of data loss due to media failure.
Extended retention allows you to retain a specific full (or synthetic full) backup for an additional period of time. For example, you may want to retain your monthly full backups for 90 days.
Extended retention rules allow you to define three additional "extended" retention periods for full (or synthetic full) backups. For example:
A backup job will be selected for extended retention based on its start time. For example: If a backup job starts at 11:55 pm on August 31st and ends at 1 am on September 1st, then it will be selected as the last full backup for the month of August and will be picked up for extended retention.
You can use a spreadsheet to calculate retention. Click here to download the spreadsheet, and then enter your data into spreadsheet.
Database Agents have unique retention rules and behaviors for logs. Refer to the Data Aging topic for your Agent for complete details on log retention rules.
You can extend the expiration date of a specific job. This will override the default retention set at the corresponding storage policy.
Note that if the data resides on removable media, the media will not be recycled until the jobs are manually aged.
You can delete a specific job if it no longer needs to be retained. This is useful to remove bad jobs that may impact your Auxiliary Copy operation. You can also use this option to remove unnecessary jobs that are preventing media from being reused.
Note that all dependent jobs will be automatically aged when a full job is selected for deletion.
You can override the pre-defined storage policy retention rules and apply new rules for specific data retention jobs. Follow the steps given below to manually specify the data retention time for a specific job.
Overriding the retention is a job based retention operation and does not affect any other jobs, it only affects the job that you are changing the retention. |
Several options are available for leveraging data aging jobs across a CommCell. These options reduce overhead on the CommServe computer and provide CommCell administrators with the flexibility to perform data aging operations for their own entities at times convenient for them. The following sections describe these options.
This option may be useful in the following scenarios:
This option may be useful in the following scenarios:
This option may be useful in the following scenarios:
Data aging operations age data according to the time zone of the associated storage policy.
You can designate the following time zones for a storage policy:
Exercise caution while changing time zones, especially when the time zone of a client is set to an earlier time. This can cause data to be aged earlier than intended. |
The CommServe time zone is the default for all storage policies. Use this option if you want your aging operations to be performed using the CommServe's time zone.
You can configure a storage policy for all clients within a specific time zone as follows:
If your storage policy is associated with clients in multiple time zones, you can set the time zone as follows:
Data aging behavior differs based on the types of copies that you are using. The following table describes the behavior for aging specific copy types.
Copy Name | Description |
Primary Copy without Secondary Copies | If data aging can be performed on a primary copy and there are no secondary copies defined, the data on the primary copy can be aged provided the data has exceeded its specified retention criteria. |
Primary Copy with Synchronous and Selective Copies | Data aging can be performed on a storage policy with synchronous and/or selective copies defined. Data is aged according to the primary copy retention rule only when all data eligible to be aged is copied to all active copies during an auxiliary copy operation. |
Secondary Copy | The data aging of a secondary copy is dependent on the selected retention criteria set for that copy. |
Incremental Storage Policy Copy | If data aging is performed on a storage policy that
has an incremental storage policy enabled, the data aging operation
counts backup cycles across both full and incremental storage policies.
Data on a full storage policy is aged based on the retention of the full
storage policy, and data on the incremental policy is aged based on the
retention rules of the incremental policy. If the incremental storage policy is also being used as a regular storage policy (and has full backups), the full backups will be also aged according to any basic and extended retention rules that are set. It is recommended that the retention rule for the full storage policy is greater than the incremental storage policy. Data on incremental policy will be aged earlier if it has shorter retention than the full storage policy. If the incremental storage policy has longer retention than a full storage policy, this may result in dangling incremental jobs. |
Source Copy that is not a Primary Copy | Data Aging can be performed on data copied during an
auxiliary copy operation from a non-primary copy source. The rules for data aging on source copies are as follows:
The following examples illustrate how data is aged from a storage policy that has three copies; primary copy Primary_01, Secondary_01, and Secondary_02. Secondary_02 uses Secondary_01 as a source copy. The retention rules for each copy are as follows:
|
Spool Copy | Allows you to use a primary copy as a temporary
holding area for protected data until it is copied to an active
synchronous copy. Once copied, the data on the primary copy is aged.
This copy has a retention rule of 0 days and 0 cycles, and hence, once an
auxiliary copy operation is performed, all data on this copy is aged when Data
Aging is run. Synchronous copies can not be deleted when a primary copy has the Spool Copy (no retention) option enabled, and there is no coverage for all of the subclients in the secondary copies. Instead, users are prompted with a warning message to change the retention period of the primary copy or to create another synchronous copy. The rule for data aging from spool copies is as follows:
|
Incremental and Differential Copies | Incremental and differential copies are retained
based on the rules defined for their associated storage policy copy.
However, you can configure data aging to age intermediate incremental
and differential copies preceding a new differential. This is useful to
minimize wasted disk space. Note that this configuration will also consider the retention days set on the storage policy copy for incremental or differential jobs. If the differential job is run, then incremental or differential jobs before the last differential in a cycle are pruned once the retention day criteria is met. For example, if your jobs ran in the following pattern:
Assuming that Do not retain intermediate incremental and differentials before a new differential option is selected and from the intermediate jobs between Full and Differential_02 once these jobs (Incremental_01, Incremental_02, Differential_01, Incremental_03, Incremental_04) meets the days retention criteria then these jobs will be aged, even if these jobs haven't met the cycle criteria. Use the following steps to configure data aging to age intermediate incremental and differential jobs: |
Data is only removed from media when all data on the media has been aged. As such, if half the data on a given media has been aged, that data will remain available for data recovery operations until the remaining data on the media is aged. Once all data on the media is aged, the media will then be recycled for future data protection operations.
Once a data aging operation deletes all data on tape media that has exceeded its retention rules, the media is returned to the associated scratch pool. Once the tape media is reused, the data originally written to it cannot be restored. Media that has an active status is not recycled back to the scratch pool until the media has a non-active status.
As the data aging operation only marks the metadata for deletion, the actual data in the media may be available for restores. You can view such aged data during browse and restores operations by enabling the following option.
You can save a spare media containing aged data and prevent the media from being overwritten, thereby allowing you to restore the data from the media at any time.
You can also save assigned media containing aged data.
If necessary, you can manually delete the contents of a media before the media's remaining data is aged. Use this option if you require the use of the media for critical data protection jobs and are certain you will no longer need any of the remaining data on that media.
Caution
Once deleted, the contents of the media are not available for data recovery operations, and the system will not automatically force the next data protection operation to be a full backup. If you delete the contents of the media, you should immediately run a full backup for all the subclients associated with the media once the operations is complete. |
Two disk capacity thresholds for managed disk space can be defined. They are:
The pre-defined thresholds for disk capacity for a disk library can be defined in the Mount Paths tab of the Library Properties (associated with a disk library) dialog box. |
When a client or an agent is deconfigured, the client's data is aged according to the corresponding storage policy copy’s retention time and cycle rules.
If necessary, you can ignore retention cycle rules for the deconfigured client. This option prevents the deconfigured client's data from being aged according to backup cycles that the client is no longer included in.
By default, the disaster recovery backup data is retained for 60 days and 60 cycles.
If you want to change the retention time for disaster recovery backup data, it is recommended that you keep the default setting as the minimum and configure extended retention rules defined as follows:
|
By default, CommCell operations tracked by Audit Trail are aged based on the following criteria:
These criteria can be changed as follows:
|
Job history data is aged according to the following criteria:
The following table illustrates how job history data is aged.
Job Type | Job Status | When It Is Aged |
All Jobs | Any | When the job's storage policy or storage policy copy is deleted |
Data Protection Job History/Disaster Recovery Backup Job History | Successful | With its associated data, which is aged based on the associated storage policy copy's defined retention rules |
Failed/Killed | 90 Days | |
Data Recovery Job History (including CDR Recovery operations) | Any | 90 Days |
Administration Job History | Any | 90 Days |
You
can change the default retention times
for job history data as follows:
|
Data Aging operations will automatically look up the deduplication store before data is deleted from the disk. Data Aging will only delete the source data when all the references to a given block is aged. So if you see older chunks in disk libraries remaining on the volume even if the original data is deleted, it might be due to the fact that deduplication reference(s) to the chunk is still valid.
If a deduplication store is offline, then that store will not be aged until all data on the store is eligible for aging.
Do not manually delete the Deduplication Store. The Deduplication Store facilitates the deduplication backup jobs and data aging jobs. If deleted, new deduplicated backup jobs cannot be performed and the existing data in the disk mount paths will never be aged.
The following procedures describe the available retention configurations for snapshots. For movement to media retention, refer to Data Aging - Getting Started.
By default, snapshots are pruned based on the number of retention days and cycles specified in the storage policy. You can configure your snapshot copy to retain a defined number of SnapProtect backup jobs. When the total number of jobs goes above the specified retention number, the remaining jobs will be pruned. This configuration is recommended for File System and File System like Agents. This feature is supported for SnapProtect operations performed using the NetApp storage array.
The NetApp Snap Management license is required for retaining snaps by number of jobs.
1. |
|
|
2. |
|
The following table describes the behavior for other operations on data aging.
operation | Description | Related Topics | ||
Erase Data |
Data that has been backed up or archived can be permanently erased. The
erased data will not be available for browsing and recovery. When data is marked for erasing, the data remains on the media until it is aged off according to the retention rules set for the data in the storage policy or storage policy copy. Data from Erase Backup/Archive operations can be aged as follows:
|
Refer to Erase Backed Up/Archived Data. | ||
Multiplexed Data |
Data protection operations can be run concurrently to the same media
using more than one data stream. This avoids having to back up data at
much slower than actual speeds of the tape.
Multiplexed data chunk sizes are determined by the type of data that is being multiplexed; file system data and database data.
Multiplexed data is aged when all jobs (multiplexed) on a single chunk have met the defined retention rules of their associated storage policy copy. To enable data multiplexing in a storage policy copy:
|
Refer to Data Multiplexing. | ||
Silo Storage | When an active Silo store has been sealed and moved to storage, all the backup jobs that went to that store must meet the retention rules (defined in their associated storage policy copy) for the store to become aged. Once all of the jobs have met their retention criteria, the entire store is considered ageable, and the Silo (tape) backup jobs are then aged. The tape designated for the Silo storage is then refreshed and available for re-use. Extended retention rules are not supported on Storage Policy Copies configured for Silo Storage. | Refer to Silo Storage. | ||
Cloud Deduplication Pruning | If you have cloud storage configured with deduplication, the pruning of the data will not be done until the store is sealed and all the backup jobs associated to that store meets the retention rules for the store to become aged. | |||
Disabled Jobs | If data aging is performed on a storage policy copy that has disabled jobs, these jobs are aged differently. If the disabled job is a full backup job, the entire cycle is marked as disabled. In this case, data aging does not count the disabled full backup as a valid cycle. If the disabled job is an incremental or differential backup and the full backup job is not disabled, the cycle is counted as a valid cycle. | Refer to Storage Policy Copy Operations. |
1. | From the CommCell Console, right click the CommServe icon and click All Tasks | Data Aging. | |
2. |
|
|
3. | Select the appropriate scheduling options and click OK. | |
4. | Data aging operation will execute according to the specified schedule. You can track the progress of the job from the Job Controller window. When the job has completed, the Job Controller displays Completed. |
|
1. | From the CommCell Browser, navigate to Storage Resources | Libraries, right-click the Media By Groups | New Media Group | Spare Media Group. | |
2. |
|
|
3. | Right-click the client computer and click Properties. | |
4. |
|
|
5. | Click Tools | Control Panel. | |
6. | Click Job Management. | |
7. | Select Queue Jobs if activity is disabled. | |
8. | If any jobs are run after the Jobs of activity is disabled, you can see in the Job Controller that the jobs status Queued. | |
9. | Re-point all the client's subclients to the No_Backups storage policy. When a user changes the storage policy association of a subclient, retention cycles are set to zero (0), therefore, only the retention days must be exceeded for data to be aged. In this case, the data in the storage policy to which the client's subclients were originally pointing to will age accordingly. Note that if the client computer's activity is re-enabled, you can point the subclients back to their original storage policy |
1. | From the CommCell Browser, right-click the client computer and click Properties. | |
2. |
|
|
If this report displays data you do not intend to age, you must fine-tune your rules accordingly. Once you run a data aging job, the unintended data will be lost.
1. |
From the CommCell Browser, click the Reports icon. |
|
2. | Expand Reports and select Data Retention Forecast and Compliance. | |
3. | Click Run. | |
4. | The report will display the data to be aged when the data aging job is run. |