To test the scan phase of a backup:Topics | How To | Related Topics
Using Wildcards to Define Content
Using Wildcards to Define Filters
Inclusions, Exclusions, and Exceptions to Exclusions
Data protection content is defined at the subclient level for File System iDataAgents. The default subclient, created during installation, has the unique characteristic of including all protected data not explicitly covered by other subclients within the backup set. Additional subclients can be created by the administrator to separate and manage a subset of the backup set data.
The content of each subclient is mutually exclusive within the backup set. This means the content, as defined by one subclient, will not be protected by another subclient.
In the case of application subclients, the inclusive feature of the default subclient means it can automatically "discover" all data requiring its protection. In some iDataAgents, this automatic discovery feature can be disabled, filtered (for example: Lotus Notes), or otherwise configured to assign content to other subclients (for example: Exchange mailboxes and GroupWise databases).
In File System iDataAgents, the default subclient content is defined as "\" or "/" which is the symbol for the root of all file systems. In this case, "all file systems" means:
In most cases, manually editing the content of the default subclient will disable its inclusive discover feature. Adding user-defined content to a subclient should be done in subclients other than the default subclient. See Configure Subclient Content for step-by-step instructions.
Regular expressions (or wildcard characters) can be used to define content in user-defined subclients. Wildcards expressions are characters such as * or ?. Regular expressions include patterns such as [a-f] or *.[l-n]df. The terms are interchangeable.
The use of regular expressions in defining content is limited to the last level in the data path. For example: F:\Users\[A-L]* is an acceptable use of regular expressions in defining content, whereas *.pst is not an acceptable use.
For File System iDataAgents, the use of regular expressions requires the explicit enabling of the subclient option to Treat characters as regular expressions. This is required for some operating systems (for example: MAC OS), because certain characters such as * or ? can be part of a legitimate file path or name. Keep in mind that this option does not appear in the default subclient since use of regular expressions in the default subclient is not supported.
| In order to operate wild cards correctly, the default subclient with the default content must exist within the backup set. |
Filtering unnecessary data from data protection operations can reduce backup/migration time, storage space, and recovery time. Most, but not all iDataAgents include some filtering capability at the subclient level.
File systems' subclient filters can be defined for a path, directory, or file level.
Some application iDataAgents such as Exchange Mailbox, Lotus Notes Database, and Lotus Notes Document also allow filters to be defined.
There are four basic types of filters that can be used:
|
Wildcards are not supported in Exception filters. |
|
Job Results files cannot be filtered from backup operations. |
Once data protection filters have been defined, entries can be added or deleted from the Subclient Properties (Filters) tab. In addition, some agents provide an editing capability that allows modification to existing filters without having to delete and re-add them. See Edit a Data Protection Filter for a Subclient for step-by-step instructions.
The use of wildcards or regular expressions in defining filters is supported for most iDataAgents, but support for specific wildcards can vary. See Supported Wildcards for more information.
| Exception filters that include wildcards and that point to folders or files below an excluded folder are not supported. |
Keep in mind that vendors of NAS file systems implement filtering differently. For example, BlueArc and Network Appliance systems support only the "*" character either before or after the character string (e.g., *tmp or cache*); EMC Celerra does not currently support filtering.
Subclient content serves as an include filter that allows you to specify the portions of client data that you want included in data protection operations. The Exclude Filter is designed to tell the system which subset of the specified content should not be included in data protection operations. The following example illustrates these concepts:
Include= /space
Exclude= /space/temp
The above example shows that the volume called "space" will be included in backups for the subclient, excluding the "temp" directory. Suppose there were a subdirectory of "temp" called "keep" that you wanted to include, you could set up an Exception to the Exclusion Filter, as shown in the next example:
Include= /spaceThis would result in the volume "space" being included in backups, excluding the "temp" directory, except for the "temp/keep" subdirectory.
Exceptions to exclusions can be thought of as patterns that should overlap (or match the same paths) as exclusions. This is useful for protecting data that would otherwise be filtered out by the Exclude filter. The Exception filter is not intended for use as a stand-alone include filter--the objects in the Exception filter must be a subset of objects in the Exclusion Filter.
Regular expressions (or wildcards) offer additional flexibility in defining the content to be included, excluded and/or excepted. The next example illustrates how wildcards may be used in filtering:
Include= /spaceThis example defines the content to be included as the "space" volume, excluding all subdirectories of "space" that begin with the letters "tem", except for the "temp/keep" subdirectory. Using wildcards in this way would exclude directories such as "space/temp", "space/templates", and "space/temporary_internet_files" from data protection operations. The Exception filter provides an exception to the exclusion of all subdirectories beginning with "tem", by allowing the "temp/keep" directory to be protected.
A more advanced example follows:
Include= C:\ and D:\ and E:\Drives C, D and E are the content to be included, excluding all directories that begin with "temp" from any level on any of those drives, except for the "templates" directory on drive C. This is just one example of the many possible combinations of wildcards that may be used in filtering. Refer to Supported Wildcards list of supported wildcards, along with examples for each.
Keep in mind the following rules with regard to filtering:
See Add/Edit a Data Protection Filter for a Subclient for step-by-step instructions.
Refer to the following for a list of wildcards supported for the respective agent type:
Select the desired question:
A good backup filter is essential in achieving your backup and restore objectives and in keeping your protected storage usage to a minimum. A backup filter prevents unnecessary files from being backed up on a client. These are generally temporary files (*.tmp, Cache or Temp directories, etc.) or files that can be recovered from another source (CD, compiling, etc.) that are not essential to system recovery, applications, or to users.
Filters should be used in situations where there is client data that doesn't need to be preserved and can be excluded from data protection and recovery operations. This will increase performance by reducing data protection and recovery time, as well as minimize the expense of storing unnecessary data.
Yes. In Windows and NetWare the swap files (pagefile.sys and _swap_mem) are automatically filtered from backups. In Unix, the /proc folder is automatically filtered. For NAS NDMP subclients with content defined at the root level, any snapshot directories are automatically excluded. In all operating systems, any mounted CD/DVD file system is automatically filtered.
Use this procedure to test the scan phase of a backup job, applicable for all file system and file system-like agents. Testing the scan phase allows you to determine which files/objects would be backed up, the number of objects, time taken for scan, etc., based on your backup options and filters.
To test the scan phase of a backup:
Restore filters are available for some agents which allow you to exclude objects that are not needed to restore the operating system or application. However, before applying these filters, we recommend that you consult with your operating system/application vendors to determine which objects do not need to be restored in order to fully recover the system or application.
Use this procedure to test a restore job without writing to media, applicable for Windows, Unix and Macintosh File System iDataAgents. Testing a restore job allows you to get a list of all files that would be restored without actually restoring them, based on your restore selections and filters.
To test a restore job:
"c:\test",""
Note that a Windows-based path is used in the above example. The path portion of the syntax will vary by operating system.
Using regular expressions (or wildcards) to define content and/or filters can be helpful in situations where a pattern of objects can be included or excluded from data protection operations, without having to specify every single object that falls into that pattern. For example, rather than typing in all file names with an extension of type "doc", you can type in *.doc.
In the exclude field you can use the following wildcard characters:
* Substitute for any number of characters (for example: *.doc)
? Substitute for any single character (for example: *.?df)
[ ] Any range of characters (for example: [a-m]*.doc)
[! ] or [^] Negates the range of characters (for example: [!ei]nsure.doc or [^fg]oo.out)
** Any directory level (for example: C:\**\temp)
Regular expression characters can be treated as literal characters by "escaping" the interpretation of a regular expression. For example: to filter out a directory named C:\[MyFolder] you would need to escape the first bracket to allow its literal interpretation. Otherwise, you would be filtering the directories C:\M, C:\y, C:\F, C:\o, etc.
The Escape character is '/' for Windows-style paths ("opposite" path delimiter of '\'), and '\' everywhere else. Hence to filter out C:\[MyFolder] use C:\/[MyFolder] (note that the \/ is a backslash forward-slash, not a capital V). On Unix, use /\[MyDirectory] to filter out /[MyDirectory].
There should be no noticeable difference in performance for filters that use regular expressions, literal paths, or any combination thereof.
Some system state files can be filtered out of backups and restores. For more information, refer to the System State page.
Filters can be used in conjunction with the "Items That Failed" list on the data protection Job History Report to eliminate backup or archive failures by excluding items which consistently fail that are not integral to the operation of the system or applications. Some items fail because they are locked by the operating system or application and cannot be opened at the time of the data protection operation. This often occurs with certain system-related files and database application files.
Also, keep in mind that you will need to run a full backup after adding failed files to the filter in order to remove them.
One of the best ways to determine which files to filter is to review the backup file list for each subclient. When you perform your first full backup of a subclient, examine the backup file list to verify your intended content is there. If you don't see the files you need, re-define the content. If you see files you don't need, then filter them out.
After your first incremental backup, examine the backup file list again. Are there any more files you don't need? If so, filter them out. Continue to examine your incremental backup file list after the next few backups until you are satisfied that your filters excluding the specified data and you are capturing only the data you want protected.
Another filtering tool is the failed file list. It should always be reviewed for possible changes to your filters. Optimally, if you implement your filters correctly, you should not have any failed files or directories in a backup. Failed files are often transitional or temporary files that are not required for restore. However, they can include production files that are locked open by a user or application. You can back up open files using software utilities such as Microsoft's Volume Shadow Service, QSnap, or St. Bernard's Open File Manager.
An entire directory can be filtered out of data protection operations if you do not wish to preserve any of the data in that directory or in any of its subdirectories. Temporary working directories often fall into this category.
At the end of this document there is a list of recommended filters. Keep in mind that all systems have differences. The recommendations listed there should be evaluated on a system by system basis to ensure that necessary files are not inadvertently excluded. One method to ensure this is to perform a full system restore test of the system.
The CommCell Console does not support the group addition of Global Filters. However, if you are comfortable with Microsoft SQL Server you could add them by editing the CommServ.dbo.GXGlobalParam table directly.
WARNING: Direct modification of the metadata is not supported. Before you attempt any modification of the Commserv database, perform a Disaster Recovery backup and ensure that it can be recovered successfully.
The CommServ.dbo.GXGlobalParam table has two fields: name and value. The Windows global filters are defined by the name "Windows FS Exclude Filters". Note that the name is not present unless a global filter entry has been previously created. Before continuing, add a global filter entry and click OK. The procedure below will overwrite the entry.
To add a group of global filters you can run an SQL statement like "UPDATE GXGlobalParam set value=<filter list> WHERE name='Windows FS Exclude Filters'.
The <filter list> is a string of filter entries separated by a space (for example: C:\Temp D:\Tmp E:\Junk). If the filter entry itself contains a space (for example: C:\Program Files\*) then those spaces are replaced by "+1" (for example: C:\Program+1Files\*).
You could use the following steps to construct a suitable string of filters:
Here's a sample of the SQL script:
UPDATE GXGlobalParam
set value = '*:\**\ntuser.dat* *:\**\usrclass.dat* *:\**\temp *:\**\tmp
*:\**\temporary+1internet+1files *:\**\*cache
*:\**\documents+1and+1settings\**\recent
*:\**\documents+1and+1settings\**\history *:\**\drvspace.* *:\**\recycled
*:\**\recycler *:\system+1volume+1information
*:\windows\downloaded+1program+1files *:\win*\hwinfo.dat
*:\win*\prefetch
*:\win*\system.da0 *:\win*\user.da0 *:\win*\history *:\win*\historique
*:\win*\$NtServicePackUninstall$ *:\win*\$NtUninstall* *:\win*\applog *:\win*\inf
*:\win*\installer *:\win*\pchealth *:\win*\servicePackFiles *:\win*\sysbckup
*:\win*\**\spool\**\*.spl *:\**\ws_ftp.log *:\**\appdata\**\microsoft\**\recent
*:\**\application+1data\microsoft\**\recent *:\**\recent\*.lnk
*:\**\windows+1update.log *:\**\windows+1update+1setup+1files
*:\program+1files\**\uninstall+1information
*:\program+1files\windowsupdate'
WHERE name = 'Windows FS Exclude Filters'
To filter all directories that begin with a particular word on all drives (e.g. My_Filter), use the global filter:
*:\**\My_Filter*\**
Or use the subclient filter as follows if you know the drive letter to the filter:
C:\**\My_Filter*\**
To filter every directory named temp on all drives, use the global filter:
*:\**\temp\**
Or use the subclient filter:
C:\**\temp\**
To filter every recycler directory on all drives, use the global filter:
*:\**\RECYCLER\**
Or use the subclient filter:
C:\**\RECYCLER\**
| Do not use "temp*" when all you need to filter is temp, as it will get "template" and other directories with the word "temp". |
The filters listed below, along with the specified file location, are the default filters recommended by the various software vendors. File extensions used are usually unique to a software application, however, this is not guaranteed. As such, there can be conflict with other applications or your file naming policy and retention requirements. Before applying any filter, make sure the path and scope of the filter is correct. After applying any filters, a thorough review of backup content is recommended to confirm that essential files are not being excluded from protection. Keep in mind that this list is not all-inclusive, nor necessarily up-to-date. We recommend that you check with your recovery administrator and operating system/application vendor for the most current information.
Select the desired topic:
*:\**\*.shs
*:\**\*.tmp
*:\**\386spart.par
*:\**\cache\*
*:\**\content.ie5\*
*:\**\documents and settings\**\*cache\*
*:\**\dsmerror.log
*:\**\dsmsched.log
*:\**\ea data. sf
*:\**\memory.dmp
*:\**\ntds\*
*:\**\pinedbg?.txt
*:\**\pinedebg.txt
*:\**\profiles\**\ntuser.dat*
*:\**\system32\perflib*.dat
*:\**\temp\*
*:\**\temporary internet files\*
*:\**\thumbs.db
*:\**\tmp\*
*:\**\win386.swp
*:\**\~*
*:\documents and settings\**\ntuser.dat
*:\documents and settings\**\ntuser.dat.log
*:\documents and settings\**\usrclass.dat
*:\documents and settings\**\usrclass.dat.log
*:\drivers\*
*:\drvspace.*
*:\recycled\*
*:\recycler\*
*:\system volume information\*
*:\win*\downloaded program files\*
*:\win*\hwinfo.dat
*:\win*\prefetch\*
*:\win*\system.da0
*:\win*\user.da0
*:\Inetpub\Mailroot\*
*:\Config.msi\*
*:\Win*\system32\Repl\Import\*
*:\Win*\system32\dhcp\backup\jet\*
Optional: Backup file types
*:\**\*.abkOptional: Disk Image file types
*:\**\*.bkf
*:\**\*.gho
*:\**\*.ghs
*:\**\*.iso
*:\**\*.pqi
*:\**\*.qic
*:\**\*.v2i
Optional: History folders and file types
*:\**\appdata\**\microsoft\**\recent
*:\**\application data\microsoft\**\recent\*
*:\**\application data\microsoft\**\r?cents\*
*:\**\cookies
*:\**\downloads.rdf
*:\**\history.ie5\*
*:\**\mozilla\**\cookies.txt
*:\**\mozilla\**\history.dat
*:\**\netscape\**\cookies.txt
*:\**\netscape.hst
*:\**\real\**\cookies.txt
*:\**\realplayer\history\*
*:\**\recent\*.lnk
*:\**\ws_ftp.log
*:\**\documents and settings\*\recent\*
*:\**\documents and settings\**\history\*
*:\program files\**\blackICE\hosts\*
*:\win*\history\*
*:\win*\historique\*
Optional: Software Update files and folders
*:\**\*.isu
*:\**\installShield Installation Information\*
*:\**\symantec\liveupdate\*
*:\**\windows update.log
*:\**\windows update setup files\*
*:\program files\**\uninstall information\*
*:\program files\windowsupdate\*
*:\win*\$NtServicePackUninstall$\*
*:\win*\$NtUninstall*\*
*:\win*\applog\*
*:\win*\inf\*
*:\win*\installer\*
*:\win*\pchealth\*
*:\win*\servicePackFiles\*
*:\win*\sysbckup\*
Optional: Spool files and folders
*:\win*\**\spool\**\*.spl
Optional: Trash files and folders
*:\**\eudora\trash.*
*:\**\imapmail\**\trash
*:\**\imapmail\**\trash.*
*:\**\mail\**\trash
*:\**\mail\**\trash.*
*:\**\netscape\**\cache.trash\*
*:\**\netscape\**\mail\trash.sbd\*
*:\**\network trash folder\*
*:\**\networkTrashFolder\*
CommCell files and directories
*:\**\CommVault Systems or
*:\**\HDS, or
*:\**\Bull StoreWay Calypso
*:\**\JobResults\*
*:\**\IndexCache\*
*:\**\CV_MAGNETIC\*
/**/.java/**/cache/*
/**/.kde*/**/cache/*
/**/.mcop/trader-cache/*
/**/.mozilla/**/*Cache*
/**/.netscape-cache/*
/**/.netscape/cache/*
/**/.netscape/xover-cache/*
/**/.opera/cache*/*
/**/.phoenix/**/Cache/*
/**/.pine-debug*
/**/.sylpheed/imapcache
/**/.sylpheed_cache
/**/Temp
/**/core
/**/dsm_error.log
/**/dsm_sched.log
/**/dsmerror.log
/**/dsmsched.log
/**/mozilla/**/*Cache/*
/**/netscape/**/Cache/*
/**/netscape/**/cache/*
/**/netscape/Cache/*
/**/temp/*
/**/tmp/*
/**/tmp/**/*
/cache/*
/etc/adsm.err
/etc/adsm.log
/var/cache/*
Optional: Backup file types
/**/.mozilla/**/prefs.bak
/**/.netscape/plugin-list.BAK
/**/.opera/opera*.adr.bak
/**/.phoenix/**/prefs.bak
/**/.sylpheed/folderlist.xml.bak
/**/.sylpheed/sylpheed.log.bak
/**/.sylpheed/sylpheedrc.bak
/**/*.dat.bak
Optional: Disk Image file type
/**/*.bkf
/**/*.gho
/**/*.ghs
/**/*.iso
/**/*.pqi
/**/*.qic
/**/*.v2i
Optional: History directories and file types
/**/.kde*/share/apps/kcookiejar/cookies/*
/**/.kde*/share/apps/RecentDocuments/*
/**/.mozilla/**/cookies.txt
/**/.mozilla/**/history.dat
/**/.mozilla/**/downloads.rdf
/**/.netscape/cookies
/**/.netscape/history.dat
/**/.netscape/history.db
/**/.netscape/history.list
/**/.opera/cookies*.dat
/**/.opera/download.dat
/**/.phoenix/**/cookies.txt
/**/.phoenix/**/downloads.rdf
/**/.phoenix/**/history.dat
/**/mozilla/**/cookies.txt
/**/ws_ftp.log
Optional category: Spool directories and file types
/spool/mqueue*/*
/var/spool/lpd/*
/var/spool/mqueue*/*
/var/spool/print/*
CommCell files and directories
/**/galaxy
/**/JobResults/*
/**/IndexCache/*
/**/CV_MAGNETIC/*
*:\**\*.tmp
*:\**\browser\cache\*
*:\**\cache\*
*:\**\cookies.txt
*:\**\dependency cache\*
*:\**\dsmerror.log
*:\**\dsmsched.log
*:\**\mozilla\**\*Cache\*
*:\**\netscape\**\cache\*
*:\**\nsuser\cache\*
*:\**\package cache\*
*:\**\pinedbg?.txt
*:\**\pinedebg.txt
*:\**\symantec Shared\VirusDefs\*
*:\**\temp\*
*:\**\temporary Internet Files\*
*:\**\thumbs.db
sys:\system\btrieve.trn
sys:\system\cmaster.dba
sys:\system\events.log
sys:\system\net$obj.sys
sys:\system\net$prop.sys
sys:\system\net$val.sys
sys:\system\secaudit.log
sys:\system\sys$log.err
sys:\system\system.log
sys:\system\tsa\tsa$temp.*
sys:\tts$log.err
sys:\vol$log.err
Optional: Backup file types
*:\**\*.bk!
*:\**\mozilla\**\prefs.bak
*:\**\netscape\prefs.bak
*:\**\netscape\prefs.old
*:\**\pinerc.old
Optional: Disk Image file types
*:\**\*.bkf
*:\**\*.gho
*:\**\*.ghs
*:\**\*.iso
*:\**\*.pqi
*:\**\*.qic
*:\**\*.v2i
Optional: History files and directories
*:\**\history.ie5\*
*:\**\mozilla\**\cookperm.txt
*:\**\mozilla\**\history.dat
*:\**\netscape.hst
*:\**\recent\*.lnk
*:\**\ws_ftp.log
Optional: Spool files
*:\**\*.qdr
Optional: Trash files and directories
*:\**\eduora\trash.*
*:\**\imapmail\**\trash
*:\**\imapmail\**\trash.*
*:\**\mail\**\trash
*:\**\mail\**\trash.*
*:\**\netscape\**\cache.trash\*
*:\**\network Trash Folder\*
CommCell files and directories
*:\**\galaxy\*
*:\**\MDBDATA\*
*:\**\MTADATA\*
*:\**\DSADATA\*
*:\**\tmp.edb
*:\**\Exchsrvr\server_name .log
*:\**\Exchsrvr\mailroot\vsi *\*
*:\**\Exchsrvr\srsdata\*
*:\**\Exchsrvr\IMCDATA
*:\**\Exchsrvr\ExchangeServer_*\Projects\*
*:\**\Exchsrvr\ExchangeServer_*\Projects\ExchangeServer_*.edb
*:\**\Exchsrvr\ExchangeServer_*\MMS.*
M:\*
*:\**\Microsoft SQL Server\mssql\backup\*
*:\**\*.[l-n]df
Note: for Unix be sure to include all cases
*.nsf*
*.ntf
s*.txn
*.box*
*:\**\SMS\Inboxes\*
*:\**\SMS\Logs\*
*:\Win*\MS\SMS\*
*:\SMSLOGON\*
*:\**\Sophos SWEEP for NT\Temp\*
*:\**\Sophos SWEEP for NT\~Temp\*
*:\**\Sophos SWEEP for NT\infected\*
*:\PROGRAM FILES\SOPHOS\MAILMONITOR\EXCHANGE\TEMP\*
*:\PROGRAM FILES\SOPHOS\MAILMONITOR\EXCHANGE\QUARANTINE\*
*:\Program Files\NAV\I2_LDVP.TMP\*
*:\**\Application Data\Symantec\Norton AntiVirus\Corporate Edition\*
*:\Program Files\NavNT\vpdebug.log
*:\Program Files\NAVMSE\Temp\*
*:\Program Files\Symantec\SAVFMSE\Temp\*
*:\Program Files\Trend\SProtect\Temp\*
*:\Program Files\Trend\SProtect\Virus\*
*:\Program Files\TrendMicro\VCS\Agent\Bin\Temp\*
*:\Program Files\CA\common\scanengine\logs\*
*:\Program Files\CA\common\scanengine\incoming\*
*:\Program Files\CA\Etrust\InoculateIT\MOVE\*
*:\**\Common\ScanEngine\ARCTEMP\*
*:\**\Etrust\Inoculate\DEBUG