Deduplication on the source (client) side identifies and
eliminates redundant data from the client. This reduces the data transfer
rate over the network from the client to the MediaAgent.
When data is
backed up for first time, all the data is stored physically. The signature
is generated and stored in the deduplication database on the client and
in the MediaAgent.
For subsequent backups, the signatures are generated in the client and
compared with the existing signatures in the deduplication store in the
MediaAgent. Based on the comparison, the following is performed:
If a signature is present in the store, then it indicates that the
data segment was already encountered in a previous backup and only the
signature reference is transmitted.
If a signature is not present in the store, then it indicates that
the data segment is new and the data as well as the signature is transmitted
to the MediaAgent.
By default, Source Deduplication is enabled in a storage policy copy. Use the
following steps to enable Source Deduplication for an existing storage policy copy,
if it was not enabled.
From the CommCell Browser, navigate to Policies | Storage Policies
| <Storage Policy>.
Right-click the storage policy copy displayed in the right pane
and then click Properties.
In the Copy Properties dialog box, click the Deduplication
tab and then click the Advanced tab.
Select the Enable Client Side Deduplication option.
This option supersedes the Perform
Client Side Deduplication option set on the client.
You can also enable the source side deduplication for a specific client.
Use the following steps only when Client Side Deduplication option is not
enabled on the Storage Policy copy and you want to enable client side
deduplication on specific clients.
From the CommCell Browser, navigate to Client Computers |
<Client>.
Right-click the client and click Properties.
In the Client Properties dialog box, click the Client
Side Deduplication tab.
Select Perform client side Deduplication box.
If Enable Client Side Deduplication
option is set on the Storage Policy copy, then this setting
will have no effect.
The signature lookup process can be optimized by setting up the cache locally
on the source. When the local cache is setup, the signature is first looked up in
the local cache. A remote lookup is initiated only when the signature is not available
in the local cache.
For each backup, though the first lookup is a remote lookup, all subsequent lookups
will be performed locally. Also, the remote lookups are performed in batches to
optimize network usage. The signature cache can grow to a maximum of 32 GB.
Use the following steps to enable signature caching on the client:
From the CommCell Browser, navigate to Client Computers |
<Client>.
Right-click the client and click Properties.
Click the Client Side Deduplication tab.
Ensure Perform client side Deduplication box is selected.
Select Enable Client Side Disk Cache box.
Click OK.
Once the client side disk cache is enabled, the signatures are generated
for any new data and a database of the signatures is maintained on the
client computer.
To verify the source side deduplication database, on the client computer,
start the windows explorer and navigate to following installation directory:
bull\<Agent>\Job Results\
It contains a folder, CV_CLDB. This
folder is the source side deduplication database directory.
During the backup process, it is recommended that to filter the source side
deduplication database directory to avoid the backup failure.
If you have Service Pack 5A or higher version installed, the source side
deduplication database directory automatically gets filtered.
If you have Service Pack version earlier than 5A, make sure to install the
latest service pack or use the following procedure to filter the source side
deduplication database folder:
From the CommCell Browser, navigate to Client Computers |
<Client> | defaultBackupSet
Right-click the <subclient> displayed in the right pane and
then click Properties.
Click Filters tab.
Under Exclude these files/folders/pattern click Browse
button and navigate bull\<Agent>\Job
Results and then click CV_CLDB folder.
When source side deduplication storage policy is configured,
compression is automatically enabled on the storage policy copy, but the
compression always happen on the client.
For most of the data types the
compression is recommended. This process works by compressing the blocks and
then generating a signature hash on the compressed block.
Use the following steps to enable data compression for all subclients
associated to storage policy:
From the CommCell Browser, navigate to Policies | Storage
Policies | <Storage_Policy>.
Right-click the primary storage policy copy displayed in
the right pane and click Properties.
Click the Deduplication tab, and then click the Settings
tab.
Select the Enable Software Compression with Deduplication
box.
This options is enabled by default. It is recommended to have
data compression enabled when using deduplication.
When this option is enabled the Use
Storage Policy Settings option is enabled by default on
the corresponding subclients.
If Enable Software Compression with Deduplication
check box is selected, the signature generation and
compression will be done on client.
Click OK.
Compression on Subclient
By default, for source side deduplication storage policy, compression always
happens on client. To turn off the compression on a specific client computer,
use the following steps
From the CommCell Browser, navigate to Client Computers | <Client_Computer> | File System | defaultBackupSet.
Right-click the <Subclient> to which the
deduplication storage policy is associated and then click Properties.
Click the Storage Device tab, and then click the Data
Transfer Option
tab.
In the Software Compression area, Using Storage Policy Settings option is
selected by default.
Select the Off option to turn off the compression.
Variable content alignment is a content aware approach to deduplication that further
reduces the amount of data stored during a database agents backup. It accomplishes this
by aligning the segment boundaries of the backup data stream as minor changes to
the data in the stream that are made between incremental backups. Therefore, the effectiveness
of deduplication increases more with this feature on client systems that experience
small changes to the backup data.
Variable content alignment is performed on the
client system and consequently you may experience some performance overhead, especially
when used together with software compression. You can enable variable content alignment
as follows:
From the CommCell Browser, right-click the <Client>
you wish to enable variable content alignment and then click Properties.
Enabling this option will consume more
disk space. This happens because a fresh copy of the deduplicated data blocks with new signature
is created for that deduplication database. Hence, this new
signature will not match the existing signatures available
in the deduplication database and thus creates a new baseline
for the deduplication database.
Whenever a new store is created, a fresh copy of the deduplicated data blocks
is created for that store. However these data blocks may also be available in the
sealed stores residing in the MediaAgent/data center. The Store Priming feature
discovers such data blocks from the previously sealed stores and uses them to baseline
the new store. In source-side deduplication, this saves the need for clients to
transfer data blocks that are already available in the MediaAgent.
Use this feature to reduce the data transfer between clients that have source-side
deduplication enabled and connect to the MediaAgent over a WAN.
This feature is not recommended if the clients exist in the data center or performance
throughput is critical.
By default, Optimize for concurrent LAN backups option is enabled on the
MediaAgent properties to ease up the IPC resource usage and to use more number
of concurrent streams for data protection operations.
For the following agents, if you have source side deduplication enabled,
Optimize for concurrent LAN backups option is not recommended. If you have
this option enabled, the system ignores the option and will use a regular
pipeline for data protection operations. If you wish to use this option to
increase streams for concurrent backups, use MediaAgent side deduplication.
For Windows Agents such as DB2, Oracle, SAP Oracle or SAP MAXDB
iDataAgents this option is
not recommended.
For all Unix Agents except Unix File System
iDataAgent this option is not
recommended.
Use the following steps to enable/disable optimize for concurrent LAN backups
on the MediaAgent:
From the CommCell Browser, navigate to Storage Resources | Media
Agents | <MediaAgent>.
Right-click the MediaAgent that you wish to optimize for concurrent
LAN backups and then click Properties.
Click the Control tab.
By default, Optimize for concurrent LAN backups check box
is selected.
Clear the Optimize for concurrent LAN backups to disable
the option.
Click OK.
Configure the Shared Memory Segments
HP-UX
The following table displays the possible range of values for each parameter:
If "*" appears
after the parameter name, this indicates that you can increment the
value of the parameter based on the number of desired streams by using
the formula
parameter value = 64 * number of desired streams
as long as the "high-end" value of the range for the parameter is not
exceeded.
Parameters
Description
Possible Range of Values
SEMMNI*
Defines the maximum number of semaphore sets in the entire system.
1 – 65535
SEMMAP
Defines the maximum number of semaphore map entries.
1 – 65535 (SEMMNI + 2)
SEMMNS*
Defines the maximum semaphores in the system
1 – 2147483647 [MAXINT] (Default value 128 is too low.)
SEMVMX
Defines the maximum value of a semaphore
32768
SHMMNI*
Defines the maximum number of shared memory segments in the entire
system
1 – 2147483647 [MAXINT]
SHMMAX
Defines the maximum allowable size of one shared memory segment
0 – 4294967295 (Available physical memory); recommended value is
2147483648
SHMSEG*
Defines the maximum number of shared memory segments that one
process can attach
0 - 32767
SEMMNU
Defines the maximum number of System V IPC system-wide processes
that can have "undo" operations pending at any given time.
1 - (nproc-4) [256]; preferred value: 1024
Use the following steps to modify the kernel parameters with System
Administrator Menu (SAM) for HP-UX File System:
1.
From the unix prompt, enter the command to open the
System Administrator Menu (SAM).
#sam
2.
In the SAM utility window, double-click
the Kernel Configuration icon.
3.
Double-click the Configurable
Parameters icon. SAM will display a screen containing a list of
configurable parameters, including current and pending values.
4.
Double-click on the parameter you wish to
change.
5.
Click New Formula/Value option and
type the
recommended value.
6.
Click OK.
7.
Restart the computer to allow the changes
to take effect.
System Management
Homepage (SMH) is the new tool introduced to manage HP-UX. Use the following steps to modify the kernel parameters
using System Management Homepage (SMH) since the System Administration Manager (SAM) is
deprecated in the current releases of HP-UX:
1.
From the unix prompt, enter the command to open the System Management
Homepage
(SMH) window.
#smh
2.
In the SMH Text User Interface, select k -Kernel Configuration
and press Enter.
DISPLAY variable is not set. Starting
the Text User Interface.
Please wait...HP-UX System Management
Homepage (Text User Interface)
Use the following formula to
calculate the value for the parameter:
parameter value = current value + (64 * number
of desired streams)
If the client is configured as iDataAgent, be sure
to allocate 10 streams per each affected parameter. If the
client is configured as a MediaAgent, be sure to allocate at least 100
additional streams per each affected parameter.
If the system is configured as both an
iDataAgent and a MediaAgent, be sure to consider the sum of
the respective parameter values for this purpose.
A Confirmation window will be displayed warning you that the requested
changes have been saved and will take effect at next boot. Press
Enter to continue.
mount: ignoring incomplete/incorrect
entry for /dev/dsk/c0t0d0 in /etc/fstab
NOTE: The requested changes could not
be applied to the currently
running system, for the following
reasons:
- The tunable 'semmni' cannot be
changed without a reboot.
* The requested changes have been
saved, and will take effect at next boot.
Tunable Value Expression
semmni (now) 2048 Default
(next boot) 2048 2048
mount: ignoring incomplete/incorrect
entry for /dev/dsk/c0t0d0 in /etc/fstab
Command Successful
press Enter Key to continue
8.
Restart the computer.
Linux
The following table displays the possible range of values for each parameter:
If "*" appears
after the parameter name, this indicates that you can increment the
value of the parameter based on the number of desired streams by using
the formula
parameter value = 64 * number of desired streams
as long as the "high-end" value of the range for the parameter is not
exceeded.
Parameters
Description
Possible Range of Values
SEMMSL*
Defines the minimum recommended semaphore value.
1 – 2147483647 [MAXINT]
SEMMNS*
Defines the maximum semaphores in the system.
1 – 2147483647 [MAXINT]
SEMOPM
Defines the maximum number of operations for each semaphore call.
100
SEMMNI*
Defines the maximum number of semaphore sets in the entire system.
1 – 65535
SHMMNI*
Defines the maximum number of shared memory segments in
the entire system.
1 – 2147483647 [MAXINT]
SHMMAX
Defines the maximum allowable size of one shared memory segment.
0 – 2147483648 [2 GB] (the high-end value is the recommended value)
SHMALL
Defines the maximum total shared memory system-wide.
2097152
Use the following steps to modify the kernel parameters:
1.
From the unix prompt, log on to the client as root.
2.
Navigate to the /proc/sys/kernel.
cd /proc/sys/kernel
3.
View the current values of the parameters in the sem
file.
For example, view the values
of all semaphore parameters. The output provides the current value for
the following parameters in the order listed: SEMMSL, SEMMNS, SEMOPM, and
SEMMNI.
Example:
cat sem
250 32000 32 128
In this example, SEMMSL=250,
SEMMNS=32000, SEMOPM=32, and
SEMMNI=128
4.
Modify the required parameter.
For example, modify the
semmns parameter.
Use the following formula to
calculate the value for the parameter:
parameter value = current value + (64 * number
of desired streams)
If the client is configured as iDataAgent, be sure
to allocate 10 streams per each affected parameter. If the
client is configured as a MediaAgent, be sure to allocate at least 100
additional streams per each affected parameter.
If the system is configured as both an
iDataAgent and a MediaAgent, be sure to consider the sum of
the respective parameter values for this purpose.
sysctl -w kernel.<parameter>="value"
For example:
sysctl -w kernel.semmns="2147483647"
5.
Restart the computer to allow the changes to
take effect.
On a redhat linux computer, you can permanently modify the parameters using
the following steps:
1.
From the unix prompt, log on to the client as root.
2.
Open the /etc/sysctl.conf file.
vi /etc/sysctl.conf
3.
Modify the required parameters.
For example, modify the
sem
parameters.
Use the following formula to
calculate the value for the parameter:
parameter value = current value + (64 * number
of desired streams)
If the client is configured as iDataAgent, be sure
to allocate 10 streams per each affected parameter. If the
client is configured as a MediaAgent, be sure to allocate at least 100
additional streams per each affected parameter.
If the system is configured as both an
iDataAgent and a MediaAgent, be sure to consider the sum of
the respective parameter values for this purpose.
Run the command to allow the changes to
take effect.
sysctl -p
Tru64
The following table displays the possible range of values for each parameter:
If "*" appears
after the parameter name, this indicates that you can increment the
value of the parameter based on the number of desired streams by using
the formula
parameter value = 64 * number of desired streams
as long as the "high-end" value of the range for the parameter is not
exceeded.
Parameters
Description
Possible Range of Values
SEMMNI*
Defines the maximum number of semaphore sets in the entire system.
1 – 65535
SEMMSL*
Defines the minimum recommended semaphore value
1 – 2147483647 [MAXINT]
SEMOPM
Defines the maximum number of operations for each semaphore call
100
SEMVMX
Defines the maximum value of a semaphore
32767
SEMUME
Defines the maximum number of undo entries per process
64
SEMAEM
Defines the maximum value for adjustment on exit
16384
SHMMIN
Defines the minimum allowable size of a single shared memory
segment
0 ( Physical Ram)
1
SHMMNI*
Defines the maximum number of shared memory segments in the entire
system
1 – 2147483647 [MAXINT]
SHMMAX
Defines the maximum allowable size of one shared memory segment
0 – 4294967295 (recommended value is 2147483648)
SHMSEG*
Defines the maximum number of shared memory segments that one
process can attach
0 - 32767
Use the following steps to modify the kernel parameters:
1.
From the unix prompt, create a backup copy of the
/etc/sysconfigtab file.
Example:
cp /etc/sysconfigtab /etc/sysconfigtab.copy
2.
Use the ipcs
command to view the current values of the parameters.
You can also view the kernel parameters in sysconfig file using the
following commands:
Example:
ipcs -m -s
shm_max=1073741824 shm_mni=1024
shm_min=1 shm_seg=256
sem_mni=128 sem_msl=128
sem_opm=100 sem_ume=64
sem_vmx=32768 sem_aem=16384
sysconfig -s
Example:
sysconfig -s
3.
Modify the required parameter value.
For example, modify the shm_mni
parameter.
Use the following formula to
calculate the value for the parameter:
parameter value = current value + (64 * number
of desired streams)
If the client is configured as iDataAgent, be sure
to allocate 10 streams per each affected parameter. If the
client is configured as a MediaAgent, be sure to allocate at least 100
additional streams per each affected parameter.
If the system is configured as both an
iDataAgent and a MediaAgent, be sure to consider the sum of
the respective parameter values for this purpose.
sysconfig -r
<parameter>="value"
Example:
sysconfig-r shm_mni="2147483647"
4.
Save the sysconfigtab
file.
Press the Esc key.
Type
:wq
5.
Restart the computer.
Solaris
By default, the kernel parameters are set during the installation. However, you
can also modify the parameters at a later point in time if needed.
For Solaris 9 and below versions
The following table displays the possible range of values for each parameter
for Solaris 9 and below:
If "*" appears
after the parameter name, this indicates that you can increment the
value of the parameter based on the number of desired streams by using
the formula
parameter value = 64 * number of desired streams
as long as the "high-end" value of the range for the parameter is not
exceeded.
Parameters
Description
Possible Range of Values
SEMMNI*
Defines the maximum number of semaphore sets in the entire system.
1 – 65535 (unsigned short)
SEMMNS*
Defines the maximum semaphores in the system.
1 – MAXINT
MAXINT = 0x7fffffff (2147483647)
SEMMSL*
Defines the minimum recommended semaphore value.
1 – MAXINT
MAXINT = 0x7fffffff (2147483647)
SHMMIN
Defines the minimum allowable size of a single shared memory segment.
0 (Physical Ram)
1
(It is recommended not to change the current value if set. If no value
exists, set it to 1.)
SHMMNI*
Defines the maximum number of shared memory segments in the entire system.
0 – MAXINT
MAXINT = 0x7fffffff (2147483647)
SHMMAX
Defines the maximum allowable size of one shared memory segment.
0 - MAXINT3264
MAXINT3264 = MAXINT (if 32-bit) or 0x7fffffffffffffff (if 64-bit)
(If the current value is less than 4199304, then set it to 4199304.)
SHMSEG*
Defines the maximum number of shared memory segments that one process
can attach.
0 - 32767 (signed short)
SEMINFO_SEMMNU
Defines the total number of undo structures supported by the System
V semaphore system.
1024
Use the following steps to modify the kernel parameters for Solaris 9 and
below:
1.
From the unix prompt, create a backup copy of the
/etc/system file.
cp /etc/system /etc/system.copy
2.
Open the /etc/system file.
vi /etc/system
3.
Change the desired parameter value as per the range given
in the above table.
For example, set the value for semmni
parameter to 55680
Use the following formula to
calculate the value for the parameter:
parameter value = current value + (64 * number
of desired streams)
If the client is configured as iDataAgent, be sure
to allocate 10 streams per each affected parameter. If the
client is configured as a MediaAgent, be sure to allocate at least 100
additional streams per each affected parameter.
If the system is configured as both an
iDataAgent and a MediaAgent, be sure to consider the sum of
the respective parameter values for this purpose.
Example:
set semsys:seminfo_semmni=55680
4.
Save the /etc/system file.
Press the Esc key.
Type :wq
5.
Restart the computer to allow the changes to take
effect.
For Solaris 10 and above versions
The following table displays the possible range of values for each parameter
for Solaris 10 and above:
project.max-shm-memory
Defines the total amount
of shared memory allowed for a project.
[Default = 25% of the physical memory]
It is recommended not to set this value. However, if you have to set the control to accommodate other
resource consumers, ensure that the value is not less than 2MB =
[2097152].
project.max-shm-ids*
Defines the maximum
number of shared memory IDs allowed for a project
16777216 [Default = 128]
project.max-sem-ids*
Defines the maximum
number of semaphores allowed for a project
16777216 [Default = 128]
For Solaris 10 and above, you can use any of the following
commands to modify the kernel parameters:
Projmod command
prctl command
Use the following steps to modify the kernel parameters in Solaris 10 and
above versions using the projmod command:
Change the desired parameter value as per the range given
in the above table.
For example, set the value for project.max-shm-ids
parameter
Use the following formula to
calculate the value for the parameter:
parameter value = current value + (64 * number
of desired streams)
If the client is configured as iDataAgent, be sure
to allocate 10 streams per each affected parameter. If the
client is configured as a MediaAgent, be sure to allocate at least 100
additional streams per each affected parameter.
If the system is configured as both an
iDataAgent and a MediaAgent, be sure to consider the sum of
the respective parameter values for this purpose.
Change the desired parameter value as per the range given
in the above table.
For example, set the value for project.max-shm-ids
parameter
Use the following formula to
calculate the value for the parameter:
parameter value = current value + (64 * number
of desired streams)
If the client is configured as iDataAgent, be sure
to allocate 10 streams per each affected parameter. If the
client is configured as a MediaAgent, be sure to allocate at least 100
additional streams per each affected parameter.
If the system is configured as both an
iDataAgent and a MediaAgent, be sure to consider the sum of
the respective parameter values for this purpose.
