Calypso® software is deployed in large enterprise environments.
Various scalability criteria needs to be addressed in order to ensure a successful
and sustainable installation.
Certain guidelines associated with the design, deployment, and support of large
CommCell® environments must be followed. Consider the following as you plan the
install and configuration of Calypso®.
The following list outlines the benefits gained by deploying Calypso®
software across multiple CommCell groups in an Enterprise environment:
General IT Maintenance
Allows for partial downtime of backup operations. For example, when a CommServe
group is taken offline for software updates/upgrades or hardware upgrades and
maintenance, the other CommCell groups will continue to function without impact
on data movement operations.
Reduce Risk Associated with Unreliable IP Networks
The strategic placement of CommServe within the shortest network distance
to the servers it protect, dramatically reduces issues associated with slow
or unreliable network connectivity between LAN clients and the CommServe.
Improved CommCell® Group Activity Monitoring and Performance
By balancing data protection activity load across multiple CommServes, a
smoother operation of the CommCell group is allowed, with better response time
in each individual CommCell console GUI.
Consolidated Processing
Reports from CommServes in multiple CommCells may be e-mailed to the same
user or appended in same folder or spreadsheet.
This helps in consolidated processing and review. However, using CommNet
as a central executive monitoring console and reporting mechanism helps spanning
multiple CommCell groups.
Load Balancing on Resources
As the data protection infrastructure evolves, Calypso®
provides an easy-to-use method for maintaining the balance of resources across
CommCell groups via CommCell Migration.
This section suggests the deployment requirements and considerations to be taken
into account for a Workgroup, Datacenter or an Enterprise environment. Each environment
has different hardware and software requirements.
It is important to complete an accurate assessment of data growth planned, during
completion of CommCell® design configuration.
This ensures that the design aligns with the scalability requirements reflected
in this document when calculating backups, tape utilization and retention requirements.
When deploying Calypso® software in virtual
environments consider the following:
The hardware/software vendors must support their products within the hypervisor.
The hypervisor vendor must support use of the hardware/software product
within the hypervisor. Refer to specific notations regarding tape device connectivity
to MediaAgents resident on Virtual Machines in the sections that follow.
The use of Calypso®software within a virtual
machine may introduce conditions (such as contention for shared resources or
other interruptions) not present during standard product certification and scalability
testing. The most commonly anticipated impacts associated with system performance
on Virtual Machines are specified, however these impacts are intentionally broad
and different use cases may yield better or lesser performance than as documented
here. Additional tuning may be required to address any resulting delay, retry,
or timeout conditions. Any such tuning is at the recommendation of the Virtual
Machine platform developer rather than Bull.
The software may experience lesser performance when used in a Virtual Machine
configuration for CommServe and MediaAgent systems. It may be necessary to increase
system resources to address observed performance issues.
As noted in the upcoming sections of this document using a hypervisor vendor’s
interface to suspend, resume, or otherwise end the operation of a production
(running services) CommServe or MediaAgent is specifically not supported. Only
the use of the Bull Service Control interface (GUI and / or
qscripts and qcommands) is supported for control.
In the event of a support escalation, Bull Customer Support
will make commercially reasonable efforts to attempt to resolve the issue within
the virtual environment. Bull reserves the right to qualify,
limit, exclude, or discontinue support for a virtualized CommServe or MediaAgent
configuration due to unforeseen incompatibilities within the hypervisor environment.
If the CommServe server is configured on a Virtual Machine (VM), then it typically
operates at a range of 60% efficiency as compared to a comparable physical server.
In this deployment model scalability limits are affected as compared to physical
server environment. The following conditions must be taken into consideration while
deploying the CommServe in virtual environment:
ERSet creation for a virtualized CommServe server is also limited to writes
to a shared magnetic library given support for tape devices in Virtualized environments.
Appropriate disk should be allocated to the CommServe server for the ERSet.
Virtualized CommCell guests should be running on VMware vSphere 4.0.x or
Microsoft Hyper-V R2 with comparable resources to a physical server. At a minimum
2 vCPUs, 4 GB vMemory and 60GB of disk should be available to the guest (Workgroup-level
CommServe).
In order to perform a storage vMotion, the virtual machine being vMotioned
must have a snapshot taken. Once the snapshot is taken, the base disk is then
copied from the source LUN to the destination LUN. This operation imposes high
IO usage on both those disks, decreasing capacity available for VM usage. Additionally,
Virtual Machine IO is directed to the snapshot, hence for every read operation
the VM must first check the snapshot file to see if the bit is there, and if
it is not, then must read it from the base disk. Once the copy is complete,
the VM's base disk is changed to the destination location, but the snapshot
(containing IO that occurred during the vMotion) still resides on the source
disk. At this time that snapshot is committed. This means those changes are
synced back to the base disk, now on the destination LUN. This also requires
IO overhead.
During a storage vMotion, every IO operation sent
by the virtual machine requires at least 2 IO operations on the physical
disk, and additional IO is being sent to the SAN, which would decrease
available IO capacity.
During any of these phases, the additional IO could cause an IO intensive
application on the virtual machine, which is sensitive to data read/write times
above and beyond what the OS timeouts are set to, to time out on some of those
requests, if the SAN is unable to cope with the additional workload and still
return read/write requests within that application's timeout settings. Even
if requests are returned fast enough to not cause a timeout, it still would
result in additional latency. Databases are among the types of applications
this might affect. vMotion operations associated with the CommServe DB will
create undesirable latency on CommServe operations. Under no circumstances should
a vMotion operation be completed with an active CommServe DB.
Storage vMotions cause additional IO load, which can cause a heavily loaded
storage device to respond more slowly than an application might expect.
Configure an Alarm using vSphere Client
As a best practice step when configuring a CommServe on a VM, it is
recommended to configure
an alarm to notify the CommCell administrator of a vMotion event. The CommCell
administrator can use this alarm to diagnose any CommServe performance issue
that follows the vMotion event.
1.
From the vSphere Client, navigate to the virtual machine.
Click the Alarms tab and select the Definitions view.
Right-click anywhere in the tab and click New Alarm...
2.
Type a name for the alarm in the Alarm name box.
Specify the purpose for this alarm in the Description box.
Select Monitor for specific events occurring on this object, for
example, VM powered On.
3.
Click the Triggers tab.
Click Add.
From the Event column, select the activity that will trigger the
alarm.
4.
Click the Actions tab.
Click Add.
From the Action column, select Send a notification email.
If a MediaAgent is configured on a Virtual Machine (VM), then it typically operates
at a range of 60% efficiency as compared to a comparable physical server. In this
deployment model scalability limits are affected as compared to physical server
environment. The following conditions must be taken into considerations while deploying
the MediaAgent in virtual environment:
The maximum number of concurrent streams supported by a single MediaAgent
is effectively 60% of the specification for a physical host. Refer to the upcoming
sections of this document for additional detail regarding MediaAgent scalability
associated with stream counts.
Virtualized MediaAgent guest servers running in a VMware environment should
be running on VMware vSphere 4.0.x with the latest version and patches applied.
In a Microsoft environment guests should be running on Hyper-V R2. At a minimum
2 vCPUs, 4 GB vMemory and sufficient disk resources to hold the data (to be
protected) should be available to the guest.
Library creation for a virtualized MediaAgent is limited to writes to a
single magnetic library or stand alone library given support for tape devices
in Virtualized environments.
For a VMware virtualized MediaAgent, Virtual disk libraries can be configured
as virtual disks within the MediaAgents guests or as attached RAW LUNs. Tape
Libraries also can be presented to a virtual guest by using SCSI pass through.
Adaptec cards are supported for SCSI pass through and iSCSI is supported for
tape library connections. It is recommended to bind the iSCSI initiator to a
separate NIC on the ESX host as it will isolate the ISCSI calls from routine
LAN traffic. For more information on SCSI pass through and hardware compatibility
consult VMware documentation:
http://www.vmware.com/support/pubs/
and VMware support.
Library creation for a virtualized MediaAgent is
also limited to writes to a single magnetic library or stand alone library
given support for tape devices in Virtualized environments.
For Hyper-V MediaAgents, disk can be passed to the virtualized Hyper-V
MediaAgent with the pass through disk feature. Note that the disk will need to be
in an offline state on the parent server to function on the guest. For more
information on pass through disks in Hyper-V refer to the Microsoft Hyper-V
R2 website:
http://www.microsoft.com/windowsserver2008/en/us/hyperv-main.aspx.
A warning is issued to Administrators when the scalability limits are approached.
The warning message advises to modify current settings or configure the entities
that exceed the scalability guidelines within a different CommCell group.
The following environments have been identified as scalability thresholds within
a single CommCell group. Threshold considerations are divided as follows:
Observe the following parameters for implementing a CommCell hierarchy.
If your CommServe is running on a
physical computer (a non-virtual machine) with Solid-state
drive (SSD), you can scale it to twice the limits stated
below.
E.g. Number of clients sustained could reach 5000
clients instead of usual 2500.
CommCell Class
Multiplexing Factor
Concurrent Job Streams
Client Count
MediaAgent Count
Drive Count
SLOT COUNT
Workgroup
2
75
50
20
20
No Limit
Datacenter
5
300
200
50
100
No Limit
MS SQL Express Edition
5
10
25
The maximum number of Client count is inclusive of MediaAgents.
25
Any client computer can be installed as a MediaAgent.
500
No Limit
Enterprise
25
1000
2500
Any client computer can be installed as a MediaAgent.
There is no
separate restriction for MediaAgent.
1000
No Limit
Notes
Multiplexing Factor Limit for a Single CommCell.
Administrator Notification are set at a
Multiplex count of 5.
Number of Active Job Streams corresponding to concurrently running
jobs.
Running Job Stream = No. of tape drives * (Multiplexing Factor + Magnetic
Writers).
This is the high watermark value in the GUI and is enforced at 1,200.
Number of Supported Clients Within a Single CommCell.
Hard Limit Notification occurs at 4,500 Clients within the CommCell.
Number of Supported MediaAgents within a Single CommCell
Express CommCells have a limit of
25 MediaAgents
The maximum Number of Concurrent Streams to a Single MediaAgent is set
at a value of 300.
Number of Supported Tape Drives within a Single CommCell
Number of Supported Slots within a Single CommCell
To improve performance of a CommCell group, enable the option for the CommCell
Console to become disconnected when inactive for a certain period of time. This
option disconnects connections from idle GUI sessions, thereby allowing other GUIs
to connect without exceeding the established parameter.
Follow the steps below to enable this option:
From the CommCell Console, select Tools | Control Panel.
From the Control Panel, select System.
From the General tab, select the Allow GUI connections
to timeout.
Enter a timeout value in GUI timeout in minutes box.
The recommended value is 30
minutes, however the default value is set to 180 minutes.
Enter a reasonable timeout value to programmatically
terminate idle connections. The lower the disconnect threshold
value, the sooner the idle GUIs will be disconnected, allowing
active GUI sessions to connect to the CommServe.
It is strongly recommended that the soft limits mentioned in the sections below
are followed, and not be exceeded.
It is also recommended to review of the Deduplication Architecture Guide when
planning for Calypso® Block Level Deduplication CommCell
group deployments.
Contact Customer Support or your Account Team for the current release of that
separate document.
To increase the number of streams for concurrent backups from large number of
clients, enable the option optimize for concurrent backups, It will increase the
current stream count limit by 200 more streams.
From the CommCell Browser, navigate to Storage Resources | MediaAgents | <MediaAgent>.
Right-click the MediaAgent that you wish to optimize for concurrent
LAN backups and then click Properties.
Click the Control tab.
Select the Optimize for concurrent LAN backups to enable
the option.
For better throughput, specify a lower value for the number of data pipes/processes
that the client uses to transfer data over a network.
From the CommCell Browser, right-click the subclient.
Click Properties.
Click Storage Device.
Click Data Transfer Option tab.
Specify the number of Network Agents as 1.
On non-UNIX computers, the default value
is 2 and a maximum of 4 can be established if necessary. On
UNIX computers the default value is 1 and a maximum of 2 can
be established if necessary.
For maximum performance and robustness, the total number of Replication Pairs
configured for the same source volume should be kept to a minimum. If multiple Replication
Pairs for the same source volume are required, the following limits must be observed.
It is important to manage concurrently running jobs, this can be done by staggering
schedules. Use multiple schedule policies on different client groups and adjust
the timing of the schedules in order to optimize scalability.
The table below displays the maximum number of concurrent jobs permitted in different
environments.
CommCell Class
Total Permitted Job
Count
Workgroup
1 to 100
Datacenter
101 to 300
Enterprise
301 to 1,000
Notes
This includes Jobs in a Waiting/Pending status.
However, there is no limit to the number of Storage Policies in a single
CommCell Group for the Enterprise environments.
Stagger the start time of jobs to be separated by a time interval of
Upto 20 minutes.
There is no software limit on the number of hardware-generated Snapshots using
Calypso®. However for limits imposed by each manufacturer’s
controllers and software, refer to your hardware provider's documentation.
This section represents scalability thresholds for a co-located CommServe and
CommNet Server, where the CommNet Server functions as reporting server for a single
CommServe.
CommCell Class
Maximum Number of
Supported Clients
Workgroup
1 to 50
Datacenter
51 to 200
Enterprise
201 to 2,500
Notes
A dedicated and stand alone CommNet Server is recommended for Multiple CommCells.
The co-located CommNet Server cannot function as reporting server
for multiple CommCells, unless the appropriate licenses are
available.
Client counts per CommCell beyond these specification require the
configuration of separate CommNet Servers.
The maximum number of CommCell groups managed by a single CommNet
Server is 25.
It is recommended to install CommNet Server
on dedicated machines, in order to achieve scalability of connecting
maximum number of clients to a CommServe.
Subclient count optimization allows managing the daily backup operations easily
for the administrator. It is recommended to periodically review the subclients in
order to determine if any redundant or unneeded subclients exist and can be removed
from the CommCell configuration. Information about each subclient is tracked by
the CommServe, and by reducing the number of subclients, there is a huge reduction
in the amount of tracking information.
Increasing Chunk Size
This parameter impacts tape backup operations. A higher chunk size gives a better
throughput.
A lower value for this setting is recommended for frequent checks against slower
data protection operations, especially when data is moving across a WAN link.
From the CommCell Console, select Tools | Control Panel.
From the Control Panel, select Media Management.
Click the Chunk Size tab.
Click the Agent for which you wish to modify the chunk size, and
modify the chunk size to 8 GB in the Chunk Size column.
Granular restores will be slower using this setting,
however, large restores will be faster.
This setting does not apply to database backups where
the Chunk Size is 16GB by default.
Click OK.
Updating Interval
It is recommended to implement a formal software update management process. This
process allows for planned software updates to be aggregated on a 30-day or 60-day
basis.
A test CommServe is recommended to be used for the completion of update validation
prior to placing scheduled updates into a production environment. This helps improve
backup stability.
It is also recommended to increase the Job manager Update Interval for your Agent,
this can be done using the following steps:
From the CommCell Console, select Tools | Control Panel.
From the Control Panel, select Job Management.
Click the Job Updates tab.
Click the Agent for which you wish to modify the update interval,
and modify the Protection (Mins) to 10.
File System Multi-Streaming employs multiple data streams per subclient for the
data protection operation, enabling the subclient's contents to be distributed to
all the streams, transmitting them in parallel to the storage media. It is recommended
to use Automatic File Multi-Streaming for larger subclients (1TB or more).
It allows the file system backup to use multiple readers for increased performance,
this configuration in turn reduces duplicate file Scan Time on client servers.
For subclients less than 1TB in total size, set the
number of readers on the sub-client to 1.
Windows File System
After upgrading to a newer release of Calypso®, delete
the system state backups.
Newly installed clients do not have the separate subclient.
Within a CommCell hierarchy, the following subsystems must be examined so that
a degradation in performance does not occur. Concerning areas are dependent upon
individual data protection objectives and the server along with the available storage
resources. The subsystem recommendations described in this section are based on
size and and expected growth trends.
No Impact
Update index on MediaAgent. Up to the number of drives controlled through each
MediaAgent.
Least Impact
Growth and rapid overwriting of log files.
Create index processes on the MediaAgent.
Approximately six backup jobs per drive per night on a MediaAgent, for a
maximum of 36 processes that may start concurrently.
Number of cache mounts in the system.
Medium Impact
Refresh updated to the GUI.
Concurrent schedule submission (600).
You may stagger up to 100 jobs at a time and 20 minutes apart.
Operating too many jobs in the job controller.
Resource Manager slowness due to parsing of high numbers of resource entities.
Reports Prune Forecast Report and Data on Media Report run
slow when amount of data is high.
High Impact
Performance degrades when large Exchange Mailbox or Lotus Notes Document
jobs are run concurrently.
Excessive number of simultaneous GUI connections to the CommServe also impact
performance.
CommServe SQL database interaction with Data Aging, Timeouts, deadlocks,
etc. impacts performance.
Flooding of event messages and alerts (if configured) impacts performance.
Tape drive thrashing (tape to drive swaps) may occur if all jobs for all
Storage Policies are flooded into the Job Controller at one time. It is recommended
that backups of Storage Policies are scheduled in groups in order to achieve
a level of efficiency so that queued jobs that are associated with one Storage
Policies have the tapes mounted in drives. This will allow jobs to stream to
“pre-mounted” tapes.
