Data Domain discussion and Notes

The following are notes from our discussions w/ EMC on deploying Data Domain w/ RMAN at client site:

Process flow is as follows
Clients -> eth -> Media/Master Server ((includes OST)-> SAN -> DD

The OST plugin first sends a request to the DD server/appliance to validate the hash of the data packet. DD uses variable segment size to check this, it is not fixed block. If DD has already seen it, the full data packet is discarded and the block’s expiration is updated on the Data Domain. This is where Boost comes in handy, as it does it inline/in-band, reducing roundtrip acks. However, when DD does not have a hit; i.e., doesn’t find a duplicate block already stored, the media server sends a second packet with all of the data. This could mean twice the number of data packets if your data does not dedupe well, which leads to poor performance and high CPU utilization all around.

If you don’t have a lot of duplicate data, the media server can take a longer time to process the same amount of data and it still had to send it anyway, negating the use of Boost. Keep the following in mind: Each backup stream is processed in one thread (typically there is 4 to 8 threads per core), so stream speed is limited by processor core speed, we typically see CPU usage at 13%.

The following were key points/considerations:

Use RMAN catalog
Use rman duplicate or clone when cloning databases. If using backup/restore for cloning, make sure to use ‘nid’ to get new db incarnation

Don’t use RMAN compression
RMAN 2 Channels per core -> define how many cores you want to consume
RMAN Maxopenfiles = 4
RMAN Filesperset = 1 <- a must for deduce to work correctly Jumbo Frames dnfs - 4 1gb links or 1 10GbE Example RMAN command

RUN
{
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE; # default
CONFIGURE DEVICE TYPE SBT_TAPE Backup TYPE to BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE ‘SBT_TAPE’
PARMS ‘SBT_LIBRARY=/u01/app/oracle/product/11.2.0.4/dbhome/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityn.com,ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome)';
ALLOCATE CHANNEL c1 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=/u01/app/oracle/product/11.2.0.4/dbhome/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityn.com,ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome)';
send ‘set username ddboostusername password XXXX servername ddomain406.viscosityn.com';
RELEASE CHANNEL c1;
}


RUN {
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE; # default
CONFIGURE DEVICE TYPE SBT_TAPE Backup TYPE to BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE ‘SBT_TAPE’
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityna.com,ORACLE_HOME=${ORACLE_HOME})';
ALLOCATE CHANNEL c1 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityna.com,ORACLE_HOME=${ORACLE_HOME})';
ALLOCATE CHANNEL c2 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityna.com,ORACLE_HOME=${ORACLE_HOME})';
ALLOCATE CHANNEL c3 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityna.com,ORACLE_HOME=${ORACLE_HOME})';
ALLOCATE CHANNEL c4 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityna.com,ORACLE_HOME=${ORACLE_HOME})';
ALLOCATE CHANNEL c5 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityna.com,ORACLE_HOME=${ORACLE_HOME})';
ALLOCATE CHANNEL c6 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=ddomain406.viscosityna.com,ORACLE_HOME=${ORACLE_HOME})';
backup filesperset 1 all database format ‘%d_DATABASE_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} database backup';
backup current controlfile format ‘%d_controlfile_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} Controlfile backup';
backup spfile format ‘%d_spfile_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} Spfile backup';
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
release channel c6;
}

▪While this backup is running, you can monitor performance from Data Domain side using the following command:
# ddboost show stats interval 5 count 100

▪To inspect storage statistics; eg, compress/dedupe rates:
# ddboost storage-unit show compression

Final thoughts

We were told that the latency should not exceed 15 ms (even for routed) (entreprise DD), plus make sure its not highly routed

We used DD Management Center it proved to be quite useful for monitoring and trending

Customer had dd2200 – Observations & Considerations
We were able to get 20 TB/hr
There was one channel == 1 thread

Watch out for Solaris with multi-threaded, very weird scheduling issues with CMT systems (16 threads per core)
Best to \test w/ non-boost then w/ boost to determine overall benefit of DDBoost. Esp for databases smaller than 300GB
Use hard mount for non-boost
Dont backup archive logs if you can avoid it
setup ifgroups for layer3 aggregation