Part Null – Adding Disks to ODA – The Procedure

Adding Solid-State Drives (SSDs) for ODA Data Storage (non-Expansion)

 

There are cases when customers will need to expand the initial storage of the Oracle Database Appliance [ODA].  There are two use cases for such a scenario; one, where storage will be expanded in the existing enclosure 0, second, when attaching an additional enclosure (en1).  The latter is referred to as a storage unit expansion.  In the example below, we will be adding a pack of solid-state drives (SSDs) for data storage into the existing Oracle Database Appliance X7-2-HA base configuration (en0) to fully populate the base storage shelf. When ordering and adding storage, to the base configuration, you can order one, two, or three 5-pack of SSDs to complete the base configuration on Oracle Database Appliance X7-2-HA.

Before adding the disks to the system, ensure that ODA is on the latest update version.

# odacli describe-component

System Version 

—————

12.2.1.4.0

 

System node Name

—————

zsc-oda0-0

 

Local System Version

—————

12.2.1.4.0

 

Component                                Installed Version    Available Version  

—————————————- ——————– ——————–

OAK                                       12.2.1.4.0            up-to-date         

 

GI                                        12.2.0.1.180417       up-to-date         

 

DB                                        12.1.0.2.180116       12.1.0.2.180417    

 

DCSAGENT                                  18.2.1.0.0            up-to-date         

 

ILOM                                      4.0.2.20.b.r123704    up-to-date         

 

BIOS                                      41021300              up-to-date         

 

OS                                        6.9                   up-to-date         

 

FIRMWARECONTROLLER                        13.00.00.00           up-to-date         

 

FIRMWAREEXPANDER                          0304                  0306               

 

FIRMWAREDISK {

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d20,c1d21,   A122                  up-to-date         

c1d22,c1d23,c2d0,c2d1,c2d2,c2d3,c2d4,                                              

c2d20,c2d21,c2d22,c2d23 ]                                                          

}

 

  1. To add three 5-pack of SSDs, insert the disks in slots 5 to 19.
 Insert disks one at a time in the slots.

 

Note:Allow at least one minute between inserting each disk to avoid flooding of disk events into oakd.

After all disks are added, move on to Step 3.

 

  1. Since the firmware of the controller is older than the disks being added. The storage firmware will need to be upgraded.

 

# odacli update-storage -v 12.2.14.0 (this update on both nodes)

 

Or

 

If any disks are older then their firware needs to be upgraded (specifically for those disks):

fwupdate list disk -n <disk name> -v

 

All disk firmware is located under directory /opt/oracle/oak/pkgrepos/thirdpartypkgs/Firmware/Disk

 

Using the output from the “fwupdate list disk” command, apply the appropriate firmware to the disk

fwupdate update disk-firmware -n <cXdX> -f <path to firmware>

 

 

  1. Run the odaadmcli show ismaster command to determine which node is the master.

# odaadmcli show ismaster



 

  1. Run the odaadmcli expand storage command from the master node.

# odaadmcli expand storage -ndisk 15 (number of disks to be added ) -enclosure enclosure number of the disks to be added, either 0 or 1

 

# odaadmcli expand storage -ndisk 15 -enclosure 0

Precheck passed.

Check the progress of expansion of storage by executing ‘odaadmcli show disk’

Waiting for expansion to finish …

Run the command immediately on the slave node.

  1. Execute the odaadmcli expand storage command on the slave node.

#odaadmcli expand storage -ndisk 15 -enclosure 0

Precheck passed.

Check the progress of expansion of storage by executing ‘odaadmcli show disk’

Waiting for expansion to finish …

 

It takes 15-20 minutes to add all of the disks to the configuration. Periodically check the status of the disks inserted to ensure the state changes from “invaid” à “newinserted” à“uninitialized” to à “installed”

 

  1. Run the odaadmcli show disk command to ensure that all disks are listed, are online, and are in a good state.

# odaadmcli show disk

 

  1. Verify that the disks in slots 5 to 19 are added to ASM

 

  1. Run the asm_script to verify that the disks in slots 5 to 19 are added to ASM. If the 15 disks are successfully added (CACHED and MEMBER), then go to Step 9.

 

su grid user /opt/oracle/oak/bin/stordiag/asm_script.sh 1 6

 

For example:

#/opt/oracle/oak/bin/stordiag/asm_script.sh 1 6 | grep CACHED

…….

/dev/mapper/SSD_E0_S05_1399652120p1 SSD_E0_S05_1399652120P1 1 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S05_1399652120p2 SSD_E0_S05_1399652120P2 3 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S06_1399645200p1 SSD_E0_S06_1399645200P1 1 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S06_1399645200p2 SSD_E0_S06_1399645200P2 3 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S07_1399646692p1 SSD_E0_S07_1399646692P1 1 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S07_1399646692p2 SSD_E0_S07_1399646692P2 3 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S08_1399649840p1 SSD_E0_S08_1399649840P1 1 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S08_1399649840p2 SSD_E0_S08_1399649840P2 3 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S09_1399649424p1 SSD_E0_S09_1399649424P1 1 NORMAL ONLINE CACHED MEMBER

/dev/mapper/SSD_E0_S09_1399649424p2 SSD_E0_S09_1399649424P2 3 NORMAL ONLINE CACHED MEMBER

 

  1. Use the odaadmcli show validation storage errors command to show hard storage errors.   Hard errors include having the wrong type of disk inserted into a particular slot, an invalid disk model, or an incorrect disk size.

# odaadmcli show validation storage errors


 

  1. Use the odaadmcli show validation storage failures command to show soft validation errors. A typical soft disk error would be an invalid version of the disk firmware.

# odaadmcli show validation storage failures


 

  1. If the disks are not added to ASM, then add them manually. As grid user, execute the sqlplus command on the master node to add the disks to ASM.

 

 

For a system with ASM Filter Driver configured, add the Oracle ASM disks as follows:

 

SQL> alter diskgroup /*+ _OAK_AsmCookie */ data add disk

AFD:SSD_E0_S05_2701256380P1’ name SSD_E0_S05_2701256380P1,

AFD:SSD_E0_S06_2701258544P1 name SSD_E0_S06_2701258544P1,

AFD:SSD_E0_S07_2701240428P1 name SSD_E0_S07_2701240428P1,

AFD:SSD_E0_S08_2701228196P1 name SSD_E0_S08_2701228196P1,

AFD:SSD_E0_S09_2701252584P1 name SSD_E0_S09_2701252584P1,

AFD:SSD_E0_S10_2701259504P1 name SSD_E0_S10_2701259504P1,

AFD:SSD_E0_S11_2701257468P1 name SSD_E0_S11_2701257468P1,

AFD:SSD_E0_S12_2701258588P1 name SSD_E0_S12_2701258588P1,

AFD:SSD_E0_S13_2701254148P1 name SSD_E0_S13_2701254148P1,

AFD:SSD_E0_S14_2701241428P1 name SSD_E0_S14_2701241428P1,

AFD:SSD_E0_S15_2701258144P1 name SSD_E0_S15_2701258144P1,

AFD:SSD_E0_S16_2701255896P1 name SSD_E0_S16_2701255896P1,

AFD:SSD_E0_S17_2701244644P1 name SSD_E0_S17_2701244644P1,

AFD:SSD_E0_S18_2701260436P1 name SSD_E0_S18_2701260436P1,

AFD:SSD_E0_S19_2701246564P1 name SSD_E0_S19_2701246564P1

power level 8;

 

 

SQL> alter diskgroup /*+ _OAK_AsmCookie */ reco add disk

AFD:SSD_E0_S05_2701256380P2 name SSD_E0_S05_2701256380P2,

AFD:SSD_E0_S06_2701258544P2 name SSD_E0_S06_2701258544P2,

AFD:SSD_E0_S07_2701240428P2 name SSD_E0_S07_2701240428P2,

AFD:SSD_E0_S08_2701228196P2 name SSD_E0_S08_2701228196P2,

AFD:SSD_E0_S09_2701252584P2 name SSD_E0_S09_2701252584P2,

AFD:SSD_E0_S10_2701259504P2 name SSD_E0_S10_2701259504P2,

AFD:SSD_E0_S11_2701257468P2 name SSD_E0_S11_2701257468P2,

AFD:SSD_E0_S12_2701258588P2 name SSD_E0_S12_2701258588P2,

AFD:SSD_E0_S13_2701254148P2 name SSD_E0_S13_2701254148P2,

AFD:SSD_E0_S14_2701241428P2 name SSD_E0_S14_2701241428P2,

AFD:SSD_E0_S15_2701258144P2 name SSD_E0_S15_2701258144P2,

AFD:SSD_E0_S16_2701255896P2 name SSD_E0_S16_2701255896P2,

AFD:SSD_E0_S17_2701244644P2 name SSD_E0_S17_2701244644P2,

AFD:SSD_E0_S18_2701260436P2 name SSD_E0_S18_2701260436P2,

AFD:SSD_E0_S19_2701246564P2 name SSD_E0_S19_2701246564P2

power level 8;



 

 

For a system without ASM Filter Driver configured, use the /dev/mapper name for ASM disks; e.g.,

SQL> alter diskgroup /*+ _OAK_AsmCookie */ data add disk

‘/dev/mapper/SSD_E0_S05_1399764284p1’ name SSD_E0_S05_1399764284p1,

  1. Confirm that the oak_storage_conf.xml file shows 24 on both nodes.

#cat /opt/oracle/oak/conf/oak_storage_conf.xml