Diggin’ into ASM Add Disk to DiskGroup operation

Recently we had to add storage to our X7-HA ODA.  This storage add includes a multi-step process, which is generally handled by the ODA OAK automation.  We simply added the disks in the slot, and the oakd dameon and workflow takes care of the device management.  The key things the oakd automation does is

  • Instantiates the disk device into the OS
  • Build partition tables
  • Create devmapper device names
  • updates the asmappl.config (***DO NOT TOUCH or EDIT THIS FILE..or apocalyptic things will HAPPEN **)
  • Generate a ASM disk add commands to added the disks to DATA and RECO diskgroups in the 80-20% pre-defined configuration.

This blog will cover the disk part and walk you through trace file.  I’ll blog about the automation stuff later

If you peak at the oakd.log or look in the ASM alert.log, you’ll see the actual command that gets executed.I have shown only the DATA dg disk add operation.    The RECO operation is the same but uses disk partitin P2.

My comments are inline:

SQL> ALTER DISKGROUP /*+ _OAK_AsmCookie */ DATA ADD DISK   <- This OAK ASM Cookie invokes ODA specific backend operations
‘AFD:SSD_E0_S05_2701246684P1’ NAME SSD_E0_S05_2701246684P1,   <– This is the list of disks that will be added to ODA
‘AFD:SSD_E0_S06_2701246400P1’ NAME SSD_E0_S06_2701246400P1,
‘AFD:SSD_E0_S07_2701243880P1’ NAME SSD_E0_S07_2701243880P1,
‘AFD:SSD_E0_S08_2701246408P1’ NAME SSD_E0_S08_2701246408P1,
‘AFD:SSD_E0_S09_2701257952P1’ NAME SSD_E0_S09_2701257952P1,
‘AFD:SSD_E0_S10_2701255368P1’ NAME SSD_E0_S10_2701255368P1,
‘AFD:SSD_E0_S11_2701247132P1’ NAME SSD_E0_S11_2701247132P1,
‘AFD:SSD_E0_S12_2701246568P1’ NAME SSD_E0_S12_2701246568P1,
‘AFD:SSD_E0_S13_2701251260P1’ NAME SSD_E0_S13_2701251260P1,
‘AFD:SSD_E0_S14_2701259824P1’ NAME SSD_E0_S14_2701259824P1,
‘AFD:SSD_E0_S15_2701255760P1’ NAME SSD_E0_S15_2701255760P1,
‘AFD:SSD_E0_S16_2701229772P1’ NAME SSD_E0_S16_2701229772P1,
‘AFD:SSD_E0_S17_2701232460P1’ NAME SSD_E0_S17_2701232460P1,
‘AFD:SSD_E0_S18_2701257420P1’ NAME SSD_E0_S18_2701257420P1,
‘AFD:SSD_E0_S19_2701253140P1’ NAME SSD_E0_S19_2701253140P1
kfdp_query: callcnt 78 grp 1 (DATA)                                  <– Its being added to DATA
kfdp_query: callcnt 79 grp 1 (DATA)
NOTE: Assigning number (1,5) to disk (AFD:SSD_E0_S05_2701246684P1) . <– Each disk is assigned a disk#
Disk 0x766f50e0 (1:5:AFD:SSD_E0_S05_2701246684P1) is being named (SSD_E0_S05_2701246684P1)
NOTE: Assigning number (1,6) to disk (AFD:SSD_E0_S06_2701246400P1)
2019-02-21 14:37:14.762*:kgfm.c@547: kgfmInitialize          <– Here the disks get initialized, using an array 
2019-02-21 14:37:14.763*:kgf.c@926: kgfArray_construct 0x7f2e648af208 len=0 nsegs=0
2019-02-21 14:37:14.763*:kgf.c@926: kgfArray_construct 0x7f2e648b75a0 len=0 nsegs=0
2019-02-21 14:37:14.763*:kgf.c@926: kgfArray_construct 0x7f2e69a676e8 len=0 nsegs=0
…<deleted repeated lines>

<– kgfmReadOak reads in OAK configuration information into memory structures

2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: max_disk_count is 100
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [max_disk_count] = [100]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=0->1 nsegs=0->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: appliance_name is ODA
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [appliance_name] = [ODA]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=1->2 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: diskstring is AFD:*
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [diskstring] = [AFD:*]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=2->3 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: file_version is 2
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [file_version] = [2]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=3->4 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: oda_version is 3
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [oda_version] = [3]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=4->5 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: jbod_count is 1
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [jbod_count] = [1]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=5->6 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: jbod_slot_count is 24 <– all 24 slots in the ODA are filled
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [jbod_slot_count] = [24]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=6->7 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: data_slot_count is 20 . <– 20 disks for DATA DG
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [data_slot_count] = [20]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=7->8 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: reco_slot_count is 20
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [reco_slot_count] = [20]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=8->9 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: redo_slot_count is 4       <– 4 disks for REDO DG
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [redo_slot_count] = [4]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=9->10 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: max_missing is 0
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [max_missing] = [0]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=10->11 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: min_partners is 2
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [min_partners] = [2]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=11->12 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: agent_sql_identifier is /*+ _OAK_AsmCookie
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [agent_sql_identifier] = [/*+ _OAK_AsmCookie ]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=12->13 nsegs=1->1
2019-02-21 14:37:14.763*:kgfm.c@1773: kgfmReadOak: rdbms_compatibility is 12.1.0.2
2019-02-21 14:37:14.763*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [rdbms_compatibility] = [12.1.0.2]
2019-02-21 14:37:14.763*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=13->14 nsegs=1->1
2019-02-21 14:37:14.764*:kgfm.c@1773: kgfmReadOak: asm_compatibility is 12.2.0.1
2019-02-21 14:37:14.764*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [asm_compatibility] = [12.2.0.1]
2019-02-21 14:37:14.764*:kgf.c@1025: kgfArray_grow 0x7f2e69a77e90 len=14->15 nsegs=1->1
2019-02-21 14:37:14.764*:kgfm.c@1773: kgfmReadOak: _asm_hbeatiowait is 100
2019-02-21 14:37:14.764*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [_asm_hbeatiowait] = [100]

This next section starts the disk association and partnership mapping

[

2019-02-21 14:37:14.769*:kgfm.c@2857: kgfmAddDisk: enc=0 slot=23 part=1 path=AFD:SSD_E0_S23_2181135920P1
2019-02-21 14:37:14.769*:kgf.c@1025: kgfArray_grow 0x7f2e648af208 len=0->1 nsegs=0->1
2019-02-21 14:37:14.769*:kgf.c@1025: kgfArray_grow 0x7f2e69a63530 len=0->24 nsegs=0->1
2019-02-21 14:37:14.769*:kgfm.c@1968: kgfmReadOak: disk 23 partners [ 2019-02-21 14:37:14.769*:kgfm.c@1970: 22 2019-02-21 14:37:14.769*:kgfm.c@1970: 21 2019-02-21 14:37:14.769*:kgfm.c@1970: 20 2019-02-21 14:37:14.769*:kgfm.c@1971: ]

[

2019-02-21 14:37:14.770*:kgfm.c@2857: kgfmAddDisk: enc=0 slot=22 part=1 path=AFD:SSD_E0_S22_2181131148P1
2019-02-21 14:37:14.770*:kgf.c@1025: kgfArray_grow 0x7f2e648af208 len=1->2 nsegs=1->1
2019-02-21 14:37:14.770*:kgfm.c@1968: kgfmReadOak: disk 22 partners [ 2019-02-21 14:37:14.770*:kgfm.c@1970: 23 2019-02-21 14:37:14.770*:kgfm.c@1970: 21 2019-02-21 14:37:14.770*:kgfm.c@1970: 20 2019-02-21 14:37:14.770*:kgfm.c@1971: ]

This repeated for every disk in the ODA

….

Define DiskGroup Attributes –

2019-02-21 14:37:14.774*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [agent_sql_identifier] = [/*+ _OAK_AsmCookie ]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [oda_version] = [1]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [jbod_count] = [1]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [jbod_slot_count] = [24]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [data_slot_count] = [20]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [reco_slot_count] = [20]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [redo_slot_count] = [4]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [diskstring] = [(null)]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [appliance_name] = [ODA]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [agent_sql_identifier] = [/* ASM Appliance Agent */]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [asm_compatibility] = [11.2.0.3]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [rdbms_compatibility] = [11.2.0.2]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [max_disk_count] = [100]
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 0 [_asm_hbeatiowait] = [0]
2019-02-21 14:37:14.775*:kgfm.c@2643: kgfmStaticCardinatliy: odaver 3 njbods 1 tmpl 1 card0 3 card1 3
2019-02-21 14:37:14.775*:kgfm.c@2643: kgfmStaticCardinatliy: odaver 3 njbods 1 tmpl 1 card0 5 card1 5
2019-02-21 14:37:14.775*:kgfm.c@2954: kgfmAddAttribute: tmpl 1 [max_missing] = [0]

NOTE: running client discovery for group 1 (reqid:7106731798842955164)

<– AT this point the disks are added and  client (database) re-discovers the diskgroup

*** 2019-02-21T14:37:20.381245-05:00
NOTE: running client discovery for group 1 (reqid:7106731798842926578)

*** 2019-02-21T14:37:21.421653-05:00
kfdp_updateReconf(): callcnt 80 grp 1 scope 0x204 .  <–Once you see kfdp_updateReconf …its DONE!
NOTE: group 1 PST updated.
PST verChk [reconf]: id=46860409, grp=1, requested=9 at 02/21/2019 14:37:21
PST verChk [reconf]: id=46860409 grp=1 completed at 02/21/2019 14:37:21