ODA Patching/Upgrade Triage

This blog will define the appropriate logs, files and command output to collect when triaging ODA related patching issues

Recently we attempted to upgrade from 12.2.1.2 to 12.2.1.3 on ODA X7.  We did the usual as part of the upgrade process.

unzip p27648057_122130_Linux-x86-64_1of3.zip

ls -l  oda-sm-12.2.1.3.0-180504-server*

/opt/oracle/dcs/bin/odacli update-repository -f oda-sm-12.2.1.3.0-180504-server1of3.zip, p27648057_122130_Linux-x86-64_2of3.zip,

p27648057_122130_Linux-x86-64_3of3.zip

/opt/oracle/dcs/bin/odacli update-dcsagent -v 12.2.1.3.0

/opt/oracle/dcs/bin/odacli describe-job -i jobid

rpm -qa |grep dcs-agent

dcs-agent-18.1.3.0.0_LINUX.X64_180504-86.x86_64

/opt/oracle/dcs/bin/odacli update-server -v 12.2.1.3.0

The update-server failed according to this log:

{

    “updatedTime” : 1530126568790,

    “startTime” : 1530126562856,

    “endTime” : 1530126568788,

    “taskId” : “TaskZJsonRpcExt_225”,

    “status” : “Success”,

    “taskResult” : “Successfully created the yum repos for patchingos”,

    “taskName” : “Creating repositories using yum”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_224”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Info”

  }, {

    “updatedTime” : 1530126894836,

    “startTime” : 1530126568799,

    “endTime” : 1530126894834,

    “taskId” : “TaskZJsonRpcExt_228”,

    “status” : “Success”,

    “taskResult” : “Successfully updated the OS”,

    “taskName” : “Applying OS Patches”,

    “taskDescription” : null,

    “parentTaskId” : “TaskParallel_227”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Info”

  },

{

    “updatedTime” : 1530126899457,

    “startTime” : 1530126894980,

    “endTime” : 1530126899455,

    “taskId” : “TaskZJsonRpcExt_236”,

    “status” : “Success”,

    “taskResult” : “Successfully updated the Firmware Disk”,

    “taskName” : “Applying Firmware Disk Patches”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_235”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Info”

  },

{

    “updatedTime” : 1530127898145,

    “startTime” : 1530127248252,

    “endTime” : 1530127898144,

    “taskId” : “TaskSequential_121”,

    “status” : “Failure”,

    “taskResult” : “DCS-10001:Internal error encountered:  apply patch using OpatchAuto on node odanode1.”,

    “taskName” : “task:TaskSequential_121”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_256”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Error”

  },

….

{

    “updatedTime” : 1530127898142,

    “startTime” : 1530127463976,

    “endTime” : 1530127898141,

    “taskId” : “TaskSequential_162”,

    “status” : “Failure”,

    “taskResult” : “DCS-10001:Internal error encountered:  apply patch using OpatchAuto on nodeodanode1.”,

    “taskName” : “task:TaskSequential_162”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_121”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Error”

  }, {

    “updatedTime” : 1530127898139,

    “startTime” : 1530127463979,

    “endTime” : 1530127898137,

    “taskId” : “TaskZJsonRpcExt_163”,

    “status” : “Failure”,

    “taskResult” : “DCS-10001:Internal error encountered:  apply patch using OpatchAuto on node odanode1″,

    “taskName” : “clusterware upgrade”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_162”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Error”

  } ],

  “createTimestamp” : 1530126459722,

  “resourceList” : [ ],

  “description” : “Server Patching”

}

When opening an SR make sure you collected the following infor for support.  This will minimize the ‘backn’forth’-ness and reduce mean time to resolve.

Please provide the output from the commands below:
1.

odacli describe-component

System Version 

—————

12.2.1.3.0

Component                                Installed Version    Available Version   

—————————————- ——————– ——————–

OAK                                       12.2.1.3.0            up-to-date          

GI                                        12.2.0.1.171017       12.2.0.1.180116     

DB                                        12.2.0.1.171017       12.2.0.1.180116     

DCSAGENT                                  18.1.3.0.0            up-to-date          

ILOM                                      4.0.0.28.r121827      4.0.0.22.r120818    

BIOS                                      41017600              41017100            

OS                                        6.9                   up-to-date          

FIRMWARECONTROLLER                        QDV1RE14              up-to-date          

ASR                                       5.7.7                 up-to-date          

# odacli describe-latestpatch

componentType   availableVersion    

————— ——————–

gi              12.2.0.1.180116     

gi              12.2.0.1.180116     

db              12.2.0.1.180116     

db              12.1.0.2.180116     

db              11.2.0.4.180116     

oak             12.2.1.3.0          

oak             12.2.1.3.0          

asr             5.7.7               

ilom            4.0.0.22.r120818    

os              6.9                 

hmp             2.4.1.0.9           

bios            41017100            

firmwarecontroller 13.00.00.00         

firmwarecontroller 4.650.00-7176       

firmwarecontroller kpyagr3q            

firmwarecontroller qdv1re14            

firmwaredisk    0r3q                

firmwaredisk    a122                

firmwaredisk    a122                

firmwaredisk    a374                

firmwaredisk    c122                

firmwaredisk    c122                

firmwaredisk    c376                

firmwaredisk    0112                

dcsagent        18.1.3.0.0          

2.  Provide/upload the following most recent configuration and diagnostic details./log files: 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchautodb/systemconfig* 

/tmp/opatchAutoAnalyzePatch.log 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchauto/core/opatch/opatch<date>.log 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchautodb/systemconfig<date>.log 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchauto/opatchauto<date>.log

/u01/app/oracle/crsdata/odanode1/crsconfig/crspatch_<hostname><date>.log 

Exadata Cloud – Post Provisioning Exadata Configuration – Part 1

Post Provisioning Exadata Configuration – Part1

After an Exadata is provisioned, ther are several post provisioning steps that need to be executed in order to allow system automation such as patching, backups, and infrastructure updates. This document will describe these steps.

All the traffic in an Exadata DB System is, by default, routed through the client network. To route backup traffic to the backup interface (BONDETH1), a static route needs to be created on each of the compute nodes in the cluster.

First identify the gateway configured for the BONDETH1 interface.

grep GATEWAY /etc/sysconfig/network-scripts/ifcfg-bondeth1 |awk -F”=” ‘{print $2}’

10.232.35.1

Review current /etc/sysconfig/network-scripts/route-bondeth1

cat /etc/sysconfig/network-scripts/route-bondeth1

10.232.35.0/24 dev bondeth1 table 211

default via 10.232.35.1 dev bondeth1 table 211

Create a new static rule for BONDETH1 and update route-bondeth1 with the following entries (per Cloud region)

Phoenix (PHX) region:

ADDRESS0=129.146.0.0

NETMASK0=255.255.0.0

GATEWAY0=10.232.35.1

 Ashburn (IAD) region):

ADDRESS0=129.213.0.0

NETMASK0=255.255.0.0

GATEWAY0=10.232.35.1

Restart the interface.

[root@dbsys ~]# ifdown bondeth1; ifup bondeth1; 


Once this change is done, you should see a new entry in the route table:

[root@~ network-scripts]# netstat -rn

Kernel IP routing table

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface

0.0.0.0         10.232.34.1     0.0.0.0         UG        0 0          0 bondeth0

10.232.34.0     0.0.0.0         255.255.255.0   U         0 0          0 bondeth0

10.232.35.0     0.0.0.0         255.255.255.0   U         0 0          0 bondeth1

129.146.0.0     10.232.35.1     255.255.0.0     UG        0 0          0 bondeth1

169.254.200.0   0.0.0.0         255.255.255.252 U         0 0          0 eth0

192.168.132.0   0.0.0.0         255.255.252.0   U         0 0          0 clib1

192.168.132.0   0.0.0.0         255.255.252.0   U         0 0          0 clib0

192.168.136.0   0.0.0.0         255.255.248.0   U         0 0          0 stib0

192.168.136.0   0.0.0.0         255.255.248.0   U         0 0          0 stib1

 

Exadata Cloud – Post Provisioning View of the system

Review of Exadata Deployment

Once the Exadata provisioning process completes (which takes around 4-5hrs for a ½ rack).  We explore to see what gets deployed:

$ cat/etc/oratab

OCITEST:/u02/app/oracle/product/12.2.0/dbhome_2:Y

+ASM1:/u01/app/12.2.0.1/grid:N       # line added by Agent

 

[grid@phxdbm-o3eja1 ~]$ olsnodes -n

phxdbm-o3eja1 1

phxdbm-o3eja2 2

phxdbm-o3eja3 3

phxdbm-o3eja4 4

 

[grid@phxdbm-o3eja1 ~]$ cat /var/opt/oracle/creg/OCITEST.ini | grep nodelist

nodelist=phxdbm-o3eja1 phxdbm-o3eja2 phxdbm-o3eja3 phxdbm-o3eja4

 

[grid@phxdbm-o3eja1 ~]$ crsctl stat res -t

—————————————————————————–

Name           Target  State        Server                   State details

—————————————————————————–

Local Resources

—————————————————————————–

ora.ACFSC1_DG1.C1_DG11V.advm

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

ora.ACFSC1_DG1.C1_DG12V.advm

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

ora.ACFSC1_DG1.dg

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE    ora.ACFSC1_DG2.C1_DG2V.advm

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE    ora.ACFSC1_DG2.dg

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE    ora.ASMNET1LSNR_ASM.lsnr

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

ora.DATAC1.dg

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE . ora.DBFS_DG.dg

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

ora.LISTENER.lsnr

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

ora.RECOC1.dg

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE . ora.acfsc1_dg1.c1_dg11v.acfs

ONLINE  ONLINE       phxdbm-o3eja1            mounted on /scratch/acfsc1_dg1,STABLE

ONLINE  ONLINE       phxdbm-o3eja2            mounted on /scratch/acfsc1_dg1,STABLE

ONLINE  ONLINE       phxdbm-o3eja3            mounted on /scratch/acfsc1_dg1,STABLE

ONLINE  ONLINE       phxdbm-o3eja4            mounted on /scratch/acfsc1_dg1,STABLE

ora.acfsc1_dg1.c1_dg12v.acfs

ONLINE  ONLINE       phxdbm-o3eja1            mounted on /u02/app_acfs,STABLE

ONLINE  ONLINE       phxdbm-o3eja2            mounted on /u02/app_acfs,STABLE

ONLINE  ONLINE       phxdbm-o3eja3            mounted on /u02/app_acfs,STABLE

ONLINE  ONLINE       phxdbm-o3eja4            mounted on /u02/app_acfs,STABLE

ora.acfsc1_dg2.c1_dg2v.acfs

ONLINE  ONLINE       phxdbm-o3eja1            mounted on /var/opt/oracle/dbaas_acfs,STABLE

ONLINE  ONLINE       phxdbm-o3eja2            mounted on /var/opt/oracle/dbaas_acfs,STABLE

ONLINE  ONLINE       phxdbm-o3eja3            mounted on /var/opt/oracle/dbaas_acfs,STABLE

ONLINE  ONLINE       phxdbm-o3eja4            mounted on /var/opt/oracle/dbaas_acfs,STABLE

ora.net1.network

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

ora.ons

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

ora.proxy_advm

ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ONLINE  ONLINE       phxdbm-o3eja4            STABLE

—————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1        ONLINE  ONLINE       phxdbm-o3eja2            STABLE

ora.LISTENER_SCAN2.lsnr

1        ONLINE  ONLINE       phxdbm-o3eja3            STABLE

ora.LISTENER_SCAN3.lsnr

1        ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ora.asm

1        ONLINE  ONLINE       phxdbm-o3eja1            Started,STABLE

2        ONLINE  ONLINE       phxdbm-o3eja2            Started,STABLE

3        ONLINE  ONLINE       phxdbm-o3eja3            Started,STABLE

4        ONLINE  ONLINE       phxdbm-o3eja4            Started,STABLE

ora.cvu

1        ONLINE  ONLINE       phxdbm-o3eja1            STABLE

ora.ocitest.db

1        ONLINE  ONLINE       phxdbm-o3eja1            Open,HOME=/u02/app/oracle/product/12.2.0/dbhome_2,STABLE

2        ONLINE  ONLINE       phxdbm-o3eja2            Open,HOME=/u02/app/o

racle/product/12.2.0

/dbhome_2,STABLE

3        ONLINE  ONLINE       phxdbm-o3eja3            Open,HOME=/u02/app/oracle/product/12.2.0

/dbhome_2,STABLE

4        ONLINE  ONLINE       phxdbm-o3eja4            Open,HOME=/u02/app/oracle/product/12.2.0

/dbhome_2,STABLE

ora.phxdbm-o3eja1.vip

1        ONLINE  ONLINE       phxdbm-o3eja1            STABLE ora.phxdbm-o3eja2.vip

1        ONLINE  ONLINE       phxdbm-o3eja2            STABLE ora.phxdbm-o3eja3.vip

1        ONLINE  ONLINE       phxdbm-o3eja3            STABLE ora.phxdbm-o3eja4.vip

1        ONLINE  ONLINE       phxdbm-o3eja4            STABLE ora.qosmserver

1        OFFLINE OFFLINE                               STABLE ora.scan1.vip

1        ONLINE  ONLINE       phxdbm-o3eja2            STABLE ora.scan2.vip

1        ONLINE  ONLINE       phxdbm-o3eja3            STABLE ora.scan3.vip

1        ONLINE  ONLINE       phxdbm-o3eja1            STABLE

—————————————————————————–

[grid@phxdbm-o3eja1 ~]$ asmcmd lsct

DB_Name  Status     Software_Version  Compatible_version  Instance_Name   Disk_Group

+APX     CONNECTED        12.2.0.1.0          12.2.0.1.0  +APX1   ACFSC1_DG1

+APX     CONNECTED        12.2.0.1.0          12.2.0.1.0  +APX1   ACFSC1_DG2

+ASM     CONNECTED        12.2.0.1.0          12.2.0.1.0  +ASM1   DATAC1

+ASM     CONNECTED        12.2.0.1.0          12.2.0.1.0  +ASM1    DBFS_DG

OCITEST  CONNECTED        12.2.0.1.0          12.2.0.0.0  OCITEST1 DATAC1

OCITEST  CONNECTED        12.2.0.1.0          12.2.0.0.0  OCITEST1  RECOC1

_OCR     CONNECTED         –                  phxdbm-o3eja1.client.phxexadata.oraclevcn.com  DBFS_DG

yoda     CONNECTED        12.2.0.1.0          12.2.0.0.0  yoda1    DATAC1

yoda     CONNECTED        12.2.0.1.0          12.2.0.0.0  yoda1    RECOC1

 

[root@phxdbm-o3eja1 ~]# df -k

Filesystem           1K-blocks     Used Available Use% Mounted on

/dev/mapper/VGExaDb-LVDbSys1

24639868  3878788  19486408  17% /

tmpfs                742619136  2465792 740153344   1% /dev/shm

/dev/xvda1              499656    26360    447084   6% /boot

/dev/mapper/VGExaDb-LVDbOra1

20511356   719324  18727072   4% /u01

/dev/xvdb             51475068  9757380  39079864  20% /u01/app/12.2.0.1/grid

/dev/xvdc             51475068  9302820  39534424  20% /u01/app/oracle/product/12.1.0.2/dbhome_1

/dev/xvdd             51475068  8173956  40663288  17% /u01/app/oracle/product/12.2.0.1/dbhome_1

/dev/xvde             51475068  6002756  42834488  13% /u01/app/oracle/product/11.2.0.4/dbhome_1

/dev/xvdg            206293688 19751360 176040184  11% /u02

/dev/asm/c1_dg12v-186

459276288  1067008 458209280   1% /u02/app_acfs

/dev/asm/c1_dg11v-186

229638144   611488 229026656   1% /scratch/acfsc1_dg1

/dev/asm/c1_dg2v-341 228589568 26597644 201991924  12% /var/opt/oracle/dbaas_acfs

 

Oracle Homes are created and mounted, though for IQN we will only be using 12.2, 12.1.0.2, and 11.2.0.4 [interim].

The   following are Exadata specific filesystems and use cases
/scratch/acfs1_dg1             –staging Exadata

/u02/app_acfs.                    – User filesystem for applications (currently empty)

/var/opt/oracle/dbaas_acfs.  –  Binary and image repository for all Exadata patching and enablement

Exadata Cloud Deployment and Considerations

I recently did a presentation and wipe-board session on Exadata Cloud deployment.  As part of that engagment, I did a small write-up on this topic.  This is a series of blogs that reflects the presentation:

Cloud Exadata Network and Platform Configuration

 Exadata DB Systems are offered in quarter rack, half rack or full rack configurations, and each configuration consists of compute nodes and storage servers. The compute nodes are each configured as a Virtual Machine (VM).

Key Operational characteristics of Exadata Cloud

  • Admins have root privileges for the compute node VMs. Thus 3rd party software can be installed, however, only supported Oracle DB versions and rpms should be implemented.

 

  • Admins do not have administrative access to the Exadata infrastructure components, including the physical compute node hardware, network switches, power distribution units (PDUs), integrated lights- out management (ILOM) interfaces, or the Exadata Storage Servers, which are all administered by Oracle.

 

  • Admins have full administrative privileges for your databases. However, application users should connect to databases via Oracle Net Services.

 

  • Admins are responsible for database administration tasks such as creating tablespaces and managing database users.

 

  • Admins should define how ssh keys will managed for users that will need compute node access.

 

 

 

 

 

 

 

 

 

 

 

Provisioning Exadata Pre-reqs

The following are network pre-reqs for provisioning Cloud Exadata DB Systems

Subnets

  • Require two separate VCN subnets: client subnet for user data traffic and backup subnet for backup traffic.
  • Define both the client subnet and the backup subnet as public subnets. Exadata requires a public subnet to support backup of the database to the Object Store.
  • Do not use a subnet that overlaps with 192.168.128.0/20. This restriction applies to both the client subnet and backup subnet.
  • Oracle requires that you use a VCN Resolver for DNS name resolution for the client subnet. It automatically resolves the Swift endpoints required for backing up databases, patching, and updating the cloud tooling on an Exadata DB System.

At the completion of the provisioning, you should have the following configured:

 

 

 

 

 

 

Security Lists and Routing

  • Each VCN subnet has a default security list that contains a rule to allow TCP traffic on destination port 22 (SSH) from source 0.0.0.0/0 and any source port. Properly configure the security list ingress and egress rules.
  • The OneCommand configuration enables TCP and ICMP traffic between all nodes and all ports in the respective subnet for client and backup subnets
  • Exadata DB System’s cloud network (VCN) must be configured with an internet gateway. Add a route table rule to open the access to the Object Storage Service Swift endpoint on CIDR 0.0.0.0/0.
  • Update the backup subnet’s security list to disallow any access from outside the subnet and allow egress traffic for TCP port 443 (https) on CIDR Ranges 129.146.0.0/16 (Phoenix region), 129.213.0.0/16 (Ashburn region)

Enable a route table with an entry that includes a Internet Gateway.  This will enable remote ssh access to the Exadata nodes

 

 

 

 

 

 

 

Provisioning Exadata

Service Console – Provision Exadata

Below are screenshot views that illustrate the provisioning of Exadata

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cloud Exadata Storage Configuration

Exadata Storage Servers use the following ASM disk groups:

DATA diskgroup – for the storage of Oracle Data base datafiles.

RECO diskgroup – primarily used for storing files related to backup and recovery, such as RMAN backups and archived redo log files.  Depending how admins choose to provision for backups on Exadata storage

approximately 40% of the available storage space is allocated to the DATA disk group and approximately 60% is allocated to the RECO disk group.

Provision for backups on Exadata storage, approximately 80% of the available storage space is allocated to the DATA disk group and approximately 20% is allocated to the RECO disk group.

DBFS and ACFS diskgroups are system diskgroups that support various operational purposes. The DBFS disk group is primarily used to store the shared Clusterware files (Oracle Cluster Registry and voting disks), while the ACFS disk groups are primarily used to store Oracle Database binaries, staging directories and metadata.

 

ACFS Snapshot – A Walk Through

This blog explores some of the new 12.2 ACFS features.  We will walk through the ACFS snapshot process flow:

 

[oracle@oracle122 log]$ acfsutil snap info /acfsmounts/acfsdata/

snapshot name:               just_before_load

snapshot location:           /acfsmounts/acfsdata/.ACFS/snaps/just_before_load

RO snapshot or RW snapshot:  RO

parent name:                 /acfsmounts/acfsdata/

snapshot creation time:      Wed Mar 22 20:36:09 2017

storage added to snapshot:   8650752   (   8.25 MB )

number of snapshots:  1

snapshot space usage: 8704000  (   8.30 MB )

[oracle@oracle122 log]$ du -sk .

18292  .


[oracle@oracle122 log]$ acfsutil snap create -w -p just_before_load just_about_batch_upload /acfsmounts/acfsdata/

acfsutil snap create: Snapshot operation is complete.

[oracle@oracle122 log]$ acfsutil snap info /acfsmounts/acfsdata

snapshot name:               just_before_load

snapshot location:           /acfsmounts/acfsdata/.ACFS/snaps/just_before_load

RO snapshot or RW snapshot:  RO

parent name:                 /acfsmounts/acfsdata

snapshot creation time:      Wed Mar 22 20:36:09 2017

storage added to snapshot:   8650752   (   8.25 MB )

snapshot name:               just_about_batch_upload

snapshot location:           /acfsmounts/acfsdata/.ACFS/snaps/just_about_batch_upload

RO snapshot or RW snapshot:  RW

parent name:                 just_before_load

snapshot creation time:      Wed Mar 22 20:42:56 2017

storage added to snapshot:   8650752   (   8.25 MB )

root@oracle122 ~]# acfsutil compress on /acfsmounts/acfsdata/log/wtf

acfsutil compress on: ACFS-05518: /acfsmounts/acfsdata/log/wtf is not an ACFS mount point

[root@oracle122 ~]# acfsutil compress info /acfsmounts/acfsdata/log/wtf

The file /acfsmounts/acfsdata/log/wtf is not compressed.

[root@oracle122 ~]# acfsutil compress info /acfsmounts/acfsdata/log/nitin

nitin             nitin_compressed 

[root@oracle122 ~]# acfsutil compress info /acfsmounts/acfsdata/log/nitin_compressed

Compression Unit size: 32768

Disk storage used:   (  60.00 KB )

Disk storage saved:  (   7.75 MB )

Storage used is 1% of what the uncompressed file would use.

File is not scheduled for asynchronous compression.

oracle@oracle122 log]$ ls -l lastlog*

-rw-r--r--. 1 oracle oracle 145708 Mar 22 12:07 lastlog

-rw-r--r--. 1 oracle oracle 145708 Mar 23 05:49 lastlog_compressed

[oracle@oracle122 log]$

[root@oracle122 ~]# acfsutil compress info /acfsmounts/acfsdata/log/lastlog_compressed

Compression Unit size: 32768

Disk storage used:   (  32.00 KB )

Disk storage saved:  ( 110.29 KB )

Storage used is 22% of what the uncompressed file would use.

File is not scheduled for asynchronous compression.

If you are curious about the other snapshop options... then look below !!

[oracle@oracle122 log]$ acfsutil snap -h

 Command Subcmd    Arguments

--------------- --------- ------------------------------------------

snap create    [-w|-r|-c] [-p parent_snap_name] <snap_name> <mountpoint>

snap create    [-w]                      - create a writeable snapshot

snap create    [-r]                      - create a read-only snapshot

snap create                                This is the default behavior

snap create    [-c]                      - create a writable snapshot of a

snap create                                snap duplicate target

snap create    [-p parent_snap_name]     - create a snapshot from a snapshot

snap delete    <snap_name> <mountpoint> - delete a file system snapshot

snap rename    <old_snap_name> <new_snap_name> <mountpoint>

snap rename                             - rename a file system snapshot

snap convert   -w|-r <snap_name> <mountpoint>

snap convert   -w                       - convert to a writeable snapshot

snap convert   -r                       - convert to a read-only snapshot

snap info      [-t] [<snap_name>] <mountpoint>

snap info                    - get information about snapshots

snap info      [-t]          - display family tree starting at next name given

snap info      [<snap_name>] - snapshot name

snap info      <mountpoint>  - mount point

snap remaster  {<snap_name> | -c} <volume_path>

snap remaster                           - make the specified snapshot

snap remaster                             the master file system.  The

snap remaster                             current master and all other

snap remaster                             snapshots will be deleted.

snap remaster                             WARNING: This operation cannot

snap remaster                             be reversed.  Admin privileges

snap remaster                             are required.  The file system

snap remaster                             must be unmounted on all nodes.

snap remaster                             The file system must not have

snap remaster                             Replication running.

snap remaster  [-c]                     - Continue an interrupted snapshot

snap remaster                             remastering.  Use the -c option,

snap remaster                             instead of the <snap_name>, to

 snap remaster                             complete an interrupted

snap remaster                             snapshot remastering.

snap remaster  [-f]                     - Force the snapshot remastering.

 snap duplicate apply     [-b] [-d {0..6}] [<snap_name>] <mountpoint>

 snap duplicate apply     -b                       - maintain backup snapshot

 snap duplicate apply     [-d {0..6}]              - set trace level for debugging

 snap duplicate apply     [<snap_name>]            - target snapshot

 snap duplicate apply     <mountpoint>             - mount point for target site

 snap duplicate create    [-r] [-i oldsnapname] [-d {0..6}] <newsnapname> <mountpoint>

 snap duplicate create    [-r]              - restart of data stream

 snap duplicate create    [-p parentsnap]   - parent snap for base site

 snap duplicate create    [-i oldsnapname]  - old snapshot name

 snap duplicate create    [-d {0..6}]       - set trace level for debugging

 snap duplicate create    <newsnapname>     - new snapshot name

 snap duplicate create    <mountpoint>      - mount point for base site

 snap quota     [[-|+]nnn[K|M|G|T|P]]<snap_name> <mountpoint>

 snap quota                              - set quota for snapshot

 

Grid Infrastructure and RAC 12.2 New Features – a Recap

The following list illustrates the new 12.2 Oracle RAC and Grid Infrastructure. This is a personal list which “I believe to be the most interesting.” I apologize to the RAC Dev team if I left out any features.

Streamlined Grid Infrastructure Installation

12.2 Grid Infrastructure software is available as an image file for download and installation. The key objective of this feature was to enable a simpler and quicker installation of Grid Infrastructure. Administrators simply prep the system by creating a new Grid home directory, appropriate users, permissions and kernel settings. Once completed, Admins extract the image file into the newly-created Grid home, and execute the gridsetup.sh script to invoke setup wizard to register the Oracle Grid Infrastructure stack with Oracle inventory. This installation approach can be used for Oracle Grid Infrastructure for Cluster and Standalone Servers configurations. This new software installation will improve large scale deployment automation as well as deployment of customized images, Patch Set Updates (PSUs) and patches.

Real Application Clusters Reader Nodes

In 12.2, Oracle extended the capability of Flex Clusters by introducing Reader nodes. Reader nodes are Leaf nodes (in a Flex Cluster) that run read-only RAC database instances. The Reader nodes are not affected by RAC reconfigurations, caused by node evictions or other cluster node membership changes, as long as the Hub Node, to which it is connected, is part of the cluster. Reader Nodes allows users to create huge reader farms (up to 64 reader nodes per Hub Node), thus enabling massive parallel processing. In this architecture, updates to the read/write instances (running on Hub nodes) are immediately propagated to the read-only instances on the Leaf Nodes, where they can be used for online reporting or instantaneous queries. Users can create services to direct queries to read-only instances running on reader nodes.

Service-Oriented Buffer Cache Access

RAC Services, which are used to allocate and distribute workloads across RAC instances, are the cornerstone of RAC workload management. There is a strong relationship between a RAC Service, a specific workload, and the database object it accesses. With 12.2 RAC, a Service- oriented buffer cache feature was introduced to improve scale and performance, by optimizing instance and node-buffer cache affinity. This is done by caching or pre-warming instances with data blocks for objects accessed where a service is expected to run.

Twelve Days of 12.2

Server Weight-Based Node Eviction

When there is a spilt-brain, or when a node eviction decision must be made, traditionally the decision was based on age, or duration of the nodes, in the cluster; i.e., nodes with a large uptime in the cluster will survive. In 12.2 RAC, Server weight-based node eviction uses a more intelligent, tie-breaker mechanism to evict a particular node or a group of nodes from a cluster. The Server Weight-based node eviction feature introspects the current load on those servers as part of the decision. Two principle mechanisms, a system inherent automatic mechanism and a user input-based mechanism is used to offer and provide guidance.

Load-Aware Resource Placement

Load-aware resource placement, prevents overloading a server with more database instances than the server is capable of running. The metrics used to determine whether an application can be started on a given server, is based on the expected resource consumption of the application, as well as the capacity of the server in terms of CPU and memory. Administrators can define database resources such as CPU (cpu_count) and memory (memory_target) to Clusterware. Clusterware uses this information to place the database instances only on servers that meet a sufficient number of CPUs, amount of memory or both.

srvctl modify database -db testdb -cpucount 8 -memorytarget 64g

Hang Manager

The Hang Manager features first became available in 11gR1. In this initial version, Hang Manager evaluated and identified system hangs, then dumped the relevant information, “wait for graph,” into a trace file. In 12.2, Hang Manager takes action and attempts to resolve the system hang. An ORA-32701 error message is logged in the alert log to reflect the hang resolution. Hang Manager also runs in both single-instance and Oracle RAC database instances. With Hang Manager, it is constantly aware of processes running in reader nodes instances, and checks whether any of these processes are blocking progress on Hub Nodes to take action, if possible.

Separation of Duty for Administering RAC Clusters

12.2 RAC introduces a new administrative privilege called SYSRAC. This privilege is used by the Clusterware agent, and removes the need to use SYSDBA privilege for RAC administrative tasks, thus reducing the reliance on SYSDBA on production systems. Note, SYSRAC privilege is the default mode for connecting to the database by Clusterware agent; e.g, when executing RAC utilities such as SRVCTL.

Rapid Home Provisioning of Oracle Software

Rapid Home Provisioning enables you to create clusters, provision, patch, and upgrade Oracle Grid Infrastructure and Oracle Database homes. It also provisions 11.2 Clusters, applications, and middleware using Rapid Home Provisioning.

Extended Clusters

In 12.2 GI Administrators can create an extended RAC cluster across two, or more, geographically separate sites. Note, each site will include a set of servers with its own storage. If a site fails, the other site acts as an active standby. 12.2 Extended Clusters can be built on initial installation or be converted from an existing (non-Flex ASM) cluster, using the ConvertToExtended script.

De-support of OCR and Voting Files on Shared Filesystem

In Grid Infrastructure 12.2, the placement of Oracle Clusterware files: the Oracle Cluster Registry (OCR), and the Voting Files, directly on a shared file system is desupported. Only ASM or NFS is supported. If you need to use a supported shared file system, either a Network File System, or a shared cluster file system instead of native disk devices, then you must create Oracle ASM disks on supported network file systems that you plan to use for hosting Oracle Clusterware files before installing Oracle Grid Infrastructure. You can then use the Oracle ASM disks in an Oracle ASM disk group to manage Oracle Clusterware files. If your Oracle Database files are stored on a shared file system, then you can continue to use shared file system storage for database files, instead of moving them to Oracle ASM storage.

ACFS 12.2 New Features – a Recap

Oracle Automatic Storage Management Cluster File System (ACFS) made it’s debut with Oracle 11.2. Many DBAs are not aware of the vast features that are available with ACFS. With each release and update to Oracle, significant enhancements have been made. With Oracle Database 12c Release 2, new feature/functionality was made to ACFS.

Snapshot Enhancements

In Oracle 12.2, Oracle extends ACFS snapshot functionality and further simplifies file system snapshot operations. The following are a few of the key new features with snapshots:

Admins can now, if needed, impose quotas to snapshots to limit amount of write operations that can be done on a snapshot. Quotas can be set on the snapshot level. Oracle also provides the capability to rename an existing ACFS snapshot, to allow more user-friendly names.

When we delete a snapshot with the “acfsutil snap delete snapshot mount_point” command, we can force a delete, even if there are open files.

There are several new capabilities with snapshot re-mastering and duplication. The new ACFS snapshot remaster capability allows for a snapshot in the snapshot registry to become the primary file system. ACFS snapshot duplication features are introduced. With the “acfsutil snap duplicate create” command, can be used to duplicate a snapshot from an existing snapshot, to a standby target file system.

The “apply” option to the “acfsutil snap duplicate” command, allows us to apply deltas to the target ACFS file system or snapshot. If this is the initial apply, the target file system must be empty. If the target had been applied before, then the apply process becomes an incremental update. Before the incremental update occurs, the contents of the target file system must match the content of the older snapshot, since the last incremental update. Also, the contents of the target snapshot cannot be modified while the apply is happening.

Additionally, ACFS snapshot-based replication now uses SSH protocols to transmit data streams.

4k Sectors and Metadata

When Admins create an ACFS file system, they have the option to create the file system with the 4096-byte metadata structure. When issuing the mkfs command, you can specify the metadata block size with the –i option; two valid options are 512 bytes or 4096 bytes. The 4096-byte metadata structure is made up of multiple 512-byte logical sectors.

If the COMPATIBLE.ADVM ASM Diskgroup attribute is set to 12.2 or greater, then the metadata block is 4096 bytes by default. If COMPATIBLE.ADVM attribute is set to less than 12.2, then the block size is set to 512 bytes. When the ADVM volume of the ACFS file system is set with 4K logical disk sector size, Direct I/O requests should be aligned on the 4K offset and be a multiple of 4k size for optimal performance.

Defragger

Very rarely would you need the defragmentation tool, due to the fact that ACFS algorithm is for allocation and coalesce-ment of free space. However, for those rare situations, when we can get into fragmented situations under heavy workloads or for compressed files, Oracle provides the defrag option to the acfsutil command. Now, we can issue “acfsutil defrag dir” or “acfsutil defrag file” commands for on-demand defragmentation.

ACFS will perform all defrag operations in the background. With the –r option of the “acfsutil defrag dir”command, you can recursively defrag subdirectories.

Compression Enhancements

ACFS compression can significantly reduce disk storage requirements for customers running databases on ACFS. Databases running on ACFS, must be of versions 11.2.0.4 or higher. ACFS compression can be enabled for specific ACFS file systems for database files, RMAN backup files, archivelogs, data pump extract files, and general purpose files. Oracle does not support redo log/flashback logs/control file compression.

When enabling ACFS compression for a file system, only new incoming files will be compressed. All existing files on the file system will remain un-compressed. Likewise, if you decide to uncompress a file system, Oracle will not de-compress files. Oracle will simply disable compression for newly created files.

To compress and uncompress ACFS file systems, execute the acfsutil compress on or acfsutil compress off commands. To view compression state and space consumption information, you can execute the “acfsutil compress info” command. The commands “acfsutil info fs” and “acfsutil info file” now support ACFS compression status.

At this time, databases with 2K or 4K block sizes are not supported for ACFS compression. ACFS compression is supported on Linux and AIX. ACFS is also supported to work with ACFS snapshot-based replication.

Loopback Devices

ACFS now supports loopback devices on the Linux operating system. With ACFS loopback device support, we can now take OVM images, templates, and virtual disks and present them as a block device. Files can be sparse or non-sparse. ACFS also supports Direct I/O on sparse images.

Metadata Collector

The metadata collector, copies metadata structures from an Oracle ACFS file system to a separate output file that can be ingested for analysis and diagnostics. The metadata collector reads the contents of the file system and all metadata is written out to a specified output file. The metadata collector can read the ACFS file system online without requiring an outage. Note, this tool is not a replacement for the file system checker command (fsck), but a supplement for additional diagnosis and support. Even though the metadata collector can read the file system while it is online, for best results, unmount the file system prior to metadata collection. The size of the output file, is directly correlated to the size of the file system that the collection is specified for. To collect metadata for a file system, invoke the “acfsutil meta” command.

The auto-resize feature, allows us to “autoextend” a file system if the size of the file system is about to run out of space. Just like an Oracle datafile that has the autoextend option enabled, we can now “autoextend” the ACFS file system to the size of the increment by option. With the –a option to the “acfsutil size” command, we can specify the increment by size.

We can also specify the maximum size or quota for the ACFS file system to “autoextend” to guard against a runaway space consumption. To set the maximum size for an ACFS file system, execute the “acfsutil size” command with the –x option.

What’s with MGTDB anyways

For those who have either upgraded or fresh-installed 12.1 (12c) Grid Infrastructure stack, will notice a new database instance (-MGMTDB) that was provisioned automagically. So what is this MGMTDB and why do I need this overhead.

Si let’s recap what the DB is and what it does…
Management Database is the central repository to store Cluster Health Monitor, the Grid Infrastructure Management Repository.

MGMT database is a container database (CDB) with one pluggable database (PDB) running. However, this database runs out of the Grid Infrastructure home.
The MGMTDB is a Rac One Node database; i.e., it runs on one node at a time, but because this is Clustered Resource, it can be started or failed over on any node in the cluster. MGMTDB is as a non-critical component of the GI stack (with no “real” hard dependencies). This means that if MGMTDB fails or becomes unavailable, Grid Infrastructure continues running

MGMTDB is configured (subject to change) with 750 MB SGA/325 MB PGA, and 5GB database size. But note that, due to the footprint MGMT’s SGA is not configured for hugepages . Since, this database is dynamically created on install, the OUI installer does not have pre-knowledge of the database that are configured or will be migrated to this cluster, thus in order to avoid any database names conflict the name “-MGMTDB” was chosen (notice the “-“). Note, bypassing MGMTDB installation is only allowed for upgrades to 12.1.0.2. New 12.1.0.2 installations or upgrades to future releases will require MGMTDB to be installed. if MGMTDB is not selected during upgrade, all features (Cluster Health Monitor (CHM/OS) etc) that depend on it will be disabled.

So if you are wondering where the datafiles and other structures are stored for this database. Well they would will be stored in the same diskgroup as OCR and VOTE However, these dtabase files can be migrated into ASM diskgroup post install.

MGMTDB will store a subset of Operating System (OS) performance data for longer term to provide diagnostic information and support intelligent workload management. Performance data (OS metrics similar to, but a subset of Exawatcher) collected by the ‘Cluster Health Monitor’ (CHM) is stored also on local disk, so when not using MGMTDB, CHM data can still be obtained from local disk but intelligent workload management (QoS) will be disabled. onger term MGMTDB will become a key component of the Grid Infrastructure and provide services for important components, because of this MGMTDB will eventually become a mandatory component in future upgrades to releases on Exadata.

See document 1568402.1 for more details.

High Level Overview of 11204 ASM Rebalance in Async ARB0

High Level look at 11204 Rebalance with Plan Optimiation and Async ARB0

 

Drop disk

 

SQL> alter diskgroup reco drop disk ‘ASM_NORM_DATA4’ rebalance power 12

here we issue the rebalance

NOTE: requesting all-instance membership refresh for group=2

GMON querying group 2 at 120 for pid 19, osid 19030

GMON updating for reconfiguration, group 2 at 121 for pid 19, osid 19030

NOTE: group 2 PST updated.

NOTE: membership refresh pending for group 2/0x89b87754 (RECO)

GMON querying group 2 at 122 for pid 13, osid 4000

SUCCESS: refreshed membership for 2/0x89b87754 (RECO)

NOTE: starting rebalance of group 2/0x89b87754 (RECO) at power 12   rebalance internally started

Starting background process ARB0    ARB0 gets started for this rebalance

SUCCESS: alter diskgroup reco drop disk ‘ASM_NORM_DATA4’ rebalance power 12

Wed Sep 19 23:54:10 2012

ARB0 started with pid=21, OS id=19526

NOTE: assigning ARB0 to group 2/0x89b87754 (RECO) with 12 parallel I/Os   ARB0 assigned to this

diskgroup rebalance. Note that it states 12 parallel I/Os

NOTE: Attempting voting file refresh on diskgroup RECO

Wed Sep 19 23:54:38 2012

NOTE: requesting all-instance membership refresh for group=2   first indications that rebalance is completing

GMON updating for reconfiguration, group 2 at 123 for pid 22, osid 19609

NOTE: group 2 PST updated.

SUCCESS: grp 2 disk ASM_NORM_DATA4 emptied    Once rebalanced relocation phase is complete, the disk is emptied

NOTE: erasing header on grp 2 disk ASM_NORM_DATA4   The emptied disk’s header is erased and set to FORMER

NOTE: process _x000_+asm (19609) initiating offline of disk 3.3915941808 (ASM_NORM_DATA4) with mask 0x7e in group 2

The dropped disk is offlined

NOTE: initiating PST update: grp = 2, dsk = 3/0xe96887b0, mask = 0x6a, op = clear

GMON updating disk modes for group 2 at 124 for pid 22, osid 19609

NOTE: PST update grp = 2 completed successfully

NOTE: initiating PST update: grp = 2, dsk = 3/0xe96887b0, mask = 0x7e, op = clear

GMON updating disk modes for group 2 at 125 for pid 22, osid 19609

NOTE: cache closing disk 3 of grp 2: ASM_NORM_DATA4

NOTE: PST update grp = 2 completed successfully

GMON updating for reconfiguration, group 2 at 126 for pid 22, osid 19609

NOTE: cache closing disk 3 of grp 2: (not open) ASM_NORM_DATA4

NOTE: group 2 PST updated.

Wed Sep 19 23:54:42 2012

NOTE: membership refresh pending for group 2/0x89b87754 (RECO)

GMON querying group 2 at 127 for pid 13, osid 4000

GMON querying group 2 at 128 for pid 13, osid 4000

NOTE: Disk in mode 0x8 marked for de-assignment

SUCCESS: refreshed membership for 2/0x89b87754 (RECO)

NOTE: Attempting voting file refresh on diskgroup RECO

Wed Sep 19 23:56:45 2012

NOTE: stopping process ARB0    All phases of rebalance are completed and ARB0 is shutdown

SUCCESS: rebalance completed for group 2/0x89b87754 (RECO)   Rebalance marked as complete

 

 

Add disk

Starting background process ARB0

SUCCESS: alter diskgroup reco add disk ‘ORCL:ASM_NORM_DATA4’ rebalance power 16

Thu Sep 20 23:08:22 2012

ARB0 started with pid=22, OS id=19415

NOTE: assigning ARB0 to group 2/0x89b87754 (RECO) with 16 parallel I/Os

Thu Sep 20 23:08:31 2012

NOTE: Attempting voting file refresh on diskgroup RECO

Thu Sep 20 23:08:46 2012

NOTE: requesting all-instance membership refresh for group=2

Thu Sep 20 23:08:49 2012

NOTE: F1X0 copy 1 relocating from 0:2 to 0:459 for diskgroup 2 (RECO)

Thu Sep 20 23:08:50 2012

GMON updating for reconfiguration, group 2 at 134 for pid 27, osid 19492

NOTE: group 2 PST updated.

Thu Sep 20 23:08:50 2012

NOTE: membership refresh pending for group 2/0x89b87754 (RECO)

NOTE: F1X0 copy 2 relocating from 1:2 to 1:500 for diskgroup 2 (RECO)

NOTE: F1X0 copy 3 relocating from 2:2 to 2:548 for diskgroup 2 (RECO)

GMON querying group 2 at 135 for pid 13, osid 4000

SUCCESS: refreshed membership for 2/0x89b87754 (RECO)

Thu Sep 20 23:09:06 2012

NOTE: Attempting voting file refresh on diskgroup RECO

Thu Sep 20 23:09:57 2012

NOTE: stopping process ARB0

SUCCESS: rebalance completed for group 2/0x89b87754 (RECO)

SQL> select NUMBER_KFGMG, OP_KFGMG, ACTUAL_KFGMG, REBALST_KFGMG from X$KFGMG;
NUMBER_KFGMG   OP_KFGMG ACTUAL_KFGMG REBALST_KFGMG
------------ ---------- ------------ -------------
           2         1           0             2
           2         32           0             2

NUMBER_KFGMG   OP_KFGMG ACTUAL_KFGMG REBALST_KFGMG
------------ ---------- ------------ -------------
           2         1           16             1
NUMBER_KFGMG   OP_KFGMG ACTUAL_KFGMG REBALST_KFGMG
------------ ---------- ------------ -------------
           2        1           16             2
           2         32           16             2


NUMBER_KFGMG   OP_KFGMG ACTUAL_KFGMG REBALST_KFGMG
------------ ---------- ------------ -------------
           2         1           16             2

			

Apply 12102 PSU1 (19392646) on a Standalone Cluster

We are about to apply 12102 PSU1 (19392646) on a Standalone Cluster. This patch applies the patch to RDBMS home as part of opatchauto.
Here’s the output in case anyone wants to compare:

[root@ol64afd OPatch]# ./opatchauto apply -analyze /mnt/hgfs/12cGridSoftware/12102-PSU1/19392646 -ocmrf ocm/bin/ocm.rsp
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation. All rights reserved.

OPatchauto version : 12.1.0.1.5
OUI version : 12.1.0.2.0
Running from : /u01/app/oracle/product/12.1.0/grid

opatchauto log file: /u01/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/19392646/opatch_gi_2014-10-30_16-03-29_analyze.log

NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.

Parameter Validation: Successful

Grid Infrastructure home:
/u01/app/oracle/product/12.1.0/grid
RAC home(s):
/u01/app/oracle/product/12.1.0/database

Configuration Validation: Successful

Patch Location: /mnt/hgfs/12cGridSoftware/12102-PSU1/19392646
Grid Infrastructure Patch(es): 19303936 19392590 19392604
RAC Patch(es): 19303936 19392604

Patch Validation: Successful
Command “/u01/app/oracle/product/12.1.0/database/OPatch/opatch version -oh /u01/app/oracle/product/12.1.0/database -invPtrLoc /u01/app/oracle/product/12.1.0/grid/oraInst.loc -v2c 12.1.0.1.5” execution failed:
bash: /u01/app/oracle/product/12.1.0/database/OPatch/opatch: No such file or directory

For more details, please refer to the log file “/u01/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/19392646/opatch_gi_2014-10-30_16-03-29_analyze.debug.log”.

Apply Summary:

Following patch(es) failed to be analyzed:
GI Home: /u01/app/oracle/product/12.1.0/grid: 19303936, 19392590, 19392604
RAC Home: /u01/app/oracle/product/12.1.0/database: 19303936, 19392604

opatchauto failed with error code 2.
[root@ol64afd OPatch]#

Need to copy 12.1.0.2.1 OPatch into Grid_HOME and DB_HOME. Now re-run

[root@ol64afd OPatch]# ./opatchauto apply -analyze /mnt/hgfs/12cGridSoftware/12102-PSU1/19392646 -ocmrf ocm/bin/ocm.rsp
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation. All rights reserved.

OPatchauto version : 12.1.0.1.5
OUI version : 12.1.0.2.0
Running from : /u01/app/oracle/product/12.1.0/grid

opatchauto log file: /u01/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/19392646/opatch_gi_2014-10-30_16-05-43_analyze.log

NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.

Parameter Validation: Successful

Grid Infrastructure home:
/u01/app/oracle/product/12.1.0/grid
RAC home(s):
/u01/app/oracle/product/12.1.0/database

Configuration Validation: Successful

Patch Location: /mnt/hgfs/12cGridSoftware/12102-PSU1/19392646
Grid Infrastructure Patch(es): 19303936 19392590 19392604
RAC Patch(es): 19303936 19392604

Patch Validation: Successful

Analyzing patch(es) on “/u01/app/oracle/product/12.1.0/database” …
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19303936” successfully analyzed on “/u01/app/oracle/product/12.1.0/database” for apply.
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19392604” successfully analyzed on “/u01/app/oracle/product/12.1.0/database” for apply.

Analyzing patch(es) on “/u01/app/oracle/product/12.1.0/grid” …
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19303936” successfully analyzed on “/u01/app/oracle/product/12.1.0/grid” for apply.
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19392590” successfully analyzed on “/u01/app/oracle/product/12.1.0/grid” for apply.
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19392604” successfully analyzed on “/u01/app/oracle/product/12.1.0/grid” for apply.

Apply Summary:
Following patch(es) are successfully analyzed:
GI Home: /u01/app/oracle/product/12.1.0/grid: 19303936, 19392590, 19392604
RAC Home: /u01/app/oracle/product/12.1.0/database: 19303936, 19392604

opatchauto succeeded.
[root@ol64afd OPatch]# ./opatchauto apply /mnt/hgfs/12cGridSoftware/12102-PSU1/19392646 -ocmrf ocm/bin/ocm.rsp
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation. All rights reserved.

OPatchauto version : 12.1.0.1.5
OUI version : 12.1.0.2.0
Running from : /u01/app/oracle/product/12.1.0/grid

opatchauto log file: /u01/app/oracle/product/12.1.0/grid/cfgtoollogs/opatchauto/19392646/opatch_gi_2014-10-30_16-07-28_deploy.log

Parameter Validation: Successful

Grid Infrastructure home:
/u01/app/oracle/product/12.1.0/grid
RAC home(s):
/u01/app/oracle/product/12.1.0/database

Configuration Validation: Successful

Patch Location: /mnt/hgfs/12cGridSoftware/12102-PSU1/19392646
Grid Infrastructure Patch(es): 19303936 19392590 19392604
RAC Patch(es): 19303936 19392604

Patch Validation: Successful

Stopping RAC (/u01/app/oracle/product/12.1.0/database) … Successful
Following database(s) and/or service(s) were stopped and will be restarted later during the session: yoda

Applying patch(es) to “/u01/app/oracle/product/12.1.0/database” …
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19303936” successfully applied to “/u01/app/oracle/product/12.1.0/database”.
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19392604” successfully applied to “/u01/app/oracle/product/12.1.0/database”.

Stopping CRS … Successful

Applying patch(es) to “/u01/app/oracle/product/12.1.0/grid” …
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19303936” successfully applied to “/u01/app/oracle/product/12.1.0/grid”.
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19392590” successfully applied to “/u01/app/oracle/product/12.1.0/grid”.
Patch “/mnt/hgfs/12cGridSoftware/12102-PSU1/19392646/19392604” successfully applied to “/u01/app/oracle/product/12.1.0/grid”.

Starting CRS … Successful

Starting RAC (/u01/app/oracle/product/12.1.0/database) … Successful

Apply Summary:
Following patch(es) are successfully installed:
GI Home: /u01/app/oracle/product/12.1.0/grid: 19303936, 19392590, 19392604
RAC Home: /u01/app/oracle/product/12.1.0/database: 19303936, 19392604

ASM Check script

Here's a little script from @racdba that does ASM check when we go onsite

#!/bin/ksh
HOST=`hostname`
ASM_OS_DEV_NM=/tmp/asmdevicenames.log
ASMVOTEDSK=/tmp/asm_votingdisks.log
GRID_HOME=`cat /etc/oratab |grep “+ASM” |awk -F “:” ‘{print $2}’`
ORACLE_HOME=$GRID_HOME
PATH=$ORACLE_HOME/bin:$PATH:
export GAWK=/bin/gawk

#
#
do_pipe ()
{
SQLP=”$GRID_HOME/bin/sqlplus -s / as sysdba”;
$SQLP |& # Open a pipe to SQL*Plus
print -p — ‘set feed off pause off pages 0 head off veri off line 500’;
print -p — ‘set term off time off’;
print -p — “set sqlprompt ””;

print -p — ‘select sysdate from dual;’;
read -p SYSDATE;

print -p — “select version from v\$instance;”;
read -p ASM_VERSION;

print -p — “select value from v\$parameter where name=’processes’;”;
read -p ASM_PROCESS;

print -p — “select value/1024/1024 from v\$parameter where name=’memory_target’;”;
read -p ASM_MEMORY;

print -p — “quit;”;
sleep 5;
}
#
function get_asminfo {
for LUNS in `ls /dev/oracleasm/disks/*`
do
echo “ASMLIB disk: $LUNS”
asmdisk=`kfed read $LUNS | grep dskname | tr -s ‘ ‘| cut -f2 -d’ ‘`
echo “ASM disk: $asmdisk”
majorminor=`ls -l $LUNS | tr -s ‘ ‘ | cut -f5,6 -d’ ‘`
dev=`ls -l /dev | tr -s ‘ ‘ | grep “$majorminor” | cut -f10 -d’ ‘`
echo “Device path: /dev/$dev”
echo “—-”
done

echo “”
echo “# ————————————————————————————————– #”;
/usr/sbin/oracleasm-discover;
}

function get_mem_info {
MEM=`free | $GAWK ‘/^Mem:/{ print int( ($2 / 1024 / 1024 + 4) / 4 ) * 4 }’`
SWAP=`free | $GAWK ‘/^Swap:/{ print int ( $2 / 1024 / 1024 + 0.5 ) }’`
HUGEPAGES=`grep HugePages_Total /proc/meminfo | $GAWK ‘{print $2}’`

echo “Physical Memory: $MEM |Swap: $SWAP”
echo “HugePages: $HUGEPAGES”
}

export ORACLE_SID=`cat /etc/oratab |grep “+ASM” |awk -F “:” ‘{print $1}’`
CHKPMON=`ps -ef|grep -v grep|grep pmon_$i|awk ‘{print $8}’`
if [ -n “$CHKPMON” ]; then
do_pipe $ORACLE_SID
echo “# ————————————————————————————————– #”;
echo “HOSTNAME: ${HOST}”
echo “GRID HOME: ${GRID_HOME}”
echo “ASM VERSION: ${ASM_VERSION}”
echo “ASM PROCESSES: ${ASM_PROCESS}”
echo “ASM MEMORY: ${ASM_MEMORY} MB”
echo “# ————————————————————————————————– #”;
get_mem_info
echo “# ————————————————————————————————– #”;
else
echo “${ORACLE_SID} is not running.”
fi

echo “# ————————————————————————————————– #”;
echo “LINUX VERSION INFORMATION:”
echo ” ”
[ -f “/etc/redhat-release” ] && cat /etc/redhat-release
[ -f “/etc/oracle-release” ] && cat /etc/oracle-release
uname -a
echo “# ————————————————————————————————– #”;

##SQLP=”sqlplus -s / as sysdba”;
##$SQLP < $ASM_OS_DEV_NM
##set feed off pause off head on veri off line 500;
##set term off time off numwidth 15;
##set sqlprompt ”;
##col label for a25
##col path for a55
##–select label,path,os_mb from v\$asm_disk;
##select label,os_mb from v\$asm_disk;
##exit;
##!

echo “ASM OS DEVICE INFORMATION:”
##cat $ASM_OS_DEV_NM
## Check for ASMLib
ASMLIBCHK=`rpm -qa |grep oracleasmlib`
if [[ -n $ASMLIBCHK ]]
then
echo “# ————————————————————————————————– #”;
echo “ASMLIB RPM: ${ASMLIBCHK}”
echo ” ”
##echo “ASM OS DEVICE INFORMATION:”
##echo ” ”
get_asminfo
else
echo “ASMLIB is NOT installed.”
fi

echo “# ————————————————————————————————– #”;

## Check OCR/Voting disks
OCR=`$GRID_HOME/bin/ocrcheck |grep “Device/File Name” |awk ‘{print $4}’`
##echo ” ”
##echo “GRID HOME is located at ${GRID_HOME}.”
echo “OCR LOCATION: ${OCR}”
echo “# ————————————————————————————————– #”;
echo ” ”

## Voting disk
$GRID_HOME/bin/crsctl query css votedisk > $ASMVOTEDSK

echo “VOTING DISK INFORMATION:”
echo ” ”
cat $ASMVOTEDSK
echo “# ————————————————————————————————– #”;

## Cleanup
if [[ -f $ASM_OS_DEV_NM ]]
then
rm $ASM_OS_DEV_NM
fi

if [[ -f $ASMVOTEDSK ]]
then
rm $ASMVOTEDSK
fi

iSCSI and enable disks for ASM

Helpful tips for my iSCSI storage buds that want to enable/provision disks for ASM

# get initiator name of host or hosts
cat /etc/iscsi/initiatorname.iscsi

# create volume on san and present to host
login to san, create volume, add initator(s) to access

# get wwid for each disk and update alias in multipath.conf
multipath -ll
or
scsi_id -g -u -s /block/sdd

# create partiton on dev
fdisk /dev/mapper/DATA18

# add device map from partition table and verify it can be seen
kpartx -a /dev/mapper/DATA18
kpartx -l /dev/mapper/DATA18

# set partition alignment
echo “2048,,” | sfdisk -uS /dev/mapper/DATA18

# create the asm disk
oracleasm createdisk DATA18 /dev/mapper/DATA18p1

#verify you can see disk
oracleasm scandisks
oracleasm listdisks