ODA Patching/Upgrade Triage

This blog will define the appropriate logs, files and command output to collect when triaging ODA related patching issues

Recently we attempted to upgrade from 12.2.1.2 to 12.2.1.3 on ODA X7.  We did the usual as part of the upgrade process.

unzip p27648057_122130_Linux-x86-64_1of3.zip

ls -l  oda-sm-12.2.1.3.0-180504-server*

/opt/oracle/dcs/bin/odacli update-repository -f oda-sm-12.2.1.3.0-180504-server1of3.zip, p27648057_122130_Linux-x86-64_2of3.zip,

p27648057_122130_Linux-x86-64_3of3.zip

/opt/oracle/dcs/bin/odacli update-dcsagent -v 12.2.1.3.0

/opt/oracle/dcs/bin/odacli describe-job -i jobid

rpm -qa |grep dcs-agent

dcs-agent-18.1.3.0.0_LINUX.X64_180504-86.x86_64

/opt/oracle/dcs/bin/odacli update-server -v 12.2.1.3.0

The update-server failed according to this log:

{

    “updatedTime” : 1530126568790,

    “startTime” : 1530126562856,

    “endTime” : 1530126568788,

    “taskId” : “TaskZJsonRpcExt_225”,

    “status” : “Success”,

    “taskResult” : “Successfully created the yum repos for patchingos”,

    “taskName” : “Creating repositories using yum”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_224”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Info”

  }, {

    “updatedTime” : 1530126894836,

    “startTime” : 1530126568799,

    “endTime” : 1530126894834,

    “taskId” : “TaskZJsonRpcExt_228”,

    “status” : “Success”,

    “taskResult” : “Successfully updated the OS”,

    “taskName” : “Applying OS Patches”,

    “taskDescription” : null,

    “parentTaskId” : “TaskParallel_227”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Info”

  },

{

    “updatedTime” : 1530126899457,

    “startTime” : 1530126894980,

    “endTime” : 1530126899455,

    “taskId” : “TaskZJsonRpcExt_236”,

    “status” : “Success”,

    “taskResult” : “Successfully updated the Firmware Disk”,

    “taskName” : “Applying Firmware Disk Patches”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_235”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Info”

  },

{

    “updatedTime” : 1530127898145,

    “startTime” : 1530127248252,

    “endTime” : 1530127898144,

    “taskId” : “TaskSequential_121”,

    “status” : “Failure”,

    “taskResult” : “DCS-10001:Internal error encountered:  apply patch using OpatchAuto on node odanode1.”,

    “taskName” : “task:TaskSequential_121”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_256”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Error”

  },

….

{

    “updatedTime” : 1530127898142,

    “startTime” : 1530127463976,

    “endTime” : 1530127898141,

    “taskId” : “TaskSequential_162”,

    “status” : “Failure”,

    “taskResult” : “DCS-10001:Internal error encountered:  apply patch using OpatchAuto on nodeodanode1.”,

    “taskName” : “task:TaskSequential_162”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_121”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Error”

  }, {

    “updatedTime” : 1530127898139,

    “startTime” : 1530127463979,

    “endTime” : 1530127898137,

    “taskId” : “TaskZJsonRpcExt_163”,

    “status” : “Failure”,

    “taskResult” : “DCS-10001:Internal error encountered:  apply patch using OpatchAuto on node odanode1″,

    “taskName” : “clusterware upgrade”,

    “taskDescription” : null,

    “parentTaskId” : “TaskSequential_162”,

    “jobId” : “5dd32b38-4d4b-4a3a-bf45-bb71cc8bf801”,

    “tags” : [ ],

    “reportLevel” : “Error”

  } ],

  “createTimestamp” : 1530126459722,

  “resourceList” : [ ],

  “description” : “Server Patching”

}

When opening an SR make sure you collected the following infor for support.  This will minimize the ‘backn’forth’-ness and reduce mean time to resolve.

Please provide the output from the commands below:
1.

odacli describe-component

System Version 

—————

12.2.1.3.0

Component                                Installed Version    Available Version   

—————————————- ——————– ——————–

OAK                                       12.2.1.3.0            up-to-date          

GI                                        12.2.0.1.171017       12.2.0.1.180116     

DB                                        12.2.0.1.171017       12.2.0.1.180116     

DCSAGENT                                  18.1.3.0.0            up-to-date          

ILOM                                      4.0.0.28.r121827      4.0.0.22.r120818    

BIOS                                      41017600              41017100            

OS                                        6.9                   up-to-date          

FIRMWARECONTROLLER                        QDV1RE14              up-to-date          

ASR                                       5.7.7                 up-to-date          

# odacli describe-latestpatch

componentType   availableVersion    

————— ——————–

gi              12.2.0.1.180116     

gi              12.2.0.1.180116     

db              12.2.0.1.180116     

db              12.1.0.2.180116     

db              11.2.0.4.180116     

oak             12.2.1.3.0          

oak             12.2.1.3.0          

asr             5.7.7               

ilom            4.0.0.22.r120818    

os              6.9                 

hmp             2.4.1.0.9           

bios            41017100            

firmwarecontroller 13.00.00.00         

firmwarecontroller 4.650.00-7176       

firmwarecontroller kpyagr3q            

firmwarecontroller qdv1re14            

firmwaredisk    0r3q                

firmwaredisk    a122                

firmwaredisk    a122                

firmwaredisk    a374                

firmwaredisk    c122                

firmwaredisk    c122                

firmwaredisk    c376                

firmwaredisk    0112                

dcsagent        18.1.3.0.0          

2.  Provide/upload the following most recent configuration and diagnostic details./log files: 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchautodb/systemconfig* 

/tmp/opatchAutoAnalyzePatch.log 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchauto/core/opatch/opatch<date>.log 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchautodb/systemconfig<date>.log 

/u01/app/12.2.0.1/oracle/cfgtoollogs/opatchauto/opatchauto<date>.log

/u01/app/oracle/crsdata/odanode1/crsconfig/crspatch_<hostname><date>.log