ASM Check script

Here's a little script from @racdba that does ASM check when we go onsite

#!/bin/ksh
HOST=`hostname`
ASM_OS_DEV_NM=/tmp/asmdevicenames.log
ASMVOTEDSK=/tmp/asm_votingdisks.log
GRID_HOME=`cat /etc/oratab |grep “+ASM” |awk -F “:” ‘{print $2}’`
ORACLE_HOME=$GRID_HOME
PATH=$ORACLE_HOME/bin:$PATH:
export GAWK=/bin/gawk

#
#
do_pipe ()
{
SQLP=”$GRID_HOME/bin/sqlplus -s / as sysdba”;
$SQLP |& # Open a pipe to SQL*Plus
print -p — ‘set feed off pause off pages 0 head off veri off line 500’;
print -p — ‘set term off time off’;
print -p — “set sqlprompt ””;

print -p — ‘select sysdate from dual;’;
read -p SYSDATE;

print -p — “select version from v\$instance;”;
read -p ASM_VERSION;

print -p — “select value from v\$parameter where name=’processes’;”;
read -p ASM_PROCESS;

print -p — “select value/1024/1024 from v\$parameter where name=’memory_target’;”;
read -p ASM_MEMORY;

print -p — “quit;”;
sleep 5;
}
#
function get_asminfo {
for LUNS in `ls /dev/oracleasm/disks/*`
do
echo “ASMLIB disk: $LUNS”
asmdisk=`kfed read $LUNS | grep dskname | tr -s ‘ ‘| cut -f2 -d’ ‘`
echo “ASM disk: $asmdisk”
majorminor=`ls -l $LUNS | tr -s ‘ ‘ | cut -f5,6 -d’ ‘`
dev=`ls -l /dev | tr -s ‘ ‘ | grep “$majorminor” | cut -f10 -d’ ‘`
echo “Device path: /dev/$dev”
echo “—-”
done

echo “”
echo “# ————————————————————————————————– #”;
/usr/sbin/oracleasm-discover;
}

function get_mem_info {
MEM=`free | $GAWK ‘/^Mem:/{ print int( ($2 / 1024 / 1024 + 4) / 4 ) * 4 }’`
SWAP=`free | $GAWK ‘/^Swap:/{ print int ( $2 / 1024 / 1024 + 0.5 ) }’`
HUGEPAGES=`grep HugePages_Total /proc/meminfo | $GAWK ‘{print $2}’`

echo “Physical Memory: $MEM |Swap: $SWAP”
echo “HugePages: $HUGEPAGES”
}

export ORACLE_SID=`cat /etc/oratab |grep “+ASM” |awk -F “:” ‘{print $1}’`
CHKPMON=`ps -ef|grep -v grep|grep pmon_$i|awk ‘{print $8}’`
if [ -n “$CHKPMON” ]; then
do_pipe $ORACLE_SID
echo “# ————————————————————————————————– #”;
echo “HOSTNAME: ${HOST}”
echo “GRID HOME: ${GRID_HOME}”
echo “ASM VERSION: ${ASM_VERSION}”
echo “ASM PROCESSES: ${ASM_PROCESS}”
echo “ASM MEMORY: ${ASM_MEMORY} MB”
echo “# ————————————————————————————————– #”;
get_mem_info
echo “# ————————————————————————————————– #”;
else
echo “${ORACLE_SID} is not running.”
fi

echo “# ————————————————————————————————– #”;
echo “LINUX VERSION INFORMATION:”
echo ” ”
[ -f “/etc/redhat-release” ] && cat /etc/redhat-release
[ -f “/etc/oracle-release” ] && cat /etc/oracle-release
uname -a
echo “# ————————————————————————————————– #”;

##SQLP=”sqlplus -s / as sysdba”;
##$SQLP < $ASM_OS_DEV_NM
##set feed off pause off head on veri off line 500;
##set term off time off numwidth 15;
##set sqlprompt ”;
##col label for a25
##col path for a55
##–select label,path,os_mb from v\$asm_disk;
##select label,os_mb from v\$asm_disk;
##exit;
##!

echo “ASM OS DEVICE INFORMATION:”
##cat $ASM_OS_DEV_NM
## Check for ASMLib
ASMLIBCHK=`rpm -qa |grep oracleasmlib`
if [[ -n $ASMLIBCHK ]]
then
echo “# ————————————————————————————————– #”;
echo “ASMLIB RPM: ${ASMLIBCHK}”
echo ” ”
##echo “ASM OS DEVICE INFORMATION:”
##echo ” ”
get_asminfo
else
echo “ASMLIB is NOT installed.”
fi

echo “# ————————————————————————————————– #”;

## Check OCR/Voting disks
OCR=`$GRID_HOME/bin/ocrcheck |grep “Device/File Name” |awk ‘{print $4}’`
##echo ” ”
##echo “GRID HOME is located at ${GRID_HOME}.”
echo “OCR LOCATION: ${OCR}”
echo “# ————————————————————————————————– #”;
echo ” ”

## Voting disk
$GRID_HOME/bin/crsctl query css votedisk > $ASMVOTEDSK

echo “VOTING DISK INFORMATION:”
echo ” ”
cat $ASMVOTEDSK
echo “# ————————————————————————————————– #”;

## Cleanup
if [[ -f $ASM_OS_DEV_NM ]]
then
rm $ASM_OS_DEV_NM
fi

if [[ -f $ASMVOTEDSK ]]
then
rm $ASMVOTEDSK
fi

FlexASM Deep Dive – Show Me the Output!!!

If you saw the first FlexASM blog you know we installed and configured FlexASM and a CDB plus a couple of PDBs. Also, this was Policy Managed with a cardinality of 2. Now let’s see what the configuration looks like, and we can break it down using the wonderful crsctl and srvctl tools

First let’s ensure we are really running in FlexASM mode:

[oracle@rac02 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled

[oracle@rac02 ~]$ srvctl status serverpool -serverpool naboo
Server pool name: naboo
Active servers count: 2

[oracle@rac01 trace]$ crsctl get node role status -all
Node ‘rac01’ active role is ‘hub’
Node ‘rac03’ active role is ‘hub’
Node ‘rac02’ active role is ‘hub’
Node ‘rac04’ active role is ‘hub’

[oracle@rac01 ~]$ crsctl stat res -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

You notice that we have 4 ASM listeners one on each node in the Cluster. You’ll see the process as the following on each node:

[oracle@rac01 ~]$ ps -ef |grep -i asmnet

ooracle 6646 1 0 12:19 ? 00:00:00 /u01/app/12.1.0/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit

ora.CRSDATA.DATAVOL1.advm
ONLINE ONLINE rac01 Volume device /dev/a
sm/datavol1-194 is o
nline,STABLE
ONLINE ONLINE rac02 Volume device /dev/a
sm/datavol1-194 is o
nline,STABLE
ONLINE OFFLINE rac03 Unable to connect to
ASM,STABLE
ONLINE ONLINE rac04 Volume device /dev/a
sm/datavol1-194 is o
nline,STABLE
The datavol1 ADVM resource runs on all the nodes where indicated it should run. In this case we are seeing that RAC03 is having some issues.
Let’s look into that a little later. But I like the fact crsctl tells something is amiss here on node3

ora.CRSDATA.dg
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
OFFLINE OFFLINE rac04 STABLE

ora.FRA.dg
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
OFFLINE OFFLINE rac04 STABLE

The crsdata and fra disk groups resource is started on all nodes except node 4

ora.LISTENER.lsnr
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

We all know, as in 11gR2, that this is the Node listener.

ora.PDBDATA.dg
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
OFFLINE OFFLINE rac04 STABLE

The pdbdata disk groups resource is started on all nodes except node 4

ora.crsdata.datavol1.acfs
ONLINE ONLINE rac01 mounted on /u02/app/
oracle/acfsmounts,ST
ABLE
ONLINE ONLINE rac02 mounted on /u02/app/
oracle/acfsmounts,ST
ABLE
ONLINE OFFLINE rac03 (2) volume /u02/app/
oracle/acfsmounts of
fline,STABLE
ONLINE ONLINE rac04 mounted on /u02/app/
oracle/acfsmounts,ST
ABLE

ACFS filesystem resource for datavol1 is started on all nodes except node3.
But I think the following has something to do w/ it :-). Need to debug this a bit later. I even tried:
[oracle@rac03 ~]$ asmcmd volenable –all
ASMCMD-9470: ASM proxy instance unavailable
ASMCMD-9471: cannot enable or disable volumes

ora.net1.network
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE
ora.ons
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE ONLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

The Network (in my case I only have only Net1) and ONS are same as in previous versions

ora.proxy_advm
ONLINE ONLINE rac01 STABLE
ONLINE ONLINE rac02 STABLE
ONLINE OFFLINE rac03 STABLE
ONLINE ONLINE rac04 STABLE

Yep, since proxy_advm is not started on node3, the filesystems won’t come online….but again, i’ll look at that later
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac03 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac04 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac01 169.254.90.36 172.16
.11.10,STABLE
ora.asm
1 ONLINE ONLINE rac03 STABLE
2 ONLINE ONLINE rac01 STABLE
3 ONLINE ONLINE rac02 STABLE

Since we have the cardinality of 3 ASM instance we have 3 ASM resources active

ora.cvu
1 ONLINE ONLINE rac01 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac01 Open,STABLE
ora.oc4j
1 ONLINE ONLINE rac01 STABLE
ora.rac01.vip
1 ONLINE ONLINE rac01 STABLE
ora.rac02.vip
1 ONLINE ONLINE rac02 STABLE
ora.rac03.vip
1 ONLINE ONLINE rac03 STABLE
ora.rac04.vip
1 ONLINE ONLINE rac04 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac02 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac03 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac04 STABLE
ora.tatooine.db
1 ONLINE ONLINE rac01 Open,STABLE
2 ONLINE ONLINE rac02 Open,STABLE

As we stated above, I specified a Policy Managed database with cardinality of 2, so I have 2 database instances running
——————————————————————————–

Here’s some other important supporting info on FlexASm:

[oracle@rac02 ~]$ srvctl config asm -detail
ASM home: /u01/app/12.1.0/grid
Password file: +CRSDATA/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

[oracle@rac02 ~]$ srvctl status filesystem
ACFS file system /u02/app/oracle/acfsmounts is mounted on nodes rac01,rac02,rac04

ANd here’s what the Database has to say about FlexASM

NOTE: ASMB registering with ASM instance as client 0x10001 (reg:1377584805)
NOTE: ASMB connected to ASM instance +ASM1 (Flex mode; client id 0x10001)
NOTE: ASMB rebuilding ASM server state
NOTE: ASMB rebuilt 2 (of 2) groups
SUCCESS: ASMB reconnected & completed ASM server state

So for the interesting part:
If you notice that ASM is not running node 4:
[oracle@rac02 ~]$ srvctl status asm -v

ASM is running on rac01,rac02,rac03
[oracle@rac02 ~]$ srvctl status asm -detail
ASM is running on rac01,rac02,rac03

So, how does a client (ocrdump, rman, asmcmd, etc..) connect to if ASM if there is no ASM on that node. Well let’s test this using asmcmd on node4. You notice that a pipe is created, a connect string is generated and passed to ASMCMD to connect remotely to ASM2 on node2!!!!

22-Sep-13 12:54 ASMCMD Foreground (PID = 14106): Pipe /tmp/pipe_14106 has been found.
22-Sep-13 12:54 ASMCMD Background (PID = 14117): Successfully opened the pipe /tmp/pipe_14106
22-Sep-13 12:54 ASMCMD Foreground (PID = 14106): Successfully opened the pipe /tmp/pipe_14106 in read mode
NOTE: Executing kfod /u01/app/12.1.0/grid/bin/kfod op=getclstype..
22-Sep-13 12:54 Printing the connection string
contype =
driver =
instanceName = <>
usr =
ServiceName = <+ASM>
23-Sep-13 16:23 Successfully connected to ASM instance +ASM2
23-Sep-13 16:23 NOTE: Querying ASM instance to get list of disks
22-Sep-13 12:54 Registered Daemon process.
22-Sep-13 12:54 ASMCMD Foreground (PID = 14106): Closed pipe /tmp/pipe_14106.