[Re-] Imaging Exadata with 12.1.2.1.1

 [Re-] Imaging Exadata with 12.1.2.1.1

Introduction
This document, based on Doc ID 1991376.1, specifically addresses re-imaging Exadata systems to 12.1.2.1.1, using standard (non-OVM) deployment via USB boot.  In this scenario, a new Exadata system was delivered with the 12.1.2.1.0 image; however, client decided to implement the latest image (12.1.2.1.1) before OEDA was executed.

Scenario
The purpose of this doc to build on the base doc and includes resolution to issues we ran into.  This doc applies to Oracle Exadata Storage Server Software Version 12.1.2.1.0 and later.  In our deployment, we re-imaged a new X5-2 that had existing 12.1.2.1.0, with version 12.1.2.1.1 released in April.

This paper describes steps necessary to image or re-image Exadata DB and cells nodes using the USB method.  Doc ID 1991376.1 covers the other available options. This document assumes that Exadata has been powered on and a hardware validation has been performed by an Oracle Field Engineer. Prior to imaging, the administrator should obtain and execute the latest OEDA to generate the configuration files (preconf.csv).

Procedure
The following are preparatory steps before imaging the Exadata:

1.     The generated preconfig.csv will need to be copied to the DB node where the image file will generated.  This should node1 (first DB node)

2.     Connect to the first DB node by connecting a laptop to the ILOM Serial Management port.  It is important to connect to serial RS232 port (115200,8,N,1) as well as the VGA port.  We encountered issues where log output was not always displayed to tty1.

3.     In ILOM, obtain the eth0 MAC address for each DB node.

Connect to /SYS/MB/NET0 and execute “show fru_macaddress”

> cd /SYS/MB/NET0
/SYS/MB/NET0

-> show fru_macaddress

/SYS/MB/NET0
Properties:
fru_macaddress = 00:10:e0:6f:b2:aa

node1 compute node macaddress – 00:10:e0:6f:8e:e6
node2 compute node macaddress – 00:10:e0:6f:b2:aa
node 1 cell node macaddress –  00:10:e0:62:b8:0a
node 2 cell macaddr –  00:10:e0:71:e8:ac
node 3 cell macaddr –  00:10:e0:72:06:b8

4.     In preconf.csv file, add the the MAC addresses in CAPs in the appropriate place for each node. (this is the 7th field the first set of empty fields in the OEDA preconf.csv “,,”)

5.     Download the Imagemaker files.  This kit is publicly available on the edelivery.oracle.com site under “Oracle Database Products >Linux x8664”.  Note, there are seperate ImageMaker files for DB nodes and cells. See MOS note 1306961.1 for more information on downloading the files.

6.     After download, unzip the first file “V75080-01.zip”. Perform unzip then untar the file.
# unzip V7508001.zip
# unzip cellImageMaker_12.1.2.1.1_LINUX.X64_150316.21.x86_64.zip
# tar pxvf cellImageMaker_12.1.2.1.1_LINUX.X64_150316.21.x86_64.tar

7.     Insert a blank USB device in the slot of the DB node where the ImageMaker was unpacked.

8.     Use makeImageMaker.sh to build the kernel, initrd and image files for the USB device.  The makeImageMedia.sh script provided with Storage Cell and Linux Database Host images is available on the Oracle Software Delivery Cloud (http://edelivery.oracle.com).  For building 12.1.2.1.0 images. The following is recommended makeImageMedia.sh options for building USB bootable media :

./makeImageMedia.sh -factory  -stit reboot-on-success –nodisktests \ -dhcp –preconf ‘/tmp/preconv.cvs’

9.     Repeat steps 6-8 to makeImageMedia.sh for the cell node using a different USB device

10.   Once the USB device has been boot formatted, place it into the slot of the server to be imaged (ensure image is for the correct type of DB node or cell).  Note, do not issue a Linux ‘mount’ against the device and ensure that the device is not mounted in Linux

11.  Reboot the DB node or cell to start the imaging process from USB.  Note that regardless of cell or DB node, the boot image message states CELLUSB_INSTALL_OS.  At this point the internal disk of DB or Cell node will be loaded with the new image.   This process took 20 minutes on a X5 Cell node and 15 minutes on DB node.  Note, because we specified reboot-on-success, it will reboot twice

12.   Validate that the nodes are imaged correctly, run imageinfo –-all

13.  Ibhosts

14.  New Exadata Systems with Exadata release 12.1.2.1.0 or higher will not have a hotspare available. The reclaimdisk.sh script will be used to reclaim diskspace from the RAID5 volume group to produce a hotspare.  This script should only be executed after imaging all the DB and cell nodes

Do not skip this step. Skipping this step will result in unused space that can no longer be reclaimed by reclaimdisks.sh.

The following steps must be completed on ALL database servers before running the OEDA installation tool:

“/opt/oracle.SupportTools/reclaimdisks.sh check”
“/opt/oracle.SupportTools/reclaimdisks.sh free reclaim”

15.  On successful re-imaging, the next step is to execute OEDA deployment. This is not covered in this document.