Clonewars – Next Gen Cloning with Oracle 12.2 Multitenancy (Part Deux)… With a Sprinkle of PDB Refresh

 

This is Part 2 of the Remote [PDB] Cloning capabilities of Oracle 12.2 Mulitenant.

Cloning Example 2:  Remote clone copy from an existing CBD/PDB into a local PDB (PDB->PDB).  In this example “darkside” is  CDB with darthmaul being the source/remote PDB and  yoda (PDB) is local target

 

SQL> select database_name from v$database;

DATABASE_NAME
--------------------------------------------------------
DARKSIDE

darkside$SQL> alter pluggable database darthmaul open;
Pluggable database altered.

SQL> select name, open_mode from v$pdbs;
NAME .    OPEN_MODE

--------------------
PDB$SEED  READ ONLY
DARTHMAUL READ WRITE

darkside$SQL> archive log list ;
Database log mode            Archive Mode
Automatic archival           Enabled
Archive destination          USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     1
Next log sequence to archive   3
Current log sequence         3

darkside$SQL> select name, open_mode from v$database;
NAME     OPEN_MODE
--------- --------------------
DARKSIDE  READ WRITE

darkside$SQL> COLUMN property_name FORMAT A30
COLUMN property_value FORMAT A30
SELECT property_name, property_value
FROM   database_properties
WHERE  property_name = 'LOCAL_UNDO_ENABLED'; 
PROPERTY_NAME                PROPERTY_VALUE
------------------------------ ------------------------------
LOCAL_UNDO_ENABLED           TRUE


$ cat darkside_create_remote_clone_user.sql
create user c##darksidecloneuser identified by cloneuser123 container=ALL;
grant create session, create pluggable database to c##darksidecloneuser  container=ALL;

$cat darkside_db_link.sql
create database link darksideclone_link
CONNECT TO c##darksidecloneuser IDENTIFIED BY cloneuser123 USING 'darkside'

Nishan$SQL> select DB_LINK,HOST from dba_db_links;
DB_LINK        HOST
------------  ---------------------------
SYS_HUB          SEEDDATA
REMOTECLONELINK  hansolo
DARKSIDECLONE_LINK darkside

darkside$SQL> select name from v$datafile;
NAME
-------------------------------------------------------------------------------

+ACFSDATA/DARKSIDE/4E63D836FC5AF80DE053B214A8C07E55/DATAFILE/system.276.942656929

+ACFSDATA/DARKSIDE/4E63D836FC5AF80DE053B214A8C07E55/DATAFILE/sysaux.277.942656929

+ACFSDATA/DARKSIDE/4E63D836FC5AF80DE053B214A8C07E55/DATAFILE/undotbs1.275.942656929

+ACFSDATA/DARKSIDE/4E63D836FC5AF80DE053B214A8C07E55/DATAFILE/users.279.942657041

+ACFSDATA/DARKSIDE/4E63D836FC5AF80DE053B214A8C07E55/DATAFILE/rey.291.942877803

+ACFSDATA/DARKSIDE/4E63D836FC5AF80DE053B214A8C07E55/DATAFILE/luke.292.942877825

darkside$SQL> show con_name
CON_NAME
-----------------------------
DARTHMAUL


darkside$SQL> create table foofighters tablespace rey as select * from obj$;
Table created.

Nishan$SQL> create pluggable database yoda from darthmaul@DARKSIDECLONE_LINK;

Pluggable database created.

Nishan$SQL> alter session set container = yoda;
Session altered.

yoda$SQL> select name, open_mode from v$pdbs;
NAME                    OPEN_MODE
----------------------------------------
YODA                   MOUNTED

yoda$SQL> select name from v$datafile;
NAME
--------------------------------------------------------------------------------
+DATA/NISHAN/4E82D233272C7273E0538514A8C00DF3/DATAFILE/system.310.942878321
+DATA/NISHAN/4E82D233272C7273E0538514A8C00DF3/DATAFILE/sysaux.311.942878321
+DATA/NISHAN/4E82D233272C7273E0538514A8C00DF3/DATAFILE/undotbs1.309.942878321
+DATA/NISHAN/4E82D233272C7273E0538514A8C00DF3/DATAFILE/users.306.942878319
+DATA/NISHAN/4E82D233272C7273E0538514A8C00DF3/DATAFILE/rey.307.942878319
+DATA/NISHAN/4E82D233272C7273E0538514A8C00DF3/DATAFILE/luke.308.942878319


Now on to Refresh the PDB

SQL>create table foofighters tablespace rey as select * from obj$

Table created.




SQL> select segment_name from dba_segments where tablespace_name = 'REY'

SEGMENT_NAME

----------------------------------------------------------------

FOOFIGHTERS




SQL> select name, open_mode from v$pdbs;

NAME            OPEN_MODE

------------------------------

PDB$SEED        READ ONLY

OBIWAN          READ WRITE

FORCEAWAKENS    MOUNTED

YODA            MOUNTED




SQL> alter pluggable database yoda open read only;

Pluggable database altered.




SQL> select segment_name from dba_segments where tablespace_name = 'REY';

no rows selected




SQL> alter session set container = yoda;

Session altered.




SQL> ALTER PLUGGABLE DATABASE CLOSE IMMEDIATE;

Pluggable database altered.




SQL> ALTER PLUGGABLE DATABASE refresh;

Pluggable database altered.




SQL> select segment_name from dba_segments where tablespace_name = 'REY';

select segment_name from dba_segments where tablespace_name = 'REY'  

ERROR at line 1:

ORA-01219: database or pluggable database not open: queries allowed on fixed

tables or views only




SQL> ALTER PLUGGABLE DATABASE open read only;

Pluggable database altered.




SQL> select segment_name from dba_segments where tablespace_name = 'REY';

SEGMENT_NAME

-----------------------------------------------------

FOOFIGHTERS

 

 

 

 

 

Clonewars – Next Gen Cloning with Oracle 12.2 Multitenancy (Part Un)

In this blog, we will walk through Oracle 12.2 Remote Cloning of PDB feature. In Oracle 12.1, remote cloning was also available, however, this required placing the productions database (which is usually the source) in read-only mode. This makes the cloning feature very inefficient to leverage. In 12.2, it is now possible to maintain the production database in read-write mode and allow for online copy of the database, this is reffered to as a “hot clone”.  The distinction between a hot clone and a cold clone is only relevant for customers running 12.1 Multitenancy. As of 12.2 all clones are hot clones, unless the source database is explicitly closed.

We will illustrate two examples of this real-world example, just the names have been changed to protect the extremely innocent. And sorry about the StarWars references.. just couldn’t help myself!!

Note, for clarity, the remote DB is source database which will cloned, and the local DB is the CDB where the PDB will cloned into.

Cloning Example 1: Remote clone copy from existing non-CDB into a local PDB (non-CDB->PDB).  In this example “hansolo” is remote non-CDB (source PDB).

Cloning Example 2: Remote clone copy from existing CBD/PDB into a local PDB (PDB->PDB). In this example “darkside” is CDB with obiwan being the source PDB and  nishan-obiwan (PDB) is local.

Cloning Example 1

Prep work and validation
 Hansolo$SQL> startup
 ORACLE instance started.
 Total System Global Area 2483027968 bytes
 Fixed Size 8795808 bytes
 Variable Size 637536608 bytes
 Database Buffers 1610612736 bytes
 Redo Buffers 7979008 bytes
 In-Memory Area 218103808 bytes
 Database mounted.
 Database opened.
Hansolo$SQL> select database_name from v$database;
 DATABASE_NAME
 ------------------------------------------------------
 HANSOLO

Nishan$SQL> select name from v$pdbs;
 NAME
 ------------------------------------------------------------------------------
 PDB$SEED
 OBIWAN

In 12.2, each PDB will have its own undo tablespace. 
This new undo management configuration is called local undo mode, and is the underlying 
design for many of the portability features in 12.2. Local Undo is the default for greefield/fresh 12.2 installs, 
for upgrades to 12.2 the Shared Undo will need to converted to Local (we won't cover that here)
 
Hansolo$SQL> SELECT property_name, property_value FROM database_properties WHERE property_name = 'LOCAL_UNDO_ENABLED ';
PROPERTY_NAME PROPERTY_VALUE
------------------------------ ------------------------------
LOCAL_UNDO_ENABLED TRUE

Hansolo$SQL> archive log list
 Database log mode Archive Mode
 Automatic archival Enabled
 Archive destination USE_DB_RECOVERY_FILE_DEST
 Oldest online log sequence 118
 Next log sequence to archive 120
 Current log sequence 120

Hansolo$SQL> select name, open_mode from v$database
NAME OPEN_MODE
--------- --------------------
HANSOLO READ WRITE

Hansolo$SQL> create tablespace kyloren datafile size 20M;

Tablespace created.

Hansolo$SQL> create tablespace MazKanata datafile size 20M

Tablespace created.

Hansolo$SQL> select tablespace_name from dba_tablespaces;

TABLESPACE_NAME
------------------------------
 SYSTEM
 SYSAUX
 UNDOTBS1
 TEMP
 USERS
 KYLOREN
 MAZKANATA

Hansolo$SQL> select current_scn from v$database;

CURRENT_SCN
 -----------
 27506427

$cat hansolo_create_remoteclone.sql
 CREATE USER cloneuser IDENTIFIED BY cloneuser123;
 GRANT CREATE SESSION, CREATE PLUGGABLE DATABASE TO cloneuser;

Hansolo$SQL>@hansolo_create_remoteclone.sql

Verify user connection

Hansolo$SQL> connect cloneuser/cloneuser123;
 Connected.

Now, prep the source environment

Nishan$SQL> select database_name from v$database;

DATABASE_NAME
-------------------------------------------------
NISHAN

Create DBLink to hansolo from nishan

$cat pdbclone_dblink.sql
CREATE DATABASE LINK remoteclonelink CONNECT TO cloneuser IDENTIFIED BY 
cloneuser123 USING 'hansolo'

Nishan$SQL> @pdbclone_dblink.sql

Nishan$SQL> select db_link, host from dba_db_links;
DB_LINK            HOST
----------------  -----------------
SYS_HUB           SEEDDATA 
REMOTECLONELINK   hansolo 

Verify connection to hansolo from forceawakens PDB

$ sqlplus cloneuser/cloneuser123@hansolo

Nishan$SQL> create pluggable database forceawakens from non$cdb@REMOTECLONELINK;

Pluggable database created.

Nishan$SQL> alter session set container = FORCEAWAKENS;
Session altered.

forceawakens$SQL> select name, open_mode from v$pdbs;
 NAME           OPEN_MODE
---------       ----------------------
FORCEAWAKENS    MOUNTED

forceawakens$SQL> select name from v$datafile;
NAME
-------------------------------------------------------------------------------
 +DATA/NISHAN/4E6DBABFDE2EBBECE0538514A8C0247B/DATAFILE/system.302.942700581
 +DATA/NISHAN/4E6DBABFDE2EBBECE0538514A8C0247B/DATAFILE/sysaux.301.942700581
 +DATA/NISHAN/4E6DBABFDE2EBBECE0538514A8C0247B/DATAFILE/undotbs1.300.942700581
 +DATA/NISHAN/4E6DBABFDE2EBBECE0538514A8C0247B/DATAFILE/users.297.942700581
 +DATA/NISHAN/4E6DBABFDE2EBBECE0538514A8C0247B/DATAFILE/kyloren.298.942700581
 +DATA/NISHAN/4E6DBABFDE2EBBECE0538514A8C0247B/DATAFILE/mazkanata.299.942700581

forceawakens$SQL> select current_scn from v$database;

CURRENT_SCN
-----------
 0

Since the source database was a non-CDB, it needs to be cleansed to be PDB-capable using the @$ORACLE_HOME/rdbms/admin/noncdb_to_pdb.sql. This is a requirement before you can open and online the PDB.

forceawakens$SQL> @$ORACLE_HOME/rdbms/admin/noncdb_to_pdb.sql

forceawakens$SQL> alter pluggable database open;

What’s with MGTDB anyways

For those who have either upgraded or fresh-installed 12.1 (12c) Grid Infrastructure stack, will notice a new database instance (-MGMTDB) that was provisioned automagically. So what is this MGMTDB and why do I need this overhead.

Si let’s recap what the DB is and what it does…
Management Database is the central repository to store Cluster Health Monitor, the Grid Infrastructure Management Repository.

MGMT database is a container database (CDB) with one pluggable database (PDB) running. However, this database runs out of the Grid Infrastructure home.
The MGMTDB is a Rac One Node database; i.e., it runs on one node at a time, but because this is Clustered Resource, it can be started or failed over on any node in the cluster. MGMTDB is as a non-critical component of the GI stack (with no “real” hard dependencies). This means that if MGMTDB fails or becomes unavailable, Grid Infrastructure continues running

MGMTDB is configured (subject to change) with 750 MB SGA/325 MB PGA, and 5GB database size. But note that, due to the footprint MGMT’s SGA is not configured for hugepages . Since, this database is dynamically created on install, the OUI installer does not have pre-knowledge of the database that are configured or will be migrated to this cluster, thus in order to avoid any database names conflict the name “-MGMTDB” was chosen (notice the “-“). Note, bypassing MGMTDB installation is only allowed for upgrades to 12.1.0.2. New 12.1.0.2 installations or upgrades to future releases will require MGMTDB to be installed. if MGMTDB is not selected during upgrade, all features (Cluster Health Monitor (CHM/OS) etc) that depend on it will be disabled.

So if you are wondering where the datafiles and other structures are stored for this database. Well they would will be stored in the same diskgroup as OCR and VOTE However, these dtabase files can be migrated into ASM diskgroup post install.

MGMTDB will store a subset of Operating System (OS) performance data for longer term to provide diagnostic information and support intelligent workload management. Performance data (OS metrics similar to, but a subset of Exawatcher) collected by the ‘Cluster Health Monitor’ (CHM) is stored also on local disk, so when not using MGMTDB, CHM data can still be obtained from local disk but intelligent workload management (QoS) will be disabled. onger term MGMTDB will become a key component of the Grid Infrastructure and provide services for important components, because of this MGMTDB will eventually become a mandatory component in future upgrades to releases on Exadata.

See document 1568402.1 for more details.

Creating PDBs part 3

Due to so many people asking me other methods besides SQLplus for provisioning PDBs; such as OEM, DBCA, etc. In this blog entry I’ll DBCA, just because its simple to show. As I mentioned in my last PDB blog,
the installer DBCA (initial DBA invocation) looks different than the subsequent (post initial db creation).

The main DBCA screen shows the following pages. We will choose Manage Pluggable Database

PDB12c 2013 08 20 17 46 20

Choose the CDB, Note you could have many CDBs on the same Node or RAC cluster

PDB12c 2013 08 30 17 51 59

We choose our PDB that we created in Part 1 of the blog

PDB12c 2013 08 30 17 52 39

Ahh..we gotta open the PDB first. As before:

CDB$ROOT@YODA> alter session set container=pdbobi;
Session altered.

CDB$ROOT@YODA> alter pluggable database pdbobi open;

Pluggable database altered.

or CDB$ROOT@YODA> alter pluggable database all open;

PDB12c 2013 08 30 17 54 22

Now we can Add support for and configure Database Vault. Additionally, Label Security can be configured.
It would have been nice to enable and modify Resource Manager as well other PDB tasks.
But I get the fact that this DBCA is really driven for the PDB operations (plug,unplug, create and destroy PDB).
Bulk of the PDB admin tasks are provided in EM

PDB12c 2013 08 30 18 14 54

Let’s do a new PDB creation for grins 🙂

PDB12c 2013 08 30 18 21 01

Specify the PDB name, storage location, and a default tablespace. Again, it would have been nice to specify a TEMP tablespace too, but that was left out

PDB12c 2013 08 30 18 22 26

Progress ….

PDB12c 2013 08 30 18 23 18

And Completion….Pretty Straightforward

PDB12c 2013 08 30 18 22 53

Creating PDB’s part 2

Once we have installed 12.1 Database Software, we can create the Container Database and the Pluggable Databases. In my case I did a software only install then manually executed DBCA

In this blog entry I’ll show the screens that walk-thru the configuration of the “first” database. I noticed that once DBCA is used to create the initial database, the capability and options (screens) for DBCA are different; i.e., it much more aligned to create/manage additional databases. I’ll show those screens in Part 3 of PDB

So let’s get started by executing
$ $ORACLE_HOME/bin/dbca

Rac01 2013 09 15 22 39 12

Choose Advanced mode for Policy Managed Database or use “Default Configuration”. Being a big promoter of Policy Managed Databases and since I have 4 RAC nodes (my best practice threshold to choose Policy Managed), I’ll choose, that.

Rac01 2013 09 15 22 39 44

I’ll pick a Global Database name and choose PDB option, and also option to choose how many PDBs to create (with prefix)

Rac01 2013 09 15 22 40 30

Pick a Server Pool name, I chose a cardinality of 2

Rac01 2013 09 15 22 40 58

Define the Management Options

Rac01 2013 09 15 22 41 20

Choose the Storage locations

Rac01 2013 09 15 22 43 31

Define Database Vault Owner and also the Separate Account Manager. Note the user name definitions

Rac01 2013 09 15 22 45 31

And now the finish

Rac01 2013 09 15 22 56 41

12c PDB Multitenancy, Schema Consolidation, and whatever

Many of you have probably have heard me speak over the years (at OOW, local user groups and at the local bars) about the virtues of simplification, rationalization, and consolidation. I mentioned the different database consolidation and multi-tenancy models: Virtualization based, Database Instance and Schema consolidation.

The following papers I wrote [when I was at Oracle] touch in detail on this topic –
http://www.oracle.com/technetwork/database/database-cloud/database-cons-best-practices-1561461.pdf

And here’s a more current version of that paper., updated for 12c and PDB.
http://www.oracle.com/us/products/database/database-private-cloud-wp-360048.pdf

For those who have done consolidation via Virtualization platforms such as VMWare or OVM know its fairly straightforward and its a simple “drag and drop”, as I say. Similarly consolidation of many databases as separate database instances on platform is also fairly straightforward. Its the consolidation of many disparate schemas into a common database that makes things interesting. Couple of key points on “why schema consolidation” from the paper:

  • The schema consolidation model has consistently provided the most opportunities for reducing operating expenses, since you only have a single big database to maintain,monitor, mange and maintain.
  • Though schema consolidation allows the best ROI (w.r.t CapEX/OPex), you are sacrificing flexibility for compaction. As I’ve stated in my presentations and papers, “…consolidation and isolation move in opposite directions” The more you consolidate the less capabilities you’ll have for isolation; in contrast, the more you try to isolate, the more you sacrifice benefits of consolidation.
  • Custom (home-grown) apps have been best fit use cases for schema consolidation, since application owners and developers have more control on how the application and schema is built.

Well, with the 12c Oracle Database feature Pluggable Database (PDB) , you now have more incentive to lean towards the schema consolidation. PDB “begins” to eliminate the typical issues that come with schema consolidation; such as namespace collisions, security, granularity of recovery.

In this 1st part of the three part series on PDB, I’ll illustrate the installation of the 12c Database with Pluggable Database feature. The next upcoming parts of the series will cover management and user isolation (security) with PDB.

But first a very, very high-level primer on terminology:

  • Root Container Database – Or the root CDB (cdb$root) is the real database (if you will), and the name you give it will be name of the instance. The CDB database owns the SGA and running processes. I can have many CDBs on the same database server (each with its own PDBs). But the cool thing is that you can have a more than one CDB, allowing DBAs to have a Database Instance consolidation model coupled a schema consolidation. For best scalability, mix in RAC and leverage all the benefits of RAC Services, QoS, and Workload Distribution. The seed PDB (PDB$SEED) is a Oracle supplied system template that the CDB can use to create new PDBs. The seed PDB is named PDB$SEED. One cannot add or modify objects in PDB$SEED.
  • Pluggable Database – The PDB databases are sub-containers that serviced by CDB resources. The true beauty of the PDB is its mobility; i.e., I can unplug and plug 12c databases into and out of CDBs. I can “create like” new PDBs from existing PDB, like full snapshots.

So, now I’ll illustrate the important/interesting and new screens of 12c Database Installer:

PDB12c 2013 08 19 17 22 42

We chose Server Class

PDB12c 2013 08 19 17 23 09

It will single instance ..for now 🙂

PDB12c 2013 08 19 17 23 37

Choose Advanced Install

PDB12c 2013 08 19 17 24 07

And now for the fun step. We choose a Enterprise Edition, as Pluggable Database feature is only available in EE

PDB12c 2013 08 19 17 24 47

The next couple of screens ask about the Oracle Home and Oracle Base location, nothing new, but look at screen for Step 11. This where the fun is. We specify the Database name, but also specify if we want to create a Container Database. If we check it, it allows us to create our first PDB database in the Container Database (CDB). In my example I speficied Yoda as my CDB name and (in keeping with Star Wars theme) I said PDB is PDBOBI

PDB12c 2013 08 19 17 27 19

We obviously choose ASM as the storage location

PDB12c 2013 08 19 17 28 18

And we have the opportunity to register EM Cloud Control this new target database.

PDB12c 2013 08 20 17 46 20

The rest of the steps/screens are standard stuff, so I won’t bore you with it. But here’s an excerpt from the database alert that shows magic underneath:

create pluggable database PDB$SEED as clone  using '/u02/app/oracle/product/12.1.0/dbhome_1/assistants/dbca/templates//pdbseed.xml'  source_file_name_convert = ('/ade/b/3593327372/oracle/oradata/seeddata/pdbseed/temp01.dbf','+PDBDATA/YODA/DD7C48AA5A4404A2E04325AAE80A403C/DATAFILE/pdbseed_temp01.dbf',
'/ade/b/3593327372/oracle/oradata/seeddata/pdbseed/system01.dbf','+PDBDATA/YODA/DD7C48AA5A4404A2E04325AAE80A403C/DATAFILE/system.271.823892297',
'/ade/b/3593327372/oracle/oradata/seeddata/pdbseed/sysaux01.dbf','+PDBDATA/YODA/DD7C48AA5A4404A2E04325AAE80A403C/DATAFILE/sysaux.270.823892297') file_name_convert=NONE  NOCOPY
Mon Aug 19 18:58:59 2013
….
…. 
Post plug operations are now complete.
Pluggable database PDB$SEED with pdb id - 2 is now marked as NEW.


create pluggable database pdbobi as clone  using '/u02/app/oracle/product/12.1.0/dbhome_1/assistants/dbca/templates//sampleschema.xml'  source_file_name_convert = ('/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/temp01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/pdbobi_temp01.dbf',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/example01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/example.275.823892813',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/system01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/system.276.823892813',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/SAMPLE_SCHEMA_users01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/users.277.823892813',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/sysaux01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/sysaux.274.823892813') file_name_convert=NONE  NOCOPY
Mon Aug 19 19:07:42 2013
….
….
****************************************************************
Post plug operations are now complete.
Pluggable database PDBOBI with pdb id - 3 is now marked as NEW.
****************************************************************
Completed: create pluggable database pdbobi as clone  using '/u02/app/oracle/product/12.1.0/dbhome_1/assistants/dbca/templates//sampleschema.xml'  source_file_name_convert = ('/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/temp01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/pdbobi_temp01.dbf',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/example01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/example.275.823892813',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/system01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/system.276.823892813',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/SAMPLE_SCHEMA_users01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/users.277.823892813',
'/ade/b/3593327372/oracle/oradata/seeddata/SAMPLE_SCHEMA/sysaux01.dbf','+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/sysaux.274.823892813') file_name_convert=NONE  NOCOPY
alter pluggable database pdbobi open restricted
Pluggable database PDBOBI dictionary check beginning
Pluggable Database PDBOBI Dictionary check complete
Database Characterset is US7ASCII
….
….

XDB installed.

XDB initialized.
Mon Aug 19 19:08:01 2013
Pluggable database PDBOBI opened read write
Completed: alter pluggable database pdbobi open restricted

I will cover more of PDB creation and management in the next blog. But I’ll leave you with this teaser of DBCA screen:

PDB12c 2013 08 20 17 46 20

A 12c Flex Cluster Installation Walk-thru

Let’s start the 12c Flex install, with the execution of the traditional runInstaller script

Rac12c1 2013 07 30 22 55 32

Let’s choose Install 12c Flex Cluster

Rac12c1 2013 07 30 22 56 04

And yes we picking English… since we live in English-land

Rac12c1 2013 07 30 22 56 29

Now let’s specify the Scan information and yes, we’ll need to define GNS and since we have to use GNS, we’ll need to get DNS domain delegation setup. In our case we have us.viscosityna-test.com as the sub-domain

Rac12c1 2013 07 30 22 57 07

This is the new stuff!! We define which nodes in the cluster will be Hubs and which will be Leaf. Note, you’ll ocassionaly hear the terms Hub and RIM be used interchangeably. It just historical!

Rac12c1 2013 07 30 22 57 42

Let’s specify the interfaces. You all have seen this screen before. But it now got a small twist to it. You can specify a separate “ASM &Private” networks.

Rac12c1 2013 07 30 22 58 32

Now the validation!

Rac12c1 2013 07 30 23 01 02

This step is new too. You have the option to configure Grid Infrastructure Repository,
which is used for storing Cluster Health Monitor (CHM) data. In 11gR2 this was stored
in a Berkley DB database and was created by default. Now this option allows users to
specify a Oracle Database to store the CHM data. This database is a single instance
database that is named MGMTDB by default. It is an internal CRS resource, which has
HA-failover capabilities. I’ll cover this topic in more detail later, but I should
mention that this is the only opportunity to create this repository; i,e, you have
uninstall/reinstall to get this GI repos option.

Rac12c1 2013 07 30 23 00 38

Now the fun stuff! Let’s create the ASM disk group. Note, that if you are configuring a GI repo, then you’ll need a minimum of 5GB disk (for testers and laptop folks).

Rac12c1 2013 07 31 17 47 57

Now define passwords

Rac12c1 2013 07 31 17 48 32

Verify…and yes we’re cool w/ the passwords

Rac12c1 2013 07 31 17 49 00

No IPMI

Rac12c1 2013 07 31 17 49 32

No define the group definitions

Rac12c1 2013 07 31 17 50 06

Where we gonna put the Oracle Home and Oracle Base

Rac12c1 2013 07 31 17 52 41

Now this really cool. I can specify the root password/credentials, for downstream root required actions.

Rac12c1 2013 07 31 18 02 05

Gotta run some fix some things, execute fixup.sh script

Rac12c1 2013 07 31 21 43 21

Now off to the races !!!

Rac12c1 2013 07 31 23 33 54

Secret Agentman – Clusterware Processes and Agents

Some of you may old enough to recall the song “Secret Agent Man” from Johnny Rivers:
There’s a man who leads a life of danger.
To everyone he meets he stays a stranger.
With every move he makes another chance he takes.
Odds are he won’t live to see tomorrow.

Well that’s how I felt when I was at a customer site recently (well maybe not exactly).

They recently had a issue with a node eviction. That in itself deserves a blog post later.
But anyways, he was asking “what are all these Clusterware processes and how do you even traverse through all the log files”.
After 15 mins of discussion, I realized I had thoroughly confused him.
So I suggested we start from the beginning and firstly try to understand Oracle Clusterware processes, agents, and relationships, then draw up some pictures. Maybe then we’ll have a better feel for hierarchy.

Let’s start with the grand master himself HAS (or OHASD)

OHASD manages clusterware daemons, including CRSD. We’ll discuss CRSD resources and startp in another blog. For now just keep in mind that OHASD starts up CRSD (at some point later in the stack), once CRSD is started, it manages the remaining startup of the stack

The “-init flag” is needed for crsctl to operate on OHASD resources,e.g. crsctl stat res ora.crsd -init
To list resources started by CRSD you would issue just “crsctl stat res”

OHASD resource startup order
ora.gipcd
ora.gpnpd -> Starts ora.mdnsd because of dependency
ora.cssd -> Starts ora.diskmon and ora.cssdmonitor because of dependency
ora.ctssd
ora.evmd
ora.crsd

OHASD has agents that work for him. These agents are oraagent, orarootagent, cssdagent and cssdmonitoragent. Each agent manages and handles very specific OHASD resources, and each agent runs as a specific user (root or, clusterware user).
For example, the ora.cssd resource (as root user) is started and monitored by the ora.cssdagent, whereas ora.asm is handled by the oraagent (running as cluster ware user).

All agent as well as other OHASD resource log files are in the CRS $ORACLE_HOME/log/hostname/agent/{ohasd|crsd}/agentname_owner/agentname_owner.log or in CRS $ORACLE_HOME/log/hostname/resource_name/resource_name.log; respectively.

To find out which agent is associated with a resource issue the following:

[root@rhel59a log]# crsctl stat res ora.cssd -init -p |grep “AGENT_FILENAME”
AGENT_FILENAME=%CRS_HOME%/bin/cssdagent%CRS_EXE_SUFFIX%

For example, for CRSD we find:

[root@rhel59a bin]# crsctl stat res ora.crsd -init -p |grep “AGENT_FILENAME”
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%

Note, an agent log file can have log messages for more than one resources, since those resources are managed by the same agent.

When I debug a resource, I start by going down the following Clusterware log file tree:
1. Start with Clusterware alert.log

2. Depending on the resource (managed by OHASD or CRSD) I look $ORACLE_HOME/logs//ohasd/ohasd.log or $ORACLE_HOME/logs//crsd/crsd.log

3. Then agent log file, as I mentioned above

4. Then finally to the resources log file itself (that’ll be listed in the agent log)

Item #2 requires a little more discussion, and will be the topic of our next discussion

My new Favorite RAC-Clusterware command

My new favorite 12c Oracle Clusterware command is the 'crsctl stat res "resource name" -dependency'

What this command does, is to provide a dependency tree structure for resource the in question.  This will display startup (default) and shutdown dependencies.  

From this we can understand the pull-up, pushdown, weak, and hard dependencies between clusterware resources 


[oracle@rac02 ~]$ crsctl stat res ora.dagobah.db -dependency
================================================================================
Resource Start Dependencies
================================================================================
---------------------------------ora.dagobah.db---------------------------------
ora.dagobah.db(ora.database.type)->
| type:ora.listener.type[weak:type]
| | type:ora.cluster_vip_net1.type[hard:type,pullup:type]
| | | ora.net1.network(ora.network.type)[hard,pullup]
| | | ora.gns<Resource not found>[weak:global]
| type:ora.scan_listener.type[weak:type:global]
| | ora.scan1.vip(ora.scan_vip.type)[hard,pullup]
| | | ora.net1.network(ora.network.type)[hard,pullup:global]
| | | ora.gns<Resource not found>[weak:global]
| | | type:ora.scan_vip.type[dispersion:type:active]
| | type:ora.scan_listener.type[dispersion:type:active]
| ora.ons(ora.ons.type)[weak:uniform]
| | ora.net1.network(ora.network.type)[hard,pullup]
| ora.gns<Resource not found>[weak:global]
| ora.PDBDATA.dg(ora.diskgroup.type)[weak:global:uniform]
| | ora.asm(ora.asm.type)[hard,pullup:always]
| | | ora.LISTENER.lsnr(ora.listener.type)[weak]
| | | | type:ora.cluster_vip_net1.type[hard:type,pullup:type]
| | | | | ora.net1.network(ora.network.type)[hard,pullup]
| | | | | ora.gns<Resource not found>[weak:global]
| | | ora.ASMNET1LSNR_ASM.lsnr(ora.asm_listener.type)[hard,pullup]
| | | | ora.gns<Resource not found>[weak:global]
| ora.FRA.dg(ora.diskgroup.type)[hard:global:uniform,pullup:global]
| | ora.asm(ora.asm.type)[hard,pullup:always]
| | | ora.LISTENER.lsnr(ora.listener.type)[weak]
| | | | type:ora.cluster_vip_net1.type[hard:type,pullup:type]
| | | | | ora.net1.network(ora.network.type)[hard,pullup]
| | | | | ora.gns<Resource not found>[weak:global]
| | | ora.ASMNET1LSNR_ASM.lsnr(ora.asm_listener.type)[hard,pullup]
| | | | ora.gns<Resource not found>[weak:global]
--------------------------------------------------------------------------------

Now the same for shutdown (pushdown) dependencies

[oracle@rac02 ~]$ crsctl stat res ora.dagobah.db -dependency -stop
================================================================================
Resource Stop Dependencies
================================================================================
---------------------------------ora.dagobah.db---------------------------------
ora.dagobah.db(ora.database.type)->
| ora.dagobah.hoth.svc(ora.service.type)[hard:intermediate]
| ora.dagobah.r2d2.svc(ora.service.type)[hard:intermediate]
--------------------------------------------------------------------------------

Why is this command and output important?  Well, in cases where a particular resource doesn't come up, you may want to understand relationship with its dependents
The reason is, if you are creating your own resource dependencies using the CRS API (formally known as CLSCRS API).

<pre>CLSCRS is a set of C-based APIs for Oracle Clusterware. The CLSCRS APIs enable you to manage the operation of entities that are managed by Oracle Clusterware. These entities include resources, resource types, servers, and server pools. You can use the APIs to register user applications with Oracle Clusterware so that the clusterware can manage them and maintain high availability. Once an application is registered, you can manage, monitor and query the application's status.  The APIs allow you to use the callbacks for diagnostic logging.

</pre>

Oh..GUID of PDB World


We have done a lot talks, sessions, blogs on Oracle 12c Pluggable Databases (PDB).  The question that seems to come up a lot is what is this long string of alphanumeric characters embedded in the database file name.

Before we answer that, let's take a trip down memory lane and understand how OMF works and its relationship with this GUID

Oracle Managed Files (OMF) was a feature introduced in 9i to minimize the overhead of managing database files.  Part of this feature is the database automatic naming of database files (on successful file creation). 
Files are named using system generated names and placed in the location as defined by the DB_CREATE_FILE_DEST init.ora .  In Data Guard configurations there are *_FILE_NAME_CONVERT, STANDBY_FILE_MANAGEMENT to assist with converting names of existing and newly created datafiles when OMF is in use.


OMF really came into play with widespread use, because of the implementation of ASM.  When ASM is used, OMF is inherently used for file management.  The ASM-OMF directory structure for datafiles traditionally consists of //DATAFILE/.  A traditional file name in ASM consists of 3 parts, ...  For example:


+PDBDATA/YODA/DATAFILE/system.258.8238921091

Note,Users are not allowed to directly create files with this naming structure, if you try you'll get a single-form file name error ORA-15046!


So what does this GUID thingy mean for 12c PDB configurations with ASM-OMF.  In addition to the OMF file naming and directory structure (discussed above), there is an embedded global unique identifier (GUID).  The GUID is globally unique immutable ID assigned to the 12c database at creation time.  Each 12c database, whether its non-CDB, CDB, or PDB, has a GUID associated with it.  Thus, with PDB, the directory structure changes for each pluggable database (PDB) in a container database (CDB). 


For pre-12c non-CDB databases, the GUID will be created when the database is upgraded to 12c.  


There are so many identifiers for a 12c database, let's make sure we get this straight. There's dbid, con_id, con_uid, and guid. The DBID is the database id embedded in the database file, control file, redo log header. The con_id is simply a container number in that specific CDB, starts with 0 and 1 is for root PDB. The con_uid is a local unique identifier within that CDB. The GUID is universal across all CDB/PDB.  Keep in mind that we can unplug a PDB from one CDB into another CDB, so the GUID provides this uniqueness and streamlines portability. More on this later!


The following query shows the different 12c database identifiers:

CDB$ROOT@YODA> select CON_ID,DBID,NAME,TOTAL_SIZE from v$pdbs;    
CON_ID      DBID     NAME                     TOTAL_SIZE
---------- ---------- -------------          -------------      
2    4066465523 PDB$SEED                      283115520      
3     483260478 PDBOBI                        917504000      
4     994649056 PDBVADER                              0

Note, that the GUID does not change throughout the life of the PDB/non-CDB. The GUID for a particular container/non-CDB can be found by querying V$CONTAINERS or v$PDBs. To assist with identifying which files belong to which PDB, an ASM directory structure of ///DATAFILE/ is used for PDBs. This is one of the main reasons a PDB should be cloned (cloning generates a new GUID) rather than copying the same PDB to multilple locations and plugging in to multiple CDBs.

See the example below, for GUID:

 
CDB$ROOT@YODA> select name, con_id from v$datafile order by con_id
NAME                                                                                    CON_ID
----------------------------------------------------------------------------------- ----------
+PDBDATA/YODA/DATAFILE/undotbs1.260.823892155                                                1
+PDBDATA/YODA/DATAFILE/sysaux.257.823892063                                                  1
+PDBDATA/YODA/DATAFILE/system.258.823892109                                                  1
+PDBDATA/YODA/DATAFILE/users.259.823892155                                                   1
+PDBDATA/YODA/DD7C48AA5A4404A2E04325AAE80A403C/DATAFILE/system.271.823892297                 2
+PDBDATA/YODA/DD7C48AA5A4404A2E04325AAE80A403C/DATAFILE/sysaux.270.823892297                 2
+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/example.275.823892813                3
+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/users.277.823892813                  3
+PDBDATA/YODA/E456D87DF75E6553E043EDFE10AC71EA/DATAFILE/obiwan.284.824683339                 3
+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/system.276.823892813                 3
+PDBDATA/YODA/DD7D8C1D4C234B38E04325AAE80AF577/DATAFILE/sysaux.274.823892813                 3
+PDBDATA/YODA/E46B24386A131109E043EDFE10AC6E89/DATAFILE/sysaux.279.823980769                 4
+PDBDATA/YODA/E46B24386A131109E043EDFE10AC6E89/DATAFILE/users.281.823980769                  4
+PDBDATA/YODA/E46B24386A131109E043EDFE10AC6E89/DATAFILE/example.282.823980769                4
+PDBDATA/YODA/E46B24386A131109E043EDFE10AC6E89/DATAFILE/system.280.823980769                 4

That long identifier, "E46B24386A131109E043EDFE10AC6E89", in the OMF name, is the GUID.

Now a similar example from ASM (asmcmd) perspective

ASMCMD [+PDBDATA] > ls -l dagobah
Type           Redund  Striped  Time             Sys  Name
                                                 Y    CONTROLFILE/
                                                 Y    DATAFILE/
                                                 Y    DD7C48AA5A4404A2E04325AAE80A403C/
                                                 Y    F2F952556B226FA5E0430B2910AC1FE5/
                                                 Y    ONLINELOG/
                                                 Y    PARAMETERFILE/
                                                 Y    PASSWORD/
                                                 Y    TEMPFILE/
PASSWORD       UNPROT  COARSE   FEB 21 23:00:00  N    orapwdagobah => +PDBDATA/DAGOBAH/PASSWORD/pwddagobah.293.840152893
PARAMETERFILE  UNPROT  COARSE   APR 09 15:00:00  N    spfiledagobah.ora => +PDBDATA/DAGOBAH/PARAMETERFILE/spfile.310.840153477

ASMCMD [+PDBDATA/TATOOINE/F2F7CA2C1F1F0593E0430A2910AC246A/datafile] > ls
SYSAUX.258.840146475
SYSTEM.286.840146475

Let's look at two examples of PDB creation and the GUID.
Example1. This example illustrates the PDB creation and GUID


SQL> CREATE PLUGGABLE DATABASE pdbhansolo admin user hansolo identified by hansolo roles=(dba);

Pluggable database created.

SQL> select * from v$pdbs ;


    CON_ID       DBID    CON_UID GUID                             NAME      OPEN_MODE  RES     OPEN_TIME             CREATE_SCN TOTAL_SIZE
---------- ---------- ---------- -------------------------------- ------------------- ------   -----------           ----------- ------------
         2 4080865680 4080865680 F13EFFD958E24857E0430B2910ACF6FD PDB$SEED   READ ONLY  NO  17-FEB-14 01.01.13.909 PM   1720768  283115520
         3 3403102439 3403102439 F2A023F791663F8DE0430B2910AC37F7 PDBHANSOLO MOUNTED        17-FEB-14 01.27.08.942 PM   1846849          0

Example two. Here, we are going to plug in (convert) a PDB from a non-CDB. Note, that we can see the GUID in manifest file. In the XML output below (from manifest xml file), you see the GUID listed for this non-CBD

<PDB>
  <pdbname>wookie</pdbname>
  <cid>0</cid>
  <byteorder>1</byteorder>
  <vsn>202375168</vsn>
  <dbid>2940614436</dbid>
  <cdbid>2940614436</cdbid>
  <guid>F2BBDF340FFE3E90E0430B2910AC097F</guid>

Now connect to the CDB and create the Wookie PDB from the manifest file

CDB_SQL>CREATE PLUGGABLE DATABASE wookie USING '/home/oracle/wookie_pdb.xml'
  NOCOPY;


Pluggable database created.

SQL> select name, open_mode from v$pdbs;


NAME                           OPEN_MODE
------------------------------ ----------
PDB$SEED                       READ ONLY
WOOKIE                         MOUNTED

SQL> select name, guid from v$pdbs;


NAME                           GUID
------------------------------ --------------------------------
PDB$SEED                       F13EFFD958E24857E0430B2910ACF6FD
WOOKIE                         F2BBDF340FFE3E90E0430B2910AC097F

Here's where the big issue comes in. Many DBAs have mentioned to me that there is no real way to identify the PDB by solely looking at the path name. We do however, know the name of the CDB its in, but that's as far as we can go. In order to determine the PDB associated with the file, you would need to login directly to PDB (not even the CDB), and get the name

Initially there are some issues w/ GUID/OMF/ASM when are files are copied, and a physical standby database is in place. There have been improvements made to the multitenant plugin operation on both the primary and standby environments, e.g., You need PSU 2 at least, then DataGuard will do the right thing when you plug on a new pdb at the primary after making sure the files are at the standby first. RMAN has been enhanced so that, when copying files between databases it recognizes the GUID and acts accordingly when writing the files.

Here are some additional RMAN considerations for GUID management

* If the clone/auxiliary instance being connected to for clone operations is a CDB root, the GUID of the RMAN target database is used to determine the directory structure to write the datafiles. Connect to the CDB root as the RMAN clone/auxiliary instance when the source database should be a 12c non-CDB or PDB that is going to be migrated and plugged into a remote CDB as a brand new PDB. This will ensure that the files copied by RMAN will be written to the GUID directory of source database for the migration.
* If the clone/auxiliary instance being connected to for clone operations is a PDB, the GUID of the auxiliary PDB will be used to determine the directory structure to write the datafiles. Connect to the destination PDB as the RMAN clone auxiliary instance when the source database is a 12c non-CDB or PDB that requires a cross platform full transportable database import and the data and files will be imported into an existing PDB. This will ensure the files copied by RMAN will be written to the GUID directory of the PDB target database for the migration.

* The enhancements for multitenant plugin operations with OMF simplify the process extensively. The manifest generated on the source non-CDB/PDB contains all of the filenames and characteristics about each file. Normally, the plugin operation would use the filenames in the manifest and look for those exact filenames or partially converted (using the SOURCE_FILE_NAME_CONVERT clause on the CREATE PLUGGABLE DATABASE....USING...statement). Since all filenames will be different when copied to a new location when OMF is used, you would need to specify full directory and filename convert pairs for EACH file being plugged in. By using the SOURCE_FILE_DIRECTORY clause on the CREATE PLUGGABLE DATABASE....USING... statement in the plugin operation, the filename in the manifest is ignored and the plugin looks for a file to match additional characteristics about the file stored in the manifest, looking for the file in the SOURCE_FILE_DIRECTORY location.