Saturday, October 15, 2016

ORA-28374: typed master key not found in wallet (TDE)

basically we are doing TDE configuration for 12c env.
we are facing the below issues

---- while trying to set master key with container=all option
SQL> ADMINISTER KEY MANAGEMENT SET KEY USING TAG 'scope:all pdbs;description:Create Key for ALL PDBS' IDENTIFIED BY enc_keystore WITH BACKUP CONTAINER = ALL ;
ADMINISTER KEY MANAGEMENT SET KEY USING TAG 'scope:all pdbs;description:Create Key for ALL PDBS' IDENTIFIED BY enc_keystore WITH BACKUP CONTAINER = ALL
*
ERROR at line 1:
ORA-46665: master keys not activated for all PDBs during REKEY

--- next tried to manually set it at PDB level

SQL> alter session set container=orcl;

Session altered.

SQL> select * from V$ENCRYPTION_WALLET;

WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID
-------------------- ---------------------------------------- ------------------------------ -------------------- --------- --------- ----------
ASM +DATA/orclt1/WALLET/ CLOSED UNKNOWN SINGLE UNDEFINED 0

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY enc_keystore;

keystore altered.

SQL> select * from V$ENCRYPTION_WALLET;

WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID
-------------------- ---------------------------------------- ------------------------------ -------------------- --------- --------- ----------
ASM +DATA/orclt1/WALLET/ OPEN_NO_MASTER_KEY PASSWORD SINGLE UNDEFINED 0

SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY enc_keystore WITH BACKUP;
ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY enc_keystore WITH BACKUP
*
ERROR at line 1:
ORA-28374: typed master key not found in wallet

==================================================================================================================================================================
now the solution follows (this problem might occure due to repeated deletion of ewallet.pk2/wallet files or retrying a earlier failed trial for TDE configuration)
==================================================================================================================================================================

STEP1 (check in both ROOT & PDBs)
==================================
 Drop and recreate all encrypted tablespaces as unencrypted (except SYSTEM), and this goes for the PDB as well. Check DBA_TABLESPACES;

Only once this is done, proceed with the following, as SYSDBA:

--- to check this use the below queries in both ROOT & PDBs

- decrypt all encrypted columns using a statement like: ALTER TABLE scott.tde MODIFY (c DECRYPT);
or drop all tables that still have an entry in dba_encrypted_columns:
select * from dba_encrypted_columns;
- remove any remaining rows not associated with OBJ$ objects from table sys.ENC$ :
select * from sys.enc$;
- move all tables out of your encrypted tablespaces (including indexes and LOB segments)
select t.owner,t.table_name, t.tablespace_name from dba_tables t,
dba_tablespaces s where t.tablespace_name=s.tablespace_name
and s.encrypted = 'YES';
- drop the encrypted tablespaces
select tablespace_name from dba_tablespaces where encrypted = 'YES';
- run this sql:
update TS$ set flags = flags - 16384 where online$ = 3 and bitand(flags,16384) = 16384 ;
commit;

1) Close the keystore to flush the master keys out of memory:

ADMINISTER KEY MANAGEMENT SET KEYSTORE CLOSE IDENTIFIED BY ... ;

2) Switch all of the REDO log files in the ROOT and PDB:

alter system switch logfile;
alter system switch logfile;
alter system switch logfile;

alter session set container=<PDB>;

alter system switch logfile;
alter system switch logfile;
alter system switch logfile;

3) Move the TDE wallet (ewallet.p12) to a backup location.


NOTE: _db_discard_lost_masterkey this hidden parameter is used below , do create a pfile before proceeding further below
sometimes database will fail to start after using this parameter (end of file error) , then from the pfile remove the entry (_db_discard_lost_masterkey) and restart DB.

STEp2 (in ROOT container)
=============================

SQL> conn / as sysdba
Connected.
SQL>
SQL>
SQL>
SQL> show con_id

CON_ID
------------------------------
1
SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT
SQL> set lines 234 pages 234
SQL> col WRL_PARAMETER for a40
SQL> select WRL_TYPE, WRL_PARAMETER, STATUS from V$ENCRYPTION_WALLET;

WRL_TYPE WRL_PARAMETER STATUS
-------------------- ---------------------------------------- ------------------------------
ASM +DATA/orclt1/WALLET/ NOT_AVAILABLE

SQL> select CON_ID,KEY_ID,KEYSTORE_TYPE,CREATOR_DBNAME,CREATOR_PDBNAME from v$encryption_keys;

no rows selected

SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '+DATA/orclt1/WALLET/' IDENTIFIED BY enc_keystore;

keystore altered.

SQL> ALTER SYSTEM SET "_db_discard_lost_masterkey"=TRUE SCOPE=MEMORY;

System altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY enc_keystore;

keystore altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY enc_keystore WITH BACKUP;

keystore altered.

SQL> select WRL_TYPE, WRL_PARAMETER, STATUS from V$ENCRYPTION_WALLET;

WRL_TYPE WRL_PARAMETER STATUS
-------------------- ---------------------------------------- ------------------------------
ASM +DATA/orclt1/WALLET/ OPEN


STEP3
=======

steps done in PDB
==========
SQL> alter session set container=orcl;

Session altered.

SQL> select WRL_TYPE, WRL_PARAMETER, STATUS from V$ENCRYPTION_WALLET;

WRL_TYPE WRL_PARAMETER STATUS
-------------------- ---------------------------------------- ------------------------------
ASM +DATA/orclt1/WALLET/ CLOSED

SQL> alter system set "_db_discard_lost_masterkey"=true;

System altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY enc_keystore WITH BACKUP;
ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY enc_keystore WITH BACKUP
*
ERROR at line 1:
ORA-46658: keystore not open in the container


SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY enc_keystore;

keystore altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY enc_keystore WITH BACKUP;

keystore altered.

SQL> select WRL_TYPE, WRL_PARAMETER, STATUS from V$ENCRYPTION_WALLET;

WRL_TYPE WRL_PARAMETER STATUS
-------------------- ---------------------------------------- ------------------------------
ASM +DATA/orclt1/WALLET/ OPEN



TDE configuration in 12C

--- below was done for a 2node 12C RAC
--- wallet location was instance specific so i did configure for 1st instance then copied all the TDE files to respective location on 2nd instance & restarted 2nd instance
--- pre check for TDE

subhen002:orclt1 > sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Fri Jul 1 15:53:38 2016

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics, Real Application Testing and Unified Auditing options

SQL> select * from V$ENCRYPTION_WALLET;

WRL_TYPE
--------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC
------------------------------ -------------------- --------- ---------
    CON_ID
----------
FILE
/dboracle/orabase/admin/orcl_tp/wallet
NOT_AVAILABLE                  UNKNOWN              SINGLE    UNDEFINED
         0



--- take backup of sqlnet.ora and made entry to both nodes as below

subhen002:orclt1 > cp -p sqlnet.ora sqlnet.ora_bkp
subhen002:orclt1 > vi sqlnet.ora
#---------------------------#
# Network Tuning Parameters #
#---------------------------#
default_sdu_size=32767
send_buf_size=1048576
recv_buf_size=1048576
tcp.nodelay=yes
sqlnet.inbound_connect_timeout=180
sqlnet.expire_time=10

#TDE entry

ENCRYPTION_WALLET_LOCATION=
 (SOURCE=
  (METHOD=FILE)
   (METHOD_DATA=
    (DIRECTORY=+DATA_DG01/$ORACLE_SID/WALLET)
   )
 )
~

--- create directories in ASM

SQL> ALTER DISKGROUP DATA_DG01 ADD DIRECTORY '+DATA_DG01/orclt1';

Diskgroup altered.

SQL> ALTER DISKGROUP DATA_DG01 ADD DIRECTORY '+DATA_DG01/orclt1/WALLET/';

Diskgroup altered.

SQL> ALTER DISKGROUP DATA_DG01 ADD DIRECTORY '+DATA_DG01/orclt2';

Diskgroup altered.

SQL> ALTER DISKGROUP DATA_DG01 ADD DIRECTORY '+DATA_DG01/orclt2/WALLET/';

Diskgroup altered.


--- check again it should show ASM path

SQL> set lines 234 pages 234
SQL> col WRL_PARAMETER for a40
SQL> select WRL_TYPE, WRL_PARAMETER, STATUS from V$ENCRYPTION_WALLET;

WRL_TYPE             WRL_PARAMETER                            STATUS
-------------------- ---------------------------------------- ------------------------------
ASM                  +DATA_DG01/orclt1/WALLET/               NOT_AVAILABLE

-- as root user and in root container execute the below sqls
SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT
SQL> show user
USER is "SYS"

SQL> set lines 234 pages 234
SQL> col WRL_PARAMETER for a40
SQL> select WRL_TYPE, WRL_PARAMETER, STATUS from V$ENCRYPTION_WALLET;

WRL_TYPE             WRL_PARAMETER                            STATUS
-------------------- ---------------------------------------- ------------------------------
ASM                  +DATA_DG01/orclt1/WALLET/               NOT_AVAILABLE

SQL> select CON_ID,KEY_ID,KEYSTORE_TYPE,CREATOR_DBNAME,CREATOR_PDBNAME from v$encryption_keys;

no rows selected

SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '+DATA_DG01/orclt1/WALLET/' IDENTIFIED BY enc_keystore;

keystore altered.

SQL> select WRL_TYPE, WRL_PARAMETER, STATUS from V$ENCRYPTION_WALLET;

WRL_TYPE             WRL_PARAMETER                            STATUS
-------------------- ---------------------------------------- ------------------------------
ASM                  +DATA_DG01/orclt1/WALLET/               CLOSED

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY enc_keystore  CONTAINER = ALL ;

keystore altered.

SQL> select * from V$ENCRYPTION_WALLET;

WRL_TYPE             WRL_PARAMETER                            STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- ---------------------------------------- ------------------------------ -------------------- --------- --------- ----------
ASM                  +DATA_DG01/orclt1/WALLET/               OPEN_NO_MASTER_KEY             PASSWORD             SINGLE    UNDEFINED          0

SQL> ADMINISTER KEY MANAGEMENT SET KEY USING TAG 'scope:all pdbs;description:Create Key for ALL PDBS' IDENTIFIED BY enc_keystore WITH BACKUP CONTAINER = ALL ;

keystore altered.

--- verify the creation in root & all PDBs

SQL> select * from V$ENCRYPTION_WALLET;

WRL_TYPE             WRL_PARAMETER                            STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- ---------------------------------------- ------------------------------ -------------------- --------- --------- ----------
ASM                  +DATA_DG01/orclt1/WALLET/               OPEN                           PASSWORD             SINGLE    NO                 0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 orcl                          READ WRITE NO
SQL> alter session set container=orcl
  2  ;

Session altered.

SQL> select * from V$ENCRYPTION_WALLET;

WRL_TYPE             WRL_PARAMETER                            STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- ---------------------------------------- ------------------------------ -------------------- --------- --------- ----------
ASM                  +DATA_DG01/orclt1/WALLET/               OPEN                           PASSWORD             SINGLE    NO                 0

SQL> select CON_ID,KEY_ID,KEYSTORE_TYPE,CREATOR_DBNAME,CREATOR_PDBNAME from v$encryption_keys;

    CON_ID KEY_ID                                                                         KEYSTORE_TYPE     CREATOR_DBNAME                 CREATOR_PDBNAME
---------- ------------------------------------------------------------------------------ ----------------- ------------------------------ ------------------------------
         0 AY/4VjtNE0+4vxqV8VtEEB4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA                           SOFTWARE KEYSTORE orcl_tp                       orcl


--- create a sample user/schema for TDE use

SQL> show con_name

CON_NAME
------------------------------
orcl
SQL> create user tde_admin identified by tde_admin;

User created.

SQL> grant CREATE TABLE, CREATE TABLESPACE to tde_admin;

Grant succeeded.

SQL> grant connect,resource to tde_admin;

Grant succeeded.

SQL> exit

---- create a encrypted tablespace and assign to tde_admin

SQL> alter session set container=orcl
  2  ;

Session altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 orcl                          READ WRITE NO
SQL> show con_name

CON_NAME
------------------------------
orcl
SQL> CREATE TABLESPACE encrypt_ts
  DATAFILE '+DATA_DG01' SIZE 1M
  ENCRYPTION USING 'AES256'
  DEFAULT STORAGE (ENCRYPT);  2    3    4

Tablespace created.

SQL> alter user tde_admin default tablespace encrypt_ts;

User altered.

SQL> show user
USER is "SYS"

--- check the tablespace should be created as encrypted

SQL> conn / as sysdba
Connected.
SQL> alter session set container=orcl;

Session altered.

SQL> select TABLESPACE_NAME,ENCRYPTED from dba_tablespaces;

TABLESPACE_NAME                ENC
------------------------------ ---
SYSTEM                         NO
SYSAUX                         NO
TEMP                           NO
USERS                          NO
AUDIT_DATA                     NO
ENCRYPT_TS                     YES

6 rows selected.

--- test it by creating a sample table in the encrypted tablespace

subhen002:orclt1 > sqlplus TDE_ADMIN/tde_admin@orcl_any

SQL*Plus: Release 12.1.0.2.0 Production on Fri Jul 1 16:26:52 2016

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics, Real Application Testing and Unified Auditing options

SQL> show user
USER is "TDE_ADMIN"
SQL> create table test (id number(10)) tablespace encrypt_ts;

Table created.


----enable autologin and restart database


SQL> conn / as sysdba
Connected.

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT
SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE '+DATA_DG01/orclt1/WALLET/' IDENTIFIED BY enc_keystore;

keystore altered.

SQL> select * from V$ENCRYPTION_WALLET;

WRL_TYPE
--------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC
------------------------------ -------------------- --------- ---------
    CON_ID
----------
ASM
+DATA_DG01/orclt1/WALLET/
OPEN                           PASSWORD             SINGLE    NO
         0


SQL> shu immediate

---- for 2nd instance TDE configure

jsut copy the wallet files to respective location for 2nd instance and restart the instance


Wednesday, June 1, 2016

ORA-29548: Java system class reported: could not identify release specified in classes.bin

In our env. after migration of DB , the below error was facing


SQL>  conn profileuserr6_0/*****
Connected.
SQL> select SHA256.encrypt('TEST') from dual;
select SHA256.encrypt('TEST') from dual
       *
ERROR at line 1:
ORA-29548: Java system class reported: could not identify release specified in
classes.bin
ORA-06512: at "PROFILEUSERR6_0.SHA256", line 9

so we have followed Doc ID 1995261.1 and resolved this

in our env the second case was matching as described in doc. i.e. both patch 19121548 & 19282015 was present


subhen_host001:testDBa1 > /dboracle/orabase/product/11.2.0.3.15.0/OPatch/opatch lsinv|grep -i "19282015"
     19909862, 16885033, 20406220, 19282015, 13359289, 19895326, 13023632
subhen_host001:testDBa1 > /dboracle/orabase/product/11.2.0.3.15.0/OPatch/opatch lsinv|grep -i "19121548"
Sub-patch  19121548; "Database Patch Set Update : 11.2.0.3.12 (19121548)"
     14095982, 17333203, 19121548, 13591624, 14523004, 13440516, 16794241


the complete opatch output was as below

subhen_host001:testDBa1 > /dboracle/orabase/product/11.2.0.3.15.0/OPatch/opatch lsinv|grep -i "patch set"
Patch description:  "Database Patch Set Update : 11.2.0.3.15 (20760997)"
Sub-patch  20299017; "Database Patch Set Update : 11.2.0.3.14 (20299017)"
Sub-patch  19769496; "Database Patch Set Update : 11.2.0.3.13 (19769496)"
Sub-patch  19121548; "Database Patch Set Update : 11.2.0.3.12 (19121548)"
Sub-patch  18522512; "Database Patch Set Update : 11.2.0.3.11 (18522512)"
Sub-patch  18031683; "Database Patch Set Update : 11.2.0.3.10 (18031683)"
Sub-patch  17540582; "Database Patch Set Update : 11.2.0.3.9 (17540582)"
Sub-patch  16902043; "Database Patch Set Update : 11.2.0.3.8 (16902043)"
Sub-patch  16619892; "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch  16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.9 (HAS Components)"


Solution : As a solution i had downloaded patch 19282015 from MOS and just ran the post install part as decribed in doc 1995261.1

below location i unzipped the patch and postinstall.sql file is there.

subhen_host001:testDBa1 > pwd
/datastore/software/19282015
subhen_host001:testDBa1 > ls -ltr
total 56
-rw-r--r--. 1 oracle oinstall    18 Oct 11  2014 README.txt
-rw-r--r--. 1 oracle oinstall   466 Oct 11  2014 postinstall.sql
-rw-r--r--. 1 oracle oinstall   354 Oct 11  2014 postdeinstall.sql
drwxr-xr-x. 4 oracle oinstall  4096 Oct 11  2014 etc/
drwxr-xr-x. 2 oracle oinstall  4096 Oct 11  2014 custom/
drwxr-xr-x. 8 oracle oinstall  4096 Oct 11  2014 files/
-rw-r--r--. 1 oracle oinstall 31139 Oct 30  2014 README.html

SQL> alter system set java_jit_enabled = FALSE scope=both sid='*';

System altered.

SQL> alter system set "_system_trig_enabled"=FALSE scope=both sid='*';

System altered.

SQL> alter system set JOB_QUEUE_PROCESSES=0  scope=both sid='*';

System altered.


subhen_host001:testDBa1 > srvctl status database -d ****_ap
Instance ****a1 is running on node subhen_host001
Instance ****a2 is running on node subhen_host002
subhen_host001:testDBa1 > srvctl stop database -d testDB_ap -o immediate
subhen_host001:testDBa1 > sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Wed Jun 1 15:12:07 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup restrict


SQL> @/datastore/software/19282015/postinstall.sql

--- post execution of script change the parameter values again.

SQL> alter system set java_jit_enabled = true scope=both sid='*';

System altered.

SQL> alter system set "_system_trig_enabled"=TRUE scope=both sid='*';

System altered.

SQL> alter system set JOB_QUEUE_PROCESSES=1000 scope=both sid='*';

System altered.

SQL> shu immediate

subhen_host001:pfcita1 > srvctl start database -d pfcit_ap

--- now the issue was resolved.

SQL> select SHA256.encrypt('TEST') from dual;

SHA256.ENCRYPT('TEST')
--------------------------------------------------------------------------------
lO4FkzXlh+UBzEv5BhPggU8Ap7CLx8ZI/YZaKvaiLMI=

Wednesday, May 11, 2016

OPatch finds invalid inventory. The patch storage directory '$ORACLE_HOME\.patch_storage' is present but Standalone Inventory '$ORACLE_HOME\sainventory' is not present

for the below issue follow MOS note:1477475.1

Opatch Commands Failing with Error: "OPatch finds invalid inventory. The patch storage directory '$ORACLE_HOME\.patch_storage' is present but Standalone Inventory '$ORACLE_HOME\sainventory' is not present." (Doc ID 1477475.1)

addtional note if needed : Steps To Recreate Central Inventory(oraInventory) In RDBMS Homes (Doc ID 556834.1)

===========================================
subhen_host:ORCL > /dboracle/orabase/product/11.2.0.4/OPatch/opatch lsinv
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home            : /dboracle/orabase/product/11.2.0.4
Oracle Home Inventory  : n/a
OPatch version         : 11.2.0.3.4
Product information    : n/a
Log file location      : /dboracle/orabase/product/11.2.0.4/cfgtoollogs/opatch/opatch2016-05-11_14-00-54PM_1.log

Patch history file: /dboracle/orabase/product/11.2.0.4/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /dboracle/orabase/product/11.2.0.4/cfgtoollogs/opatch/lsinv/lsinventory2016-05-11_14-00-54PM.txt

--------------------------------------------------------------------------------
There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------

OPatch succeeded.

==============================================
for below issue also check the similar locations , i copied the missing file from other servers and it worked

subhen_host:ORCL > pwd
/dboracle/oraInventory/ContentsXML
subhen_host:ORCL > ls -tlr
total 8
-rw-r----- 1 oracle oinstall 329 May 11 14:34 comps.xml
subhen_host:ORCL > ls -ltr
total 24
-rw-r----- 1 oracle oinstall 329 May 11 14:34 comps.xml
-rw-r----- 1 oracle oinstall 618 May 11 14:36 inventory.xml
-rw-r----- 1 oracle oinstall 292 May 11 14:36 libs.xml


subhen_host:ORCL > /dboracle/orabase/product/11.2.0.4/OPatch/opatch lsinv
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /dboracle/orabase/product/11.2.0.4
Central Inventory : /dboracle/oraInventory
   from           : /dboracle/orabase/product/11.2.0.4/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.4.0
Log file location : /dboracle/orabase/product/11.2.0.4/cfgtoollogs/opatch/opatch2016-05-11_14-06-49PM_1.log

List of Homes on this system:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73


ORA-00205: error in identifying control file during DB upgrade from 11.2.0.3 to 11.2.0.4

scenario : we were doing a upgrade from 11.2.0.3 to 11.2.0.4.
we have newly installed 11.2.0.4 software and going to upgrade the first DB
we ran into the below issue while "startup upgrade" command as per MOS note 1054033.1 we found out the issue lies with the wrong group ownership of oracle executable as shown below
after making below changes on all RAC nodes my upgrade went fine without any issue

In my case, the oracle binary is owned by
-rwsr-s--x 1 oracle oinstall
instead of
-rwsr-s--x 1 oracle asmadmin

solution:

1. Manually change the oracle binary ownership and permission:

as root user:
# cd $ORACLE_HOME/bin
# chgrp asmadmin oracle
# chmod 6751 oracle
# ls -l oracle

--below is the error snapshot
subhen_host:ORCL > sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed May 11 15:29:14 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup upgrade
ORACLE instance started.

Total System Global Area 1570009088 bytes
Fixed Size                  2253584 bytes
Variable Size             469765360 bytes
Database Buffers         1073741824 bytes
Redo Buffers               24248320 bytes
ORA-00205: error in identifying control file, check alert log for more info

Sunday, April 10, 2016

manual DB creation using datafile copy method in same server (putting DB in begin backup mode)

--- put source DB in begin backup mode and copy the intended datafiles (here in my case i copied datafiles belong to system,sysaux & UNDO)

 note: in begin backup mode user still able to connect to DB normally but a huge number of archivelof generation might be expected

-- after datafiles copy finished end backup and cross check the view v$backup_files

-- sample pfile for the easy clone or datafile copy method database creation
-- make a special note of any conflicting parameters such as service names if you are starting the new database in same server

subhen_DBHOST: vi inittarget_DB.ora
"inittarget_DB.ora" [New file]
i*.aq_tm_processes=0
*.audit_sys_operations=TRUE
*.audit_trail='DB'
*.background_core_dump='PARTIAL'
*.background_dump_dest='/dboracle/admin/target_DB/bdump'
*.backup_tape_io_slaves=TRUE
*.COMPATIBLE='10.0.0'
*.control_file_record_keep_time=40
*.control_files='/dborafiles/rpoviw1/source_DB/oradata03/target_DB/control01.ctl','/dborafiles/rpoviw1/source_DB/oradata02/target_DB/control02.ctl'
*.core_dump_dest='/dboracle/admin/target_DB/cdump'
*.cursor_sharing='EXACT'
*.DB_BLOCK_CHECKSUM='TRUE'
*.DB_BLOCK_SIZE=16384
*.db_cache_size=950m
*.db_domain=''
*.DB_FILE_MULTIBLOCK_READ_COUNT=32
*.DB_FILES=100
*.db_name='target_DB'
*.db_recovery_file_dest_size=1073741824
*.db_recovery_file_dest='/dboracle'
*.db_unique_name='target_DB'
*.dg_broker_start=FALSE
*.fast_start_parallel_rollback='FALSE'
*.global_names=FALSE
*.instance_name='target_DB'
*.JAVA_POOL_SIZE=15M
*.JOB_QUEUE_PROCESSES=2
*.large_pool_size=8M
*.log_archive_dest_1='LOCATION=/dborafiles/rpoviw1/source_DB/oraarch01/target_DB'
*.log_archive_format='target_DB_%t_%s_%r.arch'
*.log_buffer=209715200
*.LOG_CHECKPOINT_INTERVAL=0
*.LOG_CHECKPOINT_TIMEOUT=0
*.log_checkpoints_to_alert=true
*.max_dump_file_size='10240000'
*.NLS_CURRENCY='£'
*.NLS_DATE_FORMAT='DD-MON-RRRR'
*.NLS_DATE_LANGUAGE='ENGLISH'
*.NLS_ISO_CURRENCY='UNITED KINGDOM'
*.NLS_LANGUAGE='ENGLISH'
*.NLS_TERRITORY='UNITED KINGDOM'
*.open_cursors=2000
*.OPTIMIZER_DYNAMIC_SAMPLING=1
*.optimizer_features_enable='10.2.0.2'
*.optimizer_index_cost_adj=1
*.optimizer_mode='ALL_ROWS'
*.os_roles=FALSE
*.PARALLEL_MAX_SERVERS=40# CR4449 161003
*.pga_aggregate_target=1048576000
*.processes=1500
*.remote_login_passwordfile='EXCLUSIVE'
*.remote_os_authent=FALSE
*.remote_os_roles=FALSE
*.resource_limit=TRUE
*.session_cached_cursors=50
*.session_max_open_files=50
*.sga_max_size=2048m
*.sga_target=2048m
*.shadow_core_dump='PARTIAL'
*.shared_pool_reserved_size=52428800
*.shared_pool_size=500m
*.sql_trace=FALSE
*.statistics_level='TYPICAL'
*.streams_pool_size=50M
*.TIMED_STATISTICS=TRUE
*.trace_enabled=FALSE
*.UNDO_MANAGEMENT='AUTO'
*.UNDO_RETENTION=10800
*.UNDO_TABLESPACE='UNDO'
*.user_dump_dest='/dboracle/admin/target_DB/udump'
*.diagnostic_dest='/dboracle'
"inittarget_DB.ora" [New file] 71 lines, 2166 characters
subhen_DBHOST: ls -ltr
-- start with the pfile

subhen_DBHOST: . oraenv
ORACLE_SID = [target_DB] ?
The Oracle base remains unchanged with value /dboracle
subhen_DBHOST: echo $ORACLE_SID
target_DB
subhen_DBHOST: echo $ORACLE_HOME
/dboracle/product/11.2.0.3
subhen_DBHOST: sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Wed Apr 6 14:47:46 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile=inittarget_DB.ora
ORA-32006: BACKGROUND_DUMP_DEST initialization parameter has been deprecated
ORA-32006: REMOTE_OS_AUTHENT initialization parameter has been deprecated
ORA-32006: SQL_TRACE initialization parameter has been deprecated
ORA-32006: USER_DUMP_DEST initialization parameter has been deprecated
ORACLE instance started.

Total System Global Area 2138521600 bytes
Fixed Size                  2161024 bytes
Variable Size             637535872 bytes
Database Buffers         1275068416 bytes
Redo Buffers              223756288 bytes

-- below is the sample controlfile ( created by using create control file to trace command in source DB and edited to suit)

subhen_DBHOST: view create_controlfile_target_DB.sql
"create_controlfile_target_DB.sql" [Read only] 24 lines, 941 characters
CREATE CONTROLFILE SET DATABASE "target_DB" RESETLOGS
    MAXLOGFILES 16
    MAXLOGMEMBERS 2
    MAXDATAFILES 1024
    MAXINSTANCES 1
    MAXLOGHISTORY 10452
LOGFILE
  GROUP 1 (
    '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/redo_log1a.log',
    '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/redo_log1b.log'
  ) SIZE 100M BLOCKSIZE 512,
  GROUP 2 (
    '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/redo_log2a.log',
    '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/redo_log2b.log'
  ) SIZE 100M BLOCKSIZE 512,
  GROUP 3 (
    '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/redo_log3a.log',
    '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/redo_log3b.log'
  ) SIZE 100M BLOCKSIZE 512
DATAFILE
  '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/system_01.dbf',
  '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/sysaux_01.dbf',
  '/dborafiles/rpoviw1/source_DB/oradata03/target_DB/undo_01.dbf'
  CHARACTER SET WE8MSWIN1252;
~

--- mount DB

SQL> @/dboracle/product/11.2.0.3/dbs/create_controlfile_target_DB.sql

Control file created.

SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted


SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
target_DB  MOUNTED              PRIMARY

--- try to open DB

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL> alter database open RESETLOGS;
alter database open RESETLOGS
*
ERROR at line 1:
ORA-01195: online backup of file 1 needs more recovery to be consistent
ORA-01110: data file 1:
'/dborafiles/rpoviw1/source_DB/oradata03/target_DB/system_01.dbf'

-- do some log switch at source DB and give the archivelog location of primary DB

SQL> recover database using BACKUP CONTROLFILE;
ORA-00279: change 13641415476848 generated at 04/06/2016 13:27:12 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6623_908035086.arch
ORA-00280: change 13641415476848 for thread 1 is in sequence #6623


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6623_908035086.arch
ORA-00279: change 13641416081656 generated at 04/06/2016 13:43:14 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6624_908035086.arch
ORA-00280: change 13641416081656 for thread 1 is in sequence #6624
ORA-00278: log file
'/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6623_908035086.arch' no
longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6624_908035086.arch
ORA-00279: change 13641416081660 generated at 04/06/2016 13:43:16 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6625_908035086.arch
ORA-00280: change 13641416081660 for thread 1 is in sequence #6625
ORA-00278: log file
'/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6624_908035086.arch' no
longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6625_908035086.arch
ORA-00279: change 13641416101823 generated at 04/06/2016 14:16:29 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6626_908035086.arch
ORA-00280: change 13641416101823 for thread 1 is in sequence #6626
ORA-00278: log file
'/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6625_908035086.arch' no
longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6626_908035086.arch
ORA-00279: change 13641417242879 generated at 04/06/2016 14:34:22 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6627_908035086.arch
ORA-00280: change 13641417242879 for thread 1 is in sequence #6627
ORA-00278: log file
'/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6626_908035086.arch' no
longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1:
'/dborafiles/rpoviw1/source_DB/oradata03/target_DB/system_01.dbf'

-- try some more log switch at source DB and apply in target DB

SQL> recover database using BACKUP CONTROLFILE;
ORA-00279: change 13641417242879 generated at 04/06/2016 14:34:22 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6627_908035086.arch
ORA-00280: change 13641417242879 for thread 1 is in sequence #6627


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6627_908035086.arch
ORA-00279: change 13641418097028 generated at 04/06/2016 14:54:35 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6628_908035086.arch
ORA-00280: change 13641418097028 for thread 1 is in sequence #6628
ORA-00278: log file
'/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6627_908035086.arch' no
longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1:
'/dborafiles/rpoviw1/source_DB/oradata03/target_DB/system_01.dbf'


SQL> recover database using BACKUP CONTROLFILE;
ORA-00279: change 13641418097028 generated at 04/06/2016 14:54:35 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6628_908035086.arch
ORA-00280: change 13641418097028 for thread 1 is in sequence #6628


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6628_908035086.arch
ORA-00279: change 13641418097034 generated at 04/06/2016 14:54:46 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6629_908035086.arch
ORA-00280: change 13641418097034 for thread 1 is in sequence #6629
ORA-00278: log file
'/dborafiles/rpoviw1/source_DB/oraarch01/standby_1_6628_908035086.arch' no
longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1:
'/dborafiles/rpoviw1/source_DB/oradata03/target_DB/system_01.dbf'



SQL> Recover database until cancel using backup controlfile;
ORA-00279: change 13641418097034 generated at 04/06/2016 14:54:46 needed for
thread 1
ORA-00289: suggestion :
/dborafiles/rpoviw1/source_DB/oraarch01/target_DB/target_DB_1_6629_908035086.arch
ORA-00280: change 13641418097034 for thread 1 is in sequence #6629


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;

Database altered.

SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
target_DB  READ WRITE           PRIMARY