Failed RMAN Catalog Upgrade from 11.2.0.2 to 12.1.0.2 （ ORA-02296 RMAN-06004 ）

由于后期使用12c的数据库，需要对现有catalog库升级。结果悲剧了。

RMAN> upgrade catalog;
error creating modify_bcf_pdb_key_not_null
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-06004: ORACLE error from recovery catalog database: ORA-02296: cannot enable (DM02CAT.) - null values found

这个问题是由于升级12.1.0.2 catalog 库导致的。

通过trace文件可以看到升级catalog报错的sql语句:
RMAN trace shows :

DBGSQL:        RCVCAT>   alter table bcf modify(pdb_key NOT NULL) -- 失败的SQL语句。
DBGSQL:
DBGSQL:       sqlcode = 2296
DBGSQL:        error: ORA-02296: cannot enable (&OWNER.) - null values found (krmkosqlerr)
DBGSQL:    (krmkosqlerr)
DBGSQL:       EXITED krmkosqlerr
RMAN-06444: error creating modify_bcf_pdb_key_not_null
DBGMISC:      krmksqlerror called from file krmk4.c, line 4046

通过查询catalog库用户 dm02cat发现 bcf表的PDB_KEY列存在空值，所以无法修改其为not null.

SQL> conn dm02cat/dm02cat
Connected.
SQL> select count(*) from bcf where PDB_KEY is null;

COUNT(*)
----------
7

SQL> select count(*) from bcf ;

COUNT(*)
----------
    375350


Workround:

连到catalog库的用户清除mismatched记录

delete bdf where not exists (select 1 from dbinc where dbinc.dbinc_key = bdf.dbinc_key);
delete bcf where not exists (select 1 from dbinc where dbinc.dbinc_key = bcf.dbinc_key);
commit;

潜在的风险:
There seems to be a potential inconsistency in the RMAN catalog when the PDB/CDB mechanisms get introduced.
This does not necessarily happen - but it can happen.
将来如果使用这个升级了的catalog库备份12c的 PDB/CDB，可能导致RMAN catalog数据字典数据不一致。

不过好在我们近期不太会使用PDB/CDB. 所以应用了workround解决了这个问题。

Reference:
https://blogs.oracle.com/UPGRADE/entry/rman_catalog_upgrade_fails_ora
Bug 20861957 - ORA-2296 during RMAN upgrade catalog to 12c (Doc ID 20861957.8)
Bug 19677999 : CATALOG SCHEMA UPGRADE TO 12.1.0.2 FAILED