一、概述
本文将介绍如何给oracle rac数据库打最新补丁，数据库版本为11.2.0.4两节点rac，操作系统为redhat6.5。
由于打补丁的具体细节可能随着补丁包的不同而有所差异，但是大致步骤是一样的，打补丁前最好是先仔细阅读补丁包里面的readme。
二、下载相关升级包
1. 登录MOS，查阅(ID 2118136.2)，下载最新补丁包
GI PUS (Grid Infrastructure Patch Set Upate)，集群grid的补丁，里面包含了数据库补丁，所以只下这个就行了
2. 搜索"Patch 6880880"，下载最新opatch
opatch是安装补丁的程序，数据库软件安装完成后，就自带了opatch，但是版本太旧了，所以这里下载最新的opatch
至此已下载两个文件
p6880880_112000_Linux-x86-64.zip：opatch升级包
p30070097_112040_Linux-x86-64.zip：集群grid的补丁
三、升级opatch
opatch的升级比较简单，就是个文件覆盖
opatch包分别解压到每个节点的oracle和grid下的ORACLE_HOME下，一共四份。限于篇幅，我只演示节点1的opatch升级，节点2照这个步骤同样做一遍。
以下操作均使用root账户
a. 上传p6880880_112000_Linux-x86-64.zip安装包到/opt目录下
b. 升级grid的opatch
[root@rac1 ~]# cd /u01/app/11.2.0/grid # 该目录是grid下的ORACLE_HOME
[root@rac1 grid]# mv OPatch OPatch.bak # 备份老的opatch
[root@rac1 grid]# cd /opt
[root@rac1 opt]# unzip p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.0/grid
[root@rac1 opt]# chown -R grid:oinstall /u01/app/11.2.0/grid/OPatch
[root@rac1 opt]# /u01/app/11.2.0/grid/OPatch/opatch version
OPatch Version: 11.2.0.3.21 OPatch succeeded.
c. 升级oracle的opatch
[root@rac1 opt]# cd /u01/app/oracle/product/11.2.0/db_1 # 该目录是oracle下的ORACLE_HOME
[root@rac1 db_1]# mv OPatch OPatch.bak # 备份老的opatch
[root@rac1 db_1]# cd /opt
[root@rac1 opt]# unzip p6880880_112000_Linux-x86-64.zip -d /u01/app/oracle/product/11.2.0/db_1
[root@rac1 opt]# chown -R oracle:oinstall /u01/app/oracle/product/11.2.0/db_1/OPatch
[root@rac1 opt]# /u01/app/oracle/product/11.2.0/db_1/OPatch/opatch version
OPatch Version: 11.2.0.3.21 OPatch succeeded.
四、生成ocm响应文件
在root下生成ocm(Oracle Configuration Manager)响应文件（我只演示节点1，节点2照这个步骤同样做一遍）
[root@rac1 opt]# cd /u01/app/11.2.0/grid/OPatch/ocm/bin
[root@rac1 bin]# ./emocmrsp
输入一次回车和一次y，在当前目录下面就生成了ocm.rsp
五、检查grid和oracle打补丁纪录
由于节点1和节点2是一样的，所以只在节点1上执行即可
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
[grid@rac1 ~]$ su - oracle
[oracle@rac1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
内容略，只要命令不报错就没问题
六、打补丁
我只演示节点1，节点2照这个步骤同样做一遍
打补丁前停掉两节点的监听，并杀掉所有的数据库连接，避免对补丁安装产生影响
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ lsnrctl stop
以下操作均使用root账户
a. 上传p30070097_112040_Linux-x86-64.zip安装包到/opt目录下
b. 解压p30070097_112040_Linux-x86-64.zip
[root@rac1 opt]# unzip p30070097_112040_Linux-x86-64.zip
c. 打补丁
[root@rac1 opt]# cd 30070097
[root@rac1 30070097]# /u01/app/11.2.0/grid/OPatch/opatch auto /opt/30070097 -ocmrf /u01/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp
d. 检查grid和oracle打补丁纪录
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.21 Copyright (c) 2019, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/11.2.0/grid Central Inventory : /u01/app/oraInventory from : /u01/app/11.2.0/grid/oraInst.loc OPatch version : 11.2.0.3.21 OUI version : 11.2.0.4.0 Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2019-12-26_14-50-02PM_1.log Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2019-12-26_14-50-02PM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Grid Infrastructure 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. --------------------------------------------------------------------------------
[grid@rac1 ~]$ su - oracle
[oracle@rac1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.21
Copyright (c) 2019, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version : 11.2.0.3.21
OUI version : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2019-12-26_14-49-43PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2019-12-26_14-49-43PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: rac1
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Database 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.
Interim patches (2) :
Patch 29913194 : applied on Thu Dec 26 14:43:44 CST 2019
Unique Patch ID: 23112673
Patch description: "Database Patch Set Update : 11.2.0.4.191015 (29913194)"
Created on 12 Sep 2019, 21:00:41 hrs PST8PDT
Sub-patch 29497421; "Database Patch Set Update : 11.2.0.4.190716 (29497421)"
Sub-patch 29141056; "Database Patch Set Update : 11.2.0.4.190416 (29141056)"
Sub-patch 28729262; "Database Patch Set Update : 11.2.0.4.190115 (28729262)"
Sub-patch 28204707; "Database Patch Set Update : 11.2.0.4.181016 (28204707)"
Sub-patch 27734982; "Database Patch Set Update : 11.2.0.4.180717 (27734982)"
Sub-patch 27338049; "Database Patch Set Update : 11.2.0.4.180417 (27338049)"
Sub-patch 26925576; "Database Patch Set Update : 11.2.0.4.180116 (26925576)"
Sub-patch 26392168; "Database Patch Set Update : 11.2.0.4.171017 (26392168)"
Sub-patch 26609445; "Database Patch Set Update : 11.2.0.4.170814 (26609445)"
Sub-patch 25869727; "Database Patch Set Update : 11.2.0.4.170718 (25869727)"
Sub-patch 24732075; "Database Patch Set Update : 11.2.0.4.170418 (24732075)"
Sub-patch 24006111; "Database Patch Set Update : 11.2.0.4.161018 (24006111)"
Sub-patch 23054359; "Database Patch Set Update : 11.2.0.4.160719 (23054359)"
Sub-patch 22502456; "Database Patch Set Update : 11.2.0.4.160419 (22502456)"
Sub-patch 21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
Sub-patch 21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch 20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch 20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch 19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Bugs fixed:
17184721, 21174504, 20169408, 21538558, 16091637, 18092127, 17381384
15979965, 20671094, 16731148, 16314254, 18441944, 13837378, 17835048
17291347, 23105538, 28254374, 13558557, 21656630, 21842740, 17008068
18382302, 17201159, 17853498, 25427662, 21197626, 17246576, 20717359
18356166, 18681862, 18440047, 20569094, 20031873, 16875449, 20387265
19788842, 17296856, 21330264, 14010183, 17648596, 17025461, 18886413
17551063, 24719736, 17258582, 17267114, 21063322, 22507210, 17912217
17889583, 18202441, 17040764, 17478145, 16524926, 25655390, 19358317
22730454, 22148226, 18747196, 26544823, 18641419, 17036973, 18948177
17811789, 16542886, 14285317, 18009564, 17359610, 16618694, 8322815
16832076, 18247991, 16692232, 22507234, 28022101, 17570240, 13871092
24624166, 26631046, 24348685, 19429927, 17848897, 17441661, 14034426
17465741, 20273319, 19207156, 16596890, 17437634, 20506706, 18510194
21343897, 28849751, 18339044, 21453153, 17951233, 21795111, 22321741
18430495, 21787056, 22380919, 20506715, 19692824, 19469538, 17811429
17903598, 19721304, 11786053, 29511611, 18230522, 19554106, 19458377
21281607, 17612828, 6599380, 18029658, 22092979, 19516448, 17040527
22321756, 17811438, 18641461, 18682983, 14657740, 25635149, 21502702
13364795, 19490948, 21387964, 17346671, 17588480, 22351572, 18235390
26474853, 18849970, 17982832, 17889549, 19309466, 16472716, 23008056
20627866, 24908321, 20134113, 25775213, 20596234, 18331850, 18641451
17019356, 20882568, 17344412, 19461270, 21179898, 17546761, 24842886
17231779, 14521849, 18203835, 18203838, 18964939, 18203837, 17313525
22195457, 18139690, 16837842, 14106803, 22296366, 17842825, 22657942
21352646, 16360112, 22594718, 20657441, 22195441, 17389192, 26198926
14565184, 19781326, 17019345, 17205719, 18740837, 18440095, 14764829
14354737, 22195448, 17019086, 13944971, 16571443, 21868720, 17186905
17080436, 18673342, 28501075, 22905130, 17027426, 19972569, 27374796
19972568, 16833845, 19972566, 20144308, 17282229, 19972564, 16870214
16410570, 21629064, 19615136, 21354456, 26039623, 19871910, 17390431
18762750, 23007241, 25248384, 16613964, 18098207, 17957017, 17484762
18471685, 19730508, 18264060, 21538485, 17323222, 17754782, 17600719
18317531, 17852463, 17596908, 17655634, 18166013, 16228604, 20074391
27053456, 24790914, 19972570, 26482376, 20856766, 18090142, 19891090
18996843, 16042673, 19854503, 17835627, 22901797, 20334344, 17393683
20861693, 18000422, 17551709, 26575788, 23315889, 20506699, 19006849
18277454, 18456514, 17258090, 19174430, 20657411, 17174582, 25654936
17242746, 27097854, 16399083, 17824637, 21132297, 17762296, 22465352
22168163, 28612674, 18604692, 17397545, 16450169, 12364061, 20067212
19373893, 18856999, 19211724, 19463893, 19463897, 27351628, 21343775
17853456, 18373438, 18673304, 20004021, 28000269, 26030218, 21668627
16194160, 17477958, 23140259, 16538760, 12982566, 24570598, 20828947
27255377, 18259031, 20296213, 21425496, 28855981, 18293054, 17610798
19699191, 23065323, 17311728, 18135678, 18774543, 23294548, 16785708
10136473, 22551446, 19777862, 24560906, 17786518, 18315328, 25879984
18334586, 12747740, 22250006, 18096714, 19032867, 21641760, 17390160
18899974, 17232014, 20598042, 16354467, 26245237, 26679352, 17484731
18673325, 16422541, 18155762, 19827973, 14015842, 22683225, 17726838
18554871, 23177648, 18051556, 20803583, 18282562, 17922254, 15990359
21972320, 16855292, 16668584, 21343838, 20299015, 29483672, 17446237
18043064, 18093615, 17694209, 23713236, 17288409, 18308268, 20475845
17274537, 13955826, 16934803, 18841764, 17634921, 17501491, 16315398
23725036, 22683212, 17006183, 13829543, 18191164, 26746894, 22809871
17655240, 28819280, 19393542, 18384391, 29633753, 21538567, 17695685
16198143, 21847223, 28199085, 25823754, 17892268, 20142975, 19584068
17165204, 25165496, 27072923, 18604493, 18508861, 21756699, 18554763
16901385, 21532755, 18189036, 17443671, 17385178, 14829250, 17936109
20476175, 20925795, 20509482, 17478514, 27441326, 16850630, 13951456
16595641, 14054676, 15861775, 21142837, 16912439, 17299889, 17297939
23003979, 16833527, 18619917, 17798953, 17630484, 19697993, 17816865
25914276, 18607546, 17571306, 21286665, 16898135, 17341326, 16819962
26910644, 17851160, 17586955, 20558005, 19049453, 21051840, 17587063
16956380, 18328509, 25042823, 14735792, 25423453, 14133975, 29033139
19718981, 18061914, 18522509, 16233738, 17518652, 21051833, 18765602
20294666, 23194294, 20860659, 18272672, 20324049, 18199537, 17332800
13609098, 22502493, 18384537, 14338435, 17945983, 27710072, 16392068
21067387, 17752995, 21097043, 21051862, 16863422, 17237521, 25505382
29483723, 18244962, 19544839, 28357401, 19433930, 24433711, 24717859
17156148, 18973907, 23026585, 17449815, 17877323, 18180390, 17088068
17037130, 20004087, 21422580, 19466309, 11733603, 25505371, 18610915
21051858, 18084625, 29027694, 18674024, 26243698, 21051852, 18091059
18306996, 16306373, 25369547, 19930276, 17787259, 19915271, 18193833
20631274, 20513399, 16344544, 26439748, 25879656, 14692762, 18614015
22782647, 17346091, 18413820, 19297917, 18228645, 17721717, 13960236
18685892, 18436307, 11883252, 19888853, 21756677, 17891943, 19475971
22353199, 16384983, 19121551, 27825893, 25634317, 12816846, 17982555
17761775, 17227073, 13936038, 22243719, 17265217, 25505394, 17071721
16721594, 18262334, 21756661, 17891946, 15913355, 17672719, 17602269
17239687, 17042658, 25555252, 17238511, 21059919, 17811456, 17284817
17752121, 20879889, 28806384, 21380789, 19601762, 17394950, 17011832
28305362, 16579084, 22195465, 16875230, 14602788, 28790634, 18325460
27567477, 30275351, 26569225, 24476265, 24476274, 12611721, 18674465
16903536, 17006570, 19689979, 28076295, 16043574, 18783224, 22836801
14705949, 24662775, 16494615, 21526048, 17392698, 19197175, 16069901
17811447, 29200700, 27870645, 28876684, 17308789, 24835538, 22195477
17865671, 17343514, 19013183, 17325413, 18316692, 16180763, 30275359
17348614, 14368995, 21983325, 17393915, 16285691, 19788303, 19211433
20331945, 17883081, 17705023, 24316947, 17614227, 23571055, 19578350
22195485, 14084247, 24975421, 26078387, 23115139, 13645875, 23328639
16777840, 19727057, 21698350, 14852021, 18744139, 18674047, 17716305
19285025, 18482502, 17622427, 19289642, 27534509, 25947799, 22195492
14458214, 20869721, 21172913, 17767676, 18723434, 25505407, 17786278
19258504, 17082983, 17365043, 21351877, 13498382, 18331812, 16065166
25489607, 16685417, 21566639, 18031668, 22893153, 17551674, 16943711
19272701, 21517440, 25897615, 17649265, 13866822, 18094246, 24528741
17783588, 14245531, 17082359, 18280813, 26007010, 20448824, 23330119
16268425, 19487147, 25600421, 18018515, 17302277, 17215560, 24411921
19271443, 25764020, 14176370, 17016369, 20777150, 16756406, 23330124
22977256, 20441797, 19769489, 28100487, 17545847, 25093656, 18260550
13853126, 17227277, 17551699, 23536835, 30237239, 25957038, 24652769
20725343, 19207117, 9756271, 17495022, 18868646, 17614134, 26667023
17546973, 19680952, 18704244, 26667015, 17050888, 18273830, 18828868
17360606, 16992075, 24563422, 17375354, 12905058, 18362222, 21429602
17254374, 27086138, 28364007, 26667032, 17571039, 17468141, 18436647
17570606, 17235750, 21168487, 17279227, 16220077, 16929165
Patch 29938455 : applied on Thu Dec 26 14:37:54 CST 2019
Unique Patch ID: 23033728
Patch description: "OCW Patch Set Update : 11.2.0.4.191015 (30070097)"
Created on 23 Aug 2019, 01:51:24 hrs PST8PDT
Bugs fixed:
18328800, 19270660, 18691572, 18508710, 20038451, 21251192, 22162062
20365005, 21232394, 17336871, 17387214, 17750548, 17617807, 14497275
20219458, 17733927, 18180541, 23757020, 18962892, 17292250, 17378618
16759171, 20110156, 17843489, 17065496, 13991403, 21694632, 18419139
17273020, 22762046, 17336884, 17155238, 17336889, 18261183, 18053580
20012766, 21245437, 20218012, 17013634, 17886392, 20995001, 24422155
17039197, 16317771, 17947785, 10052729, 20340620, 22353346, 16237657
20317221, 18199185, 15917869, 18399991, 20186278, 18024089, 17374271
16849642, 20246071, 20746251, 14270845, 23621494, 20552947, 18414137
18882642, 17001914, 17927970, 29323944, 14378120, 16346413, 15986647
17336898, 18068871, 21222147, 16206997, 18143836, 21982225, 25490238
19168690, 18343490, 20235511, 21875360, 16613232, 19276791, 17722664
20440643, 12928658, 18226143, 18520351, 16249829, 18952577, 17172091
18229842, 16076412, 18265482, 20676340, 17818075, 20091753, 18231837
14373486, 17483479, 20136892, 20551654, 18120545, 18729166, 13843841
17405302, 21225209, 18709496, 18330979, 18744838, 14525998, 18187697
17087371, 20531190, 14385860, 20598625, 18348155, 19479503, 12928592
17516024, 18370031, 17764053, 19272663, 17551223, 14671408, 18272135
14207615, 24692493, 21255373, 17500165, 18875012, 14769643, 18464784
25656952, 19558324, 18848125, 19241857, 30158572, 17955615, 14851828
14693336, 20315294, 16284825, 17352230, 20014326, 17238586, 17089344
17405605, 17531342, 21327402, 26546632, 19398098, 17159489, 17640316
13823394, 16543190, 22024217, 17983675, 20795241, 28973538, 17481314
18346135, 28553832, 17598201, 16281493, 15986311, 17208793, 19601468
18700935, 18999857, 14076173, 18428146, 16709532, 17435488, 18352845
18352846, 20408163, 17592037, 19616601, 17387779, 17391726, 14777968
15851860, 16206882, 20141091, 15832129, 17305100, 21113068, 20175174
19885321, 16901346, 17985714, 25591658, 18536826, 17780903, 18752378
18946768, 16876500, 16875342, 17769597, 19955755, 16429265, 18336452
23186035, 17273003, 17209968, 25484507, 16988311, 19319357, 20094984
17046460, 17059927, 18053631, 16867761, 18774591, 21442094, 20235486
19359787, 15869775, 17447588, 19642566, 21152052, 16798862, 15920201
--------------------------------------------------------------------------------
OPatch succeeded.七、执行catbundle.sql
其实大部分情况是不需要执行这一步的，具体可以参考本人写的另外一篇博客https://www.cnblogs.com/ddzj01/p/12100935.html
在任意一个节点执行
[oracle@rac1 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@rac1 admin]$ sqlplus / as sysdba
SQL> @catbundle.sql psu apply
SQL> quit
至此rac补丁就已经安装完毕了！使用grid账号执行lsnrctl status可以看到监听被自动启动了，通过观察打补丁的日志，可以看到在打补丁的过程中会自动检测冲突、重启实例等。
八、回退数据库补丁
数据库在做变更时，当然要考虑回退方案了，接下来介绍如何回退数据库补丁
在每个节点用root依次执行
[root@rac1 ~]# cd /opt/30070097
[root@rac1 30070097]# /u01/app/11.2.0/grid/OPatch/opatch auto /opt/30070097 -rollback -ocmrf /u01/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp
如果前面有执行catbundle.sql，那么回退补丁的时候，在任意一个节点执行sql文件
[oracle@rac1 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@rac1 admin]$ sqlplus / as sysdba
SQL> @catbundle_PSU_ORCL_ROLLBACK.sql # 注意这里可能每个人都不一样，orcl是我的实例名的前缀
SQL> quit
九、总结
本文给大家介绍了rac打补丁步骤，仅作参考，实际应以补丁包中的readme为准。我在写这篇文章的时候，找了三个朋友要集群的打补丁文档，发觉每一个人写的都不一样，而且都跟补丁文档中不一致。这让人头大，后来索性自己照着补丁文档来做，发觉其实打补丁也比较简单，oracle在简化步骤上面下了很大的功夫，根本用不到检测冲突，关闭实例什么的，这些都在打补丁过程中自动完成了。
我在第一次做这个实验的时候，打补丁就失败了，失败的原因也很弱智，是因为实验机器的空间满了，清了垃圾数据，一顿操作，搞到集群都起不来。所以打补丁还是有一定的风险，在打补丁前做好备份，在万不得已的情况下还能通过备份去恢复数据库。
总而言之，多做实验，多总结文档，网上的和别人给的都不一定对，自己总结的才是最好的！