【RAC】rac环境下的数据库备份与还原

一.1  BLOG文档结构图

【RAC】rac环境下的数据库备份与还原-LMLPHP

一.2  前言部分

一.2.1  导读

各位技术爱好者,看完本文后,你可以掌握如下的技能,也可以学到一些其它你所不知道的知识,~O(∩_∩)O~:

① rac环境下的数据库备份与还原

② rman恢复数据库的一般步骤

③ rac环境的简单操作

注意:本篇BLOG中代码部分需要特别关注的地方我都用黄色背景和红色字体来表示,比如下边的例子中,thread 1的最大归档日志号为33,thread 2的最大归档日志号为43是需要特别关注的地方。

List of Archived Logs in backup set 11

Thrd Seq     Low SCN    Low Time            Next SCN   Next Time

---- ------- ---------- ------------------- ---------- ---------

1    32      1621589    2015-05-29 11:09:52 1625242    2015-05-29 11:15:48

1    33      1625242    2015-05-29 11:15:48 1625293    2015-05-29 11:15:58

2    42      1613951    2015-05-29 10:41:18 1625245    2015-05-29 11:15:49

2    43      1625245    2015-05-29 11:15:49 1625253    2015-05-29 11:15:53

本文如有错误或不完善的地方请大家多多指正,ITPUB留言或QQ皆可,您的批评指正是我写作的最大动力。

 

一.2.2  实验环境介绍

目标库:11.2.0.1  RHEL6.5  rac环境

一.2.3  相关参考文章链接

rac安装系列:

【推荐】 一步一步搭建11gR2 rac+dg之结尾篇(十)

http://blog.itpub.net/26736162/viewspace-1328156/

【推荐】 一步一步搭建11gR2 rac+dg之DG SWITCHOVER功能(九)

http://blog.itpub.net/26736162/viewspace-1328050/

一步一步搭建11gR2 rac+dg之配置单实例的DG(八)

http://blog.itpub.net/26736162/viewspace-1298735/

一步一步搭建11gR2 rac+dg之DG 机器配置(七)

http://blog.itpub.net/26736162/viewspace-1298733/

一步一步搭建11gR2 rac+dg之安装rac出现问题解决(六)

http://blog.itpub.net/26736162/viewspace-1297128/

一步一步搭建oracle 11gR2 rac+dg之database安装(五)

http://blog.itpub.net/26736162/viewspace-1297113/

一步一步搭建 oracle 11gR2 rac+dg之grid安装(四)

http://blog.itpub.net/26736162/viewspace-1297101/

【推荐】 一步一步搭建oracle 11gR2 rac+dg之共享磁盘设置(三)

http://blog.itpub.net/26736162/viewspace-1291144/

【推荐】 一步一步搭建oracle 11gR2 rac+dg之环境准备(二)

http://blog.itpub.net/26736162/viewspace-1290416/

【推荐】 一步一步搭建 oracle 11gR2 rac + dg 之前传 (一)

http://blog.itpub.net/26736162/viewspace-1290405/

静默安装系列

【推荐】 【DBCA -SILENT】静默方式安装11gR2 oracle数据库软件

http://blog.itpub.net/26736162/viewspace-1589769/

【推荐】 【DBCA -SILENT】静默安装之rac数据库安装

http://blog.itpub.net/26736162/viewspace-1586352/

【推荐】 【DBCA -SILENT】静默安装如何启用归档模式

http://blog.itpub.net/26736162/viewspace-1585925/

【推荐】  DBCA静默方式建库

http://blog.itpub.net/26736162/viewspace-1448220/

一.2.4  本文简介

本文基于如何将单实例的数据库备份恢复到rac环境下,至于rac环境的备份集如何恢复到单实例及rac环境的备份集恢复到rac环境的实验请参考相关文章链接部分。

实验的一些数据库环境参考如下表格:

项目

source db

target  db

db 类型

rac环境

rac环境

db version

11.2.0.1

11.2.0.1

ORACLE_SID

orastrac1 和 orastrac2

orastrac1 和 orastrac2

db_name

orastrac

orastrac

主机IP地址:

192.168.1.31

192.168.1.32

192.168.1.31

192.168.1.32

一.3  实验部分

一.3.1  实验目标

验证rac数据库环境下的备份和恢复操作。

一.3.2  查看rac环境

target库已经有4个库,3个rac库处于close状态,orastrac处于online状态。

[root@node2 ~]# crsstat

Name                           Type                       Target     State      Host

------------------------------ -------------------------- ---------- ---------  -------

ora.ARCH.dg                    ora.diskgroup.type         ONLINE     ONLINE     node1

ora.DATA.dg                    ora.diskgroup.type         ONLINE     ONLINE     node1

ora.LISTENER.lsnr              ora.listener.type          ONLINE     ONLINE     node1

ora.LISTENER_SCAN1.lsnr        ora.scan_listener.type     ONLINE     ONLINE     node1

ora.OVDISK.dg                  ora.diskgroup.type         ONLINE     ONLINE     node1

ora.TEST.dg                    ora.diskgroup.type         ONLINE     ONLINE     node1

ora.asm                        ora.asm.type               ONLINE     ONLINE     node1

ora.db.db                      ora.database.type          OFFLINE    OFFLINE

ora.eons                       ora.eons.type              ONLINE     ONLINE     node1

ora.gsd                        ora.gsd.type               OFFLINE    OFFLINE

ora.jmrac.db                   ora.database.type          ONLINE     ONLINE     node1

ora.jmrac.haha.svc             ora.service.type           ONLINE     ONLINE     node1

ora.net1.network               ora.network.type           ONLINE     ONLINE     node1

ora.node1.ASM1.asm             application                ONLINE     ONLINE     node1

ora.node1.LISTENER_NODE1.lsnr  application                ONLINE     ONLINE     node1

ora.node1.gsd                  application                OFFLINE    OFFLINE

ora.node1.ons                  application                ONLINE     ONLINE     node1

ora.node1.vip                  ora.cluster_vip_net1.type  ONLINE     ONLINE     node1

ora.node2.ASM2.asm             application                ONLINE     ONLINE     node2

ora.node2.LISTENER_NODE2.lsnr  application                ONLINE     ONLINE     node2

ora.node2.gsd                  application                OFFLINE    OFFLINE

ora.node2.ons                  application                ONLINE     ONLINE     node2

ora.node2.vip                  ora.cluster_vip_net1.type  ONLINE     ONLINE     node2

ora.oc4j                       ora.oc4j.type              OFFLINE    OFFLINE

ora.ons                        ora.ons.type               ONLINE     ONLINE     node1

ora.ora11g.db                  ora.database.type          OFFLINE    OFFLINE

ora.registry.acfs              ora.registry.acfs.type     ONLINE     ONLINE     node1

ora.scan1.vip                  ora.scan_vip.type          ONLINE     ONLINE     node1

[root@node2 ~]# crsstat | grep ora.database.type

ora.db.db                      ora.database.type          OFFLINE    OFFLINE

ora.jmrac.db                   ora.database.type          OFFLINE    OFFLINE

ora.ora11g.db                  ora.database.type          OFFLINE    OFFLINE

[root@node2 ~]#

[root@node2 ~]#  crsstat | grep ora.database.type

ora.db.db                      ora.database.type          ONLINE     OFFLINE

ora.jmrac.db                   ora.database.type          OFFLINE    OFFLINE

ora.ora11g.db                  ora.database.type          OFFLINE    OFFLINE

ora.orastrac.db                ora.database.type          ONLINE     ONLINE     node1

[root@node2 ~]#

一.3.3  创建测试数据

[oracle@node2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jun 2 13:32:34 2015

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> show parameter name

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_file_name_convert                 string

db_name                              string      orastrac

db_unique_name                       string      orastrac

global_names                         boolean     FALSE

instance_name                        string      orastrac2

lock_name_space                      string

log_file_name_convert                string

service_names                        string      orastrac

SQL> create table lhr.rac_test as select * from dba_objects;

Table created.

SQL> select count(1) from lhr.rac_test;

COUNT(1)

----------

72469

SQL>

一.3.4  source库执行备份操作

备份脚本如下,注意控制文件需要最后备份,在节点2上执行备份:

run

{

allocate channel c1 type disk;

allocate channel c2 type disk;

backup database format  '/home/oracle/rman_back/full_%n_%T_%t_%s_%p.bak';

sql 'alter system archive log current';

backup archivelog all format '/home/oracle/rman_back/arch_%d_%T_%s_%p.bak' delete input;

backup current controlfile format '/home/oracle/rman_back/ctl_%d_%T_%s_%p.bak';

release channel c1;

release channel c2;

}

执行过程如下:

[oracle@node2 ~]$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Tue Jun 2 13:36:21 2015

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORASTRAC (DBID=1317814272)

RMAN> list backupset;

using target database control file instead of recovery catalog

specification does not match any backup in the repository

RMAN> run

2> {

3>  allocate channel c1 type disk;

4>  allocate channel c2 type disk;

5>  backup database format  '/home/oracle/rman_back/full_%n_%T_%t_%s_%p.bak';

6>  sql 'alter system archive log current';

7>  backup archivelog all format '/home/oracle/rman_back/arch_%d_%T_%s_%p.bak' delete input;

8>  backup current controlfile format '/home/oracle/rman_back/ctl_%d_%T_%s_%p.bak';

9>  release channel c1;

10>  release channel c2;

11> }

allocated channel: c1

channel c1: SID=33 instance=orastrac2 device type=DISK

allocated channel: c2

channel c2: SID=51 instance=orastrac2 device type=DISK

Starting backup at 02-JUN-2015 13:36:32

channel c1: starting full datafile backup set

channel c1: specifying datafile(s) in backup set

input datafile file number=00001 name=+DATA/orastrac/datafile/system.330.881251509

input datafile file number=00004 name=+DATA/orastrac/datafile/users.327.881251511

input datafile file number=00006 name=+DATA/orastrac/datafile/undotbs2.351.881252419

channel c1: starting piece 1 at 02-JUN-2015 13:36:33

channel c2: starting full datafile backup set

channel c2: specifying datafile(s) in backup set

input datafile file number=00002 name=+DATA/orastrac/datafile/sysaux.331.881251461

input datafile file number=00005 name=+DATA/orastrac/datafile/example.332.881251463

input datafile file number=00003 name=+DATA/orastrac/datafile/undotbs1.329.881251463

channel c2: starting piece 1 at 02-JUN-2015 13:36:33

channel c1: finished piece 1 at 02-JUN-2015 13:38:38

piece handle=/home/oracle/rman_back/full_ORASTRAC_20150602_881328992_9_1.bak tag=TAG20150602T133632 comment=NONE

channel c1: backup set complete, elapsed time: 00:02:05

channel c1: starting full datafile backup set

channel c1: specifying datafile(s) in backup set

channel c2: finished piece 1 at 02-JUN-2015 13:38:38

piece handle=/home/oracle/rman_back/full_ORASTRAC_20150602_881328993_10_1.bak tag=TAG20150602T133632 comment=NONE

channel c2: backup set complete, elapsed time: 00:02:05

channel c2: starting full datafile backup set

channel c2: specifying datafile(s) in backup set

including current SPFILE in backup set

channel c2: starting piece 1 at 02-JUN-2015 13:38:39

including current control file in backup set

channel c1: starting piece 1 at 02-JUN-2015 13:38:42

channel c2: finished piece 1 at 02-JUN-2015 13:38:42

piece handle=/home/oracle/rman_back/full_ORASTRAC_20150602_881329118_12_1.bak tag=TAG20150602T133632 comment=NONE

channel c2: backup set complete, elapsed time: 00:00:03

channel c1: finished piece 1 at 02-JUN-2015 13:38:43

piece handle=/home/oracle/rman_back/full_ORASTRAC_20150602_881329118_11_1.bak tag=TAG20150602T133632 comment=NONE

channel c1: backup set complete, elapsed time: 00:00:01

Finished backup at 02-JUN-2015 13:38:43

sql statement: alter system archive log current

Starting backup at 02-JUN-2015 13:38:56

current log archived

channel c1: starting archived log backup set

channel c1: specifying archived log(s) in backup set

input archived log thread=1 sequence=1 RECID=5 STAMP=881255375

input archived log thread=2 sequence=1 RECID=4 STAMP=881252641

input archived log thread=1 sequence=2 RECID=7 STAMP=881256606

input archived log thread=3 sequence=1 RECID=6 STAMP=881255612

input archived log thread=2 sequence=2 RECID=8 STAMP=881257349

input archived log thread=1 sequence=3 RECID=9 STAMP=881257704

input archived log thread=2 sequence=3 RECID=12 STAMP=881257709

channel c1: starting piece 1 at 02-JUN-2015 13:39:00

channel c2: starting archived log backup set

channel c2: specifying archived log(s) in backup set

input archived log thread=1 sequence=4 RECID=10 STAMP=881257707

input archived log thread=1 sequence=5 RECID=11 STAMP=881257708

input archived log thread=2 sequence=4 RECID=13 STAMP=881329134

channel c2: starting piece 1 at 02-JUN-2015 13:39:00

channel c2: finished piece 1 at 02-JUN-2015 13:39:01

piece handle=/home/oracle/rman_back/arch_ORASTRAC_20150602_14_1.bak tag=TAG20150602T133859 comment=NONE

channel c2: backup set complete, elapsed time: 00:00:01

channel c2: deleting archived log(s)

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_1_seq_4.366.881257705 RECID=10 STAMP=881257707

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_1_seq_5.368.881257709 RECID=11 STAMP=881257708

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_2_seq_4.360.881329131 RECID=13 STAMP=881329134

channel c2: starting archived log backup set

channel c2: specifying archived log(s) in backup set

input archived log thread=1 sequence=6 RECID=14 STAMP=881329136

input archived log thread=2 sequence=5 RECID=16 STAMP=881329139

input archived log thread=1 sequence=7 RECID=15 STAMP=881329138

channel c2: starting piece 1 at 02-JUN-2015 13:39:03

channel c1: finished piece 1 at 02-JUN-2015 13:39:03

piece handle=/home/oracle/rman_back/arch_ORASTRAC_20150602_13_1.bak tag=TAG20150602T133859 comment=NONE

channel c1: backup set complete, elapsed time: 00:00:03

channel c1: deleting archived log(s)

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_1_seq_1.357.881255367 RECID=5 STAMP=881255375

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_2_seq_1.355.881252641 RECID=4 STAMP=881252641

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_1_seq_2.362.881256605 RECID=7 STAMP=881256606

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_3_seq_1.361.881255613 RECID=6 STAMP=881255612

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_2_seq_2.363.881257349 RECID=8 STAMP=881257349

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_1_seq_3.365.881257705 RECID=9 STAMP=881257704

archived log file name=+ARCH/orastrac/archivelog/2015_06_01/thread_2_seq_3.367.881257709 RECID=12 STAMP=881257709

channel c2: finished piece 1 at 02-JUN-2015 13:39:03

piece handle=/home/oracle/rman_back/arch_ORASTRAC_20150602_15_1.bak tag=TAG20150602T133859 comment=NONE

channel c2: backup set complete, elapsed time: 00:00:00

channel c2: deleting archived log(s)

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_1_seq_6.359.881329131 RECID=14 STAMP=881329136

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_2_seq_5.370.881329139 RECID=16 STAMP=881329139

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_1_seq_7.358.881329137 RECID=15 STAMP=881329138

Finished backup at 02-JUN-2015 13:39:03

Starting backup at 02-JUN-2015 13:39:05

channel c1: starting full datafile backup set

channel c1: specifying datafile(s) in backup set

including current control file in backup set

channel c1: starting piece 1 at 02-JUN-2015 13:39:06

channel c1: finished piece 1 at 02-JUN-2015 13:39:07

piece handle=/home/oracle/rman_back/ctl_ORASTRAC_20150602_16_1.bak tag=TAG20150602T133905 comment=NONE

channel c1: backup set complete, elapsed time: 00:00:01

Finished backup at 02-JUN-2015 13:39:07

released channel: c1

released channel: c2

RMAN>

RMAN> list backupset summary;

List of Backups

===============

Key     TY LV S Device Type Completion Time      #Pieces #Copies Compressed Tag

------- -- -- - ----------- -------------------- ------- ------- ---------- ---

7       B  F  A DISK        02-JUN-2015 13:38:30 1       1       NO         TAG20150602T133632

8       B  F  A DISK        02-JUN-2015 13:38:35 1       1       NO         TAG20150602T133632

9       B  F  A DISK        02-JUN-2015 13:38:40 1       1       NO         TAG20150602T133632

10      B  F  A DISK        02-JUN-2015 13:38:42 1       1       NO         TAG20150602T133632

11      B  A  A DISK        02-JUN-2015 13:39:01 1       1       NO         TAG20150602T133859

12      B  A  A DISK        02-JUN-2015 13:39:01 1       1       NO         TAG20150602T133859

13      B  A  A DISK        02-JUN-2015 13:39:03 1       1       NO         TAG20150602T133859

14      B  F  A DISK        02-JUN-2015 13:39:06 1       1       NO         TAG20150602T133905

RMAN>

一.3.5  删除spfile、控制文件和数据文件

删除asm文件,毁掉整个数据库:

[root@node2 rman_back]# srvctl stop database -d ORASTRAC

[root@node2 rman_back]# srvctl status database -d ORASTRAC

Instance orastrac1 is not running on node node1

Instance orastrac2 is not running on node node2

[root@node2 rman_back]# su - grid

[grid@node2 ~]$ asmcmd

ASMCMD> cd data

ASMCMD> ls

DB/

DB_UNKNOWN/

JMRAC/

MYRAC/

ORA11G/

ORASTRAC/

ASMCMD> rm -rf ORASTRAC/

ASMCMD> ls

DB/

DB_UNKNOWN/

JMRAC/

MYRAC/

ORA11G/

ASMCMD>

[oracle@node2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jun 2 13:55:46 2015

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATA/orastrac/parameterfile/spfile.335.881250575'

ORA-17503: ksfdopn:2 Failed to open file +DATA/orastrac/parameterfile/spfile.335.881250575

ORA-15056: additional error message

ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATA/orastrac/parameterfile/spfile.335.881250575

ORA-17503: ksfdopn:2 Failed to open file +DATA/orastrac/parameterfile/spfile.335.881250575

ORA-15173: entry 'orastrac' does not exist in directory '/'

ORA-06512: at line 4

SQL>

一.3.6  恢复数据库过程

一.3.6.1  恢复spfile

首先利用rman恢复spfile:

[oracle@node2 ~]$ ORACLE_SID=orastrac2

[oracle@node2 dbs]$  rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Tue Jun 2 14:15:52 2015

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> set dbid 1317814272

executing command: SET DBID

RMAN> startup nomount;

startup failed: ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATA/orastrac/parameterfile/spfile.335.881250575'

ORA-17503: ksfdopn:2 Failed to open file +DATA/orastrac/parameterfile/spfile.335.881250575

ORA-15056: additional error message

ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATA/orastrac/parameterfile/spfile.335.881250575

ORA-17503: ksfdopn:2 Failed to open file +DATA/orastrac/parameterfile/spfile.335.881250575

ORA-15173: entry 'orastrac' does not exist in directory '/'

ORA-06512: at line 4

starting Oracle instance without parameter file for retrieval of spfile

Oracle instance started

Total System Global Area     158662656 bytes

Fixed Size                     2211448 bytes

Variable Size                 96469384 bytes

Database Buffers              54525952 bytes

Redo Buffers                   5455872 bytes

RMAN> restore spfile to '+DATA/orastrac/parameterfile/spfileorastrac.ora' from  '/home/oracle/rman_back/full_ORASTRAC_20150602_881329118_12_1.bak';

Starting restore at 02-JUN-2015 14:16:47

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=24 device type=DISK

channel ORA_DISK_1: restoring spfile from AUTOBACKUP /home/oracle/rman_back/full_ORASTRAC_20150602_881329118_12_1.bak

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of restore command at 06/02/2015 14:16:49

ORA-19870: error while restoring backup piece /home/oracle/rman_back/full_ORASTRAC_20150602_881329118_12_1.bak

ORA-32015: unable to restore SPFILE

ORA-17502: ksfdcre:4 Failed to create file +DATA/orastrac/parameterfile/spfileorastrac.ora

ORA-15056: additional error message

ORA-17502: ksfdcre:4 Failed to create file +DATA/orastrac/parameterfile/spfileorastrac.ora

ORA-15173: entry 'orastrac' does not exist in directory '/'

创建相关路径后继续执行:

ASMCMD> mkdir orastrac

ASMCMD> cd orastrac

ASMCMD> mkdir parameterfile

ASMCMD> pwd

+data/orastrac

ASMCMD> cd parameterfile

RMAN> restore spfile to '+DATA/orastrac/parameterfile/spfileorastrac.ora' from  '/home/oracle/rman_back/full_ORASTRAC_20150602_881329118_12_1.bak';

Starting restore at 02-JUN-2015 14:18:07

using channel ORA_DISK_1

channel ORA_DISK_1: restoring spfile from AUTOBACKUP /home/oracle/rman_back/full_ORASTRAC_20150602_881329118_12_1.bak

channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete

Finished restore at 02-JUN-2015 14:18:08

RMAN>

查看spfile:

ASMCMD> ls

spfileorastrac.ora

ASMCMD>

修改pfile文件后重启数据库:

[oracle@node2 dbs]$ more initorastrac2.ora

SPFILE='+DATA/orastrac/parameterfile/spfileorastrac.ora'

[oracle@node2 dbs]$

[oracle@node2 dbs]$  rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Tue Jun 2 14:21:18 2015

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DUMMY (not mounted)

RMAN> shutdown abort;

using target database control file instead of recovery catalog

Oracle instance shut down

RMAN> startup nomount;

connected to target database (not started)

Oracle instance started

Total System Global Area     409194496 bytes

Fixed Size                     2213856 bytes

Variable Size                360712224 bytes

Database Buffers              41943040 bytes

Redo Buffers                   4325376 bytes

RMAN>

一.3.6.2  恢复控制文件

[oracle@node2 dbs]$  rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Tue Jun 2 14:22:52 2015

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORASTRAC (not mounted)

RMAN>  startup nomount force;

Oracle instance started

Total System Global Area     409194496 bytes

Fixed Size                     2213856 bytes

Variable Size                360712224 bytes

Database Buffers              41943040 bytes

Redo Buffers                   4325376 bytes

RMAN> restore controlfile   from '/home/oracle/rman_back/ctl_ORASTRAC_20150602_16_1.bak';

Starting restore at 02-JUN-2015 14:23:30

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=33 instance=orastrac2 device type=DISK

channel ORA_DISK_1: restoring control file

channel ORA_DISK_1: restore complete, elapsed time: 00:00:01

output file name=+DATA/orastrac/controlfile/control01.ctl

output file name=+DATA/orastrac/controlfile/current.329.881331811

output file name=+ARCH/orastrac/controlfile/current.364.881257391

Finished restore at 02-JUN-2015 14:23:32

RMAN>

RMAN> alter database mount;

database mounted

released channel: ORA_DISK_1

RMAN>

一.3.6.3  还原及恢复数据文件

查看备份集及还原数据库:

RMAN>  list backupset;

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ --------------------

7       Full    440.72M    DISK        00:01:57     02-JUN-2015 13:38:30

BP Key: 7   Status: AVAILABLE  Compressed: NO  Tag: TAG20150602T133632

Piece Name: /home/oracle/rman_back/full_ORASTRAC_20150602_881328993_10_1.bak

List of Datafiles in backup set 7

File LV Type Ckp SCN    Ckp Time             Name

---- -- ---- ---------- -------------------- ----

2       Full 1079018    02-JUN-2015 13:36:33 +DATA/orastrac/datafile/sysaux.331.881251461

3       Full 1079018    02-JUN-2015 13:36:33 +DATA/orastrac/datafile/undotbs1.329.881251463

5       Full 1079018    02-JUN-2015 13:36:33 +DATA/orastrac/datafile/example.332.881251463

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ --------------------

8       Full    610.17M    DISK        00:02:03     02-JUN-2015 13:38:35

BP Key: 8   Status: AVAILABLE  Compressed: NO  Tag: TAG20150602T133632

Piece Name: /home/oracle/rman_back/full_ORASTRAC_20150602_881328992_9_1.bak

List of Datafiles in backup set 8

File LV Type Ckp SCN    Ckp Time             Name

---- -- ---- ---------- -------------------- ----

1       Full 1078985    02-JUN-2015 13:36:33 +DATA/orastrac/datafile/system.330.881251509

4       Full 1078985    02-JUN-2015 13:36:33 +DATA/orastrac/datafile/users.327.881251511

6       Full 1078985    02-JUN-2015 13:36:33 +DATA/orastrac/datafile/undotbs2.351.881252419

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ --------------------

9       Full    80.00K     DISK        00:00:02     02-JUN-2015 13:38:40

BP Key: 9   Status: AVAILABLE  Compressed: NO  Tag: TAG20150602T133632

Piece Name: /home/oracle/rman_back/full_ORASTRAC_20150602_881329118_12_1.bak

SPFILE Included: Modification time: 02-JUN-2015 11:16:55

SPFILE db_unique_name: ORASTRAC

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ --------------------

10      Full    9.33M      DISK        00:00:04     02-JUN-2015 13:38:42

BP Key: 10   Status: AVAILABLE  Compressed: NO  Tag: TAG20150602T133632

Piece Name: /home/oracle/rman_back/full_ORASTRAC_20150602_881329118_11_1.bak

Control File Included: Ckp SCN: 1079103      Ckp time: 02-JUN-2015 13:38:38

BS Key  Size       Device Type Elapsed Time Completion Time

------- ---------- ----------- ------------ --------------------

11      15.97M     DISK        00:00:01     02-JUN-2015 13:39:01

BP Key: 11   Status: AVAILABLE  Compressed: NO  Tag: TAG20150602T133859

Piece Name: /home/oracle/rman_back/arch_ORASTRAC_20150602_14_1.bak

List of Archived Logs in backup set 11

Thrd Seq     Low SCN    Low Time             Next SCN   Next Time

---- ------- ---------- -------------------- ---------- ---------

1    4       1057778    01-JUN-2015 17:48:24 1057780    01-JUN-2015 17:48:24

1    5       1057780    01-JUN-2015 17:48:24 1057786    01-JUN-2015 17:48:25

2    4       1057893    01-JUN-2015 17:48:27 1079126    02-JUN-2015 13:38:49

BS Key  Size       Device Type Elapsed Time Completion Time

------- ---------- ----------- ------------ --------------------

12      6.78M      DISK        00:00:01     02-JUN-2015 13:39:01

BP Key: 12   Status: AVAILABLE  Compressed: NO  Tag: TAG20150602T133859

Piece Name: /home/oracle/rman_back/arch_ORASTRAC_20150602_13_1.bak

List of Archived Logs in backup set 12

Thrd Seq     Low SCN    Low Time             Next SCN   Next Time

---- ------- ---------- -------------------- ---------- ---------

1    1       1027357    01-JUN-2015 16:09:51 1049321    01-JUN-2015 17:09:25

1    2       1049321    01-JUN-2015 17:09:25 1056322    01-JUN-2015 17:30:05

1    3       1056520    01-JUN-2015 17:30:48 1057778    01-JUN-2015 17:48:24

2    1       1028434    01-JUN-2015 16:23:50 1028446    01-JUN-2015 16:24:01

2    2       1049792    01-JUN-2015 17:15:06 1057412    01-JUN-2015 17:42:28

2    3       1057776    01-JUN-2015 17:48:24 1057893    01-JUN-2015 17:48:27

3    1       1049702    01-JUN-2015 17:13:10 1049717    01-JUN-2015 17:13:32

BS Key  Size       Device Type Elapsed Time Completion Time

------- ---------- ----------- ------------ --------------------

13      5.05M      DISK        00:00:00     02-JUN-2015 13:39:03

BP Key: 13   Status: AVAILABLE  Compressed: NO  Tag: TAG20150602T133859

Piece Name: /home/oracle/rman_back/arch_ORASTRAC_20150602_15_1.bak

List of Archived Logs in backup set 13

Thrd Seq     Low SCN    Low Time             Next SCN   Next Time

---- ------- ---------- -------------------- ---------- ---------

1    6       1060589    02-JUN-2015 10:16:25 1079130    02-JUN-2015 13:38:49

1    7       1079130    02-JUN-2015 13:38:49 1079140    02-JUN-2015 13:38:56

2    5       1079126    02-JUN-2015 13:38:49 1079144    02-JUN-2015 13:38:58

RMAN> restore database;

Starting restore at 02-JUN-2015 14:25:59

Starting implicit crosscheck backup at 02-JUN-2015 14:25:59

allocated channel: ORA_DISK_1

Crosschecked 7 objects

Finished implicit crosscheck backup at 02-JUN-2015 14:25:59

Starting implicit crosscheck copy at 02-JUN-2015 14:25:59

using channel ORA_DISK_1

Finished implicit crosscheck copy at 02-JUN-2015 14:26:00

searching for all files in the recovery area

cataloging files...

cataloging done

List of Cataloged Files

=======================

File Name: +arch/ORASTRAC/CONTROLFILE/current.369.881258379

File Name: +arch/ORASTRAC/ARCHIVELOG/2015_06_02/thread_2_seq_6.358.881329829

File Name: +arch/ORASTRAC/ARCHIVELOG/2015_06_02/thread_1_seq_8.370.881329829

File Name: +arch/ORASTRAC/ARCHIVELOG/2015_06_02/thread_2_seq_7.359.881329831

using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

channel ORA_DISK_1: restoring datafile 00002 to +DATA/orastrac/datafile/sysaux.331.881251461

channel ORA_DISK_1: restoring datafile 00003 to +DATA/orastrac/datafile/undotbs1.329.881251463

channel ORA_DISK_1: restoring datafile 00005 to +DATA/orastrac/datafile/example.332.881251463

channel ORA_DISK_1: reading from backup piece /home/oracle/rman_back/full_ORASTRAC_20150602_881328993_10_1.bak

channel ORA_DISK_1: piece handle=/home/oracle/rman_back/full_ORASTRAC_20150602_881328993_10_1.bak tag=TAG20150602T133632

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:45

channel ORA_DISK_1: starting datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

channel ORA_DISK_1: restoring datafile 00001 to +DATA/orastrac/datafile/system.330.881251509

channel ORA_DISK_1: restoring datafile 00004 to +DATA/orastrac/datafile/users.327.881251511

channel ORA_DISK_1: restoring datafile 00006 to +DATA/orastrac/datafile/undotbs2.351.881252419

channel ORA_DISK_1: reading from backup piece /home/oracle/rman_back/full_ORASTRAC_20150602_881328992_9_1.bak

channel ORA_DISK_1: piece handle=/home/oracle/rman_back/full_ORASTRAC_20150602_881328992_9_1.bak tag=TAG20150602T133632

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:45

Finished restore at 02-JUN-2015 14:27:31

RMAN> RUN

2> {

3> set until sequence 7 thread 1;

set until sequence 5 thread 2;

recover database;

4> 5> 6> }

executing command: SET until clause

executing command: SET until clause

Starting recover at 02-JUN-2015 14:28:42

using channel ORA_DISK_1

starting media recovery

channel ORA_DISK_1: starting archived log restore to default destination

channel ORA_DISK_1: restoring archived log

archived log thread=2 sequence=4

channel ORA_DISK_1: reading from backup piece /home/oracle/rman_back/arch_ORASTRAC_20150602_14_1.bak

channel ORA_DISK_1: piece handle=/home/oracle/rman_back/arch_ORASTRAC_20150602_14_1.bak tag=TAG20150602T133859

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:03

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_2_seq_4.367.881332133 thread=2 sequence=4

channel ORA_DISK_1: starting archived log restore to default destination

channel ORA_DISK_1: restoring archived log

archived log thread=1 sequence=6

channel ORA_DISK_1: reading from backup piece /home/oracle/rman_back/arch_ORASTRAC_20150602_15_1.bak

channel ORA_DISK_1: piece handle=/home/oracle/rman_back/arch_ORASTRAC_20150602_15_1.bak tag=TAG20150602T133859

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:03

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_1_seq_6.365.881332137 thread=1 sequence=6

channel default: deleting archived log(s)

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_2_seq_4.367.881332133 RECID=26 STAMP=881332134

channel default: deleting archived log(s)

archived log file name=+ARCH/orastrac/archivelog/2015_06_02/thread_1_seq_6.365.881332137 RECID=27 STAMP=881332137

media recovery complete, elapsed time: 00:00:00

Finished recover at 02-JUN-2015 14:28:58

RMAN>

告警日志:

Tue Jun 02 14:26:14 2015

Full restore complete of datafile 5 +DATA/orastrac/datafile/example.331.881331961.  Elapsed time: 0:00:13

checkpoint is 1079018

last deallocation scn is 965277

Full restore complete of datafile 3 +DATA/orastrac/datafile/undotbs1.353.881331961.  Elapsed time: 0:00:13

checkpoint is 1079018

last deallocation scn is 1075330

Undo Optimization current scn is 1074104

Tue Jun 02 14:26:40 2015

Full restore complete of datafile 2 +DATA/orastrac/datafile/sysaux.332.881331961.  Elapsed time: 0:00:39

checkpoint is 1079018

last deallocation scn is 1026156

Tue Jun 02 14:26:50 2015

Full restore complete of datafile 6 +DATA/orastrac/datafile/undotbs2.335.881332007.  Elapsed time: 0:00:04

checkpoint is 1078985

Undo Optimization current scn is 1074104

Full restore complete of datafile 4 +DATA/orastrac/datafile/users.357.881332007.  Elapsed time: 0:00:05

checkpoint is 1078985

Tue Jun 02 14:27:28 2015

Full restore complete of datafile 1 +DATA/orastrac/datafile/system.334.881332007.  Elapsed time: 0:00:41

checkpoint is 1078985

last deallocation scn is 1025383

Undo Optimization current scn is 1074104

Tue Jun 02 14:28:47 2015

alter database recover datafile list clear

Completed: alter database recover datafile list clear

alter database recover datafile list

1 , 2 , 3 , 4 , 5 , 6

Completed: alter database recover datafile list

1 , 2 , 3 , 4 , 5 , 6

alter database recover if needed

start until cancel using backup controlfile

Media Recovery Start

Serial Media Recovery started

ORA-279 signalled during: alter database recover if needed

start until cancel using backup controlfile

...

alter database recover logfile '+ARCH/orastrac/archivelog/2015_06_02/thread_2_seq_4.367.881332133'

Media Recovery Log +ARCH/orastrac/archivelog/2015_06_02/thread_2_seq_4.367.881332133

ORA-279 signalled during: alter database recover logfile '+ARCH/orastrac/archivelog/2015_06_02/thread_2_seq_4.367.881332133'...

Tue Jun 02 14:28:58 2015

alter database recover logfile '+ARCH/orastrac/archivelog/2015_06_02/thread_1_seq_6.365.881332137'

Media Recovery Log +ARCH/orastrac/archivelog/2015_06_02/thread_1_seq_6.365.881332137

ORA-279 signalled during: alter database recover logfile '+ARCH/orastrac/archivelog/2015_06_02/thread_1_seq_6.365.881332137'...

alter database recover cancel

Media Recovery Canceled

Completed: alter database recover cancel

一.3.6.4  RESETLOGS打开数据库并验证数据

RMAN>  alter database open resetlogs;

database opened

RMAN>

RMAN> exit

Recovery Manager complete.

[oracle@node2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jun 2 14:32:25 2015

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select count(1) from lhr.rac_test;

COUNT(1)

----------

72469

SQL>

SQL> set line 9999

SQL> col HOST_NAME format a10

SQL> select INSTANCE_NAME,HOST_NAME,VERSION,STARTUP_TIME,STATUS,ACTIVE_STATE,INSTANCE_ROLE,DATABASE_STATUS from gv$INSTANCE;

INSTANCE_NAME    HOST_NAME  VERSION           STARTUP_TIME        STATUS       ACTIVE_ST INSTANCE_ROLE      DATABASE_STATUS

---------------- ---------- ----------------- ------------------- ------------ --------- ------------------ -----------------

orastrac2        node2      11.2.0.1.0        2015-06-02 14:22:55 OPEN         NORMAL    PRIMARY_INSTANCE   ACTIVE

SQL> select INST_ID,name , open_mode, log_mode,force_logging from gv$database;

INST_ID NAME      OPEN_MODE            LOG_MODE     FOR

---------- --------- -------------------- ------------ ---

2 ORASTRAC  READ WRITE           ARCHIVELOG   NO

SQL>

告警日志:

Tue Jun 02 14:30:03 2015

alter database open resetlogs

Archived Log entry 28 added for thread 1 sequence 7 ID 0x4e8bbda6 dest 1:

Expanded controlfile section 11 from 28 to 172 records

Requested to grow by 144 records; added 6 blocks of records

Archived Log entry 29 added for thread 1 sequence 8 ID 0x4e8bbda6 dest 1:

Archived Log entry 30 added for thread 1 sequence 9 ID 0x4e8bbda6 dest 1:

ARCH: Archiving disabled thread 2 sequence 7

Archived Log entry 31 added for thread 2 sequence 7 ID 0x4e8bbda6 dest 1:

Archived Log entry 32 added for thread 2 sequence 5 ID 0x4e8bbda6 dest 1:

Archived Log entry 33 added for thread 2 sequence 6 ID 0x4e8bbda6 dest 1:

RESETLOGS after incomplete recovery UNTIL CHANGE 1079126

Resetting resetlogs activation ID 1317780902 (0x4e8bbda6)

Tue Jun 02 14:30:16 2015

Setting recovery target incarnation to 4

Tue Jun 02 14:30:16 2015

This instance was first to open

Picked broadcast on commit scheme to generate SCNs

Tue Jun 02 14:30:17 2015

Assigning activation ID 1317937735 (0x4e8e2247)

LGWR: STARTING ARCH PROCESSES

Tue Jun 02 14:30:17 2015

ARC0 started with pid=36, OS id=24160

ARC0: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC0: STARTING ARCH PROCESSES

Tue Jun 02 14:30:18 2015

ARC1 started with pid=37, OS id=24164

Tue Jun 02 14:30:18 2015

ARC2 started with pid=38, OS id=24168

ARC1: Archival started

ARC2: Archival started

ARC1: Becoming the 'no FAL' ARCH

ARC1: Becoming the 'no SRL' ARCH

Tue Jun 02 14:30:18 2015

ARC3 started with pid=39, OS id=24172

ARC2: Becoming the heartbeat ARCH

Thread 2 opened at log sequence 1

Current log# 4 seq# 1 mem# 0: +ARCH/orastrac/onlinelog/group_4.352.881252545

Successful open of redo thread 2

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

Tue Jun 02 14:30:18 2015

SMON: enabling cache recovery

Instance recovery: looking for dead threads

Instance recovery: lock domain invalid but no dead threads

ARC3: Archival started

ARC0: STARTING ARCH PROCESSES COMPLETE

Redo thread 1 internally disabled at seq 1 (CKPT)

ARC3: Archiving disabled thread 1 sequence 1

Archived Log entry 34 added for thread 1 sequence 1 ID 0x0 dest 1:

Successfully onlined Undo Tablespace 5.

Dictionary check beginning

Tue Jun 02 14:30:22 2015

Errors in file /u01/app/oracle/diag/rdbms/orastrac/orastrac2/trace/orastrac2_dbw0_23597.trc:

ORA-01157: cannot identify/lock data file 201 - see DBWR trace file

ORA-01110: data file 201: '+DATA/orastrac/tempfile/temp.333.881251803'

ORA-17503: ksfdopn:2 Failed to open file +DATA/orastrac/tempfile/temp.333.881251803

ORA-15012: ASM file '+DATA/orastrac/tempfile/temp.333.881251803' does not exist

Errors in file /u01/app/oracle/diag/rdbms/orastrac/orastrac2/trace/orastrac2_dbw0_23597.trc:

ORA-01186: file 201 failed verification tests

ORA-01157: cannot identify/lock data file 201 - see DBWR trace file

ORA-01110: data file 201: '+DATA/orastrac/tempfile/temp.333.881251803'

File 201 not verified due to error ORA-01157

Dictionary check complete

Verifying file header compatibility for 11g tablespace encryption..

Verifying 11g file header compatibility for tablespace encryption completed

SMON: enabling tx recovery

Re-creating tempfile +DATA/orastrac/tempfile/temp.333.881251803 as +DATA/orastrac/tempfile/temp.313.881332223

Database Characterset is ZHS16GBK

No Resource Manager plan active

Starting background process GTX0

Tue Jun 02 14:30:23 2015

GTX0 started with pid=40, OS id=24179

Starting background process RCBG

Tue Jun 02 14:30:23 2015

RCBG started with pid=41, OS id=24183

replication_dependency_tracking turned off (no async multimaster replication found)

Tue Jun 02 14:30:27 2015

Starting background process QMNC

Tue Jun 02 14:30:27 2015

QMNC started with pid=42, OS id=24187

LOGSTDBY: Validating controlfile with logical metadata

LOGSTDBY: Validation complete

Tue Jun 02 14:30:48 2015

Completed: alter database open resetlogs

Tue Jun 02 14:30:50 2015

db_recovery_file_dest_size of 3882 MB is 4.43% used. This is a

user-specified limit on the amount of space that will be used by this

database for recovery-related files, and does not reflect the amount of

space available in the underlying filesystem or ASM diskgroup.

Tue Jun 02 14:30:51 2015

Starting background process CJQ0

Tue Jun 02 14:30:51 2015

CJQ0 started with pid=48, OS id=24265

可以看到数据已经恢复,但是实例只启动了一个,我们来启动第二个实例:

一.3.6.5  修复另一个节点的pfile

[oracle@node1 dbs]$ more initorastrac1.ora

SPFILE='+DATA/orastrac/parameterfile/spfileorastrac.ora'

[oracle@node1 dbs]$

[oracle@node1 dbs]$ srvctl modify database -d orastrac  -p +DATA/orastrac/parameterfile/spfileorastrac.ora

[oracle@node1 dbs]$ srvctl start database -d orastrac

[oracle@node1 dbs]$

[oracle@node1 dbs]$ srvctl status database -d orastrac

Instance orastrac1 is running on node node1

Instance orastrac2 is running on node node2

[oracle@node1 dbs]$

[oracle@node2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jun 2 14:45:43 2015

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> set line 9999

SQL> col HOST_NAME format a10

SQL> select INSTANCE_NAME,HOST_NAME,VERSION,STARTUP_TIME,STATUS,ACTIVE_STATE,INSTANCE_ROLE,DATABASE_STATUS from gv$INSTANCE;

INSTANCE_NAME    HOST_NAME  VERSION           STARTUP_TIME        STATUS       ACTIVE_ST INSTANCE_ROLE      DATABASE_STATUS

---------------- ---------- ----------------- ------------------- ------------ --------- ------------------ -----------------

orastrac2        node2      11.2.0.1.0        2015-06-02 14:44:30 OPEN         NORMAL    PRIMARY_INSTANCE   ACTIVE

orastrac1        node1      11.2.0.1.0        2015-06-02 14:44:33 OPEN         NORMAL    PRIMARY_INSTANCE   ACTIVE

SQL> select INST_ID,name , open_mode, log_mode,force_logging from gv$database;

INST_ID NAME      OPEN_MODE            LOG_MODE     FOR

---------- --------- -------------------- ------------ ---

1 ORASTRAC  READ WRITE           ARCHIVELOG   NO

2 ORASTRAC  READ WRITE           ARCHIVELOG   NO

SQL>

一.3.7  实验总结

采用asm来存储的rac环境下的数据库备份和还原与单实例下os文件系统的备份和还原基本一样,不同的是recover的时候,需要设置2个thread。

一.4  总结

到此所有的处理算是基本完毕,过程很简单,但是不同的场景处理方式有很多种,我们应该学会灵活变通。

一.5  About Me

...........................................................................................................................................................................................

本文作者:小麦苗,只专注于数据库的技术,更注重技术的运用

ITPUB BLOG:

本文地址:http://blog.itpub.net/26736162/viewspace-1682237/

本文pdf版:  提取码:af2d

QQ:642808185 若加QQ请注明你所正在读的文章标题

创作时间地点:2015-06-02 10:00~ 2015-06-02 16:00 于外汇交易中心

<版权所有,文章允许转载,但须以链接方式注明源地址,否则追究法律责任!>

...........................................................................................................................................................................................

04-14 10:19