更新时间:2022-09-15 22:31:57
1:周末遭遇停电,新配置的内网测试数据库active dataguard 环境自动关闭,今早启动主库后发现报错如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
[oracle@db1 ~]$ sqlplus /nolog SQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 17 17:00:57 2013 Copyright (c) 1982, 2009, Oracle. All rights reserved. SQL> conn /as sysdba Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 1.3362E+10 bytes Fixed Size 2217952 bytes Variable Size 6777997344 bytes Database Buffers 6576668672 bytes Redo Buffers 4960256 bytes Database mounted. ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], [] |
2:第一反应先recover下database,再尝试拉起数据库,问题依旧
1
2
3
4
5
6
7
8
|
SQL> recover database; Media recovery complete. SQL> alter database open; alter database open * ERROR at line 1 :
ORA- 00600 : internal error code, arguments: [kcratr_nab_less_than_odr], [ 1 ],
[ 6 ], [ 301353 ], [ 301354 ], [], [], [], [], [], [], []
|
3:于是看alert日志和相关的trace文件,综合判断可能控制文件出现问题
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
|
[oracle@db1 ~]$ tail -f alert_db.log Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_ora_32462.trc:
ORA- 00600 : internal error code, arguments: [kcratr_nab_less_than_odr], [ 1 ], [ 6 ], [ 301353 ], [ 301354 ], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_ora_32462.trc:
ORA- 00600 : internal error code, arguments: [kcratr_nab_less_than_odr], [ 1 ], [ 6 ], [ 301353 ], [ 301354 ], [], [], [], [], [], [], []
ORA- 600 signalled during: ALTER DATABASE OPEN...
Trace dumping is performing id=[cdmp_20130617170117]
Mon Jun 17 17 : 02 : 13 2013
Sweep [inc][ 24153 ]: completed
Sweep [inc2][ 24153 ]: completed
Mon Jun 17 17 : 02 : 38 2013
ALTER DATABASE RECOVER database Media Recovery Start started logmerger process
Parallel Media Recovery started with 4 slaves
Mon Jun 17 17 : 02 : 38 2013
Recovery of Online Redo Log: Thread 1 Group 3 Seq 6 Reading mem 0
Mem# 0 : /u01/app/oracle/oradata/DB/onlinelog/o1_mf_3_8vpmjgoq_.log
Mem# 1 : /u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_3_8vpmkm9x_.log
Media Recovery Complete (db) Completed: ALTER DATABASE RECOVER database Mon Jun 17 17 : 02 : 54 2013
alter database open Beginning crash recovery of 1 threads
parallel recovery started with 3 processes
Started redo scan Completed redo scan read 152 KB redo, 0 data blocks need recovery
Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_ora_32462.trc (incident= 24154 ):
ORA- 00600 : internal error code, arguments: [kcratr_nab_less_than_odr], [ 1 ], [ 6 ], [ 301353 ], [ 301354 ], [], [], [], [], [], [], []
Incident details in : /u01/app/oracle/diag/rdbms/db1/db/incident/incdir_24154/db_ora_32462_i24154.trc
Mon Jun 17 17 : 02 : 55 2013
Trace dumping is performing id=[cdmp_20130617170255]
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_ora_32462.trc:
ORA- 00600 : internal error code, arguments: [kcratr_nab_less_than_odr], [ 1 ], [ 6 ], [ 301353 ], [ 301354 ], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_ora_32462.trc:
ORA- 00600 : internal error code, arguments: [kcratr_nab_less_than_odr], [ 1 ], [ 6 ], [ 301353 ], [ 301354 ], [], [], [], [], [], [], []
ORA- 600 signalled during: alter database open...
Mon Jun 17 17 : 03 : 13 2013
Sweep [inc][ 24154 ]: completed
Sweep [inc2][ 24154 ]: completed
|
4:于是生成控制文件trace控制脚本对控制文件执行恢复
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
|
SQL> select open_mode from v$database; OPEN_MODE -------------------- MOUNTED SQL> alter database backup controlfile to trace as '/tmp/1.ctl' ;
Database altered. [oracle@db1 ~]$ cat /tmp/ 1 .ctl
CREATE CONTROLFILE REUSE DATABASE "DB" NORESETLOGS FORCE LOGGING ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 292
LOGFILE GROUP 1 (
'/u01/app/oracle/oradata/DB/onlinelog/o1_mf_1_8vpmdkl9_.log' ,
'/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_1_8vpmdryt_.log'
) SIZE 512M BLOCKSIZE 512 ,
GROUP 2 (
'/u01/app/oracle/oradata/DB/onlinelog/o1_mf_2_8vpmfqmw_.log' ,
'/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_2_8vpmhk2z_.log'
) SIZE 512M BLOCKSIZE 512 ,
GROUP 3 (
'/u01/app/oracle/oradata/DB/onlinelog/o1_mf_3_8vpmjgoq_.log' ,
'/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_3_8vpmkm9x_.log'
) SIZE 512M BLOCKSIZE 512
--STANDBY LOGFILE -- GROUP 4 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_4_8vpq6nvy_.log' ,
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_4_8vpq7wk8_.log'
-- ) SIZE 512M BLOCKSIZE 512 ,
-- GROUP 5 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_5_8vpqbh6s_.log' ,
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_5_8vpqcmbj_.log'
-- ) SIZE 512M BLOCKSIZE 512 ,
-- GROUP 6 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_6_8vpqf3rz_.log' ,
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_6_8vpqfv5w_.log'
-- ) SIZE 512M BLOCKSIZE 512 ,
-- GROUP 7 (
-- '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_7_8vpqgw0j_.log' ,
-- '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_7_8vpqhcql_.log'
-- ) SIZE 512M BLOCKSIZE 512
DATAFILE '/u01/app/oracle/oradata/DB/datafile/o1_mf_system_8vpm8hf3_.dbf' ,
'/u01/app/oracle/oradata/DB/datafile/o1_mf_sysaux_8vpm8hjq_.dbf' ,
'/u01/app/oracle/oradata/DB/datafile/o1_mf_undotbs1_8vpm8hl4_.dbf' ,
'/u01/app/oracle/oradata/DB/datafile/o1_mf_users_8vpm8hn1_.dbf'
CHARACTER SET ZHS16GBK; SQL> shutdown immediate ORA- 01109 : database not open
Database dismounted. ORACLE instance shut down. SQL> startup nomount ORACLE instance started. Total System Global Area 1 .3362E+ 10 bytes
Fixed Size 2217952 bytes
Variable Size 6777997344 bytes
Database Buffers 6576668672 bytes
Redo Buffers 4960256 bytes
SQL> @/tmp/ 1 .ctl;
Control file created. |
5:重建控制文件后,对数据库进行recover,成功打开数据库
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
SQL> alter database open; alter database open * ERROR at line 1 :
ORA- 01113 : file 1 needs media recovery
ORA- 01110 : data file 1 :
'/u01/app/oracle/oradata/DB/datafile/o1_mf_system_8vpm8hf3_.dbf' SQL> recover database; Media recovery complete. SQL> alter database open; Database altered. SQL> select open_mode,database_role from v$database; OPEN_MODE DATABASE_ROLE -------------------- ---------------- READ WRITE PRIMARY |
6:通过观察alert日志,添加临时表空间数据文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
[oracle@db1 ~]$ tail -f alert_db.log Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_m001_381.trc:
ORA- 25153 : Temporary Tablespace is Empty
Mon Jun 17 17 : 17 : 25 2013
Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_j007_400.trc:
ORA- 25153 : Temporary Tablespace is Empty
Errors in file /u01/app/oracle/diag/rdbms/db1/db/ trace /db_j007_400.trc:
ORA- 12012 : error on auto execute of job 12696
ORA- 25153 : Temporary Tablespace is Empty
ORA- 06512 : at "DBSNMP.BSLN_INTERNAL" , line 2073
ORA- 06512 : at line 1
Mon Jun 17 17 : 18 : 20 2013
alter tablespace temp add tempfile Completed: alter tablespace temp add tempfile SQL> alter tablespace temp add tempfile; Tablespace altered. |
7:观察physical database,发现出现如下问题
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
[root@db2 ~]# su - oracle [oracle@db2 ~]$ sqlplus /nolog SQL*Plus: Release 11.2 . 0.1 . 0 Production on Mon Jun 17 17 : 18 : 50 2013
Copyright (c) 1982 , 2009 , Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 1 .3362E+ 10 bytes
Fixed Size 2217952 bytes
Variable Size 6777997344 bytes
Database Buffers 6576668672 bytes
Redo Buffers 4960256 bytes
Database mounted. ORA- 10458 : standby database requires recovery
ORA- 01196 : file 1 is inconsistent due to a failed media recovery session
ORA- 01110 : data file 1 :
'/u01/app/oracle/oradata/DB2/datafile/o1_mf_system_08oc7c38_.dbf' SQL> alter database open read only; alter database open read only * ERROR at line 1 :
ORA- 10458 : standby database requires recovery
ORA- 01196 : file 1 is inconsistent due to a failed media recovery session
ORA- 01110 : data file 1 :
'/u01/app/oracle/oradata/DB2/datafile/o1_mf_system_08oc7c38_.dbf' |
8:于是关闭physical standby,在主库上启动system表空间的热备份,拷贝相应的文件至从库指定位置,重新同步正常
1
2
3
4
5
6
7
8
9
10
11
|
SQL> shutdown immediate ORA- 01109 : database not open
Database dismounted. ORACLE instance shut down. SQL> alter tablespace system begin backup; Tablespace altered. [oracle@db1 datafile]$ pwd /u01/app/oracle/oradata/DB/datafile [oracle@db1 datafile]$ scp o1_mf_system_8vpm8hf3_.dbf root@db2:/tmp SQL> alter tablespace system end backup; Tablespace altered. 本文转自斩月博客51CTO博客,原文链接http://blog.51cto.com/ylw6006/1223802如需转载请自行联系原作者 ylw6006 |