[20150529]ORA-16664.txt

--今天在例行检查中.执行 dgmgrl 出现ora-16664错误.
--检查drcdbcn1.log文件,发现如下错误.

05/25/2015 16:34:58
Site dbcndg returned ORA-16664.
Site dbcndg2 returned ORA-16664.
Data Guard Broker Status Summary:
  Type                        Name                             Severity  Status
  Configuration               dbcn                              Warning  ORA-16607
  Primary Database            dbcn                              Success  ORA-00000
  Physical Standby Database   dbcndg                              Error  ORA-16664
  Physical Standby Database   dbcndg2                             Error  ORA-16664

$ oerr ora  16664
16664, 0000, "unable to receive the result from a database"
// *Cause:  During execution of a command, a database in the Data Guard
//          broker configuration failed to return a result.
// *Action: Check Data Guard broker logs for the details of the failure.
//          Ensure network communication is working properly amongst the
//          members of the configuration. Fix any possible network problems
//          and reissue the command.

--检查相应dg的drcdbcndg2.log文件:
05/29/2015 15:18:40
Connection to database dbcn returns ORA-01017.
Please check database dbcn is using a remote password file,
its remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and the SYS password is the same as this database.
Failed to connect to remote database dbcn. Error is ORA-01017
Failed to send message to site dbcn. Error code is ORA-01017.

--很明显按照提示,数据库的sys口令被修改了.而实际上的情况是dg的日志应用是正常的,why什么呢?
--实际上口令根本没有改变.
--只好建立新的口令文件,再检查问题消失!(我一般不建立口令文件,而是从服务器拷贝)

$ xxd -c 16 orapwdbcndg2.xxx >| /tmp/a1.txt
$ xxd -c 16 orapwdbcndg2 >| /tmp/a2.txt
$ diff  /tmp/a1.txt /tmp/a2.txt
43,45c43,45
---
> 00002a0: 0000 0000 1000 0000 1f6c 6a1d 373c 0d87  .........lj.7> 00002b0: 341a a3ce a467 0b30 319c 0026 9f40 8dcb  4..?g.01..&.@.?
> 00002c0: 7759 8363 f8eb 2c00 5359 5300 0000 0000  wY.c?,.SYS.....
50,51c50,51
---
> 0000310: 1f7c e9a6 313b f7cf 3149 da0b b16d 01a3  .|?1;.?I?.m..
> 0000320: 9cdb 8e8c 20ad 5153 2695 e166 c132 d300  .?. .QS&.醘??

--才想起来11G可能前几天修改profile,也修改了口令.
--恢复后我重新修改sys的口令(还是原来的密码),我发现即使我修改成原来的口令,马上出现:

DGMGRL> show configuration
Configuration - dbcn
  Protection Mode: MaxPerformance
  Databases:
    dbcn    - Primary database
    dbcndg  - Physical standby database
      Error: ORA-16664: unable to receive the result from a database

    dbcndg2 - Physical standby database
      Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR

$  ll  orapwdbcn1
-rw-r----- 1 oracle oinstall 1536 2015-05-29 15:53:40 orapwdbcn1

$ ll orapwdbcn2
-rw-r----- 1 oracle oinstall 1536 2014-10-28 17:04:04 orapwdbcn2

$  stat orapwdbcn1
  File: `orapwdbcn1'
  Size: 1536            Blocks: 8          IO Block: 4096   regular file
Device: fc03h/64515d    Inode: 10698784    Links: 1
Access: (0640/-rw-r-----)  Uid: ( 1001/  oracle)   Gid: ( 1001/oinstall)
Access: 2015-05-29 16:03:03.000000000 +0800
Modify: 2015-05-29 15:53:40.000000000 +0800
Change: 2015-05-29 16:03:03.000000000 +0800

$ stat orapwdbcn2
  File: `orapwdbcn2'
  Size: 1536            Blocks: 8          IO Block: 4096   regular file
Device: fc03h/64515d    Inode: 5996564     Links: 1
Access: (0640/-rw-r-----)  Uid: ( 1001/  oracle)   Gid: ( 1001/oinstall)
Access: 2015-05-29 16:03:40.000000000 +0800
Modify: 2014-10-28 17:04:04.000000000 +0800
Change: 2015-05-29 16:03:40.000000000 +0800

--还发现一个bug,这样修改口令文件,仅仅在一个实例上修改.oracle真tmd的变态!
--再做一个检查就知道了,拷贝口令文件到dg后:

--在RAC实例1上执行:
DGMGRL> show configuration
Configuration - dbcn
  Protection Mode: MaxPerformance
  Databases:
    dbcn    - Primary database
    dbcndg  - Physical standby database
    dbcndg2 - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

--在RAC实例2上执行:
DGMGRL> show configuration
Configuration - dbcn
  Protection Mode: MaxPerformance
  Databases:
    dbcn    - Primary database
    dbcndg  - Physical standby database
      Error: ORA-16664: unable to receive the result from a database
    dbcndg2 - Physical standby database
      Error: ORA-16664: unable to receive the result from a database
Fast-Start Failover: DISABLED
Configuration Status:
ERROR

--我把实例1的口令文件拷贝到实例2:
--在RAC实例2上执行:
DGMGRL> show configuration
Configuration - dbcn
  Protection Mode: MaxPerformance
  Databases:
    dbcn    - Primary database
    dbcndg  - Physical standby database
    dbcndg2 - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

--我把dg在使用原来的口令文件,问题再次出现!
--看来千万不要乱改sys口令.

--在各个机器检查一遍,dgmgrl显示都是正常了.
--这个问题到底是什么原因引起的,要找测试环境来测试看看.