Louis.Kim 의 제 3막

Hotbackup 도중 베리타스 에서 DB를 내리게 될 경우..

SQL> alter tablespace users begin backup;

Tablespace altered.

-----------------

Alert Log

SQL> alter database datafile '/oradata/EAIDB/users01.dbf' end backup;

Database altered.

SQL> alter database open;

Database altered.

SQL> recover database;

Media recovery complete.

SQL> alter database open;

Database altered.

2015-10-10 12:49:30.874: [ COMMCRS][9526]clsc_receive: (114771870) error 2

2015-10-10 21:54:31.062: [ COMMCRS][9526]clsc_receive: (114771870) error 2

2015-10-13 10:41:30.444: [  CRSEVT][11008]32CAAMonitorHandler :: 0:Could not join /oracle/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

2015-10-13 10:41:30.449: [  CRSEVT][11008]32CAAMonitorHandler :: 0:Action Script /oracle/crs/bin/racgwrap(check) timed out for ora.glvndbp05.vip! (timeout=60)

2015-10-13 10:41:30.449: [  CRSAPP][11008]32CheckResource error for ora.glvndbp05.vip error code = -2

2015-10-13 10:43:03.482: [  CRSEVT][11011]32CAAMonitorHandler :: 0:Could not join /oracle/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

2015-10-13 10:43:03.482: [  CRSEVT][11011]32CAAMonitorHandler :: 0:Action Script /oracle/crs/bin/racgwrap(check) timed out for ora.glvndbp05.vip! (timeout=60)

2015-10-13 10:43:03.482: [  CRSAPP][11011]32CheckResource error for ora.glvndbp05.vip error code = -2

2015-10-13 10:44:36.509: [  CRSEVT][11017]32CAAMonitorHandler :: 0:Could not join /oracle/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

[CRSEVT] CAAMonitorHandler :: 0:Could not join .../crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

[CRSEVT] CAAMonitorHandler :: 0:Action Script ../crs/bin/racgwrap(check) timed out for 오라클.vip! (timeout=60)

10g RAC 환경에서 있는 bug 인데 racgmain check 데몬이 비정상적으로 fork 되면서 메모리 사용율이 올라가게 되어 결국 나중엔 시스템을 사용할수 없는 지경까지 이르게 됨.

 oracle 26024     1  0  Dec  6  ?         0:00 /oracle/crs/bin/racgmain check

 oracle 23218     1  0  Dec  6  ?         0:00 /oracle/crs/bin/racgmain check

 oracle 23179     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 27277     1  0  Dec  6  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle  1028     1  0  Dec  5  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle  7991     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 15324     1  0  Dec  3  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 14314     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 10895     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle   404     1  0  Dec  3  ?         0:00 /oracle/ora10/bin/racgmain check

해결책은 아래와 같이 CRS bundle #2 patchset을 적용시키거나 workaround 방법을 써서 조치해 주어야 함.

=====================================================================================

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.1.0.6

Information in this document applies to any platform.

Oracle Server Enterprise Edition - Version: 10.1.0.2 to 10.2.0.4

Symptoms

System slows down and many "racgmain check" processes may appear in ps output.  CRS log would show the following messages.

oracle@HA5-ZW05:[/home/oracle] ps -ef|grep "racgmain check"|wc -l

1290

~~~~

CAAMonitorHandler :: 0:Action Script /opt/oracle/product/crs/bin/racgwrap(check) timed out for ora.harac1.vip! (timeout=60)

CheckResource error for ora.harac1.vip error code = -2

CAAMonitorHandler :: 0:Could not join /opt/oracle/product/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, 

other: Abnormal termination of the child

~~~~

Cause

crsd.bin invokes the racgmain to check the status of the resources that are managed by CRS. The racgmain is invoked through the wrapper script racgwrap. 

If the resource action timed out, crsd kills the action script, which is racgwrap, while racgmain process will not be killed. Over time, this might create lot of orphan racgmain processes in the system. This would eventually slow down the due to the resource contention at the OS level.

Internal bug:6196746  addresses this issue.

Solution

This is fixed in 11.1.0.7 patchset.. If you are running into this issue in 10gR2, please go ahead and apply 10.2.0.4 patchset and the latest CRS bundle patch. This fix is included  in CRS bundle patch from bundle #2 onwards.

Following option could be used as a temporary workaround until the patch is applied.

1.  Make a copy of racgwrap located under $ORACLE_HOME/bin and $CRS_HOME/bin on ALL Nodes

2.  Edit the file racgwrap and modify the last 3 lines from:

~~~

$ORACLE_HOME/bin/racgmain "$@"

status=$?

exit $status

to:

# Line added to fix for Bug 6196746

exec $ORACLE_HOME/bin/racgmain "$@"

~~~

3.  Kill all the orphan racgmain processes running.

$ ps -ef|grep "racgmain check"

oracle 18701 1 0 Aug 1 ? 0:00 /oracle/product/10.2.0/database/bin/racgmain check

oracle 14653 1 0 Aug 1 ? 0:00 /oracle/product/10.2.0/database/bin/racgmain check

oracle 24517 1 0 Aug 1 ? 0:00 /oracle/product/10.2.0/database/bin/racgmain check

$ kill -9 <PID of racgmain>

Symptoms

Symptom 1:

In 10g RAC on unix platform, VIP on one nodes always shows "UNKNOWN". 

Symptom 2:

When try to start it up, it report following error:

CRS-1028: Dependency analysis failed because of:

CRS-0223: Resource 'ora.rac-test1.vip' has placement error.  

Symptom 3:

In CRSD log, find following error:

2015-03-19 10:51:09.772: [  CRSRES][3737213248][ALERT]0`ora.rac-test1.vip` on member `rac-test1` has experienced an unrecoverable failure.

2015-03-19 10:51:09.772: [  CRSRES][3737213248]0Human intervention required to resume its availability.

2015-03-19 10:51:09.772: [  CRSEVT][3741415744]0CAAMonitorHandler :: 0:Could not execute /u01/app/oracle/product/10.2.0/crs_1/bin/racgwrap(stop) for ora.rac-test2.vip

category: 1234, operation: scls_canexec, loc: , OS error: 0, other: no exe permission, file /u01/app/oracle/product/10.2.0/crs_1/bin/racgwrap    ===> No execute permission for this file.

Solution

1. Shutdown all the resource of this node: instance/asm/nodeapps/crs

2. Change permission of those 2 files to 751 on the node with issue

chmod 751 /u01/app/oracle/product/10.2.0/crs_1/racg/admin/racgwrap

chmod 751 /u01/app/oracle/product/10.2.0/crs_1/bin/racgeut

3. Then you can startup all the resource and check whether VIP is online.

* 이미지가 작게 보일 것 같아서 아래와 같이 유사 내용을 가져왔습니다.

INFO: Problem in configuration: PRCN-2061 : Failed to add listener ora.LISTENER.lsnr
INFO: PRCN-2065 : Port(s) 1521 are not available on the nodes given
INFO: PRCN-2067 : Port 1521 is not available across node(s) "hww-poc1-VIP,hww-poc2-VIP"
INFO: Oracle Net Listener Startup:
INFO:     Listener does not exists.
INFO: Check the trace file for details: /home/grid/app/grid/cfgtoollogs/netca/trace_Ora11g_gridinfrahome1-1410287PM2700.log
INFO: Oracle Net Services configuration failed.  The exit code is 1

[root@rac1 install]# pwd

/oracle/grid/crs/install

[root@rac1 install]# ./rootcrs.pl -deconfig -force -verbose

Using configuration parameter file: ./crsconfig_params

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Delete failed, or completed with errors.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'

CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'rac1'

CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'

CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

Successfully deconfigured Oracle Restart stack

[root@rac2 install]# ./rootcrs.pl -deconfig -force -verbose

Using configuration parameter file: ./crsconfig_params

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Stop failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Delete failed, or completed with errors.

CRS-4047: No Oracle Clusterware components configured.

CRS-4000: Command Stop failed, or completed with errors.

You must kill ohasd processes or reboot the system to properly 

cleanup the processes started by Oracle clusterware

ACFS-9313: No ADVM/ACFS installation detected.

Either /etc/oracle/olr.loc does not exist or is not readable

Make sure the file exists and it has read and execute access

Failure in execution (rc=-1, 256, No such file or directory) for command /etc/init.d/ohasd deinstall

Successfully deconfigured Oracle Restart stack

[root@rac1 oracle]# $GRID_HOME/root.sh

[root@rac2 oracle]# $GRID_HOME/root.sh

[설명] 양쪽 노드에서 아래와 같이 Grid가 설치되어있는 홈 디렉토리에서 작업을 진행 하도록 하겠습니다. 우선 현재 crs, asm 등등 리소스를 확인 합니다. 어차피 띄워져 있어도 자동으로 모두 제거 하도록 하겠습니다. 아래 작업은 노드 1, 노드 2에서 모두 해주셔야 합니다.

[설명] 로컬 인벤토리에 있는 데이터 파일도 모두 제거 합니다.( 제거하지 않을 경우 그리드 설치시 에러가 발생 됩니다.)
[root@rac2 oraInventory]# $ORACLE_HOME/oraInventory/rm -rf *

[설명] 환경설정이 되어있으므로 아래와 같이 모두 제거 합니다. 
[root@rac2 u01]# rm -rf /etc/ora*

[설명] 데몬이 설정되어 있다면 rootdeinstall.sh를 반드시 수행 하셔야 합니다.  이후 아래 파일을 제거 하시기 바랍니다.
rm -f /etc/init.d/init.cssd 
rm -f /etc/init.d/init.crs 
rm -f /etc/init.d/init.crsd 
rm -f /etc/init.d/init.evmd 
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
rm -Rf /etc/oracle/scls_scr
rm -f /etc/inittab.crs 
cp /etc/inittab.orig /etc/inittab

[oracle@rac1:/oracle/grid]$ cd deinstall/

[oracle@rac1:/oracle/grid/deinstall]$ ./deinstall

Checking for required files and bootstrapping ...

Please wait ...

Location of logs /tmp/deinstall2015-09-11_09-37-29AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START #########################

## [START] Install check configuration ##

Checking for existence of the Oracle home location /oracle/grid

Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster

Oracle Base selected for deinstall is: /oracle/app/oracle

Checking for existence of central inventory location /oracle/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home /oracle/grid

The following nodes are part of this cluster: rac1,rac2

Checking for sufficient temp space availability on node(s) : 'rac1,rac2'

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2015-09-11_09-37-29AM/logs//crsdc.log

Enter an address or the name of the virtual IP used on node "rac1"[rac1-vip]

 > 

The following information can be collected by running "/sbin/ifconfig -a" on node "rac1"

Enter the IP netmask of Virtual IP "192.168.131.120" on node "rac1"[255.255.255.0]

 > 

Enter the network interface name on which the virtual IP address "192.168.131.120" is active

 > 

Enter an address or the name of the virtual IP used on node "rac2"[rac2-vip]

 > 

The following information can be collected by running "/sbin/ifconfig -a" on node "rac2"

Enter the IP netmask of Virtual IP "192.168.131.130" on node "rac2"[255.255.255.0]

 > 

Enter the network interface name on which the virtual IP address "192.168.131.130" is active

 > 

Enter an address or the name of the virtual IP[]

 > 

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2015-09-11_09-37-29AM/logs/netdc_check2015-09-11_10-08-47-AM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER,LISTENER_SCAN1]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2015-09-11_09-37-29AM/logs/asmcadc_check2015-09-11_10-08-48-AM.log

ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: 

ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is: /oracle/grid

The cluster node(s) on which the Oracle home deinstallation will be performed are:rac1,rac2

Oracle Home selected for deinstall is: /oracle/grid

Inventory Location where the Oracle home registered is: /oracle/app/oraInventory

Following RAC listener(s) will be de-configured: LISTENER,LISTENER_SCAN1

ASM was not detected in the Oracle Home

Do you want to continue (y - yes, n - no)? [n]: y

A log of this session will be written to: '/tmp/deinstall2015-09-11_09-37-29AM/logs/deinstall_deconfig2015-09-11_09-37-37-AM.out'

Any error messages from this session will be written to: '/tmp/deinstall2015-09-11_09-37-29AM/logs/deinstall_deconfig2015-09-11_09-37-37-AM.err'

######################## CLEAN OPERATION START ########################

ASM de-configuration trace file location: /tmp/deinstall2015-09-11_09-37-29AM/logs/asmcadc_clean2015-09-11_10-08-51-AM.log

ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2015-09-11_09-37-29AM/logs/netdc_clean2015-09-11_10-08-51-AM.log

De-configuring RAC listener(s): LISTENER,LISTENER_SCAN1

De-configuring listener: LISTENER

    Stopping listener: LISTENER

    Warning: Failed to stop listener. Listener may not be running.

Listener de-configured successfully.

De-configuring listener: LISTENER_SCAN1

    Stopping listener: LISTENER_SCAN1

    Warning: Failed to stop listener. Listener may not be running.

Listener de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...

Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...

Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...

Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...

Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

---------------------------------------->

The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on  the local node after the execution completes on all the remote nodes.

Run the following command as the root user or the administrator on node "rac2".

/tmp/deinstall2015-09-11_09-37-29AM/perl/bin/perl -I/tmp/deinstall2015-09-11_09-37-29AM/perl/lib -I/tmp/deinstall2015-09-11_09-37-29AM/crs/install /tmp/deinstall2015-09-11_09-37-29AM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2015-09-11_09-37-29AM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Run the following command as the root user or the administrator on node "rac1".

/tmp/deinstall2015-09-11_09-37-29AM/perl/bin/perl -I/tmp/deinstall2015-09-11_09-37-29AM/perl/lib -I/tmp/deinstall2015-09-11_09-37-29AM/crs/install /tmp/deinstall2015-09-11_09-37-29AM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2015-09-11_09-37-29AM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode

Press Enter after you finish running the above commands

<----------------------------------------

**여기서 Enter 치지 말고, console 를 하나 더 열어 root 로 위에 빨간색 명령어 진행

[root@rac2 ~]# /tmp/deinstall2015-09-11_09-37-29AM/perl/bin/perl -I/tmp/deinstall2015-09-11_09-37-29AM/perl/lib -I/tmp/deinstall2015-09-11_09-37-29AM/crs/install /tmp/deinstall2015-09-11_09-37-29AM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2015-09-11_09-37-29AM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Using configuration parameter file: /tmp/deinstall2015-09-11_09-37-29AM/response/deinstall_Ora11g_gridinfrahome1.rsp

...

**기다리면 succeeded 등이 뜨면서 끝날때까지 기다린다.(난.. failed..Y Y)

Successfully deconfigured Oracle clusterware stack on this node <--이런식으로..

[root@rac1 ~]# 

[root@rac1 ~]# root

-bash: root: command not found

[root@rac1 ~]# /tmp/deinstall2015-09-11_09-37-29AM/perl/bin/perl -I/tmp/deinstall2015-09-11_09-37-29AM/perl/lib -I/tmp/deinstall2015-09-11_09-37-29AM/crs/install /tmp/deinstall2015-09-11_09-37-29AM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2015-09-11_09-37-29AM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode

...

Successfully deconfigured Oracle clusterware stack on this node

다시 deinstall 하는 곳으로 넘어와서 Enter를 친다.

<----------------------------------------

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START

Detach Oracle home '/oracle/grid' from the central inventory on the local node : Done

Delete directory '/oracle/grid' on the local node : Done

Delete directory '/oracle/app/oraInventory' on the local node : Done

Delete directory '/oracle/app/oracle' on the local node : Done

Detach Oracle home '/oracle/grid' from the central inventory on the remote nodes 'rac2' : Done

Delete directory '/oracle/grid' on the remote nodes 'rac2' : Done

Delete directory '/oracle/app/oraInventory' on the remote nodes 'rac2' : Done

Delete directory '/oracle/app/oracle' on the remote nodes 'rac2' : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2015-09-11_09-37-29AM' on node 'rac1'

Clean install operation removing temporary directory '/tmp/deinstall2015-09-11_09-37-29AM' on node 'rac2'

## [END] Oracle install clean ##

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################

Following RAC listener(s) were de-configured successfully: LISTENER,LISTENER_SCAN1

Oracle Clusterware is stopped and successfully de-configured on node "rac1"

Oracle Clusterware is stopped and successfully de-configured on node "rac2"

Oracle Clusterware is stopped and de-configured successfully.

Successfully detached Oracle home '/oracle/grid' from the central inventory on the local node.

Successfully deleted directory '/oracle/grid' on the local node.

Successfully deleted directory '/oracle/app/oraInventory' on the local node.

Successfully deleted directory '/oracle/app/oracle' on the local node.

Successfully detached Oracle home '/oracle/grid' from the central inventory on the remote nodes 'rac2'.

Successfully deleted directory '/oracle/grid' on the remote nodes 'rac2'.

Successfully deleted directory '/oracle/app/oraInventory' on the remote nodes 'rac2'.

Successfully deleted directory '/oracle/app/oracle' on the remote nodes 'rac2'.

Oracle Universal Installer cleanup was successful.

Run 'rm -rf /etc/oraInst.loc' as root on node(s) 'rac1,rac2' at the end of the session.

Run 'rm -rf /opt/ORCLfmap' as root on node(s) 'rac1,rac2' at the end of the session.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

[oracle@rac1:/oracle/grid/deinstall]$ 

[root@rac1 ~]# rm -rf /etc/oraInst.loc

[root@rac1 ~]# rm -rf /opt/ORCLfmap

----------------------------------------------------------

[root@rac2 ~]# rm -rf /opt/ORCLfmap

[root@rac2 ~]# rm -rf /etc/oraInst.loc