Louis.Kim 의 제 3막

# service o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot.  The current values will be shown in brackets ('[]').  Hitting

ENTER without typing an answer will keep that current value.  Ctrl-C

will abort.

Load O2CB driver on boot (y/n) [n]: y

Cluster stack backing O2CB [o2cb]: 

Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfscluster1

Specify heartbeat dead threshold (>=7) [31]: 

Specify network idle timeout in ms (>=5000) [30000]: 

Specify network keepalive delay in ms (>=1000) [2000]: 

Specify network reconnect delay in ms (>=2000) [2000]: 

Writing O2CB configuration: OK

Loading filesystem "configfs": OK

Mounting configfs filesystem at /sys/kernel/config: OK

Loading stack plugin "o2cb": OK

Loading filesystem "ocfs2_dlmfs": OK

Creating directory '/dlm': OK

Mounting ocfs2_dlmfs filesystem at /dlm: OK

Setting cluster stack "o2cb": OK

Checking O2CB cluster configuration : Failed

# o2cb_ctl -C -n ocfscluster1 -t cluster -a name=ocfscluster1

# o2cb_ctl -C -n ocfsrac1 -t node -a number=0 -a ip_address=192.168.131.100 -a ip_port=7777 -a cluster=ocfscluster1

# o2cb_ctl -C -n ocfsrac2 -t node -a number=1 -a ip_address=192.168.131.110 -a ip_port=7777 -a cluster=ocfscluster1

rac1:/root>service o2cb status

Driver for "configfs": Loaded

Filesystem "configfs": Mounted

Stack glue driver: Loaded

Stack plugin "o2cb": Loaded

Driver for "ocfs2_dlmfs": Loaded

Filesystem "ocfs2_dlmfs": Mounted

Checking O2CB cluster "ocfscluster1": Offline

rac1:/root>service o2cb start ocfscluster1

Setting cluster stack "o2cb": OK

Registering O2CB cluster "ocfscluster1": OK

Setting O2CB cluster timeouts : OK

2015-10-10 12:49:30.874: [ COMMCRS][9526]clsc_receive: (114771870) error 2

2015-10-10 21:54:31.062: [ COMMCRS][9526]clsc_receive: (114771870) error 2

2015-10-13 10:41:30.444: [  CRSEVT][11008]32CAAMonitorHandler :: 0:Could not join /oracle/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

2015-10-13 10:41:30.449: [  CRSEVT][11008]32CAAMonitorHandler :: 0:Action Script /oracle/crs/bin/racgwrap(check) timed out for ora.glvndbp05.vip! (timeout=60)

2015-10-13 10:41:30.449: [  CRSAPP][11008]32CheckResource error for ora.glvndbp05.vip error code = -2

2015-10-13 10:43:03.482: [  CRSEVT][11011]32CAAMonitorHandler :: 0:Could not join /oracle/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

2015-10-13 10:43:03.482: [  CRSEVT][11011]32CAAMonitorHandler :: 0:Action Script /oracle/crs/bin/racgwrap(check) timed out for ora.glvndbp05.vip! (timeout=60)

2015-10-13 10:43:03.482: [  CRSAPP][11011]32CheckResource error for ora.glvndbp05.vip error code = -2

2015-10-13 10:44:36.509: [  CRSEVT][11017]32CAAMonitorHandler :: 0:Could not join /oracle/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

[CRSEVT] CAAMonitorHandler :: 0:Could not join .../crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child

[CRSEVT] CAAMonitorHandler :: 0:Action Script ../crs/bin/racgwrap(check) timed out for 오라클.vip! (timeout=60)

10g RAC 환경에서 있는 bug 인데 racgmain check 데몬이 비정상적으로 fork 되면서 메모리 사용율이 올라가게 되어 결국 나중엔 시스템을 사용할수 없는 지경까지 이르게 됨.

 oracle 26024     1  0  Dec  6  ?         0:00 /oracle/crs/bin/racgmain check

 oracle 23218     1  0  Dec  6  ?         0:00 /oracle/crs/bin/racgmain check

 oracle 23179     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 27277     1  0  Dec  6  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle  1028     1  0  Dec  5  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle  7991     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 15324     1  0  Dec  3  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 14314     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle 10895     1  0  Dec  4  ?         0:00 /oracle/ora10/bin/racgmain check

 oracle   404     1  0  Dec  3  ?         0:00 /oracle/ora10/bin/racgmain check

해결책은 아래와 같이 CRS bundle #2 patchset을 적용시키거나 workaround 방법을 써서 조치해 주어야 함.

=====================================================================================

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.1.0.6

Information in this document applies to any platform.

Oracle Server Enterprise Edition - Version: 10.1.0.2 to 10.2.0.4

Symptoms

System slows down and many "racgmain check" processes may appear in ps output.  CRS log would show the following messages.

oracle@HA5-ZW05:[/home/oracle] ps -ef|grep "racgmain check"|wc -l

1290

~~~~

CAAMonitorHandler :: 0:Action Script /opt/oracle/product/crs/bin/racgwrap(check) timed out for ora.harac1.vip! (timeout=60)

CheckResource error for ora.harac1.vip error code = -2

CAAMonitorHandler :: 0:Could not join /opt/oracle/product/crs/bin/racgwrap(check)

category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, 

other: Abnormal termination of the child

~~~~

Cause

crsd.bin invokes the racgmain to check the status of the resources that are managed by CRS. The racgmain is invoked through the wrapper script racgwrap. 

If the resource action timed out, crsd kills the action script, which is racgwrap, while racgmain process will not be killed. Over time, this might create lot of orphan racgmain processes in the system. This would eventually slow down the due to the resource contention at the OS level.

Internal bug:6196746  addresses this issue.

Solution

This is fixed in 11.1.0.7 patchset.. If you are running into this issue in 10gR2, please go ahead and apply 10.2.0.4 patchset and the latest CRS bundle patch. This fix is included  in CRS bundle patch from bundle #2 onwards.

Following option could be used as a temporary workaround until the patch is applied.

1.  Make a copy of racgwrap located under $ORACLE_HOME/bin and $CRS_HOME/bin on ALL Nodes

2.  Edit the file racgwrap and modify the last 3 lines from:

~~~

$ORACLE_HOME/bin/racgmain "$@"

status=$?

exit $status

to:

# Line added to fix for Bug 6196746

exec $ORACLE_HOME/bin/racgmain "$@"

~~~

3.  Kill all the orphan racgmain processes running.

$ ps -ef|grep "racgmain check"

oracle 18701 1 0 Aug 1 ? 0:00 /oracle/product/10.2.0/database/bin/racgmain check

oracle 14653 1 0 Aug 1 ? 0:00 /oracle/product/10.2.0/database/bin/racgmain check

oracle 24517 1 0 Aug 1 ? 0:00 /oracle/product/10.2.0/database/bin/racgmain check

$ kill -9 <PID of racgmain>

Symptoms

Symptom 1:

In 10g RAC on unix platform, VIP on one nodes always shows "UNKNOWN". 

Symptom 2:

When try to start it up, it report following error:

CRS-1028: Dependency analysis failed because of:

CRS-0223: Resource 'ora.rac-test1.vip' has placement error.  

Symptom 3:

In CRSD log, find following error:

2015-03-19 10:51:09.772: [  CRSRES][3737213248][ALERT]0`ora.rac-test1.vip` on member `rac-test1` has experienced an unrecoverable failure.

2015-03-19 10:51:09.772: [  CRSRES][3737213248]0Human intervention required to resume its availability.

2015-03-19 10:51:09.772: [  CRSEVT][3741415744]0CAAMonitorHandler :: 0:Could not execute /u01/app/oracle/product/10.2.0/crs_1/bin/racgwrap(stop) for ora.rac-test2.vip

category: 1234, operation: scls_canexec, loc: , OS error: 0, other: no exe permission, file /u01/app/oracle/product/10.2.0/crs_1/bin/racgwrap    ===> No execute permission for this file.

Solution

1. Shutdown all the resource of this node: instance/asm/nodeapps/crs

2. Change permission of those 2 files to 751 on the node with issue

chmod 751 /u01/app/oracle/product/10.2.0/crs_1/racg/admin/racgwrap

chmod 751 /u01/app/oracle/product/10.2.0/crs_1/bin/racgeut

3. Then you can startup all the resource and check whether VIP is online.

rac1:/root>mkfs.ocfs2 -b 4K -C 32K -N 2 -L "OCFS2Filesystem" /dev/sdb1

mkfs.ocfs2 1.8.0

Cluster stack: classic o2cb

/dev/sdb1 is apparently in use by the system; will not make a ocfs2 volume here!

<--이렇게 에러가 - _ -;;;

아래와 같이 상태 확인

rac1:/root>dmsetup status

rac-vote01: 0 614400 linear 

rac-control02: 0 204800 linear 

rac-undotbs2: 0 1638400 linear 

rac-control01: 0 204800 linear 

rac-undotbs1: 0 1638400 linear 

rac-system: 0 1638400 linear 

rac-users: 0 1638400 linear 

rac-redo06: 0 409600 linear 

rac-spfile: 0 204800 linear 

rac-temp: 0 1638400 linear 

rac-redo05: 0 409600 linear 

rac-redo04: 0 409600 linear 

rac-sysaux: 0 1638400 linear 

rac-redo03: 0 409600 linear 

rac-vote03: 0 614400 linear 

rac-redo02: 0 409600 linear 

rac-ocr02: 0 614400 linear 

rac-vote02: 0 614400 linear 

rac-redo01: 0 409600 linear 

rac-control03: 0 204800 linear 

rac-ocr01: 0 614400 linear 

젠장 내 잘못이다..기존에 raw device 구성 중에 이 짓을 했으니...ㅠㅠ

아래와 같이 지워주자..

rac1:/root>dmsetup remove_all

다시 진행해 보자..!!

rac1:/root>mkfs.ocfs2 -b 4K -C 32K -N 2 -L "OCFS2Filesystem" /dev/sdb1

mkfs.ocfs2 1.8.0

Cluster stack: classic o2cb

Label: OCFS2Filesystem

Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super xattr indexed-dirs refcount discontig-bg

Block size: 4096 (12 bits)

Cluster size: 32768 (15 bits)

Volume size: 21467922432 (655149 clusters) (5241192 blocks)

Cluster groups: 21 (tail covers 10029 clusters, rest cover 32256 clusters)

Extent allocator size: 12582912 (3 groups)

Journal size: 134184960

Node slots: 2

Creating bitmaps: done

Initializing superblock: done

Writing system files: done

Writing superblock: done

Writing backup superblock: 3 block(s)

Formatting Journals: done

Growing extent allocator: done

Formatting slot map: done

Formatting quota files: done

Writing lost+found: done

mkfs.ocfs2 successful