There was activity @Storage Level by System administrator last night.
today morning 6:00 PM, I got call from support engineer that DB Node1 services not working after storage level activity
DB S/W 10G 10.2.0.4.0 + ASM (2 Node RAC)
Oracle Sun Solaris 10 64bit Platform
DB Node2 is up and running
DB Node1 cluster stack not started automatically after rebooted server
tail -100f /var/adm/messages
Jul 20 07:43:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:43:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:43:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:43:57 node1 pseudo: [ID 129642 kern.info] pseudo-device: bmc0
Jul 20 07:43:57 node1 genunix: [ID 936769 kern.info] bmc0 is /pseudo/bmc@0
Jul 20 07:44:08 node1 scsi: [ID 799468 kern.info] ssd22 at scsi_vhci0: name g600a0b800067e686000003b54c2841d4, bus address g600a0b800067e686000003b54c2841d4
Jul 20 07:44:08 node1 genunix: [ID 936769 kern.info] ssd22 is /scsi_vhci/ssd@g600a0b800067e686000003b54c2841d4
Jul 20 07:44:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:44:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:44:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:45:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:45:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:45:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:46:19 node1 scsi: [ID 193665 kern.info] sd1 at mpt0: target 1 lun 0
Jul 20 07:46:19 node1 genunix: [ID 936769 kern.info] sd1 is /pci@780/pci@0/pci@9/scsi@0/sd@1,0
Jul 20 07:46:19 node1 pseudo: [ID 129642 kern.info] pseudo-device: rsm0
Jul 20 07:46:19 node1 genunix: [ID 936769 kern.info] rsm0 is /pseudo/rsm@0
Jul 20 07:46:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:46:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:46:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:47:39 node1 xntpd[242]: [ID 774427 daemon.notice] time reset (step) 1.544583 s
Jul 20 07:47:53 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:47:53 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:47:53 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268
$cat /tmp/crsctl.1268
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]
./crsctl query css votedisk
./ocrcheck
compare the output of vote and ocr disks using above commands
ls -ltr /dev/rdsk/c1
configure permission
chown -h orarac:oinstall c4t600A0B800067E686000003B74C2843F3d0s0
chown -R 664 c4t600A0B800067E686000003B74C2843F3d0s0
select * from v$asm_disk;
SELECT * FROM v$asm_diskgroup;
DB2
=====
GROUP_NUMBER DISK_NUMBER COMPOUND_INDEX INCARNATION MOUNT_S HEADER_STATU MODE_ST STATE REDUNDA LIBRARY TOTAL_MB FREE_MB NAME FAILGROUP LABEL PATH UDID PRODUCT CREATE_DA MOUNT_DAT REPAIR_TIMER READS WRITES READ_ERRS WRITE_ERRS READ_TIME WRITE_TIME BYTES_READ BYTES_WRITTEN
------------ ----------- -------------- ----------- ------- ------------ ------- -------- ------- ------------------- ---------- ------------ ---------- ------- ------------------------------------------------------ -------- --------- --------- ------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------------
0 0 0 0 CLOSED FOREIGN ONLINE NORMAL UNKNOWN System 3048 0 /dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0 0
1 0 16777216 4042313616 CACHED MEMBER ONLINE NORMAL UNKNOWN System 204752 187619 DATA_0000 DATA_0000 /dev/rdsk/c4t600A0B800067E686000003C54C297E76d0s0 29-JUN-10 22-JAN-14 0 3109195 649757 0 0 71598.79 74535.59 5.2792E+10 5044413440
1 1 16777217 4042313617 CACHED MEMBER ONLINE NORMAL UNKNOWN System 204752 187574 DATA_0001 DATA_0001 /dev/rdsk/c4t600A0B800067E686000003C74C297EBDd0s0 29-JUN-10 22-JAN-14 0 4662501 911380 0 0 85038.87 83353.91 7.5551E+10 9475180032
1 2 16777218 4042313618 CACHED MEMBER ONLINE NORMAL UNKNOWN System 204752 187622 DATA_0002 DATA_0002 /dev/rdsk/c4t600A0B800067E686000003C94C297EFFd0s0 29-JUN-10 22-JAN-14 0 4974687 914113 0 0 75236.7 66627.38 7.9971E+10 9682139648
2 0 33554432 4042313619 CACHED MEMBER ONLINE NORMAL UNKNOWN System 298948 224182 FLASH_0000 FLASH_0000 /dev/rdsk/c4t600A0B800067E686000003CB4C297F64d0s0 29-JUN-10 22-JAN-14 0 28049 55004 0 0 12685.85 1070.03 9655395840 316669952
GROUP_NUMBER“|”NAME “|”SECTOR_SIZE“|”BLOCK_SIZE“|”ALLOCATION_UNIT_SIZE“|”STATE “|”TYPE “|”TOTAL_MB“|”FREE_MB“|”HOT_USED_MB“|”COLD_USED_MB“|”REQUIRED_MIRROR_FREE_MB“|”USABLE_FILE_MB“|”OFFLINE_DISKS“|”COMPATIBILITY “|”DATABASE_COMPATIBILITY “|”VOTIN
------------“|”----------“|”-----------“|”----------“|”--------------------“|”----------“|”--------“|”--------“|”-------“|”-----------“|”------------“|”-----------------------“|”--------------“|”-------------“|”-------------------------“|”-------------------------“|”-----
7“|”DATA “|” 512“|” 4096“|” 4194304“|”UNMOUNTED “|”EXTERN “|” 0“|” 0“|” 0“|” 0“|” 0“|” 870316“|” 0“|”10.2.0.0.0 “|”10.2.0.0.0 “|”N
8“|”FLASH “|” 512“|” 4096“|” 4194304“|”MOUNTED “|”EXTERN “|” 870396“|” 870316“|” 0“|” 80“|” 0“|” 870316“|” 0“|”10.0.0.0.0 “|”10.2.0.0.0 “|”N
DB1
=======
GROUP_NUMBER DISK_NUMBER COMPOUND_INDEX INCARNATION MOUNT_S HEADER_STATU MODE_ST STATE REDUNDA LIBRARY TOTAL_MB FREE_MB NAME FAILGROUP LABEL PATH UDID PRODUCT CREATE_DA MOUNT_DAT REPAIR_TIMER READS WRITES READ_ERRS WRITE_ERRS READ_TIME WRITE_TIME BYTES_READ BYTES_WRITTEN
------------ ----------- -------------- ----------- ------- ------------ ------- -------- ------- ------------------- ---------- ------------ ---------- ------- ------------------------------------------------------ -------- --------- --------- ------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------------
0 0 0 0 CLOSED FOREIGN ONLINE NORMAL UNKNOWN System 3048 0 /dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0 0
0 1 1 0 CLOSED FOREIGN ONLINE NORMAL UNKNOWN System 3048 0 /dev/rdsk/c4t600A0B800067E686000003B54C2841D4d0s0 0
0 2 2 0 CLOSED MEMBER ONLINE NORMAL UNKNOWN System 204752 0 /dev/rdsk/c4t600A0B800067E686000003C74C297EBDd0s0 29-JUN-10 22-JAN-14 0
0 3 3 0 CLOSED MEMBER ONLINE NORMAL UNKNOWN System 204752 0 /dev/rdsk/c4t600A0B800067E686000003C94C297EFFd0s0 29-JUN-10 22-JAN-14 0
2 0 33554432 4042700921 CACHED MEMBER ONLINE NORMAL UNKNOWN System 298948 224182 FLASH_0000 FLASH_0000 /dev/rdsk/c4t600A0B800067E686000003CB4C297F64d0s0 29-JUN-10 22-JAN-14 0 12 1 0 0 .05 0 53248 4096
GROUP_NUMBER“|”NAME “|”SECTOR_SIZE“|”BLOCK_SIZE“|”ALLOCATION_UNIT_SIZE“|”STATE “|”TYPE “|”TOTAL_MB“|”FREE_MB“|”HOT_USED_MB“|”COLD_USED_MB“|”REQUIRED_MIRROR_FREE_MB“|”USABLE_FILE_MB“|”OFFLINE_DISKS“|”COMPATIBILITY “|”DATABASE_COMPATIBILITY “|”VOTIN
------------“|”----------“|”-----------“|”----------“|”--------------------“|”----------“|”--------“|”--------“|”-------“|”-----------“|”------------“|”-----------------------“|”--------------“|”-------------“|”-------------------------“|”-------------------------“|”-----
7“|”DATA “|” 512“|” 4096“|” 4194304“|”MOUNTED “|”EXTERN “|” 870396“|” 870316“|” 0“|” 80“|” 0“|” 870316“|” 0“|”10.0.0.0.0 “|”10.2.0.0.0 “|”N
8“|”FLASH “|” 512“|” 4096“|” 4194304“|”MOUNTED “|”EXTERN “|” 870396“|” 870316“|” 0“|” 80“|” 0“|” 870316“|” 0“|”10.0.0.0.0 “|”10.2.0.0.0 “|”N
Workaround
==========
1. raw devices ownership and permission reassignment
–OCR/VOTE
chown -h orarac:oinstall c100230492348294
chmod -R 664 c100230492348294
–DB
chown -h orarac:oinstall c24294kwhfksjhfk
chmod -R 777 c24294kwhfksjhfk
2. crsctl stop crs
3. crsctl start crs
4. found that one disk group is not mount in DB1 becuase one disk is not configured permission properly
5. Repeat step 1 to 3.
🙂
