CLSU-00100: operating system function: opendir failed with error data: 2

CLSU-00100: operating system function: opendir failed with error data: 2

[root@localhost ~]# /u01/app/12.1.0.2/grid/bin/crsctl check crs
Oracle Clusterware infrastructure error in CRSCTL (OS PID 33435): ADR home path /u01/app/oracle/diag/crs/localhost/crs does not exist; ADR initialization will try to create it

CRS-4639: Could not contact Oracle High Availability Service

http://www.oracle-scn.com/asm-not-starting-after-changing-the-hostname-or-ip-address-of-the-server/
https://mehmeteser.wordpress.com/2013/02/21/rename-host-has/
http://www.toadworld.com/platforms/oracle/b/weblog/archive/2016/01/23/how-to-fix-quot-cant-open-scr-home-dir-scls-scr-getval-quot-error

hostname csm-clstrnode1

crsctl start crs

ORA-15087: disk ‘string’ is formatted as an ext2/ext3 or OCFS2 file system

Problem:
ORA-15087: disk ‘string’ is formatted as an ext2/ext3 or OCFS2 file system

Document:
Cannot See ASM Disks From Asmca when using udev rules on rhel 6 (Doc ID 1911888.1)

SOLUTION
Clear the header by writing 25 MB of null, after clearing the header ASMCA and KFOD can able to see the devices
For example:

# dd if=/dev/zero of= bs=1M count=25

How to find raw partition from asm disk ?

If your ASM is using RAW devices then its pretty easy to find out by just looking into your /etc/sysconfig/rawdevices (hope your DBA had made some comments as to which RAW devices is for which).

If you are using ASMLib then its kinda hard to find out the exact mapping of your ORACLEASM stamped device to the Linux Physical Device.

To make it simple, here are the steps to find the Physical Device that is mapped to your ASMLib disk:
Run the following as a root:
1. /etc/init.d/oracleasm listdisks — lists all the disks that are in your ASM
2. /etc/init.d/oracleasm querydisk -d disk_name_that_you_wan_to_find_physical_device (this name is from the above result)
3. Note the values in braces [ n1, n2] from the above command result
4. do ls -al /dev/ |grep n1 |grep n2 — n1 and n2 are the values from the above step you noted

That gives you the device number its mapped to.

#/etc/init.d/oracleasm querydisk -d TIER2_DATA01
Disk “DG1_DISK1” is a valid ASM disk on device [253, 26]

#ls -l /dev |grep 253 |grep 26
brw-rw—- 1 root root 253, 26 Nov 3 12:03 dm-26

From the above dm-26 is the physical device that DG1_DISK1 ASM Disk is mapped to

http://sanoralife.blogspot.in/2012/02/how-to-find-mapping-of-asm-disk-to.html
https://blogs.oracle.com/AlejandroVargas/entry/mapping_asm_disks_to_physical

Reconfigure RAC from single scan vip to three scan vip

Artical: Reconfigure rac from single scan vip to three scan vip

Step-1
======
Node-1 Configured as DNS Server.
yum install bind-chroot

Step-2
======
[root@ocm1 ~]# cat /var/named/chroot/etc/named.conf
options {
listen-on port 53 { 127.0.0.1; 192.168.56.101; };
directory “/var/named”;
dump-file “/var/named/data/cache_dump.db”;
statistics-file “/var/named/data/named_stats.txt”;
/*
* If there is a firewall between you and nameservers you want
* to talk to, you might need to uncomment the query-source
* directive below. Previous versions of BIND always asked
* questions using port 53, but BIND 8.1 uses an unprivileged
* port by default.
*/
// query-source address * port 53;

// My Additions
// Forwarder: Anything this DNS can’t resolve gets forwarded to my ISPs DNS.
forwarders { 192.168.56.1; };
// End My Additions
};

zone “hitgon.com.” IN {
type master;
file “hitgon.com.zone”;
allow-update { none; };
};

zone “1.168.192.in-addr.arpa.” IN {
type master;
file “1.168.192.in-addr.arpa”;
allow-update { none; };
};

include “/etc/rndc.key”;
Step-3
======
[root@ocm1 ~]# cat /var/named/chroot/var/named/hitgon.com.zone
$TTL 86400
@ IN SOA localhost root.localhost (
42 ; serial (d. adams)
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum
IN NS localhost
localhost IN A 127.0.0.1
ocm1 IN A 192.168.56.101
ocm2 IN A 192.168.56.102
ocm1-vip IN A 192.168.56.103
ocm2-vip IN A 192.168.56.104
ocm1-priv IN A 10.10.0.1
ocm2-priv IN A 10.10.0.2
ocm-scan IN A 192.168.56.5
ocm-scan IN A 192.168.56.6
ocm-scan IN A 192.168.56.7

Step-4
======
[root@ocm1 ~]# cat /var/named/chroot/var/named/1.168.192.in-addr.arpa
$ORIGIN 1.168.192.in-addr.arpa.
$TTL 1H
@ IN SOA ocm.hitgon.com. root.ocm.hitgon.com. ( 2
3H
1H
1W
1H )
1.168.192.in-addr.arpa. IN NS ocm.hitgon.com.

101 IN PTR ocm1.hitgon.com.
102 IN PTR ocm2.hitgon.com.
103 IN PTR ocm1-vip.hitgon.com.
104 IN PTR ocm2-vip.hitgon.com.
5 IN PTR ocm-scan.hitgon.com.
6 IN PTR ocm-scan.hitgon.com.
7 IN PTR ocm-scan.hitgon.com.

Step-5
======
#Activate and configure the named service that starts when you boot the machine. In a terminal , run as root:

service named start
chkconfig named on
Step-6
======
To finish the DNS , modify the file ” /etc/resolv.conf ” both RAC Nodes . It should look like this:
cat /etc/resolv.conf
hitgon.com search
nameserver 192.168.56.101

Step-7
======
Change the name of the hostname in Node1 .
Edit the file ” / etc / sysconfig / network ” and modify the HOSTNAME variable as follows :

HOSTNAME = ocm1.hitgon.com
In addition to change dynamically execute the following statement:

hostname ocm1.hitgon.com

Change the name of the hostname in Node2 .
Edit the file ” / etc / sysconfig / network ” and modify the HOSTNAME variable as follows :

HOSTNAME = ocm2.hitgon.com
In addition to change dynamically execute the following statement:

hostname ocm2.hitgon.com
Step-8 cat /etc/hosts should be looks line as below in Node1 and Node2
======
[root@ocm1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.56.101 ocm1.hitgon.com ocm1
192.168.56.102 ocm2.hitgon.com ocm2

192.168.56.103 ocm1-vip.hitgon.com ocm1-vip
192.168.56.104 ocm2-vip.hitgon.com ocm2-vip

10.10.0.1 ocm1-priv.hitgon.com ocm1-priv
10.10.0.2 ocm2-priv.hitgon.com ocm2-priv

NOTE-1: The SCAN IPs MUST NOT be in the /etc/hosts file. This will result in only 1 SCAN IP for the entire cluster.

NOTE-2:
If you configured the IP addresses in a DNS server, then, as the root user, change the hosts search order in
/etc/nsswitch.conf on all nodes as shown here:
Old:
hosts: files nis dns
New:
hosts: dns files nis

After modifying the nsswitch.conf file, restart the nscd daemon on each node using the following
command:
# /sbin/service nscd restart

Step-9 Verify with nslookup in all nodes – all node should show configured three IPs
======
[root@ocm1 ~]# nslookup ocm-scan.hitgon.com
Server: 192.168.56.101
Address: 192.168.56.101#53

Name: ocm-scan.hitgon.com
Address: 192.168.56.7
Name: ocm-scan.hitgon.com
Address: 192.168.56.5
Name: ocm-scan.hitgon.com
Address: 192.168.56.6

[root@ocm2 ~]# nslookup ocm-scan.hitgon.com
Server: 192.168.56.101
Address: 192.168.56.101#53

Name: ocm-scan.hitgon.com
Address: 192.168.56.6
Name: ocm-scan.hitgon.com
Address: 192.168.56.7
Name: ocm-scan.hitgon.com
Address: 192.168.56.5

Step-10 Check scan configuration
========
[root@ocm1 ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.CLSTR.dg
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.DATA.dg
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.FRA.dg
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.LISTENER.lsnr
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.asm
ONLINE ONLINE ocm1 Started
ONLINE ONLINE ocm2 Started
ora.gsd
OFFLINE OFFLINE ocm1
OFFLINE OFFLINE ocm2
ora.net1.network
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.ons
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.registry.acfs
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 OFFLINE OFFLINE
ora.cvu
1 ONLINE ONLINE ocm1
ora.oc4j
1 ONLINE ONLINE ocm1
ora.ocm.db
1 ONLINE ONLINE ocm1 Open
2 ONLINE OFFLINE Instance Shutdown
ora.ocm1.vip
1 ONLINE ONLINE ocm1
ora.ocm2.vip
1 ONLINE ONLINE ocm2
ora.scan1.vip
1 ONLINE ONLINE ocm1

[root@ocm1 ~]# /u01/app/11.2.0/grid/bin/srvctl config scan
SCAN name: ocm-scan, Network: 1/192.168.56.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /ocm-scan/192.168.56.105
Step-11 Modify scan configuration
========
[root@ocm1 ~]# /u01/app/11.2.0/grid/bin/srvctl modify scan -n ocm-scan.hitgon.com

[root@ocm1 ~]# /u01/app/11.2.0/grid/bin/srvctl modify scan_listener -u

[root@ocm1 ~]# /u01/app/11.2.0/grid/bin/srvctl config scan
SCAN name: ocm-scan.hitgon.com, Network: 1/192.168.56.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /ocm-scan.hitgon.com/192.168.56.7
SCAN VIP name: scan2, IP: /ocm-scan.hitgon.com/192.168.56.5
SCAN VIP name: scan3, IP: /ocm-scan.hitgon.com/192.168.56.6

[root@ocm1 ~]# /u01/app/11.2.0/grid/bin/srvctl start scan_listener
[root@ocm1 ~]#
[root@ocm1 ~]# /u01/app/11.2.0/grid/bin/srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node ocm2
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node ocm1
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node ocm1
[oracle@ocm1 ~]$ crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.CLSTR.dg
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.DATA.dg
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.FRA.dg
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.LISTENER.lsnr
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.asm
ONLINE ONLINE ocm1 Started
ONLINE ONLINE ocm2 Started
ora.gsd
OFFLINE OFFLINE ocm1
OFFLINE OFFLINE ocm2
ora.net1.network
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.ons
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
ora.registry.acfs
ONLINE ONLINE ocm1
ONLINE ONLINE ocm2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ocm2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE ocm1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE ocm1
ora.cvu
1 ONLINE ONLINE ocm1
ora.oc4j
1 ONLINE ONLINE ocm1
ora.ocm.db
1 ONLINE ONLINE ocm1 Open
2 ONLINE OFFLINE Instance Shutdown
ora.ocm1.vip
1 ONLINE ONLINE ocm1
ora.ocm2.vip
1 ONLINE ONLINE ocm2
ora.scan1.vip
1 ONLINE ONLINE ocm2
ora.scan2.vip
1 ONLINE ONLINE ocm1
ora.scan3.vip
1 ONLINE ONLINE ocm1
Step-12 Change in scan listener initialization parameter – remote_listener
=======
sqlplus / as sysdba
SQL> show parameter listener

NAME TYPE VALUE
———————————— ———– ——————————
listener_networks string
local_listener string (DESCRIPTION=(ADDRESS_LIST=(AD
DRESS=(PROTOCOL=TCP)(HOST=192.
168.56.104)(PORT=1521))))
remote_listener string ocm-scan:1521
SQL> alter system set remote_listener=’ocm-scan.hitgon.com:1521′;

$srvctl stop database -d ocm
$srvctl start database -d ocm

Step-13 Verify the Database Connectivity using scan vip
=======
[oracle@ocm1 ~]$ cat /u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

OCM =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ocm-scan.hitgon.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ocm)
)
)

[oracle@ocm2 ~]$ cat /u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnames.ora
# tnsnames.ora.ocm2 Network Configuration File: /u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnames.ora.ocm2
# Generated by Oracle configuration tools.

OCM =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ocm-scan.hitgon.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ocm)
)
)
[oracle@ocm1 ~]$ tnsping ocm

TNS Ping Utility for Linux: Version 11.2.0.3.0 – Production on 09-JUL-2016 12:59:20

Copyright (c) 1997, 2011, Oracle. All rights reserved.

Used parameter files:
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ocm-scan.hitgon.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ocm)))
OK (0 msec)
[oracle@ocm1 ~]$
[oracle@ocm1 ~]$ sqlplus scott/tiger@ocm

SQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 9 12:59:28 2016

Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> exit

[oracle@ocm2 ~]$ tnsping ocm

TNS Ping Utility for Linux: Version 11.2.0.3.0 – Production on 09-JUL-2016 11:58:17

Copyright (c) 1997, 2011, Oracle. All rights reserved.

Used parameter files:
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ocm-scan.hitgon.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ocm)))
OK (0 msec)
[oracle@ocm2 ~]$ sqlplus scott/tiger@ocm

SQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 9 11:58:24 2016

Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> exit
Reference
=========
http://facedba.blogspot.in/2014/12/configuring-three-ips-for-scan-listener.html
https://oracle-base.com/articles/linux/dns-configuration-for-scan
http://www.freeoraclehelp.com/2011/09/11gr2-rac-scan-dns-bind-configuration.html

RAC to RAC DG – ARCHIVELOG GAP RECOVERY SCENARION

On primary :

SELECT ARCH.THREAD# “Thread”, ARCH.SEQUENCE# “Last Sequence Received”, APPL.SEQUENCE# “Last Sequence Applied”, (ARCH.SEQUENCE# – APPL.SEQUENCE#) “Difference” FROM
(SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
(SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
WHERE
ARCH.THREAD# = APPL.THREAD#
ORDER BY 1;

On Standby :

SELECT ARCH.THREAD# “Thread”, ARCH.SEQUENCE# “Last Sequence Received”, APPL.SEQUENCE# “Last Sequence Applied”, (ARCH.SEQUENCE# – APPL.SEQUENCE#) “Difference” FROM
(SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
(SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
WHERE
ARCH.THREAD# = APPL.THREAD#
ORDER BY 1;

SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM V$ARCHIVE_GAP;

=============================ARCHIVELOG GAP RECOVERY SCENARION====================================================

On Standby :

SELECT TO_CHAR(CURRENT_SCN) FROM v$DATABASE;

—————
24848626949

On Primary :

rman target /

run
{
allocate channel c1 type disk format ‘/SMP_Logs/rman/%U.rmb’;
backup incremental from scn 24848626949 database;
BACKUP CURRENT CONTROLFILE FOR STANDBY FORMAT ‘/SMP_Logs/rman/ForStandbyCTRL.bck’;
}

SCP backup* to standby;

On Standby:

srvctl stop database -d piapdr

startup nomount

rman target /

restore standby controlfile from ‘/u01/rman/ForStandbyCTRL.bck’;

sqlplus / as sysdba >> Both node

ALTER DATABASE MOUNT;

rman target /

catalog start with ‘/u01/rman/’;

catalog start with ‘+ORADATA1’;
catalog start with ‘+ORADATA2’;

report schema;

Note : Check Size column is shouldn’t be contain zero (0) values if yes than follow below steps for each datafiles which is has 0 values.

run
{
set newname for datafile 1 to ‘+ORADATA1/piapdr/datafile/SYSTEM.326.909536701’;
}

run
{
restore datafile 2;
}
run

run

{
set newname for datafile 2 to ‘+ORADATA1/piapdr/datafile/SYSAUX.325.909536707’;
}

run
{
restore datafile 2;
}
run

{
set newname for datafile 3 to ‘+ORADATA1/piapdr/datafile/UNDOTBS1.328.909536651’;
}

run
{
restore datafile 3;
}
run

{
set newname for datafile 4 to ‘+ORADATA1/piapdr/datafile/USERS.324.909536825’;
}

run
{
restore datafile 4;
}

run

{
set newname for datafile 44 to ‘+oradata2/piapdr/datafile/WEBINPWIFI.367.909537767’;
}

run
{
restore datafile 44;
}

ALTER DATABASE OPEN READ ONLY; ===== Both nodes.

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT USING SESSION;