Jumbo Frames for Oracle RAC Interconnect

Story of jumbo frame and network tuning !!!!!

Let’s first discuss how RAC traffic works before continuing. Environment for the discussion is: 2 node cluster with 8K database block size,

UDP protocol is used for cache fusion. (BTW, UDP and RDS protocols are supported in UNIX platform;)

UDP protocol, fragmentation, and assembly

UDP Protocol is an higher level protocol stack, and it is implemented over IP Protocol ( UDP/IP). Cache Fusion uses UDP protocol to send packets over the wire.

MTU defines the Maximum Transfer Unit of an IP packet. Let us consider an example of MTU set to 1500 in a network interface. One 8K block transfer cannot be performed with just one IP packet  as the IP packet size (1500 bytes) is less than 8K. So, one transfer of UDP packet of 8K size is fragmented to 6 IP packets and sent over the wire. In the receiving side, those 6 packets are reassembled to create one UDP buffer of size 8K. After the assembly, that UDP buffer is delivered to an UDP port of a UNIX process. Usually, a foreground process will listen on that port to receive the UDP buffer.

Yesterday night we had already enable some port in switch as jumbo frame.

Jumbo Frames for RAC Interconnect

https://blogs.oracle.com/XPSONHA/entry/jumbo_frames_for_rac_interconn_1

https://blogs.oracle.com/glennf/entry/jumbo_frames_with_oracle_rac

 

We can configure the jumbo frame for Oracle RAC Interconnect before go live.

 

Moral of the story: Implement Jumbo Frames for Oracle RAC interconnects… It is a best practice after all 🙂

Bug 13332363

Sorry team… 😦

We cannot go ahead with DB V 11.2.0.3.0 suffering some bug while enabling jumbo frame in Solaris platform

Although we are running under RHEL we cannot take risk so first we have to check in EC Lab and then only we can decide

So jumbo frame is now subject of R&D in EC Lab with some later oracle releases and hope so it working stable … 🙂

 

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:”Times New Roman”;
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:”Times New Roman”;
mso-bidi-theme-font:minor-bidi;}

Oracle MetaLink - How to Modify Private Network Information in Oracle Clusterware

Case III. Changing private network MTU only

For example, private network MTU is changed from 1500 to 9000 (enable jumbo frame),
network interface name and subnet remain the same.

ping 10.80.184.60 -n 10

ping 10.80.184.61 -n 10

1. Shutdown Oracle Clusterware stack on all nodes

Note:- Node-1 & Node-2 one by one

/u01/app/11.2.0/grid/crsctl stop crs

2. Configure the jumbo frame on L3 switch for Oracle RAC Interconnect VLAN port only.

   Action perform by Mr. Pravin

3. Make the required network change of MTU size at OS network layer, ensure private network is available with the desired MTU size,
    ping with the desired MTU size works on all cluster nodes

  Note:- Oracle RAC Interconnect Private VLAN using the eth3 so we will take permananet changes on network level.

 cp /etc/sysconfig/network-scripts/ifcfg-eth3 /etc/sysconfig/network-scripts/ifcfg-eth3.bkup

  Note-(Add MTU="9000" in /etc/sysconfig/network-scripts/ifcfg-eth3 file)

 vi /etc/sysconfig/network-scripts/ifcfg-eth3

 MTU="9000"

  :wq!

service network restart

ping 10.80.184.60 -n 10

ping 10.80.184.61 -n 10

4. Restart Oracle Clusterware stack on all nodes.

Note:- Node-1 & Node-2 one by one

/u01/app/11.2.0/grid/crsctl start crs

Happy Testing !!!!! 🙂

Jumbo Frame is L2 characteristic, so the effectiveness is dependent upon NIC, Switch, and OS Drivers. In summary you cannot vouch up on the lab results for Jumbo Frame.

RAC post deployment require the load testing and monitoring environment for at least a week.

Good luck!!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.