Story of jumbo frame and network tuning !!!!!
Let’s first discuss how RAC traffic works before continuing. Environment for the discussion is: 2 node cluster with 8K database block size,
UDP protocol is used for cache fusion. (BTW, UDP and RDS protocols are supported in UNIX platform;)
UDP protocol, fragmentation, and assembly
UDP Protocol is an higher level protocol stack, and it is implemented over IP Protocol ( UDP/IP). Cache Fusion uses UDP protocol to send packets over the wire.
MTU defines the Maximum Transfer Unit of an IP packet. Let us consider an example of MTU set to 1500 in a network interface. One 8K block transfer cannot be performed with just one IP packet as the IP packet size (1500 bytes) is less than 8K. So, one transfer of UDP packet of 8K size is fragmented to 6 IP packets and sent over the wire. In the receiving side, those 6 packets are reassembled to create one UDP buffer of size 8K. After the assembly, that UDP buffer is delivered to an UDP port of a UNIX process. Usually, a foreground process will listen on that port to receive the UDP buffer.
Yesterday night we had already enable some port in switch as jumbo frame.
Jumbo Frames for RAC Interconnect
We can configure the jumbo frame for Oracle RAC Interconnect before go live.
Moral of the story: Implement Jumbo Frames for Oracle RAC interconnects… It is a best practice after all 🙂
Sorry team… 😦
We cannot go ahead with DB V 220.127.116.11.0 suffering some bug while enabling jumbo frame in Solaris platform
Although we are running under RHEL we cannot take risk so first we have to check in EC Lab and then only we can decide
So jumbo frame is now subject of R&D in EC Lab with some later oracle releases and hope so it working stable … 🙂
/* Style Definitions */
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-fareast-font-family:”Times New Roman”;
mso-bidi-font-family:”Times New Roman”;
Oracle MetaLink - How to Modify Private Network Information in Oracle Clusterware Case III. Changing private network MTU only For example, private network MTU is changed from 1500 to 9000 (enable jumbo frame), network interface name and subnet remain the same. ping 10.80.184.60 -n 10 ping 10.80.184.61 -n 10 1. Shutdown Oracle Clusterware stack on all nodes Note:- Node-1 & Node-2 one by one /u01/app/11.2.0/grid/crsctl stop crs 2. Configure the jumbo frame on L3 switch for Oracle RAC Interconnect VLAN port only. Action perform by Mr. Pravin 3. Make the required network change of MTU size at OS network layer, ensure private network is available with the desired MTU size, ping with the desired MTU size works on all cluster nodes Note:- Oracle RAC Interconnect Private VLAN using the eth3 so we will take permananet changes on network level. cp /etc/sysconfig/network-scripts/ifcfg-eth3 /etc/sysconfig/network-scripts/ifcfg-eth3.bkup Note-(Add MTU="9000" in /etc/sysconfig/network-scripts/ifcfg-eth3 file) vi /etc/sysconfig/network-scripts/ifcfg-eth3 MTU="9000" :wq! service network restart ping 10.80.184.60 -n 10 ping 10.80.184.61 -n 10 4. Restart Oracle Clusterware stack on all nodes. Note:- Node-1 & Node-2 one by one /u01/app/11.2.0/grid/crsctl start crs
Happy Testing !!!!! 🙂
Jumbo Frame is L2 characteristic, so the effectiveness is dependent upon NIC, Switch, and OS Drivers. In summary you cannot vouch up on the lab results for Jumbo Frame.
RAC post deployment require the load testing and monitoring environment for at least a week.