Emulex NIC 환경에서 RX Packet Drop 발생시 조치

RHEL 5, 6.x 계열 Linux 및 Emulex사 NIC 환경에서 Network 지연등의 증상과 함께 ethtool 혹은 netstat 명령등으로 수신 Packet의 Drop 내역 수반된다면 아래와 같은 조치를 통해 증상을 개선할 수 있다.

우선 결론부터 말하자면 Emulex사 NIC를 사용하는 RHEL 5, 6.x 계열 Linux를 사용하는 환경에서,

NIC Parameter 중 TCP 패킷의 최대 분할 전송단위(Jumbo frame)인 MTU 기본값은 1500으로 정의되어 있으나

이를 iSCSI 등의 TCP/IP 기반 IP-SAN등의 성능개선을 위해 MTU 9000 등으로 조정했을 경우,

수신되는 Packet의 사이즈에 비해 허용가능한 Flagment Size 가 작기 때문에 RX Packet에 대한 Drop등 유실이 발생하게 된다.

물론 TCP IP 프로토콜은 Packet Header에 Sequence 한 Packet 번호를 함께 전송토록 하고 있어, Packet의 Drop등 유실이 발생하더라도 유실된 Packet을 재전송토록 요청하고, 이를 재조립연산하는 보완 체계를 가지고 있다.

 

하지만 Network를 기반으로 하는 서비스 입장에서는 TCP Packet의 유실과 재요청, 이를 통한 재조립연산이 잦을 수록 서비스 지연요소로 작용하기 때문에 근본적인 해결이 필요하다.

 

 

1.  OS 버전정보 확인 및 NIC 와 Packet 유실내역 확인

 

[root@TESTDB ~]# [root@TESTDB ~]# lsb_release -a LSB Version:       :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarchDistributor ID:   OracleServer Description:       Oracle Linux Server release 6.4 Release: 6.4 Codename:        n/a [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# cat /proc/net/bonding/bond1 | grep eth Currently Active Slave: eth1 Slave Interface: eth1 Slave Interface: eth3 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# ethtool -S eth1 | grep rx_drops_no_frags      rxq0: rx_drops_no_frags: 92230      rxq1: rx_drops_no_frags: 0      rxq2: rx_drops_no_frags: 0      rxq3: rx_drops_no_frags: 0      rxq4: rx_drops_no_frags: 0 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# ethtool -S eth3 | grep rx_drops_no_frags      rxq0: rx_drops_no_frags: 0      rxq1: rx_drops_no_frags: 0      rxq2: rx_drops_no_frags: 0      rxq3: rx_drops_no_frags: 0      rxq4: rx_drops_no_frags: 0 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# netstat -n -I=eth1 Kernel Interface table Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg eth1       9000   0 4815501439      0  92230     67 4767736176      0      0      0 BMsRU [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# netstat -n -I=eth3 Kernel Interface table Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg eth3       9000   0      586      0    582     69        2      0      0      0 BMsRU [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]#

 

 

2. Server 및 NIC H/W 모델 확인과 함께 NIC Driver 정보 확인

 

 

[root@TESTDB ~]# [root@TESTDB ~]# dmidecode -t system | grep -i produc            Product Name: ProLiant DL980 G7 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# find /sys/devices/ -type l | grep -i eth /sys/devices/virtual/net/bond0/slave_eth0 /sys/devices/virtual/net/bond0/slave_eth2 /sys/devices/virtual/net/bond1/slave_eth1 /sys/devices/virtual/net/bond1/slave_eth3 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0/net/eth4/subsystem /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0/net/eth4/device /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.1/net/eth5/subsystem /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.1/net/eth5/device /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.2/net/eth6/subsystem /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.2/net/eth6/device /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.3/net/eth7/subsystem /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.3/net/eth7/device /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.0/net/eth0/subsystem /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.0/net/eth0/device /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.0/net/eth0/master /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.1/net/eth1/subsystem /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.1/net/eth1/device /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.1/net/eth1/master /sys/devices/pci0000:00/0000:00:07.0/0000:0b:00.0/net/eth2/subsystem /sys/devices/pci0000:00/0000:00:07.0/0000:0b:00.0/net/eth2/device /sys/devices/pci0000:00/0000:00:07.0/0000:0b:00.0/net/eth2/master /sys/devices/pci0000:00/0000:00:07.0/0000:0b:00.1/net/eth3/subsystem /sys/devices/pci0000:00/0000:00:07.0/0000:0b:00.1/net/eth3/device /sys/devices/pci0000:00/0000:00:07.0/0000:0b:00.1/net/eth3/master [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# lspci | grep -i eth 04:00.0 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) 04:00.1 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) 04:00.2 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) 04:00.3 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) 0b:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (rev 02) 0b:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (rev 02) 11:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (rev 02) 11:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (rev 02) [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# lsmod | grep -i be2net be2net                 94523  0 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# modprobe -l | grep be2net kernel/drivers/net/benet/be2net.ko [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# yum provides */be2net.ko Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock Loading support for kernel ABI … 중략 kernel-uek-2.6.39-400.211.1.el6uek.x86_64 : The Linux kernel Repo        : installed Matched from: Filename    : /lib/modules/2.6.39-400.211.1.el6uek.x86_64/kernel/drivers/net/benet/be2net.ko   [root@TESTDB ~]# [root@TESTDB ~]#

 

3. Emulex be2net Driver 이하 rx_frag_size Default 값 확인 및 관련 Config Tunnig 후 System Reboot 처리

 

[root@TESTDB ~]# [root@TESTDB ~]# cat /sys/module/be2net/parameters/rx_frag_size 2048 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# vi /etc/modprobe.d/be2net.conf options be2net rx_frag_size=8192 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]# init 6 [root@TESTDB ~]# [root@TESTDB ~]# [root@TESTDB ~]#

 

 

※ 참고 자료

https://access.redhat.com/solutions/659453

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0106342

http://h17007.www1.hp.com/docs/enterprise/servers/cloudsystem/8.1/webhelp/content/s_TS-hosts-isc.html

 

For RHEL5 this options line can be added to the file /etc/modprobe.conf.
For RHEL6 this options line can be added to a file /etc/modprobe.d/be2net.conf.
For Suse linux 10 and 11 the options line can be added to the file /etc/modprobe.conf.local. 
For Ubuntu Linux 12.xx the options line can be added to a file /etc/modprobe.d/be2net.conf.