Ethernet Driver 오류 발생시 해당 Device의 정보 확인 과정

Linux 시스템을 운영하다 보면 간혹  Ethernet Card의 Driver오류나 Hardware Fault가 발생할 때가 있다.

이럴때 아래와 같이 해당 Ethernet의 파트 교체나 드라이버 패치등을 위해 Hardware Level의 Device 정보나 PCI Bus 정보등을 파악할 필요가 있다.

 

 

1. /var/log/messages 시스템 로그에 Ethernet Device의 Link Down 및 Driver 오류 발생

 

[root@TEST01 /]# [root@TEST01 /]# cat /var/log/messages . . . 생략 . . . Apr  3 12:38:52 TEST01 kernel: nx_nic[eth2]: Device is DOWN. Fail count[8] Apr  3 12:38:52 TEST01 kernel: nx_nic[eth2]: Firmware hang detected. Severity code=0 Peg number=0 Error code=0 Return address=0 Apr  3 12:38:52 TEST01 kernel: nx_nic[eth2]: Load stored FW Apr  3 12:38:54 TEST01 kernel: nx_nic: File FW[nx3fwct.bin] version[4.0.579:0] Apr  3 12:38:55 TEST01 kernel: nx_nic: Gen2 strapping detected Apr  3 12:38:56 TEST01 kernel: nx_nic HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter Board S/N Chip id 0x1 Apr  3 12:38:57 TEST01 last message repeated 2 times Apr  3 12:38:58 TEST01 kernel: nx_nic[eth4]: NIC Link is up Apr  3 12:39:02 TEST01 kernel: nx_nic[eth2]: Device is UP.

 

 

2. 위에서 확인된 nx_nix 드라이버 모듈의 상태 및 정보 확인 과정

 

[root@TEST01 /]# [root@TEST01 /]# lsmod | grep nx_nic nx_nic                254700  0 [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# modprobe -l | grep nx_nic /lib/modules/2.6.18-194.el5/extra/hp-nx_nic/driver/nx_nic.ko /lib/modules/2.6.18-194.el5/extra/hp-nx_nic/nx_xport.ko [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# uname -r 2.6.18-194.el5 [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# rpm -qa *-release enterprise-release-5-0.0.22 [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# cat /etc/enterprise-release Enterprise Linux Enterprise Linux Server release 5.5 (Carthage) [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# yum provides */nx_nic.ko Loaded plugins: security kmod-hp-nx_nic-4.0.579-2.x86_64 : hp-nx_nic kernel module(s) Repo        : installed Matched from: Filename    : /lib/modules/2.6.18-194.el5/extra/hp-nx_nic/driver/nx_nic.ko [root@TEST01 /]# rpm -qa | grep kmod-hp-nx_nic kmod-hp-nx_nic-4.0.579-2 [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# [root@TEST01 /]# rpm -qi kmod-hp-nx_nic-4.0.579-2 Name        : kmod-hp-nx_nic               Relocations: (not relocatable) Version     : 4.0.579                           Vendor: QLogic, Corporation. Release     : 2                             Build Date: 2012년 02월 25일 (토) 오후 01시 57분 13초 Install Date: 2012년 02월 25일 (토) 오후 03시 04분 07초      Build Host: TEST01 Group       : System Environment/Kernel     Source RPM: hp-nx_nic-4.0.579-2.src.rpm Size        : 4317296                          License: GPL Signature   : (none) Packager    : Hewlett-Packard Company URL         : http://www.qlogic.com Summary     : hp-nx_nic kernel module(s) Description : This package provides the hp-nx_nic kernel modules built for the Linux kernel 2.6.18-194.el5 for the x86_64 family of processors. [root@TEST01 /]# [root@TEST01 /]# ## HP DL시리즈 Ethernet 관련 드라이버 패치 링크 - http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=4038767&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_096edf3739ef43fdba3198ac5e%257CswEnvOID%253D%257CitemLocale%253D%257CswLang%253D%257Cmode%253D5%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

 

 

3. Link Down 및 오류로 확인된 eth2 Ethernet의 사용 유무 확인을 통해 미사용 Ethernet 임을 확인

 

[root@TEST01 network-scripts]# ip addr | grep -w "inet"     inet 127.0.0.1/8 scope host lo     inet 192.168.20.7/23 brd 192.168.21.255 scope global eth8     inet 192.168.200.7/23 brd 192.168.201.255 scope global eth10     inet 192.168.2.7/23 brd 192.168.3.255 scope global eth9     inet 172.30.1.27/23 brd 172.30.1.255 scope global eth11     inet 10.217.27.34/24 brd 10.217.27.255 scope global bond0     inet 10.217.27.32/24 brd 10.217.27.255 scope global secondary bond0:1     inet 172.31.200.25/24 brd 172.31.200.255 scope global bond1     inet 169.254.193.76/16 brd 169.254.255.255 scope global bond1:1 [root@TEST01 network-scripts]# [root@TEST01 /]# [root@TEST01 /]# cat /proc/net/bonding/bond0 | grep -i "interface" Slave Interface: eth1 Slave Interface: eth7 [root@TEST01 /]# [root@TEST01 /]# cat /proc/net/bonding/bond1 | grep -i "interface" Slave Interface: eth13 Slave Interface: eth14 [root@TEST01 /]# [root@TEST01 /]#

 

 

4. 시스템에서 사용중인 Ethernet Device들의 PCI Bus 주소 정보와 맵핑된 Ethernet Name 확인

 

[root@TEST01 devices]# [root@TEST01 devices]# find /sys/devices/ -type l | grep eth /sys/devices/pci0000:a0/0000:a0:09.0/0000:a1:00.1/net:eth15 /sys/devices/pci0000:a0/0000:a0:09.0/0000:a1:00.0/net:eth14 /sys/devices/pci0000:a0/0000:a0:07.0/0000:a4:00.1/net:eth12 /sys/devices/pci0000:a0/0000:a0:07.0/0000:a4:00.0/net:eth13 /sys/devices/pci0000:50/0000:50:00.0/0000:51:00.0/0000:52:04.0/0000:54:00.1/net:eth11 /sys/devices/pci0000:50/0000:50:00.0/0000:51:00.0/0000:52:04.0/0000:54:00.0/net:eth9 /sys/devices/pci0000:50/0000:50:00.0/0000:51:00.0/0000:52:02.0/0000:53:00.1/net:eth10 /sys/devices/pci0000:50/0000:50:00.0/0000:51:00.0/0000:52:02.0/0000:53:00.0/net:eth8 /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.1/net:eth7 /sys/devices/pci0000:00/0000:00:05.0/0000:11:00.0/net:eth6 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.3/net:eth5 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.2/net:eth4 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.1/net:eth3 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0/net:eth2 /sys/devices/pci0000:00/0000:00:01.0/0000:0e:00.1/net:eth1 /sys/devices/pci0000:00/0000:00:01.0/0000:0e:00.0/net:eth0 [root@TEST01 devices]# [root@TEST01 devices]# [root@TEST01 devices]# [root@TEST01 devices]# find /sys/devices/ -type l | grep "eth" | grep "04:00" /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.3/net:eth5 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.2/net:eth4 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.1/net:eth3 /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0/net:eth2 [root@TEST01 devices]# [root@TEST01 devices]#

 

 

5. 상기 4번에서 확인된 PCI Bus 주소정보를 기준으로 해당 Device 정보와 Mac address 정보 확인

 

[root@TEST01 network-scripts]# [root@TEST01 network-scripts]# lspci | grep -i "eth" | grep -i "^04:00" 04:00.0 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) 04:00.1 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) 04:00.2 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) 04:00.3 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42) [root@TEST01 network-scripts]# [root@TEST01 network-scripts]# [root@TEST01 ~]# [root@TEST01 ~]# for LIST in `lspci | grep -i "ethernet" | cut -d "." -f 1 | uniq`; do dmidecode | grep ${LIST} -B7 && echo "================"; done Handle 0xCA7F, DMI type 202, 9 bytes OEM-specific Type  Header and Data:   CA 09 7F CA 7F 11 10 08 FF Handle 0xD100, DMI type 209, 36 bytes HP BIOS NIC PCI and MAC Information  NIC 1: PCI device 04:00.0, MAC address 3C:4A:92:F9:3B:34  NIC 2: PCI device 04:00.1, MAC address 3C:4A:92:F9:3B:35  NIC 3: PCI device 04:00.2, MAC address 3C:4A:92:F9:3B:36  NIC 4: PCI device 04:00.3, MAC address 3C:4A:92:F9:3B:37 ================  Designation: PCI-E Slot10  Type: x4 PCI Express Gen 2 x8  Current Usage: In Use  Length: Long  Characteristics:   3.3 V is provided   PME signal is supported  Bus Address: 0000:0e:00.0 ================  Designation: PCI-E Slot11  Type: x8 PCI Express Gen 2 x16  Current Usage: In Use  Length: Long  Characteristics:   3.3 V is provided   PME signal is supported  Bus Address: 0000:11:00.0 ================  Designation: PCI-E Slot 12  Type: x8 PCI Express Gen 2 x16  Current Usage: In Use  Length: Short  Characteristics:   3.3 V is provided   PME signal is supported  Bus Address: 0000:a1:00.0 ================  Designation: PCI-E Slot 13  Type: x8 PCI Express Gen 2 x16  Current Usage: In Use  Length: Short  Characteristics:   3.3 V is provided   PME signal is supported  Bus Address: 0000:a4:00.0 ================ [root@TEST01 ~]#

 

 

6. 해당 서버의 하드웨어 Product 정보와 해당 Ethernet Device에 종속된 Ethernet Channel 의 Link 상태 확인

 

[root@TEST01 devices]# [root@TEST01 devices]# dmidecode | egrep 'Product Name|Vendor'  Vendor: HP  Product Name: ProLiant DL980 G7 [root@TEST01 devices]# [root@TEST01 devices]# [root@TEST01 /]# [root@TEST01 /]# for LIST in `find /sys/devices/ -type l | grep "eth" | grep "04:00"`; do echo "`echo ${LIST} | cut -d "/" -f 7` - `cat ${LIST}/address` - `cat ${LIST}/operstate`"; done net:eth5 - 3c:4a:92:f9:3b:37 - down net:eth4 - 3c:4a:92:f9:3b:36 - up net:eth3 - 3c:4a:92:f9:3b:35 - down net:eth2 - 3c:4a:92:f9:3b:34 - down [root@TEST01 /]# [root@TEST01 /]#