Linux Installation and Configuration of Intel® Ethernet FCoE

SUSE* Linux Enterprise Server 11 FCoE Installation

Red Hat* Enterprise Linux 6 FCoE Installation

Linux FCoE Configuration

SUSE* Linux Enterprise Server 11 (SLES 11) FCoE Installation

Select FCoE during OS Installation

In Installation Settings, click the Software headline to go to Software Selection and System Tasks.
In Software Selection and System Tasks, click Details to enter a new window.
Enter Search tab
In Search box, type "fcoe" and click "Search"
Select open-fcoe package and click "Accept"
Click OK for questions to get back to "Installation Settings"

Using Yast to install after OS Installation

NOTE: If FCoE is installed during OS installation, then this is not required.

Select Software in the left pane
Select Software Management in the right pane
Type "fcoe" in the search box, then click Search button
"open-fcoe" package will show in right pane
Select "open-fcoe" package, then click Accept
Verify "Automatic Changes" and click OK
Click Install to install package

Red Hat* Enterprise Linux 6 (RHEL 6) FCoE Installation

Select FCoE during OS Installation

Select your desired installation profile (Basic Server, Database Server, Web Server, etc).
Set up your repositories as desired, and select the Customize now button before clicking Next to go to the software package selection screen.
Under Base System, select FCoE Storage Client. Make sure that the optional package Intel LLDP Agent is included under Optional packages.
Continue with OS installation.

Using YUM to install after OS Installation

NOTE: If FCoE is installed during OS installation, then these steps are not required.

Either configure the Red Hat Network (RHN) or mount the installation DVD. Please refer to the Red Hat documentation to learn how to register your repositories with YUM.
root# yum -y install fcoe*
1. You will notice several dependency packages being automatically installed.

Linux Intel® Ethernet FCoE Configuration

NOTE: The following examples assume that the network interface named "eth3" is the FCoE interface. Your interface name is likely to be different. Please substitute your interface name when "eth3" is used below.

Verifying the Network Interface

After the FCoE package has been installed, activate the interface. Then ensure that link is established and active at 10G link speed to the network switch.

root# ifconfig eth3 up

root# ethtool eth3

Settings for eth3:

Speed: 10000Mb/s
Link detected: yes

NOTE: If you do not see link detected, check cabling and correct before proceeding.

Configuring Link Layer Discovery Protocol agent daemon (lldpad)

lldpad is a utility used to initiate and configure the Link Layer Discovery Protocol (LLDP). LLDP enables the IEEE Data Center Bridging extensions (DCBX) so that FCoE VLAN discovery, parameter negotiations and session initiation can be accomplished.

Start lldpad service and configure to start at boot time.

root# service lldpad start

Starting lldpad: [done] [ OK ]

root# chkconfig lldpad on

NOTE: There is no output from this command but it will enable lldpad to automatically start when the system is booted.

Adjust DCB settings on every physical interface to be used for FCoE with these commands:

NOTE: dcbtool makes change to /var/lib/lldpad/lldpad.conf.

root# dcbtool sc eth3 dcb on

root# dcbtool sc eth3 dcb on
Version: 2
Command: Set Config
Feature: DCB State
Port: eth3
Status: Successful

root# dcbtool sc eth3 app:fcoe e:1

root# dcbtool sc eth3 app:fcoe e:1
Version: 2
Command: Set Config
Feature: Application FCoE
Port: eth3
Status: Successful

root# dcbtool sc eth3 pfc e:1 a:1 w:1

(this optional command ensures pfc is configured in the default way (not necessary to do on a fresh system)

root# dcbtool sc eth3 pfc e:1 a:1 w:1
Version: 2
Command: Set Config
Feature: Priority Flow Control
Port: eth3
Status: Successful

root# dcbtool go eth3 pfc

(this optional command is used to verify that the settings are correct)

root# dcbtool go eth3 pfc
Version: 2
Command: Get Oper
Feature: Priority Flow Control
Port: eth3
Status: Successful
Oper Version: 0
Max Version: 0
Errors: 0x00 - none
Oper Mode: true
Syncd: true
pfcup: 0 0 0 1 0 0 0 0

root# dcbtool go eth3 app:fcoe

(this optional command is used to verify that the settings are correct)

root# dcbtool go eth3 app:fcoe
Version: 2
Command: Get Oper
Feature: Application FCoE
Port: eth3
Status: Successful
Oper Version: 0
Max Version: 0
Errors: 0x00 - none
Oper Mode: true
Syncd: true
appcfg: 08

Configuring Intel® Ethernet FCoE

Create a configuration file for the VLAN interface. To copy cfg file, run command:

NOTE: The sample file is cfg-ethx. Copy it to a file which reflects your FCoE network interface name (eg. cfg-eth3).

root# cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-eth3

Start fcoe service and configure to start at boot time.

root# service fcoe start

Starting FCoE initiator service: [ OK ]

root# service fcoe status

Created interfaces: eth3.100-fcoe

NOTE: In the example above, an FCoE interface was created for eth3 using VLAN 100. "No interfaces added" indicates that the FIPVLAN protocol was not able to acquire a vlan and therefore an interface was not created. This must be corrected before continuing. This issue may be because you are not connected to a DCB enabled switch or DCB is disabled on this switch port. Contact your network administrator for further help.

root# chkconfig fcoe on

NOTE: There is no output from this command but it will enable FCoE to automatically start when the system is booted.

Verify that a vlan interface has been created.

root# ifconfig -a

eth3 Link encap:Ethernet HWaddr 00:1B:21:60:62:75
...
eth3.100-fcoe Link encap:Ethernet HWaddr 00:1B:21:60:62:75

...

NOTE: The interface name "eth3.100-fcoe" indicates that the FCoE Initialization Protocol (FIP) negotiated with the switch and established an FCoE session using VLAN 100. This process usually happens automatically for all DCBX enabled switches.

No FCoE ports configured

NOTE: This message indicates that the FIP VLAN Discovery failed. Please see your network administrator and ensure that this port is configured for DCBX and FCoE.

Check that vlan interface can obtain FC-ID. Run command

root# fcoeadm -i

Description: 82599EB 10-Gigabit Network Connection
Revision: 01
Manufacturer: Intel Corporation
Serial Number: 001B21606274
Driver: ixgbe 2.0.62-k2
Number of Ports: 2

Symbolic Name: fcoe v0.1 over eth3.100-fcoe
OS Device Name: host3
Node Name: 0x1000001B21606277
Port Name: 0x2000001B21606277
FabricName: 0x2064000DECA33A01
Speed: 10 Gbit
Supported Speed: 10 Gbit
MaxFrameSize: 2112
FC-ID (Port ID): 0x4A0064
State: Online

NOTE: The important values to note are that the state is "Online", the Port Name and the FC-ID. Your storage administrator may need the Port Name and the FC-ID in order to identify your connection when allocating zones for you.

If you see the FCoE interface and an FCID from the switch, you are ready to zone and provision your storage. Please see your storage administrator. Once your storage administrator has properly provisioned your storage, you may continue to configure.

Verify that you have been assigned on a storage target.

root# fcoeadm -t
Interface: eth3.100-fcoe
...
FC-ID (Port ID): 0x4A06EF
State: Online

LUN ID Device Name Capacity Block Size Description
------ ----------- -------- ---------- -----------
0 /dev/sdb 25.00 GB 512 DGC RAID 10 (rev 0429)

NOTE: This example shows that a LUN has been provisioned and assigned to /dev/sdb. This may happen automatically but usually you must request a LUN from your storage administrator.

Verify that the file system has identified the SAN storage.

root# lsscsi
[0:0:0:0] cd/dvd TSSTcorp CDDVDW SH-S223L SB02 /dev/sr1
[1:0:0:0] disk ATA ST3160811AS 3.AA /dev/sda
[2:0:1:0] cd/dvd MATSHITA DVD-ROM SR-8178 PZ16 /dev/sr0
[3:0:0:0] disk DGC RAID 10 0429 /dev/sdb

Basic Intel® Ethernet FCoE Commands

show interface info: fcoeadm -i

root# fcoeadm -i
Description: 82599EB 10-Gigabit Network Connection
Revision: 01
Manufacturer: Intel Corporation
Serial Number: 001B21606274
Driver: ixgbe 2.0.62-k2
Number of Ports: 2

Symbolic Name: fcoe v0.1 over eth3.100-fcoe
OS Device Name: host3
Node Name: 0x1000001B21606277
Port Name: 0x2000001B21606277
FabricName: 0x2064000DECA33A01
Speed: 10 Gbit
Supported Speed: 10 Gbit
MaxFrameSize: 2112
FC-ID (Port ID): 0x4A0064
State: Online

show LUNs: fcoeadm -l

root# fcoeadm -l
Interface: eth3.100-fcoe
Roles: FCP Target
Node Name: 0x50060160C46029CC
Port Name: 0x50060160446029CC
Target ID: 0
MaxFrameSize: 2048
OS Device Name: rport-3:0-1
FC-ID (Port ID): 0x4A06EF
State: Online

LUN #0 Information:
OS Device Name: /dev/sdb
Description: DGC RAID 10 (rev 0429)
Ethernet Port FCID: 0x4A0064
Target FCID: 0x4A06EF
Target ID: 0
LUN ID: 0
Capacity: 25.00 GB
Capacity in Blocks: 52428799
Block Size: 512 bytes
Status: Attached

show targets: fcoeadm -t

root# fcoeadm -t
Interface: eth3.100-fcoe
Roles: FCP Target
Node Name: 0x50060160C46029CC
Port Name: 0x50060160446029CC
Target ID: 0
MaxFrameSize: 2048
OS Device Name: rport-3:0-1
FC-ID (Port ID): 0x4A06EF
State: Online

LUN ID Device Name Capacity Block Size Description
------ ----------- -------- ---------- -----------
0 /dev/sdb 25.00 GB 512 DGC RAID 10 (rev 0429)

show statistics: fcoeadm -s

root# fcoeadm -s eth3.100-fcoe 1

eth3.100-fcoe interval: 1                                               Err   Inv     IvTx Link Cntl Input Input Output Output
Seconds TxFrames TxBytes RxFrames RxBytes Frms CRC Byte Fail Reqs Requests MBytes Requests MBytes
--------   ---------- -------- ---------- -------- ----- ----   ---- ---- ----   --------- -------- --------- --------
0            102          5424      103           6444       0       0     0      0     6       79           526      0        0
1            102          5424      103           6444       0       0     0      0     6       79           526      0        0
2            102          5424      103           6444       0       0     0      0     6       79           526      0        0