C H A P T E R 4

Network Configuration

This chapter describes how to manually configure the network interface and attributes, including optional methods to automate network configuration.

This chapter contains the following topics:

- Section 4.1, Configuring Network Host Files

- Section 4.2, Configuring Jumbo Frames

- Section 4.3, Configuring Link Aggregation

- Section 4.4, Configuring VLANs

##

4.1 Configuring Network Host Files

After installing the driver software as described in Chapter 3, you can create a `hostname.nxge` file to configure each of the Netra CP32x0 ARTM-10G interfaces.

You must designate both an IP address and a host name for each Netra CP32x0 ARTM-10G Ethernet interface in the `/etc/hosts` file.

Adhere to the following guidelines when assigning host names:

- The `/etc/hostname.nxge` file must contain the host name for the appropriate `nxge` interface.

- The host name must have an IP address listed in the `/etc/hosts` file.

- The host name must be different from any other host name of any other interface, for example: `/etc/hostname.nxge0` and `/etc/hostname.nxge1` cannot share the same host name.

The following example shows the `/etc/hostname.nxge` file required for a system called zardoz that has an `nxge` driver (zardoz-11).

    # cat /etc/hostname.nxge0
    zardoz
    # cat /etc/hostname.nxge1
    zardoz-11

Note - To preserve `nxge` instance numbers for interfaces between reboots, only one product that uses the `nxge` driver can be used in a given slot. For example, numbers can change if between a series of reboots, a Netra CP32x0 ARTM-10G in one slot is removed and replaced by a different `nxge` driver-based network interface product, and later a Netra CP32x0 ARTM-10G reinstalled into that same slot.

### To Manually Configure Network Host Files

This method enables you to manually manipulate a network interface via the command line interface. Note that any changes made using this method will be lost when the system reboots.

1. At the command line, search the `/etc/path_to_inst` file for `nxge` interfaces.

The ARTM reports two `nxge` interfaces, and for each, includes three descriptive parameters:

- Instance path name in quotes

- Instance Number (integer, no quotes)

- Instance driver name in quotes

    # grep nxge /etc/path_to_inst
    # "/pci@7c0/pci@0/pci@9/network@0" 0 "nxge"
    # "/pci@7c0/pci@0/pci@9/network@0,1" 1 "nxge"

2. Write down the device path and instance, which in the example is `/pci@7c0/pci@0/pci@9/network@0` `0`.

Your Netra CP32x0 ARTM-10G device path and instance will be similar. You need this information later to make changes to the `nxge.conf` file.

3. Set up the Netra CP32x0 ARTM-10G’s `nxge` interface.

4. Use the `ifconfig` command to assign an IP address to the network interface.

Type the following at the command line, replacing ip-address with the Netra CP32x0 ARTM-10Gs IP address:

    # ifconfig nxge0 plumb ip-address up

Refer to the `ifconfig`(1M) man page and the Solaris documentation for more information.

5. Create an appropriate entry in the `/etc/hosts` file for each active `nxge` interface.

Follow the guidelines for assigning IP addresses and host names, described at the beginning of this Section 4.1, Configuring Network Host Files.

    # cat /etc/hosts
    #
    # Internet host table
    #
    127.0.0.1 localhost
    129.144.10.57 zardoz loghost
    129.144.11.83 zardoz-11

### To Automatically Configure Network Host Files

This method enables you to create a file that provides a persistent setup, which is automatically loaded at each system reboot.

1. Create an `/etc/hostname.nxge` x file.

Where x is the instance number of the `nxge` interface you plan to use. For example, to auto-configure a Netra CP32x0 ARTM-10Gs `nxge` interface with instance 0, create an `/etc/hostname.nxge0` file, where 0 is the number of the `nxge` interface. If the instance number were 1, the file name would be `/etc/hostname.nxge1`.

Note - Do not create an `/etc/hostname.nxge` file for any Netra CP32x0 ARTM-10Gs you plan to leave unused.

2. Create an appropriate entry in the `/etc/hosts` file for each active `nxge` interface.

Follow the guidelines for assigning IP addresses and host names, described at the beginning of Section 4.1, Configuring Network Host Files.

    # cat /etc/hosts
    #
    # Internet host table
    #
    127.0.0.1 localhost
    129.144.10.57 zardoz loghost
    129.144.11.83 zardoz-11

##

4.2 Configuring Jumbo Frames

Configuring Jumbo Frames enables Ethernet interfaces to send and receive packets larger than the standard 1500 bytes. However, the actual transfer size depends on the switch capability and the device driver capability.

Note - Refer to the documentation that came with your switch for commands to configure Jumbo Frames.

### 4.2.1 Displaying Layer Configuration in a Solaris Environment

The Jumbo Frames checking occurs at Layer 2 or Layer 3, depending on the configuration method.

To Display Layer 2 Configuration

View the maximum transmission unit (MTU) configuration of a `nxge` instance at any time with the `kstat` command:

    # kstat nxge:0 | grep mac_mtu

The `kstat` `mac_mtu` variable represents the complete size of the Ethernet frame, which includes the Ethernet header, maximum payload, and CRC. The value of this variable should be equal to or less than the MTU configured on the switch.

To Display Layer 3 Configuration

View the Layer 3 configuration by using the `dladm` command with the `show-link` option.

    # dladm show-link
    e1000g0 type: non-vlan mtu: 1500 device: e1000g0
    e1000g1 type: non-vlan mtu: 1500 device: e1000g1
    e1000g2 type: non-vlan mtu: 1500 device: e1000g2
    e1000g3 type: non-vlan mtu: 1500 device: e1000g3
    nxge0 type: non-vlan mtu: 9194 device: nxge0
    nxge1 type: non-vlan mtu: 9194 device: nxge1
    nxge38001 type: vlan 38 mtu: 9194 device: nxge1

### 4.2.2 Displaying Driver Statistics

This section describes how to display all the statistics that the driver tracks for an interface.

To Show Driver Statistics in a Solaris Environment

1. Use the `kstat` command to display driver statistics.

The output can be further filtered to display specific data of interest. The following example displays the receive packet counts on all eight receive DMA channels on `nxgeinterface` `1`.

    # kstat nxge:1 |grep rdc_packets
    rdc_packets 798982054
    rdc_packets 792546171
    rdc_packets 803941759
    rdc_packets 805674872
    rdc_packets 798714912
    rdc_packets 799293256
    rdc_packets 806470537
    rdc_packets 805413540

2. Use the `kstat` command to display driver statistics of a VLAN interface.

    # kstat nxge:38001
    module: nxge instance: 38001
    name: nxge38001 class: net
    brdcstrcv 0
    brdcstxmt 0
    collisions 0
    crtime 3842.493000352
    ierrors 0
    ifspeed 10000000000
    ipackets 2116069805
    ipackets64 6411037101
    multircv 0
    multixmt 0
    norcvbuf 0
    noxmtbuf 0
    obytes 2757388874
    obytes64 23380264381002
    oerrors 0
    opackets 37606022
    opackets64 4332573318
    rbytes 2937141290
    rbytes64 47178857920554

To Show Driver Statistics in a Linux Environment

1. Use the `ifconfig` utility to display driver statistics.

    # ifconfig eth6
    eth6 Link encap:Ethernet HWaddr 00:14:4F:83:9E:1A
    inet addr:192.168.11.189 Bcast:192.168.11.255
    Mask:255.255.255.0
    inet6 addr: fe80::214:4fff:fe83:9e1a/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:3 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:256 (256.0 b) TX bytes:0 (0.0 b)

In the previous example output, MTU indicates the complete size of the Ethernet frame, which includes the Ethernet header, maximum payload, and CRC. The value of MTU should be equal to or less than the MTU configured on the switch.

2. If you want to see only MTU, use the following command:

    # ifconfig eth6 | grep MTU
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

3. Use the `ethtool` utility with the `-S` option to get more detailed information.

This option displays a large amount of statistics maintained by the driver. Output can be further filtered to display specific data of interest using `grep`.

    # ethtool -S eth9 |grep rx_pac
    rx_packets: 748274
    rx_packets: 828
    rx_packets: 112
    rx_packets: 189088
    rx_packets: 134
    rx_packets: 196085
    rx_packets: 177884
    rx_packets: 93
    rx_packets: 184050

### 4.2.3 Enabling Jumbo Frames

Enable Jumbo Frames if you want to configure Ethernet interfaces to send and receive packets larger than the standard 1500 bytes.

#### 4.2.3.1 Enabling Jumbo Frames in a Solaris Environment

This section describes how to enable Jumbo Frames in both a SPARC and an x86 environment.

To Enable Jumbo Frames in a Solaris Environment

1. Enable Jumbo Frames for a port using the `nxge.conf` file.

    name = "pciex108e,abcd" parent = "/pci@780/pci@0/pci@8/network@0"
    unit-address = "0"
    accept-jumbo=1;

2. Reboot the system.

    # reboot -- -r

3. Set the maximum transmission unit (MTU) for maximum performance.

    # ifconfig nxge mtu 9194

#### 4.2.3.2 Enabling Jumbo Frames in a Linux Environment

This section describes how to enable Jumbo Frames in Linux environment.

To Enable Jumbo Frames in a Linux Environment

1. Ensure that the `nxge` software is installed.

    # modprobe nxge

2. Plumb the Netra CP32x0 ARTM-10G 10GbE SFP+ interface.

The xxx.xxx.xx.xxx represents the IP address of the interface.

    # ifconfig eth2 xxx.xxx.xx.xxx up

3. Set the maximum transmission unit (MTU) to specify jumbo frames size.

Where x designates the instance number of the interface.

    # ifconfig ethnumber mtu 9194

##

4.3 Configuring Link Aggregation

Link aggregation enables one or more network links to be aggregated together to form a link-aggregation group. This link-aggregation group appears to MAC clients as a regular link. Link aggregation is defined by IEEE 802.3ad and provides the following benefits:

- Increased bandwidth

- Linearly incremental bandwidth

- Load sharing

- Automatic configuration

- Rapid configuration and reconfiguration

- Deterministic behavior

- Low risk of duplication or incorrect ordering

- Support of existing IEEE 802.3 MAC clients

### 4.3.1 Configuring Link Aggregation in a Solaris Environment

This section explains how to configure link aggregation.

To Configure Link Aggregation in a Solaris Environment

1. Aggregate `nxge0` and `nxge1` to form an aggregation and use a random number as key.

2. Unplumb the interfaces to be aggregated:

    # ifconfig down unplumb nxge0
    # ifconfig down unplumb nxge1

3. Create a link aggregation group with a random number as a key, without specifying mode.

This example uses 33.

    # dladm create-aggr -d nxge0 -d nxge1 33

When the command returns, one line appears in the `/etc/aggregation.conf` file and indicates that the default mode is off. For example:

    # tail -1 /etc/aggregation.conf
    33 L4 2 nxge0/0,nxge1/0 auto off short
    # dladm show-aggr
    key: 33 (0x0021) policy: L4 address: 0:3:ba:d8:9d:e8 (auto)
    device address speed duplex link state
    nxge0 0:3:ba:d8:9d:e8 10000 Mbps full up standby
    nxge1 0:3:ba:d8:9d:e9 10000 Mbps full up standby
    # dladm show-link aggr33
    aggr33 type: non-vlan mtu: 1500 aggregation: key 33

4. Plumb up the interface `aggrkey`, which is `aggr33` in the following example:

    # ifconfig aggr33 plumb
    # ifconfig aggr33
    aggr33: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 8
    inet 0.0.0.0 netmask 0
    ether 0:3:ba:d8:9d:e8
    # ifconfig aggr33 192.168.1.1/24 broadcast + up
    # ifconfig aggr33
    aggr33: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 8
    inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
    ether 0:3:ba:d8:9d:e8

5. Show link aggregation status again.

The state now is attached:

    # dladm show-aggr
    key: 33 (0x0021) policy: L4 address: 0:3:ba:d8:9d:e8 (auto)
    device address speed duplex link state
    nxge0 0:3:ba:d8:9d:e8 10000 Mbps full up attached
    nxge1 0:3:ba:d8:9d:e9 10000 Mbps full up attached

6. Use the `dladm` `show-aggr` `-s` command to display statistics:

    # dladm show-aggr -s
    key: 33 ipackets rbytes opackets obytes %ipkts %opkts
    Total 0 0 16 1182
    nxge0 0 0 16 1182 - 100.0
    nxge1 0 0 0 0 - 0.0

7. Use the`dladm` `show-aggr` `-L` command to display LACP specific information:

    # dladm show-aggr -L
    key: 33 (0x0021) policy: L4 address: 0:3:ba:d8:9d:e8 (auto)
    LACP mode: off LACP timer: short
    device activity timeout aggregatable sync coll dist defaulted expired
    nxge0 passive short yes no no no no no
    nxge1 passive short yes no no no no no

For more information, refer to the man pages for the `dladm` command.

### 4.3.2 Configuring Link Aggregation in a Linux Environment

Many administrators familiar with Linux refer to Link Aggregation as “NIC Bonding.” Both refer to the IEEE802.3ad standard that defines Dynamic link aggregation.

To Configure Link Aggregation in a Linux Environment

1. Modify the `/etc/modprobe.conf` file for the 2.6 kernels file to add the following lines:

    alias bond0 bonding
    options bonding max_bonds=2 mode=4 miimon=1000

Where:

- bond0 is the bonding device.

- `max_bonds` is the number of bond interfaces to be created.

- `mode` specifies the bonding policies.

- `miimon` is the frequency in milliseconds that MII link monitoring occurs.

Refer to Linux documentation for more information.

2. Load the bonding driver:

    # modprobe bonding

3. Configure the `bond0` interface.

In this example, `bond0` is the master of the two interfaces `eth4` and `eth5`.

    # ip addr add 192.12.38.64/24 brd + dev bond0
    # ip link set dev bond0 up
    # ifenslave bond0 eth4 eth5

##

4.4 Configuring VLANs

VLANs enable you to split your physical LAN into logical subparts, providing an essential tool for increasing the efficiency and flexibility of your network.

VLANs are commonly used to separate groups of network users into manageable broadcast domains, to create logical segmentation of workgroups, and to enforce security policies among each logical segment.

Each defined VLAN behaves as its own separate network. The traffic and broadcasts of each VLAN are isolated from the others, increasing the bandwidth efficiency within each logical group.

Although VLANs are commonly used to create individual broadcast domains or separate IP subnets, a server can have a presence on more than one VLAN simultaneously.

The Netra CP32x0 ARTM-10G supports multiple VLANs on a per port or per interface basis, allowing very flexible network configurations. With multiple VLANs on an ARTM, a single physical port can have a logical presence on multiple IP subnets.

By default, 128 VLANs can be defined for each VLAN-aware ARTM port in the chassis. However, you can increase this number by changing the system parameters. If your network does not require multiple VLANs, you can use the default configuration, in which case no further configuration is necessary.

| Caution - If you change any of the VLAN configuration parameters, you must reboot the system before the changes take effect. If you make changes and do not reboot, your system might experience configuration problems.

VLANs can be created according to various criteria, but each VLAN must be assigned a VLAN tag or VLAN ID (VID). The VID is a 12-bit identifier between 1 and 4094 that identifies a unique VLAN. For each network interface (`nxge0` and `nxge1`), 4094 possible VLAN IDs can be selected per port for up to 4 ports.

Tagging an Ethernet frame requires adding a tag header to the frame. Insert the header immediately following the destination MAC address and the source MAC address. The tag header consists of two bytes of Ethernet Tag Protocol identifier (TPID, 0x8100) and two bytes of Tag Control Information (TCI). TABLE 4-1 shows the Ethernet tag header format.

TABLE 4-1 Ethernet Tag Header Format
TAG Header Description | Octet
TPID, high byte (value = 0x81) | 1
TPID, low byte (value = 0x00) | 2
Priority (3 bits) | CFI (1 bit) | VLAN ID (4bits) | 3
VLAN ID (8 bits) | 4

By default, a single VLAN is configured for every port. This configuration groups all ports into the same broadcast domain, just as if there were no VLANs at all. VLAN tagging for the switch port is turned off.

Note - If you configure a VLAN virtual device for an ARTM, all traffic sent or received by that ARTM must be in VLAN-tagged format.

### 4.4.1 Configuring VLANs in a Solaris Environment

This section describes how to configure VLANs in a Solaris environment.

To Configure VLANs in a Solaris Environment

1. Create one `/etc/hostname.nxge` x file for each VLAN that will be configured for the installed ARTM.

Where x is the instance number of the `nxge` interface you plan to use. For example, to configure a Netra CP32x0 ARTM-10Gs `nxge` interface with instance 0, create a `hostname.nxge0` file, where 0 is the number of the `nxge` interface. If the instance number were 1, the file name would be `hostname.nxge1`.

2. Enter the VLAN ID (VID) and the physical point of attachment (PPA), as in the following example.

    VLAN logical PPA = 1000 * VID + Device PPA
    nxge123000 = 1000*123 + nxge

This format limits the maximum number of PPAs (instances) you can configure to 1000 in the `/etc/path_to_inst` file.

For example, on a server blade with an Netra CP32x0 ARTM-10G having an instance of 0, belonging to a member of two VLANs (with VID 123 and 224), you would use `nxge123000` and `nxge224000`, respectively, as the two VLAN PPAs.

For more information about VID formats, see the introduction to this Section 4.4, Configuring VLANs.

3. Use `ifconfig` to configure a VLAN virtual device.

    # ifconfig nxge123000 plumb up
    # ifconfig nxge224000 plumb up

The output of `ifconfig -a` on a system having VLAN devices `nxge123000` and `nxge224000` would be as follows:

    # ifconfig -a
    lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232
    index 1
    inet 127.0.0.1 netmask ff000000
    hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500
    index 2
    inet 129.144.131.91 netmask ffffff00 broadcast
    129.144.131.255
    ether 8:0:20:a4:4f:b8
    nxge123000: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>
    mtu 1500 index 3
    inet 199.199.123.3 netmask ffffff00 broadcast
    199.199.123.255
    ether 8:0:20:a4:4f:b8
    nxge224000: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>
    mtu 1500 index 4
    inet 199.199.224.3 netmask ffffff00 broadcast
    199.199.224.225
    ether 8:0:20:a4:4f:b8

4. On the switch, set VLAN tagging and set VLAN ports to coincide with the VLANs you have set up on the server.

For formats, see the introduction to Section 4.4, Configuring VLANs.

Continuing the examples used in Step 2, you would set up VLAN ports 123 and 224 on the switch.

Refer to the documentation that came with your switch for specific instructions for setting VLAN tagging and ports.

### 4.4.2 Configuring VLANs in a Linux Environment

This section describes how to configure VLANs in a Linux environment.

To Configure VLANs in a Linux Environment

1. Ensure that the `nxge` driver is installed:

    # modprobe nxge

2. Plumb the Sun Dual 10GbE SFP+ PCIe ExpressModule interface.

Where xxx.xxx.xx.xxx = the IP address of the interface.

    # ifconfig eth2 xxx.xxx.xx.xxx up

3. Insert the VLAN module:

    # /sbin/modprobe 8021q

4. Add the VLAN instance (VID):

    # /sbin/vconfig add eth2 5

For more information about VID formats, see the introduction to Section 4.4, Configuring VLANs.

5. Configure the `nxge` VLAN (`eth2` in the following example):

    # ifconfig eth2.5 xxx.xxx.xx.xxx up

Copyright © 2008 Sun Microsystems, Inc. All Rights Reserved.