PCIe SR-IOV on Neoverse N1 SDP

Introduction

The Neoverse N1 System Development Platform (N1SDP) supports two PCIe root ports each supporting the standard PCIe features including the Single Root I/O Virtualization (SR-IOV) feature. This document gives an overview of how to enable and test the SR-IOV feature on N1SDP.

Note: Due to the PCIe limitations in N1SDP platform (pcie-support), the SR-IOV feature has not been completely validated.

Pre-Requisite

  1. Latest software stack synced by following steps given in user guide

  2. Ensure that the Linux tree in the synced workspace contains the SR-IOV and PCI ACS override patches. This should have been already applied when syncing the code.

Steps to verify SR-IOV

  1. Add the kernel config CONFIG_IXGBEVF=y to the file linux/arch/arm64/configs/defconfig. This is required to enable the drivers for the mentioned Intel card.

  2. Build the software stack and flash the Ubuntu image onto the boot device. Do initial boot of the board which installs Ubuntu and perform second boot which boots Ubuntu kernel.

  3. Login to target Ubuntu console and edit the file /etc/default/grub. Add pcie_acs_override=id:13b5:0100 option to the GRUB_CMDLINE_LINUX_DEFAULT. Save this file and run update-grub command.

  4. Reboot the board from Ubuntu console using reboot now command.

  5. Now Linux probes and assigns separate IOMMU groups for all PCIe devices.

  6. Virtual functions can be enabled from sysfs using following command:

    echo 63 > /sys/bus/pci/devices/0001:01:00.0/sriov_numvfs

    Note that in test environment the Intel card’s ethernet port 0 is identified in Segment:1 Bus:1 Dev:0 Function:0

Limitations

  1. SR-IOV feature is only supported in CCIX slot and not in the PCIe slots. This is due to the on-board PCIe switch not supporting the ARI capability to which the PCIe slots are connected.

  2. Only Intel X540-T2 card has been validated for the SR-IOV feature.

References

Copyright (c) 2020-2021, Arm Limited. All rights reserved.