User Guide

Contents

• User Guide

  • Introduction

  • Host prerequisites

    • Packages

    • CMake

    • Installing repo tool

  • Syncing and building the source code

    • Syncing

      • Check dependencies

      • Fetch Tools

    • Building the Software Stack

      • Firmware only

      • Ubuntu Distribution

      • Minimal BusyBox

      • Prebuilt board firmware

  • Software Components

    • Firmware

      • Firmware image creation

      • Trusted Firmware-A

      • System Control Processor (SCP)

      • UEFI EDK2

    • File-system

      • Disk image creation

      • GRUB

      • Linux

      • BusyBox

      • Ubuntu

  • Running the software distribution on N1SDP

    • Setting up the N1SDP

  • Update firmware on microSD card

    • L3 Cache Enablement

    • Boot Minimal BusyBox Image on N1SDP

    • Boot Ubuntu on N1SDP

    • Building Kernel Modules Natively

Introduction

The Neoverse N1 System Development Platform (N1SDP) is an enterprise class reference board based on the Neoverse N1 core.

This document is a guide on how to fetch, build from source, and run an Arm Reference Platforms software stack on N1SDP, including a Linux distribution based on either the Ubuntu server distribution or a minimal BusyBox based root filesystem.

The synced workspace includes:

• Scripts to build the board firmware, Linux kernel, and Ubuntu server distribution image.

• Ubuntu server distribution, sources for Linux Kernel, EDK2, firmware and BusyBox.

• Other supporting board files and prebuilt binaries.

Host prerequisites

• Host PC is running Ubuntu Linux 20.04 LTS.

• Minimum 50GB of storage space.

• Commands provided in this guide are executed from a bash shell environment.

Packages

To install all the required packages, run:

sudo apt-get update
sudo apt-get install autoconf autopoint bison build-essential curl \
   device-tree-compiler flex git libssl-dev m4 mtools pkg-config \
   python3-distutils python-is-python2 unzip uuid-dev

CMake

While cmake is available to install with apt-get, this does not install the required version i.e. 3.18.4 or later. Following commands can be used for checking the cmake version and removing it if the version is less than 3.18.4.

sudo cmake --version
sudo apt-get remove cmake

An alternate way to install the cmake tool is by running the following command.

sudo snap install cmake --classic

Configure Git, run:

git config --global user.email "[email protected]"
git config --global user.name "Your Name"

Installing repo tool

Follow the instructions provided in the repo README file to install the repo tool.

NOTE: The repo tool which gets installed using apt-get command sometimes return errors, in such a case it is recommended to install repo using the following steps from the repo README file:

mkdir -p ~/.bin
PATH="${HOME}/.bin:${PATH}"
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/.bin/repo
chmod a+rx ~/.bin/repo

Syncing and building the source code

Syncing

The N1SDP software stack supports two software distributions:

• A minimal BusyBox root filesystem.

• Ubuntu server.

The instructions below provide the necessary steps to sync these distributions.

Create a new folder that will be your workspace and will henceforth be referred to as <n1sdp_workspace> in these instructions:

mkdir <n1sdp_workspace>
cd <n1sdp_workspace>

Run the following commands to fetch the software stack sources.

repo init \
    -u https://git.gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-manifest.git \
    -b <BRANCH> \
    -g <GROUP> \
    -m <MANIFEST FILE> \
    --depth=6
repo sync

Note: If repo sync fails prompting failure of server certificate verification, run sudo apt-get install ca-certificates to update the ca-certificates and then re-run repo sync.

Supported options for <BRANCH> are:

• refs/tags/<TAG> (This option fetches a particular release tag pointed by the <TAG> field. The latest release tag is N1SDP-2022.06.22)

• master (This option fetches the master branch used for code development and could contain patches that are not available in the above branches/tags.)

Supported options for <MANIFEST FILE> are:

• n1sdp.xml (This option uses the manifest file pointing to master branch for a few components. However, please be aware that such untagged changes have not yet been fully verified and should be considered unstable until they are tagged in an official release.)

• pinned-n1sdp.xml (This option uses the manifest file pointing to pinned version of the components. The pinned version used is determined by <BRANCH> option.)

Supported options for <GROUP> are:

• ubuntu

• bsp

• busybox

The -g <GROUP> option downloads the software components just for the specified group to save time and storage space. <GROUP> is a comma separated option where multiple, comma-separated options can be provided as shown below:

• -g bsp (Only downloads software components required for BSP)

• -g busybox (Downloads software components for both BSP and BusyBox)

• -g ubuntu (Downloads software components for both BSP and Ubuntu)

• -g busybox,ubuntu (Downloads software components for BSP, BusyBox and Ubuntu)

NOTE: To sync the source code with the complete history of components, drop the depth option from the repo init command for the given filesystem/distribution.

Check dependencies

To ensure that all the required dependent packages were installed, run:

./build-scripts/check_dep.sh

"no missing dependencies detected" should get displayed.

Fetch Tools

The tools required to build the software components are fetched using build-scripts/fetch-tools.sh, it is executed as part of build-all.sh.

The script must be run separately when building individual software components for the first time.

./build-scripts/fetch-tools.sh -f none

Building the Software Stack

To build all the software components for a given platform and filesystem, a top level build-all.sh script is provided. Execute the following command to trigger the software build.

./build-scripts/build-all.sh -f <FILESYSTEM> <CMD>

Supported options for <FILESYSTEM> are:

• ubuntu

• busybox

• none

Filesystem option none builds and packages only the firmware components.

Supported options for <CMD> are:

• clean

• build

• all

Note: If CMD option is not passed, build is performed by default.

Firmware only

Build the firmware files.

./build-scripts/build-all.sh -f none

Ubuntu Distribution

N1SDP provides support for installation and boot of standard Ubuntu 18.04 distribution. The distribution is installed on a disk and since the installed image is persistent it can be used for multiple boots.

Build the firmware and Ubuntu distribtution:

./build-scripts/build-all.sh -f ubuntu

The bootable image will be available at the location <n1sdp_workspace>/output/n1sdp/ubuntu.img>

Minimal BusyBox

Build the firmware and BusyBox based root filesystem:

./build-scripts/build-all.sh -f busybox

The bootable image will be available at the location <n1sdp_workspace>/output/n1sdp/busybox.img>

Prebuilt board firmware

The board firmware prebuilt binaries are made available as part of the release in the directory bsp/n1sdp-board-firmware/, these binaries can be copied to the onboard SD card to boot the platform to UEFI EDK2 shell console.

There are two methods for updating the microSD card with the firmware binaries:

1. The microSD card from the N1SDP can be removed from N1SDP and can be mounted on a host machine using a card reader.

2. The USB debug cable when connected to host machine will show the microSD partition on host machine which can be mounted.

   $> sudo mount /dev/sdX1 /mnt
   $> sudo rm -rf /mnt/*
   $> sudo cp -a bsp/n1sdp-board-firmware/* /mnt/
   $> sudo umount /mnt
   

NOTE: replace sdX1 with the device and partition of the SD card. Option (2) above is typically preferred, as removing the microSD card requires physical access to the motherboard inside the N1SDP’s tower case.

NOTE:

Please ensure to use the recommended PMIC binary. Refer to page potential-damage for more info. If a PMIC binary mismatch is detected, a warning message is printed in the MCC console recommending the user to switch to appropriate PMIC image. On MCC recommendation ONLY, please update the MB/HBI0316A/io_v123f.txt file on the microSD using the below commands.

Example command to switch to 300k_8c2.bin from the host PC

$> sudo mount /dev/sdX1 /mnt
$> sudo sed -i '/^MBPMIC: pms_0V85.bin/s/^/;/g' /mnt/MB/HBI0316A/io_v123f.txt
$> sudo sed -i '/^;MBPMIC: 300k_8c2.bin/s/^;//g' /mnt/MB/HBI0316A/io_v123f.txt
$> sudo umount /mnt

Software Components

Firmware

Each firmware component has a separate script to build that component. After the firmware components have been built they must be packaged using build-firmware-image.sh.

To make customizations to the firmware

# first make sure to build all the firmware components
<n1sdp_workspace>/build-scripts/build-all.sh -f none

# modify the firmware component

# build the component
<n1sdp_workspace>/build-scripts/build-[COMPONENT].sh -f none

# repackage the firmware files
<n1sdp_workspace>/build-scripts/build-firmware-image.sh -f none

As an alternative the one-liner <n1sdp_workspace>/build-scripts/build-all.sh -f none will trigger the build of all firmware scripts.

NOTE: The valid firmware [COMPONENT] names are :

"arm-tf" : To build Trusted Firmware-A
"scp"    : To build SCP and MCP ROM and RAM firmware binaries
"uefi"   : To build EDK2 for N1SDP

Firmware image creation

Packages the firmware files in a format suitable for the N1SDP.

Build script

<n1sdp_workspace>/build-scripts/build-firmware-image.sh

Output

<n1sdp_workspace>/output/n1sdp/firmware/scp_rom.bin <n1sdp_workspace>/output/n1sdp/firmware/scp_fw.bin <n1sdp_workspace>/output/n1sdp/firmware/mcp_rom.bin <n1sdp_workspace>/output/n1sdp/firmware/mcp_fw.bin <n1sdp_workspace>/output/n1sdp/firmware/fip.bin <n1sdp_workspace>/output/n1sdp/firmware/n1sdp-board-firmware_primary.tar.gz <n1sdp_workspace>/output/n1sdp/firmware/n1sdp-board-firmware_secondary.tar.gz

Trusted Firmware-A

Based on Trusted Firmware-A.

Build script	<n1sdp_workspace>/build-scripts/build-arm-tf.sh
Checkout path	<n1sdp_workspace>/bsp/arm-tf
Output	<n1sdp_workspace>/output/n1sdp/intermediates/tf-bl1.bin <n1sdp_workspace>/output/n1sdp/intermediates/tf-bl2.bin <n1sdp_workspace>/output/n1sdp/intermediates/tf-bl31.bin <n1sdp_workspace>/output/n1sdp/intermediates/n1sdp-single-chip.dtb <n1sdp_workspace>/output/n1sdp/intermediates/n1sdp-multi-chip.dtb <n1sdp_workspace>/output/n1sdp/intermediates/n1sdp_fw_config.dtb <n1sdp_workspace>/output/n1sdp/intermediates/n1sdp_tb_fw_config.dtb <n1sdp_workspace>/output/n1sdp/intermediates/n1sdp_nt_fw_config.dtb

System Control Processor (SCP)

Based on SCP Firmware.

Build script	<n1sdp_workspace>/build-scripts/build-scp.sh
Checkout path	<n1sdp_workspace>/bsp/scp
Output	<n1sdp_workspace>/output/n1sdp/intermediates/scp-ram.bin <n1sdp_workspace>/output/n1sdp/intermediates/scp_rom.bin <n1sdp_workspace>/output/n1sdp/intermediates/mcp-ram.bin <n1sdp_workspace>/output/n1sdp/intermediates/mcp_rom.bin

UEFI EDK2

Based on UEFI EDK2.

Build script	<n1sdp_workspace>/build-scripts/build-uefi.sh
Checkout path	<n1sdp_workspace>/bsp/uefi/edk2
Output	<n1sdp_workspace>/output/n1sdp/intermediates/uefi.bin

File-system

Each file-system component has a separate script to build that component. Some build scripts depend on the value of -f <FILESYSTEM>. After the file-system components have been built they must be packaged into a disk image using build-disk-image.sh.

# first make sure that all the components have been built at least once
<n1sdp_workspace>/build-scripts/build-all.sh -f <FILESYSTEM>

# modify the component

# build the component
<n1sdp_workspace>/build-scripts/build-[COMPONENT].sh -f <FILESYSTEM>

# repackage the disk image
<n1sdp_workspace>/build-scripts/build-disk-image.sh -f <FILESYSTEM>

NOTE:

The valid filesystem [COMPONENT] names are :

"grub" : To build the GRUB utilities
"linux": To build the Linux kernel image
"perf" : To build the perf tool

The valid [FILESYSTEM] names are :

"busybox" : To build for BusyBox filesystem
"ubuntu"  : To build for Ubuntu filesystem distribution

Disk image creation

Create a disk image for -f <FILESYSTEM>. FILESYSTEM is either busybox or ubuntu.

Build script	<n1sdp_workspace>/build-scripts/build-disk-image.sh
Output	<n1sdp_workspace>/output/n1sdp/<FILESYSTEM>.img <n1sdp_workspace>/output/n1sdp/intermediates/<FILESYSTEM>.esp.img

GRUB

Based on grub.

Build script	<n1sdp_workspace>/build-scripts/build-grub.sh
Checkout path	<n1sdp_workspace>/grub
Output	<n1sdp_workspace>/output/n1sdp/intermediates/grub/output/grubaa64.efi

Linux

Based on Linux 5.18.4 for N1SDP.

Build script	<n1sdp_workspace>/build-scripts/build-linux.sh
Checkout path	<n1sdp_workspace>/linux
Output	<n1sdp_workspace>/output/n1sdp/intermediates/kernel_Image_<FILESYSTEM>

BusyBox

Build a minimal root filesystem based on BusyBox

Build script	<n1sdp_workspace>/build-scripts/build-busybox.sh
Checkout path	<n1sdp_workspace>/busybox
Output	<n1sdp_workspace>/output/n1sdp/intermediates/busybox.initramfs <n1sdp_workspace>/output/n1sdp/intermediates/busybox.root.img

Ubuntu

Build Ubuntu server distribution based on Ubuntu 18.04

Build script	<n1sdp_workspace>/build-scripts/build-ubuntu.sh
Checkout path	<n1sdp_workspace>/tools/ubuntu_bionic
Output	<n1sdp_workspace>/output/n1sdp/intermediates/ubuntu.esp.img <n1sdp_workspace>/output/n1sdp/intermediates/ubuntu.root.img

Running the software distribution on N1SDP

This section provides steps for:

• Setting up the N1SDP with the required board firmware

• Preparing a bootable disk

• Boot the supported software distributions (Minimal BusyBox or Ubuntu Server).

Setting up the N1SDP

After powering up or rebooting the board, any firmware images placed on the board’s microSD will be flashed into either on-board QSPI flash or copied into the DDR3 memory via the IOFPGA.

Configure COM Ports

Connect a USB-B cable between your host PC and N1SDP’s DBG USB port, then power ON the board. The DBG USB connection will enumerate as four virtual COM ports assigned to the following processing entities, in order

ttyUSB<n>   - Motherboard Configuration Controller (MCC)
ttyUSB<n+1> - Application Processor (AP)
ttyUSB<n+2> - System Control Processor (SCP)
ttyUSB<n+3> - Manageability Control Processor (MCP)

Please check ls /dev/ttyUSB* in Linux to identify <n>.

Use a serial port application such as minicom to connect to all virtual COM ports with the following settings:

115200 baud
8-bit word length
No parity
1 stop bit
No flow control

Note: Some serial port applications refer to this as “115200 8N1” or similar.

• An example usage to open the MCC console using minicom is as follows: (Ensure to have minicom package pre-installed to the host using "sudo apt-get install minicom")

  sudo minicom -s /dev/ttyUSB<n>
  

  Select Serial port setup from the user interface and configure the port settings mentioned above. Select Exit to apply the settings and to proceed with the port connection.

  Refer to the COM port nomenclature mentioned above in this section to choose the desired port to establish the connection with. For more information on minicom usage, refer to Minicom Manual.

Before running the deliverables, ensure both BOOT CONF switches are in the

OFF position, then issue the following command in the MCC console:

Cmd> USB_ON

This will launch the microSD card in the N1SDP board as a mass storage device in the host PC.

Note: After updating firmware as mentioned in the Update firmware on microSD card section, ensure to use USB_OFF command to disconnect the microSD from the host PC.

Enter the following command on the MCC console window to ensure time is correctly set. This is required for the first distribution boot to succeed:

Cmd> debug
Debug> time
Debug> date
Debug> exit

Update firmware on microSD card

The board firmware files are located in <n1sdp_workspace/output/n1sdp/firmware/> after the firmware source build.

Single chip mode:

n1sdp-board-firmware_primary.tar.gz    : firmware to be copied to microSD of N1SDP board in single chip mode.

Multi chip mode:

n1sdp-board-firmware_primary.tar.gz    : firmware to be copied to microSD of primary board.
n1sdp-board-firmware_secondary.tar.gz  : firmware to be copied to microSD of secondary board.

There are two methods for populating the microSD card:

1. The microSD card from the N1SDP can be removed from N1SDP and can be mounted on a host machine using a card reader,

2. The USB debug cable when connected to host machine will show the microSD partition on host machine which can be mounted.

Option (2) above is typically preferred, as removing the microSD card requires physical access to the motherboard inside the N1SDP’s tower case.

The instructions to extract the board firmware package onto the microSD card is as shown below:

$> sudo mount /dev/sdX1 /mnt
$> sudo rm -rf /mnt/*
$> sudo tar --no-same-owner -xzf n1sdp-board-firmware_primary.tar.gz -C /mnt/
$> sudo umount /mnt

NOTE:

• Replace sdX1 with the device and partition of the SD card.

• Follow the similar steps to install the firmware on secondary board with n1sdp-board-firmware_secondary.tar.gz for multi chip mode.

Firmware tarball package contains IOFPGA configuration files, SCP, TF-A, and UEFI binaries.

NOTE: Please ensure to use the recommended PMIC binary. Refer to page potential-damage for more info.

If a PMIC binary mismatch is detected, a warning message is printed in the MCC console recommending the user to switch to appropriate PMIC image. On MCC recommendation ONLY, please update the MB/HBI0316A/io_v123f.txt file on the microSD using the below commands.

Example command to switch to 300k_8c2.bin from the host PC

$> sudo mount /dev/sdX1 /mnt
$> sudo sed -i '/^MBPMIC: pms_0V85.bin/s/^/;/g' /mnt/MB/HBI0316A/io_v123f.txt
$> sudo sed -i '/^;MBPMIC: 300k_8c2.bin/s/^;//g' /mnt/MB/HBI0316A/io_v123f.txt
$> sudo umount /mnt

L3 Cache Enablement

By default, L3 cache is disabled for use. To enable/disable L3 cache support, follow these steps:

1. Run USB_ON command to mount the on-board microSD card on the host PC.

2. Open the file “MB/HBI0316A/io_v123f.txt”.

3. For user to enable/disable L3 cache support, edit the SCC BOOT_GPR1 register in the following manner.

   • To enable L3 cache, update the SOCCON with offset 0x1184 and set the value 0x00000001. The line should now read,

     SOCCON: 0x1184 0x00000001 ; SoC SCC BOOT_GPR1

   • To disable L3 cache, update the SOCCON with offset 0x1184 and set the value 0x00000000. The line should now read,

     SOCCON: 0x1184 0x00000000 ; SoC SCC BOOT_GPR1

4. Save and close the file.

Boot Minimal BusyBox Image on N1SDP

Preparing a bootable disk with BusyBox root filesystem

A bootable disk (USB stick or SATA drive) can be prepared by flashing the image generated from the source build. The image will be available at the location <n1sdp_workspace>/output/n1sdp/busybox.img>

This is a bootable GRUB image comprising Linux kernel and BusyBox binaries. The partitioning and packaging is performed during the build phase.

Use the following commands to prepare the GRUB image on a USB stick or SATA drive:

$ lsblk
$ sudo dd if=busybox.img of=/dev/sdX conv=fsync bs=1M
$ sync

Note: Replace /dev/sdX with the handle corresponding to your USB stick or SATA drive, as identified by the lsblk command.

Booting the board with BusyBox image

Insert the bootable disk created earlier. Shutdown and reboot the board by issuing the following commands to the MCC console:

Cmd> SHUTDOWN
Cmd> REBOOT

Enter the UEFI menu by pressing Esc on the AP console as the EDK2 logs start appearing; from here, enter the UEFI Boot Manager menu and then select the disk.

By default the Linux kernel will boot with ACPI configurations:

* BusyBox N1SDP (ACPI)
  BusyBox N1SDP SINGLE CHIP (Device Tree)
  BusyBox N1SDP MULTI CHIP (Device Tree)

The system will boot into a minimal BusyBox Linux image environment.

Boot Ubuntu on N1SDP

Preparing a bootable Ubuntu disk

A bootable disk (USB stick or SATA drive) can be prepared by formatting it with the distribution image created during source build. The image will be available at the location <n1sdp_workspace/output/n1sdp/ubuntu.img>.

This is a bootable GRUB image comprising Linux kernel and an Ubuntu Server 18.04 file system. The partitioning and packaging is performed during the build.

Use the following commands to burn the GRUB image to a USB stick or SATA drive:

$ lsblk
$ sudo dd if=ubuntu.img of=/dev/sdX bs=1M conv=fsync
$ sync

Note: Replace /dev/sdX with the handle corresponding to your USB stick or SATA drive, as identified by the lsblk command.

Booting the board with Ubuntu image

Insert the bootable disk created earlier and connect the ethernet cable to a working internet connection. This is REQUIRED on first boot in order to successfully download and install necessary Ubuntu packages. Installation will fail if an internet connection is not available.

NOTE: It is also observed that the installation may fail if more than one storage device is present on N1SDP, the error log is as shown below:

Booting `Install Ubuntu on N1SDP Platform'
error: disk `hd1,msdos2' not found.
error: you need to load the kernel first.

Press any key to continue...

Failed to boot both default and fallback entries.

Press any key to continue...

Therefore, it is always recommended to have only one storage device on N1SDP on which you want to install the Ubuntu software.

Shutdown and reboot the board by issuing the following commands to the MCC console:

Cmd> SHUTDOWN
Cmd> REBOOT

Enter the UEFI menu by pressing Esc on the AP console as the EDK2 logs start appearing; from here, enter the UEFI Boot Manager menu and then select the burned disk.

Ubuntu 18.04 will boot in two stages; the first boot is an installation pass, after which a second boot is required to actually enter the Ubuntu Server environment.

To reboot the board after the first boot installation pass has completed, from MCC console:

Cmd> REBOOT

The system will boot into a minimal Ubuntu 18.04 environment.

Login as user root with password root, and install any desired packages from the console:

# apt-get install <package-name>

Building Kernel Modules Natively

Native building of kernel modules typically requires kernel headers to be installed on the platform. However, a bug in deb-pkg currently causes host executables to be packed rather than the target executables.

This can be worked around by building and installing the kernel natively on the platform.

Boot the N1SDP board with Ubuntu filesystem and login as root.

Install all the required packages, run:

apt-get install -y git build-essential bc bison flex libssl-dev

Native build fails with 5.18.4 kernel which is under discussion here and will be fixed in later versions of the kernel. As a workaround for this issue, newer version of gcc can be installed with the following steps:

apt-get install software-properties-common
add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get update
apt-get install gcc-9 g++-9
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9

To build the kernel natively use the following steps:

git clone -b n1sdp https://git.gitlab.arm.com/arm-reference-solutions/linux.git
cd linux/
mkdir out
cp -v /boot/config-5.18.4+  out/.config
make O=out -j4
make O=out modules_install
make O=out install
update-grub
sync

Reboot the board and when Grub menu appears, select the Advanced Boot Options -> 5.18.4 kernel for booting.

---

Copyright (c) 2021-2022, Arm Limited. All rights reserved.