uClinux on the NetFPGA Embedded System Stanford High-Performance Networking Group

I joined the project fall of 2006 and have been working on porting the operating system and software toolchain necessary to enable students to easily develop and deploy their software so

uClinux on the NetFPGA

Embedded System Stanford High-Performance Networking Group

Justin Erickson

I would like to thank the Stanford High-Performance Networking Group for providing

me the opportunity to work on this project and for all their support throughout the project

I would especially like to thank Greg Watson and Paul Hartke for guiding me through all

the difficulties

Table of Contents

1 Introduction 5

2 History of the NetFPGA Project 7

2.1 Project Motivation 7

2.2 NetFPGA v1 Design 7

2.3 Problems with NetFPGA v1 8

3 Overview of the NetFPGA Platform 9

3.1 Hardware Overview 9

3.2 Software Overview 11

3.3 Solutions to NetFPGA problems 12

4 Software Implementation 13

4.1 Overview of Challenges 13

4.2 Overview of Build Process 14

4.2.1 Building the Base System 14

4.2.2 Building uClinux 15

4.2.3 Uploading the Configuration to the Board 16

4.3 Building on the Common XUPV2P Board 16

4.4 Building on NetFPGA Board 18

5 Test Software 20

5.1 Overview 20

5.2 Building the Test Application 20

5.3 Results 21

6 Conclusion 22

6.1 Current Status 22

6.2 Lessons Learned 22

6.3 Future 23

Appendix: NetFPGA with uClinux Design Tutorial 25

Installing the Software Tools 25

Creating the Hardware from Xilinx Platform Studio (XPS) 26

Building uClinux 30

Uploading the Hardware and uClinux to the NetFPGA Board 31

Adding Custom User Applications to uClinux 33

Finding Help 33

References 34

1 Introduction

This thesis will detail my role in the Stanford High-Performance Networking group working on the NetFPGA project NetFPGA is a customizable software and hardware solution aimed at allowing students to create network hardware that most accurately simulates real-world network hardware I am specifically working on the second-

generation NetFPGA board

The overall idea is to develop a platform that provides computer science and electrical engineering students full control to develop both the software and hardware for a networkdevice The NetFPGA board combines memory, a processor, and a FPGA (Field

Programmable Array) on a standard PCI board, which will easily be integrated into standard PCs today The FPGA chip is a Xilinx technology which can dynamically alter itself to implement user-customized hardware, while the memory and processor enable software to run on the board as well The FPGA chip has enough logic components to run

a processor within it rather than the embedded processor, which is what we decided to do

in this project for reasons to be further discussed in Section 3

The NetFPGA project is a group within the Stanford High-Performance Networking group and funded by the National Science Foundation About ten people are currently working on the project, and it has been in development since about 2004

I joined the project fall of 2006 and have been working on porting the operating system and software toolchain necessary to enable students to easily develop and deploy their software solutions on the NetFPGA board As I will detail in Section 4, we decided to port the uClinux operating system, which is a variant of Linux specifically catered to small embedded systems due to the NetFPGA’s memory constraints and the user

community’s knowledge of Linux

After getting the toolchain up and running, I developed test software as a proof of

concept that the NETFPGA project was a viable platform for developing networking devices and to ensure that the necessary software toolchain and libraries were configured properly

To provide context for my discussion of my part of the project, this thesis provide

background in Sections 2 and 3 Section 2 will specifically provide a brief description of the whole NetFPGA project It will start with the overall motivations and design for the first NetFPGA project and finish with the results and problems that second-generation NetFPGA board aims to correct Section 3 will provide an architectural overview of the new NetFGPA design and how we will overcome the issues encountered in the original NetFPGA design Section 3 will also introduce the software architecture that was the focus of my work

Sections 4 and 5 will detail my specific contribution to the NetFPGA project and the challenges I encountered in getting the necessary software toolchain in place Section 4 will describe porting uClinux to the NetFPGA hardware platform using the Xilinx and Microblaze toolchain Section 5 will detail the test software I designed to demonstrate theviability of NetFPGA as practical and realistic development platform for networking devices Section 6 will discuss the current status of the NetFPGA project as well as lessons learned and the future of NetFPGA

The Appendix will contain a tutorial on how to create a working uClinux system for the NetFPGA board As explained in Section 4, it is far easier to reference a base

configuration for a similar design than to create an entirely new configuration because theXilinx tools are designed for full flexibility and as such do not prevent non-working configurations For more information on porting uClinux to platforms similar to

2 History of the NetFPGA Project

implementations because the long-term trend has been towards designing customized hardware for faster implementations of networking algorithms Another problem with software implementations is that they are inherently slower than customized hardware solutions

Thus, to provide more effective simulation of networking solutions, the NetFPGA projectaims to create an inexpensive and practical networking platform that allows users to develop customized hardware

2.2 NetFPGA v1 Design

The original NetFPGA board contains three Field Programmable Arrays (FPGAs), which are chips that accept a Hardware Description Language such as Verilog and dynamically alter themselves to implement the described hardware logic Two of these FPGAs are available for the user to customize for their individual hardware solution, while the third

is used to control the interaction between the user’s hardware and the Ethernet controller Each FPGA has its own SRAM memory bank The NetFPGA board also has an Ethernet controller with eight Ethernet 10 Mb/s ports The Ethernet ports serve the obvious

providing a means of receiving the new FPGA descriptions from a NetFPGA server using

a customized protocol designed to control the FPGA Figure 1 shows a visualization of the overall architecture [2]

Figure 1: Architectural Overview of the NetFPGA Board [2]

2.3 Problems with NetFPGA v1

While NetFPGA succeeded in its original goal of providing a cost- and time-effective environment to test and run customized network hardware, it has some fundamental shortcomings that prevent it from achieving widespread use outside the Stanford

community The largest drawback of NetFPGA v1 is the difficulty in setting up and configuring a new cluster of NetFPGA boards The NetFPGA boards required a custom rack and special networking switches [2] Another issue is that the boards cannot run software While the trend in industry is towards hardware implementations of networkingsolutions, many industry networking devices implement the more advanced functionality

in software in an embedded system

3 Overview of the NetFPGA Platform

3.1 Hardware Overview

In general, the design of the new NetFPGA board is similar to the original NetFPGA board It, however, has an embedded processor as well as the capability to run a processorwithin one of its FPGAs, thus enabling software to run in addition to the custom

hardware While the board can run as a self-sustained unit, it is built to run in a standard full-sized PCI slot on a host box The PCI connection allows the host machine to interact with the NetFPGA board for configuration and debugging Instead of three FPGAs, the new NetFPGA board uses two FPGAs, a larger Vertex2Pro and a smaller Spartan-II The Spartan-II is programmed to handle the PCI bus communications and exposes a simpler interface to the Virtex2Pro chip, which handles the board’s networking capabilities and run the user-defined hardware (see Figure 2) [3]

Figure 2: Block diagram of NetFPGA’s two FPGAs The red circle is where the user

design runs [3]

The NetFPGA board contains four gigabit Ethernet ports, 4MB of SRAM, and a

PowerPC processor The inclusion of the SRAM bank and processor enable the board to run user-designed software as well as hardware Furthermore, the Vertex2Pro is large enough to store and run Xilinx’s softcore Microblaze processor, which is a simple

processor designed to run inside an FPGA

As the PCI protocol does not change, there will seldom be a need to change the

Spartan-II configuration, so it will automatically load the PCI protocol implementation from a Flash chip at startup

From the user’s perspective, managing the specific implementation details of the memorycontroller, PCI interface, and multiple MAC arbiter is generally uninteresting and

superfluous to the design, so the default NetFPGA configuration gives the user a

simplified abstraction of the actual hardware by providing a virtual LAN layer Therefore,the user design interacts with the virtual LAN layer, PCI interface exposed by the

Spartan-II, and the SRAM controller The PCI interface will also be used to push the user FPGA design into the system and provide advanced debugging capabilities Since the user design will change frequently, it will not be stored statically on the board but rather downloaded to the FPGA whenever necessary Figure 3 shows the simplified user view ofthe NetFPGA board

Figure 3: Simplified user view of NetFPGA [3]

3.2 Software Overview

Unlike the original NetFPGA, the second-generation NetFPGA is designed to allow software to run on the board on the embedded processor Despite these improvements, thenew NetFPGA board has some strict limitations that were a critical factor in deciding the software architecture These limitations are the amount of logic components available in the Vertex2Pro FPGA, which effectively limit the complexity of the hardware it can implement, and a total of 4MB of SRAM that must be shared between the embedded hardware and software While these limitations may at first seem like a design

miscalculation because all fully-fledged desktop operating systems require far more resources, the cost efficiency that they contribute is critical to achieving wide-spread adoption at other universities

In order to adhere to these limitations, we decided to use uClinux with a Microblaze softcore processor rather than the PowerPC uClinux is an operating system heavily based

on Linux but designed to run on systems with significantly less processing and memory resources such as DVD players and other embedded systems By default, uClinux also contains the standard C libraries the user will need to develop useful applications and can

be easily extended to include additional user applications By using uClinux, we enable

users to leverage any existing knowledge of Linux to decrease the learning curve of interacting with the NetFPGA system, while also respecting the board’s limitations.

The choice to use Microblaze rather the PowerPC was twofold First, the uClinux

community has documented experiences of porting uClinux to other Microblaze systems, and, thus, by using Microblaze, we hoped to utilize these resources to minimize the difficulty of porting uClinux to the NetFPGA board Second, the Microblaze processor was designed to consume as few resources as possible, while a PowerPC CPU was designed with performance in mind and thus generally has a larger runtime footprint than the Microblaze processor

3.3 Solutions to NetFPGA problems

As discussed in Section 2, the main problems preventing increased market penetration of original NetFPGA platform were the complications of initially setting up the board and the inability to run software on an embedded system By using the standard full-size PCI form factor, the second-generation NetFPGA board can fit in most existing PC boxes The original NetFPGA relied on a complicated, proprietary, network protocol to transfer FPGA configurations to the board, which required some careful setup to configure

properly By using standard PCI bus to interact with the board, the user will find it will beeasier for the user to upload to interact with the new NetFPGA board Additionally, the PCI connection can be configured to allow the host PC to view and provide advanced debugging capabilities to the user that were not available on the original NetFPGA

4 Software Implementation

4.1 Overview of Challenges

My main objective for setting up NetFPGA’s software platform was to get uClinux running on the board uClinux contains the necessary software libraries and operating system functionality to run custom C applications, such as software routers or switches The first challenge was familiarizing myself with the necessary tools to build and deploy uClinux onto the NetFPGA board, namely, Xilinx’s toolchain and the uClinux build environment Part of my challenge here was to gain enough familiarity with the hardwaredesign to understand how the operating system will interact with the hardware to make a functional system

To help gain familiarity with the tools and understand how to port uClinux to a new system, I first created an uClinux implementation on the Xilinx XUPV2P board using a known reference implementation for the board The Xilinx XUPV2P board is a

commonly available board that members of the uClinux community have already had experience with Because the NetFPGA board has many similarities to the XUPV2P board, this allowed me to work in a known environment where I could utilize the

resources of the user community to help me understand and debug the issues that arose The largest challenges for me personally were learning about hardware designs for embedded systems and how to use the Xilinx environment, which was clearly designed with maximum flexibility for knowledgeable users

After getting uClinux to run successfully on the XUPV2P boards and gaining enough familiarity to apply my knowledge of the toolset to the new NetFPGA environment, I worked on porting the XUPV2P configuration to the NetFPGA board

As mentioned in the Section 3, a major limitation of the NetFPGA board is the 4MB total

of SRAM banks that needs to be shared between the hardware and software As network hardware generally requires memory to buffer incoming or outgoing packets, it is

important that the software not monopolize the memory Therefore, my next challenge was to strip out the superfluous components of uClinux to minimize its memory usage

Once I configured uClinux to run in an acceptable amount of memory, I worked on developing software to interact with the Ethernet controller as a proof of concept that the system was capable of running user developed software for networking devices This part

of the project will be detailed in Section 5

4.2 Overview of Build Process

This section will detail how I built the necessary components to get uClinux running on

an embedded system with Microblaze, which in my case is either the XUPV2P or

NetFPGA boards First I used the Xilinx tools to develop the hardware system and the software interface according to the board’s and uClinux’s configuration specifications Then I built uClinux to adhere to the hardware specifications of this system Finally, I uploaded the system binaries to the board, which configures the FPGAs to implement the hardware, and then I uploaded the uClinux image to run on the hardware The following subsections will provide a brief overview of the build process (see Appendix for specific details)

4.2.1 Building the Base System

Xilinx Platform Studio (XPS) is a GUI solution for building hardware configurations that

is packaged with Xilinx’s Embedded Development Kit (EDK) In addition to XPS, the

XPS itself comes with a Base System Builder (BSB) wizard, which guides users through the configuration of the basic components of an embedded system.

As each board can have different peripherals and components, every board needs a configuration specification called a Board Support Package (BSP), which XPS can load

to provide the user with the correct configuration options during the BSB wizard As we will discuss in the next section, uClinux also provides a BSP that provides XPS with the necessary specifications to configure the system to work with uClinux Once the user has run the BSB wizard, the user can then modify both the software and hardware

configuration of the base system Of course, if an advanced user prefers, he can bypass the XPS altogether and manually design the board specification himself

During my project, I used the XPS to design the basic hardware configuration as well as the interface with uClinux operating system When building the system, XPS outputs a configuration file for uClinux, which is used in building uClinux to shape the build to the correct hardware configuration The uClinux build environment will be described in the following section

4.2.2 Building uClinux

As explained in the Section 3, uClinux is a version of Linux, which is specifically

designed to work on limited resource configurations such as NetFPGA In order to work within this limitation, uClinux does not provide memory management and thus has no notion of virtual memory addresses In general, from a user perspective, most of the complexity of working within such an environment is handled by the uClinux build process, and thus the user can effectively interact with the uClinux system just as he would with Linux

In order to build uClinux on a Microblaze board such as NetFPGA and XUPV2P, we need the Microblaze toolchain and the uClinux BSP, as well as the uClinux source code

compatible with Microblaze processor, which we will use to actually build uClinux The uClinux BSP, just like the NetFPGA BSP, provides the uClinux specifications to allow XPS to design a system configuration that will be compatible with your uClinux build XPS will also output a configuration file, called auto-config.in, which the Microblaze compiler will use to configure uClinux to work with your system configuration.

4.2.3 Uploading the Configuration to the Board

Uploading the configuration is a two-part process First we upload the hardware

configuration generated through XPS, and then we upload the uClinux operating system image we built with the Microblaze toolchain We can use Xilinx’s Impact tool to upload the hardware to the FPGAs This will get the underlying hardware running Then we connect to the NetFPGA board through the debug module, Xilinx Multiprocessor

Debugger (XMD), to upload the directly into the SRAM banks We then initialize some registers, including the program counter, and fire off the processor If everything is done correctly, uClinux will then boot up

4.3 Building on the Common XUPV2P Board

As mentioned Section 4.1, before working on the custom-designed NetFPGA board, I built uClinux to work on an XUPV2P board, a commonly used board for embedded systems While this board is obviously different from the target NetFPGA board, the similarities are large enough that I was able to port the design from the XUPV2P board tothe NetFPGA with a few modifications

The first challenges for me were simply setting up the board and properly configuring theXilinx tools for the XUPV2P board Some things I had to learn was how to use the Board

Unfortunately, there was no way to know if the board was configured properly until I actually started using the board, so I started with Jason Wu’s “Microblaze uClinux: Creating a Simple uClinux ready Microblaze Design” tutorial in order to familiarize myself with the Xilinx tools and the Microblaze hardware environment [1] In addition tolearning about the Xilinx tools, I also had to gain some knowledge about that underlying hardware to debug the many problems I encountered Even though Jason Wu’s tutorial was extremely useful, the tutorial was based on a slightly different board than the

XUPV2P that I was using, so I had to determine what changes were necessary to port his design to the XUPV2P (see Appendix for specific details) After building the hardware, I then had to learn how to build uClinux and upload everything to the board Once again, the differences between the XUPV2P and the board used in the tutorial meant that I had

to learn how to adapt the tutorial configuration for the XUPV2P board

The next challenge was figuring out what needs to be changed in the configuration to portthe design from the XUPV2P board to the NetFPGA board Since there are many

potential issues that may arise in building the configuration for the new board, it was easiest to test as many of a similar configuration on the XUPV2P board first because I already had a working configuration The first big difference between the XUPV2P boardand the NetFPGA board was the lack of a serial connector for outputting to the host machine I eventually learned that I could trick uClinux to output to the debug terminal rather than try to output over a dedicated output port by simply switching the addresses for the UART, which handles the serial communications, and the debug module (see Appendix for more details)

Another issue was adjusting to the memory constraints of the NetFPGA board since the XUPV2P had 256MB of RAM instead of the 4MB of the NetFPGA board In order to testthis, I added artificial constraints on XUPV2P configuration to force uClinux and the hardware to recognize only 4MB of the available 256MB on the board Fortunately, the test showed that uClinux by default is capable of booting in only 4MB