System Design and Development

History of Systems Design and Development Within the SPPDG

Over the years the SPPDG has conducted a number of complete system development projects, the first as long ago as the early 1970's (see History section). During the 1980's the group developed a Digital Radio Frequency Memory (DRFM) containing a 4-bit, 1 gigasample/second (Gs/s) gallium arsenide A/D converter and 200 other commercial components. During the early 1990's we developed an eight-channel fully-synchronous data acquisition system, with each channel collecting digital data at 2 gigabits/second; channel sensitivities allowed data collection from signals with amplitudes as small as 50 mV. From 1991 to 1993 we developed a very small (for the time period) six-channel global positioning system (GPS) receiver, using unpackaged integrated circuits mounted on both sides of a laminate (i.e., organic) multichip module (MCM); the system consumed 1.3 Watts. This GPS receiver held the world’s record as being the smallest such system for nearly five years. The unit was designed, simulated, and assembled in our laboratory using integrated circuits acquired in a collaborative project from Motorola, Inc.

In the period between 1994 and 1997 the Mayo team, in collaboration with MIT Lincoln Laboratory, undertook the development of a prototype all-digital radar-like receiver. Operating in the 420-450 MHz band, the experimental digital receiver used a high sample rate monolithic A/D converter integrated circuit (fabricated in gallium arsenide by Rockwell Science Center) to digitize the RF signal directly, without any requirement for down-conversion of the analog signal. Mayo's task was to create a system package for the ADC integrated circuit and for the gallium arsenide demultiplexer chip that would not distort the incoming analog signal before reaching the input to the ADC, while protecting the ADC from the noise generated by the adjacent demultiplexer. An MCM was created for this pair of chips tailored to the special characteristics of the ADC and the demultiplexer.

Beginning in 1998 and continuing through 2002, the Mayo group designed and fabricated a novel all-digital receiver using an indium phosphide 7 gigasample/sec monolithic A/D converter (clocked at 2.5 GHz) and three indium phosphide demultiplexer chips to demonstrate, successfully, that the reception of signals in the second (and higher) Nyquist bands was feasible.

Beginning in 2002, the Mayo team developed processor systems, housed in rack-mountable drawers, that could accept digital optical data streams at OC-192 rates (9.6 gigabits/second), convert the optical data stream to electrical signals, conduct processing operations on the electrical data stream (using seven large Xilinx Virtex Field Programmable Gate Arrays [FPGAs] in each chassis), reconvert the processed electrical stream back into an optical bit stream, and retransmit the optical signal onto an output fiber. Although the Mayo group typically does not fabricate more than one or two examples of any given processor, this project was an exception, since the sponsor requested that additional systems be fabricated after the first two had been proven to work as intended. A rigorous test of these systems is illustrated in the figure below (Fig.1), in which an optical bit stream generated by an Agilent bit-error-rate detector (BERT) system was run in series through five consecutive chasses (seven chasses appear in the figure, but only the five leftmost stacked units were tested in this particular run). The entire set of five systems operated continuously at 9.6 gigabits/second for 12 hours without a single detected bit error.

Following the completion of the 10 Gb/s cryptographic processor pictured above, we embarked on a project to develop a similar system, again processing an incoming unencrypted optical bit stream and outputting an encrypted stream, at 40 Gb/s. This system, which was implemented on a single large circuit board, appears in the next figure. This unit was specifically designed not to support only the well-known cryptographic algorithms such as AES, but instead to be able to support any number of encryption algorithms by programming the them into FPGAs, whose contents could thus be changed at will. This physical structure allowed the 40 Gb/s system to be used experimentally to identify which algorithms would best suit any given cryptographic environment. A photograph of the 40 Gb/s processor, which internally we referred to as the “Quickarticle Processor”, appears later in this web page, with a description of several of the stages in its development. Note the six fan towers located around the circuit board, under each of which was the highest-capacity FPGA that was available to us when this system was designed.

Description of the Approach Used by SPPDG to Develop a Complex System

A foundational component in systems engineering is the system’s architecture, which describes the system’s structure, behavior, and capability. When described in detail, the system architecture is the basis for development of the constituent circuits by providing the inputs to, outputs from, states of, and functional operations within the circuits. As the circuits themselves are developed, inevitable discovery refines details of the circuits, which percolate upward to iteratively shape the architecture. The figure below (Fig.2) depicts an architecture for a 1 Tb/s encryptor that processes Internet Protocol packets.

Once an initial system architecture becomes available, system-level performance metrics can be estimated. For example, the system’s physical size and construction can be estimated relatively early in the development cycle. Shown here (Fig.3) are three such estimates that assuming different types of integrated circuits and packaging technology. The FPGA-Based Example assumes the use of commodity 28 nm CMOS field-programmable gate arrays (FPGAs) and commodity printed circuit board (PCB) packaging. The inefficiencies within these technologies results in a relatively large, power-hungry system. Alternatively, integrating the same functionality into custom 32 nm CMOS/SOI application-specific integrated circuits (ASICs) enables tighter integration, which reduces the power needed to move the data between functions, and eliminates unnecessary circuitry, as shown in the ASIC-Based Example. The Advanced Technology Example leverages the advantages of novel packaging technologies to tightly integrate multiple ASICs together, thus further reducing size and power. With each incremental use of technology, there is corresponding increments in development costs – in both financial impact and elapsed time-to-develop. FGPAs offer rapid prototyping using commodity components, which is a relatively inexpensive implementation approach for small quantities of the completed system. Alternatively, ASICs can be quite expensive to develop and require significant calendar time for design and fabrication, but the production cost of individual units, in quantity, tends to be less. The trade space of such options needs to be considered both for every type of system and for each application.

Similarly, advances in semiconductor technology offer simplifications as well. Below is a comparable system at a slower data rate (400 Gb/s versus 1 Tb/s from above), which itself reduces the physical size due to the need for fewer components. But in addition, the example below (Fig.4) uses the next-generation FPGAs that use 20 nm FinFET CMOS, which provide approximately twice the logical resources per FPGA, which correspondingly reduces the off-chip interconnect, power, and size.

A Recent Example of a System Development Conducted Within SPPDG

Below (Fig.5) is an example of prototype system developed by the SPPDG several years ago, a 40 Gb/s programmable network processor. External line cards (not shown) interface with 40 Gb/s Ethernet through optical cables. The system then inspects, parses, modifies, and reformats packets. The PCB itself is 15.5” by 26” in size, 0.122” thick, has 32 metal layers, has 20,000 drills, and nearly 5,000 components. Through successive and redundant data paths, the system provides a rapid-development platform for experimenting with new traffic-processing algorithms.

To develop the system shown above, SPPDG systems engineers developed a layered set of architectures, of which one layer is shown below (Fig.6). Here, the signal paths for the main data bus are shown, which architects use to verify conceptual operation and to describe requirements for individual subcircuits.

As the architecture takes shape, physical implementation begins. Developing systems involves a broad collection of expertise, including thermal management, mechanical stress analysis, electronic assembly, material science, etc. As one example of such, expertise, the figure below (Fig.7) shows the design of one of the 32 layers within the PCB. This particular layer has 30 “islands” where each island is used to deliver a particular voltage to the FPGAs. Each island is designed to ensure a low-impedance path from the voltage regulator to the FPGA over a broad frequency range. In addition, each island must be electrically isolated from its neighbors to prevent interference.

Using the PCB design as the basis, full enclosures can be engineered. As illustrated below (Fig.8), the enclosure development involves similar expertise, including thermal management with fans, ducting, and ventilation; mechanical design of the physical enclosure and stress points; power delivery from the AC supply to the intermediate regulator; and signal integrity analysis of the cabling.

The Mayo group’s primary role is often to develop prototype systems, as shown here. When doing so, we work closely with volume manufacturers to ensure that the prototypes that we develop can readily transition to high-volume production with little or no change in the design.

 

Information updated Thursday, October 8, 2015 ⋀ Back to top