PEX8648-BB50BCF - PLX Technology, Inc.

PCIe Gen2, 5.0GT/s 48-l ne, 12-port PCIe Switch a

Features Features

 PEX 8648 Vitals  PEX 8648 Vitals

o 48-lane, 12-port PCIe Gen2 switch o 48-lane, 12-port PCIe Gen2 switch

- Integrated 5.0 GT/s SerDes - Integrated 5.0 GT/s SerDes

o 27 x 27mm2, 676-pin FCBGA package o 27 x 27mm2, 676-pin FCBGA package

o Typical Power: 3.7 Watts o Typical Power: 3.7 Watts

 PEX 8648 Key Features  PEX 8648 Key Features

o Standards Compliant o Standards Compliant

- PCI Express Base Specification, r2.0

(backwards compatible w/ PCIe r1.0a/1.1)

- PCI Express Base Specification, r2.0

(backwards compatible w/ PCIe r1.0a/1.1)

- PCI Power Management Spec, r1.2 - PCI Power Management Spec, r1.2

- Microsoft Vista Compliant - Microsoft Vista Compliant

- Supports Access Control Services - Supports Access Control Services

- Dynamic link-width control - Dynamic link-width control

- Dynamic SerDes speed control - Dynamic SerDes speed control

o High Performance o High Performance

- Non-blocking switch fabric - Non-blocking switch fabric

- Full line rate on all ports - Full line rate on all ports

- Packet Cut-Thru with 140ns max packet

latency (x16 to x16)

- Packet Cut-Thru with 140ns max packet

latency (x16 to x16)

- 2KB Max Payload Size - 2KB Max Payload Size

- Read Pacing (bandwidth throttling) - Read Pacing (bandwidth throttling)

- Dual-Cast - Dual-Cast

o Flexible Configuration o Flexible Configuration

- Ports configurable as x1, x2, x4, x8, x16 - Ports configurable as x1, x2, x4, x8, x16

- Registers configurable with strapping

pins, EEPROM, I2C, or host software

- Registers configurable with strapping

pins, EEPROM, I2C, or host software

- Lane and polarity reversal - Lane and polarity reversal

- Compatible with PCIe 1.0a PM - Compatible with PCIe 1.0a PM

o Dual-Host & Fail-Over Support o Dual-Host & Fail-Over Support

- Configurable Non-Transparent port - Configurable Non-Transparent port

- Moveable upstream port - Moveable upstream port

- Crosslink port capability - Crosslink port capability

o Quality of Service (QoS) o Quality of Service (QoS)

- Eight traffic classes per port - Eight traffic classes per port

- Weighted round-robin source

port arbitration

- Weighted round-robin source

port arbitration

o Reliability, Availability, Serviceability o Reliability, Availability, Serviceability

- 3 Hot Plug Ports with native HP Signals - 3 Hot Plug Ports with native HP Signals

- All ports hot plug capable thru I2C

(Hot Plug Controller on every port)

- All ports hot plug capable thru I2C

(Hot Plug Controller on every port)

- ECRC and Poison bit support - ECRC and Poison bit support

- Data Path parity - Data Path parity

- Memory (RAM) Error Correction - Memory (RAM) Error Correction

- INTA# and FATAL_ERR# signals - INTA# and FATAL_ERR# signals

- Advanced Error Reporting - Advanced Error Reporting

- Port Status bits and GPIO available - Port Status bits and GPIO available

- Per port error diagnostics - Per port error diagnostics

- Performance Monitoring - Performance Monitoring

• Per port payload & header counters • Per port payload & header counters

Version 1.0 2009

PEX 8648

The ExpressLaneTM PEX 8648 device offers PCI Express switching

capability enabling users to add scalable high bandwidth, non-blocking

interconnection to a wide variety of applications including servers,

storage systems, and communications platforms. The PEX 8648 is

well suited for fan-out, aggregation, and peer-to-peer applications.

High Performance & Low Packet Latency

The PEX 8648 architecture supports packet cut-thru with a maximum

latency of 140ns (x16 to x16). This, combined with large packet memory

and non-blocking internal switch architecture, provides full line rate on all

ports for performance-hungry applications such as servers and switch

fabrics. The low latency enables applications to achieve high throughput and

performance. In addition to low latency, the device supports a packet payload

size of up to 2048 bytes, enabling the user to achieve even higher throughput.

Data Integrity

The PEX 8648 provides end-to-end CRC (ECRC) protection and Poison bit

support to enable designs that require end-to-end data integrity. PLX also

supports data path parity and memory (RAM) error correction as packets

pass through the switch.

Flexible Register & Port Configuration

The PEX 8648’s 12 ports can be configured to lane widths of x1, x2, x4, x8,

or x16. Flexible buffer allocation, along with the device's flexible packet

flow control, maximizes throughput for applications where more traffic

flows in the downstream, rather than upstream, direction. Any port can be

designated as the upstream port, which can be changed dynamically. The

PEX 8648 also provides

several ways to

configure its registers.

The device can be

configured through

strapping pins, I2C

interface, host

software, or an optional

serial EEPROM. This

allows for easy debug

during the development

phase, performance

monitoring during the

operation phase, and

driver or software

upgrade. Figure 1

shows some of the PEX

8648’s common port

configurations.

Figure 1. Common Port Configurations

PEX 8648

11 x4 2 x44 x8

10 x4 6x42 x8

Figure 1. Common Port Configurations

PEX 8648

11 x4 2 x44 x8

10 x4 6x42 x8

Dual-Host & Failover Support

The PEX 8648 product supports a Non-Transparent

(NT) Port, which enables the implementation of multi-

host systems in communications, storage, and blade

server applications. The NT port allows systems to

isolate host memory domains by presenting the

processor subsystem as an endpoint rather than another

memory system. Base address registers are used to

translate addresses; doorbell registers are used to send

interrupts between the address domains; and scratchpad

registers (accessible by both CPUs) allow inter-

processor communication (see Figure 2).

CPU

Prim a ry Ho st

Primary Host

Root

Complex

Root

Complex

End

Point

End

Point End

Point

End

Point End

Point

End

Point

PEX 8648

CPU

Secondary Host

Non-Transparent

Port

Figure 2. Non-Transparent Port

CPU

Prim a ry Ho st

Primary Host

CPU

Prim a ry Ho st

Primary Host

Root

Complex

Root

Complex

Root

Complex

Root

Complex

End

Point

End

Point End

Point

End

Point End

Point

End

Point

End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point

PEX 8648

CPU

Secondary Host

CPU

Secondary Host

Non-Transparent

Port

Figure 2. Non-Transparent Port

Dual Cast

The PEX 8648 supports Dual Cast, a feature which

allows for the copying of data (e.g. packets) from one

ingress port to two egress ports allowing for higher

performance in dual-graphics, storage, security, and

redundant applications.

Read Pacing

The Read Pacing feature allows users to throttle the

amount of read requests being made by downstream

devices. When a downstream device requests several

long reads back-to-back, the Root Complex gets tied up

in serving this downstream port. If this port has a narrow

link and is therefore slow in receiving these read packets

from the Root Complex, then other downstream ports

may become starved – thus, impacting performance. The

Read Pacing feature enhances performances by allowing

for the adequate servicing of all downstream devices.

Hot Plug for High Availability

Hot plug capability allows users to replace hardware

modules and perform maintenance without powering

down the system. The PEX 8648 hot plug capability

feature makes it suitable for High Availability (HA)

applications. Three downstream ports include a

Standard Hot Plug Controller. If the PEX 8648 is used in

an application where one or more of its downstream

ports connect to PCI Express slots, each port’s Hot Plug

Controller can be used to manage the hot-plug event of

its associated slot. Every port on the PEX 8648 is

equipped with a hot-plug control/status register to

support hot-plug capability through external logic via the

I2C interface.

SerDes Power and Signal Management

The PEX 8648 supports software control of the SerDes

outputs to allow optimization of power and signal

strength in a system. The PLX SerDes implementation

supports four levels of power – off, low, typical, and

high. The SerDes block also supports loop-back modes

and advanced reporting of error conditions, which

enables efficient management of the entire system.

Interoperability

The PEX 8648 is designed to be fully compliant with the

PCI Express Base Specification r2.0, and is backwards

compatible to PCI Express Base Specification r1.1 and

r1.0a. Additionally, it supports auto-negotiation, lane

reversal, and polarity reversal. Furthermore, the PEX

8648 is tested for Microsoft Vista compliance. All PLX

switches undergo thorough interoperability testing in

PLX’s Interoperability Lab and compliance testing at

the PCI-SIG plug-fest.

Applications

Suitable for host-centric as well as peer-to-peer traffic

patterns, the PEX 8648 can be configured for a wide

variety of form factors and applications.

Host Centric Fan-out

The PEX 8648, with its symmetric or asymmetric lane

configuration capability, allows user-specific tuning to a

variety of host-centric applications. Figure 3 shows a

typical server design where the root complex provides a

PCI Express link that needs to be expanded to a larger

number of smaller ports for a variety of I/O functions. In

this example, the PEX 8648 has a 16-lane upstream port,

and five downstream ports using x8 and x4 links.

The PEX 8648 can also be used to create PCIe Gen1 (2.5

Gbps) ports. The PEX 8648 is backwards compatible

with PCIe Gen1 devices. Therefore, the PEX 8648

enables a Gen 2 native Chip Set to fan-out to Gen 1

endpoints. In Figure 3, the PCIe slots connected to the

PEX 8648’s downstream ports can be populated with

either PCIe Gen1 or PCIe Gen 2 devices. Conversely,

the PEX 8648 can also be used to create Gen 2 ports on

a Gen 1 native Chip Set in the same fashion.

CPU

x16

Memory

x16

Endpoint

Chipset

Endpoint

PCIe Gen1 or PCIe Gen2 slots

PEX 8648

Figure 3. Fan-in/out Usage

Communications Systems

The PEX 8648 can also be utilized in communications

applications. Figure 4 shows a PEX 8648 being used in a

router to connect multiple endpoints in a line card to the

control module using PowerPC. The link-widths for each

module can be configured as required. The peer-to-peer

communication feature of the PEX 8648 allows the

endpoints to communicate with each other without any

intervention or management by the host.

Memory

PowerPC

End

Point

End

Point

End

Point

End

Point End

Point

End

Point

End

Point

End

Point

Control

Module

Line Card

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

End

Point

PEX 8648

x8 or x4

Figure 4. Router Usage

Backplane Communication

With 48 lanes and 12 ports, the PEX 8648 is well suited

for backplane applications requiring high connectivity (a

large number of ports). Figure 5 represents an ATCA or

MicroTCA backplane application with two switch fabric

blades and multiple AMC blades. In this example, two

PEX 8648s provide peer-to-peer data exchange for up to

20 AMC blades connecting to the switch fabric. The

PEX 8648 utilizes its NT port to isolate the hosts on the

switch fabrics and the remaining 10 downstream ports

on each switch are used to fan-out to the 20 AMC blades

(or 10 AMC Carrier Modules).

10-20 AMC or 10 AMC

Carrier Modules Fabric

Backplane

To other

Fabric

PEX 8648

CPU

Figure 5. ATCA/MicroTCA Backplane Usage

Fail-Over Storage Systems with Dual Cast

The PEX 8648’s Dual Cast feature proves to be very

useful in storage systems. In Figure 6, the Dual Cast

feature enables the PEX 8648 to copy data going to its

four downstream ports to the backup system and vice

versa (see yellow traffic patterns) in one transaction as

opposed to having to execute two separate transactions

to send data to the redundant chassis. By offloading the

task of backing up data onto the secondary system,

processor and system performance is enhanced. Non-

Transparent (NT) ports are used to isolate the host

domains of the backup system from the primary system.

PEX 8648

8 Disk Chassis

x4 x4

8 Disk Chassis

x4 x4

8 Disk Chassis

x4 x4

8 Disk Chassis

x4 x4

CPU

Memory

Chipset

CPU

CPU CPU

Memory

Chipset

Memory

Chipset

Backup

System

PEX

8648

PEX

8648

PEX 8616

NT NT

PEX 8616

All

x8s All

x8s

PEX 8616

Figure 6. Dual Cast in Storage Systems

Software Usage Model Development Tools

From a system model viewpoint, each PCI Express port

is a virtual PCI to PCI bridge device and has its own set

of PCI Express configuration registers. It is through the

upstream port that the BIOS or host can configure the

other ports using standard PCI enumeration. The virtual

PCI to PCI bridges within the PEX 8648 are compliant

to the PCI and PCI Express system models. The

Configuration Space Registers (CSRs) in a virtual

primary/secondary PCI to PCI bridge are accessible by

type 0 configuration cycles through the virtual primary

bus interface (matching bus number, device number, and

function number).

PLX offers hardware and software tools to enable rapid

customer design activity. These tools consist of a

hardware module (PEX 8648RDK), hardware

documentation (available at www.plxtech.com), and a

Software Development Kit (also available at

www.plxtech.com).

ExpressLane PEX 8648RDK

The PEX 8648RDK is a hardware module containing the

PEX 8648 which plugs right into your system. The

PEX 8648RDK can be used to test and validate customer

software, or used as an evaluation vehicle for PEX 8648

features and benefits. The PEX 8648RDK provides

everything that a user needs to get their hardware and

software development started.

Interrupt Sources/Events

The PEX 8648 switch supports the INTx interrupt

message type (compatible with PCI 2.3 Interrupt signals)

or Message Signaled Interrupts (MSI) when enabled.

Interrupts/messages are generated by PEX 8648 for hot

plug events, doorbell interrupts, baseline error reporting,

and advanced error reporting.

Software Development Kit (SDK)

PLX’s Software Development Kit is available for

download at www.plxtech.com/sdk. The software

development kit includes drivers, source code, and GUI

interfaces to aid in configuring and debugging the

PEX 8648.

Product Ordering Information

Part Number Description

PEX8648-BB50BC 48-Lane, 12-Port PCI Express Switch (27x27mm2)

PEX8648-BB50BC F 48-Lane, 12-Port PC I Ex press Switch, Pb-F ree (27x 2 7m m2)

PLX Technology, Inc.

870 Maude Ave.

PEX8648-BB RDK PEX 8648 Rapid Development Kit

Please visit the PLX Web site at http://www.plxtech.com or contact PLX sales at 408-774-9060 for sampling.

Sunnyvale, CA 94085 USA

info@plxtech.com

www.plxtech.com

Inc., which may be registered in some jurisdiction. All other product names that appear in this material are for identification purposes only and are acknowledged to be

trademarks or registered trademarks of their respective companies. Information supplied by PLX is believed to be accurate and reliable, but PLX Technology, Inc. assumes no

responsibility for any errors that may appear in this material. PLX Technology, Inc. reserves the right, without notice, to make changes in product design or specification.

PEX8648-SIL-PB-1.0 04/09