The Origins of our PCIe Expansion Products
Cyclone Microsystems is a developer and manufacturer of high performance Intelligent I/O boards and embedded systems. Four years ago when we launched our first PCIe version of our popular PCI Express IOP, an exciting project requirement arose requiring the aggregation of up to 80 PCIe IOP boards under a single host. PCIe hosts then and now have few PCIe slots. Most enterprise PC vendors figure that fast LAN accommodates "expansion" for most general IT users. However, most embedded systems developers need to integrate a wide range of hardware - they need plentiful PCIe slots. This need drove us to develop our PCI Express Expansion Systems. 

Our Expansion Systems address the needs of embedded system developers. Consequently, our PCI Express expansion systems are an integral component of a wide range of embedded systems. These applications range from audio/graphic workstations to seismic exploration.

What PCIe is - and is not
PCI Express is a fast local bus using 2.5 and 5.0 Gb/s SerDes technology for Gen1 and Gen2, respectively. Using the PCI SIG 1.0 Cabling Standard, Cyclone has extended the capabilities of slot-limited host computers and enabled them as the foundation for I/O intensive embedded systems. As a local bus, the system requires a single root complex to control the system resources and provide system clocking. Our PCI Express products are built as downstream devices for a host system root complex.

In addition to single-attached PCIe Expansion Systems, we have successfully tested multi-tier hierarchies as well as multiple expansion systems plugged into a single host. Both topologies work successfully.

PCI Express Expansion Systems are not suitable for interconnecting multiple host systems for a failover or high reliability application. Other advanced interconnect technologies like Infiniband are currently better suited and specifically designed for such applications.

There are efforts underway to create multi-domain switches which will enable multi-host systems. However, with today’s products, this is currently not an viable option. 

For PCI Express specifications, please see: http://www.pcisig.com/specifications/pciexpress/

Host Operating Systems
Our PCI Express systems are host system agnostic and work with all PCI Express based host computer systems. The host BIOS is the key to supporting bus expansion as the BIOS needs the capability to recognize multiple bus segments behind a bridge. To the BIOS, the expansion backplane’s switch appears in this manner.

We have many programs successfully deploying our Expansion Systems with Windows, Apple (both Pentium and PPC), and Sun.

Approved Host Computer List
Not all early PCI Express system's BIOSs supported additional bus segments to accommodate PCIe expansion. Now most all do. Consequently, we do not have a list of compatible products as all the systems we’ve tested and heard from customers from Dell, HP, IBM, Supermicro, Sun, and Apple have the proper BIOS capability. A small few that encountered difficulties have been resolved with a BIOS update from the host system manufacturer.

As mentioned above, our PCI Express Expansion systems are host processor and vendor agnostic. The sole dependency is on the host BIOS's ability to recognize and enumerate a large quantity of bridged bus segments. The host OS sees the expansion system's switch as a large quantity of bridged bus segments.

Our customers are successfully using our expansion systems with Pentium, Xeon, PPC, and UltraSPARC T1 processors.

Expansion System Drivers
None required. The host BIOS recognizes the expansion backplane switch and the boards and are automatically enumerated.

Bandwidths over the Expansion Cable
PCI Express Gen1 is based on SerDes Lanes of 2.5 Gb/s bi-directional traffic. Consequently, a x8 Gen1 link, or eight lanes, yields a bi-directional bandwidth of 20 Gb/s. After subtracting 8b10b encoding and bus overhead, we measure a sustained practical bandwidth of 14.7 Gb/s. The PCIe Expansion cable's PCIe lanes are dynamically assigned to each expansion slot/ board by the expansion system switch according to the bandwidth needs of each board.

Gen2 uses 5.0 Gb/s SerDes lanes yielding 80Gb/s x16 lanes and 40 Gb/s x8 lanes.

Common Start Up Questions/Challenges

Power Switch

The Expansion Chassis power supply is turned on by a signal from the host system via the expansion cable. The power switch on the expansion chassis serves no purpose (expect for keeping the Expansion system price low by using a generic ATX chassis). Consequently, is not possible to power cycle the Expansion chassis independently of the host system.

Drivers

No drivers are needed as the host system BIOS recognizes the expansion system's switches and enumerates the entire system

Physical Board Seating

Mechanically securing the installed expansion boards by their face panels to the expansion chassis is highly advised. This is especially important for up-plugged boards (smaller link card into a larger connector)

Xilinx FPGA based Link Boards

Our support experience is that early versions of the PCI Express interface IP from Xilinx do not fully conform to the PCI Express Specification. Please insure that you have the most current version.

PCIe-409 Downshifting (Superceded by the new PCIe-425 Host Bus to Expansion Cable Adapter)
The PCIe-409 uses bus repeaters to provide the lowest bus latency possible. However, the PCIe-409 does not support down-shifting. Down-shifting is plugging a larger lane board into a smaller lane routed connector. In the case of the x8 PCIe-409, it must be plugged into the true routed x8 lane slot. Do not be deceived by the mechanical size of the PCIe slot connector. Verify that the slot has the correct quantity of lanes routed to it. This is particularly important on many of Apple’s host computer products where all/most PCI Express Expansion Slots are provisioned with x16 or x8 connectors. However, each connector’s lane are rationed and assigned by configurable boot parameters. (See next listing for MAC PRO)  

Apple Mac PRO Challenge (PCIe-409, PCIe-416 HBA Reprise)
Many of Apple’s PCI Express host computer products have bus connectors that are mechanically either x16 or x8. However, each connector’s lane assignments are configurable boot parameters. See Apple's support note at http://docs.info.apple.com/article.html?artnum=304122 Please insure that HBA products that do not support down-shifting, like the repeater-based PCIe-409 and PCIe416s, are assigned their respective x8 and x16 lane assignment. 

Audio Processing: The difference between the 600-2704 and 600-2705 Systems

The 2704 and 2705 use the same expansion backplane and chassis, but differ only by the use of different Host Bus to Expansion Cable technology. The 2704 uses the switch based PCIe-408 HBA while the 2705 uses the repeater based PCIe-409 HBA. The repeater based technology used on the PCIe-409 (and 2705 system) shows slightly better performance (8-11% better reads for small packet sizes, unchanged on writes) but is unable to be downshifted into a smaller lane width - it requires a fully routed x8 slot. The PCIe-408 is slightly slower but can be down-shifted to a x4 lane width.

We developed a the repeater based technology to shave system latencies in small packet, latency dependent systems like audio and video processing. Consequently, the 600-2705 is the recommended choice for use with ProTools and Universal Audio and other audio processing applications.

Manufacturing Test Beds: PCIe Boards

Hot Swap vs Host System Reboot
The 600-2701 supports low level hot plug capabilities. This essentially provides the event signaling and slot power ramping circuitry. This hardware capability does not initially allow for autonomous power cycling of the Expansion Chassis for efficient manufacturing test. The full hot-plug feature was dependent on Microsoft’s inclusion of hot-plug higher level capabilities in a subsequent Windows release.  These features would allow for the dynamic allocation and de-allocation of resources from the systems without rebooting the host. To date, Microsoft has removed Hot Plug from future Windows releases.

However, some enterprising engineers have used the 2701’s low level hot plug features to detect the event signaling and power down the requested slot. A custom board driver then re-writes the shadow configuration space settings back to hardware. Using this approach, manufacturing test groups have been able to remove and reinstall boards under test with the host system remaining live.

Slot Savers
For test environments, we recommend the use of expendable “PCIe Slot savers”. We successfully use products from Adexelec for this purpose."http://www.adexelec.com/pciexp.htm#PEXP8-EX"www.adexelec.com/pciexp.htm#PEXP8-EX

They also have x4 and x16 Straight risers. Please keep in mind that the mechanical fence needs to rise an equivalent distance.

Shake and Bake
Our products use commercial grade switches. Consequently, our products are designed to conform to commercial temperature specifications. Unless you isolate the expansion backplane from the test chamber with slot ribbon cables, Cyclone products can not be used in extended temperature environments.  

Down Plugging Hazards
Down-plugging is the plugging of a smaller link card into a larger link connector. PCI Express fully supports down plugging. However, even in a test environment, insure that the board’s face panels are mechanically secured to the chassis fence to insure the boards-under-test are seated firmly in the slot. It doesn’t take much rocking motion before the false negative test fallout becomes a crisis!

Purchasing

OEM Applications     http://www.cyclone.com/purchase/process.php  

End-Users               E-commerce Site      

Dealers and VARS are welcome to distinguish themselves through orders at the higher volume levels.