- Cloud ServicesDeep Learning
- ProductsRackmount &
- Cloud Services
- Deep Learning Servers
- Rackmount Servers
- Blade Servers
- Legacy Products
- SolutionsFor Today's
- IndustriesHPC and more
- CompanyAbout Cirrascale
- ContactGet in Touch
About Our Technology/ PCIe Switch Riser Technology
Cirrascale multi-device PCIe Switch Riser peering solutions for highly parallel applications are different from other supporting hardware implementations. Our switch riser allows PCIe-compliant devices to communicate as if they are all on the same bus... because they are.
Gone are the days of needing a bounce-buffer in host memory, or leaving GPU DMA engines unused because they couldn't address other devices in the system. Our PCIe switch riser technology reduces intercard latency while helping to maintain a consistent performance level between GPUs or other PCIe-compliant devices such as coprocessors, InfiniBand® cards, RAID controllers, FPGAs, or PCIe flash storage. Cirrascale has been able to overcome these issues, and achieve near linear performance scaling with its remarkable design.
Expanding PCIe on a Single Root Complex
Cirrascale GPU compute servers are different from other GPU supporting hardware implementations. Most of the hardware configurations available today only provide maximum performance between specific pairs of GPUs; and since GPUs are paired up, jobs requiring communication between arbitrary GPUs experience a performance impact. Additionally, there can be a significant performance impacts with trying to scale more than four GPUs on multi-socket systems. These have been persistent problems for customers who are pushing the limits of GPUs with large, complex data-sets and calculations, or where data must be streamed between GPUs. Cirrascale has been able to overcome these issues, and achieve near linear performance scaling with its design.
Maximize PCIe Bandwidth
Cirrascale is a strong believer in utilizing a technology to its fullest potential whenever possible and GPUs and GPU Accelerators are no exception. If the GPU has a PCIe Gen3 x16 link, then it should use it when communicating with other GPUs — any other GPUs. Our switch riser technology allows us to scale and peer multiple PCIe x16 Gen3 cards on a single root complex ensuring that the maximum PCIe bandwidth is available utilized for inter-card communication.
Minimize Intercard Latency and Obtain Consistent Performance Between GPUs
Our switch riser allows GPUs to communicate closely together because they are all on the same bus. Say goodbye to needing a bounce-buffer in host memory, or to leaving GPU DMA engines unused. This greatly reduces intercard latency and helps to maintain consistent performance levels between GPUs.
Enable GPU-Centric Development and Usage
Since most all of the GPU traffic is passed between the GPUs directly via the Cirrascale PCIe switch riser, a very negligible amount of host resources are needed to perform GPU work. Additionally, with a single address space and simultaneous inter-card communication at full PCIe x16 Gen3 speeds, software can spend more time doing work than thinking about when to schedule data copies.
Supports the Largest Number of GPU Offerings
We work closely with our technology partners to ensure you're given the broadest offerings for your application. The Cirrascale GB Series supports both professional and consumer cards from the leading manufacturers including ground-breaking GPU accelerators, such as the NVIDIA® Tesla® K80 Dual-GPU Accelerators.
Configurable for True Expandability
What makes the Cirrascale line of PCIe Switch Risers so unique is their ability to go beyond their obvious uses. Beyond just the ability to expand PCIe fabric and enable up to 160 PCIe lanes in one system, the PCIe Switch Risers are programable to enable additional features such as load balancing, bridging and virtual switching.
Load Balancing Method
When one of the PCIe slots is over-subscribed (think multiple GPUs trying to talk to the host at the same time), we can adjust how the traffic is balanced among the slots. There are a few options, but the two high-level choices are "Equal Bandwidth" and "Equal Packets". "Equal Bandwidth" attempts to ensure that all users of the over-subscribed slot get an equal share of the available bandwidth on that slot. For example, if two PCIe x16 Gen3 GPUs try and send data to the host simultaneously, using the single PCIe x16 Gen3 upstream port, they will each get (effectively) a PCIe x8 Gen3 amount of bandwidth. The "Equal Packets" mode, which is how we typically configure the Switch Riser, instead of balancing out bandwidth, balances out PCIe TLP packets. Just like in Ethernet and many other packetized protocols, PCIe packets on the bus are not always the same size. This mode attempts to balance the number of packets sent across the over-subscribed link, rather than the bandwidth.
The most straightforward way to configure the Cirrascale Switch Riser products is in "Transparent Bridge" mode. In this configuration, all downstream devices (think GPUs, IB cards, etc.) are visible to the upstream device (read: PCIe Root Complex) without any special configuration. Device enumeration and data is "transparently" routed to the appropriate device, with no drivers or unique configuration necessary. This is what allows products using the Cirrascale Switch Risers to "just work" with nearly any operating system and host architecture. The other mode of operation is "Non-Transparent Bridge" mode. As the name would suggest, this mode requires special configuration by the upstream device to query and expose downstream devices – usually in the form of a kernel-space driver. The reason this would be desired would be to support unique, multi-host configurations, where devices such as GPUs, could be shared between multiple hosts. In "Transparent Bridge" mode this doesn't work because each host believes it has exclusive access to the devices and things go south quickly. When configured as a "Non-Transparent Bridge", the hosts can communicate using other means, to negotiate who will configure ownership of each device, then program the Cirrascale PCIe Switch Riser accordingly.
Virtual Switch Mode
Much of the value of the Cirrascale PCIe Switch Riser comes from the fact that it allows multiple downstream devices, such as GPUs, to communicate with each other directly. By maximizing the number of downstream slots, the Switch Riser maximizes the number of devices that can partake in that communication. An alternate mode, called "Virtual Switch Mode" can convert one (or more) downstream slots into an upstream slot, and associate specific downstream ports with it. For example, instead of Slot 0 being upstream and Slots 1-5 on an SR3615 being downstream, the Cirrascale Switch Riser can be configured so that Slot 0 is upstream, and grouped with Slots 1 and 2. Slots 3 and 4 would also be configured as downstream, Slot 5 would now be an upstream port. This effectively converts the SR3615 into a pair of SR3312 switches. Note that because PCIe cannot have any routing loops, and has no concept of "Spanning Tree" or similar discovery protocols, this removes the ability for devices in slots 1 and 2 to communicate directly with those in slots 3 and 4 — the SR3615 is essentially cut into two. This can be advantageous if more per-device upstream bandwidth is desired, but is an uncommon configuration for Cirrascale products. Most designs benefit more from the addition of a 2nd Cirrascale Switch Riser than in reducing the slot-count of existing risers.