Archive for the ‘Backplane’ Category
Practical Modeling of High-speed Channels
As Dave Dunham from Molex Corp. likes to say, “When designing high-speed serial links beyond 10 GB/s, everything matters”. In order to ensure first time success at these speeds, accurate channel modeling is a prerequisite. This is especially true for long backplane channels.
Although many EDA tools include the latest and greatest models for conductor surface roughness and wide-band dielectric properties, obtaining the right parameters to feed the models is always a challenge. Often the only sources are from data sheets alone. But in most cases, the numbers do not translate directly into parameters needed for the EDA tools. So how do we get these parameters?
One way is to follow the design feedback method which involves designing, building and measuring a test coupon, then extracting the parameters through tuning simulation to measurement. Although this method is pretty practical and accurate, a significant amount of expertise and equipment is required to design, build and measure the test coupon, which takes significant amount of time and money.
But, as Eric Bogatin often likes to say, “Sometimes an OK answer NOW! is better than a good answer late.” As a high-speed signal integrity practitioner and backplane architect, I often have to come up with an answer sooner, rather than later because of the impact to time and cost to my clients. And that’s why I have been motivated over the last few years to research and develop simple methodologies to accurately determine parameters to feed into modern EDA tools.
If you went to this year`s EDICon 2017 in Boston, and attended the High-speed Digital Symposium session, you would have heard me speak on a “Practical Modeling of High-speed Channels Based on Data Sheet Input”, which was the title of my presentation.
For those of you who could not attend, I have made available an annotated slide deck. You can download a copy from my web site.
What you will learn:
-
How to use my Cannonball model to determine Huray roughness parameters from data sheet alone
-
How to determine effective dielectric constant due to roughness from data sheets alone
-
How to apply these parameters in the latest version of Polar Si9000e Field Solver
-
How to pull it all together using Keysight ADS software
And this is an example of simulation results compared to measurements you can expect to see:
Non-contact Interconnect: When Crosstalk Is Your Friend
Originally published in, The PCB Design Magazine, November 2012 issue.
In normal PCB designs, crosstalk is usually an unwanted effect, due to electro-magnetic coupling, of two or more traces routed in close proximity to one another. We usually consider it to be our enemy, in any high-speed design, and go to great lengths to avoid it. So how, you may ask, can crosstalk ever be your friend?
To answer that question, I would like to start out by taking you back to the fall of 1994. This was the era of wide parallel busses running up to 33 MHz across backplanes. High-speed serial, point-point interfaces, and serdes technology, as we know and love today, was just a twinkle in some bright young engineer’s eye.
Nortel, a.k.a. Northern Telecom at the time, was looking to replace the computing module shelf of the DMS Supernode platform because it was projected to run out of steam a few years later. In order to address the issue, the system architects decided that a scalable, multi-processing, shared memory, computing architecture was needed to replace it.
My job was to develop a concept to package all these cards in a shelf, and then design a backplane to interconnect everything. It quickly became evident that a single shared bus could not support the bandwidth required for multi-processing. Nor could multiple parallel buses solve the problem, because of the lack of high-density backplane connector technology needed for all the I/O. Even if we had a suitable connector, and it could magically fit within the confines of the card slot, then the layer count of the backplane would have grown exponentially.
No, something else was needed. Fortunately, Bell Northern Research (BNR), the R&D lab of Nortel where I was working at the time, had an advanced technology group, that liked to play in the sand. I remember going to a meeting one day to see some presentations on some of the neat technology they were playing with.
One presentation they gave, was of a unique non-contact interconnect technology. I immediately saw the practical application that technology offered for our architecture, and it instantly became my friend. It allowed us to eventually invent a patented, proprietary point to multi-point interconnect solution, running at 1GB/s per pair [1].
The non-contact technology actually relied on controlled electro-magnetic coupling, or simply crosstalk. See Figure 1. In this simple high-level block diagram, each card on the shelf would transmit their data differentially across the backplane. As the differential pairs traversed through the connector fields of the card slots, the transmit signal was edge-coupled to an adjacent small trace, about three quarters of an inch long, connected to the respective receiver pin. After the last card slot, the transmit differential pairs switched layers where they returned back to the originating card and were terminated.
The beauty of this architecture was that each card only needed one set of transmitters to broadcast its data to all the other cards. Since each card had enough receivers to listen to the other cards, the point to multi-point interconnect achieved the equivalent of a multipoint to multipoint architecture; but without the overhead of additional pins and PCB layers. Furthermore, an effective line rate of 1GB/s was achieved using simple, inexpensive 2mm connectors; the same ones chosen for compact PCI standard.
Figure 2 is a photograph of an inner layer, double-sided core of the backplane, prior to lamination and drilling. It shows the couplers in more detail. The round pads are for the connector vias, and are used to attach the coupler traces to the connector pins. The rows of pads on the left are for one card slot, while the rows of pads on the right are for another card slot.
If we look at the two traces entering the picture from the bottom left side, we can see how they are routed through the connector field. These two traces are part of a differential pair where each are routed as single-ended traces, i.e. with no coupling to one another. As these traces approach the first row of pads, they jog down to minimum spacing to ensure close coupling to the coupler traces attached to the pads. The close spacing continues to ensure maximum coupling to the next set of pads, where the pattern stars all over again at the bottom right. This pattern repeats all the way up the photo for each differential pair.
You may be astute to notice that the bottom coupler trace connects to a pad at each end, while the mate coupler, above it does not. When two, coplanar parallel traces are in close proximity to one another, there are two types of crosstalk generated; backward or Near-End crosstalk (NEXT); and forward or Far-End crosstalk (FEXT).
As the transmit signal propagates, from left to right in the photo, the rising edge of the signal initiates NEXT at the beginning of the coupled length. The NEXT voltage saturates after a critical length equal to the risetime divided by twice the propagation delay; where the risetime is in seconds, and propagation delay is in seconds per unit length. It stays saturated for twice the time delay of the coupled length. Because of differential signalling, the NEXT voltages are of opposite phase on the respective couplers.
At the coupler pin, there is a reflection caused by the via. Since the couplers, at the far-end, are not terminated, in the characteristic impedance, and left open, any secondary reflections due to coupler via reflects back towards the receiver, again with opposite phase. When both reflections arrive back at the receiver, they will add together and add additional noise to the eye, causing inter-symbol interference, as shown by the shoulder in Figure 3(A). By leaving one end open, and shorting the other one to ground, means that any secondary noise will have the same phase, and when they arrive at the receiver, they will cancel, thereby eliminating the inter-symbol interference and increasing the eye amplitude as shown in Figure 3(B).
You will notice that the eye waveforms do not resemble the traditional eye diagram we are used to seeing. Instead we observe a typical NEXT eye, when the coupled length is short, compared to the bit time. There is also a line right in the middle.
Figure 4 can help to explain the reason. The blue waveform is the NEXT voltage, seen at the near-end of the coupler, in response to the red transmitted waveform. Notice that there are only pulses at an edge transition of the transmitted waveform. A rising edge creates a positive pulse, and a falling edge generates a negative pulse. The duration of each pulse is twice the time delay of the coupler length.
The receiver uses simple peak-detectors and latch to regenerate the signal back to the original waveform. A positive going pulse is detected by the positive peak-detector. When it crosses the positive voltage threshold (+Vth), it sets the latch output to logic high. The output remains high until a negative pulse crosses the negative threshold (-Vth), of the negative peak-detector, and resets the latch to logic low.
And that is how crosstalk can be your friend! Of course the small coupled crosstalk signal means we have to guard against CROSSTALK from other digital signals on board. But that’s nothing that mixed signal layout design rules can’t solve. ……Wait a minute! ……We both share the same enemy? …….. Who would have thought an old Proverb, “The enemy of my enemy is my friend” [sic], would apply here too?
Reference:
[1] L. Simonovich et al, U.S. Patent 6,091,739, “HIGH SPEED DATA BUS UTILIZING POINT TO MULTI-POINT INTERCONNECT NON-CONTACT COUPLER TECHNOLOGY ACHIEVING A MULTI-POINT TO MULTI-POINT INTERCONNECT.”
[2] J. Williamson et al, U.S. Patent 6,016,086, “NOISE CANCELLATION MODIFICATION TO NON-CONTACT BUS.”
[3] Alexandre Guterman, Robert J.Zani, “Point-to-Multipoint Gigabit Backplane Design”, IEEE International Symposium on EMC, May 11-16, 2003.
Backplane High Level Design –the Secret to Success
In a previous design note on Backplane Architecture and Design, I touched briefly on the concept of a Backplane High Level Design (HLD). In this design note, I will touch on key aspects that go into this process, using a simple fictitious system architecture as a straw-man, to demonstrate the principle.
For any new backplane design, I always recommend starting with a HLD. It helps you capture your thoughts, in an organized manner, and later provides the road map to follow for detailed design of the backplane. It also facilitates concurrent design of the rest of the system by the rest of the design team.
I like to use PowerPoint to capture the HLD information, but any other graphical based tool could be used. Later on in the design process, the drawings in the HLD document are reused in a more formal design specification document.
One of the first things I do, when coming on board a project, is capture the system architecture in a series of functional block diagrams starting from the high-level system block diagram, as shown in Figure 1. This is an example of what you might receive from the system architect at the beginning of a project.
Each block diagram details how the respective circuit packs, or other components of the system, interconnect to one another; complete with the number of signal I/Os for that function. For example, Figure 2 below shows the system data path and system control plane block diagrams. It illustrates one possible way of how you would arrange the circuit pack blocks, as they would appear in a shelf, when viewed from the front. Whenever possible, I like to arrange the blocks this way, because it presents a consistent look and feel throughout the documentation; from mechanical views, to connector placement, and route planning.
Preliminary Route Planning:
After all the functional block diagrams are completed, I usually go through a preliminary route planning exercise. The idea here is to gain some intuition for the final routing strategy, and to uncover any hidden issues that may surface down the road.
This is the most crucial step in any backplane design. Usually at this stage of the project, the system packaging architect is busy developing the shelf packaging concept, and is looking for feedback on connectors and card locations, so he (or she) can complete the common features drawing. The common features drawing defines all the x-y coordinates for all connectors and other mechanical parts on the backplane.
An example of a preliminary routing plan strategy diagram is shown in Figure 3. Each color represents two routing layers; for a total of 6 layers. The heavy black lines represent the high-speed serial link bundles of the data path; routed completely from SW1 and SW4 to LC1-10. The partially routed heavy red and blue lines, follow the exact same route plan as the heavy black lines, except they terminate to the respective color-coded SW cards. The beauty of this comes later, when the actual routing of the backplane takes place. Because the routing is identical, except for the source and destinations, it is a simple copy and paste exercise to replicate the routing on 5 of the 6 layers. The only editing required is at each end of the links. As you can appreciate, this is a huge time saver in completing the final layout!
When the preliminary route plane is complete, a pin-list summary for each circuit pack is compiled using an Excel spreadsheet. The pin-list summarizes the minimum number of pins needed per circuit pack for the function. Later on, it helps to drive the selection and number of connectors.
After completing the preliminary route planning exercise, and pin-list summary, you will gain a sense for:
-
the number of routing layers you will need
-
circuit pack connector signal grouping and partitioning
-
connector selection criteria for density
-
minimum vertical routing channel space needed between connectors
-
worst case topologies for signal integrity analysis
Backplane Connector Selection:
Large companies invest a lot of money and time to qualify a connector family. There is always strong pressure to reuse connectors from one system design to another because of cost. Qualifying a new connector is no trivial task. It takes a significant development effort to model, characterize and test the connectors. If you try to qualify a new connector, at the same time as designing a new system, you run the risk of delaying the overall program if serious issues develop along the way. Sometimes though, reusing the same connector just won’t cut it. For whatever the reason, one day you will be forced to look at other connectors.
Choosing the right connector for any new system is the most important aspect for any backplane design; regardless if it is reuse of a previous connector, or looking at new ones. The connector is the lifeblood of the backplane because it ultimately drives minimum slot pitch and circuit board height. It must be capable of supporting current and next generation high-speed signaling standards, and be robust enough to withstand multiple insertions. Factors such as pin density, pin pitch, pairs per row, overall size, skew, and crosstalk are examples to consider in this process.
Preliminary Stack-up:
In any high-speed serial link architecture, the data plane links are the most critical signals. They are the ones that usually define the total number of routing layers for the final PCB stack-up. When we include 4 layers, for redundant power distribution, to the 6 routing layers, the minimum number of layers for the backplane will be 18 layers as shown in Figure 4.
The right half of the figure gives counter-bore details. Another name often used is back-drilling. It is a common procedure done on backplanes to minimize via stubs, which is a killer for multi-gigabit serial links.
Detailed Route Plan:
Usually, around this time in the project schedule, the mechanical architect has put together a preliminary common features drawing, showing the preliminary connector placement. We use this drawing as a template to do a more detailed routing plan analysis.
By studying the preliminary route plan and pin-list, we can come up with a strategy to organize and partition the signals within the connector, and perform a more detailed routing analysis. This process can take a few iterations before it is optimum, but eventually, we end up with a more detailed routing plan as summarized in Figure 5. Each illustration here represents two routing layers per drawing. One layer is for Tx and the other is for Rx.
Vertical Routing Channel Analysis:
Before we sign-off on connector placement and route plan though, we need to verify there is enough space between connectors for the vertical routing channels. Otherwise, this may be a deal breaker for the chosen connector; slot pitch; total number of layers; or even the whole system packaging concept. If you do not have enough space here, there will be compromises needed somewhere else to accommodate it. The worst case scenario is having to double the number of layers, or having to choose a higher cost connector.
Signal Integrity Analysis:
Finally preliminary channel simulations must be done before we can sign-off on the backplane physical architecture concept. Now that all the detailed routing analysis is complete, we can easily establish several topologies to analyze.
One example of a worst case reference topology is highlighted in Figure 6. During this stage, we use Manhattan distance to estimate trace lengths.
After procuring the connector models, and developing circuit models to represent the via structures, I like to use Agilent ADS to capture and simulate the topologies. An example of the circuit topology, and simulation results are summarized in Figure 7.
Here, the topology was simulated at 10GB/s. The S-parameters are compared against the IEEE 802.3 10BaseKR spec. You would normally do this for every topology of interest. Later on, during the detailed design phase of the program, I would get 3D models of the vias built and use actual routed lengths from the backplane and circuit pack cards to confirm the design.
Summary:
Hopefully by now, you can appreciate the backplane architecture and design can be a complex beast to tame, and get right the first time. There are many complex interrelated steps that require the due diligence and meticulous planning to be successful. We have only scratched the surface here. You can download the full white paper titled, “ Backplane Architecture High-Level Design” , from which this design note is based upon, and an example of the PowerPoint HLD document from our website at: www.lamsimenterprises.com.
If you would like more information on our signal integrity and backplane services, or how we can help you achieve your next high-speed design challenge, email us at: info@lamsimenterprises.com.
Backplane Architecture and Design
According to Wiktionary, an Architect is: “A person who plans, devises or contrives the achievement of a desired result.” Because the backplane is the key component in any system architecture, the sooner you consider the backplane’s physical architecture near the beginning of a project, the more successful the project will be. If you think about it in the same way as designing a building, you would never consider building it without first engaging a building architect to plan and oversee the detailed design. Likewise, the backplane architect plans and oversees the physical backplane design before any layout is ever started. He or she works closely with a system-packaging engineer to satisfy the system requirements before any concept becomes final. Sometimes the original system architecture needs revisions due to physical limitations the backplane imposes. This can only be established with due diligence and planning during the high-level design stage.
Unlike other circuit pack designs used in the system, the backplane is much like the keel of a ship of which the rest of the ship’s construction depends on for support and structural integrity throughout its lifetime. Backplanes need to be right the first time so that circuit packs can interoperate together day one and be capable of supporting future system upgrades as technology advances. Once the system has been deployed into the field, it is next to impossible to change the backplane to correct any deficiencies or to upgrade for performance like you can by redesigning the plug-in circuit packs.
The seasoned backplane architect is a unique individual usually tasked to turn the system architect’s ideas and dreams, like the system block diagram example shown to the left, into reality. An often-misunderstood profession, backplane architects wear many hats to accomplish their goals. Often they must juggle the design requirements from many disciplines and decide on the best trade-offs for the final design. They must converse fluently with system architects, mechanical designers, circuit pack designers, connector suppliers, PCB layout designers, ASIC/FPGA and software engineers. They must be organized and meticulous in their documentation and design. But, most importantly, they must have a sound knowledge of mechanical, PCB layout/fabrication, signal integrity, power and EMC issues.
The greatest danger in leaving the backplane design as an afterthought is the connector selection and pin-out definition. If left to system packaging engineers and board designers to define, they may not be optimum for either performance or system cost. Many times system architects and packaging engineers will merely take the total number of signals and choose a connector with the highest pin density per inch without considering PCB routing or signal integrity implications. Inefficient routing of the traces leads to an increase in layer count and results in a thicker board. Thicker boards leads to higher hole aspect ratios and longer vias affecting high speed performance. Additional layer count impacts common equipment cost.
The high-level design stage is where the physical backplane architecture starts to take shape. It uncovers potential layout routing issues and gives you the confidence the design will work the first time. The importance of this stage cannot be overstated. It primarily drives these key activities:
-
Sanitizes the system architecture.
-
Defines the final selection of appropriate connectors.
-
Defines the connector signal partitioning and circuit pack pin-outs.
-
Provides the routing plan and design rules for layout.
-
Defines the net topologies for signal integrity analysis and link budgeting.
-
Facilitates the mechanical design of shelf and system packaging.
-
Defines the minimum slot pitch for optimum routing channels.
-
Facilitates early circuit pack floor planning and final card size.
-
Facilitates ASIC and FPGA pin selection for optimum routing to backplane connectors.
-
Estimates PCB layer count and board thickness.
-
Establishes an estimate for system cost of goods to support the business case.
Proper route planning and connector pin-out definition is vital for optimum performance. When done correctly, the final schematic capture and actual PCB layout will flow smoothly with no surprises. As an example, the left half of the figure (labeled HLD Plan) shows a sample of an inner layer high-level design route plan I did using Framemaker as the drawing tool on a design before any schematic was ever captured or pin-outs defined. Everything was planned from the number of layers to how the tracks needed to break out of the connector fields. The right half of the figure is the actual layout done in Cadence Allegro showing the inner layer routing of the artwork. The due diligence done in the high-level design stage made the actual layout fairly trivial. If you forgo this step, the worst-case scenario is the project will need to be reset to redesign shelf mechanicals or redefine card pin-outs causing delay in meeting time to market objectives and ballooning R&D costs. It’s a classic case of pay me now or pay me later.
At Lamsim Enterprises Inc., we can help you with these or any other design challenges you may have by providing innovative signal integrity and backplane solutions. Visit us at our web site at: lamsimenterprises.com .
Backplane Architecture Terms and Definitions
The following is a list of common terms and definitions associated with system architecture and backplane design:
Backplane
A backplane is a multi-layered printed circuit board assembly serving as the backbone of a system. Its purpose is to interconnect several printed circuit board assemblies called circuit packs or cards using plug in connectors to form a complete system. These cards plug into one side of the shelf assembly. Usually in mission critical system applications like central office telco or data centers, the backplane is passive meaning it does not contain active semiconductor devices permanently attached as part of the final assembly. Usually only connectors are the only components, but occasionally capacitors and resistors are also used. Active backplanes on the other hand, contains active components and often found in enterprise or consumer grade applications
Midplane
A midplane is similar to a backplane in function except that the circuit packs plug into both sides of the shelf assembly. In these systems, cards with I/O cabling from the faceplate plug into one side of the shelf, while non-I/O circuit pack plug in on the other side. Some midplane architectures have the front card plugged in orthogonally to the rear cards for high speed applications.
Parallel Bus Topologies
Parallel bus topologies carry data words in parallel on multiple traces from card-slot to card-slot across a backplane or from chip to chip on a circuit pack. Up until the late 1990’s, most system architectures used this form of interconnect. Due to signal integrity and timing issues associated with some parallel bus architectures with 10 to 16 card slots, the speed of the bus was limited to 25-66 MHz Two popular industry standard systems still using parallel busses today are CompactPCI and VMEbus.
The main issue with a parallel bus topology is fault tolerance where a single point of failure on the bus can bring down the entire system. Mission critical systems often had to employ redundant busses to guard against single point failures.
As performance demand increased, newer high speed system architectures were designed using serial technology in a point-to-point or point-to-multi-point switched fabric topologies.
Switched Fabric
Switched fabric, or just plain fabric, is the term most popular used in telecommunications and high-speed networks, including InfiniBand, Fiber Channel, PCIe, ATCA and other proprietary fabric based architectures. In these architectures, all data passes through the fabric before continuing to its destination. It offers better total throughput than parallel busses because traffic is spread across multiple physical links. It manages and controls all functions of the network and acts as a repeater for the data flow.
Single Star Topology
Star topologies are one of the most common high-speed serial topologies used in networks today. The advantage is it reduces the chance of network failure by connecting all of the systems to a central node. A failure of a link from any peripheral node to the central node results in the isolation of that peripheral node from all others. As a result, the rest of the systems remain unaffected.
In its simplest form, a single star topology consists of one central hub node interconnected point-to-point to other peripheral nodes resembling a spoke wheel or star configuration. When implemented in a backplane, the central node is usually the switched fabric card and the peripheral nodes are line cards. The fabric card switches messages between the other line cards in the network. The line cards usually have faceplate I/O connectors to connect to other shelves in a network.
The main disadvantage with a single star topology is high dependence of the system on the functioning of the central fabric. Failure of the fabric card can bring down the entire system. Because of this, mission critical systems employ two fabric cards for redundancy in a dual star topology configuration.
Dual Star/Multi-star Topology
The dual star or multi-star topology is similar to the star network topology except it has two or more central hub nodes interconnected point-to-point to other peripheral nodes. When implemented in a backplane application, these central nodes are usually the switched fabric cards and peripheral nodes are the line cards. The additional fabric(s) provides redundancy in mission critical system applications in case of failure, or for upgrading fabric card hardware.
Fully Connected Mesh Topology
A fully connected mesh topology, when applied to a backplane application, does not have one central fabric node(s) as in the case of star topologies. Instead, each line card node connects with all other line card nodes forming a mesh. Its major disadvantage is the number of connections grows significantly with the number of nodes. This requires additional backplane connector pins and layers to interconnect them. Because of this, it is impractical for large systems and only used when there are a small number of cards needing to be interconnected.
Bert Simonovich's Design Notes 