Re: [802.3BA] 802.3ba XR ad hoc next step concern

["Abbott, John S Dr" <AbbottJS@xxxxxxxxxxx>]

The John Petrilla email asks whether there really is interest in longer lengths for the 100GbE MM solution.  Steve Swanson submitted a presentation to the XR ad hoc in June http://www.ieee802.org/3/ba/public/AdHoc/MMF-Reach/swanson_xr_01_0608.pdf  which summarized the responses of 20 data center Corning fiber customers, all of whom expressed the need for longer length. One customer indicated half the fiber links were over 100meters, and another customer with a data center of 550K sq ft indicated the infrastructure included the 100m-185m range. 

 

It at first seems hard to reconcile a customer response that “over half the links are >100m” with other presentations suggesting only 1% of the links are over 100m, so let me attempt an explanation.

 

1. Corning is surveying fiber customers and asking about installed fiber; the customers are responding in the context of OM3 fiber used for Ethernet and Fibre Channel.  There REALLY ARE a significant portion of the fiber links over 100m.
2. The surveys by Flatman and DiMinico on installed links referenced in http://ieee802.org/3/ba/public/jan08/flatman_01_0108.pdf include copper links as well.  Depending on the date of past surveys, 60-80% of the links reported are copper.  These are, not surprisingly, the shortest links, so including the copper data makes optical fiber look like a small part of the data center picture. These shorter links definitely need to be included in the thinking for 100GbE applications.  The HPC application is also a high number count, short-reach application, and the 100GbE HPC application represents a shift to using fiber instead of copper.  However, we need to be careful that this emphasis on the shorter links doesn’t lead us down a path to think we can just forget about 100m+ and a significant part of the current fiber infrastructure in data centers. 
3. I think that there really are two application ‘spaces’ – (1) short-distance applications where fiber is replacing longer lengths of copper at 100GbE, where number count is high and there is a need to eliminate any non-value-added cost, and (2) applications where MM fiber currently plays a role and a cost-conscious 100GbE solution is needed which can meet the distance requirement.  The discussions in the XR ad hoc (and John Petrilla’s comments) have looked for synergy between the two application spaces and have identified workable solutions.

 

Although the 100m solution works for replacing short (i.e. copper) links, the failure rate at longer lengths does not meet IEEE reliability when normalized by the number of fiber links (not the total number of copper and fiber links) .  The lowest cost solution for the 100m-150m distance is the same OM3 cable proposed for 100m but with one of the enhancements suggested by the XR ad hoc. 

 

Let me again emphasize that the 100m+ need is real today and the 100m+ market is real today. 

 

 

---

From: PETRILLA,JOHN [mailto:john.petrilla@xxxxxxxxxxxxx]
Sent: Wednesday, August 20, 2008 11:23 PM
To: STDS-802-3-HSSG@xxxxxxxxxxxxxxxxx
Subject: [802.3BA] 802.3ba XR ad hoc next step concern

 

Colleagues

 

I’m concerned that the proposal of creating a new objective is leading us into a train wreck.  This is due to my belief that it’s very unlikely that 75% of the project members will find this acceptable.  This will be very frustrating for various reasons, one of which, almost all the modules expected to be developed will easily support the desired extended link reaches, will be discussed below.

 

I don’t want to wait until our next phone conference to share this in the hope that we can make use of that time to prepare a proposal for the September interim.  I’ll try to capture my thoughts in text in order to save some time and avoid distributing a presentation file to such a large distribution.  I may have a presentation by the phone conference.

 

Optical modules are expected to either have a XLAUI/CLAUI interface or a PMD service interface, PPI.  Both are considered.

 

A previous presentation, petrilla_xr_02_0708, http://ieee802.org/3/ba/public/AdHoc/MMF-Reach/petrilla_xr_02_0708.pdf has shown that modules with XLAUI/CLAUI interfaces will support 150 m of OM3 and 250 m of OM4.  These modules will be selected by equipment implementers primarily because of the commonality of their form factor with other variants, especially LR, and/or because of the flexibility the XLAUI/CLAUI interface offers the PCB designer.  Here the extended fiber reach comes for no additional cost or effort.  This is also true in PPI modules where FEC is available in the host.

 

Everyone is welcome to express their forecast of the timing and adoption of XLAUI/CLAUI MMF modules vs baseline MMF modules.

 

To evaluate the base line proposal for its extended reach capability, a set of Monte Carlo, MC, analyses were run.  The first MC evaluates just a Tx distribution against an aggregate Tx metric.  This is to estimate the percentage removed by the aggregate Tx test.  The second MC evaluates the same Tx distribution in combination with an Rx distribution and 150 m of worst case OM3.  The third MC repeats the second but replaces the 150 m of OM3 with 250 m of worst case OM4.  Worst case fiber plant characteristics were used in all link simulations.

 

The Tx distribution characteristics follow.  All distributions are Gaussian.

  Min OMA, mean = -2.50 dBm, std dev = 0.50 dBm (Baseline value = -3.0 dBm)

  Tx tr tf, mean = 33.0 ps, std dev = 2.0 ps (Example value = 35 ps)

  RIN(oma), mean = -132.0 dB/Hz, std dev = 2.0 dB (Baseline value = -128 to -132 dB/Hz, Example value = -130 dB/Hz)

  Tx Contributed DJ, mean = 11.0 ps, std dev = 2.0 ps (Example value = 13.0 ps)

  Spectral Width, mean = 0.45 nm, std dev = 0.05 nm (Baseline value = 0.65 nm).

  Baseline values are from Pepeljugoski_01_0508 and where no baseline value is available Example values from petrilla_02_0508 are used.

 

All of the above, except spectral width, can be included in an aggregate Tx test permitting less restrictive individual parameter distributions than if each parameter is tested individually.  In this example distributions are chosen such that only the mean and one std dev of the distribution satisfy the target value in the link budget spreadsheet.  If the individual parameter is tested directly to this value the yield loss would be approximately 16%.

 

The Rx distribution characteristics follow.  Again, all distributions are Gaussian.

  Unstressed sensitivity, mean = -12.0 dBm, std dev = 0.75 dB (Baseline value = -11.3 dBm)

  Rx Contributed DJ, mean = 11.0 ps, std dev = 2.0 ps (Baseline value = 13.0 ps)

  Rx bandwidth, mean = 10000 MHz, std dev = 850 MHz (Baseline value = 7500 MHz).

 

For the Tx MC, only 2% of the combinations would fail the aggregate Tx test.

 

For the 150 m OM3 MC, only 2% of the combinations would have negative link margin and fail to support the 150 m reach.  This is less than the percentage of modules that would have been rejected by the Tx aggregate test and a stressed Rx sensitivity test and very few would actually be seen in the field.

 

For the 250 m OM4 MC, only 8% of the combinations would have negative link margin.  Here approximately half of these would be due to transmitters and receivers that should have been caught at their respective tests.

 

The above analysis is for a single lane.  In the case of multiple lane modules, the module yield loss will increase depending on how tightly the lanes are correlated.  Where module yield loss is high, module vendors will adjust the individual parameter distributions such that more than one std dev separates the mean from the spread sheet target value.  This will reduce the proportion of modules failing the extended link criteria. Also, any correlation between lanes results in a module distribution of units that are shipped having fewer marginal lanes than where the lanes are independent.    

 

So while there’s a finite probability that a PPI interface module doesn’t support the desired extended reaches, the odds are overwhelming that it does.

 

Then with all of one form factor and more than 92% of the other form factor supporting the desired extended reach, the question becomes, ‘what’s a rational and acceptable means to take advantage of what is already available?’  A new objective would enable this but, as stated above getting a new objective for this is at best questionable.  Further, it’s expected that one would test to see that modules meet the criteria for the new objective, set up part numbers, create inventory, etc. and that adds cost.  Finally, users, installers, etc. are intelligent and will soon find this out and will no longer accept any cost premium for modules that were developed to support extended reach - they will just use a standard module.  There’s little incentive to invest in an extended reach module development.

 

I’ll make a modest proposal: Do nothing – just hook up the link.  Do nothing to the standard and when 150 m of OM3 or 250 m of OM4 is desired – just plug in the fiber.  The odds are overwhelming that it will work.  If something is really needed in the standard, then generate a white paper and/or an informative annex describing the statistical solution.

 

Background/Additional thoughts:

 

Even with all the survey results provided to this project, it’s not easy to grasp what to expect for a distribution of optical fiber lengths within a data center and what is gained by extending the reach of the MMF baseline beyond 100 m.  Here’s another attempt.

 

In flatman_01_0108, page 11, there’s a projection for 2012.  There for 40G, the expected adoption percentage of links in Client-to-Access (C-A) applications of 40G is 30%, for Access-to-Distribution (A-D) links, it is 30%, and for Distribution-to-Core (D-C)links it is 20%.  While Flatman does not explicitly provide a relative breakout of link quantities between the segments, C-A, A-D & D-C, perhaps one can use his sample sizes as an estimate.  This yields for C-A 250000, for A-D 16000 and for D-C 3000.  Combining with the above adoption percentages yields an expected link ratio of C-A:A-D:D-C = 750:48:6.

 

Perhaps Alan Flatman can comment on how outrageous this appears.

 

This has D-C, responsible for 1% of all 40G links, looking like a niche.  Arguments over covering the last 10% or 20% or 50% of D-C reaches does not seem like time well spent.  Even A-D combined with D-C, AD+DC, provides only 7% of the total.

 

Similarly for 100G:  the 2012 projected percentage adoption for C-A:A-D:D-C is 10:40:60 and link ratio is 250:64:18.  Here D-C is responsible for 5% of the links and combined with A-D generates 25% of the links.  Now the last 20% of AD+DC represents 5% of the market.

 

Since the computer architecture trend leads to the expectation of shorter link lengths and there are multiple other solutions that can support longer lengths, activating FEC, active cross-connects, telecom centric users prefer SM anyway, point-to-point connections, etc., there is no apparent valid business case supporting resource allocation for development of an extended reach solution.