Re: [HSSG] The List

[Paul Kolesar <PKOLESAR@xxxxxxxxxxxx>]

Jack,
thanks for taking the time to provide
deeper explanations of some details.

I understand the comparison you made
between a pair of SNAP12s and a pair of QSFPs.  I do not dispute that
the pair of QSFPs involve more optical alignment steps.  But this
labor content is offset by the QSFP having smaller optical ICs with commensurately
higher yield, and fewer elements to align per IC that should reduce the
amount of labor per IC.  Additionally, and perhaps more to the point
of the HSSG applications, with a 4-lane application, like 40G, only a single
QSFP is used, not a pair.  So while the SNAP12 may be more efficient
per lane, that efficiency does not necessarily offset the cost of the unused
capacity it offers for applications that need only four lanes.  While
it would be possible to occupy all twelve lanes to deliver 40G, such as
is the case with the currently available SNAP12 modules, the electrical
muxing device that would be required between these twelve optical lanes
and the fewer electrical I/O lanes that most future designs would target,
would provide a less favorable cost scenario.  This last point also
speaks to the item that Frank Chang raised some days ago (on June 28) about
using SNAP12 for 40G.  

To me a key goal is optimally matching
the technology choices to the application.  One optimization is lane
rate matching.  At this point in the technology evolution of electronics
and optics, if the electrical lane rate has advanced to 10G, then the optimal
optical lane rate should not be lower since optical solutions commonly
operate at 10G today.  

Regards,
Paul Kolesar
CommScope Inc.
EnterpriseŽ Solutions
1300 East Lookout Drive 
Richardson, TX 75082 
Phone:  972.792.3155
Fax:      972.792.3111
eMail:   pkolesar@commscope.com

"Jack Jewell" <Jack.Jewell@picolight.com>

07/03/2007 11:02 AM

To	<PKOLESAR@SYSTIMAX.COM>, <STDS-802-3-HSSG@listserv.ieee.org>
cc
Subject	RE: [HSSG] The List

Paul,
 
I admire your willingness to take
on this difficult task and your efforts to get the root of the matter.
 You're right about cost being an uncomfortable subject, especially
in the margin-challenged module business.  But I expect anyone involved
with shipping both SFP's and parallel modules in volume will agree that
the SFP platform is the most cost-effective one to transceive a given number
of Gb/s.  Just take the cost of a 4G SFP and ask if a 4Gb/s/ch QSFP
(or POP4) can be made for less than 4X that cost.  The same will be
true at 10Gb/s/ch, due to very high volumes in 8GFC.  Only with comparable
volumes and quite a few years of technology maturation will parallel modules
be able to approach or even surpass the cost-effectiveness of the SFP platform.
 
The seemingly inverted TOSA cost
(a 1x4 vs 4 singlets) does NOT apply to the Tx/Rx ICs.  A quad or
a 12-element driver or receiver IC utilizes essentially the same process/package/test
technology as a singlet, with less handling and labor and chip area.  Thus
a NX array IC is less than N times that of a singlet IC - unless there
is a large disparity in volume/demand/availability.  Singlet and array
 TOSA/ROSA packages are quite different.  You can find dozens
of contract manufacturers throughout the far east that perform high-quality
TO-can packaging for extremely low cost.  The techniques are known,
equipment is prelevant, barrier to entry is low.  Not yet so for array
packaging.  Equipment is custom.  CM's with know-how are few
and far between; ones with experience are even fewer.  For QSFP, there
is the additional task of mounting the VCSEL and PIN chips with 1's of
um registration accuracy (to align with the same fiber ribbon).  Alignment
has more degrees of freedom than for a serial TOSA, which must be dealt
with.  All this in volumes that are on the order of 1% (probably far
less, actually) than that of TO-cans.
 
The above is not meant to paint
a dismal picture for parallel.  On the contrary, parallel is the appropriate
approach to deliver the interconnection density and integration required
for 100GigEthernet and many other bandwidth-intensive applications.  4
SFP's occupy roughly 3X more width than a QSFP; comparing widths between
12 SFP's to a pair of SNAP12's gets downright ludicrous.  But there
is the cost barrier to array packaging discussed above.  That barrier
occurs as soon as you get away from a singlet, e.g. a 1x2.  As you
go to higher integration levels, the cost barrier is spread out over more
channels and per/channel cost decreases.  Compare a "4+4"
parallel approach (e.g. QSFP or POP4) vs a "12x" approach (e.g.
SNAP12 or any other that aligns a 1x12 array).  Two QSFP modules require
the mounting of 4 optical ICs (2 VCSEL arrays and 2 PIN arrays) and 4 electrical
ICs (2 drivers and 2 receivers).  In each case, the VCSEL and PIN
arrays must be mounted to 1's of um accuracy in registration.  The
two QSFPs can deliver 80Gb/s.  In contrast, a Tx/Rx pair of SNAP12's
require the mounting of only 2 optical ICs and 2 electrical ICs.  Registration
between optical ICs is not an issue since only one optical IC aligns to
a fiber ribbon.  The SNAP12 pair should cost less than the QSFP pair
since it involves fewer parts and is simpler, i.e. it's MORE INTEGRATED.
 Yet the SNAP12 pair delivers 100Gb/s (or even 120Gb/s), compared
to 80Gb/s.  Reduced costs for parallel optical technology will occur
and will arrive sooner if the industry settles on an appropriate level
of integration - not too high and not too low.  Given the (relative)
prevalence of 12-fiber ribbons and MTP connectors and modules in the field
and the match to 100Gig Ethernet, I think 12 Tx and 12 Rx is the appropriate
level of integration.
 
Cheers,
Jack

---

From: Paul Kolesar [mailto:PKOLESAR@SYSTIMAX.COM]
Sent: Monday, July 02, 2007 5:10 PM
To: STDS-802-3-HSSG@listserv.ieee.org
Subject: Re: [HSSG] The List


Jack, 
I appreciate your detailed responses.   

Regarding your statements that I am filling a role that does not match
my affiliation, I have to agree.  But I am filling a role that needs
to be filled.  I am trying to stimulate the discussion in an area
that has become one of the remaining factors that will set direction for
the next several years work.  There are certain dynamics that stifle
this type of discussion from occurring to the level necessary between the
expected parties so that folks like myself can have sufficient understanding.
 For example, cost is not usually a comfortable subject to explore
in detail within a competitive arena.  So I offer my analysis/contributions
and questions/comments as a vehicle to draw out controversy and agreement,
and I welcome more transceiver experts to join in.

Regarding your comments on item 5), it is clear that more work needs to
be done on the content of kolesar_01_0507.  I will look to submit
a revised version with more explanation for the July meeting.  Part
of the problem is that there is no explanation for the approach I took,
which I think has lead to misunderstandings.  The first order explanation
is that the XFP column is the basis for the transceiver component comparisons
and each row is independent.  That means that each element of the
XFP components is rated 4x (times 4 because there are four XFPs used for
4x10G LAG), regardless of their relative cost to each other.  The
only row were these are summed together is in the "weighted average".
  

I am surprised regarding your assertions that four discrete OSAs cost less
than one 4-element array OSA, albeit with volume effects factored in.  I
sure hope this relationship does not come to haunt the IC industry, or
we'll have quad devices costing more than four times their single counterparts,
and integration will fail to be a means to lower cost.  

Regards,
Paul Kolesar
CommScope Inc.
EnterpriseŽ Solutions
1300 East Lookout Drive 
Richardson, TX 75082 
Phone:  972.792.3155
Fax:      972.792.3111
eMail:   pkolesar@commscope.com

"Jack Jewell" <Jack.Jewell@picolight.com>

06/29/2007 05:58 PM

To	<PKOLESAR@SYSTIMAX.COM>, <STDS-802-3-HSSG@listserv.ieee.org>
cc
Subject	RE: [HSSG] The List

Thanks Paul for undertaking the task of this cost analysis.  I must
say though, it's most curious that the task of 40G module cost analysis
has been left to a fiber supplier.  You're truly dedicated.  I've
been traveling without email access, hence the delayed response.
  
My responses are in red.
 For you CrackBerry Warriors without colored text, the responses start
on the next line following Paul's questions, preceded by JJ
- . 
  
Jack

---

From: Paul Kolesar [mailto:PKOLESAR@SYSTIMAX.COM]
Sent: Wednesday, June 27, 2007 4:57 PM
To: STDS-802-3-HSSG@listserv.ieee.org
Subject: Re: [HSSG] The List


Jack, 
thank you for chiming in.  We need more of this type of input.  Your
response is provocative on several points, so consequently I have some
questions and comments about your statements:  

1) First, SFP+ achieves the 300m SR objective with similar ease as XFP.

What is absorbing or mitigating the jitter that is handled by the CDR in
the XFP?  Is it an external CDR, and EDC function, reduced temperature
range, or improved signal quality from the upper level divides to which
the SFP+ interfaces?  Or was the presence of the CDR in the XFP unwarranted
in the first place?  Or are you saying that SFP+ supports a 300m link
despite being noncompliant to 10GBASE-S specs?  
JJ - The SFP+ architecture utilizes Tx pre-emphasis and Rx equalization
in the PHY IC, and limits the FR4 traces to ~8" rather than 12".
 XFP does not need these features due to the CDR.  Modern PHY
chip include these features with only incremental cost, but they were not
available when XFP was defined.  The comments directly relate to the
SFP+ limiting version (no EDC), which is analogous to the XFP.  With
EDC, the comments still apply, and additional benefits are available.

2) While a reduced-reach 10G PMD might reduce costs, it's a retreating
approach that requires the customer to purchase higher-cost transceivers
to fill in the reach gap, e.g. for reaches between 100m and 300m.

I completely agree that this is a retreating approach that is not a preferred
solution if it can be avoided, especially if all that is required is a
simple (low cost and power) EDC function built into the receiver's TIA.
   
JJ - Check.  EDC
functionality however is not in the receiver's TIA, but on the PHY chip
(better yet!).  The TIA simply needs a linear response.

3) A reduced operating temperature range offers immediate cost savings
for some customers without compromising reach and without any new standards
specifications. 

I do not think that the 802.3 standard specifies operating temperature
range.  If so, this improvement is available without violation of
the 10GBASE-S spec.  But then there is the market requirement on temp
range to consider, which can be just as imposing as the standards specs.
 This leads back to the questions of item 1.  
JJ - That's why it involves no new standards specifications and also why
only "some" customers are able to enjoy this benefit.

4) For QSFP,.  Here again, EDC offers great advantage, and ignoring
it would be a mistake.  But crosstalk introduces module-level limitations
that might motivate reach reduction or other modification to the PMD.

The crosstalk in a transceiver is another source of jitter.  I accept
that the crosstalk in a single lane transceiver is likely to be easier
to mitigate than that in a multi-lane transceiver that shares a common
housing/circuit board between the transmit and receive functions. The degree
of added impairment is the issue, and whether the jitter impairment causes
a reduction in reach or other parameter trade-offs will need to be sorted
out during the setting of specifications.  But if EDC and reduced
temp range are means to lower cost for SFP+, then they will be as well
for QSFP.  I think this may need to be sorted out in the lab as companies
design these components into their test fixtures and systems.  That
is why I encourage those with experience in this area to afford us the
benefit of that experience.  Existence proof is a powerful argument.
 
JJ - Of course EDC and reduced temp range benefit SFP+ and QSFP (and anything
else) about equally, hence my comment on EDC. For
a parallel module at 10G/ch, crosstalk and power variation add to the difficulty
and motivate some relief elsewhere.  That is why although I'm fine
with a 300m reach for 10G serial, I support a shorter reach objective for
10G parallel without EDC.  Adding EDC provides a tradeoff between
further spec relaxation and increased reach.

5) Comparing costs, the SFP+ will be lower than QSFP for the foreseeable
future (per lane for the same spec). 

Does this claim rely on volumes favoring SFP+?  From what I can tell,
the intrinsic costs favor QSFP relative to 4xSFP+.  If you disagree
with this, please provide the particulars of where we differ by addressing
the constituent comparisons in kolesar_01_0507.  
JJ - This claim is based on the reality of present SFP+ costs and
lack of any visible path for
QSFP to reach parity.  Making a cost comparison for equal volumes
is irrelevant when the reality is that one module will continue to have
far higher volumes than the other - again for the forseeable future.  Before
there is any 40G Ethernet market, the tsunami of 8G Fibre-Channel, utilizing
10G technology, will drive down the costs for the serial ICs.  The
balance of TOSA/ROSA costs between 4xSFP+ and QSFP is reversed on your
chart.  I think you underestimate the impact of product maturity on
TOSA/ROSA packaging costs as well.  Also in the 4xSFP+, the PCB and
Pkg costs are too high. 

Thanks for helping to increase understanding of these issues.

Regards,
Paul Kolesar
CommScope Inc.
EnterpriseŽ Solutions
1300 East Lookout Drive 
Richardson, TX 75082 
Phone:  972.792.3155
Fax:      972.792.3111
eMail:   pkolesar@commscope.com

Jack Jewell <Jack.Jewell@PICOLIGHT.COM>

06/27/2007 11:41 AM

Please respond to
Jack Jewell <Jack.Jewell@PICOLIGHT.COM>

To	STDS-802-3-HSSG@listserv.ieee.org
cc
Subject	Re: [HSSG] The List

Input from a transceiver vendor with experience and interest in both serial
and parallel modules: 
 
First, SFP+ achieves the 300m SR objective with similar ease as XFP.  While
a reduced-reach 10G PMD might reduce costs, it's a retreating approach
that requires the customer to purchase higher-cost transceivers to fill
in the reach gap, e.g. for reaches between 100m and 300m..  A reduced
operating temperature range offers immediate cost savings for some customers
without compromising reach and without any new standards specifications.
 Use of a linear receiver and EDC and relaxed Tx specs can also achieve
the 300m reach at reduced cost.  For this application, the amount
of compensation required is far less than what is required for LRM.  EDC
is becoming widely available, to the point of being a standard feature
of PHY ICs, so the cost of this EDC is becoming insignificant.  Forward-looking
standards efforts will achieve their full impact by making use of EDC.
 
For QSFP,.  Here again, EDC offers great advantage, and ignoring it
would be a mistake.  But crosstalk introduces module-level limitations
that might motivate reach reduction or other modification to the PMD.
 
Comparing costs, the SFP+ will be lower than QSFP for the forseeable future
(per lane for the same spec).  The only advantage of QSFP over SFP+
is density and possibly simpler cabling.  However, a 12-channel parallel
module pair, e.g. SNAP12, offers greater density than QSFP and 2.5-3X the
bandwidth at <2X the cost.   
 
Jack

---

From: Dove, Dan [mailto:dan.dove@HP.COM]
Sent: Wednesday, June 27, 2007 9:32 AM
To: STDS-802-3-HSSG@listserv.ieee.org
Subject: Re: [HSSG] The List

Paul, 
 
Regarding SFP+, I am very familiar with the technology and have been tracking
the SFF-8431 development. The architecture re-distribution of cost that
SFP+ offers will have a substantial impact on cost, especially when combined
with the higher density we can achieve with smaller geometry ASICs and
multiport PHYs that will come with it. 
 
As for QSFP, I am less familiar with whether or not it will provide a cost
improvement over SFP+ or be capable of meeting the existing SR spec. This
is something for the QSFP experts to consider, but like I said, a shorter
10G PMD might be the avenue to take rather than an identity challenged
40G spec. 
 
Regarding LAG, my conversation with HP Server architects indicates there
are a number of avenues for improvement of LAG under development.
 
I cited these areas in my earlier message and would appreciate them being
addressed rather than ignored. 

Dan

---

From: Paul Kolesar [mailto:PKOLESAR@SYSTIMAX.COM]
Sent: Wednesday, June 27, 2007 8:04 AM
To: STDS-802-3-HSSG@listserv.ieee.org
Subject: Re: [HSSG] The List


Dan, 
I can't tell how successfully SFP+ and QSFP will be at meeting the existing
10GBASE-S spec.  If they can, a new shorter distance 10G PMD would
not be of value.  If they can't, then a new PMD spec may be worth
while.  Those attempting to implement these lower cost platforms need
to weigh in to provide guidance.  In the event that either the QSFP
and/or SFP+ can meet 10GBASE-S specs in multiple vendor's platforms, or
that a new shorter distance spec is developed that allows lower cost, the
performance issues of LAG will remain.  I believe Howard's presentations
on LAG have indicated that improving LAG would not be without compromise,
leading me to conclude that, however improved, LAG performance could not
become equivalent to a 40G pipe.  Developing a 40G spec would ensure
a solution that simultaneously addresses these cost and performance issues.

Regards,
Paul Kolesar
CommScope Inc.
EnterpriseŽ Solutions
1300 East Lookout Drive 
Richardson, TX 75082 
Phone:  972.792.3155
Fax:      972.792.3111
eMail:   pkolesar@commscope.com

"Dove, Dan" <dan.dove@HP.COM>

06/26/2007 09:24 PM

Please respond to
"Dove, Dan" <dan.dove@HP.COM>

To	STDS-802-3-HSSG@listserv.ieee.org
cc
Subject	Re: [HSSG] The List

Hi Paul, 

Good points. I was not really expecting to see a significant cost differential
at the PMD although its a good argument that a 100m PMD would be less expensive.
If this is the case, why not do another 10G PMD focused on lowering the
cost of server interconnect? I believe that would be a smaller project
and have a much less significant impact on 100G development.

Thanks, 

Dan

---

From: Paul Kolesar [mailto:PKOLESAR@SYSTIMAX.COM]
Sent: Tuesday, June 26, 2007 6:16 PM
To: STDS-802-3-HSSG@listserv.ieee.org
Subject: Re: [HSSG] The List


Dan, 
thanks for your detailed thoughts and proposals.  I appreciate the
points you made regarding the volume effect of 10G components on the cost
comparison.  The presentation I submitted for the May interim looked
at the intrinsic cost factors and did not attempt to include volume in
the equation.  But volume certainly can be a significant factor.  Your
suggestion to look into its impact when comparing 4x10G LAG to 40G is reasonable,
but complicated at the PMD level.  As my May presentation shows there
are a few ways to implement LAG on MMF.  One uses the XFP, another
the SFP+, still another the QSFP.  Today the XFP is shipping to the
10GBASE-S spec, and supports 300m transmission.  Designs using SFP+
and QSFP will be more challenged to meet this spec due to jitter, so it
remains to be seen how successfully these lower cost form factors can substitute
for the XFP in 10GBASE-S compliant LAG.  However, a reduced distance
requirement, such as that stated in the HSSG objectives, would greatly
improve the chances that QSFP will suffice for "40GBASE-S".  So
while volume is important, these unanswered questions on suitability make
it impossible from my vantage point to determine how the volumes for 10GBASE-S
will be divided among XFP, SFP+, and QSFP.  And the effects of volume
on production costs are better left to those who manufacture the devices.
 Perhaps individuals with such insights will offer some scenarios.
  


Regards,
Paul Kolesar
CommScope Inc.
EnterpriseŽ Solutions
1300 East Lookout Drive 
Richardson, TX 75082 
Phone:  972.792.3155
Fax:      972.792.3111
eMail:   pkolesar@commscope.com

"Dove, Dan" <dan.dove@HP.COM>

06/26/2007 02:45 PM

Please respond to
"Dove, Dan" <dan.dove@HP.COM>

To	STDS-802-3-HSSG@listserv.ieee.org
cc
Subject	Re: [HSSG] The List

My fellow colleagues , 

Last week I sent out a list of items that I felt need to be addressed to
ensure that a 40G PAR would be justified. At a subsequent EA teleconference
intended to build concensus in the HSSG, I offered to review the presentations
made in support of 40G Economic Feasibility and comparing 40G vs 4x10 LAG
performance to ensure that I was not being too harsh in my consideration
of the material that was presented. 

Over the weekend, I reviewed every presentation I could find on these subjects
so that I could be comfortable that I was not being unfair in my concerns.
Fortunately, it was not a huge task as there are not that many to review.

After doing so, I found myself less convinced in the validity of
some presentations that were made. This statement is not made to criticize
my colleagues, but to honor the concept of peer review which requires that
we review and criticize, otherwise we might as well just upload them to
a server and forget about them. 

Specifically, I disagreed with cost arguments made on the assumption that
10G cost remains a constant, when in fact I anticipate substantial reductions
in 10G cost over the next few years at a rate much faster than today due
to a few factors; 

1) Higher density/lower cost optical form factors (SFP+) allowing better
utilization of switch infrastructural cost and QSFP for NICs.
2) Smaller geometry CMOS allowing higher port densities to work in synergy
with PMD cost reductions. 
3) Integration of XFI / SFI interfaces directly into ASICs or multi-port
PHYs driving 10G cost further downward. 
4) Higher volumes / commoditization of 10G driving cost down much faster
than the current trajectory. 

While 40G can leverage some of these elements, it cannot leverage the volume
that feeds the downward cost spiral. So in 4 years, a 40G switch port cost
is going to be based on low-volume, freshly designed and un-amortized silicon
used primarily for server interconnect, whereas a 10G port cost will be
based on amortized, high-volume silicon being used in a huge array of applications.
Having different trajectories, the relative cost for 40G will be higher
than presented. This is true for 100G as well, but who is arguing a need
for 100G based on cost? It is bandwidth that drives 100G demand.

In addition, I found presentations claiming that LAG was insufficient to
address server I/O bandwidth needs, yet those presentations failed to address
upcoming technology enhancements like TRILL and its impact combined with
I/O Virtualization, perhaps with a physical manifestation of QSFP and MPO
optics which I believe can lead to graceful performance scaling for servers
that does not demand an intermediate IEEE standard. In other words, activities
and technologies are advancing which will parse server network access into
multiple conversations that can then be put onto a LAG group with much
higher than presented performance levels. 

Now, I realize that I am swimming upstream here by asking that the proponents
for "40G now"  to complete a task that took the 100G proponents
almost a year to accomplish, in less than 6 months, but then I am not asking
them to do that. My first choice, the one I proposed in Geneva,
was that we move 100G forward (because it is DONE) and that we continue
to work on 40G (until it is done). 

This appears to be a minority position because apparently some people will
accept an unproven 40G proposal rather than risk 100G. Others think that
40G is proven sufficiently and are demanding "40G now" or they
will not allow a 100G PAR to go forward. Those in the latter camp must
either be unconvinced of my concerns, or they think my concerns are insufficient
to justify any further work being done to justify a 40G project. 

I can accept differences of opinion. 

What I cannot do, however, is pretend that these issues do not exist, or
that the work we would have to spend getting a 40G standard done is not
going to delay the much needed 100G aggregation solution our customers
demand. I cannot ignore what I perceive as holes in the 40G presentations.

So, to provide a little more direction to my colleagues in the "40G
now or the HSSG stalls" crowd, I am asking you to include relative
cost trajectories in your analysis of 40G vs 10G cost models, and to include
technology enhancements to LAG (TRILL, I/O Virtualization, QSFP, MPO) in
your performance analysis. 

If you feel that this is unnecessary, I am requesting that you communicate
this position to me as soon as possible so that I can prepare a presentation
on these areas of concern for the July meeting. 

Respectfully, 

Dan Dove 
Dove Networking Solutions - Serving ProCurve Networking by HP