Brad
Please see my comment below in red
Brad Booth wrote:
9D1E2BDCB5C57B46B56E6D80843439EB04DC471A@SDCEXCHANGE01.ad.amcc.com"
type="cite">
That's definitely one way it can
be interpreted. There is some apples and oranges comparison going on
there, and I'm not sure how the VG got added to the 802.3 mix. ;-)
I do recall the battle of VG vs Base-T, definitely on paper and in
press VG looked very good with Base-T late start.
9D1E2BDCB5C57B46B56E6D80843439EB04DC471A@SDCEXCHANGE01.ad.amcc.com"
type="cite">
But with optics, I would agree
that the tougher choices haven't always been made, even when the
writing was on the wall. The biggest complaint with 10GbE was all the
possible port types (thank you WAN interface sublayer). The 10GBASE-SR
PHY is doing well in the market though, and that's partially due to the
fact that it is the only 850nm wavelength PHY for that space. Interestingly
enough though, implementations can be achieved with either linear or
limiting components.
The conclusion in this presentation given in 802.3ae with many
supporters
http://www.ieee802.org/3/10G_study/public/nov99/cornejo_1_1199.pdf
state "Suppliers will leverage new development in the SR(2Km) and
VSR(<500m) OC-192 applications to take advantage of economies of
scale
and reduce the cost to customer". The only leverage here was the name
SR where in IEEE it is associated with 850 nm where in SONET world
it is associated with 1.3 um laser
:-) . As you stated 10GBASE-SR is doing
significantly better than other 10GbE PMDs because of lower cost.
Still due to poor choices made on SR PMD during the 802.3ae development
the 10GBASE-SR SFP+ modules are 2.5x more expensive today
than the the FC 8.5 Gig SFP+ SR. Furthermore, 10 GbE did not achieve
the traditional Ethernet market growth like 1GbE SR or 1GBASE-T.
In contrast during the same period "the dot com bust" 4Gig FC was
introduced and went from 0 to ~100% market share replacing the 2 Gig
FC,
because at volume there was no cost difference.
And,
if you put a linear at one end and a limiting at the other, they will
communicate.
Not only linear and limiting will communicate with each other but you
can go to 220 m or 300 m with linear to linear.
9D1E2BDCB5C57B46B56E6D80843439EB04DC471A@SDCEXCHANGE01.ad.amcc.com"
type="cite">
That's what I do like
about Ali's proposal. He has shown that it is possible to do 300m of
MMF with an linear approach. That indicates to the task force there is
more margin in a linear approach than in a limiting approach;
therefore, having more margin to play with, the linear approach with a
100m MMF reach should be able to become the lowest cost solution for
the largest volume of the MMF market. That's a huge benefit. Rather
than trying to pushing the limits and slowing the adoption curve, there
is an implementation option which should make 100G MMF up to 100m a
fiscally viable option.
Thanks,
Ali
9D1E2BDCB5C57B46B56E6D80843439EB04DC471A@SDCEXCHANGE01.ad.amcc.com"
type="cite">
"Let the market decide" was how
we ended up with 100BASE-TX, instead of 100BASE-T4, 100BASE-T2, or
100BASE-VG. The 802.3 working group did a poor job of making tough
decisions and minimizing the number of options to be presented to the
industry.
What a mess.
But I
think 100BASE-TX is the most widely deployed of the various 802.3
interfaces. There have been a few billion shipped so far.
KB
"Let the market decide" is a
really, really bad way to write a standard. The IEEE 802.3 working
group has done a very good job of making tough decisions and minimizing
the number of options to be presented to the industry. To create a
reach objective that can only be satisfied by one implementation is a
poor choice as it reduces the ability of component vendors to compete
based upon their respective implementation strategies. As the current
objective is written, the reach is achievable with limiting and linear
TIA's and may be achievable with lower cost components.
Just my 2 cents,
Brad
Petar
Thanks for sending the pointer to the top 500 list and I do see the
server at TJW.
In November 2007, 2 systems appeared in the TOP500 list.
They did not show a picture or how big is the server, but based on your
remarks it is small enough to fit in modest room.
I assume the Intra-links with the Blue Gene might be proprietary or
IB. What does clustering system Intra-links has do to
with the Ethernet network connection.
I assume still some of the users in TJW lab may want to connect with
higher speed Ethernet to this server, very likely you will need
links longer than 100 m. In addition higher speed Ethernet may be used
to cluster several Blue Gene system for fail over,
redundancy, disaster tolerance, or higher performance which will
require links longer than 100 m.
We are both in agreements that parallel ribbon fiber will provide the
highest density in near future. The module form factors with a gearbox
will be 3-4x larger. Here is a rough estimate of BW/mm (Linear face
plate) for several form factors:
Speed Media Sig. Form Factor
Bandwidth (Gb/mm)
10GbE 1x10G SFP+ (SR/LR/LRM/Cu ) 1.52
(Assumes stacked cages)
40 GbE 4x10G QSFP (SR or direct attach)
4.37 (Assumes stacked cages)
40 GbE TBD If assumed Xenpak (LR)
0.98
100 GbE 10x10G CSFP (SR or direct attach) 3.85
(The proposed connector already is stacked )
100 GbE 4x25G CFP (LR)
1.23
As you could see here the form factors which allow you to go >100 m
will be several time larger and not compatible
with the higher density solution based on nx10G. Linear nx10G as given
in
http://www.ieee802.org/3/ba/public/jan08/ghiasi_02_0108.pdf
can extend the reach to 300 m on OM3 fiber and relax the transmitter
and jitter budget.
You have stated strongly you see no need for more than 100 m, but we
have also heard from other who stated
there is a need for MMF for more than 100 m especially if you have to
change the form factor for more than
100m! Like FC and SFP+ we can define limiting option for 100 m and
linear option for 300 m, and
let the market decide.
Thanks,
Ali
Petar Pepeljugoski wrote:
OF4D7F1939.EE7C74E4-ON8525740D.000B4235-8525740D.000CD5A7@us.ibm.com"
type="cite">
Frank,
You are missing my point. Even the
best case stat, no matter how you twist it in your favor, is based on
distances from yesterday. New servers are much smaller, require shorter
interconnect distances. I wish you could come to see the room where
current #8 on the top500 list of supercomputers is (Rpeak 114 GFlops),
maybe you'll understand then.
Instead of trying to design
something that uses more power and goes unnecessarilly longer
distances, we should focus our effort towards designing energy
efficient, small footprint, cost effective modules.
Regards,
Petar Pepeljugoski
IBM Research
P.O.Box 218 (mail)
1101 Kitchawan Road, Rte. 134 (shipping)
Yorktown Heights, NY 10598
e-mail: petarp@us.ibm.com
phone: (914)-945-3761
fax: (914)-945-4134
Petar;
Depending on the sources
of link statistics, 100m OM3 reach objective actually covers from 70%
to 90% of the links, so we are talking about that 100m isnot even close
to 95% coverage.
Regards
Frank
From: Petar Pepeljugoski [mailto:petarp@US.IBM.COM]
Sent: Friday, March 14, 2008 5:09 PM
To: STDS-802-3-HSSG@listserv.ieee.org
Subject: Re: [802.3BA] Longer OM3 Reach Objective
Hello Jonathan,
While I am sympathetic with your view of the objectives, I disagree and
oppose changing the current reach objective of 100m over OM3 fiber.
From my previous standards experience, I believe that all the
difficulties arise in the last 0.5 dB or 1dB of the power budget (as
well as jitter budget). It is worthwhile to ask module vendors how much
would their yield improve if they are given 0.5 or 1 dB. It is
responsible for most yield hits, making products much more expensive.
I believe that selecting specifications that penalize 95% of the
customers to benefit 5% is a wrong design point.
You make another point - that larger data centers have higher bandwidth
needs. While it is true that the bandwidth needs increase, you fail to
mention is that the distance needs today are less than on previous
server generations, since the processing power today is much more
densely packed than before.
I believe that 100m is more than sufficient to address our customers'
needs.
Sincerely.
Petar Pepeljugoski
IBM Research
P.O.Box 218 (mail)
1101 Kitchawan Road, Rte. 134 (shipping)
Yorktown Heights, NY 10598
e-mail: petarp@us.ibm.com
phone: (914)-945-3761
fax: (914)-945-4134
I am a consultant with over 25 years experience
in data center
infrastructure design and data center relocations including in excess
of 50
data centers totaling 2 million+ sq ft. I am currently engaged in data
center projects for one of the two top credit card processing firms and
one
of the two top computer manufacturers.
I'm concerned about the 100m OM3 reach objective, as it does not cover
an
adequate number (>95%) of backbone (access-to-distribution and
distribution-to-core switch) channels for most of my clients' data
centers.
Based on a review of my current and past projects, I expect that a 150m
or
larger reach objective would be more suitable. It appears that some of
the
data presented by others to the task force, such as Alan Flatman's Data
Centre Link Survey supports my impression.
There is a pretty strong correlation between the size of my clients'
data
centers and the early adoption of new technologies such as higher speed
LAN
connectivity. It also stands to reason that larger data centers have
higher bandwidth needs, particularly at the network core.
I strongly encourage you to consider a longer OM3 reach objective than
100m.
Jonathan Jew
President
J&M Consultants, Inc
jew@j-and-m.com
co-chair BICSI data center standards committee
vice-chair TIA TR-42.6 telecom administration subcommittee
vice-chair TIA TR-42.1.1 data center working group (during development
of
TIA-942)
USTAG representative to ISO/IEC JTC 1 SC25 WG3 data center standard
adhoc
|