Re: [802.3BA] [HSSG] Reach Ad Hoc
Following the 12/06/07 Reach Ad Hoc conference call, I sent an email to
the HSSG reflector (appended at the end of this email) requesting
continued end user input on reach distribution for 10km and shorter
applications.
My email was forwarded to several end user email lists, for example
NANOG (North American Network Operators Group). I subsequently received
a number of emails from different data center operators commenting on
their needs. This gave me the opportunity to follow up with some more
detailed exchanges to better try to understand how optics are being
used. While it would have been preferable for those emails exchanges to
have occurred on the HSSG reflector, at least I can share the insights I
gained.
More then half the emails objected to reducing the 10km reach objective.
There were three broad categories for why a 3km or 4km reach objective
was seen an unacceptable. For a portion of the end users, 3km or 4km was
not enough reach; they needed something a bit more then 4km for example
5km. For another portion, the extra 10km link budget was used to
accommodate much higher number of connectors / patch panels then
nominally in the standard, or just to deal with sloppy patch cord
handling. The last portion of the end users had a significant fraction
of links between 4km and 10km.
The emails that preferred a 3km or 4km reach objective were in three
broad categories of what was seen as the volume sweet spot, for which
minimum cost was paramount. The three sets of reaches that were seen as
capturing most of the volume were: 1) 500m to 1km, 2) <2km, 3) <4km.
I received permission from several of the end users to share their
emails with the HSSG reflector, and they are appended below. These
emails are meant to be exemplary and there is no intent to imply that
they accurate reflect the distribution of opinions for any of the end
user groups.
My conclusion from these exchanges is that there are many different ways
in which optics are used. End users have adopted the existing 10km reach
standard to a variety of applications, and we should be very careful
before making changes that may break many applications that we are not
even aware of.
At this point, I support keeping the existing 10km reach objective, and
do not support changing to a shorter reach.
I would also encourage anyone on the HSSG reflector, who is also a
member of any of the end user groups, like NANOG, to share their views,
or summarize their take on the views of their group on the HSSG
reflector. This will add to the credibility of this discussion.
Chris
> -----Original Message-----
> From: Sam Silvester [mailto:sam.silvester@gmail.com]
> Sent: Thursday, December 13, 2007 3:13 PM
> To: Chris Cole
> Subject: Re: IEEE 40GE & 100GE
>
> Hi Chris.
>
> Working for a reasonably small (geographically speaking - within a
> single state / capital city) carrier, as long as (per your earlier
> comments) 10km optics are happy talking to each other with no
> attenuator required, I see very little value in a 4km reach version.
>
> If we're talking about a cost ratio of between 1.15 and 1.3, this just
> reinforces the fact in my opinion.
>
> Internally i.e. within our PoPs, we use 10km optics currently at 1GE
> and 10GE (even for < 500m runs). We have some deployment of 40km but
> in general due to geography, 10km does the job nicely.
>
> I'd suggest even if a 4km optic came along, I'd not bother as the "end
> user price difference" most likely wouldn't excite me enough to have
> to carry another line of spares. I realise this is more or less in
> line with other comments, but I figured you might like the input
> anyway.
>
> Excluding "short runs" of 500m, I'd say the ratio of <4km links vs
> <10km links is about 50/50.
>
> Note that I expect in an enterprise-style deployment the number may
> well stack up with the 90% figure you mentioned in your email. I'm
> still not sure a 1.15 - 1.3 ratio makes that worthwhile...
>
> Regards,
>
> Sam
> -----Original Message-----
> From: Justin M. Streiner [mailto:streiner@cluebyfour.org]
> Sent: Wednesday, December 12, 2007 4:51 AM
> To: nanog@merit.edu
> Cc: Chris Cole
> Subject: Re: IEEE 40GE & 100GE
>
> On Wed, 12 Dec 2007, Robert E. Seastrom wrote:
>
>> A practical question here: does anyone know offhand if 4km reach is
>> adequate for interbuilding access (i.e., DC[124] to DC3) access at
>> Equinix Ashburn, including worst-case interior wiring and cross
>> connects? I'm thinking that's cutting it close. The enterprise
>> people are substantially less likely to find themselves with a lot of
>> interconnections in a GCE (Ginormous Campus Environment) than we are,
>> and I suspect that skews the 90% number a bit. Folks who are more
>> familiar with the layout of other facilities may wish to chime in
>> here.
>
> I'm not in any of the Equinix facilities, but I do run a decent-sized
> urban campus network and a 3km-4km distance limitation would be
> cutting it really close for me in some cases. Some of the 10G links
> on my backbone today do require multiple physical cross-connects,
> which would eats into the link budget. Most of my backbone connections
> work find with 10G-LX4 ptics, but there are a few places where 10G-ER
> is needed.
>
> I haven't read the draft spec yet to see what's being proposed for a
> link budget at 3/4/10km, but that's just as important as the physical
> distance.
>
> jms
>
> -----Original Message-----
> From: Randy Whitney [mailto:randy.whitney@verizonbusiness.com]
> Sent: Wednesday, December 12, 2007 8:46 AM
> To: Chris Cole
> Subject: Feedback on 100GE Proposed
>
> Chris,
>
> Per cross-posting to lists I am on I am providing you feedback on the
> proposed change to the 100GE standard.
>
> For my deployment needs, 3km or 4km would *not* be sufficient in many
> cases, so I would be strongly opposed to reducing the std down to 3km
> or 4km.
>
> Best Regards,
> --
> Randy S Whitney +1 703 886 5642
(0)
> Verizon Business, Global Peering Manager +1 703 886 0512
(F)
> 22001 Loudoun County Pkwy, Ashburn, VA 20147 +1 202 997 8174
(M)
> randy.whitney@verizonbusiness.com uursw (AIM,
YAHOO)
>
> -----Original Message-----
> From: Robert E. Seastrom [mailto:rs@seastrom.com]
> Sent: Wednesday, December 12, 2007 11:20 AM
> To: Chris Cole
> Cc: Justin M. Streiner; nanog@merit.edu
> Subject: Re: IEEE 40GE & 100GE
>
>
> "Chris Cole" <chris.cole@finisar.com> writes:
>
>> One of the points made by Ted Sealy from Sprint is that they take
>> advantage of the extra link budget in 10GBASE-LR 10km link budget to
>> account for extra connector loss, etc.
>
> Ted Seely and I are of the same mind on this. 2 dB sounds like plenty
> for connector loss right up until you have to deal with multiple patch
> bays in a structured system with amateurishly applied mechanical
> splices. The difference between noting that the loss is a little high
> but the link still works so you roll with it, and having to spend time
> on the phone arguing with someone who thinks 24 dB link loss is A-OK,
> will make the slight additional up front cost for the better grade
> optics look very inexpensive indeed...
>
> ---rob
> -----Original Message-----
> From: Dale W. Carder [mailto:dwcarder@wisc.edu]
> Sent: Wednesday, December 12, 2007 12:53 PM
> To: Chris Cole
> Subject: Fwd: IEEE 40GE & 100GE
>
>
> Hi Chris,
>
> I appreciate the opportunity to provide some end-user perspective from
> our environment, the University of Wisconsin @ Madison and WiscNet.
>
> Most of what I saw in the links below was about link budget, but what
> about ORL? We got bit pretty hard migrating from 1G to 10G through
> old hand-polished SMF panels which had never been a problem before.
> We ended up cutting & fusion splicing on new pigtails in numerous
> locations.
>
> Data Center / super-computing environments:
> - high performance connections are all within 300 meters,
> with both FDDI grade MMF (largely abandoned) and also SMF-28.
>
> Campus environment:
> - about (200) buildings all within 3km of our central
> aggregation points on SMF-28.
> - roughly 40% of point-to-point links have (2) patches
> - roughly 60% of point-to-point links have (4) patches
> (via an intermediate building's patch-panel)
>
> Metro Ethernet environments:
> - about (10) links 8-10km, (4) 10-12km.
> - We are running 10G-LX today on a few links beyond 10km.
> - most have 6-8 patches
>
> Regional Long-Haul environment:
> - (30) 80-100km links
>
> We are quite interested in what happens with HSSG, let me know if
> there's anything else we can help with.
>
> Dale
>
> --
> Dale W. Carder - Network Engineer
> University of Wisconsin - Madison / WiscNet
> http://net.doit.wisc.edu/~dwcarder
>
> -----Original Message-----
> From: Brian Dickson [mailto:briand@ca.afilias.info]
> Sent: Tuesday, December 11, 2007 6:17 PM
> To: Chris Cole
> Subject: Re: [HSSG] Reach Ad Hoc
>
> Hi, Chris,
>
> Bora Akyol forwarded your summary message on the 10 km vs 3/4 km
issue,
> to the NANOG mailing list, indicating that the IEEE needs operator
> feedback on this issue.
>
> I'm pretty familiar with who needs to be deploying 40G and 100G, and
> where and how they are likely to need these interfaces.
>
> There are primarily two places we expect to see *lots* of these
> connections.
>
> One is - service provider backbones. These are the internal network
> links that form major ISP/NSP network infrastructure. Those will fall
> into a number of categories, ranging from DWDM, to CWDM, to
> direct-over-fibre.
>
> The other is, the interconnections between ISPs. The number of
> connections on these ingress/egress points is likely to be higher by
an
> order of magnitude - anywhere from 2x to 10x the links, proportionally
> speaking.
>
> It is worth noting that even when traffic levels may only be
approaching
>
> 10Gbps, that 100G is *very* attractive, notwithstanding the price.
Burst
>
> capacity is what IP network engineering is all about, not average
usage.
> And, even if an ISP is small compared to his neighbor, the links have
to
>
> operate at one speed, which *needs* to be the highest-common-multiple,
> not the lowest-common-denominator. When in doubt, go high, is the
rule.
>
> And, also worth noting is, that once the technology is available, the
> switching capacity is anticipated to meet the line rate head-on.
> Line-rate many-port switches will be the norm, and access to those is
> best achieved at 100G.
> Getting those cards affordable is a huge deal, and getting them
> delivered in a timely fashion by the parts manufacturing community is
> something we are *very* worried about. Even 10G cards are late to the
> game.
>
> The places where ISPs interconnect, are a few dozen locations
worldwide,
> and those have the following common characteristics:
>
> A very large building, with cages, suites, and cabinets
> A meet-me area where all interconnections generally run
> Some big colocation providers who re-sell some of the building
> real-estate, with their own meet-me areas
> Hundreds to tens of thousands of companies colocating their equiment,
> especially ISPs
> Interconnections run from ISPs to common switching equipment (an
> Internet eXchange, or IX)
> Interconnections directly between ISPs bypassing the IX
>
> The former (IX connections) typically reach much further down the list
> of customers, many 10s to a few hundreds
> The latter (private peering) typically use the same media type (SMF,
or
> sometimes MMF) regardless of speed - 100M, 1000M, 10G, 40G, 100G
>
> The buildings will typically be telco hotels (vertical, up to 30 or so
> floors), no more than 50m long and 30m wide, max fibre distance of
300m
> typically, but as many as 8-10 patches per link, or as few as zero.
> Some other buildings are more warehouse type - one floor only, but as
> big as 300m x 500m or more, max fibre distance typically 1.5-2km, with
> probably fewer patches (2, 4, 6 are typical numbers).
>
> Based on this info, I would suggest 4km over 3km, but that 4km is very
> much of interest to the buying community, who (based on volume of
links)
> are definitely interested in keeping costs down. Many of those folks
> were bitten in the @ss by OC-192 costs that were about 5x higher than
> they should have been.
>
> Hope this information is helpful, and if you would like me to put you
in
> touch with some folks who operate the neutral data centers and run the
> fibres at them, I would be happy to oblige.
>
> Thanks for allowing me to provide some input,
>
> Brian Dickson
> Afilias Canada
>
-----Original Message-----
From: Chris Cole [mailto:chris.cole@FINISAR.COM]
Sent: Tuesday, December 11, 2007 4:22 PM
To: STDS-802-3-HSSG@LISTSERV.IEEE.ORG
Subject: Re: [HSSG] Reach Ad Hoc
During the November HSSG meeting, optics vendors made a presentation
proposing changing the 10km reach objective to 3km or 4km. One of my
motivations for working on the proposal was informal input from a number
of 100GE end users that >90% of their data center and short interconnect
needs would be met by a reach objective less then 4km (versus 10km.)
With such a reach distribution, a 4km or less optimized reach objective
would result in overall cost savings.
As part of the HSSG effort to review this proposal, numerous requests,
both informal as well as from the HSSG chair and Reach Ad Hoc chair,
have been made for contributions to quantify the 10km and under reach
distribution. While the optics vendors as suppliers can accurately
represent the relative costs of optics alternatives, they can not
represent end user requirements.
To date, we have seen no end user presentation or data supporting
changing the 10km reach objective to 4km or less. Unless such
contributions are forthcoming, it is likely that there will be no
motivation to make the change. This sentiment can be seen in the 12/7
Reach Ad Hoc conference call minutes.
I would encourage any HSSG participant that views their volume 100GE SMF
needs as better met by a 4km or shorter reach objective to make a
contribution containing reach distribution data in support of this
position. Otherwise we will move forward with the existing approved
objectives.
Chris
________________________________________
From: Andy Moorwood [mailto:amoorwood@EXTREMENETWORKS.COM]
Sent: Tuesday, December 11, 2007 6:03 AM
To: STDS-802-3-HSSG@LISTSERV.IEEE.ORG
Subject: [HSSG] Reach Ad Hoc
Colleagues, the meeting notes from our call last week are now posted on
the IEEE website
http://www.ieee802.org/3/hssg/public/reach/MeetingNotes_r1_1207.pdf
Thank you for your contributions
Andy