Dan,
Sorry if this is a repeat to you, but you
said you were not getting emails at your APM address to which it was first
sent.
Paul
---
So far I am not getting hung up on the
discrepancies in the DC size distributions. Even if we knew them
precisely, we would still not be able to tell the cabling topologies used
within them. That is why I prefer the approach of looking at the cable
lengths themselves. A decade ago this would not have been possible, but
the advent and popularity of pre-terminated cabling now allows us that
insight. The cabling data and analysis I presented represents a good view
of how our customer base builds data centers. Those customers can be
generally classified as from the “Fortune 1000”, so are also very
relevant as consumers of our next-generation output.
The survey and cabling distribution data
presented to 802.3 over the years does indicate that data centers are
evolving. Denser servers have accompanied shorter cabling spans to their
access switches, and with that, an increase in switch-to-switch cable lengths
for the channels that carry the aggregated traffic. Whether 10G server
access channels remain dominated by very short twinax copper or migrate to
longer reach UTP as 10GBASE-T becomes lower in power consumption remains to be
seen, but I think it is clear that there will be a higher percentage of closer
access switch placement topologies like ToR, MoR, and EoR than in the past,
basically because there are more servers in close proximity to each other.
So far these topologies have driven an overall increase in the switch-to-switch
reach requirements.
One of the other trends is the advent of
the cloud. This is somewhat synonymous with data center hosting where the
internet services for many customers are housed in larger and larger data
centers. This is supported by the observation that mega data centers only
came into existence in the past 5 years or so. These are the primary
locations where traffic demand is so high that 100G services are
essential.
You also bring up the
“containerized” data center, the growing trend where pods are built
in trailer-sized containers and added as demand necessitates. If your
assertion that smaller volume means more energy efficiency, these pods may have
that advantage too. Yet it is not clear to me how they impact reach
requirements. For example, these pods are delivered by truck, so for the equivalent
of a large data center the container “parking lot” would require
truck-sized access aisles that must co-exist with cabling pathways. Such
ingress and egress requirements may increase reach requirements even though the
containers are relatively small compared to a big building.
All this leaves me with the view that
pre-term cabling data are likely the best means of assessing channel length
distributions and their trends. Please realize that the data I presented
included that from 2010, a year when the shorter reach limitations of 40GE and
100GE would have been known. Despite the fact that the reach shrunk from
300 m for 10GE to 100 – 150 m for 40/100GE, the channel length
distribution got longer compared to 2005 data. Because of this I
can’t simply agree that if we reduce the reach again that it will drive
acceptable data center design changes, because the data is telling me that
channel lengths longer than 100 m are common. So while I agree that a 100
m reach satisfies the large majority of channel needs (because the pre-term
data analysis supports that view), we cannot determine if that reach is optimal
without knowing the cost of solutions that will be required to satisfy the
longer channels, for it is only in the context of the total solution set where
total cost of ownership can be determined.
I also agree that this analysis should
attempt to include not only the initial hardware cost, but the operational
costs associated with power consumption. However, this operational cost
aspect is drawn out over long periods of time, so it cannot be simply added to
the initial cost. So strictly speaking the complication of the time-value
of money comes into play. But as a first pass, without time-value
factored in, it may be the case that an extra couple
Lastly, you mentioned using FEC as a reach
booster. FEC does provide additional reach for SM solutions because it
translates into more power budget by allowing the receiver to operate below
sensitivity or with noisier signals because FEC patches up errors caused by
noise. However FECs efficacy for MM optics that are constrained not by noise
but by slow VCSELs and detectors and chromatic dispersion is really
questionable. These are not noise impairments that occasionally cause an
error, but rather distortion impairments that insert structural pulse shape
problems affecting many bit patterns. Such distortions can swamp FEC
capability and are much better compensated by equalization techniques that are
designed to tackle distortion problems rather than noise.
So I agree that we should not get hung up
on the DC size distribution data, but we should follow thru with our present
course of analysis before setting objectives.
Regards,
Paul
From: Dan Dove
[mailto:dan.dove@xxxxxxxxxxxxxxxxxx]
Sent: Monday, December 12, 2011
1:28 PM
To:
Subject: Re: [802.3_100GNGOPTX]
Emerging new reach space (RESEND)
For some
reason, this message is not going to the reflector, and I am not receiving
messages at my APM address.
Date:
Thu, 08 Dec 2011 12:56:45 -0800
To: 100G Group <
Subject: Re: [802.3_100GNGOPTX] Emerging new reach space
Hi all;
I wonder if we are spending our time on the wrong question, ie; "What is
the current distribution of fiber in data centers?"
Its useful to see where fiber is being installed at 10G and to understand the
reaches various applications have used, however, it seems to me that a few
key principles are going to drive the future installations and reach
requirements. To an extent, what we produce will be a driving force rather than
a response to the market.
Principles;
Data Centers are extremely sensitive to energy consumption and Total Cost of
Ownership (TCO) factors related to it.
Energy consumption requires thermal management, and efficiency is a function of
Initial investment in IT equipment itself is based on performance demands and
thus likely to be an independent (or loosely dependent) variable from reach.
Not to be simplistic, but this leads me to the euphemism "Build it, and
they will come".
More specifically, if we find that X meters of MMF is what we can support
without exceeding a bend in the cost curve, as long as X is reasonable and
usable, people will figure out how to optimize around it.
Between papers on "containerized data centers" and "energy
efficient warehouse scale data centers" I see a trend growing. Relatively
small clusters of racks which contain Top of Rack (TOR) or Middle of Rack (MOR)
switches that tie servers together using very short (most likely copper) links.
These switches are then tied together with a fiber-run (probably two) to a
cluster switch. The total geometry of this cluster wants to be small because
air-flow defines the thermal efficiency. A large, spread out cluster requires a
lot of air to cool it. The thermal performance is based on velocity, thus a
smaller, faster air column is going to work better than a big, slow one.
Even for massive data centers, they are likely to segregate the air columns, and
maintain smaller clusters that are tied together with floor switches or meshes.
This all means that the future distribution of fiber may look very different
from what historically has been the case. The size of the clusters will be
based upon energy efficiency and thermal management more than the reach of the
fiber within the cluster. The distance between cluster switches *may* be
supportable by MMF, and a 100m reach allows for a pretty significant sized
building to be supported before you have to even consider SMF as a link between
them.
Below is an example of a 150,000 square foot data center layout.
With today's server technology, the number of servers in such a building can
start to challenge other design limits than cable reach. Power distribution,
thermal management, ingress/egress data capacity become a bigger challenge than
the distance between switches.
I am only saying this to question if we are beating ourselves up over a
non-issue. If we can get 100m of reach as Jonathan King's presentation in
November stated, I think we are going to satisfy everything from a 150,000 sq
ft data center down. The only question that might remain, and its more of a
Task Force question, is whether we need two PMDs (one for cluster links, one
for floor links) and this depends on the cost delta between a 40m reach and
100m reach. As Jonathan points out, the difference could be achieved simply by
enabling/disabling FEC with a common PMD.
Personally, I am ready to support a 100m objective and believe that we can show
technical feasibility and market potential based on available information.
Some more work on economic feasibility would be appreciated, and additional
supporting presentations on MP and TF would be helpful, but we are very close
if we can simply conclude that 100m is a reasonable target without getting
bogged down on inconsistencies between sources of historical data.
Dan