RE: 10xGbE on DWDM

[Paul Gunning <paulg@xxxxxxxxxxxx>]

Hi Bill &c.,

I very much welcome the opportunity to add to the debate
on 10xGbE. My background is in ultrafast photonic networks.
I'm afraid my knowledge of electronics is suspect and equally
my appreciation of the finer points of the protocol stack leaves
a lot to be desired. But indulge me as I'm learning a 
great deal from the 10xGbE discussion about "esoteric" 
things such as 8B/10B coding, thermal loads etc. which are 
all very new and interesting to me. So interesting, in fact,
that I bought Rich Siefert's book on Gigabit Ethernet and 
also a wonderful book called: High-Speed Digital Design : 
A Handbook of Black Magic by Johnson and Graham.

However let me try to express some of the potentially 
"disruptive" technological solutions that photonics can 
bring to the table.

The photonics community has been investigating
the issues that you mention for the past ten years. So 
spare a thought for optical regeneration as the eventual 
replacement for electrical regeneration when the serial 
bit rate increases.
 
Many industrial research teams: Alcatel, NTT, Nortel and 
BT and university teams: TUD-Denmark, ETH-Zurich, HHI-Berlin,  
have had active research programmes in the optical regeneration
area for the past decade. Prototypical devices have been built
and demonstrated, but a market hasn't yet emerged because
electronics has alays provided a cheaper alternative. I'm not 
convinced that 10xGbE will kickstart the market for optical
regeneration---a pity. However I believe that 100xGbE WILL!

Please allow me to expose the arguments of my colleague, Julian
Lucek, in particular his distilla of the "4th quadrant." 

	http://www.labs.bt.com/people/lucekjk/4thquad.htm

Consewuently, it's important to think a little 
less of the notion of optical transparency as 
touted by DWDM, because it's not really optically 
transparent because it requires periodic 3R 
OPTOELECTRONIC regeneration! Instead ponder DIGITAL 
OPTICS and how that WILL impact 100xGbE. 

For example an issue that will become more important 
is the impact of the inherent timing jitter from the 
transmitter/source AND the timing jitter converted 
from any wavelength jitter from the transmitter/source 
due to dispersive optical fibre propagation.  

To this end, my colleagues and I knocked something 
together last summer in the lab which we called an 
Asynchronous Digital Optical REgenerator (ADORE.) 
It all-optically, digitally regenerated optical data 
packets without the need for preamble bits to effect 
Rx synchronistion via an electrical phase locked loop. 

Moreover each network node can have an independent 
local clock; every clock within the network being 
nominally synchronous. Global synchronisation IS NOT
required much the same as with current Ethernet networks. 
In fact the technique allows the construction of 
10+ Gbit/s that have a commonality with the switched 
Ethernet we have today i.e. all the attached 
computers and switches have independent clocks.

This gives great freedom because it allows the
construction off arbitrary network architectures.
But most usefully since it is an all-optical 
regenerator, error-free and UNLIMITED propagation 
distances are realisable.

A paper resulted from the work:
	10Gbit/s Asynchronous Digital Optical Regenerator
	P. Gunning, A.D.Ellis, I.D. Phillips, J.K.Lucek, 
	D.G. Moodie, and D. Cotter, 
	OFC/IOOC '99, San Diego, Paper: TuJ5
that I presented in February.

An expanded version of the paper will appear in the July 1999
edition of Photonics Technology Letters:
	10Gbit/s Asynchronous Digital Optical Regenerator
	I. D. Phillips, P. Gunning, A.D.Ellis, J.K.Lucek, 
	D.G. Moodie, A. E. Kelly, and D. Cotter, 
	vol.11,no.7, July 1999.

The key components were (x2) Electroabsorption Modulators
(EAMs) which are just reverse-biased, high-speed multi-quantum 
well (MQW) photodiodes. The remainder was optical "plumbing." 
There is no reason why an ADORE cannot be implemented on a planar
silica "chip" and mass-produced given the will and commercial demand.

Error detection? My colleague Alistair Poustie has demonstrated
all-optical parity checking at 1Gbit/s and he assures me the same
technique can be applied at 100Gbit/s (or 10Gbit/s):
	All-optical parity checking
	A. J. Poustie, K. J. Blow, A. E. Kelly and R. J. Manning
	OFC/IOOC '99, San Diego, Paper: TuJ6

In the lab he has implemented an all-optical counter. I hope you can
appreciate that many of the necessary functions implemented currently
in electronics can also be implemented all-optically for bit rates
far in excess what is currently feasible with optoelectronics. Most
importantly the Digital Optical data is mainteined in its native state
without recourse to optoelectronic cnversions.

Now correct me if I'm wrong but isn't timing management 
in SONET/SDH something of a black art? What with all 
the different timing strata and the necessity that 
all nodes have a direct and hierarchical "path" 
(without nasty timing loops) to the primary clock source? 

Also why are Cisco moving away from full-blown SDH/SONET 
functionality with their DPT technology? 


http://www.cisco.com/warp/public/cc/cisco/mkt/servprod/opt/dpt/tech/dpta_wp.htm

Note: Synchronisation between adjacent nodes is not effected 
via SONET/SDH with DPT. What are the deployment plans 
for 40Gbit/s SONET/SDH? Are solutions available now? Soon?
Bear in mind the lessons of ATM: ATM can currently offer 
622Mbit/s, yet GBe, for a fraction of the cost, can 
provide 1000Mbit/s. ATM is perceived as complex, 
GbE simple.

SONET/SDH can provide ~10Gbit/s now, yet 10xGBe will provide this
soon... ;) Beyond 10Gbit/s, when will SONET/SDH provide 40Gbit/s?
And by then will "100xGbE" = 10 * 10xGbE be available...with 
"proper"-serial 100xGbE soon to follow?

Also why can't 10xGbE, 100xGbE not be used for high-speed 
interconnects for emerging system/storage area networks 
(SANs)?

Ultimately the most compelling advantage of Ethernet lies
in its modularity and simplicity. Sure even I can plug 
the RJ45 into the wall socket beside me and it works!!!! 
I fear that some of the solutions proposed for 10xGbE so 
far have lost sight of this....and the fact that 100xGbE,
...1xTbE will inevitably follow.

	Keep it simple, keep it serial!


Finally, a plea for tolerance and respect. I think 10xGbE 
is where two communities: Datacomms and Photonics are 
meeting for the first time we really have a lot to 
learn from each other. 

Paul.

p.s.  The EDFA window is not narrow. It's 25THz from
1535 -> 1565 nm! Indeed there is an extra 25THz available
from 1580 -> 1610 nm using next generation EDFAs. 
Plenty of room for expansion!

The filters and optical sources used for DWDM based 
on Arrayed Waveguide Grating (AWG) technology will 
cost next to nothing when Moore Law economics prevails. 
Just like IC's before them, its only "processed sand" 
festooned with a few choice elements.

For Campus LAN installations Blown fibre is second to none.
The story of its invention is worth recalling. The two guys
involved, Steve Cassidy and Mick Reeve, were tasked to 
find a technique to allow post-installation of optical 
fibre within plastic conduits. They started-off by 
attaching a "mini-parachute" from an "airborne" Action 
Man (GI-Joe) figure (p.p.s. Be Careful: Watch out for Dr. X!!!):

	http://www.hasbro.co.uk/actionman/
	
to the fibre end which was then placed in the conduit and
blown by compressed air to facillitate deployment. It worked 
a treat. Until one day the mini-parachute became detached...
and still the fibre happily threaded it's way through the 
conduit "floating" on a cushion of air. There are many 
different tradenames for the system the one I know about 
is Bicc Blolite, which I think is now owned by Corning...

	http://www.bicc.com/aboutus/mn_network_revolution.html

I would encourage network instalation managers to consider 
upgrading "once and for all" to 8/125um, single mode fibre 
via blown fibre so that they can reap the benefits of 
10xGBe...and, in time, 1xTbE!!!
-- 
Paul Gunning
Futures Lab
BT Laboratories

Phone:	+44 1473 647049
Fax:	+44 1473 646885
http://www.labs.bt.com/people/gunninp


Bill St. Arnaud wrote:
> 
> Paul:
> 
> I share your vision. The problem I see with 10XGbE on long haul DWDM systems
> is the operational issues.  This is where SONET excels, particularly if you
> have to do intermediate electrical regeneration.  Most SONET/DWDM
> manufacturers have well established, proprietary OAM systems for fault
> management, etc
> 
> However as data transparent DWDM networks reach greater and greater
> distances than there is no reason why we can't use 10xGBE on long haul DWDM
> systems.
> 
> However, DWDM systems are inherently more costly because of the narrow EDFA
> window, so the wavelengths have to be closely spaced requiring special
> optical filters and very stable lasers.
> 
> Ledcor and other fiber installation companies routinely use air blowing
> rather than fiber pull strings.  You can blow a fiber a much greater
> distance than you can pull it.
> 
> Bill
> 
> -------------------------------------------
> Bill St Arnaud
> Director Network Projects
> CANARIE
> bill.st.arnaud@xxxxxxxxxx
> http://tweetie.canarie.ca/~bstarn
> 
> 
> 
> 
> 
> > -----Original Message-----
> > From: paulg [mailto:paulg]On Behalf Of Paul Gunning
> > Sent: Friday, May 28, 1999 1:32 PM
> > To: Iain Verigin; bill.st.arnaud@xxxxxxxxxx
> > Cc: bin.guo@xxxxxxx; rtaborek@xxxxxxxxxxxxxxxx;
> > dwmartin@xxxxxxxxxxxxxxxxxx; stds-802-3-hssg@xxxxxxxx;
> > sachs@xxxxxxxxxxxxxx; widmer@xxxxxxxxxx widmer@xxxxxxxxxx
> > widmer@us.http://m.doubleclick.net/viewad/208245-textlinkdot.gif
> > Subject: Re: Wide Area Networking for the Rest of US - the debate on BER
> > a nd other issues
> >
> >
> > Iain, Bill &c.
> >
> > Why not...
> >
> >       Layer 3         IP
> >
> >       Layer 2         Ethernet (10GbE = serial 10Gbit/s ->
> > 12.5GBaud with 8B/10B)
> >
> >       Layer 1         Optics    (N * 12.5 GBaud via N WDM channels)
> >
> > then have a generic PHY termination onto which one of three possible
> > optical "modules":
> >
> > a)    850nm VCSELs for short reach using multimode fibre (<300m)
> >
> > b)    1300nm Fabry-Perot or VCSELs for short-to-intermediate reach (<30km)
> >
> > c)    1550nm temperature controlled DFB lasers for long reach (>30km)
> >
> > can be plugged? Want to increase capacity? Use WDM! (Just consider
> > each WDM channel as a "virtual" fibre!)
> >
> > Option b) can make use of mature 1300nm TWSOA (Travelling Wave
> > Semiconductor
> > Optical Amplifier) technology to extend the link length.
> >
> > Option c) can make use of mature EDFA technology to extend the
> > transmission span. Moreover the use of temperature control "locks"
> > the position of the DFB wavelength i.e channel, within the EDFA
> > bandwidth
> > which is 50 THz of bandwidth (anyone for Terabit Ethernet?)
> > assuming 1535nm <-> 1565nm; 1580nm <->1610nm are used.
> >
> > Consequently a long-distance, optically-amplified, fibre
> > transmission path could happily carry a mixture of
> > SONET/SDH _AND_ 10 GbE traffic simultaneously. For example,
> > 1535nm <-> 1565nm would carry 16 * 10 GbE channels;
> > 1580nm <-> 1610nm would carry 16 * OC-192/STM-48 channels?
> > Each distinct channel is assigned a distinct wavelength.
> > Use simple passive "blue/red" WDM couplers to MUX/DEMUX
> > the SONET wavelength band from the GbE wavelength band
> > at the termination points at either end of the trabsmission
> > span.
> >
> >       That's 320 Gbit/s of data!!!
> >
> > For the 10GbE standard is it _REALLY_ necessary to map 10 GbE
> > into SONET/SDH frames? Why not aim for simplicity? For long distances
> > over grey fibre (i.e. partially "lit"= available optical amplifer
> > bandwidth not fully exploited; dark fibre is "unlit") this is
> > desirable. Remember SONET and 10 * GbE can co-exist happily. Moreover
> > 1300nm intermediate haul 10 GbE could also be carried along with the
> > 1550nm traffic mix!
> >
> > Has anyone considered exploiting Air Blown Fibre e.g.
> >
> >       http://www.vector-resources.com/abf.htm
> >
> > as a simple method for installing and/or upgrading the fibre
> > plant within a building or campus to single-mode?
> >
> > Finally, serial 100 GbE will be possible soon. It will use a
> > pragmatic mix of optics and electronics. Therefore the choices
> > that are adopted for 10GbE will have a crucial bearing on the
> > ease with which serial 100GbE will become a reality.
> >
> > (Note that by using WDM as "virtual" fibre 10 * 100GbE = 1 TbE !)
> >
> > Keep it serial, keep it simple.
> >
> > I would welcome your comments and criticisms of the above.
> >
> > Paul.
> >
> >
> > Iain Verigin wrote:
> > >
> > > As I know it, there could be three IP stacks at 10G.  Two exist today at
> > > lower rates, they are:
> > > 1. IP/Ethernet - General use is intra-CO (LAN) interconnect.
> > > 2. IP/Packet-over-Sonet (POS) RFC 1619 - General use is inter-CO (WAN)
> > > interconnect.
> > >    (Note IETF RFC 1619 has been revised to include OC-192c.)
> > >
> > > I agree with Bill that we should investigate a third stack that would be
> > > consistent with today's WAN infrastructure.
> > > 3. IP/Ethernet/SONET.
> > > * there are many reasons for this. I believe one of the
> > strongest is that it
> > > removes the need for Layer 3 processing that POS imposes.  A negative is
> > > that it adds more bytes, an Ethernet tax, which may not be
> > desirable on long
> > > haul links.
> > > * the IEEE could work on this because no other standards body is.
> > >
> > > Iain Verigin
> > > PMC-Sierra
> > >
> > > -----Original Message-----
> > > From: Bill St. Arnaud [mailto:bill.st.arnaud@xxxxxxxxxx]
> > > Sent: May 28, 1999 5:52 AM
> > > To: bin.guo@xxxxxxx; rtaborek@xxxxxxxxxxxxxxxx;
> > > dwmartin@xxxxxxxxxxxxxxxxxx
> > > Cc: stds-802-3-hssg@xxxxxxxx; sachs@xxxxxxxxxxxxxx; "widmer@xxxxxxxxxx
> > > widmer@xxxxxxxxxx widmer"@us.ibm.com
> > > Subject: Wide Area Networking for the Rest of US - the debate on BER and
> > > other issues
> > >
> > > All:
> > > I have been following the interesting debate about BER. Let me
> > bring some
> > > further issues into the debate.
> > >
> > > I am assuming that on WAN and long haul GbE the upper layer
> > protocol will
> > > only be IP.
> > >
> > > On most IP links, even ones with BERs of 10^-15 there is about
> > 1-3% packet
> > > loss and retransmission.  This is due to a number of factors but most
> > > typically it relates to TCP flow control mechanism from server bound
> > > congestion (not network congestion) and the use of WRED in routers.
> > >
> > > So, on most IP links the packet loss due to BER is
> > significantly less than
> > > that due to normal TCP congestion.  As long as that ratio is
> > maintained it
> > > is largely irrelevant what the absolute BER value is.  There
> > will be many
> > > more retransmissions from the IP layer than there will be at
> > the physical
> > > layer due to BER.
> > >
> > > Other protocols like Frame Relay and SNA are a lot more
> > sensitive to high
> > > BERs.  IP ( in particular TCP/IP) is significantly more robust
> > and can work
> > > quite effectively in high BER environments e.g. TCP/IP over barbed wire.
> > >
> > > I would like to suggest that the 802.3 HSSG group consider an 2
> > solutions
> > > for 10xGbE WAN:
> > > (1) native 10xGbE using 8b/10b; and
> > > (2)10xGbE mapped to a SONET STS OC-192 frame
> > >
> > > For extreme long haul solutions SONET makes a lot of sense as a
> > transport
> > > technology.  However for intermediate long haul (up to 1000 km) and WAN
> > > native 10xGbE is more attractive. Native GbE can be either
> > transported on a
> > > transparent optical network or carried directly on a CWDM system with
> > > transceivers. In medium range networks coding efficiency is not
> > as important
> > > as it is in long haul networks. If coding efficiency is
> > important then in my
> > > opinion, it does not make sense to invent a new coding scheme for 10xGbE
> > > when it would be just as easy to map it to a SONET frame.
> > >
> > > The attraction of native 10xGbE for the WAN is that it is a "wide area
> > > networking solution for the rest of us".  You don't need to
> > hire specialized
> > > SONET engineers to run and manage your networks.  The 18 year
> > old kid who is
> > > running your LAN can now easily learn to operate and manage a WAN.
> > >
> > > In Canada and the US, there are several vendors who are willing
> > to sell dark
> > > fiber at a very reasonable cost.  Right now the cost of
> > building a WAN with
> > > 10xGbE and CWDM is substantially less (for comparable data
> > rates) than using
> > > SONET equipment.
> > >
> > > Bill
> > >
> > > -------------------------------------------
> > > Bill St Arnaud
> > > Director Network Projects
> > > CANARIE
> > > bill.st.arnaud@xxxxxxxxxx
> > > http://tweetie.canarie.ca/~bstarn
> > >
> > >
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: owner-stds-802-3-hssg@xxxxxxxxxxxxxxxxxx
> > > > [mailto:owner-stds-802-3-hssg@xxxxxxxxxxxxxxxxxx]On Behalf Of
> > > > bin.guo@xxxxxxx
> > > > Sent: Thursday, May 27, 1999 7:28 PM
> > > > To: rtaborek@xxxxxxxxxxxxxxxx; dwmartin@xxxxxxxxxxxxxxxxxx
> > > > Cc: stds-802-3-hssg@xxxxxxxx; sachs@xxxxxxxxxxxxxx; "widmer@xxxxxxxxxx
> > > > widmer@xxxxxxxxxx widmer"@us.ibm.com
> > > > Subject: RE: 1000BASE-T PCS question
> > > >
> > > >
> > > >
> > > > Rich,
> > > >
> > > > The DC balance can be directly translated into jitter (when timing is
> > > > concerned) and offset (when threshold slicing is concerned).  You
> > > > only need
> > > > to deal with the former if the signal is 2-level NRZI, while
> > you need to
> > > > deal with both if multi-level signal modulation is used.
> > > >
> > > > For long term DC imbalance, it translates into low frequency
> > jitter and if
> > > > it's low enough(<1 KHz ?), it's called baseline wonder.  For
> > > > short term, it
> > > > relates to Data Dependent Jitter, which is more difficult for timing
> > > > recovery to handle since it's not from system or channel imparity, and
> > > > therefore it's harder to compensate.
> > > >
> > > > When you have a lot of jitter margin, for example in lower
> > speed clocking,
> > > > the amount of jitter, translated from DC drift resulted from data
> > > > imbalance
> > > > coupled by AC circuit, percentage wise is a small portion of the clock
> > > > period and therefore does not contribute to much of the eye
> > > > closing.  On the
> > > > other hand, for high speed clocking at 10G (100 ps?), the jitter
> > > > translated
> > > > from the same amount of DC drift can be a significant portion
> > of the clock
> > > > period, so contributes to much large percentage wise jitter which
> > > > results in
> > > > reduced eye opening -- higher BER.
> > > >
> > > > Dave said in his mail that "The limiting factor is enough RX
> > optical power
> > > > to provide a sufficiently open eye." but you still have to
> > deal with the
> > > > data dependent jitter due to DC imbalance generated after
> > O/E, that can
> > > > close the eye further again.
> > > >
> > > > Bin
> > > >
> > > > ADL, AMD
> > > >
> > > > > -----Original Message-----
> > > > > From:       Rich Taborek [SMTP:rtaborek@xxxxxxxxxxxxxxxx]
> > > > > Sent:       Thursday, May 27, 1999 3:23 PM
> > > > > To: David Martin
> > > > > Cc: HSSG_reflector; Sachs,Marty; Widmer,Albert_X
> > > > > Subject:    Re: 1000BASE-T PCS question
> > > > >
> > > > >
> > > > > Dave,
> > > > >
> > > > > Do you know of any research or other proofs in this area?
> > You say that
> > > > > lower speed SONET links regularly achieves BERs of < 10 E-15. I have
> > > > > substantial experience with mainframe serial links such as ESCON(tm)
> > > > > where the effective system BERs are in the same ballpark. SONET uses
> > > > > scrambling with long term DC balance and ESCON uses 8B/10B
> > with short
> > > > > term DC balance. The following questions come to mind:
> > > > >
> > > > > - How important is DC balance?
> > > > > - How does this importance scale in going to 10 Gbps?
> > > > >
> > > > > I'll see if I can get some 8B/10B experts to chime in here
> > if you can
> > > > > get scrambling experts to bear down on the same problem.
> > > > >
> > > > > --
> > > > >
> > > > > >(text deleted)
> > > > > >
> > > > > >The point here is that the SONET scrambler is not the
> > limiting issue in
> > > > > >achieving low error rates. The issue is having enough
> > photons/bit, or
> > > > > >optical SNR (eye-Q) to accurately recover the data.
> > > > > >
> > > > > >...Dave
> > > > > >
> > > > > >David W. Martin
> > > > > >Nortel Networks
> > > > > >+1 613 765-2901
> > > > > >+1 613 763-2388 (fax)
> > > > > >dwmartin@xxxxxxxxxxxxxxxxxx
> > > > > >========================
> > > >