Re: On two clock issue
Larry,
Sorry about the delay in getting back to you. I took this issue to the person
who is recognized as being THE authority on synchronization and clocking within
MCI WorldCom. He agrees that independent, privately owned fiber implementations
can be plesiosynchronious, the transmit data rate defining the clock rate for
the far end receiver, with the transmit and receiver clocks running independent
within a T/R port. This is how LANs work today. This can also be implemented
when using leased fiber where there are not any active systems as part of the
fiber services, such as active DWDM. This would include the intelligent Metro
DWDM systems that do independent clock recovery separately on the transmit and
receive of GbE service interfaces. He does insist that clocking be
isosynchronious when 10GbE is implemented as path or tributary to active types
of systems, similar to how POS interfaces work today, with the SONET/SDH systems
providing line timing to both the transmit and receive clocks of the T/R port.
Thank you,
Roy Bynum
MCI WorldCom
Larry Miller wrote:
> Roy,
>
> I agree that not even SONET/SDH is truly synchronous in the mechanical gear
> train sense. As I understand it,
>
> 1) Data is transmitted with placeholders for padding flags and padding at a
> prescribed timing accuracy. This accuracy is such that
>
> 2) The receiver, also using its own local time base of prescribed accuracy
> can, once it get synchronized, find these timing marks "by Braille", i.e.,
> by just counting off time cycles. In a point to point (terminating) link, I
> guess the principal use of this is to ignore them. If passing data through,
> these must be preserved or modified.
>
> 3) More importantly, "3R" regenerators and add/drop multiplexers can modify
> the padding flags and amounts to keep the system in long-term synch over
> multiple hops without having to fiddle with the actual data content in the
> cells, since these padding flags and bits or bytes occur at specified (i.e.,
> "known" times and, consequently, positions, in the cell) from the start of
> synch attainment.
>
> Without this cut/pad to fit capability at all levels in SDH, you have the
> old problem of elasticity buffer crawl that you get with, say, back-to-back
> RS-232 UARTS running off separate crystals. ("Trad" Ethernet uses the IPG to
> allow for avoidance of multi-hop and speed-dependent data pileup or suckout,
> and RS-232 uses the start and stop bits for the same purpose.)
>
> The respective clocks' precision(s) must be compatible with the allowable
> amount of padding, in addition to other timing disturbances.
>
> Is my understanding incorrect?
>
> On I Gb we allow quite a lot of time variation (typically 200 ps peak to
> peak), and even somewhat more at low frequencies. (The receiver PLL can
> follow low frequencies, but you have to have some limit to keep FIFO sizes
> reasonable. We measure our crystal clock oscillators (of the $8.00 variety)
> with a 634 kHz single pole high-pass filter. I think that Fibre Channel also
> does this. The 100 PPM is a "DC" frequency specification, so the overall
> speeds (baud rates) of the two directions of the link could differ by as
> much as 0.1% and still be conforming.)
>
> Anyway, what timing accuracy do you think is needed? What do you get for
> specifications at the PMD level in systems you do?
>
> Thanks,
>
> Larry Miller
>
> -----Original Message-----
> From: Roy Bynum <rabynum@xxxxxxxxxxx>
> To: Larry Miller <l_d_miller@xxxxxxxxxxxxx>
> Cc: HSSG <stds-802-3-hssg@xxxxxxxx>
> Date: Tuesday, July 20, 1999 4:17 PM
> Subject: Re: On two clock issue
>
> >Larry,
> >
> >Actually, Packet Over SONET/SDH interfaces in routers today do not have the
> same
> >level of synchronization as true SONET/SDH. Payload synchronization alarms
> are
> >turned off on the transport systems to allow this. In reality, the router
> >interfaces use line timing derived from the transport system, just like any
> >other WAN interface. Synchronization is achieved in the SONET/SDH
> transport
> >system, not the data end system. The issue of tight synchronization
> >requirements is unfounded.
> >
> >Thank you,
> >Roy Bynum
> >MCI WorldCom
> >
> >Larry Miller wrote:
> >
> >> Roy,
> >>
> >> Ethernet is (so far) asynchronous.
> >>
> >> Each transmitter runs at its own speed and phase (within the allowed
> >> tolerances, which are huge compared to SONET, typically +/-100 ppm), and
> >> each receiver recovers its clock from the received data.
> >>
> >> And, Yes, they free run (each Gigabit Ethernet receiver switches over to
> its
> >> local transmit clock in the absence of incoming signals to minimize
> receive
> >> PLL acquisition time), and, Yes, the transmit and receive data rates
> differ.
> >> One of the main purposes of the Inter-Packet Gap (IPG is to allow for
> this).
> >>
> >> The only exception I know of is 1000BASE-T where the timing/phase
> accuracy
> >> of the echo cancellation and line compensation filters (simultaneous
> >> transmit/receive on same copper wires) is so tight that one end of the
> link
> >> has to supply clock for both directions, and a master-slave relationship
> is
> >> negotiated and set up for the link (but not synchronous to the cosmos--
> >> still +/-100 ppm for the clock used for both directions).
> >>
> >> The POS line cards I have seen need considerable buffering and flow
> control
> >> to get into/out of the synchronous world.
> >>
> >> Some of you guys are trying to hammer Ethernet into SONET/SDH in a big
> way.
> >>
> >> This interface between the synchronous and asynchronous worlds, as has
> been
> >> pointed out by many here, will not go away, but will simply move into
> >> greater or less costly or inconvenient points in the data paths.
> >>
> >> Since the synchronous world requires terrifically accurate time bases
> >> (single Ethernet links can tolerate very large timing differences in
> >> comparison, like 10X that of SONET without even thinking about it much--
> >> hey, it's A-synchronous) you should expect and will get a lot of
> resistance
> >> to forcing the Ethernet world into the expensive SONET performance
> >> requirements and costs.
> >>
> >> Cheers,
> >>
> >> Larry Miller
> >> Nortel Networks
> >>
> >> -----Original Message-----
> >> From: Roy Bynum <rabynum@xxxxxxxxxxx>
> >> To: rtaborek@xxxxxxxxxxxxxxxx <rtaborek@xxxxxxxxxxxxxxxx>
> >> Cc: HSSG <stds-802-3-hssg@xxxxxxxx>
> >> Date: Monday, July 19, 1999 8:50 PM
> >> Subject: Re: On two clock issue
> >>
> >> >
> >> >Rich,
> >> >
> >> >Just a note of clarification. If the receive clock is separate from the
> >> system clock,
> >> >what controls the transmit clock. Does each link transmit bit rate
> provide
> >> the clock
> >> >to the other system's receiver? Would this tend to cause free running
> and
> >> separate
> >> >transmit and receive data rates? How do existing OC48C POS interfaces
> use
> >> line timing
> >> >from the carrier LTE to control receive as well as transmit speeds? If
> >> 802.3 were to
> >> >be used on anything other than enclosed LANs, what would be available to
> >> provide line
> >> >timing? Are there alternatives?
> >> >
> >> >Thank you,
> >> >Roy Bynum
> >> >MCI WorldCom
> >> >
> >> >Rich Taborek wrote:
> >> >
> >> >> Devendra,
> >> >>
> >> >> We're taking a few too many tangents here. To bring everyone else up
> to
> >> speed, the
> >> >> root issue is a MAC/PLS rate of 10.0 Gbps vs. 9.584640 Gbps. I
> asserted
> >> that a
> >> >> 9.584640 Gbps would complicate the clock design of Ethernet equipment
> due
> >> to the
> >> >> requirement to support a rate which is not a multiple of slower
> Ethernet
> >> MAC/PLS
> >> >> local clocks.
> >> >>
> >> >> Receive clocks are separate from local clocks. I believe that most
> >> Ethernet
> >> >> equipment supporting multiple interfaces DOES NOT use the receive
> clock
> >> from any
> >> >> interface as its local clock.
> >> >>
> >> >> Devendra Tripathi wrote:
> >> >>
> >> >> > Rich,
> >> >> >
> >> >> > One clarification here. The use of two clocks is almost always
> there.
> >> >> > Ofcourse
> >> >> > some designs carry the retreived clock all the way to host fifo
> >> interface
> >> >> > and some
> >> >> > terminate it early. I am doubtful, though, if anyone uses that clock
> >> for
> >> >> > transmit
> >> >> > purpose. Because in that case, the reference clock (which is taken
> as
> >> transmit
> >> >> > clock by PHYs) also has to be retrieved one, thus making a loop in
> the
> >> PLL.
> >> >> >
> >> >> > Thanks,
> >> >> > Tripathi.
> >> >> >
> >> >> > At 10:17 AM 7/19/99 -0700, you wrote:
> >> >> > >
> >> >> > >Roy,
> >> >> > >
> >> >> > >This one's pretty simple. The MAC/PLS clock for 10 Mbps Ethernet is
> >> based
> >> >> > on the
> >> >> > >transport of bits. At 100 Mbps, it's nibbles (4 bits). At 1 Gbps,
> it's
> >> >> > octets. At 10
> >> >> > >Gbps, several proposals suggest 8, 16 and 32 bit chunks. The local
> >> clock
> >> >> > for Ethernet
> >> >> > >equipment which performs speed matching is easily derived from
> MAC/PLS
> >> >> > clock of the
> >> >> > >slowest interface supported. Deriving a 9.584640 is not so
> >> >> > straightforward. Reverse
> >> >> > >deriving lower speed clocks is also not so straightforward. The
> >> simplest
> >> >> > solution is
> >> >> > >to use two clocks. This solution increases implementation cost and
> >> will
> >> >> > significantly
> >> >> > >complicate clocking design.
> >> >> > >
> >> >> > >Best Regards,
> >> >> > >Rich
> >> >> > >
> >> >> > >--
> >> >> > >
> >> >> > >Roy Bynum wrote:
> >> >> > >
> >> >> > >> Rich,
> >> >> > >>
> >> >> > >> I hear much about the single system clock issue. In the past,
> the
> >> data
> >> >> > transport
> >> >> > >> signal clock and the data system bus and processing clock were
> >> separate.
> >> >> > For many
> >> >> > >> systems the data processing would exceed the transport signal in
> >> order
> >> >> > to maintain
> >> >> > >> control. If that is the case, then your argument about a single
> >> clock
> >> >> > does not
> >> >> > >> hold. I could be wrong. Would one of the system implementation
> >> people
> >> >> > please
> >> >> > >> define how a data signal clock is derived in today's systems. Is
> >> >> > internal 802.3
> >> >> > >> frame processing done at a higher rate than the transport signal
> >> rates
> >> >> > in today's
> >> >> > >> GbE L2 switches? Is there a real issue with a separate transport
> >> signal
> >> >> > rate or
> >> >> > >> even a separate transport data rate?
> >> >> > >>
> >> >> > >> Thank you,
> >> >> > >> Roy Bynum
> >> >> > >> MCI WorldCom
> >> >> > >>
> >> >> > >> Rich Taborek wrote:
> >> >> > >>
> >> >> > >> > Hon Wah,
> >> >> > >> >
> >> >> > >> > Thanks for kicking this issue off again!
> >> >> > >> >
> >> >> > >> > Hon Wah Chin wrote:
> >> >> > >> >
> >> >> > >> > > Perhaps a difficult number to remember, but with the +-
> 100ppm
> >> >> > tolerance
> >> >> > >> > > and a bit rate that needs only to fit within about 200ppm of
> the
> >> >> > nominal
> >> >> > >> > > SONET number we should be able to choose a round number with
> 4
> >> >> > digits in it.
> >> >> > >> > >
> >> >> > >> > > ---
> >> >> > >> > >
> >> >> > >> > > As I understand the presentations in Montreal on speed,
> >> >> > >> > > a strong advantage of choosing this OC-192 payload rate is
> >> >> > >> > > to transport the signal over SONET OC-192 equipment. This
> would
> >> >> > >> > > be from a "10Gb/s Ethernet" port out to SONET gear, which is
> >> really
> >> >> > >> > > a PMD external interface rather than a definition for the
> >> MAC/PLS
> >> >> > interface
> >> >> > >> > > and data rate.
> >> >> > >> >
> >> >> > >> > In my conversations with several folks on both sides of the
> issue
> >> >> > during the
> >> >> > >> > Montreal meetings, I've come to the conclusion that the root
> >> reasons
> >> >> > to select
> >> >> > >> > either a 10 or 9.584640 Gbps are purely ease-of implementation
> >> based
> >> >> > and have no
> >> >> > >> > architectural basis whatsoever. I believe this to be true on
> both
> >> >> > sides of the
> >> >> > >> > argument with the choice of one over the other, rendering the
> >> >> > implementation
> >> >> > >> > (i.e. product cost) of the losing side only slightly more
> >> difficult.
> >> >> > Please
> >> >> > >> > allow me to explain the basis of this contention:
> >> >> > >> >
> >> >> > >> > 1) SONET, and specifically synchronous transport, is legacy in
> the
> >> MAN
> >> >> > and WAN,
> >> >> > >> > will never be replaced by Ethernet completely or even quickly.
> >> >> > Ethernet will
> >> >> > >> > make inroads into "green-field" applications, but SONET will be
> >> king
> >> >> > for some
> >> >> > >> > time to come;
> >> >> > >> >
> >> >> > >> > 2) Ethernet, and specifically packet-based transport, is legacy
> in
> >> the
> >> >> > LAN, is
> >> >> > >> > growing in its dominance in the LAN, and will likely gain
> market
> >> share
> >> >> > in the
> >> >> > >> > LAN as well as encroach on other non-traditional Ethernet
> >> transports
> >> >> > including
> >> >> > >> > MAN, SAN, and some WAN. I don't include WAN access in WAN.
> Instead
> >> I
> >> >> > include WAN
> >> >> > >> > access in LAN or MAN;
> >> >> > >> >
> >> >> > >> > 3) The existing WAN infrastructure does a great job of
> >> transporting
> >> >> > Ethernet
> >> >> > >> > packets end-to-end today. However, much protocol conversion and
> >> >> > equipment to map
> >> >> > >> > between packets and TDM bits exists in mapping Ethernet to the
> WAN
> >> at
> >> >> > each end.
> >> >> > >> > Considerable savings can be realized by architecting a more
> >> seamless
> >> >> > Ethernet to
> >> >> > >> > SONET connection. This issue seems to be at the root of the 10
> vs.
> >> >> > 9.584640 Gbps
> >> >> > >> > issue.
> >> >> > >> >
> >> >> > >> > 4) There seems to be no intent by either side to consider any
> >> other
> >> >> > changes but
> >> >> > >> > speed as a HSSG objective. Therefore, Ethernet will remain a
> >> simple,
> >> >> > general
> >> >> > >> > purpose, packet-based transport, and SONET will remain a
> specific
> >> purpose
> >> >> > >> > (MAN/WAN), synchronous transport no matter which way the
> decision
> >> goes.
> >> >> > >> >
> >> >> > >> > 5) Consider a Ethernet to OC-192 line card (feeding a fiber or
> >> >> > wavelength) in
> >> >> > >> > operation. Assume that receive and transmit paths are separate
> on
> >> the
> >> >> > SONET side
> >> >> > >> > and related (i.e. full duplex) on the Ethernet side:
> >> >> > >> > a) Ethernet -> SONET @ 9.584640 Gbps: The Ethernet side can
> >> >> > continuously feed
> >> >> > >> > the SONET link with no flow control required.
> >> >> > >> > b) Ethernet -> SONET @ 10 Gbps: The Ethernet side must be
> flow
> >> >> > controlled to
> >> >> > >> > prevent over-feeding the SONET link
> >> >> > >> > c) SONET -> Ethernet @ 9.584640 or 10 Gbps: The Ethernet side
> >> can
> >> >> > continuously
> >> >> > >> > source SONET data but will flow control or drop packets
> downstream
> >> >> > whenever the
> >> >> > >> > network is congested.
> >> >> > >> >
> >> >> > >> > Therefore, the issue boils down to one of implementation of
> >> existing
> >> >> > Ethernet
> >> >> > >> > mechanisms such as 802.3x flow control or a reasonable
> facsimile
> >> on
> >> >> > the line
> >> >> > >> > card versus complicating the implementation of all Ethernet
> >> products
> >> >> > which must
> >> >> > >> > support a MAC/PLS rate which is not a multiple of 10. These
> >> >> > implementation
> >> >> > >> > difficulties include multiple clocks which may "beat" against
> each
> >> >> > other, not
> >> >> > >> > being able to easily feed 10 slower links into one faster one,
> and
> >> >> > numerous
> >> >> > >> > other difficulties which are best listed by Ethernet product
> >> >> > implementers.
> >> >> > >> >
> >> >> > >> > My intention is not to make light of the problem but rather to
> >> agree
> >> >> > with a
> >> >> > >> > solution direction along the line proposed by Dan Dove of HP at
> >> the
> >> >> > Montreal
> >> >> > >> > meeting. I believe that Dan's general direction was to tradeoff
> a
> >> simple
> >> >> > >> > architectural change with respect to MAC operation to enable
> cost
> >> >> > effective 10
> >> >> > >> > Gbps to SONET implementations. I don't particularly agree with
> >> resolving
> >> >> > >> > implementation cost issues between two dominant legacy
> protocols
> >> by
> >> >> > tweaking
> >> >> > >> > with the underlying architecture, but I'll raise my hand in
> >> support of
> >> >> > this
> >> >> > >> > solution to the problem.
> >> >> > >> >
> >> >> > >> > Such a solution would enable the implementation of a 10 Gbps
> >> Ethernet
> >> >> > to SONET
> >> >> > >> > OC-192 line card without requiring a full MAC.
> >> >> > >> >
> >> >> > >> > I'll let Dan fill in the details of his proposal so I don't get
> it
> >> >> > wrong if it
> >> >> > >> > is still applicable.
> >> >> > >> >
> >> >> > >> > Best Regards,
> >> >> > >> > Rich
> >> >> > >> >
> >> >> > >> > --
> >> >> > >> >
> >> >> > >> > > Given a raw continuous bit stream at the PMD, some scheme for
> >> >> > >> > > framing packets would be needed. 10M used a carrier, 100M
> used
> >> coding,
> >> >> > >> > > 1000M used coding. Using coding where the PMD speed is fixed
> at
> >> >> > 9.58Gb/s
> >> >> > >> > > would mean a further speed reduction (probably 10-20%) at the
> >> MAC/PLS
> >> >> > >> > > interface. The discussion at the meeting has already started
> to
> >> >> > consider ways
> >> >> > >> > > of
> >> >> > >> > > reducing the useful throughput at the MAC/PLS below the data
> >> >> > clocking rate.
> >> >> > >> > > An
> >> >> > >> > > alternative framing scheme presented to HSSG, which has a
> >> smaller
> >> >> > throughput
> >> >> > >> > > reduction, requires a packet length header -- a departure
> from
> >> >> > previous 802
> >> >> > >> > > practice.
> >> >> > >> > >
> >> >> > >> > > In considering the advantage of leveraging SONET OC-192
> >> transport
> >> >> > >> > > we should also consider the issues which come up in actually
> >> getting
> >> >> > >> > > the hoped-for benefits. It would also be worthwhile to
> >> carefully
> >> >> > consider
> >> >> > >> > > what volume forecasts for the OC-192 components can be
> >> documented, in
> >> >> > >> > > evaluating the advantage to be gained. Counting IEEE802.3
> >> 10Gb/s data
> >> >> > >> > > ports (however the definition works out) to get 2 million
> ports
> >> sounds
> >> >> > >> > > good, but I found the forecast of 2,000,000 OC-192 ports in
> 2000
> >> rather
> >> >> > >> > > surprising.
> >> >> > >> > >
> >> >> > >> > > -hwc
> >> >> > >
> >> >> > >-------------------------------------------------------------
> >> >> > >Richard Taborek Sr. Tel: 650 210 8800 x101 or 408 370 9233
> >> >> > >Principal Architect Fax: 650 940 1898 or 408 374 3645
> >> >> > >Transcendata, Inc. Email: rtaborek@xxxxxxxxxxxxxxxx
> >> >> > >1029 Corporation Way http://www.transcendata.com
> >> >> > >Palo Alto, CA 94303-4305 Alt email: rtaborek@xxxxxxxxxxxxx
> >> >> > >
> >> >>
> >> >> --
> >> >>
> >> >> Best Regards,
> >> >> Rich
> >> >>
> >> >> -------------------------------------------------------------
> >> >> Richard Taborek Sr. Tel: 650 210 8800 x101 or 408 370 9233
> >> >> Principal Architect Fax: 650 940 1898 or 408 374 3645
> >> >> Transcendata, Inc. Email: rtaborek@xxxxxxxxxxxxxxxx
> >> >> 1029 Corporation Way http://www.transcendata.com
> >> >> Palo Alto, CA 94303-4305 Alt email: rtaborek@xxxxxxxxxxxxx
> >> >
> >> >
> >