RE: Long distance links
Dan:
You are looking at the problem from the point of view of a MAC talking to a
PHY attached to an Optical network. I agree that about 1 packet time for
transmit and 1-2 for receive is required for this configuration.
The issue is what happens when a MAC delivers a 10 Gigbit data steam to a
10 Gigabit PHY which is attached to a link which eventually reaches an
Optical network. At the optical network a transform is performed to
continue the link. The device at the juncture must flow control the link to
slow the data rate to 9.584640. To do so it must have enough buffer to
cover 2*delay*bandwidth. Since this buffer depends on delay it has a direct
dependence on the link length (making the solution scale poorly). A
reasonable design point for wide area equipment is about 25 msec (typical
routers use 200 msec. which is where the design point needs to be for
general applications).
Doing the math on 25 msec gives us 2*25msec*400Mbps/8bpB = 2.5 Mbytes. If
we recalculate the buffer requirement based on the standard router design
point of 200msec we get 2*200msec*400Mbps/8bpB = 20 Mbytes.
Paul
At 10:09 AM 8/31/99 -0600, DOVE,DANIEL J (HP-Roseville,ex1) wrote:
>
>Hi Paul,
>
>First off, thanks for the White Paper reference. I found it informative
>and have forwarded copies off to some people here that are wondering why
>10 Gigabit Ethernet is necessary. I have some questions about the need for
>"several megabytes of memory plus half-switch" on the PHY if a 10.0000 MAC
>is used with an IPG stretching HOLD function though.
>
>I believe that a single packet is all you would have to buffer in that
>configuration. You could begin receiving the packet from the MAC, insert the
>data into the OC-192 stream (including headers) and hold the MAC from
>sending any more packets until you have completed sending the complete
>frame.
>
>As for the "half-switch", that is quite an overstatement when you consider
>that this device is strictly a rate-matching mechanism. There is no address
>hashing, L3 protocol, or other forwarding calculation.
>
>Based on my experience with memory cost and ASIC complexity, I can't believe
>that such elements pose more than a few dollars of cost in a system that
>will have PHY optics that run many hundreds of dollars.
>
>In addition, the 9.58 vs 10.0000 MAC speed aspect should be clarified
>a bit. If I have a switch that runs 100/1000/10000, I am using a 125MHz
>clock oscillator that is PLL multiplied up to 1.25GHz for 1Gig and I only
>have to widen the bus to operate at 10Gig. Everything in the MAC chip is
>synchronous except for the incoming FIFOs which are QUICKLY brought into
>that same clock domain. If I had my druthers, the PHY would perform
>frequency conversion as well to bring everything at the MAC into a single
>clock domain. Using a 9.953280G clock rate, or a 9.58464G clock rate will
>require FIFOs on my transmit side and complicate that ASIC design quite a
>bit. Since that clock is only required for an OC-192 line rate, and the PHY
>has a FIFO to accomodate headers and rate adjustments anyway, it seems like
>the PHY is a better place to add the functionality.
>
>This way, those PHYs that are running in a campus can operate with the same
>clock rate at a lower overall system cost because we are using a single
>clock domain for the ASIC and no FIFOs for the transmit side.
>
>Either way, the cost addition is negligible, but the complexity should go
>where it really belongs (IMO) which is on the PHY that requires it.
>
>Thanks again,
>
>Dan Dove
>
>___________ _________________________________________________________
>_________ _/ ___________ Daniel Dove Principal Engineer __
>_______ _/ ________ dan_dove@xxxxxx LAN PHY Technology __
>_____ _/ ______ Hewlett-Packard Company __
>____ _/_/_/ _/_/_/ _____ Workgroup Networks Division __
>____ _/ _/ _/ _/ _____ 8000 Foothills Blvd. MS 5555 __
>_____ _/ _/ _/_/_/ ______ Roseville, CA 95747-5555 __
>______ _/ ________ Phone: 916 785 4187 __
>_______ _/ _________ Fax : 916 785 1815 __
>__________ _/ __________________________________________________________
>
>
>> -----Original Message-----
>> From: pbottorf@xxxxxxxxxxxxxxxxxx [mailto:pbottorf@xxxxxxxxxxxxxxxxxx]
>> Sent: Monday, August 30, 1999 3:18 PM
>> To: DOVE,DANIEL J (HP-Roseville,ex1); HSSG
>> Subject: RE: Long distance links
>>
>>
>>
>> Dan:
>>
>> At 02:11 PM 8/30/99 -0600, DOVE,DANIEL J (HP-Roseville,ex1) wrote:
>> >
>> >Hi All,
>> >
>> >Have we come to concensus yet?
>> >
>> >I keep seeing things flying across the reflector about how
>> we need SONET
>> >OAM&P, SONET's 9.58464 data rate at the MAC/PLS interface,
>> etc... I thought
>> >that the initial issue was having SONET's signalling at the PHY PMD
>> >interface.
>>
>> I believe management and rate are separate topics. They are
>> coupled in the
>> sense that both become an issue when considering Ethernet in the WAN.
>>
>> >
>> >For those who want a 10.0000 Gigabit Ethernet that operates
>> as a "Local Area
>> >Network" (LAN) and allows us to hook our switches together in wiring
>> >closets, buildings, and campuses cost effectively, we have
>> an architecture
>> >that will work.
>> >
>> > [ 10 Gig MAC ]->10 Gig MII-> 10 Gig PHY (SMF/MMF/Cu)
>> >
>> >For those who want their long-distance WAN solution
>> >
>> > [ 10 Gig MAC ]->10 Gig MII-> 9.58464 SONET PHY (SMF)
>> >
>>
>> The WAN PHY is not SONET. SONET does not specify a method for carring
>> Ethernet frames. SONET requires full management. SONET has
>> very tight clock
>> requirements. The reason we are getting confused is that we
>> are calling the
>> WAN PHY SONET and, in so doing, are carrying more bagage than
>> Ethernet can
>> stand. I believe what is being proposed is the use of some OC-192
>> technology for the WAN PHY which might be called 'SONET
>> light' or 'Thin
>> SONET', but is not OC-192. The issue we are debating is how
>> much of SONET
>> needs to be retained to fill the market requirement for easy
>> interface and
>> operation over existing WAN networks. To make it clear we are
>> not talking
>> about SONET we should refer to the PHY as the WAN PHY.
>>
>> >which uses HOLD as a mechanism to prevent overflow of the SONET PHY
>> >on outbound traffic. Now, I have a few questions about this
>> SONET PHY that
>> >are due to my inexperience with SONET signalling. I am willing to
>> >expose my naivete in the interest of gaining knowledge on
>> the specific
>> >problems with my vision on this matter.
>> >
>> >1) Assuming the HOLD mechanism, what is essentially wrong
>> with using a
>> >10.0000 Gbps MAC/PLS interface?
>>
>> What is wrong with this is it forces the PHY to use a
>> different transmit
>> clock than the MAC since the PHY wants to send a 9.584640
>> Gbps data stream
>> on a 9.953280 Gbaud carrier. A better solution is to use a
>> 9.953280 Gbaud
>> clock at the MAC with a HOLD mechanism. This will allow the
>> MAC and PHY
>> transmitters to be synchronous.
>>
>> If a word-by-word HOLD is used, then the PHY will not require any
>> buffering. Using any of the IPG stretching techniques,
>> including IPG HOLD,
>> will force the PHY to have about 1 packet worth of transmit
>> and 2 packets
>> worth of receive queue. The reason these queues are much larger than
>> required for the rate match is because the transmitter must
>> synchronously
>> transmit overheads mixed in the data stream. While the
>> overheads are being
>> transmitted the PHY must either be able to HOLD the MAC or
>> must buffer as
>> many bytes as the overhead area. On reception the PHY must
>> either be able
>> to put the receiver in HOLD over the overhead area or must
>> have sufficient
>> packet data buffered to feed an entire packet to the MAC
>> synchronously.
>>
>> >
>> >2) Why can't OAM&P functionality be performed in the SONET
>> "PHY" where
>> >it renders the least impact on the overall system architecture?
>>
>> Yes, the WAN PHY could perform OAM&P without impact on the
>> upper layers.
>> The issue is not implementation, but is cost. For connections
>> over photonic
>> networks OAM&P function are required, however for LAN
>> connections they are
>> an overkill. It is very desirable to have a PHY which can
>> operate with or
>> without OAM&P so those who need it pay for it, but those who
>> do not can
>> have cheap interfaces.
>>
>> >
>> > 2a) Can't we use MDIO/MDC to allow management access to the
>> necessary
>> > PHY parameters in case processing is required?
>>
>> Yes
>>
>> >
>> >Now I totally realize that I am using the term "PHY" loosely
>> as the SONET
>> >PHY function may require extensive processing capability. But from an
>> >architectural/specification perspective, it makes it much
>> more clean than to
>> >burden a short-haul system with the same complexities as a
>> long-haul system.
>>
>> This is why SONET is not proposed for the PHY. The OAM&P function does
>> require considerable processing which is expensive for in building
>> applications while very valuable for WAN applications.
>>
>> >
>> >Now, if I have a vision that is workable (subject to debate)
>> why are we
>> >continuing to discuss a 9.58464 MAC/PLS interface or OAM&P
>> functionality in
>> >the MAC?
>> >
>> >Thanks,
>> >
>> >Dan Dove
>> >HP Workgroup Networks
>> >
>> >PS: Is there a good white-paper on SONET signalling on the web?
>>
>> Have a look at:
>>
>http://www.nortelnetworks.com/products/library/collateral/87003.25-08-99.pdf
>
>>
>>
>>
>Paul A. Bottorff, Director Switching Architecture
>Enterprise Solutions Technology Center
>Nortel Networks, Inc.
>4401 Great America Parkway
>Santa Clara, CA 95052-8185
>Tel: 408 495 3365 Fax: 408 495 1299 ESN: 265 3365
>email: pbottorf@xxxxxxxxxxxxxxxxxx
>
>
Paul A. Bottorff, Director Switching Architecture
Enterprise Solutions Technology Center
Nortel Networks, Inc.
4401 Great America Parkway
Santa Clara, CA 95052-8185
Tel: 408 495 3365 Fax: 408 495 1299 ESN: 265 3365
email: pbottorf@xxxxxxxxxxxxxxxxxx