RE: Unified PMD vs. Unified PHY

It is true that the average frame size is significantly smaller than the maximum, but this doesn't change the fact that IPG header compression isn't going to get you where you need to be. You can bring the WAN PHY a little closer to the LAN PHY bandwidth. One percent closer. Not close enough to remove the need for rate matching, etc, but it's closer.

But discussing the average size of frames is not helpful, any more than saying "most links aren't 100% utilized and so why not go with 8Gb and call it 10Gb". Unless, of course, it is always qualified: i.e. "with IPG header compression we can recover 3% of the data rate (assuming the current average internet traffic)". Without the qualification, we should only discuss the worst case (1%).

Claiming that the WAN PHY can meet the 10.000Gb rate using header compression, as has been done, is totally misleading.

-Simon

P.S. "Caveat Emptor" applies to the use of Jumbo frames. It is one thing to change a few parameters in code to support a different MTU. It's another thing to ensure that it works in all corner cases. For example, many components (PHYs, MACs, etc) include elasticity FIFOs the depth of which is calculated by knowing the worst case difference in clock rates and the largest packet size. Are all the PHY/MAC vendors in the link path making their elasticity FIFOs six times larger to support Jumbo Frames? If they don't specifically say they are, they probably aren't. I guess, either way, it'll usually work. Mostly, even, but not always.

-----Original Message-----
From: owner-stds-802-3-hssg@xxxxxxxx [mailto:owner-stds-802-3-hssg@xxxxxxxx]On Behalf Of Roy Bynum
Sent: Tuesday, March 14, 2000 9:35 PM
To: McCormick, Corey
Cc: 802.3ae
Subject: Re: Unified PMD vs. Unified PHY

Corey,

Actually, the reflector is probably a very good place to discuss this. I think that you have an interesting question that should be addressed. It goes directly to the efficiency of improving MAC transfer rate by IPG compression.

The average datagram size that I refereed to was originally from Internet MCI and then again from vBNS and UUNet. These are diverse sources, in both time and typical access. Internet MCI was functional before the massive numbers of high speed dial up ISPs came into being. The vBNS network is primarily universities with direct circuit access. UUNet is a NSP that supports dial ISPs and commercial services with direct circuit access. That the average datagram size has not changed speaks more to the applications on the desk tops and in the homes than to the access method. As more and more voice, which uses a very small datagram, and video, which uses a large datagram, become even more prevalent, the weighting at the extremes of frame sizes will increase, but I don't think that will change the overall average Ethernet frame size at the higher bandwidths such as 10 Gigabit.

While the P802.3ae TF has stated that it will not take on the issue of increasing the MTU size, I have been told that almost all of the GbE vendors support jumbo frames. One of the problems of having larger MTU size years ago was that the technology and cost required to support any larger frames was a paradigm based probem. It goes back to issue why the original microprocessors chips only supported a limited amount of memory, or why the original PC only supported 640K of application memory. Hindsight is allways 20/20. Those of us that have been around long enough and worked in enough diverse environments, to understand how and why things got where they are, have the oportunity to perhaps avoid some of the same types of mistakes.

Thank you,

Roy Bynum

----- Original Message -----

From: McCormick, Corey

To: Roy Bynum

Sent: Tuesday, March 14, 2000 10:57 PM

Subject: RE: Unified PMD vs. Unified PHY

I did not know if this was correct for the reflector, so I thought I would take this offline.

While the average size of ~380 may be correct, this new standard is for the future. The current ~380 byte size I believe is due to the presence of predominately a Wintel architecture throughout the Internet community. The current installed base of Wintel 95/98 dial-up networking uses a default IP MTU of 576 for all connections less than ISDN 2B (128Kbit/sec). That means to me:

Dial-up connections over the next few years are being replaced with technologies that push the average packet size way up:

* Cable-modems, Satellite and xDSL - 1500 MTU
* E business-business transactions - 1500 MTU
* VPN tunnels. - default MTU + VPN wrappers
* Faster speeds mean more graphics-rich traffic as the users continue to crave higher bandwidth commodities (Video, Audio, Visually Interactive content, etc...) These tend to fill up the TCP receive window with more full-size packets versus smaller ones.

* Selective acknowledgements being added to the installed base now mean the 20-40 byte ack packets that drive the average down low now will continue to wane.

* Win2K support of RFC-1323 TCP options for very large TCP receive windows (640Kbytes+) will mean many more (400+) full-size packets can be "in flight" before the small ack packets are seen.

I believe these will all push the average packet size up quite a bit, while voice over IP will tend to want to drive it down.

I guess what I am saying is that I think 400 maybe too small to assume. I am not arguing for or against the Uniphy just sharing some experience...

As a side note, where I hang my work hat these days, we are using 9Kbyte Jumbo Frame Ethernet since we cannot afford the CPU overhead of even single GigE speeds on current systems. The TCP segmentation/checksum operations are just too great with these small (1500) MTU packets to utilize the speed of our GigE pipes. As the adapters/OS interfaces get smarter, some of this will get better I am sure, but much larger would be better for almost everything except the Internet. Vendor complain about the compatibility and ASIC troubles, but the users have to live with things for many years after the ASICs are done and the products are no longer being sold. Current architectures just do not have the capability to do all this well. We are only running normal business apps SAP, Database, WP, Printing, etc... for these things. (no specialty seismic, simulation, etc...) Besides, even Bob Metcalf said if he could change one thing, it would be to make the MTU larger.

Thanks for your time, you all have a lot to work out, but I am sure you will get there.

Good Luck...

-Corey McCormick

CITGO Petroleum

-----Original Message-----
From: Roy Bynum [mailto:rabynum@xxxxxxxxxxxxxx]
Sent:   Tuesday, March 14, 2000 8:34 AM
To:     Benjamin J. Brown; 802.3ae
Subject:        Re: Unified PMD vs. Unified PHY

Ben,

The expense in transfer rate is an addtional 3% above the ~4% of the SONET
framing. This makes the total bandwidth expense of the Unified PHY close to
7%. This is almost half of the overhead cost of ATM.

With the proposal of IPG compression in the PHY, most of the ~4% overhead of
the SONET framing can be recovered. The overhead recovery will be more
effective with small frames than with large frames, but I believe that it
will average out. At present, I have been told that the average IP datagram
on the Internet is 380 bytes. This is the same as it was two years ago, so
it does not seem to be shifting very much. From this information, an
average of 400 bytes can be somewhat safely used to determine the average
overhead recovery that can be achieved with frame stuffing as proposed by
Nortel and Lucent. With a reduction of the IPG by 10 bytes, using an
average 400 byte frame (with current IPG, 420bytes), 2.3% average overhead
recovery can be added to the MAC transfer rate.

With IPG recovery using frame stuffing, the overhead cost of the WAN phy
becomes ~1.7%. Compared to the ~7% overhead of the 64B/66B proposal, that is
a difference of 6.3%.   This makes the cost of the unifed PHY at least 6.3%
greater than the seperate WAN PHY. I think that the original compromise and
the objectives as stated are correct, there needs to be seperate LAN and WAN
PHYs.

Thank you,
Roy Bynum

----- Original Message -----
From: Benjamin J. Brown <bebrown@xxxxxxxxxxxxxxxxxx>
To: 802.3ae <stds-802-3-hssg@xxxxxxxx>
Sent: Monday, March 13, 2000 8:50 AM
Subject: Re: Unified PMD vs. Unified PHY

>
>
> Roy,
>
> Let's please keep this on the reflector so everyone can follow
> along with the discussion. This way, others with similar concerns
> or questions won't be kept in the dark.
>
> A question has been raised regarding how tightly coupled the
> XAUI and 64b/66b encodings are or need to be. Several people,
> including me, have voiced the opinion that there shouldn't
> be any requirement that 64b/66b uses the encoding of XAUI.
>
> As for the expense in transfer rate, I'm a little confused. I
> believe Howard Frazier pointed out that over WAN, the 64b/66b
> encoding scheme is somewhat less efficient (3%?) than a
> scrambled encoding. I agree this is an issue worth discussing
> but it is a PCS issue, not a PMD one.
>
> Look at a serial PHY. From the MAC to the PCS is an XGMII.
> Some implementations may choose to extend this XGMII using
> XAUI but this interconnect is optional. The PCS should not
> require any features of the XAUI. The PCS encodes the MAC
> data from the XGMII then this data is serialized and driven
> onto the fiber. The encoding scheme within the PCS is the
> factor which determines the required baud rate on the fiber.
>
> Because we chose to make as an objective the support of a
> WAN compatible PHY, we chose a baud rate of 9.95328 G for
> the PMA/PMD. To share this PMA/PMD with serial LAN solutions
> (in order to reduce the number of discreet PMA/PMDs in the
> standard), we'd like to choose an encoding scheme for the
> LAN which shares this baud rate (or something close enough
> that works). We're kind of working this problem backwards.
>
> We'd also like to have a common encoding scheme (or as
> common as possible) between the WAN and the LAN. For both
> of these reasons, we're looking at 64b/66b and scrambling.
> Both of these can support a common baud rate necessary to
> reduce the number of PMA/PMDs and a common encoding scheme
> necessary to support the results of Jonathan's survey.
>
> Ben
>
> Roy Bynum wrote:
> >
> > Ben,
> >
> > Gb-Mtr is an acronym that I created because I quickly got tired of
> > repeatedly spelling out "Gigbit MAC transfer rate". The real question
was
> > not relative to the baud rate of a LAN PMD vs a WAN PMD, but the
confusion
> > that has been introduced by the effort to "unify" the PHY. XAUI/64B66B
> > encoding makes XAUI a requirement, and efforts to reduce the PMD rate to
a
> > single common is going to be very expensive in transfer rate. By
abandoning
> > the "Hari" based 8B10B block encoding, the frame stuffing proposals by
> > Nortel and Lucent give the ability recover much if not all of the MAC
> > transfer rate.
> >
> > Johnathan has been using the object of having common PMDs as the reason
for
> > supporting a PHY that provides a specific vendor the ability to maintain
the
> > 8B10B to be required at the MAC chip. The issue is to segregate the
issue
> > of common PMDs from that of a common PHY, so that the requirement for
8B10B
> > can be released.
> >
> > Thank you,
> > Roy Bynum
> >
> > ----- Original Message -----
> > From: Benjamin J. Brown <bebrown@xxxxxxxxxxxxxxxxxx>
> > To: 802.3ae <stds-802-3-hssg@xxxxxxxx>
> > Sent: Sunday, March 12, 2000 3:27 PM
> > Subject: Re: Unified PMD vs. Unified PHY
> >
> > >
> > >
> > > Roy,
> > >
> > > I realize you asked your question to Jonathan, but if you don't
> > > mind I'll try an answer to this.
> > >
> > > In support of the WAN, the serial PMDs (and PMAs) must support
> > > a 9.95328 Gbaud rate. I think it was fairly clear from early
> > > on that using an 8b10b encoding for the LAN would require a
> > > 12.5 Gbaud rate and that the PMA/PMD for LAN & WAN could not
> > > be identical (as the WAN PMA/PMD doesn't simply scale up in
> > > baud rate).
> > >
> > > I believe that is the idea behind the 64b/66b and SLP proposals
> > > as these encodings require 10.3125 and 10.000 Gbaud rates,
> > > respectively. These baud rates are within the range of current
> > > WAN PMA/PMDs to achieve. This means for the serial PMA/PMDs,
> > > a single solution can be generated (or perhaps 2 - longwave
> > > and shortwave) and dialed with an appropriate oscillator to
> > > support the WAN rate (9.95328 Gbaud) or the LAN rate (10.3125
> > > or 10.000 Gbaud).
> > >
> > > The PMA/PMD cares little about the content of the data going
> > > onto or coming off of the fiber. The encoding affects the baud
> > > rate in order to account for overhead.
> > >
> > > BTW: What is a Gb-Mtr?
> > >
> > > Ben
> > >
> > > Roy Bynum wrote:
> > > >
> > > > Johnathan,
> > > >
> > > > I was intending to ask you why you did not ask about unified PMDs
> > > > separate from a unified PHY as part of your survey but did not get a
> > > > chance. At the 10GEA technical meeting you were very adamant about
> > > > getting consensus for a small set of PMDs. I agree that having a
small
> > > > group of PMDs is preferable. Having a unified PHY in order to have
a
> > > > small set of PMDs may not be preferable.
> > > >
> > > > The cost of the unified PHY, as presented, so far has been very high
in
> > > > the form of lost transfer rate. As it is, the unified PHY, as
> > > > presented, does not meet the objective to have a 10.000 Gigabit MAC
> > > > data transfer rate (Gb-Mtr). Separate PHYs, LAN and WAN do meet the
> > > > objectives. Additionally, one of the scramble encoded WAN PHY
> > > > presentations was able to achieve an average 10.000 Gb-Mtr transfer
rate
> > > > by using IPG compression, which can be inferred to meet the 10.000
> > > > Gb-Mtr objective in addition to the 9.548 Gb-Mtr objective.
> > > >
> > > > A unified PMD set can support the block encoded LAN PHY and the
scramble
> > > > encoded WAN PHY, allowing both to meet the 10.000 Gb-Mtr objective.
> > > > This will allow the PMD people to concentrate on the technologies of
the
> > > > PMDs with the consideration of a signaling range to support both
PHYs.
> > > > It will also simplify the marketing of 10GbE by reducing the
confusion
> > > > about distances and fiber types.
> > > >
> > > > As was demonstrated in some of the previous presentations (SUPI and
OIF
> > > > SERDES), it is possible to have unified PMDs without having a
unified
> > > > PHY. If the question had been asked, would it have made a
difference to
> > > > separate the issues? If they are separate issues, as a I believe
they
> > > > are, then should the survey be redone with that segregation? Would
this
> > > > have put less pressure on group to have a unified PHY and changed
the
> > > > scaling of the responses?
> > > >
> > > > Thank you,
> > > > Roy Bynum
> > >
> > >
> > > --
> > > -----------------------------------------
> > > Benjamin Brown
> > > Router Products Division
> > > Nortel Networks
> > > 1 Bedford Farms,
> > > Kilton Road
> > > Bedford, NH 03110
> > > 603-629-3027 - Work
> > > 603-629-3070 - Fax
> > > 603-798-4115 - Home
> > > bebrown@xxxxxxxxxxxxxxxxxx
> > > -----------------------------------------
>
>
> --
> -----------------------------------------
> Benjamin Brown
> Router Products Division
> Nortel Networks
> 1 Bedford Farms,
> Kilton Road
> Bedford, NH 03110
> 603-629-3027 - Work
> 603-629-3070 - Fax
> 603-798-4115 - Home
> bebrown@xxxxxxxxxxxxxxxxxx
> -----------------------------------------