Re: Reply to note by R. Taborek, `Hari Byte vs. Word Striping', dated26 Nov 1999
widmer@xxxxxxxxxx wrote:
>
> This is a note in reply to Rich Taborek's note (26 Nov 1999) on Byte vs.
> Word striping.
>
> Important Characteristics of the word-striped Hari Proposal
>
> A list of reasons, why I believe a word-striped Hari is the most attractive
> technical solution for the 10 GbE and Fibre Channel application is
> summarized here:
>
> 1) The word-striped Hari keeps a 4-byte transport structure intact
> through all levels and configurations of data transmission with a uniform
> orientation (parallel/serial). (For idles, skips, and initialization, the
> byte-striped proposal uses on the 4 lanes a combination of parallel and
> serial formats in a single transport layer resulting in a two dimensional
> operating space, which is at the root of many of the difficulties for
> defining suitable `code blocks' and arriving at simple models and
> implementations). The word-striped proposal does not allow any parallel
> formats in the serial transmission domain. No deviations from this
> structure are needed for the benefit of synchronization (byte, word,
> frequency compensation) regardless of the number of transmission lanes. So
> any protocol which is compatible with the 4-byte wide structure can
> readily be accommodated and others can be made compatible without much
> pain, as is the case for 10 GbE framing per Howard Frazier, without its
> skip and idle definitions.
On the contrary, Byte-Striping naturally transports 4-byte data increments
delivered by the 10 GbE MAC across a 4-byte wide (32-bit) XGMII, across a
parallel arrangement of 4 serial lanes (Hari), for transmission across the
medium. Subsequent to PMD reclocking to remove system jitter, each of the 4
proposed PMD variants, Serial, MAS, WWDM and Parallel Optics may then
re-transmit 4-bytes of data either directly (WWDM or Parallel Optics) or recode
or multiplex the received 4-byte stream as a single serial stream (Serial or
MAS).
Byte-Striping over a parallel arrangement of serial lanes is analogous to the
simple and natural scaling of a single signal into parallel signals in order to
provide increased throughput (e.g. a parallel bus, two dimensional space, etc.).
I see no difficulties in the proposed definitions of 'code-blocks' for 10 GbE
employing Byte-Striping. These definitions effectively illustrate the
straightforward progression of data through the 10 GbE Data Link and Physical
layers. Please see:
http://grouper.ieee.org/groups/802/3/10G_study/public/nov99/taborek_1_1199.pdf,
page 16
Any protocol is compatible with Byte-Striping. Byte-Striping is not limited to
supporting only those protocols defined as 4-byte structures easily.
Word-Striping, on the contrary, requires that the MAC, PHY, and PMD or
combinations including all of these elements buffer data for transport in fixed
granularities of 4-bytes, synchronously, in each lane.
Byte-Striping seems to have a significant advantage over Word-Striping on this
point.
> 2) It has a much wider native skew tolerance making special deskew
> circuits superfluous and saving significant high-speed (baud and fractional
> baud intervals) circuit complexity and associated power consumption.
I agree that Word-Striping has the native skew tolerance to accommodate 2-bytes
of lane skew. However, lane deskew, much like parallel bus deskew is a process
that does not require a fixed blocking of data. Lane deskew is a process which
may be executed at initialization time based on a pattern as simple as the
Ethernet Idle pattern 'KR'. The latter lane deskew can also accommodate 2-bytes
of lane skew and does not requires any specific fixed blocking of data on each
serial lane.
Deskew can only be performed on easily recognizable boundaries such as commas.
This is the case for both Byte and Word-Striping. For both cases deskew is only
performed during the IPG. For 10 GbE, The KR Idle pattern results in a 40-bit
pattern with the same natural skew tolerance afforded by Word-Striping with no
negative effects.
I'm tired of arguing circuit complexity and power consumption. This will have to
be proven in implementations and will not be proven until a chip is powered-on.
I wouldn't consciously push a proposal that is more complex and higher power
(that's why I'm not pushing Word-Striping).
No Word-Striping advantage is evident on this point.
> 3) Much simpler, less state-ful logic, much cleaner interfaces, and
> more flexibility in mapping link protocols onto the link coding, fewer
> special cases of the "can remove X bytes if and only if they a re preceded
> Y bytes and followed by Z bytes" variety.
I assume that your talking about Insert/Remove protocol for clock tolerance
compensation. 10 GbE has no such requirements. Historically, Ethernet has
performed this function at the decoded data level. I expect the same for 10 GbE.
Word-Striping requires a complex, logic intensive and protocol dependent
Insert/Remove protocol that I've fully described in my original note, criteria
#8) PMD Clock Tolerance Compensation. I will be shortly be proposing a
Byte-Striped 10 Gigabit Fibre Channel mapping which will eliminate this current
FC protocol dependency and the requirement to Insert/Remove hundreds of 4-byte
Ordered-Sets as part of Insert/Remove protocol. This proposal will bring 10 GFC
in line with InfiniBand and 10 GbE proposals in support of a truly protocol
independent Insert/Remove protocol and enable the development of protocol
independent PMDs.
Byte-Striping seems to have a significant advantage over Word-Striping on this
point.
> 4) Idle or skip insertion is done independent of the coding, and
> independent of the number of lanes ( An idle is a 4-byte word inserted into
> the stream of 4-byte words whenever necessary -- no variation/dependency in
> format vs. number of lines in the interface). These operations are done in
> the stream of deserialized, word boundary aligned words, in coded or
> uncoded form.
Idle Insert/Remove in 10 GbE is performed above the MAC in granularities of one
byte during the IPG. The IPG is uncoded and, as such is independent of coding
and number of lanes.
Skip is only pertinent at the encoded data level (by the PHY) since Idles are
encoded as KR and and 'R' column is called a 'skip'. Skip Insertion is automatic
and Skip Removal simply deletes an 'R' column when necessary. Skip processing is
independent of the number of lanes.
No Word-Striping advantage is evident on this point.
> 5) Frame format and coding (except for disparity control) independent
> of the number of lanes.
Byte-Striping is independent of Frame Format, coding (including disparity
control which never crosses lane boundaries) and independent of the number of
lanes.
No Word-Striping advantage is evident on this point.
> 6) Word sync between every packet with minimum Ethernet IPG.
Word(per lane) sync is not a GbE requirement nor is it a 10 GbE objective.
Byte-Striping supports link synchronization on each lane given a minimum
Ethernet IPG.
No Word-Striping advantage is evident on this point.
> 7) Much greater flexibility in control sequences -- don't run out of
> control characters, since link control is done with control words instead
> of a single K character.
Control sequences are not limited by Byte-Striping.
However, control sequences beyond the necessary delimiters have historically
been regarded as an undesirable complexity by the 802.3 committee.
No Word-Striping advantage is evident on this point.
> 8) Almost all the logic is done on a per-word basis, rather than a
> per-byte basis. -- Simpler logic clocking.
Almost only counts in horseshoes and hand grenades :-)
This is probably the biggest fallacy surrounding Byte-Striping, and it's my
fault. I have to apologize for this. I now realize that I should have called
Byte-Striping 'Column-Striping' in the first place.
ALL Byte-Striped data is presented in full columns corresponding to 4-byte words
at exactly the same processing rate as for Word-Striping, but in a much more
straightforward fashion. I've addressed this issue in my previous note, criteria
#5) Data Processing Rate.
No Word-Striping advantage is evident on this point.
> 9) It's very intuitive, and easier to describe, construct, and
> simulate than the per-byte interfaces.
I don't think so. Just compare the following two presentations:
http://grouper.ieee.org/groups/802/3/10G_study/public/nov99/ritter_1_1199.pdf
http://grouper.ieee.org/groups/802/3/10G_study/public/nov99/taborek_1_1199.pdf
No Word-Striping advantage is evident on this point.
> 10) Doing line de-crossing (i.e. to get lines 0-3 to connect to lines
> 3-0 in mirrored order, with the de-crossing done inside the chips) is
> trivially easy, adds 0 ns of extra latency, and adds a very few extra
> gates. Line de-crossing on the chip can save significant area on the card
> in complex systems.
This issue is orthogonal to striping granularity and does not belong in this
list.
I believe I have referred to this issue as lane misordering during Hari
meetings. It is a useful function for chip-to-chip interconnects and is likely
to be supported by InfiniBand. I see little value in only resolving mirrored
misordering since intentional misordering can significantly ease trace routing
and mirrored misordering is a special misordering case which may not alleviate
the routing for a specific Hari interface.
> Albert Widmer Phone: 914 945-2047 Email:
> widmer@xxxxxxxxxx
>
> IBM T.J. Watson Research Center
>
> Yorktown Heights, NY 10598-0218
--
Best Regards,
Rich
-------------------------------------------------------------
Richard Taborek Sr. Tel: 408-330-0488 or 408-370-9233
Chief Technology Officer Cell: 408-832-3957
nSerial Corporation Email: rtaborek@xxxxxxxxxxxxx
2500-5 Augustine Dr. Alt email: rtaborek@xxxxxxxxxx
Santa Clara, CA 95054