RE: Does Ten-Gigabit Ethernet need fault tolerance?
Joe,
A question and a suggestion:
1. Are you suggesting that Fault Tolerence is a requirement for 10 Gig
Ethernet or for all Ethernet. Or, if FT is added to 10Gig Ethernet, is it of
any particular value if 10, 100, and 1000 BASE-* don't have it?
A1: If all Ethernet: you should ask for a call for interest in 802.3 and
bring presentations supporting the requirement (5 criteria, etc).
A2: If only 10 Gig Ethernet: you should bring a presentation supporting the
requirment to the next 802.3 HSSG meeting. Expect questions about how this
supports the 5 criteria. Expect questions about why only 10 Gig Ethernet.
jonathan
> -----Original Message-----
> From: gwinn@xxxxxxxxxx [mailto:gwinn@xxxxxxxxxx]
> Sent: Friday, July 16, 1999 2:15 PM
> To: stds-802-3-hssg@xxxxxxxx
> Subject: Does Ten-Gigabit Ethernet need fault tolerance?
>
>
>
> The purpose of this note is to present a case for inclusion of fault
> tolerance in 10GbE, and to offer a suitable proven technology for
> consideration. However, no salesman will call.
>
> Why Fault Tolerance? Ten-Gigabit Ethernet is going to be a relatively
> expensive, high-performance technology intended for major backbones,
> perhaps even nibbling at the bottom end of the wide-area network (WAN)
> market. In such applications, high availability is very much
> desired; loss
> of such a backbone or WAN is much too disruptive (and
> therefore expensive)
> to be much tolerated, and this kind of a market will gladly pay a
> reasonable premium to achieve the needed fault tolerance.
>
> Why add Fault Tolerance now? Because it's easiest (and thus
> cheapest) if
> done from the start, and because having FT built in and
> therefore becoming
> ubiquitous will be a competitive discriminator, neutralizing
> one of the
> remaining claimed advantages of ATM.
>
> Isn't Fault Tolerance difficult? In hub-and-spoke (logical
> star, physical
> loop) topologies such as GbE and10GbE, it's not hard to
> achieve both fault
> tolerance (FT) and military-level damage tolerance (DT). In
> networks of
> unrestricted topology, it's a lot harder. The presence of
> bridges does not
> affect this conclusion.
>
> How do I know that FT is so easily achieved? Because it's
> already been
> done, may be bought commercially, and is in use on one
> military system and
> is proposed for others. The FT/DT technology mentioned here
> was developed
> on a US Navy project, and is publically available without intellectual
> property restrictions. Why was the technology made public?
> To encourage
> its adopotion and use in COTS products, so that defense
> contractors can buy
> FT/DT lans from catalogs, rather than having to develop them again and
> again, at great risk and expense.
>
> What is the difference between Fault Tolerance and Damage
> Tolerance? In
> fault tolerance, faults are rare and do not correlate in
> either time or
> place. The classic example is the random failure of hardware
> components.
> (Small acts of damage, such as somebody tripping over a wire
> or breaking a
> connector somewhere, are treated as faults here because they
> are also rare
> and uncorrelated.) In damage tolerance, the individual faults
> are sharply
> correlated in time and place, and are often massive in
> number. The classic
> military example is a weapon strike. In the commercial world,
> a major power
> failure is a good example. Damage tolerance is considered
> much harder to
> accomplish than fault tolerance. If you have damage
> tolerance, you also
> have fault tolerance, but fault tolerance does not by itself
> confer damage
> tolerance.
>
> How is this Damage Tolerance achieved? All changes in LAN
> segment topology
> (the loss or gain of nodes (NICs), hubs, or fibers) are
> detected in MAC
> hardware by the many link receivers, which report both loss
> and acquisition
> of modulated light. This surveillance occurs all the time on
> all links, and
> is independent of data traffic. Any change in topology provokes the
> hardware into "rostering mode", the automatic exploration of
> the segment
> using a flood of special "roster" packets to find the best path, where
> "best" is defined as that path which includes the maximum
> number of nodes
> (NICs).
>
> Just how fault tolerant and damage tolerant is this scheme?
> A segment will
> work properly with any number of nodes and hubs, if sufficient fibers
> survive to connect them together, and will automatically
> configure itself
> into a working segment within a millisecond of the last fault. If the
> number of broken fibers is less than the number of hubs, all surviving
> nodes will remain accessible, regardless of the fault pattern. If the
> number of fiber breaks is equal to or greater than the number of hubs,
> there is a simple equation to predict the probability of loss
> of access to
> a typical node due to loss of hubs and/or fibers, given only
> the number of
> hubs and the probability of any fiber breaking: Pnd[p,r]=
> ((2p)(1-p))^r,
> where p is the probability of fiber breakage and r is the number of
> surviving hubs (which ranges from zero to four in a quad system). This
> equation is exact (to within 1%) for fiber breakage
> probabilities of 33% or
> less, and applies for any number of hubs.
>
> The simplicity of this equation is a consequence of the
> simplicity of this
> protocol, which is currently implemented in standard-issue FPGAs (not
> ASICs), and works without software intervention. It can also be
> implemented in firmware.
>
> To give a numerical example, in a 33-node 4-hub segment, loss
> of 42 fibers
> (16% of the segment's 264 fibers) would lead to only 0.5% of the nodes
> becoming inaccessible, on average. Said another way, after 42
> fiber breaks,
> there are only five chances out of a thousand that a node will become
> inaccessible. This is very heavy damage, with one fiber in
> six broken. To
> take a more likely example, with three broken fibers, all
> nodes will be
> accessible, and with four broken fibers, there is less than
> one chance in a
> million that a node will become inaccessible. Recovery takes
> two ring tour
> times plus settling time (electrical plus mechanical),
> typically less than
> one millisecond in ship-size networks, measured from the last fault.
> Chattering and/or intermittent faults can be handled by a number of
> mechanisms, including delaying node entry by up to one
> second. Few current
> LAN technologies approach this degree of resilience, or speed
> of recovery.
>
> In commercial systems and some military systems, a dual-ring
> solution is
> sufficient. Up to quad-ring solutions are comercially
> available, needed
> for some military systems. However, the ability to support up to quad
> redundant systems should be provided in 10GbE, for two
> reasons. First,
> quad is needed for the military market, which may be economically
> significant in the early years of 10GbE. Second, quad
> provides a clear
> growth path and a way to reassure non-military customers that
> their most
> stringent problems can be solved: One can ask them if their
> needs really
> exceed those of warships duelling with supersonic missiles.
>
> The basic technical document, the RTFC Principles of
> Operation, is on the
> GbE website as "http://grouper.ieee.org/ groups/802/3/
> 10G_study/public/
> email_attach/ gwinn_1_0699.pdf" and "http://grouper.ieee.org/
> groups/802/3/10G_study/ public/ email_attach/
> gwinn_2_0699.pdf". I was a
> member of the team that developed the technology, and am the author of
> these documents.
>
> Although these documents assume RTFC, a form of distributed
> shared memory,
> the basic rostering technology can easily be adapted for Gigabit and
> Ten-Gigabit Ethernet as well. For nontechnical reasons, RTFC
> originally
> favored smart nodes connected via dumb hubs. However, the
> overall design
> can be somewhat simplified if one goes the other way, to dumb
> nodes and
> smart hubs. This also allows the same dumb nodes to be used
> in both non-FT
> and FT networks, increasing node production volume, and does not force
> users to throw nodes away to upgrade to FT.
>
> I therefore would submit that 10GbE would greatly benefit from fault
> tolerance, and also that it's very easily achieved if included in the
> original design of 10GbE.
>
> Joe Gwinn
>
>