Re: [8023-CMSG] Proposed Upper Layer Compatibility Objective
Hugh,
Your second paragraph describes what I think a large number of us are
talking about. The edge devices (i.e. adapters) and switch fabrics are
where some of these issues are presenting themselves. The only
assistance 802.3 provides is PAUSE or nothing. As you mentioned, PAUSE
creates false congestion and can move the congestion point. Stopping
traffic for output queue X would reduce the congestion in that queue,
while preventing dropping packets destined for non-congested queues.
The term that some have been using for this is traffic prioritization.
Maybe the term is close to something used by people in their companies
or something used in 802.1, I don't know. Would the term
"priority-based PAUSE" or "queue PAUSE" be better?
Thanks,
Brad
-----Original Message-----
From: owner-stds-802-3-cm@LISTSERV.IEEE.ORG
[mailto:owner-stds-802-3-cm@LISTSERV.IEEE.ORG] On Behalf Of Hugh Barrass
Sent: Wednesday, May 19, 2004 12:30 PM
To: STDS-802-3-CM@LISTSERV.IEEE.ORG
Subject: Re: [8023-CMSG] Proposed Upper Layer Compatibility Objective
Ben (also Brad),
Personally, I prefer that the MAC should not get more "intelligent" as a
general principle. Efforts to increase MAC intelligence in the past
(think, BLAM) have failed, whereas efforts to decrease intelligence
(think, full-duplex) have succeeded.
The difficulty with signaling any form of queue state from the receiver
to the transmitter is that the queue state could (or should) only
reflect input queue state whereas the useful information would (or
should) reflect output queue state. Therefore queue state signaling is
only really useful for (certain) edge devices or for devices with less
than line rate switch fabrics. If a device can drain its input queues at
line rate then congestion will only occur when a destination queue
fills. This would mean that the useful backpressure indication would say
"please reduce (or stop) traffic destined for my output queue X."
Traffic destined for other output queues should not be stopped. Pausing
traffic for all destinations would create false congestion in the paused
device (which may then start dropping frames destined for the
non-congested paths). Upper layer protocols have evolved assuming that
packets will be dropped at congestion points and any pause protocol
tends to move the congestion points artificially.
I won't say that I'm dead against making the MAC more "intelligent" but
I think that any such changes must be handled with extreme care as it
will create interactions between the new MAC intelligence and other
"intelligent" entities (including L2 & L3 QOS, network & traffic
management and analysis).
Hugh.
Benjamin Brown wrote:
> Gary,
>
> I understand that there are different types of traffic and that these
> different types require different network capabilities. Some desire
> loss-less traffic, others can tolerate losses but those packets that
> arrive need low latency. Perhaps I was being too general in my
> question/comment. I could substitute dropped frames for low
> latency. I could probably choose a better term than "highest
priority".
> Each DSCP or 802.1 priority could have different characteristics
> and I'll presume bridges know the difference.
>
> However, I think the point stands. DiffServ classifies data. 802.1
> provides a way of prioritizing these classifications into different
> queues with different queue-emptying algorithms and ways of
> favoring priorities. It would seem advantageous to be able to
> share buffer congestion information between bridges across
> links in order to enable a better picture of the state of the layer
> 2 network.
>
> While the round-trip latency of large, unconstrained networks may
> inhibit the feasibility of this, it seems quite likely that small,
well-
> constrained networks should be able to make good use of this
> capability.
>
> Regards.
> Ben
>
> McAlpine, Gary L wrote:
>
>> Also regarding the "Re:[8023-CMSG] Questions" response from Norm Finn
>> Sun 5/16/2004 9:11 AM.
>>
>> Ben,
>>
>> Good line of thought. I only see one issue with it (which I think may
be
>> a general problem with the assumptions about prioritization in layer
2).
>> If we consider a broad spectrum of traffic types and their service
>> requirements, I don't believe there is a direct correlation between
>> "discardability" and "priority". For instance, some traffic (storage
>> blocks as an example) might best (from a system perspective) be
>> transported at a lower priority, but with a minimized loss rate.
While
>> other traffic (voice as an example) might best be transported at a
high
>> priority to minimize latency and the latency variations, but with a
>> relaxed loss rate. I assume a DSCP contains the context to determine
>> "discardability" and "priority". Is there a and existing standard for
>> communcating "discardability" between layer 3 and layer 2?
>>
>> Gary
>>
>> -----Original Message-----
>> From: owner-stds-802-3-cm@LISTSERV.IEEE.ORG
>> [mailto:owner-stds-802-3-cm@LISTSERV.IEEE.ORG] On Behalf Of Benjamin
>> Brown
>> Sent: Monday, May 17, 2004 11:18 AM
>> To: STDS-802-3-CM@LISTSERV.IEEE.ORG
>> Subject: Re: [8023-CMSG] Proposed Upper Layer Compatibility Objective
>>
>>
>>
>> Jeff,
>>
>> Let me see if I understand what you're trying to say. Differentiated
>> Services (DiffServ - DS), as an example upper layer protocol (ULP),
>> applies DS code points (DSCPs) to its traffic. These DSCPs are a
>> tag that classify different types of traffic. Since ULPs see layer 2
as
>> a simple pipe, it neither expects nor desires any kind of support for
>> these various DSCPs. In other words, if congestion occurs and packets
>> must be dropped in the layer 2 "pipe" it would make no difference to
>> DS what kind of traffic was dropped. It could be done totally
randomly.
>>
>> However, should the layer 2 provide some level of support, it would
>> be advantageous to DS if there was some type of mapping between
>> DSCPs and layer 2 priority tags. These priority tags could then be
>> used to separate the traffic into priority queues. Now, when buffer
>> congestion occurs and packets must be dropped, the lower priority
>> packets can be dropped before the higher ones. As long as the
>> mapping is done right, the packets that DS thinks are the most
>> important are least likely to be dropped.
>>
>> Assuming I'm close on the above, why then is it so hard to take this
>> one more step? If Ethernet can make advances so that the highest
>> priority traffic is even less likely to drop packets in the face of
>> congestion,
>> further aiding DS (and other ULPs) to maintain the traffic flow of
the
>> traffic they think is the most important, why is that bad? Indeed,
why
>> isn't that justification for this project?
>>
>> The whole approach of creating priorities within 802.1 (P802.1Q)
>> and now assigning a mapping between DSCPs and 802.1 priorities
>> so that everybody does it the same (isn't this being done in
P802.1ad?)
>> is considered very worthwhile to most people. It seems to me that
>> it may be equally worthwhile to extend this further into layer 2 so
>> that a mechanism exists to communicate, not only within a bridge
>> but between bridges, when buffer congestion is occurring for these
>> different priorities so that steps can be taken to preserve the
highest
>> priority traffic.
>>
>> Thanks,
>> Ben
>>
>> Jeff Warren wrote:
>>
>> Hi Gary,
>>
>> I agree we can't ignore "How does what we do at L2 impact UPL". Yikes
-
>> there are many ULP's, not just DS.
>>
>> I agree that "We leave the discard decisions up to the UPL".
>>
>> I can't tell you exactly how CM will impact DS until I know what CM
>> does? I am consulting with one of the original DS authors, Steve
Blake,
>> he and I use to work together a few companies ago when we were both
at
>> IBM, this was when DS was standardized. We discussed this point this
>> morning and the general feeling was that a FDX point-to-point (PTP)
L2
>> Ethernet link should not be making implicit or explicit selective
packet
>> drop decisions, that this would in a negative way impact the
operation
>> of DS.
>>
>> If what CM does is to channelize traffic across a PTP link (high or
low
>> BW, short or long distance, inside a chassis across an Ethernet
>> backplane or externally across a physical copper or optical link)
into
>> 'N' number of Classes (CoS transmission buffers) then when one Class
is
>> "some how" determined to be using too much BW and that Class is
"Turned
>> OFF" for some period of time there is a high likelihood that packets
in
>> that Class will be dropped. When this happens you've just impacted DS
>> ability to do its drop probability calculations properly...........
more
>> to come on this as we better understand what CM does.
>>
>> NOTE: 802.1p has already defined a L2 "marker" that is used as a form
of
>> L2 rate control. 802.1p is used by intermediate L2 switches (between
>> router hops; DS Hops) to provide intermediate L2 class of service
>> prioritization. Most switch/router products that are worth purchasing
>> use this feature along with DSCP (L3) to prioritize traffic across
their
>> available ingress & egress ports.
>>
>>
>> Regards,
>>
>> - Jeff
>> ----- Original Message -----
>> From: McAlpine, Gary L
>> To: STDS-802-3-CM@LISTSERV.IEEE.ORG
>> Sent: Thursday, May 13, 2004 11:48 AM
>> Subject: Re: [8023-CMSG] Proposed Upper Layer Compatibility Objective
>>
>>
>> Jeff,
>>
>> I am perfectly fine with NOT having such an objective if you think we
>> can get away with it. I'm not sure we can ignore how, what we propose
to
>> do at L2, will affect the operation of the upper layers.
>>
>> On the subject of DiffServ, I would simply propose that, within the
>> limited scope of the high speed, short range, L2 interconnects we are
>> trying to enhance, we leave the discard decisions up to the upper
layers
>> while maintaining the desired latency and traffic differentiation
>> qualities within layer 2. We have shown through simulations that
layer 2
>> rate control mechanisms can eliminate (or significantly reduce frame)
>> discards within layer 2 (effectively pushing the discard decision up
to
>> L3). We have also shown that rate control combined with
prioritization
>> in layer 2 can maintain excellent latency and traffic differentiation
>> qualities.
>>
>> Maybe you can explain to us how this is likely to affect the
operation
>> of DiffServ. I haven't dug deep enough into DiffServ to know if it is
>> counting on L2 devices to discard. My intuition is telling me it is
>> likely to improve the operation of DiffServ, as well as other upper
>> layer protocols of interest.
>>
>> Gary
>>
>>
>>
>>
>> -----Original Message-----
>> From: owner-stds-802-3-cm@LISTSERV.IEEE.ORG
>> [mailto:owner-stds-802-3-cm@LISTSERV.IEEE.ORG] On Behalf Of Jeff
Warren
>> Sent: Wednesday, May 12, 2004 7:58 PM
>> To: STDS-802-3-CM@LISTSERV.IEEE.ORG
>> Subject: Re: [8023-CMSG] Proposed Upper Layer Compatibility Objective
>>
>>
>> Gary,
>>
>> You mentioned improved operation of DiffServ as a goal for CM. DS is
a
>> collection of a number of RFC's, here's the basic set of DS RFCs.
>> RFC2474 "Definition of the Differentiated Services Field (DS Field)
in
>> the IPv4 and IPv6 Headers"
>> RFC2475 "An Architecture for Differentiated Services"
>> RFC2597 "Assured Forwarding PHB"
>> RFC2598 "An Expedited Forwarding PHB"
>> What did you have in mind for supporting ULP's such as DS? For
example
>> this L3 protocol's purpose in life is to decide drop probabilities
for
>> individual packets on a hop-by-hop basis. For example assured
forwarding
>> has three levels of drop precedence (red, yellow, green). Then
there's
>> expedited forwarding in the highest priority queue, plus best effort,
>> etc......
>>
>> I've been wondering how an "Ethernet" standard is going to assist a
L3
>> protocol such as DS by classification MAC traffic? Would you propose
>> duplication of or cooperation with DS protocols? Maybe you let DS do
its
>> thing and present a CM enabled Ethernet egress port with remarked
>> packets that are fortunate enough to have passed across the devices
>> fabric w/o being dropped. This CM enabled Ethernet port would then
align
>> the offered MAC load to the queues it has available, it does this for
>> example by inspecting DSCP values and comparing these DSCP values to
a
>> predefined buffer table. Bla bla bla......
>>
>> The lights haven't gone off for me yet, I don't see the value in CM
>> supporting DS because switch manufactures have already figured this
one
>> out and implemented this concept of multiple egress queues working at
>> line rate. Plus these implementations require global knowledge of
>> networking policies to configure them properly, and more importantly
to
>> standardize them were talking about linking behavior of ingress and
>> egress ports, knowledge of system wide (e.g. switch or router) buffer
>> management capabilities, all very much vendor specific capabilities
that
>> would be very difficult to get everyone to agree to.
>>
>> Regards,
>>
>> - Jeff Warren
>>
>>
>>
>> ----- Original Message -----
>> From: McAlpine, Gary L
>> To: STDS-802-3-CM@LISTSERV.IEEE.ORG
>> Sent: Wednesday, May 12, 2004 5:44 PM
>> Subject: [8023-CMSG] Proposed Upper Layer Compatibility Objective
>>
>>
>> My A.R. from last meeting (thank you Ben).
>> Here's a first shot at a Upper Layer Compatibility objective:
>> The objective is:
>> "To define 802.3 congestion control support that, at a minimum, will
do
>> nothing to degrade the operation of existing upper layer protocols
and
>> flow/congestion control mechanisms, but has the explicit goal of
>> facilitating the improved operation of some existing and emerging
>> protocols, over 802.3 full-duplex link technology."
>>
>> If we can narrow the scope and still make it meaningful, I'm all for
it.
>>
>> I have attached RFC3168 (on ECN) as a reference. It contains a very
good
>> overview of congestion control at the TCP and IP layers. I would also
>> consider this and DiffServ as examples of existing ULPs we would want
to
>> do our part to improve the operation of. IMO, what we can do at the
>> 802.3 level to better support these will also provide the support we
>> need for improved operation of some emerging protocols such as iSCSI
and
>> RDMA.
>> Gary <<rfc3168.txt>>
>>
>>
>> --
>> -----------------------------------------
>> Benjamin Brown
>> 178 Bear Hill Road
>> Chichester, NH 03258
>> 603-491-0296 - Cell
>> 603-798-4115 - Office
>> benjamin-dot-brown-at-ieee-dot-org
>> (Will this cut down on my spam???)
>> -----------------------------------------
>>
>>
>>
>
> --
> -----------------------------------------
> Benjamin Brown
> 178 Bear Hill Road
> Chichester, NH 03258
> 603-491-0296 - Cell
> 603-798-4115 - Office
> benjamin-dot-brown-at-ieee-dot-org
> (Will this cut down on my spam???)
> -----------------------------------------
>