Thread Links Date Links
Thread Prev Thread Next Thread Index Date Prev Date Next Date Index

Re: A telephony carrier industry perspective


Khaled,

I have not worked on "congested" networks for a while.  I have been
designing and implimenting data networks based on application bandwidth
subscription defintions.  I do not have any specific data from the
"congested" networks that I have inheireted from others.  

The cyclic nature of "congestion" can be directly observed on a LAN that
does not have proper broadcast domain segregation.  It is a combination
of applicatin derivative processes, frame sizes, numbers of systems
attempting to "talk", and the delays in the network.  I have not seen
any mathmatical models that have properly represented that cyclic
process.  

				Thank you,
					Roy Bynum


Khaled Amer wrote:
> 
> Roy,
> I agree with your assessment and concerns.
> 
> As the ex-chairman of the 'killed' QoS/flow control study group, I've been
> having similar thoughts, and have been looking into investigating these
> issues.
> 
> > I agree with your assessment of the "best effort" IP based Internet.
> > Unreliable WAN transport system protocols go all the way back to the
> > old BiSync days. What happens when the WAN transport becomes reliable?
> > Do you still need the unreliable, best effort protocols? If you had
> > a reliable WAN transport at the same cost of an unreliable one, which
> > one would you pick?
> 
> Agree ... well put!
> 
> > The TCP flow control at the end systems only works relative to
> > retransmissions, which tend to make the congestion worse for longer
> > period of time. What happens to mission critical, time critical
> > applications in that environment?
> 
> Agree again ....
> 
> > Have you ever calculated the amount of data that gets dropped per
> > second at a 10Gb rate. Can you imagine the amount of money that a
> > carrier would loose because they lost a massive amount of their
> > customers' data traffic?
> 
> Even though the point seems to be obvious, do you have any data or figures
> to support this?
> 
> > I think that you would be surprised at how fast the "congestion"
> > resolves itself in an L2 environment.
> 
> Again ... do you have any data (measurements or models) to show this? I'm
> looking for such data, and doing some analysis myself.
> 
> >From my point of view, and based on the analysis that I've been doing, as
> well as research I've been reading, it seems to me that flow control at L2
> is needed in general, and becomes more and more important as we scale up LAN
> speeds. I also believe that it should be done in a fashion that
> differentiates between different types of traffic.
> 
> Khaled Amer
> 
> ----- Original Message -----
> From: Roy Bynum <RBYNUM/0004245935@xxxxxxxxxxx>
> To: Andrew Smith <andrew@xxxxxxxxxxxxxxxxxxx>
> Cc: IEEE HSSG <stds-802-3-hssg@xxxxxxxx>
> Sent: Wednesday, May 19, 1999 10:13 PM
> Subject: RE: A telephony carrier industry perspective
> 
> >
> > Andrew,
> >
> > Cascading of congestion flow-control will that will cause the
> > degradation of the whole enterprise network because of one circuit,
> > will only happen on a network that only has one primary circuit. I
> > think that you would be surprised at how fast the "congestion"
> > resolves itself in an L2 environment.
> >
> > I agree with your assessment of the "best effort" IP based Internet.
> > Unreliable WAN transport system protocols go all the way back to the
> > old BiSync days. What happens when the WAN transport becomes reliable?
> > Do you still need the unreliable, best effort protocols? If you had
> > a reliable WAN transport at the same cost of an unreliable one, which
> > one would you pick?
> >
> > The TCP flow control at the end systems only works relative to
> > retransmissions, which tend to make the congestion worse for longer
> > period of time. What happens to mission critical, time critical
> > applications in that environment? You don't put those kind of
> > applications on an "Internet". This is the reason that carriers like
> > MCI WorldCom have companies that out source their data networks to the
> > carriers, which in turn provides major revenue. This is the reason
> > that data services such as Frame Relay make so much money for
> > carriers.
> >
> > From a carrier viewpoint, unreliable is unsupportable. I have been
> > told by MCI WorldCom's operations and management group, that an
> > unreliable 10 GbE will not be deployed as a service in our network. I
> > don't know about other carriers, we will stay with reliable SONET
> > transport. We can make service level agreements about the reliability
> > of the transport on a SONET system. We can not make service level
> > agreements on an unreliable 10 GbE. We can do maintenance on SONET
> > transport systems without having a major effect on our customers'
> > traffic. We can not do that on an unreliable 10 GbE.
> >
> > Have you ever calculated the amount of data that gets dropped per
> > second at a 10Gb rate. Can you imagine the amount of money that a
> > carrier would loose because they lost a massive amount of their
> > customers' data traffic?
> >
> >                                    Thank you,
> >                                    Roy Bynum
> >
> >
> >
> >
> >
> > Date:     Tue May 18, 1999 12:24 pm  CST
> > Source-Date: Tue, 18 May 1999 11:23:57 -0700
> > From:     Andrew Smith
> >           EMS: INTERNET / MCI ID: 376-5414
> >           MBX: andrew@xxxxxxxxxxxxxxxxxxx
> >
> > TO:     * ROY BYNUM / MCI ID: 424-5935
> > CC:       IEEE HSSG
> >           EMS: INTERNET / MCI ID: 376-5414
> >           MBX: stds-802-3-hssg@xxxxxxxx
> > Subject:  RE: A telephony carrier industry perspective
> > Message-Id: 99051818240626/INTERNETGWDN3IG
> > Source-Msg-Id:
> <D0805D3B448BD211A7990008C7B18130142D09@xxxxxxxxxxxxxxxxxxxxxxx>
> > U-Content-return: allowed
> > U-X-Mailer: Internet Mail Service (5.5.2448.0)
> >
> >
> > Roy,
> >
> > This is very similar territory to that explored recently in the IEEE802
> > QoS/Flow-control study group (see
> > http://grouper.ieee.org/groups/802/QOSFC/index.html).
> >
> > Congestion is usually separated into 2 types: short-term, due to
> unforeseen
> > traffic peaks, and long-term, due to sustained levels of traffic in excess
> > of provisioned capacity. I think you are merging the two types in your
> > message. The latter type of congestion is what is usually known as
> > "over-subscription" and requires what you term "subscription control".
> >
> > Queue buffering can be used to handle short-term congestion but is not a
> > solution for the second type. It can be used to absorb peaks that occur
> > naturally due to well-behaved data flows traversing switching nodes but
> > should not be thought of as a good mechanism for handling uncontrolled
> > bursty sources. In your scenario, with persistent overload and using
> > per-link flow control, "flow-control will continue to cascade back" until
> > the whole network becomes equivalent to a shared network segment with an
> > *aggregate* bandwidth equivalent to that of the congested link: this is
> > likely not going to be popular with your customers.
> >
> > In the presence of data sources that react to congestion feedback (through
> > packet dropping or ECN), the offered load can be throttled at the sources
> to
> > match the provisioned capacity: this is how the best-effort Internet works
> > today using TCP. There are well-understood packet-dropping mechanisms e.g.
> > RED that are commonly implemented these days in routers in the Internet
> core
> > that lead to efficient and fair overall network utilisation for TCP
> sources:
> > in the Internet architecture, dropping data to signal congestion is
> > considered a Good Thing (tm). Uncontrolled (open-loop) data sources are a
> > potential problem in such a network and must be controlled by shaping at
> or
> > near the source and suitable provisioning if they produce enough traffic
> to
> > cause problems.
> >
> > At the IP layer, mechanisms are being standardised (see
> > http://www.ietf.org/html.charters/diffserv-charter.html and RFC 2475 in
> > particular) that are widely deployed today in routers for providing
> > differential services for different traffic classes on a link and within
> > switching nodes: this model assumes the presence of traffic shapers at
> > inputs that enforce a per-class contract ("SLA") with the data source that
> > can take account of the burstiness of the sources and allows a provider to
> > provision appropriately for such pre-agreed peaks without reliance on
> > flow-control - in fact, per-link flow control would make it difficult for
> a
> > provider to meet such SLAs. The SLAs provide the financial controls to
> make
> > this model work.
> >
> > It is unclear today (to me at least) what added benefit link-layer flow
> > control can offer on top of TCP in such networks: any economically viable
> > form of flow control is likely to offer, at best, per-class control,
> rather
> > than the more desirable per-microflow control. Per-class control penalises
> > all microflows that share the class, even those that are well behaved.
> > Per-microflow control is expensive and would, in any case, duplicate in
> many
> > ways functions that are already provided at the transport layer. Certainly
> > there is scope for some mathematical modelling of such scenarios and there
> > are also demonstrable scenarios where such detailed flow control is
> > worthwhile, particularly those using low-speed, expensive transmission
> > lines: I'm not sure that 10G Ethernet falls into that category.
> >
> > Persistent over-subscription of WAN links for todays enterprise networks
> is
> > a solved problem: you either buy a larger SLA or you use bandwidth
> > management solutions such as class-based queueing to ensure that the right
> > traffic gets through. I really don't hear a lot of people asking for
> > link-level flow control to solve this problem.
> >
> > But we digress ...
> >
> > Andrew
> >
> > ****************************************************************
> > Andrew Smith                              tel: +1 (408) 579-2821
> > Extreme Networks                          fax: +1 (408) 579-3000
> > 3585 Monroe St.                   http://www.extremenetworks.com
> > Santa Clara CA 95051-1450        em:  andrew@xxxxxxxxxxxxxxxxxxx
> > ****************************************************************
> >
> >
> > > -----Original Message-----
> > > From: Roy Bynum [mailto:RBYNUM/0004245935@xxxxxxxxxxx]
> > > Sent: Tuesday, May 18, 1999 7:46 AM
> > > To: Andrew Smith
> > > Cc: IEEE HSSG
> > > Subject: RE: A telephony carrier industry perspective
> > >
> > >
> > > Andrew,
> > >
> > > I have been designing and implementing all sorts of data networking
> > > environments, LAN/MAN/WAN, for some time. One of the headaches is
> > > understanding the business application requirements and cost trade-off
> > > on link circuit bandwidth. Because intermediate/gateway systems
> > > historically do not have the ability to signal back when an output
> > > circuit becomes overloaded (the new term is "over-subscribed"), queue
> > > buffering has been a major issue. The problem is a network that has
> > > circuits that consistently become overloaded and do so for extended
> > > lengths of time, such that the queues become overloaded. Data network
> > > architects tend to design large enterprise WAN networks based on a
> > > consistent loading with an attempt to limit the peaks to close to the
> > > maximum circuit bandwidth.
> > >
> > > With "flow-control" using GbE I can design WAN data networks closer to
> > > the consistent loading with less concern for peak traffic. I have set
> > > up a test WAN network using Native data GbE over Optical DWDM and
> > > SONET DATS to test this. When the L2/L3 GbE switches are properly
> > > configured, I can only attempt to overload the network "circuits".
> > > When the flow-control is applied all the way to the data source
> > > systems, the flow-control will continue to cascade back such that
> > > over-subscription does not occur. Unlike a traditional TDM WAN circuit
> > > based design, no data is lost during peak "overloading", which tends
> > > to prevent more problems because of re-transmissions. As this is
> > > better understood and implemented, it will change the bandwidth
> > > allocation process of designing enterprise networks.
> > >
> > >                                    Thank you,
> > >                                    Roy Bynum
> > >
> > >
> > >
> > > Date:     Mon May 17, 1999  5:27 pm  CST
> > > Source-Date: Mon, 17 May 1999 16:26:58 -0700
> > > From:     Andrew Smith
> > >           EMS: INTERNET / MCI ID: 376-5414
> > >           MBX: andrew@xxxxxxxxxxxxxxxxxxx
> > >
> > > TO:     * ROY BYNUM / MCI ID: 424-5935
> > > CC:       IEEE HSSG
> > >           EMS: INTERNET / MCI ID: 376-5414
> > >           MBX: stds-802-3-hssg@xxxxxxxx
> > > Subject:  RE: A telephony carrier industry perspective
> > > Message-Id: 99051723270571/INTERNETGWDN1IG
> > > Source-Msg-Id:
> > > <D0805D3B448BD211A7990008C7B18130142CFC@xxxxxxxxxxxxxxxxxxxxxxx>
> > > U-Content-return: allowed
> > > U-X-Mailer: Internet Mail Service (5.5.2448.0)
> > >
> > > Roy,
> > >
> > > At the risk of waking some sleeping dogs, could I ask you to
> > > elaborate a
> > > little on your statement:
> > >
> > > 'In addition to being less expensive, GbE over Native Data
> > > SONET DATS will
> > > provide "subscription control" though "flow control" That
> > > "subscription
> > > control" will prevent over-subscribing the WAN links, which
> > > is a big problem
> > > for enterprise data network designers and architects.'
> > >
> > > Andrew
> > >
> > > ****************************************************************
> > > Andrew Smith                              tel: +1 (408) 579-2821
> > > Extreme Networks                          fax: +1 (408) 579-3000
> > > 3585 Monroe St.                   http://www.extremenetworks.com
> > > Santa Clara CA 95051-1450        em:  andrew@xxxxxxxxxxxxxxxxxxx
> > > ****************************************************************
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Roy Bynum [mailto:RBYNUM/0004245935@xxxxxxxxxxx]
> > > > Sent: Monday, May 17, 1999 4:12 PM
> > > > To: bill.st.arnaud
> > > > Cc: IEEE HSSG
> > > > Subject: RE: A telephony carrier industry perspective
> > > >
> > > >
> > > >
> > > > Bill,
> > > >
> > > > Internet IP will continue to be what the market
> > > requirements reflect.
> > > > Slow restoration times are acceptable in that environment. The UUNet
> > > > Long Haul GbE is part of what I am doing. I am the one that put the
> > > > Long Haul Optical switching/Metro DWDM/SONET DATS evaluation of GbE
> > > > together.
> > > >
> > > > As a bit irony, dependable GbE is turning out to be less expensive
> > > > than undependable IP over TDM, ATM, or POS. Unless MPLS comes in a
> > > > respectable price break, GbE over Native Data SONET DATS
> > > will still be
> > > > less expensive.
> > > >
> > > > In addition to being less expensive, GbE over Native Data SONET DATS
> > > > will provide "subscription control" though "flow control" That
> > > > "subscription control" will prevent over-subscribing the WAN links,
> > > > which is a big problem for enterprise data network designers and
> > > > architects.  Combine that with "priority queueing" and most of what
> > > > MPLS was supposed to do has already been accomplished by GbE.  You
> > > > guys as IEEE have done a greater job than you knew.
> > > >
> > > > Dependable, high quality transport of Native data traffic
> > > such as GbE
> > > > and 10GbE is probably going to be a different market, one that "best
> > > > effort" is unacceptable to. If there is a market that provides the
> > > > profit margin that will sustain 10GbE, it will not be the
> > > Internet as
> > > > it is today.
> > > >
> > > > I do know that I have been given the requiement that
> > > carriers can not
> > > > support a data service over long haul systems that does not provide
> > > > "SONET like" functionality. The reason that I joined this
> > > study group
> > > > is to provide that insight to the standards developers. If
> > > that is GbE
> > > > or 10GbE over SONET then the issue is already resolved. All that
> > > > remains is to determine what the LAN application requirements are,
> > > > then the standard can be defined.
> > > >
> > > >
> > > > Date:     Mon May 17, 1999  3:23 pm  CST
> > > > Source-Date: Mon, 17 May 1999 17:17:51 -0400
> > > > Fromm:     bill.st.arnaud
> > > >           EMS: INTERNET / MCI ID: 376-5414
> > > >           MBX: bill.st.arnaud@xxxxxxxxxx
> > > >
> > > > TO:     * ROY BYNUM / MCI ID: 424-5935
> > > > Subject:  RE: A telephony carrier industry perspective
> > > > Message-Id: 99051721234042/INTERNETGWDN1IG
> > > > Source-Msg-Id:
> > > > <NBBBJIMEPHPGCNGAHPMFIEIJELAA.bill.st.arnaud@xxxxxxxxxx>
> > > > U-Importance: Normal
> > > > U-X-Mailer: Microsoft Outlook IMO, Build 9.0.2212 (4.71.2419.0)
> > > > U-X-MIMEOLE: Produced By Microsoft MimeOLE V5.00.2314.1300
> > > > U-X-MSMail-priority: Normal
> > > > U-X-Priority: 3 (Normal)
> > > >
> > > > Roy:
> > > >
> > > > Again no disagreement.  I don't think traditional SONET or
> > > > ATM networks will
> > > > disappear. The model we advocate is the same that Frontier is
> > > > now deploying:
> > > > IP/DWDM for best efforts, slow restoral traffic on one set of
> > > > wavelengths,
> > > > IP over SONET on another set of wavelengths for those
> > > > services that need
> > > > fast restoral and security of SONET, and IP over ATM over
> > > > SONET on another
> > > > set of wavelengths for fine grained QoS services.
> > > >
> > > > I agree with you that the driving force for GbE is cost.  It makes a
> > > > dramatic difference to the overall cost of the network.
> > > >
> > > > But I believe GbE can also make an equal dramatic difference on the
> > > > transport side on medium, long haul links up to 1000 km.
> > > Your sister
> > > > company UUNet has already demonstrated that on some long haul
> > > > GbE systems.
> > > > But I agree with you this type of link is probably only
> > > good for best
> > > > efforts IP traffic.
> > > >
> > > > Bill
> > > >
> > > > -------------------------------------------
> > > > Bill St Arnaud
> > > > Director Network Projects
> > > > CANARIE
> > > > bill.st.arnaud@xxxxxxxxxx
> > > > http://tweetie.canarie.ca/~bstarn
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Roy Bynum [mailto:RBYNUM/0004245935@xxxxxxxxxxx]
> > > > > Sent: Monday, May 17, 1999 4:54 PM
> > > > > To: bill.st.arnaud
> > > > > Cc: IEEE HSSG
> > > > > Subject: RE: A telephony carrier industry perspective
> > > > >
> > > > >
> > > > > Bill,
> > > > >
> > > > > I have an IP base video conferencing demonstration
> > > > application that is
> > > > > full motion, full resolution. It uses MPEG2 compression and
> > > > drives the
> > > > > IP data at 7Mbs bidirectional. Part of the demonstration of the
> > > > > reliability of GbE over optical and SONET Native Data
> > > > systems is to do
> > > > > a simulated fiber break. Over a normal IP network, there
> > > is a major
> > > > > loss of data and thus video synchronization, sometimes
> > > the "call" is
> > > > > even dropped. Over a Native data network, optical or
> > > SONET DATS, if
> > > > > you blink, you miss the cut. It is that kind of traffic path
> > > > > restoration quality that will be required by future, real time,
> > > > > visual, and virtual applications. This is the quality of
> > > > traffic path
> > > > > restoration that needs to be implemented in Metro/MAN and WAN
> > > > > environments. I do not believe that any one protocol and/or
> > > > system can
> > > > > do that and still be cost effective.
> > > > >
> > > > > Fromm looking at Cisco's DPT description, it looks a lot like a
> > > > > SONET/SDH BLSR ring. In duplicating the alternate path coupled
> > > > > architecture of SONET/SDH, Cisco could well duplicate the
> > > > restoration
> > > > > functionality of SONET/SDH. Although what value it will
> > > > have over long
> > > > > haul systems that are geared for very high bandwidth, I
> > > am not sure.
> > > > > Within a LAN environment, that type of restoration is probably not
> > > > > required. Within a Metro MAN environment, we have found that GbE
> > > > > combined with optical path protected DWDM is very cost effective.
> > > > > Cisco will have to come in at a VERY "cheep" cost in order
> > > > to justify
> > > > > the deployment of their systems.  I expect to have some
> > > test systems
> > > > > before too long.  I am looking forward to finding out.
> > > > >
> > > > > The bottom line to this is cost. Preliminary evaluations
> > > are showing
> > > > > that GbE already is so cost effective that it is less expensive to
> > > > > have 10 GbE interfaces on a router than it is to have 4 OC48 POS
> > > > > interfaces. Combine that with the less expensive interfaces on the
> > > > > DWDM and SONET DATS equipment it becomes even more attractive. The
> > > > > capital cost of IP over "Ethernet" or what I am now calling
> > > > Native IP
> > > > > is less expensive over a MAN or WAN environment than the
> > > > existing TDM
> > > > > WAN systems. 10GbE must compete in this environment. In
> > > order for it
> > > > > to be deployed, it must provide high native data bandwidth,
> > > > very cost
> > > > > effectively with the specific service, maintenance, and operations
> > > > > support that each very different environment requires.
> > > > >
> > > > >
> > > > >                               Thank you,
> > > > >                               Roy Bynum
> > > > >
> > > > >
> > <much good stuff chopped from below here>