| Thread Links | Date Links | ||||
|---|---|---|---|---|---|
| Thread Prev | Thread Next | Thread Index | Date Prev | Date Next | Date Index |
I've been following the newsgroup recently and noticed discussion relative to jumbo frames in the context of frame preemption, some argued the inevitability of 64KB packets would necessitate preemption and others claimed that Jumbo frames do not make sense outside of the LAN. While not lending to the debate either way, I thought I would help lend some background on Jumbo Frames :
1. It is unlikely Ethernet Jumbo Frames extend much past 9180 bytes (I'm picking 9180 out of the air, it could be 9000 or 9500)
"Jumbo frames" extends ethernet to around 9180 bytes. Why 9180?
1. Because ethernet uses a 32 bit CRC that loses its effectiveness above about 12000 bytes.
2. 9180 bytes was large enough to carry an 8 KB application datagram (e.g. NFS) plus packet header overhead.
3. The MTU size for IP over ATM is 9180 bytes (which was adopted since the IP MTU for SMDS was defined as 9180)
At 64 KB we reach the limit of an IPv4 datagram, while IPv6 allows for packets up to 4 GB in size. For ethernet however, the 32 bit CRC limit is hard to change, so don't expect to see ethernet frame sizes much above 9180 bytes anytime soon.
2. Jumbo frames have a place in both the LAN and the WAN
Smaller frames usually mean more CPU interrupts and more processing overhead for a given data transfer size. Often the per-packet processing overhead sets the limit of TCP performance in the LAN environment. Alteon has an often cited study showing an example where jumbo frames provided 50% more throughput with 50% less CPU load than 1500 byte frames.
At first approximation, WAN Jumbo Frames make less sense: compare the efficiency of a 9,000-byte large-packet system with a standards-based 1,500-byte system. The standard packet gets 1,500 data bytes out of 1,538 bytes of frame and overhead, or 97.5% efficiency. The nonstandard packet gets 9,000 data bytes out of 9,038 bytes, or 99.6% efficiency. To put it another way, the difference in time required to send a 1M-byte file is only 0.1 msec.
This argument is shortsighted in that it doesn’t take into account the reality of the WAN links and TCP.
The performance of TCP over wide area networks has been extensively studied and modeled. One landmark paper by Matt Mathis explains how TCP throughput has an upper bound based on the following parameters:
Throughput <= ~0.9 * MSS / (rtt * sqrt(packet_loss))
So maximum TCP throughput is directly proportional to the Maximum Segment Size (MSS, which is MTU minus TCP/IP headers). All other things being equal, you can double your throughput by doubling the packet size! This relationship seems to have escaped most of the arguments surrounding jumbo frames. [Packet_loss may also increase with MSS size, but does so at a sub-linear rate, and in any case has an inverse square effect on throughput, i.e. MSS size still dominates throughput.]
In the local area network or campus environment, rtt and packet loss are both usually small enough that factors other than the above equation set your performance limit (e.g. raw available link bandwidths, packet forwarding speeds, host CPU limitations, etc.). In the WAN however, rtt and packet loss are often rather large and something that the end systems cannot control. Thus their only hope for improved performance in the wide area is to use larger packet sizes.
Let's take an example: Ottawa to Vancouver Round Trip Time (rtt) is about 40 msec, and let's say packet loss is 0.1% (0.001). With an MTU of 1500 bytes (MSS of 1460), TCP throughput will have an upper bound of about 8.3 Mbps! And no, that is not a window size limitation, but rather one based on TCP's ability to detect and recover from congestion (loss). With 9000 byte frames, TCP throughput could reach about 51 Mbps.
Or let's look at that example in terms of packet loss rates. Same round trip time, but let's say we want to achieve a throughput of 500 Mbps (half a "gigabit"). To do that with 9000 byte frames, we would need a packet loss rate of no more than 1x10-5 With 1500 byte frames, the required packet loss rate is down to 2.8x10-7! While the jumbo frame is only 6 times larger, it allows us the same throughput in the face of 36 times more packet loss.
Vish Nandlall
Nortel Networks
ESN 393-1350
External (613) 763-1350
vnandlal@xxxxxxxxxxxxxxxxxx