Kapil and Dan,
Absolute numbers would be great to see; however, we still need to understand what incremental differences create meaningful competitive advantage of one datacenter
providing services over another.
Jeff
From: Dan Dove [mailto:dan.dove@xxxxxxxxxxxxxxxxxx]
Sent: Thursday, July 16, 2015 12:29 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC
Hi Kapil,
Good addition to the discussion.
A break down of entire end-to-end latency is essential. This would include stack, PHY/PMD, cable, switch, cable, PHY/PMD, stack.
An analysis that shows how an increase in latency on the PHY/PMD impacts performance of useful applications in addition to benchmark tests would be helpful.
Regards,
On 7/15/15 8:55 PM, Kapil Shrikhande wrote:
Hi Mike,
Thanks for offering to do this. That will be really a useful thing.
Looking at switch data-sheets using previous generation chips, we know that switch latency has been in the 100s of nanoseconds range, and therefore the added
latency from FEC could be a good chunk of that. But what I want to understand better is – how does this FEC latency stack-up in the context of the end-end latency, e.g. latency of a packet from server A going to server B in a Data Center. And should we (or
shouldn’t we) also consider including the latency of the server (application) stack in this end-end latency. And can we estimate the impact of PMD latency to the latency of the “job” or Application, or is that too complex (I suspect)... This is not an easy
task; and is specific to Data center types, workloads, etc. but getting a few viewpoints on this topic will greatly help the discussion. So, thanks for doing this.
Kapil
Hi folks,
I had a hall chat with Mark today and agreed to bring in a presentation to an upcoming ad-hoc to lay out the points I made during comments after Joel’s presentation
on the value of a no-FEC solution. It sounds like there is value in explaining the impact of latency in the market spaces we compete in as well as the relative importance of the factors influenced by using FEC. I’m not sure I can get something ready for
next week but will aim for the week after.
Best regards,
Mike Andrewartha
Well said !
I can only add that I love my Mum, and apple pie is nice.
J
From: Brad Booth [mailto:bbooth@xxxxxxxx]
Sent: Wednesday, July 15, 2015 1:00 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC
No disagreement on doing things for the right reasons.
The issue is, which reason is the right one? :-) For some applications, latency is a big issue. For others, power could be an issue. And was mentioned on Monday, the ability to not have to create an engineered link could address issues.
The great thing is that the task force is considering these things and truly looking at if we can do a 3m no FEC. The task force is not ignoring the ask, and that's great because for 25G Ethernet, that server to switch link is going to
be the dominant volume.
I know I'm preaching to the choir when I say that getting this right will be a big win for the specification and the market. I believe that's why we're seeing the passion and dedication from our fellow colleagues to bring us data. :-) All
goodness in my books.
On Wed, Jul 15, 2015 at 12:34 PM, Jonathan King <jonathan.king@xxxxxxxxxxx> wrote:
Hi Brad, I understood there’s a case to be made for 3m based on latency, and enlarging the application
area for the standard. Likewise, I was just addressing the power argument. Let’s do things for the right reasons.
Best wishes
jonathan
From: Brad Booth [mailto:bbooth@xxxxxxxx]
Sent: Wednesday, July 15, 2015 12:28 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC
I was only providing clarification relative to the FEC power information.
You may have missed the points my colleague made yesterday about latency and BMP being ranked higher on the list than power. FEC increases latency, and in some applications latency
is a measured performance parameter. Therefore, to be competitive with a 3m solution, the end user may have to consider doing an engineered link. Once that reach becomes an engineering, it is no longer a standards-based commodity component which impacts the
market size for the standard. It is beneficial to the market and to end users to maximize the volume for a standards-based solution, and that can be achieved with a little more work for a 3m no FEC solution.
On Tue, Jul 14, 2015 at 8:28 PM, Jonathan King <jonathan.king@xxxxxxxxxxx> wrote:
Hi Brad, I don’t disagree with any of your points 1 to 4
and….
if you remove the FEC gain, the SNR has to be made up in other ways – FEC is an efficient way of buying SNR. Agree,
using FEC means adding the power for implementing the code/decode circuitry, but it allows power reduction in other parts of the link. Net result, less power burn for ‘with FEC’ links, leaving
more power for the things customers really care about etc…
best wishes
jonathan
From: Brad Booth [mailto:bbooth@xxxxxxxx]
Sent: Tuesday, July 14, 2015 6:03 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC
I think there a few factors that have to be considered, but unfortunately there was so much discussion about the validity of the data, that the key points were truly missed.
1) Datacenters have a power constraint. There is not unlimited power.
2) Even if the power is 25 mW for FEC, FEC has to be used on both ends of the link; therefore, the power number is double.
3) The power supplied is about 5-6x the power used due to losses inherent in electronics, transformers, etc.
4) Power is amplified by the scale. What seems small at a single point in the architecture is amplified when you move to hyper-scale. At 1M servers, that's 50 kW for FEC which is
equivalent to about 4-5 server racks.
Finally, power spent on FEC (if not required) takes power away from other things that customers really care about like VMs, security, etc.
Just some extra food for thought.
On Tue, Jul 14, 2015 at 4:26 PM, Scott Kipp <skipp@xxxxxxxxxxx> wrote:
802.3by,
Yesterday, there was considerable debate about power consumption due to the RS-FEC used in 25GbE. In goergen_3by_02_0715.pdf, the power consumption for RS-FEC was approximated
to be 300mW. I did a quick search and found these estimates:
http://www.ieee802.org/3/bm/public/sep12/wang_01_0912_optx.pdf = 45mW on slide 3
http://www.ieee802.org/3/bj/public/mar12/gustlin_01_0312.pdf = 90 mW on page 6
http://www.ieee802.org/3/bs/public/adhoc/logic/oct21_14/wangz_01_1014_logic.pdf, Wangz
concludes: In practice, either KR4 or KP4 FEC is easy to implement and not much power-consuming.
A conservative estimate is 100mW for 100GBASE-KR4. 25GBASE-KR4 will consume 1/4 of this power or 25mW based on a one of the four lanes.
I propose that 25mW is a practical power consumption for RS-FEC. Let’s calculate the cost of the power consumption for the server over a 3-year lifespan.
|
|
Goergen_3by_02_0715
|
My proposal
|
|
Power of RS-FEC (mW)
|
200 (value used in calculation)
|
25
|
|
Power Consumption over 3 year lifespan of server (kWh)
|
5.25
|
0.66
|
|
Cost of kWh ($/kWh)
|
0.11
|
0.05
|
|
Power cost/server for RS-FEC
|
$0.58
|
$0.03
|
Another correction that I wanted to make is to the estimate is the cost of electricity. Hyperscale data centers that consume megawatts of power buy electricity at wholesale rates.
Many are located near electric power plants so that Google pays about $0.04/kWh according to this article:
http://www.oregonlive.com/silicon-forest/index.ssf/2013/09/google_reaches_new_data_center.html
The wholesale electricity industry is regulated and current pricing is available for this site and shows electricity ranges in 2015 has varied from $0.036/MWh to $0.065/MWh. Read
more here:
http://www.eia.gov/electricity/wholesale/
I used a conservative $0.05/kWh.
My estimate is that the cost of RS-FEC per server is about $0.03 over the life of the server. Three cents is basically in the noise for our purposes and should not be a fundamental
driver for 3m cabling without FEC.
Kind regards,
Scott Kipp
|