Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC

[Dan Dove <dan.dove@xxxxxxxxxxxxxxxxxx>]

Hi Brad,

While your particular implementation might be confidential, I know that standard NIC vendors and server vendors publish data for their implementations. Often they use standardized benchmark performance suites, sometimes generally available applications.

We could understand the impact for the general user, the "broad market" so to speak, with that kind of data.

Once we understand the end-to-end latency for broad market equipment, and more important, its relative impact on performance in broad market applications, we can quantify the impact of FEC upon that performance in specific terms.

Otherwise, you could perhaps have some of your engineering folks perform the benchmarking tests with a variable latency factor (easily constructed using fiber-optic interfaces of various lengths) and chart performance as you increase channel latency. All of that can be done in relative numbers to maintain some confidentiality on the absolute performance.

Dan Dove
Chief Consultant
Dove Networking Solutions
530-906-3683 - Mobile

On 7/16/15 11:10 AM, Brad Booth wrote:

Jeff,

Excellent point.

Kapil, Dan,

As I'm sure you can understand, the application and stack latency information is confidential. The task force should focus on the latency numbers that it has a direct ability to influence.

Cheers,
Brad

On Thu, Jul 16, 2015 at 12:41 PM, Jeffery Maki <jmaki@xxxxxxxxxxx> wrote:

Kapil and Dan,

 

Absolute numbers would be great to see; however, we still need to understand what incremental differences create meaningful competitive advantage of one datacenter providing services over another.

 

Jeff

 

 

From: Dan Dove [mailto:dan.dove@xxxxxxxxxxxxxxxxxx]
Sent: Thursday, July 16, 2015 12:29 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC

 

Hi Kapil,

Good addition to the discussion.

A break down of entire end-to-end latency is essential. This would include stack, PHY/PMD, cable, switch, cable, PHY/PMD, stack.

An analysis that shows how an increase in latency on the PHY/PMD impacts performance of useful applications in addition to benchmark tests would be helpful.

Regards,

Dan Dove
Chief Consultant
Dove Networking Solutions
530-906-3683 - Mobile

On 7/15/15 8:55 PM, Kapil Shrikhande wrote:

Hi Mike,  

Thanks for offering to do this. That will be really a useful thing.

Looking at switch data-sheets using previous generation chips, we know that switch latency has been in the 100s of nanoseconds range, and therefore the added latency from FEC could be a good chunk of that. But what I want to understand better is – how does this FEC latency stack-up in the context of the end-end latency, e.g. latency of a packet from server A going to server B in a Data Center. And should we (or shouldn’t we) also consider including the latency of the server (application) stack in this end-end latency. And can we estimate the impact of PMD latency to the latency of the “job” or Application, or is that too complex (I suspect)... This is not an easy task; and is specific to Data center types, workloads, etc. but getting a few viewpoints on this topic will greatly help the discussion.   So, thanks for doing this.

 

Kapil

 

 

From: Mike Andrewartha [mailto:Mike.Andrewartha@xxxxxxxxxxxxx]
Sent: Wednesday, July 15, 2015 7:02 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC

 

Hi folks,

 

I had a hall chat with Mark today and agreed to bring in a presentation to an upcoming ad-hoc to lay out the points I made during comments after Joel’s presentation on the value of a no-FEC solution.  It sounds like there is value in explaining the impact of latency in the market spaces we compete in as well as the relative importance of the factors influenced by using FEC.  I’m not sure I can get something ready for next week but will aim for the week after. 

 

Best regards,

 

Mike Andrewartha

 

From: Jonathan King [mailto:jonathan.king@xxxxxxxxxxx]
Sent: Wednesday, July 15, 2015 2:17 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC

 

Well said !

I can only add that I love my Mum, and apple pie is nice.

 

J

 

From: Brad Booth [mailto:bbooth@xxxxxxxx]
Sent: Wednesday, July 15, 2015 1:00 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC

 

Jonathan,

 

No disagreement on doing things for the right reasons.

 

The issue is, which reason is the right one? :-) For some applications, latency is a big issue. For others, power could be an issue. And was mentioned on Monday, the ability to not have to create an engineered link could address issues.

 

The great thing is that the task force is considering these things and truly looking at if we can do a 3m no FEC. The task force is not ignoring the ask, and that's great because for 25G Ethernet, that server to switch link is going to be the dominant volume.

 

I know I'm preaching to the choir when I say that getting this right will be a big win for the specification and the market. I believe that's why we're seeing the passion and dedication from our fellow colleagues to bring us data. :-) All goodness in my books.

 

Cheers,
Brad

 

 

 

On Wed, Jul 15, 2015 at 12:34 PM, Jonathan King <jonathan.king@xxxxxxxxxxx> wrote:

Hi Brad,  I understood there’s a case to be made for 3m based on latency, and enlarging the application area for the standard. Likewise, I was just addressing the power argument. Let’s do things for the right reasons.

Best wishes

jonathan

 

From: Brad Booth [mailto:bbooth@xxxxxxxx]
Sent: Wednesday, July 15, 2015 12:28 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC

 

Jonathan,

 

I was only providing clarification relative to the FEC power information.

 

You may have missed the points my colleague made yesterday about latency and BMP being ranked higher on the list than power. FEC increases latency, and in some applications latency is a measured performance parameter. Therefore, to be competitive with a 3m solution, the end user may have to consider doing an engineered link. Once that reach becomes an engineering, it is no longer a standards-based commodity component which impacts the market size for the standard. It is beneficial to the market and to end users to maximize the volume for a standards-based solution, and that can be achieved with a little more work for a 3m no FEC solution.

 

Cheers,
Brad

 

 

On Tue, Jul 14, 2015 at 8:28 PM, Jonathan King <jonathan.king@xxxxxxxxxxx> wrote:

Hi Brad, I don’t disagree with any of your points 1 to 4

and….

if you remove the FEC gain, the SNR has to be made up in other ways – FEC is an efficient way of buying SNR.  Agree, using FEC means adding the power for implementing the code/decode circuitry, but it allows power reduction in other parts of the link.  Net result, less power burn for ‘with FEC’ links, leaving more power for the things customers really care about etc…

 

best wishes

jonathan

 

From: Brad Booth [mailto:bbooth@xxxxxxxx]
Sent: Tuesday, July 14, 2015 6:03 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption of of 25G RS-FEC

 

I think there a few factors that have to be considered, but unfortunately there was so much discussion about the validity of the data, that the key points were truly missed.

 

1) Datacenters have a power constraint. There is not unlimited power.

2) Even if the power is 25 mW for FEC, FEC has to be used on both ends of the link; therefore, the power number is double.

3) The power supplied is about 5-6x the power used due to losses inherent in electronics, transformers, etc.  

4) Power is amplified by the scale. What seems small at a single point in the architecture is amplified when you move to hyper-scale. At 1M servers, that's 50 kW for FEC which is equivalent to about 4-5 server racks.

 

Finally, power spent on FEC (if not required) takes power away from other things that customers really care about like VMs, security, etc.

 

Just some extra food for thought.

 

Thanks,
Brad

 

On Tue, Jul 14, 2015 at 4:26 PM, Scott Kipp <skipp@xxxxxxxxxxx> wrote:

802.3by,

 

Yesterday, there was considerable debate about power consumption due to the RS-FEC used in 25GbE.  In goergen_3by_02_0715.pdf, the power consumption for RS-FEC was approximated to be 300mW.  I did a quick search and found these estimates:

http://www.ieee802.org/3/bm/public/sep12/wang_01_0912_optx.pdf = 45mW on slide 3

http://www.ieee802.org/3/bj/public/mar12/gustlin_01_0312.pdf = 90 mW on page 6

http://www.ieee802.org/3/bs/public/adhoc/logic/oct21_14/wangz_01_1014_logic.pdf, Wangz concludes: In practice, either KR4 or KP4 FEC is easy to implement and not much power-consuming.

 

A conservative estimate is 100mW for 100GBASE-KR4.  25GBASE-KR4 will consume 1/4 of this power or 25mW based on a one of the four lanes.

 

I propose that 25mW is a practical power consumption for RS-FEC.  Let’s calculate the cost of the power consumption for the server over a 3-year lifespan.

 

	Goergen_3by_02_0715	My proposal
Power of RS-FEC (mW)	200 (value used in calculation)	25
Power Consumption over 3 year lifespan of server (kWh)	5.25	0.66
Cost of kWh ($/kWh)	0.11	0.05
Power cost/server for RS-FEC	$0.58	$0.03

 

Another correction that I wanted to make is to the estimate is the cost of electricity.  Hyperscale data centers that consume megawatts of power buy electricity at wholesale rates.  Many are located near electric power plants so that Google pays about $0.04/kWh according to this article:

http://www.oregonlive.com/silicon-forest/index.ssf/2013/09/google_reaches_new_data_center.html

 

The wholesale electricity industry is regulated and current pricing is available for this site and shows electricity ranges in 2015 has varied from $0.036/MWh to $0.065/MWh.  Read more here:

http://www.eia.gov/electricity/wholesale/

 

I used a conservative $0.05/kWh.

 

My estimate is that the cost of RS-FEC per server is about $0.03 over the life of the server.  Three cents is basically in the noise for our purposes and should not be a fundamental driver for 3m cabling without FEC.

 

Kind regards,

Scott Kipp