Hi Joel,
I think we need to keep the big picture in mind at all times.
If you have the lights automatically shut off in a room too
quickly to save energy, someone might stumble and fall requiring
ambulances, hospitals and various other response activities that
might not have been required had you simply been more diligent in
taking the big picture into mind.
Softening the robustness of a specification can/will lead to
increases in service calls. Service calls cost energy, time, and
$.
I am 100% in support of end-users, or system-vendors adding
requirements onto their own specifications. When I was at HP, we
NEVER accepted a marginally compliant solution. We always put more
into it than was required to ensure we minimized service calls.
Not only because we wanted to save in support cost, but to
minimize negative impact to our customers.
The cost of a marginal solution is much greater than materials or
support costs... and I know our end-users are completely aware of
that. They will ensure sufficient margin even if they have to
mandate super-compliant implementations.
But for the vast majority of enterprise end-users, they will
expect plug-n-play with high reliability. They don't get into the
subtle difference in performance that might have been achieved if
they had only engineered their links for no-FEC support. They
would see the 25G w/FEC solution as substantially better than the
10G solution they used before and would be happy with that.
To them, the question would not be "If a tree fell in the forest
and it had no-FEC, would it have fallen faster?"
The vast majority of end-users just do not get into that level of
detail on their networking gear. They expect it to work reliably
and interoperate with other devices. They don't get into shaving
small bits of performance out of a standard solution. Those that
do, can shave at-will. FEC will have Auto-Negotiation capability
and they can engineer their links.
Of course, this is just my opinion and we can all agree to
disagree.
On 7/15/15 10:06 AM, Joel Goergen
(jgoergen) wrote:
Adee
I will continue to advocate for solutions that are power
efficient, scalable, and implement or have the ability to
implement as low a power solution as possible. Regardless of
the amount of power, as a group, we have to stop throwing power
at a solution.
When you leave a room, do you shut the lights off or leave
them on? I don’t think it costs much either way, but we should
be trying to shut the light off so as not to waste power. Why
spend the money to leave the light on when you could put it
towards something else.
Power makes a difference and we should always be asking the
question are we designing the most efficient ethernet possible
for the objectives we want to meet.
Joel
"the SERDES designs are complete and we are
just cleaning up the COM code (Afe and SNDR changes
are just fixes) and taking out some of the margin"
I beg to differ – (a) SERDES designs are still
ramping up, future designs that comply to the standard
will be more power optimized, and possibly save power
when FEC is enabled and provide more margin. (b) The
discussions led by Joel proved that there is very
little margin to be gained from COM parameters
assuming the AFE parameters are not improved (which
may cause higher power consumption). We were not even
shown any data to convince that no-FEC on 3 m is
feasible with worst-case instances of current
products.
Having said that, the numbers shown yesterday,
and especially with Scott's addition below, make
"power saving" a very unconvincing argument. We could
continue discussing power as a strawman to prove that
no-FEC is bad for energy saving. So I suggest that we
stop even mentioning power.
I was convinced from the discussion yesterday
that latency is a metric that the market looks at, and
therefore is important. Let's concentrate on that.
Thanks Tom, good point, power is not the
justification, the impact of latency is.
Jonathan,
One of the points that I tried to make during
the discussion was that we can easily achieve 3m no
FEC with no changes to the existing 25G SERDES. This
means that although in a perfect world you might be
able to take advantage of FEC to design a lower power
SERDES, reality is that the SERDES designs are
complete and we are just cleaning up the COM code (Afe
and SNDR changes are just fixes) and taking out some
of the margin.
I think we need to move the discussion away
from power and over to latency. Does anyone have an
idea on the impact of added latency? I looked up low
latency ethernet switches and found specs ranging from
190ns to 350ns. Adding 82ns of latency (x2 for in and
out) looks like a doubling of latency to me.
Tom
Hi Brad, I don’t
disagree with any of your points 1 to 4
and….
if you remove
the FEC gain, the SNR has to be made up in other ways
– FEC is an efficient way of buying SNR. Agree, using
FEC means adding the power for implementing the
code/decode circuitry, but it allows power reduction
in other parts of the link. Net result, less power
burn for ‘with FEC’ links, leaving
more power for the things customers
really care about etc…
best wishes
jonathan
From: Brad Booth [mailto:bbooth@xxxxxxxx]
Sent: Tuesday, July 14, 2015 6:03 PM
To: STDS-802-3-25G@xxxxxxxxxxxxxxxxx
Subject: Re: [STDS-802-3-25G] Power Consumption
of of 25G RS-FEC
I think there a few factors that
have to be considered, but unfortunately there was
so much discussion about the validity of the data,
that the key points were truly missed.
1) Datacenters have a power
constraint. There is not unlimited power.
2) Even if the power is 25 mW for
FEC, FEC has to be used on both ends of the link;
therefore, the power number is double.
3) The power supplied is about
5-6x the power used due to losses inherent in
electronics, transformers, etc.
4) Power is amplified by the
scale. What seems small at a single point in the
architecture is amplified when you move to
hyper-scale. At 1M servers, that's 50 kW for FEC
which is equivalent to about 4-5 server racks.
Finally, power spent on FEC (if
not required) takes power away from other things
that customers really care about like VMs, security,
etc.
Just some extra food for thought.
On Tue, Jul 14, 2015 at 4:26 PM,
Scott Kipp <skipp@xxxxxxxxxxx>
wrote:
802.3by,
Yesterday, there was
considerable debate about power consumption
due to the RS-FEC used in 25GbE. In
goergen_3by_02_0715.pdf, the power consumption
for RS-FEC was approximated to be 300mW. I
did a quick search and found these estimates:
http://www.ieee802.org/3/bm/public/sep12/wang_01_0912_optx.pdf
= 45mW on slide 3
http://www.ieee802.org/3/bj/public/mar12/gustlin_01_0312.pdf
= 90 mW on page 6
http://www.ieee802.org/3/bs/public/adhoc/logic/oct21_14/wangz_01_1014_logic.pdf,
Wangz concludes: In practice, either KR4 or
KP4 FEC is easy to implement and not much
power-consuming.
A conservative estimate is
100mW for 100GBASE-KR4. 25GBASE-KR4 will
consume 1/4 of this power or 25mW based on a
one of the four lanes.
I propose that 25mW is a
practical power consumption for RS-FEC. Let’s
calculate the cost of the power consumption
for the server over a 3-year lifespan.
|
|
Goergen_3by_02_0715
|
My
proposal
|
|
Power
of RS-FEC (mW)
|
200
(value used in calculation)
|
25
|
|
Power
Consumption over 3 year lifespan of
server (kWh)
|
5.25
|
0.66
|
|
Cost
of kWh ($/kWh)
|
0.11
|
0.05
|
|
Power
cost/server for RS-FEC
|
$0.58
|
$0.03
|
Another correction that I
wanted to make is to the estimate is the cost
of electricity. Hyperscale data centers that
consume megawatts of power buy electricity at
wholesale rates. Many are located near
electric power plants so that Google pays
about $0.04/kWh according to this article:
http://www.oregonlive.com/silicon-forest/index.ssf/2013/09/google_reaches_new_data_center.html
The wholesale electricity
industry is regulated and current pricing is
available for this site and shows electricity
ranges in 2015 has varied from $0.036/MWh to
$0.065/MWh. Read more here:
http://www.eia.gov/electricity/wholesale/
I used a conservative
$0.05/kWh.
My estimate is that the
cost of RS-FEC per server is about $0.03 over
the life of the server. Three cents is
basically in the noise for our purposes and
should not be a fundamental driver for 3m
cabling without FEC.
Kind regards,
Scott Kipp
|