Study Group Members,
I share the regret, expressed in several posts to this list, that large internet data center operators are unwilling to make their requirements known in an open non confidential manner. I would like to forward to the group a paper, recommended by a colleague, that may help close this information gap.
“The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
Luiz André Barroso and Urs Hölzle”
www.morganclaypool.com
ISBN: 9781598295566 paperback
ISBN: 9781598295573 ebook
The paper may be viewed, without charge, at the Morgan & Claypool site, but please be aware of the restrictive notices on page (iv)
http://www.morganclaypool.com/doi/pdf/10.2200/S00193ED1V01Y200905CAC006
Copies may also be purchased at internet book sites.
The paper, 120 pages in all, describes the specific challenges faced when applications implemented by internet content providers, such as Google and Microsoft, require many thousands, even tens of thousands of servers. Indeed the usage model of the data center changes from “a place to house servers” to a “building to host an application”.
The introduction, pages 1 to 11, gives insight into why these warehouse computers differ from traditional data centers and how this impacts the need for communication bandwidth within the data center.
The ideal system, as described by Barosso and Hölzle, would be one where the cross sectional communication bandwidth of the data center would equal to the bandwidth of the servers, i.e. a network without over subscription. In such a system the application developer can freely locate functions throughout the network, optimally distributing load and minimizing computational and HVAC hotspots. The authors admit that economic considerations cannot support such a model and that over subscription levels of 5:1 are evident between racks of servers (80 servers per rack), 10 racks in a group (800 servers). Using the terminology as per kolesar_02_0911_NG100GOPTX, page 4, citing barbieri_01_0107.pdf, this over subscription would refer to the “access" to “distribution” network layers.
http://www.ieee802.org/3/100GNGOPTX/public/sept11/kolesar_02_0911_NG100GOPTX.pdf
Decreasing the relative cost of these access to distribution layer links would enable warehouse scale computer builders to reduce the level over subscription and get closer to their ideal system. Throughout the paper the authors use a system wide approach to find the lowest cost. By this I mean reducing cost in one area is not beneficial if it results in an increase in overall costs, since cost is just shifted from one area to the other.
Such considerations should play a part in the determination of a reach objective. As we increase the reach to include an ever higher percentage of the links as described in kolesar_02_0911_NG100GOPTX, we should be cognizant of the increase in relative cost to achieve this increased reach; and evaluate if, when considered at a network level with a distribution of link lengths as per Paul’s presentation, we are decreasing overall cost, or not.
Best Regards
Andy