Datacenters

Cloud computing

Cloud computing is the delivery of on-demand computing resources, everything from application to data centers, over the Internet on a pay-for-use basis (IBM).

• was first used in the 1996 Compaq business plan.

Principles of cloud computing:

• computing as a utility
• purchase how ever much you want, whenever you need it
• service ranges from access to raw (virtual) machines to higher level: distributed storage, web services, etc.
• Access provided via the internet
• Pay-for-use pricing

What enables cloud computing?

• Data centers! Data centers are typically located near a body of water so it can cool down. There are racks of servers connected through ethernet cables.

Datacenter Equipment

Equipment inside a DC:

• Server (in rack format): 8 cores, 16 GB DRAM, 4x1 TB disk. They have specific physical dimensions. Servers are put into a rack
• Rack: fits 40-80 servers + Ethernet. LAN (1-10 Gbps, now increased to ~40 Gbps), switch in the middle ("rack switch", "top of the rack", "ToR switch"). Server racks are grouped to a cluster switch. Servers on one rack can communicate with each other because they have a direct link.
• Cluster (aka array): 16-32 server racks + larger LAN switch ("array switch"). 10x faster => cost 100x.

Datacenter Network

• another layer sits on top of the aggregation switches, called the core switch. The core is connected to the internet. Note that the core switch is always connected to every single aggregation switch to provide redundancy. Core switches are very expensive (higher in hierarchy = fewer ports = expensive).

Datacenter networks have ten to hundreds of thousands of hosts, often closely coupled, in close proximity:

• e-commerce (e.g., Amazon)
• content servers (e.g., Netflix, YouTube, Apple, Microsoft)
• search engines, data mining (e.g., Google)
• social networks (e.g., Facebook, Twitter, Instagram)

Challenges

Challenges: multiple applications, each serving massive number of clients. How big exactly?

• 1 million server/sites (microsoft/amazon/google)
• >$1 billiion to build one site (facebook)
• >$20 million per month to operate (microsoft 2009)
• but only O(10-100) sites

Datacenter network challenges:

• Not enough bandwidth: What if two servers want to talk to each other and they are in different racks or aggregations? Oversubscription: less bandwidth in the ToR-Agg links than all the servers' bandwidth in the rack. Oversubscription ratio between bandwidth underneath and bandwidth above is 40:1 for rack to aggregation level, 200:1 for core level
• Not enough paths between server switches. This causes load balancing issues and failure recovery issues.

Solution:

• Instead of using expensive switches, use cheaper commodity switches and use redundancy to connect them.
• each rack is connected to two aggregate switches and they are connected to four core switches.

Let's look at a web service in detail.

1. you go search something on Google
2. it goes from the internet to a server on a rack.
3. one of the servers in your datacenter will receive your request
4. Your request will go from a top-level aggregator to several mid-level aggregators, which will divide the queries into smaller parts (one fetches the jhu website's images for its preview while another one does something else) to its workers.
5. once workers finish with your query, it sends the response back to the mid-layer aggregators.
6. Sends to top layer aggregator
7. the user gets the search results.

Datacenter Traffic

Partition-aggregation traffic: the load is divided up and sent to many servers on different racks.

• north-south traffic describes when the traffic enters the datacenter and it gets distributed between the levels
• east-west traffic describes how the work is partitioned and divided between many servers.

End to end response time is <200ms between receiving the user query in the browser and displaying the results.

• RTT = O(10) to 100 ms
• What remains?

Data center traffic characteristics. This paper instrumented a large cluster used for data mining and identified distinctive traffic patterns

• traffic patterns are highly volatile: a large number of distinctive patterns in a day
• traffic patterns are unpredictable: weak correlation between patterns.

Because of the lack of pattern, traffic-aware optimization needs to be done frequently and rapidly.

Datacenter workloads: Mice and elephants

• short messages (e.g., query, coordination) -> low latency
• large flows (e.g., data update, backup) -> high throughput

Incast: synchronized fan-in congestion.

• if all the workers send a response to its aggregator at the same time, then some of the packets would be dropped. After some time, the aggregator would realized its missing some results from a TCP timeout (like 300ms), then the packet would be sent back to the aggregator. This is a traffic pattern that happens in datacenter (and does not happen in LAN or internet). This is because of the way the partition-aggregation is designed.
• This is pretty serious. It causes jittering trades of median for high percentiles.

Trasport requirements

Datacenter transport requirements. The challenge is how to achieve these together:

1. low latency: short messages, queries
2. high throughput: continuous data updates, backups
3. high burst tolerance: incast

High throughput vs low latency.

• baseline fabric latency (propagation + switching) is 10 microseconds.
• High throughput requires buffering for rate mismatches, but this significantly adds to queuing latency.

Data Center TCP

Recall the TCP algorithm.

• additive increase $W \gets W+1 \text{ per round-trip time}$
• multiplicative decrease: $W \gets W/2 \text{ per drop or ECN mark}$
• ECN = explicit congestion notifcation

Data center TCP: extract multi-bit feedback from single-bit stream of ECN marks.

• reduce window size based on faction of marked packets.

For example,

ECN Marks	TCP	DCTCP
101110111	Cut window by 50%	Cut window by 40%
000000001	Cut window by 50%	Cut window by 5%

The algorithm:

• Switch side: mark packets when $\text{queue length > K}$
• Sender side: maintain running average of fraction of packets marked ($\alpha$): each RTT: $F = \frac{\text{\# of marked ACKs}}{\text{Total \# of ACKs}} \implies \alpha \leftarrow (1-g)\alpha - gF$ $W \gets (1-\frac{\alpha}{2})$

Adaptive window decreases: decrease factor between 1 and 2.