Why You Shouldn't Build Your Own Datacenters

By Yiren Lu — Sep 19, 2021

Tweet by Hemant Mohapatra

now: SaaS & Crypto @lightspeedindia, past: investor @a16z, product BD @Google, engg @AMD; lifelong fan of physics, poetry and piano // http://hmohapatra.com

Note: This tweet is from 2019 so the Lyft cloud numbers may be off, but I thought it was so insightful I had to share.

So @lyft is paying $8m/mo to @AWS -- almost $100m/yr! Each ride costs $.14 in AWS rent. I keep hearing they could build their own DC & save. My early days at @Google cloud, heard the same from customers: "at scale, owning is cheaper". It wasn't - they all came around. Here's why:

Construction of a mid-sized Enterprise DC (just 5000sqft), at just "tier3" availability (3 9s) will cost around 40m. If you want 5 9s redundancy you will need 1-2 failovers, so 3x that. Incld racks, cooling, power, construction and land. Using a colo @Equinix will likely save 20%

But your DC costs will amortize over 10 years, correct? Yes. But there's more. Construction will take 12-24mos. For that time, company loses focus, hires non-core engg, vendors, and planners that understand bldg codes, fire safety, env rules, security, maintenance etc.

Then for 10 years you have: ongoing support, maintenance & repair, costs of power, heating/cooling, and biometric security of physical assets. Power bills alone run in xxMs that's why Google DCs are so remote and near Geo/hydro/solar power sources.

Moreover, you need to build for 10yrs out, not today, so you'll likely either keep building more and more, or overbuild capacity by 50-100%. Your initial estimate of 40m (x3) is now 60-80m (x3).

Next comes some of the most expensive stuff: fiber! Without gigabit connectivity you are toast. Building your own undersea cables will cost 100s of Ms, so you'll be beholden to buying dark fiber from tier1/2 telcos and pay exorbitant rates for intercontinental traffic.

Next comes the outages: no matter what you do, you'll never have hot-swappable everything managed by ultra fast robotic arms that replace hard drives in seconds. Your hw will fail at rack/server level at 2-3x the avg cloud. Massive costs to missing SLAs to the biz that you bear.

Finally comes certifications for PCI, for HIPAA, for Gov, bla bla bla. You'll coordinate with the consultants day and out and it'll take anywhere between 3-12 months to get most of your infra certified to run your biz the right way.

I ran a "TCO" analysis at Google to convince a large customer why GCP was better. The numbers were clear, but customer wasn't convinced. Went to Verizon cloud, which shut down, then HP cloud, which also got shut down. Went on-prem. Then came to us. Zynga story is well known too.

Few rare examples of this working in parts. Netflix & Dropbox. When companies reach "internet scale" & have to do a lot of customization on the stack, running own infra may make sense IFF you have the GMs to cover 2yrs CAPEX & plan upfront. Both firms still use the cloud a lot.

E.g. if you are +30% of the internet traffic (nflx) it doesn't make sense to pay rent to telcos any more and feed their margins. You have the volume and stable demand to justify ownership. For the rest, cloud is where they'll live and die.