AWS to bare metal two years later: Answering your questions about leaving AWS
In this technical retrospective (2025), @ndhandala shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Post-ZIRP & AI (2023–2026)
COMMUNITY SCORE ★ 727 points
TOTAL DISCUSSIONS 491 comments
Something I've noticed about all public clouds is that they promise a unique value proposition, and then entirely fail to deliver it for almost all customers.
Specifically, they promise to provide a "small, rapidly growable slice of a very big thing". I.e.: You can create an empty S3 account for $0.00, fill it with a few megabytes of initial data for $0.00001, and then if you suddenly need petabytes then it scales smoothly and beautifully up past any reasonable scale. You get billed for that at an exorbitant rate, but the point is that you can do it without having to rearchitect anything.
"This Works Great(tm)" for three specific categories of customers:
- Tiny organisations that expect to grow big suddenly: Startups, and maybe the few orgs that have rare or annual events and nothing in between. Think electronic voting systems and the like.
- Small international companies that need global presence on the cheap. SaaS vendors, IoT, and a few niche organisations are pretty much the only ones in this category.
- Enormous organisations that need large scale but can't be bothered (or can't afford) managing that. There was an AWS talk about a customer that needed about 1 PB of storage which these days is "just" 500 x 20 TB disks, but they needed burst IOPS far in excess of that, on the order of 100,000 disks. The "thin slice" model of the cloud works great for this, because S3 has millions of disks behind it, and each customers' data is spread out over those disks.
Everybody else in the "medium" category is sold a bunch of bullshit.
Cloud VMs are between 5x and 10x as expensive as the on-prem equivalents, all costs factored in. I've seen the numbers from CIOs and CTOs, they all got told "the cloud is cheaper", and their costs skyrocketed as soon as they went to the cloud. You still need engineers. You still need "deployments". You still need sysops. You still need to update your VMs and their software somehow. Nothing changes really, except suddenly VMware starts looking cheap in comparison.
The "You can scale, the cloud is flexible, you can..." marketing is a load of bollocks.
First, Pay-as-you-Go pricing is on average 7x on-prem VM pricing. The only way to bring this down to merely 3x the on-prem cost is to LOCK IN the compute using "reservations" of some sort, typically for 3-year periods. This is NOT FLEXIBLE BY DEFINITION!
The whole marketing of the cloud revolves around the flexibility, but all of their pricing and cost optimisation revolves around getting customers to lock in spending for years and years.
Similarly, Spot-priced anything is so unreliable that it is completely unusable for almost all "enterprise" customers, even for non-production use.
Seriously, can you imagine telling a developer that costs $200/hour that they can't do their work because their DEV instance is gone for a day because some other tenant needed it more!?
This underlying issue with VM pricing means that any service that is built on top of VMs inherits the same pricing model with lock-in contracts, totally negating the scalability benefits and global presence of the public cloud. If you want S3 in every region with 10 MB each... that's cheap. If you want just one VM in every region... no longer cheap. If you want any VM-based service in every region... also not cheap. Oh... you wanted DEV/TST/UAT/GREEN/BLUE? With high availability? Get the CFO on the line, he'll need to approve your budget!
"Just engineer your software to be cloud native!" is what you inevitably hear from apologists. Sure, sure... I'll get right on that. I mean sure, the public cloud vendors failed to do so for like two thirds of their own first-party products, but I'm sure I'll have better luck! Let's see.. my "medium sized org" has... checks notes... about 1,000 unique pieces of software deployed on VMs, of which 800 are CotS vendor products, 600 of which require Windows Server and have a GUI configurator. This will go... smooth.
For 90% of the potential customers out there, the big businesses, the enterprises, the universities, governments, and the like... it's just a more expensive data centre that someone else runs for them.
The only significant advantages to public cloud vs on-prem I've seen are:
- Faster networking, with a well-engineered 100 Gbps or 200 Gbps core in most clouds. This includes Internet uplinks of similar spec. These are rare in private hosting.
- Three-way zone redundancy instead of the typical two-way.
- Zone redundant services that are just a "checkbox".
- Separation of duties where the layer 2 network and hypervisor are managed by a vendor instead of internal staff. (This can be difficult to arrange internally for medium sized orgs needing high security, there aren't enough IT admin staff for true separation.)
... that's about it.
Running EKS on AWS was their problem. If they didn't run EKS on AWS, they would've had a considerably simpler setup running Amazon Linux, not having to upgrade Kubernetes every 3 quarters, managing network security using security groups instead of having open internal networking, and running in a single AZ would've eliminated intra-AZ costs. In large data centers like us-east-1, an individual AZ is actually internally striped for extra redundancy, and you are much more likely to experience regional downtime than single AZ downtime, especially if you have a stable workload and do not rely on tech beyond rock-solid basics (EC2, VPC, ELB, S3, EBS). If you're willing to operate a single bare metal rack in a DC, you should be willing to run in a single AWS AZ.
I don't know how much time they spend configuring/dealing with Kubernetes, but I bet it's a large chunk of the 24 hour engineer-hours per quarter. But this is not a required expense: "EKS had an extra $1,260/month control-plane fee". Running EKS adds a massive IAM policy maintenance overhead, whereas a non-EKS (EC2 w/ golden AMIs) setup results in drastically simpler IAM policies.
NAT gateways are ~$50 a month, plus data transfer. Setting up a gateway VPC endpoint to S3 will avoid having to pay transfer charges to S3.
They were at 90% reservation capacity, so they should be using reservations for greater savings and in fact, running stable workloads with reservations is something that AWS excels at. Reservation means that you will be able to terminate and re-launch instances even when there's a spike in demand from other users--your instance capacity is guaranteed.
Running the basics on VMs also effectively avoids vendor lock-in. Every cloud provider supports VMs with a RedHat clone, VPCs, load balancing, networked storage, access controls, object storage and a fixed size fleet with auto-relaunch on instance failure.
With a consistent workload, they would have very likely escaped the downtime from AWS a week ago as well, because, as per AWS, "existing EC2 instances that had been launched prior to the start of the event remained healthy and did not experience any impact for the duration of the event".
With Terraform and automation for building launchable images, you can stand up a cluster quickly in any region with secure networking, including in a separate AWS account, in the same region, for the sake of testing.
With AWS, you can set up automatic EBS backups of all your data to snapshots trivially, and even send them to a 3rd locked-down account, so they can't be accidentally wiped.
I'm involved in a fairly large academic cloud deployment, sited in a 15MW data center built and shared by a few large universities.
There are huge advantages of scale to computer operations in a few areas:
- facility: the capital and running cost of a purpose-built datacenter is far cheaper per rack than putting machines in existing office-class buildings, as long as it's a reasonable size - ours is ~1000 racks, but you might get decent scale at a quarter of that. (also one fat network pipe instead of a bunch of slow ones)
- purchasing: unlike consumer PCs, low-volume prices for major vendor servers are wildly inflated, and you don't get decent prices until you buy quite a few of them.
- operations: people come in integer units, and (assuming your salary ranges are bounded) are only competent in small number of technical areas each. Whether you have one machine or 1000s you need someone who can handle each technology your deployment depends on, from Kubernetes to network ops; multiply 4x for those requiring 24/7 coverage, or accept long response times for off-hours failures.
That last one is probably the kicker. To keep salary costs below 50% of your total, assuming US pay rates and 5-year depreciation since machines aren't getting faster as quickly as they used to, you probably need to be running tens of millions of dollars in hardware.
Note that a tiny deployment of a few machines in a tech company is an exception, since you have existing technical staff who can run them in their spare time. (and you have other interesting work for them to do, so recruiting and retention isn't the same problem as if their only job was to babysit a micro-deployment)
That's why it can be simultaneously true that (a) profit margins on AWS-like services are very high, and (b) AWS is cheaper than running your own machines for a large number of companies.
Sounds like they did the right thing for their business model.
I think as AWS grows and changes the curve of the target audience is changing too. The value proposition is "You can get Cloud service without having a dedicated Cloud team," but there are caveats:
- AWS is complicated enough that you will still need a team to integrate against it. The abstractions are not free and the ones that are leaky will bite you without dedicated systems engineers to specialize in making it work with your company's goals.
- For small companies with little compute need, AWS is a good option. Beyond a certain scale... It is worth noting that big companies build their own datacenters, they don't rely on someone else's Cloud. Amazon, Google, and Microsoft don't run on each other.
- Recently, the cost model has likely changed if a company pokes their head up and runs the numbers, there's, uh, quite a few engineers with deep knowledge of how to build a scalable cloud infrastructure available to hire now for some reason. In fact, a savvy company keeping its ear to the ground can probably snap up some high-tier talent very soon (https://www.reuters.com/business/world-at-work/amazon-targets-many-30000-corporate-job-cuts-sources-say-2025-10-27/).
It really depends on where your company's risk and cost models are. Running on someone else's cloud just isn't the only option.
Am I just naive? How is a uptime SaaS product saving over a million year on managed colo vs AWS? Was every API route in it's own EC2 instance?
AWS is expensive sure, but over a million dollars a year? For this product specifically?.
I got some clarification from their earlier posts and it looks like they were intentionally avoiding any AWS platform features:
I just think if you are going to deploy on AWS, then treat it AWS like managed-colo, then your bill is going to be high. I understand how that seems unfair, but AWS isn't really in the business of selling virtual machines. If you sit down and ask yourself how you got here, it just seems like you committed yourself to wasting money. If I knew I just needed some linux boxes from the start, there are better choices than AWS.