Lessons Learned from Scaling to Multi-Terabyte Datasets
In this technical retrospective (2024), @thunderbong shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Post-ZIRP & AI (2023–2026)
COMMUNITY SCORE ★ 149 points
TOTAL DISCUSSIONS 45 comments
Since you are already running on AWS, I wonder if you would consider AWS serverless EMR an option?
https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html
I guess the pricing will be cheaper than renting entire machines since you are doing something like micro-spot allocations for map or reduce jobs.
We have done a lot of analytical and also batched workflows with AWS Athena. This is really cheap and also, once you get a hang of it, you get surprised how much you can achieve with only SQL.
I am hoping serverless EMR would be roughly equivalent, just with another language...
Servers grow much bigger. 256GB of RAM is pretty much a standard rack server on a major cloud. On AWS, ultra memory with 24TB of RAM is one API call away from you :-). The article took note of that, but tooling on one node puzzles me.
So, multi TB can still be a single node with DuckDB. It's also rather a small ClickHouse cluster. It sounds easier to use DuckDB than the proposed tools.
In many use cases, if it's not 24/7, you can use a data lake (e.g. Iceberg) and query it if needed. Databricks seems a way to go, since the author uses Spark.
When I first read the title I had in my head that it said multi-petabyte. When I think Multi-terabyte, I think 'yea, that is mostly in RAM'. 4TBs of RAM is sort of the new lower norm in DB machines, and GCP and AWS have high RAM machines at 12TB and 24TB. Not that these are not good ideas about scaling to multi-terabytes.. it is just interesting how much data has grown in size.
https://github.com/Eventual-Inc/Daft Is also great at these types of workloads since it’s both distributed and vectorized!
A “multi-terabyte dataset” is something that fits into a single machine, and can be loaded into Clickhouse in a few minutes.