Project Tungsten: Bringing Spark Closer to Bare Metal
In this technical retrospective (2015), @mateiz shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Cloud & Mobile Era (2014–2019)
COMMUNITY SCORE ★ 59 points
TOTAL DISCUSSIONS 11 comments
I'm not sure I completely buy their claim that network/disk IO is no longer the bottleneck in many situations. Their recent NSDI paper (https://kayousterhout.github.io/trace-analysis/) is compute bound for a 20 node cluster, but they never evaluate it in a larger cluster with fixed data size. The availability of on-demand cloud resources would potentially make it easier to solve the bottleneck by just increasing the cluster size.
And on a related note, congratulations to Matei Zaharia for winning the ACM Best Dissertation Award for his work on Spark: http://awards.acm.org/doctoral_dissertation/
Are there JIRA issues related to these ongoing projects? I'd be interested to see the current thoughts on llvm code gen and the "Spark Binary Map."
This is really exciting and important Matei, can't wait to see it in action.
* Matei Zaharia is the creator of Apache Spark
That's a clever name. Somebody at spark has apparently used a TIG welder before.