How we built a new, fast file transfer protocol
In this technical retrospective (2022), @mcharawi shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
Core Principles for Independent Builders:
Defaulting to simpler architectures (such as monolithic frameworks, SQLite, or managed VPS) prevents early engineering bankruptcy.
Evaluating true compute and egress costs reveals that high cloud bills can often be slashed by 80%+ through judicious re-platforming.
Code you write at 2 AM as a solo engineer must be easy to debug six months later without complex microservice dependencies.
Community Debate Themes:
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
CATEGORY Frugal Engineering →
HISTORICAL ERA Pandemic Bubble (2020–2022)
COMMUNITY SCORE ★ 90 points
TOTAL DISCUSSIONS 63 comments
So it was a lot of effort, and now they're inviting Big-G and Cloudflare to contact them to possibly achieve a paltry 30%-ish speed increase for certain scenarios? Or are they inviting app devs who want faster video uploads to reach out? What is the actual use case where the sometimes 30% improvement matters and actually moves the needle?!
Why hasn't Tachyon been working with their prospective customers and warming them up the whole time, or at least working the social and investor nets and reaching out proactively already?
This strategy is kind of like being a dweeb at a poorly lit school dance and hoping the most popular girl at the dance somehow notices you're wearing shoes that let you float a centimeter in the air. Cool trick, bud.
Presumably it's not a $10/mo service contract. Is this really an effective strategy when building and selling to enterprise these days? To me it sounds like a risky and hard way to make less money than what is possible using tried and true product development strategies. To be fair, I have also made this mistake before. It was embarrassing enough as a solo-founder, and seems less forgivable with larger founding group sizes, because it means more folks agreed to support and follow such a sub-optimal harebrained scheme :)
You all sound like very capable software engineers, and I know it's both fun and satisfying to build and make The Thing.
Good luck, sincerely.
p.s. You may also consider pursuing some of the medium sized targets like Backblaze, Rackspace, Larry Ellisons Oracle OCI, or Microshaft Azure.
(sorry, I couldn't resist having some fun at the end, though the suggestion is serious!)
This article was interesting and also frustrating to read.
1. There are very few numbers. In particular, improvement in performance under various circumstances is _not_ given! If you dig around you can find their transfer time application [1], but there is no discussion on that page.
2. The basis for the improvement is not spelled out. (References are given, but you have to know the field - "acronyms only".) If I understand correctly, their contribution is the improved measures of congestion used. Their landing page just touts "don't use TCP"... which sounds like Step 0 of a very long process.
I admit, the title is basically accurate: "how to build" not "the performance of".
tl;dr: Start with existing work, simulate and improve incrementally.
I don't know anything about the field, but this article didn't lead me to understand any better. I'd love to know the real numbers they observed, which approaches didn't pan out, are they effectively using an error correcting code?
Anyway, it's certainly not an academic paper - just an advertisement.
[1] https://www.trytachyon.com/file-transfer-calculator
The article has essentially no technical information on how the protocol works, besides the unlabeled flow rate graphs which show TCP rate popping up and down and theirs more or less constant. Their rate appears much less stable than FASP's. OTOH, FASP is extremely expensive.
I have updated the Wikipedia page on the FASP protocol they compete with, to provide more detail on how it works. Theirs might work similarly.
Curiously, FASP usage only really took off after the product got a comprehensive GUI file management app, with scheduling and a speed control you could drag up and down. Being able to transfer files overseas several times as fast as the competition was not enough.
FASP was on HN a couple of years ago, https://news.ycombinator.com/item?id=21898072
Always good to see more in this space! Long fat networks (LFNs or "elephants") are everywhere, especially once you start moving data between continents.
I've had success personally with UFTP, but you explicitly set the transmit rate. Don't forget to enable encryption/authentication if you want the downloads to be verified! You'll get silent UDP corruption otherwise: http://uftp-multicast.sourceforge.net/
IMVHO the main issue in file transfer today is that in 2022 most people still do not have a public ip (like an IPv6 global ones) so most people still have NAT traversal issues and need to relay on third parties or not-so-performant more or less distributed networks...
The second main issue is that most do not own a personal domain name with a subdomain per personal host (like {desktop,craphone,laptop}.mydomain.tld etc).
Those two issues are so big IMVHO that push all others aside...