The new browser-based puppeteer REPL, and how we built it
In this technical retrospective (2021), @mrskitch shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
Nice! We built something like this 3 years ago while I was still at Intoli. Instead of Puppeteer we used our own Web Extensions API based framework called Remote Browser [1], the core of which was written by my cofounder.
The tour is still up at [2]. The servers that actually run the Remote Browser have since gone down, but interestingly you can still run the tour. That's because if you don't change the code in the REPL window, you get cached results (except step 7/7 which scrapes Hacker News and won't work). To get those results, we built a little tour "recorder" that would be run on every release. If I remember correctly, we allowed some dynamic ES6 imports through a custom Babel compiler for the code that's input, which also allows first level async stuff, which still works :)
[1]: https://github.com/intoli/remote-browser [2]: https://intoli.com/tour/
I've used Puppeteer extensively and its the most unstable tool that I've come across in recent years. Its full of race conditions and the waitForSomething functions will sometimes work, sometimes not.
It's usable, but it's very aggravating and uncomfortable to use.
Could you elaborate a little on ‘puppeteer in the browser’?
Is puppeteer running on a webserver then the repl connecting to it? Or is puppeteer completely contained within each users browser?
Hey, I have a question. On your home page it says that the usage-based plan doesn't offer the Live Debug, how does one debug scripts on that plan?
Does this run Puppeteer on Lambda/Docker? One thing that I’m struggling to do is run Chromium on AWS Lambda as the new container image.
Hey, Joel here, mostly responsible for this tool. Happy to answer questions — one thing not well covered is getting puppeteer to run in the browser, especially a webworker. Can talk more about it if there’s interest!