Ask HN: Get feedback on my bootstrapped web scraping as a service project?
Hi HN,
I worked on https://scrape.it for the past 5 months or so. I would like some feedback please.
Thank you.
Published in 2014, this chronicle by @notastartup details the mechanics of building a profitable, self-funded business without venture backing.
The community focused on B2B customer acquisition channels, cold email vs SEO distribution, and balancing full-time employment with side-project development.
The UI for defining the scraper targeting rules is an A+ in my book. I've used a number of visual tools for this over the years but most try to do to much and the usefulness suffers. Yours doesn't.
Your price however feels off by about a factor of somewhere between 4 and 8. I haven't really got your system usage data to fully analyse but it feels too high a price. If your using full desktops and chrome browsers and the overhead is higher than I'm expecting based on past experience then perhaps the pricing is off by a factor of only 2
That also doesn't even account for the ability to take the scrape rules, reformat them to run with a headless scraper like scrapy when the website is suitable and is able to be scraped by scrapy, offering a reduced cost scraping option powered this way, throttled back to "human speed" is probably best priced around $1 or $2 a month per crawl thread.
I'm not saying the chrome browser powered crawling part couldn't be so inefficient you need to charge $20 to make 1 crawler pay for itself, but if that's the case, it's a growth limiter you should consider ways to optimise.
Right now your UI is about 10 times better than the open source Portia UI from scrapinghub, but your also 10x the cost of operating a Portia crawler. It's just something to keep in mind.
TL;DR? Nice product, but a bit expensive.
It looks great, very good job.
One question you should be prepared to answer is how you compare to alternatives:
- Services like Kimonolabs and Import.io
- Building a scraper with Scrapy
- Using Scrapehub
From what I understand the price point of the above is lower (some are free) while still offering the same features. What can you offer that they don't?
Needs more details, a couple of examples that instantly spring to mind:
Do you spread requests out across multiple IP's to avoid bans / rate limiting?
Do you run javascript on the pages?
It looks quite easy to use from your video.
Which cloud provider do you use for your backend crawl, and do you run the scraper from a single node or from multiple nodes?
Page itself looks nice, but why you've decided to have a custom scroll? I think it would be so much better without it. Will try to look into app itself later.
clickable link: https://scrape.it since you can't click the one above.