Show HN: How I built a trading signal by scraping Nasdaq for short interest

While I'd love to take all the credit (blame?), the reformed academic in my feels compelled to admit that the idea to look for predictive value in stock loan data is not original to me. The finance literature has some fascinating articles on this dating back as far as the late 80s (look for Desai 2002, J of Finance, Asquith 2005 J Fin Econ, or most recently http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1570451).

The intuition behind this signal as a market inefficiency, or 'anomaly' is that the market sees short sellers as informed investors, the so called 'smart money', and there is a herding effect to follow their trades which generates abnormal returns. The same logic can be applied to disclosed insider trades or institutional holdings filings made public via the SEC's EDGAR database.

Fawce's slick implementation of a 'Days to Cover' signal is a great way to highlight the power of aiming new tools like Quantopian at freely available public data stores (which exist expressly to increase market transparency). And sure, it doesn't go the whole way for you on execution details like borrow costs, liquidity etc. but those aspects tend to be unique to each trader.

Back testing is a real bitch. I've been building my own app for back testing recently, my specific interest being how published insider buys (SEC Form 4 transactions) affect the prices of stocks in the short near and long term. You can get dividend data and stock splits easily enough from some public feeds. But where do you get a database of ticker changes, bankruptcy events, and spin-offs, especially on the OTC markets? You can't unless you're willing to shell out a lot of money. Back testing properly is probably out of the cost range of the individual investor.

Some examples:

* Lehhman's ticker changes on the way down

* GM going bankrupt and then coming back from the dead!

* Skye International used to trade under SKYY (at 0.35c/share), but now SKYY tracks a cloud SaaS ETF 20.60/share). Think you got a big win using that strategy that including buying SKYY? Think again!

My comment in the last thread opened with a post from this source:

Past performance does not guarantee future results" is still the operative principle here. Data-mining discovers patterns, but it doesn't lead to deep insight into causes, and markets are perturbed by many events that you don't put into your training algorithm. "The market can remain irrational longer than you can remain solvent" is still important investment advice.

You can never build a trading signal just by scraping historical data, unless you like losing your shirt.

Can you tell I'm reading Antifragile: Things That Gain from Disorder just now? I'm very sensitive to errors in statistical thinking today.

I have for two years now been playing around with Algorithmic trading as a hobby and I am amazed by people who think wave riders or simple mathmatical transforms will get them profits in the market. I have found that the best method is still a good mix of modeling and trader input. I don't think a model exists that you can just turn on and have it print you money. So attempts like this to make one of those really are a waste of time. Your systems should be tuned to listen to you and then take what input you have and do what you cannot ( make decision in sub-second windows )