How we built the largest database of police use of force in N.J. history
In this technical retrospective (2018), @danso shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
A database that lists the officers responsible for force by name, frequency and race. Resulting in neat graphs that strongly suggests that police violence is caused by two factors: individuals and/or race.
Police brutality is a serious problem and data like this should be available, but several things here are (maybe intentionally) provocative:
1. Listing the officers by name is a kind of online vigilantism that ultimately might threaten their personal safety. It doesn't sit well with me at least. This data should be anonymized.
2. The prominence of graphs that conflict race of officer with race distribution of the community. Race should really be stripped from this kind statistic. Race is a social construct based on superficial attributes that does not carry any explanatory meaning. It mostly serves to underpin racist prejudice, which in turn serves to blur out the real reasons for a problem.
Hello, I've never commented on Hacker News before so please forgive me if I am breaking any rules, but I worked on this project. Let me know if you have any questions about our process. - Erin Petenko, data reporter for NJ.com
Proof: https://twitter.com/EPetenko/status/1069664096042270720
See also http://killedbypolice.net/
It attempts to list all police killings in the US. Unfortunately it looks like they stopped in the middle of this year due to lack of support, but there is five years of interesting past data still there.
If you like this sort of reporting, please make sure you are a paying subscriber for your local newspaper. We must support this sort of essential journalism with our pocketbooks.
True journalism. ACLU should do a nationwide one, they'd raise the money in a hearbeat.
If this topic interests you, worth pointing out some similar projects, such as the Citizens Police Data Project of Chicago, which currently has 247K police complaints (and puts the code/data on Github) [0]. Note that it's about complaints -- not about all use of force, as in the NJ database. It also covers a single agency -- the Chicago PD. Ostensibly, the Chicago PD has put in the work to create a centralized database, making it relatively straightforward to get the bulk data once the records request/lawsuit succeeds. It's a much different situation when trying to corral the records from multiple agencies, all of which have their own IT systems and practices.
VICE did a mass records request for shooting incidents (again, a subset of use of force) from the 50 largest police departments. It took them 9 months to collect the data [1]
[0] https://invisible.institute/police-data/
[1] https://news.vice.com/en_us/article/a3jjpa/nonfatal-police-shootings-data