Hetzner Cloud – Data loss incident (redux)
Dear Customer
We regret to inform you that there has been a loss of snapshots from your following Storage Box that were created in the period from 15 May 2023 11:00 CEST and 12 July 2023 13:00 CEST. Snapshots created before or after this period and the contents of your Storage Box are not affected!
On 25 July 2023, we discovered that due to a software error, snapshots created for Storage Boxes were partially deleted. This affected both the snapshots created manually (via web interface or Robot Web Service) and those created automatically as part of snapshot plans. Unfortunately, we were unable to restore the snapshots deleted during the above-mentioned period.
The protection of your data is important to us. We take this matter very seriously and have already rectified the problem and immediately reported the incident to the Bavarian State Office for Data Protection Supervision.
We apologise for this inconvenience and ask for your understanding.
If you have any questions, please do not hesitate to contact us. Please use the contact form in your administration interface under the menu item “Support”.
Kind regards
Hetzner Online ---- * Previous incident, 15 months ago: https://news.ycombinator.com/item?id=31015840
In this technical retrospective (2023), @asgrdz shares practical architectural lessons learned from running, scaling, or migrating production software under lean constraints.
HN engineers debated the tradeoffs between cloud convenience and raw server economics, database indexing strategies, and resisting premature architectural complexity.
Data loss is terrifying. I am building a tool to help in data transfers.
I always assume my data is more secure in S3 than my on prem NAS. I also assume that 0.00001% accounts for anything from one file to an entire data center worth of bytes.
This is bad, but storage boxes are already a backup target and the content of the backup target (actual files stored in in box) are not affected... "only" snapshots.