MinIO: A Bare Metal Drop-In for AWS S3

Is HDFS nice? I did a lot of research before settling on Ceph for our in-house storage cluster, and I don't remember even considering HDFS and I don't really know why. Ceph also is a drop-in for S3 for bare metal clusters.

I've been running Ceph for about a year now, and the start up was a bit rough. We are actually on second hand hard drives, that had a lot of bad apples, and the failures weren't actually very transparent to deal with, which was a bit of a disappointment. Maybe my expectations were too high, but I was hoping it would just sort of fix itself (i.e. down the relevant drive, send me a notification, and ensure continuity). I feel I had to learn way too much about Ceph to be able to operate it properly. Besides that the performance is also not stellar, it apparently scales with CPU frequency, which is a bit astonishing to me, but I've never designed a distributed filesystem so who am I to judge.

I was looking for something that would scale with the company. Now we've got 70 drives, maybe next year 100 and the next year 200. Now all our drives are 4TB, but I'd like to switch them out for 14TB or 18TB drives as we go along. We're not in a position to just drop 100k on a batch of shiny state of the art machines at once. Many filesystems assume the number of drives in your cluster never changes, it's crazy.

MinIO is something I'll look into. And, as another example to the article's, it might also come in handy for some data needs for factories with imperfect Internet reliability (e.g., when the main submarine cable between the factory and AWS Singapore gets severed :).

This first example from the article sounds very valid, but is still personally funny to me, because it's related to the first use I made of S3, but in the opposite direction (due to different technical needs than the article's):

If an Airline has a fleet of 100 aircraft that produce 200 TB of telemetry each week and has poor network connectivity at its hub.

Years ago, I helped move a tricky Linux-filesystem-based storage scheme for flight data recorder captures to S3. I ended up making a bespoke layer for local-caching, encryption (integrating a proven method and implementation, not rolling own, of course), compression, and legacy backward-compatibility.

That was a pretty interesting challenge of architecture, operations, and systems-y software development. And the occasional non-mainstream technical requirements we encounter are why projects like this MinIO are interesting.

At work, we use MinIO as a replacement of the S3 Api on our CI servers since we dont want to call production APIs for integration testing.

One of the challenges we had was "pre-filling" the MinIO server with testdata. Some tests require reading a testfile from our mocked S3 API. We wanted to have those testfiles instantly available at MinIO startup, but couldn not get that to work with docker volume mounts, MinIO simply would not recognize the files and serve a 404.

Has anyone got that working or a proposal for an alternative solution? We resorted to uploading the files via the application on startup (if it is in "testing mode"), but that does feel like a dirty hack.

I don't know when this was written, but MinIO does not have a great story (or really any story) around horizontal scalability. Yes, you can set it up in "distributed mode", but that is completely fixed at setup time and requires a certain number of nodes right from the beginning.

For anyone who wants HA and horizontal elastic scalability, checkout SeaweedFS instead, it is based on the Facebook "Haystack" paper: https://github.com/chrislusf/seaweedfs

Amazon S3 on Outposts (more info at https://aws.amazon.com/s3/outposts/ ) runs on-premises, offers durable storage, high throughput, the S3 API, currently scales to 380 TB, and doesn't require you to watch for and deal with failing disks.

I believe that it addresses many of the OP's reasons for deciding against S3.