Bare Metal K8s Clustering at Scale

To everyone asking "why would you ever do this", ask them, or look for videos of the talk/etc., but don't assume the decision was poorly thought through. That seems prejudicial at best (and I say that as someone that regularly pushes back on web software folks I work with trying to build huge-company-scale orchestration layers when they don't need them).

I saw these guys talk at QCon. It was a fascinating talk, and an excellent example of SRE adaptability and nonstandard, uncommon innovation given unusual constraints.

Not speaking for them, just from my memories of the talk and the following Q/A, but their reasons for this stack were primarily:

- They couldn't run in the cloud because connectivity to their sites is often terrible.

- They mostly ran IoT stuff from the k8s clusters--automated kitchen equipment like fryers and fridges, order tracking/status screens, building control systems, and metrics aggregation so they can see how businesses are doing.

- Because of the bad connectivity, a "fetch"/"push" (from the k8s clusters at the edge) model was needed for deployments/logging/administration/getting business data back up to the cloud.

- They explicitly did not process payments.

- k8s was used primarily for ease of deployment and providing a base layer of clustered reliability for pretty simple services. Since the boxes in the cluster were running in often-unventillated racks/closets full of junk in random restaurants, having that base layer was very important to them. Other solutions were evaluated and they chose k8s after consideration.

- Unlike typical IoT/automation setups here, they wanted to be able to experiment, monitor, and deploy software without the traditional industrial control practice of "take shit down, flash your controller (call a tech if you don't understand that), spin it up, and if it breaks you're down until we ship a new control unit or you manually fail over to a backup".

- However, they didn't want to fall into the IoT over-the-air update security pitfalls (it would really suck if someone hacked your fridge's temperature control system and gave a week's worth of customers salmonella). As a result they spent a ton of time making very good (and simultaneously very simple) deployment/update authorization and tracking tools. They chose the "pull" model and keying/security layers explicitly to avoid having to think about tons of open remote-access vectors and/or site hijacking.

- The k8s tooling (and some of their own) allowed easy, remote rollbacks to "default/clean state" in case something went wrong, which was critical given that downtime might compromise a restaurant and having a "reset button" automated in was important for ease-of-use by nontechnical, overworked site managers.

- The clustering allowed individual nodes to fail (which they will, because unreliable environments), and people to manually yank ones with confidence.

- While, as some commenters pointed out, the leader (re)election system chosen might be unacceptably slow/randomized for, say, a cloud database, it is perfectly sufficient for failing over a control system in a restaurant. A few seconds of delay on an order tracking screen, or a system reboot/state-loss of in-flight orders is vastly preferable than some split-brain situation making the restaurant accidentally cook 1.25x the correct number of sandwiches for hours, to go to waste.

It's important to understand their use case: they needed to basically ship something with the reliability equivalent of a Comcast modem (totally nontechnical users unboxed it, plugged it in, turned it on, and their restaurant worked) to extremely poorly-provisioned spaces (not server rooms) in very unreliable network environments. For them, k8s is an (important) implementation detail. It lets them get close to the substrate-level reliability of a much more expensive industrial control system in their sites (with clustering/reset/making sure everything is containerized and therefore less likely to totally break a host), while also letting them deploy/iterate/manage/experiment with much more confidence and flexibility than such systems provides.

I think this is a great story of using new tools for a novel (or at least unusual) purpose, and getting big benefits from it.

Brian, Caleb: great talk, great writeup. Sorry HN is . . . being HN. Keep at it.

Edit: QCon talk summary is here: https://www.infoq.com/news/2017/07/iot-edge-compute-chick-fil-a. If you have any employees/friends that went, they should have access to the video. It may be made public at some point, too.

This type of deployment is a perfect fit for NixOS. Immutable deployments with zero configuration drift, easy rollback and options to both push and pull the system configuration updates. It's also easy to customize the system to the hardware unlike CoreOS or Rancher while providing pre-built binaries of all the dependencies.

Setting up a single-node kubernetes is basically adding one line to the system config:

``` services.kubernetes.roles = ["master" "node"]; ```

The intetesting bit they miss detail on is why they are running k8 at the edge in restaurants.

The only reason i can think of that is they get to push point of sale software out by using K8s from some central system. I cant think of a worse use/abuse of k8 as a software updafe system if that's what they are doing.

The other reason is they distributed their compute and resturants pay the power bill but that sounds just as silly.

Curious to know why you would use k8s at the edge

This is great. Does anyone have guides on how to do the cluster creation bootstrap on public clouds where you don't get a known DNS name ahead of time and master nodes may come and go? Ie I want to bake an AMI and create an ASG so that we can turn it on and it will self cluster, create certs, etc and can add and remove nodes at the whim of the ASG.

Wish the article had more context for readers.

These days it feels like everybody needs to throw in Kubernetes at everything introducing complexity for the sake of being cool.

I guess those of us that likes to run non-distributed software for small scale applications are the new grumpy grey beards....