Thread by @lizthegrey

I'm excited to see @alicegoldfuss's talk on container operations for the first time! #VelocityConf

@alicegoldfuss She's telling us about the idyllic picture painted by advertisements (for instance, for cars) with the fancy 0-60 numbers... vs. the tiny small words "laboratory setting" or "test track". #VelocityConf

That's how @alicegoldfuss feels about container talks. Presenter comes out and says "BAM, I will revolutionize your infrastructure? What's that? Run more than one container? No problem! [with shipping container images overlaid on cars]" #VelocityConf

"Is this why sysadmins are so mad?" @alicegoldfuss wants to see more "Docker WTFs" and fewer "Docker 101s". We need to be able to make better decisions about containers. And she calls herself "Edward DevOpsHands" #VelocityConf

She ran containers in prod since 2015 since Docker 0.6, paving the road behind them; and now she's on the k8s platform team at GitHub.

So to recap, what *is* a container? #VelocityConf

We have a host, which is a box, and an OS, which is a box, and a container daemon, which is a box, and on top of that you have containers WHICH ARE SMALLER BOXES.

But it's COMPLETELY LIES. -- @alicegoldfuss #VelocityConf

But to your kernel, they're all just running processes. python processes, ruby processes, and then the dockerd and kubelet. They're not hidden in boxes, they're processes like any others, born from tarballs, anchored to namespaces, controlled by cgroups. #VelocityConf

Containers are just a bunch of native linux features stapled together. There is no such singular concept in the kernel.

Namespaces determine what a process can see. cgroups determine what a process can use.

"I thought it was whales..." #VelocityConf

Four lessons to give us a better sense of whether containers are right for us:
(1) Containers have strengths when used for the right purposes -- for scaling stateless applications and making them portable and more easily revised/deployed. Data in, data out. #VelocityConf

(2) Disaster recovery. Avoids that developer's laptop that runs a critical piece of our infrastructure every day at 3am. We deploy our apps in the same way to the same places, making environments reproducible.

We can by default redeploy/restart even others' apps. #VelocityConf

(3) Development environments. We can run dev built applications in the same way as our production applications, and tear down cleanly afterwards.

And we don't have to repro on our laptop. #VelocityConf

Lesson number 2: containers have weaknesses:

Stateful apps [ed: like hairspray]. Databases, for instance. If you're not having billions of requests per second, you probably don't need to containerize databases.

You *can* get stability/recovery/provisioning BUT... #VelocityConf

It's cheaper to do that with a cloud provider doing that work with the right tooling for you! Even if it's more money to the cloud provider, it's cheaper than your development time to do things right.

Are you *sure* you want to do that in-house? #VelocityConf

There are some ways to handle this, such as using network-mounted storage backing containerized databases, but then you get network bound.

Or you can save the data locally, but then you get no benefits except sounding good on resume. #VelocityConf

"Seriously, consider going to the cloud for your databases. It will save you time and hardship. You didn't really want that HN blogpost anyways." --@alicegoldfuss #VelocityConf

Lesson 3: It's never "just" containers. Lots of decisions to make. How will you build your tarballs? You don't have to use docker in prod. Orchestration matters too. Managing clusters, SLOs, and automating your networking and access controls. #VelocityConf

Here are all your choices for k8s networking: calico, flannel, cilium, linkerd, istio, envoy...

and how many of them will survive the next few years? and will your engineers be able to submit bugs and patches? #VelocityConf

but now you have to integrate with your infrastructure. for instance, deployment. Do you need to build new integration and tooling? which will you fix first if they both break?

How are you going to monitor? provision? and *debug* the extra abstraction. #VelocityConf

You need processes for gradually rolling out the new platform to users; it'll take a year to get a container platform off the ground with just one service running on it.

and you'll be coping with a hybrid/chimera for a while. set your expectations accordingly. #VelocityConf

Lesson number 4 (final lesson) -- even if you're going to disregard the rest of Alice's advice and do "Kubernetes in the App Store.":

Containers need headcount. hypothetical "Why not just give it to ops?" #VelocityConf

Deploying containers is like building a new datacenter with a completely new software stack requiring a completely new team, not just adding one more jenkins to your pile of jenkins. #VelocityConf

Make sure you hire the right people -- ops specialists who know the skeletons, deployment folks who understand the landscape, tooling/glue writers, and monitoring/metrics specialists. Kernel engineers who can debug interactions. Networking specialists. #VelocityConf

Get people who understand infosec and build it in from the start. And developer advocates or internal sponsors who can get people to adopt. #VelocityConf

and you need, last, but not least, a product & project manager. When something fails, you need the breathing room and heatshielding.

You can pull these from other teams, but BACKFILL THEM or @alicegoldfuss will get mad at you for trying to be sneaky. She sees all. #VelocityConf

You need a team, not necessarily one person for each role, but 6-8 people in total ideally, and 4+ people at minimum. Oncall of less than 4 people is untenable and will cause your platform to plummet into the sea.

And give them space to experiment and succeed. #VelocityConf

[ed: this talk is so good and I wish I could also capture all of @alicegoldfuss's quips including about the LKML, but instead I just got the best concrete points]

Recap: benefits/drawbacks of containers, don't use them for stateful, use cloud providers instead #VelocityConf

And invest sufficiently in the team if you're building a container platform.

There's no one right answer to "should I run containers in prod?" If you have stateless services, a large platform, and lots of resources, go for it. #VelocityConf

If your databases have nicknames, you have a monolith, and a small team with no org support... don't. Unless you want to spite Alice.

Don't do it for the blog post. "it would be rad..." DON'T DO IT, yells @alicegoldfuss. #VelocityConf

"Containers are just a tool", says @alicegoldfuss. "They're a little tricky, but if you have a good team, you'll get there in the end." --@alicegoldfuss [fin and raucous applause] #VelocityConf