Thread by @lizthegrey

Final talk I'll be getting to at #VelocityConf before I dash to Toronto: @IanColdwater on improving container security on k8s.

@IanColdwater She focuses on hardening her employer's cloud container infrastructure, including doing work on k8s.

She also was an ethical hacker before she went into DevOps and DevSecOps. #VelocityConf

She travels around doing competitive hacking with CTFs. It's important to think like an attacker rather than assuming good intents and nice user personas that use our features in the way the devs intended things to be used. #VelocityConf

"How often do we have evil user personas? Not all users are good." --@IanColdwater

Having people who can think like attackers involved in the process of securing your software makes your software more secure. #VelocityConf

As an attacker, they want to pwn you. And they are not going to tell you the myriad of ways they can, that's for the resources slide.

How did we get here? In the beginning, there were VMs. and then containers. [ed: messed up their pronouns, sorry, fixed from here] #VelocityConf

Containers bring benefits, but add complexity. So we need orchestration systems to help us automate management and deployment. Like Kubernetes!

It's very active and being changed very fast. 90 day dev cadence. k8s used to be wide open, but is getting better. #VelocityConf

It used to not support multitenancy and gave everything root by default prior to v1.5.

The attack surface is going down over time. However, often what happens in prod stays in prod so there are older k8s clusters floating around. Hopefully not on your servers. #VelocityConf

How to attack and defend k8s depends upon how it's configured. It's hard to give hardening advice on something so generic.

Most people find k8s to be hard. Because it's so complicated, it's defaulted to being wide open first with security as afterthought. #VelocityConf

Security is the top challenge for k8s users. Real consequences: Tesla had their clusters compromised and made to run cryptominers. Ditto Weight Watchers. Shopify too (via bug bounty).

All wide open with no password. #VelocityConf

Even security professionals aren't immune - one CTF had the organizers' own k8s compromised and the participants stole flags that way.

@IanColdwater's advice is that you can't stay still in k8s land to stay secure; keep on top of your updates. #VelocityConf

k8s is only as secure as the OS and hardware it runs on. It shares a kernel with your host, so... maybe don't be vulnerable to Spectre, Meltdown, etc. #VelocityConf

Start with your threat model - what are you trying to protect, who are you protecting from, and what are the capabilities on each side.

And then they make a hilarious joke. "Your threat model is not my threat model, and that's okay." --@IanColdwater #VelocityConf

Nation-states are very different from random script kiddies. You know your systems better than everyone else does, so figure out the answers to these questions and make strategic decisions.

Don't spend money protecting against 0-days while you have weak passwords. #VelocityConf

Hackers don't give a shit if your system is internal, or about your business case, or about the fact that a system is unmaintained because someone left and nobody replaced them.

The consequences are real. #VelocityConf

They may be motivated by different things -- but they tend to go through these steps:
(1) Discovery
(2) Enumeration (including of versions, exploitable holes)
(3) Getting in
(4) Privilege escalation and lateral movement (incl downloading from internet)
(5) & repeat. #VelocityConf

"If a container is privileged and I have root access to that container, I also have access to the host and can do whatever I like with it. Your job as a defender is to make sure I can't get that far." --@IanColdwater #VelocityConf

and then you can run cryptominers on someone's cloud account.

Do you know the full graph of what's running what and has access to what? Often, the answer is that you don't have that inventory. #VelocityConf

Then you can figure out how to lock it down -- it shouldn't be possible to communicate between things that don't need to talk to each other.

Remember the principle of least privilege. #VelocityConf

Three threat models @IanColdwater is encouraging us to consider: external attackers, application compromise, and credential compromise.

They suggest that you may have various open ports by default, so you can definitely be found. ex: Shodan for finding open ports #VelocityConf

There are millions of exposed SSH keys and thousands of open etcd ports out there. It is illegal to follow that, but many people don't care. And it's reindexing the whole web every 24 hours. You don't have to be exposed that long. #VelocityConf

If you do have externally visible ports, make sure that they require authentication. Don't just give out insecure access to the kubelet. That's bad and you don't want that to happen to your clusters. #VelocityConf

Limit ssh access to k8s nodes. Use an SSH bastion if you do need to leave ssh on.

Onwards to credential compromise. Don't fall for social engineering, including pasting in commands from Stack Overflow without seeing what they do... (oops) #VelocityConf

"kubectl create -f is the new curl | /bin/bash." --@IanColdwater

And developers are more overconfident and more likely to click phishes than sales or marketing. And we have more permissions than most. #VelocityConf

Likewise, don't run: docker build some-nice-url.com/foo.git

Keep your credentials encrypted and don't just leave them lying around. Don't mishandle secrets or accidentally post them to Github. Limit access. #VelocityConf

Passwords in environment variables are easy to get; don't base64 to obfuscate, obfuscation is not encryption. Use Hashicorp Vault or an equivalent [ed: see @sethvargo's talk].

Encrypt everything. #VelocityConf

Container compromise: backdoored images have been downloaded millions of times from Docker Hub.

Do you understand what's in your containers and what they do? And are your own apps written as securely as possible?

Check your containers for vulnerabilities. #VelocityConf

Don't just leave vulnerabilities lying around, patch or mitigate as they're a ticking time bomb.

If one container in a cluster is compromised, it's possible to move across entire cluster. Kernel attacks & container breakouts make your attack surface much larger. #VelocityConf

kubelet is by default unauthenticated and can move across the cluster. So set a password. and secure your API server and etcd. #VelocityConf

etcd only recently started encrypting data at rest. Upgrade your k8s setup, your etcd, and so on and so forth.

Firewall your etcd off from everything else. Remember, only allow communication between what needs to communicate. #VelocityConf

Privilege escalation: set allowPrivilegeEscalation: false, and run your containers as users, not as root.

Set pod security policies.

Segment movement with admission controllers. Firewall things. #VelocityConf

Here are some controls to consider: network policies, admission controllers, RBAC, pod security policies, resource quotas, and logging and monitoring. Each of these is an hour long talk. So research each of them! #VelocityConf

Limit visibility, and make sure that attackers can't find additional credentials to use.

For defense in depth, repeat all of your measures and defenses at each point. Limit blast radius of compromise. Random script kiddies will give up if it's hard. #VelocityConf

"If you aren't the easiest target or lowest hanging fruit, then they'll leave and pick another target." --@IanColdwater #VelocityConf

Principle of least privilege. Again. Limit access, traffic, communication. Really.

In general: update, patch, update. Make your local defaults are secure out of the box. Mind your secrets.

Log and monitor from outside the compromise domain to avoid tampering. #VelocityConf

Much of this is standard security advice...

Don't leave admin/admin or open s3 permissions. Avoid opportunistic low hanging fruit attacks.

"Attackers are lazy like the rest of us and use automation." --@IanColdwater #VelocityConf

And they believe in you! With rainbows! It'll just take some work. So go read some resources. You can do this! [fin] #VelocityConf