Thread by @lizthegrey

First up at #VelocityConf this morning: @tammybutow on chaos engineering to increase resilience!

@tammybutow She's been doing chaos engineering since 2009. But she wants to encourage people to do Chaos Days the same way that they do Hack Days - to build resilience into teams and products. #VelocityConf

Chaos isn't about going rogue and setting everything on fire; instead, it's much more scientific and collaborative.

You need to do thoughtful, planned experiments to reveal weaknesses. #VelocityConf

Make sure you're doing things safely and that you think they have a chance of success.

Start small, using test users and staging environments. Be thoughtful about the blast radius. #VelocityConf

What chaos experiments can you run?
(1) Test infrastructure failures -- host and container level failures. Simulate cpu/io/disk bottlenecks and failures. Also test your dependencies, including your cloud provider. #VelocityConf

(1.5) simulate network failures, inviting network engineers to help you out.
(2) Application failures. Inject faults into the application code and see if it causes user-visible failures.

Full-stack chaos: inject at all layers of stack. #VelocityConf

Think about your business continuity planning. But Chaos Engineering is about doing the testing at smaller scale, more often, rather than doing BCP once a year. #VelocityConf

Think as well about Game Days (e.g. simulating or actually pulling cables from machines in datacenters), Capture the Flag, and Hack Days/Hack Weeks.

Why should we care? Think about big outages in the press. Reduce frequency of big outages. #VelocityConf

She set out to reduce outage frequency by 20% but instead had a 10x reduction in outage frequency in first 3 months by simulating and practicing until she got the big wins.

0 Sev0 incidents for 12 months after the 3 month period. #VelocityConf

How do you do this? Actually think through your dependencies and work together with them to generate action items for improving your systems.

We should think about durability and data inconsistency (c.f. @kilobitten's talk) #VelocityConf

Can you actually detect corruption? How do you know?

And are you keeping your oncall engineers fresh if you have a very low oncall load? #VelocityConf

Gain understanding of your system's weaknesses, and strengthen products pre-launch.

And involve every facet of your business in sourcing ideas for testing resiliency. #VelocityConf

You can test out new technology (e.g. k8s) and make sure that it behaves in the way you anticipate. If not, you have bugs to fix!

You can also learn how your failures impact customers. For instance, Netflix discovered that... #VelocityConf

if they gracefully degraded by not showing the top banner image, rather than showing a blank image during failure, the UI looks much better. #VelocityConf

And this can generate feature requests for cloud providers to support better failovers between regions.

Make sure your team gains the skills they need through Chaos Days. It'll take 90 days to plan your first Chaos Day. #VelocityConf

What preparation do you need to do?
(1) Know your top 5 critical systems,
(2) Have monitoring and alerting [ed: including SLOs!]
(3) know the cost of downtime
(4) Know what new things you want to battle-test.

#VelocityConf

Make sure you have the right access to perform your experiments, and that the set of people who can do it (and watch it!) is the right size.

Where are you coordinating your team? What happens if you cause a real problem? Test your physical contingencies too. #VelocityConf

It takes a team to pull off a Chaos Day -- have your VPeng or CTO around, an administrative business partner, engineering managers/directors, engineers, and new engineers and interns!

If the newest person can do it, so can everyone else. #VelocityConf

Go forth and run your Chaos Day, and let @tammybutow know how your survival story went! Let's share and learn as an industry! [fin] #VelocityConf