Next up in track 2: @_a12k on Bootstrapping SRE and starting culture change. #SREcon
USDS, working on Veterans Administration. Combining organic devops teams from multiple applications into one single shared SRE team. #SREcon
Expanded from a few initial national priorities like healthcare.gov to hundreds of applications across a dozen plus agencies. #SREcon
Goals: streamlining appeals processes, managing/discovering benefits online, healthcare applications. No more fillable PDFs that barely work. #SREcon
327,923+ veterans have applied for healthcare online since July 2016. UX studies and working with real customer veterans. #SREcon
Service level objectives. >99.9% availability based on continuous reachability monitoring. 2 minutes a week or less of downtime. [ed: what is the SLI though?] #SREcon
Several different organic teams: caseflow, vets.gov. what if we standardized on same tech stack and combined oncalls? #SREcon
Have a group of people who specialize in tools and best practices. Efficiency gains. Internal goal of changing relationships between developers and ops. Evangelize process to broader VA. #SREcon
The VA is the ultimate large enterprise environment. --@_a12k#SREcon
Environment slide: jenkins, terraform, jenkins, circleci, ansible, k8s, all running on AWS govcloud. Mixture of tech. #SREcon
Socialize SRE concepts through VA organization and have teams rotate into new SRE org to learn. #SREcon
4 Billion (B) dollars of spending on VA? USGov? IT. Top 10 enterprises in the world if appropriations = revenue. #SREcon
[ed: this is looking very familiar -- I give similar advice to start small with one team *then* evangelize rather than smearing SRE mandate across many teams.] #SREcon
Define scope, get the SRE team to work for the initial scope, choose best practices, do engineering/culture work to conform to the chosen practices. #SREcon
Next step: package best practices so they can be consumed by other VA teams. Make them generic and usable as an engagement model. 132 on-premises datacenters in the US to be migrated to public cloud. #SREcon
Inclusion is important. Have people celebrate each others' victories. It empowers people to speak up and share ideas. --@_a12k#SREcon
Cultivate a culture where it's safe to ask questions. Asking questions lets you discover why we do things, and figure out whether there are different choices we could make. #SREcon
Respect different peoples' abilities and skillsets that they bring to the table. USDS is staffed by volunteers from all walks. #SREcon
Implement a *blameless* postmortem culture. Lifted straight from the @srebook. [ed: we didn't invent it, it was jointly invented at same time at other places too e.g. Etsy :)] #SREcon
And celebrate successes, even small ones. use a victories channel to give people party parrots. Reward people for incremental progress. #SREcon
Acknowledge the impostor syndrome. It's real. To work with accomplished and smart people like @mattcutts make you feel like you don't belong. But you do. Remind each other of that. #SREcon
Change is hard, sometimes people are skeptical, but use the USDS values to effect cultural change. #SREcon
Use momentum to your advantage. Build something small that people see works. Build street cred. #SREcon
Be open and share. Show people how you do things. You are there to help. #SREcon
Hire and empower great people. You have to hire people who are very good at their jobs. But not rockstars.
[ed: this leaves open a problem with training. but if you're doing a cultural change, you're training people around you rather than onboarding/training new hires] #SREcon
Find the truth and tell the truth. Point out where processes don't work. Provide examples to drive change. Ask to see "where is the source code?" #SREcon
Go where the work is. There's no substitute to sitting side by side with someone and their processes [ed: if it's a physical presence team. if a digital team, join them in slack!] #SREcon
and finally, @USDS is hiring!! @_a12k makes a strong pitch for serving the US as a citizen by making the government work better [fin] #SREcon
• • •
Missing some Tweet in this thread? You can try to
force a refresh
Final talk I'll be getting to at #VelocityConf before I dash to Toronto: @IanColdwater on improving container security on k8s.
@IanColdwater She focuses on hardening her employer's cloud container infrastructure, including doing work on k8s.
She also was an ethical hacker before she went into DevOps and DevSecOps. #VelocityConf
She travels around doing competitive hacking with CTFs. It's important to think like an attacker rather than assuming good intents and nice user personas that use our features in the way the devs intended things to be used. #VelocityConf
My colleague @sethvargo on microservice security at #VelocityConf: traditionally we've thought of traditional security as all-or-nothing -- that you put the biggest possible padlock on your perimeter, and you have a secure zone and untrusted zone.
@sethvargo We know that monoliths don't actually work, so we're moving towards microservices. But how does this change your security model?
You might have a loadbalancer that has software-defined rules. And you have a variety of compartmentalized networks. #VelocityConf
You might also be communicating with managed services such as Cloud SQL that are outside of your security perimeter.
You no longer have one resource, firewall, loadbalancer, and security team. You have many. Including "Chris." #VelocityConf
The problems we're solving: (1) why are monoliths harder to migrate? (2) Should you? (3) How do I start? (4) Best practices #VelocityConf
.@krisnova is a Gaypher (gay gopher), is a k8s maintainer, and is involved in two k8s SIGs (cluster lifecycle & aws, but she likes all the clouds. depending upon the day). And she did SRE before becoming a Dev Advocate! #VelocityConf
"just collect data and figure out later how you'll use it" doesn't work any more. #VelocityConf
We used to be optimistic before we ruined everything.
Mozilla also used to not collect data, and only had data on number of downloads, but its market share went down because they weren't measuring user satisfaction and actual usage. #VelocityConf