About the talk
Why do we test on production? Why not avoid the risks?
Thorough testing before merging to master is great, but it doesn’t cover the unknowns. Staging on shared environments tends to be slow, unreliable and costly to support. Why not just learn from the only true environment by conducting safe and efficient experiments?
This talk is based on my experience of "shifting to the right" testing within the context of back-end systems of bol.com (one of the biggest online retailers of the Netherlands; logistics and purchasing domains), where correctness is often a bigger concern than performance, and recovery might require a bit more than users hitting the refresh button of their browser.
Testing on production is often associated with A/B testing or canary releases, but those aren't always the best - or even applicable - techniques. We will look instead at shadow and dry runs, controlled experiments, survival of the fittest; how to apply these techniques and what to be aware of.
00:04 A confession to make: I do tests. On production
00:40 Start to enjoy it: more than an opportunity
01:04 Today’s agenda
01:42 A little bit about bol.com
02:29 An error because of the scale can be very quickly quite costly
02:34 The back end and front ends
02:59 What is testing?
03:53 3 types of environments: isolation, staging, and production
05:11 First example: business context
08:03 Sometimes we would miss some messages…
11:25 Second example: database change
13:47 Sources idea or reference generation
13:49 Outside or within of your shadow area?
16:50 Last example of shadow: shadow across the system
20:02 When something really happens in productions then theoretically arrives in learning
20:22 The important point: one toggle to rule them all
22:06 The other techniques: golden order a.k.a. “lucky” customer
23:29 White-listing: a little bit more formalized technique
25:38 Another technique: test a.k.a. Sandbox accounts
26:08 Dry Run/Preview: perform a calculation without side-effects
26:47 Buddy service connectivity
27:56 The last example: survival of the fittest
30:29 How do we do performance testing?
31:17 Another very useful technique
31:51 We don't do engineering like monkeys
32:39 Testing in production - testing environment
33:07 The small but very important side note
34:02 Examples: best used together
36:02 The last drop. Conclusion
Hi good, 00:04 Afternoon. 00:06 My name is Mycola. 00:06 And I have a confession to make. 00:09 I do test. 00:12 And in production. 00:12 When I first realized this. 00:17 I had that feel feeling of sorrow because I thought. 00:19 So this is a sign of deficiency in our testing strategy. 00:23 We should test everything before we go to production. 00:27 How's the excuse of lack of resources and time it was a little bit of a week. 00:30
But we can doing this and all the time I started to get used to this testing in production. 00:35 Start to enjoy it. 00:40 And I started to see it more as an opportunity rather than seeing. 00:40 This is what I would like to talk about today. 00:47 So. 00:51 A sketch of a button just intro, 00:51 little bit boring part. 00:54 Many people find it, 00:54 but I think it's important to give the context where I'm talking or they're coming from. 00:57 Why uh this works for me? 01:01
Then I will go into examples my favorite technique is shadow shadow around there will be 3 examples. 01:04 Other techniques are little bit operational concerns. 01:10 A reflection and conclusion. 01:15 And I will be using this. 01:18 Droplet as a reminder to me. 01:20 Keep my mouth. 01:22 Hydrated. 01:25 And if you have questions. 01:28 I would like to give us some time at the end, 01:30 but if you have something. 01:33 The short or something doesn't make sense. 01:34 This would be a good moment also to ask. 01:37
So I work in bowl.com. 01:42 It's a Dutch webshop. 01:44 Um. 01:44 And also platform for Commerce. 01:48 It's a small if you look from the side, 01:50 but in our countries relatively big. 01:53 It's a common so we don't have plans that will be crashing. 01:57 If we do a mistake. 02:01
Buy this talk
Buy this video
With ConferenceCast.tv you get access to our library of the world's best conference talks.