At this point, I don't think any of them (Banner included) saw Hulk as his own person (see Ragnarok Hulk arc). They all knew that Hulk needed something to pull him back in if he got mad.
Hulk Buster was Tony's trial run at hulk damage control.
It failed spectacularly.
#TestInProduction
Social Security Website Crashes Blamed on DOGE Software Update
#DOGEthieves and bad programmers
#Testinproduction
gizmodo.com/social-secur...
So it appears Firefox is going to have desktop pwa functionality soon. Well I installed and tested my PWA and crap, two more bugs. 🤣
#testinproduction
notepad (not responding)
It is nice to know that Notepad still has awesome feature of "not responding" when opening too big file.
#Windows11 #TestInProduction
This is probably one of my favorite talks I’ve given to date. youtu.be/XqJM_VfK88I
#symfony #upsun #testinproduction
Testing in production requires us to organize our infra blast radius, and to have a blameless culture instead of risk aversion.
Infrastructure as code is an abstraction... not everything maps 1:1 but it's close! [fin] #TestInProduction
For A/B testing, you need to check whether you are meeting your SLOs, as well as being cheap enough...
Write down your hypotheses about how you'll improve SLOs or decrease cost with your changes. Then test it. #TestInProduction
Put higher risk tolerance apps in a taint group.
Once new VPC is spun up and all the instances live in it, then we can tear down the old one. [ed: I... wonder how one does this without AWS deciding it's gonna delete/recreate once network[1] becomes network[0]] #TestInProduction
We also need to have canaries that we can verify that our networks are functional and can connect to/from the outside... before we migrate VMs into the subnets. #TestInProduction
One way of doing this is to create more than one copy of the network, using a toggle variable to control whether 1 or 2 copies exist, and then the second nested copy has different properties.
[ed: we love this technique and use it at Honeycomb too!] #TestInProduction
If you can shift work left and use a staging environment, it might help you.
But, with a network, says @joatmon08, there's no practical way to simulate a full production scale network.
What can we apply? Feature flags. Preserve state, and test rollforward/back #TestInProduction
Next up is @joatmon08 on infrastructure testing in prod! #TestInProduction
So now, having rolled out 35% of all traffic, they were able to finish everything to 100% having enough confidence they'd caught everything. 15-month project to the finish line in the last 2 days!
Make sure to clean up migration intermediate state debt! [fin] #TestInProduction
So now they needed to convert individual API servers (1/47 of traffic each) to PHP7 only.
Next, they needed to do the cron jobrunners, migrating jobs one at a time (high traffic first), and then finally moving all them over. #TestInProduction
But not all users accept cookies, and also there was an opcache corruption. Fixing issues one at a time, they got to 20% rolled out to the app servers.
The next step after that was trying to roll out PHP7 to API servers, most of whose clients didn't cookie. #TestInProduction
They needed observability in order to know whether not their experiment was working. [ed: so far they're only using monitoring to tell amount of traffic...]
But, uh, caching defeats this, unless we use a separate cache key based on presence of PHP7 cookie. #TestInProduction
In order to route traffic between PHP7, vs. HHVM, they used cookies to decide where Apache should route traffic. That way, gradual changes could be made.
A custom extension would allow setting the flag for ambitious users, or routing to specific servers. #TestInProduction
The PHP FPM now provides better performance for their workload than HHVM, although HHVM was originally much much faster than PHP back in the PHP5 days. #TestInProduction
Next up at #TestInProduction is @manjiki of Wikipedia, on moving from HHVM to PHP7 with as few people noticing as possible, by testing in prod!
A few minutes late to @wiredferret's excellent talk on planning for failures, in order to avoid creating disasters. Good to see friends interested in #TestInProduction who aren't @honeycombio ;)
Instead of shifting your tests left, perhaps shift your ability to test rightward towards production.
Have synthetics [ed: or have SLOs!] that expose whether the system is working in prod and let you start debugging if it's not working. #TestInProduction #CloudNativeLondon
So let's talk #TestInProduction. That doesn't mean no pre-release testing. You still need automated testing.
Citing @copyconstruct, have fake versions of your services rather than spinning up the entire stack to test one component. #CloudNativeLondon
"It's not just testing in production, it's optimizing for noticing when things go wrong and fixing them." --@sarahjwells #TestInProduction #CloudNativeLondon
and they fail in very new and interesting ways. only prod has the full complexity of all your real users.
Example: publishing articles stopped working because an endpoint failed out of their chain of 12. they failed over//rolled back. #TestInProduction #CloudNativeLondon
