Context is that I maintain mostly-automated, post-only accounts at @ComplexSysGrp and @CriticalPointTV on Twitter for people who want to follow updates about e.g. the #CriticalPointWarStories podcast without having to listen to me rant about e.g. California housing politics.
3/5
His internship at Akamai turned into a global incident with multimillion-dollar consequences for a major tech company. Here's how it went #CriticalPointWarStories
youtu.be/-I40YOtE59Q
Scaling software isn't just about servers and data centers. Abhijit Mehta explains how consumer products face unique challenges: "Scale can mean something that has nothing to do with backend technology." #CriticalPointWarStories
youtu.be/k450kfb8NPk
Ever wonder how all the Swifties can download Taylor's latest album without breaking the Internet? Abhijit Mehta breaks down Akamai's pioneering technology called consistent hashing which enables this. #CriticalPointWarStories
youtu.be/NVUY4fvY89A
From merge to outage: A cautionary tale about the limits of feature flags and the challenge of rollbacks in software systems in this episode with former Dropbox software engineer Kate Rudolph! #CriticalPointWarStories
youtu.be/8QKdruRs4Ys
As invaluable as human QA is, they can never find all the bugs in a game. "A game comes out, and let's say there's 100,000 players. Those players play for an hour. They just wiped out three months of QA time." #CriticalPointWarStories
youtu.be/UupPvHocUQg
"It looked like everything was fine. But nothing was actually happening." Julia Lunetta on the dangers of silent failures in complex systems. #SoftwareEngineering #Monitoring #CriticalPointWarStories
youtu.be/2f9ifyVxg1s
"Akamai had a scale and sophistication that... could absorb a lot of those things." Abhijit Mehta on how Akamai's distributed network handled DDoS attacks with ease. #Cybersecurity #NetworkResilience #CriticalPointWarStories
youtu.be/NVUY4fvY89A
"Players are no longer going to accept like, 'Oh, we had server issues.' 'You had server issues on the last four launches. What have you learned?'" Zachery Johnson on rising player expectations for game launches #CriticalPointWarStories
youtu.be/UupPvHocUQg
"Not just testing that nothing explodes, but making sure that everything is still happening as it's supposed to." Julia Lunetta on the importance of positive and negative testing. #QA #SoftwareTesting #CriticalPointWarStories
youtu.be/2f9ifyVxg1s
Discover how Akamai's innovative approach to #IncidentManagement, born out of necessity, influenced the industry standard, in this #CriticalPointWarStories bonus episode with Abhijit Mehta!
youtu.be/NVUY4fvY89A
"Computers are deterministic except when they're not." –Julia Lunetta dropping truth about #software engineering on the #CriticalPointWarStories podcast
youtu.be/2f9ifyVxg1s
"When you have to do a rollback, it's not without consequences. Any time you change the code, you can break things." Valuable lessons from a Dropbox incident that offered free watermelons to everyone! 🍉 #CriticalPointWarStories
youtu.be/8QKdruRs4Ys
"I deleted the company's entire source code repository at my first internship" - @kevinriggle@bsky.social
"I accidentally routed Yahoo's traffic to Japan" -Willie
What's your biggest tech oops? Share below & watch the episode! #CriticalPointWarStories
youtu.be/-I40YOtE59Q
From breaking user logout to accidental watermelon promises, a rollback had unintended consequences for this Dropbox intern. #CriticalPointWarStories
youtu.be/8QKdruRs4Ys
From consistent hashing to incident management, Akamai's influence on the internet is everywhere. Abhijit Mehta and I explore how this "Velvet Underground of SRE" shaped today's software engineering culture. #CriticalPointWarStories
youtu.be/NVUY4fvY89A
"If I have a performance drop that even drops that by 1%, that's a lot of money." Abhijit Mehta on the high stakes of ad platform updates at Twitter. #CriticalPointWarStories
youtu.be/k450kfb8NPk
Usually latency is measured in milliseconds, but one time Willie Williams made a mistake which caused Yahoo's to be measured in thousands of miles. Watch here #CriticalPointWarStories
youtu.be/-I40YOtE59Q
"I basically took out production and our production monitoring in one fell swoop." Julia Lunetta shares a cautionary tale about shared filesystems and single points of failure. #DevOps #IncidentManagement #CriticalPOintWarStories
youtu.be/2f9ifyVxg1s
As an Akamai intern, Willie didn't know who Yahoo's CTO was, but when a configuration change he made broke their site, he learned in a hurry. #CriticalPointWarStories
youtu.be/-I40YOtE59Q
"The sanctity of the economy was at the forefront of our mind." Zachery Johnson explains why preventing item duplication was crucial in Diablo II: Resurrected, even during a major incident #CriticalPointWarStories
youtu.be/UupPvHocUQg
"As developers, be completely cognizant that players are your game. If you have no players, it doesn't really matter what you've built, so respect your players and do right by them." #CriticalPointWarStories
youtu.be/UupPvHocUQg
"I put a like, five or six line comment at the top of this 'task' file basically saying, do not touch this." Sometimes, the most important code isn't code at all. Julia Lunetta's story on the #CriticalPointWarStories podcast
youtu.be/2f9ifyVxg1s
"Changing my definition of 'done' and learning at what points things become available to different people" was an important part of what this Dropbox intern learned in her time at the company #CriticalPointWarStories
youtu.be/8QKdruRs4Ys
"Write something like you would want it to be written if you had to debug it at three in the morning, six months from now." Good advice for #coding and especially good advice for documentation. #CriticalPointWarStories
youtu.be/2f9ifyVxg1s
"I have a reverence for the database I did not have before. I understand the power it wields. We talk about single source of truth or a single point of failure, that big old box of state is what that is." #CriticalPointWarStories
youtu.be/UupPvHocUQg
😱 "ElasticSearch is there. It's the same host that was there... But it is entirely empty. There is not a drop of data in it." A tale of vanishing data and rapid recovery from Patrick O'Doherty #CriticalPointWarStories
youtu.be/RswXKVTali0
"He Broke Turkish Twitter" - not quite, but Abhijit Mehta ran into the infamous "Turkish i bug" while rolling out a major ad platform update at Twitter. #CriticalPointWarStories
youtu.be/k450kfb8NPk
"Do you know who [Yahoo's CTO] is? Google him." Not the way you want your boss to introduce you to the very angry man on the other end of the phone! #CriticalPointWarStories
youtu.be/-I40YOtE59Q
"There are little things that can bite you, and setting up the test environment to catch things like that is, in general, tricky." The challenges of global software deployment are not just technological but also cultural. #CriticalPointWarStories
youtu.be/k450kfb8NPk
