April 2026 Outage Post-Mortem - Jim's Pckt Hey all! I'm Jim, and I do system-y things at Bluesky. I'm here to give you some details about what happened on Monday of this week that caused Bluesky to go...

I wrote a short post-mortem on the Bluesky outage we had on Monday. First time using pckt, it was a really nice experience tbh!

pckt.blog/b/jcalabro/a...

I'm mostly offline until next week.

in this case it looks there are more than a thousand accounts created? we don't have simple affordances to clean up a situation that large yet.

I'd recommend starting fresh with "pds2" hostname or something like that.

"atmosphere account"?

probably don't teach about 375 Pearl St, 33 Thomas, or Secaucus either.

colo prices can be surprisingly good right in NYC, especially for high-bandwidth, lower-power-density stuff (eg, not GPUs)

Lexicon Service Docs - at:// pizza thoughts Riffing on some ways to make network services more legible to humans and machines

as you know there have been a couple ideas floated for more of an open aggregation "app store", where anybody can submit projects.

I wrote a bit about how those could be linked to interoperable "app modality" declarations and data schemas

eurosky (with @thisismissem.social ) has built a "portal" for apps. some pros of this approach are explicit curation and room for deeper partnership/integration
portal.eurosky.tech

🫂

another option would be switching over to jetstream, which can filter by collection (but provides different guarantees, doesn't do backfill, different interface, etc)

with tap bandwidth do you mean ingress or egress?

the full firehose does add up to a lot of ingress every month, but that is no-cost with many hosting providers. in theory you could split out just the tap part to one of those, and keep the rest of your setup where it is

The first freight Shinkansen bullet train, operated by East Japan Railway Co.

Related to this, we've been slowly rolling out a higher resolution limit for our image uploads, from 2000px^2 to 4000px^2.

Until the below image size increase rolls out (soon), image sizes are still capped at 1 MB, so FYI you won't see immediate quality increase in all cases just yet.

Wide shot of an elevated gantry at work

Close shot of the gantry and some suspended concrete blocks

Closer shot of the blocks

Gantry from the front where you can see how the blocks connect

The launching gantry for the SkyTrain extension is getting close to my future local train station. It is cool to see how it lifts and lines up those big concrete blocks for the elevated guideway. surreylangleyskytrain.gov.bc.ca #SurreyBC

alex I know right where you are. you must go to the Sembcorp Cool House, and then Haw Par Villa.

(curious if there are other ways to avoid that dynamic; or if i'm misinformed. it used to be a concern for web accessibility incentives)

I think this is good to explore!

gotta be careful with incentive structure: if folks get more distribution if they put a dummy string in alt text, then might do that. similar to gaming web a11y scores. maybe a first layer check for that is enough

hrm! yeah the reference PDS inserts that for you automatically with createRecord.

definitely wrong '$type' should be rejected, and I sorta feel like SDKs / client apps should include it, and maybe "MUST" (would be a transition).

I think @divy.zone liked PDS being helpful; u still feel that way?

client devs (you) should be setting it on all requests you want proxied/forwarded via the PDS.

the app.bsky-specific "automatic" behavior is deprecated and we will try to find a way to incentivize devs not to use it when we get a chance (new PDS impl "should" not need to support that)

who is developing a reputation for taste in atmospheric design?

eg, in the 2010s era, I would refer back to Github URL path structure, APIs, and product design (even though I didn't use github directly very much).

who is doing that for atproto lexicons, apps, etc?

Yes it is thematically appropriate to mute the word "antimemetics"

as far as I can tell they are not publicly releasing the model? and are doing a good job helping upstream projects patch flaws (which are actual dangerous security bugs).

I'm skeptical of most of the AI industry and hype cycle, but this specific project seems good.

Bluesky Hackathon in Tokyo Vol.1 (2026/04/18 11:45〜) ## 🗼 Bluesky Hackathon in Tokyo Vol.1 開催のお知らせ 🗼 Blueskyハッカソンを初開催します！ Blueskyを便利に使うツールやATプロトコルを活用したサービスなど、あなたのアイディアを形にしてみませんか。 🔷🔷🔷 4月13日にはBluesky Meetupイベントもあります。両方へのご参加も大歓迎です！🔷🔷🔷 ## 📆 イベント詳細 * 日...

Tokyo. April 18. First Bluesky hackathon in Japan.

Build anything on ATProto — tools, bots, feeds, visualizations. Sponsored by Bluesky. Lunch and snacks included.

Engineers, designers, planners — all welcome.

428lab.connpass.com/event/387583/

yes! we have tools and can improve things somewhat

I don't have any confidence that US/Israel will stick to anything. US/Israel could still attack in the next 24hr, or in a month, etc

probably a good time at least, like, upgrade to modern/current/supported versions of base OS, runtimes, network daemons.

that debian oldoldstable box running python3.5 which has somehow made it this far probably ain't going to make it

what's your ai strategy?
what's your quantum computing strategy?
what's your ww3 strategy?
what's your collapse of consensus reality strategy?
what's your restructuring of global supply chains strategy?
what's your pedophile billionaire cabal blackmail ring strategy?

if you can get even a few dozen regular active human accounts first (eg, early supporters, testflight users, etc) that is a better start

that isn't necessarily due to bad intent or negligence, it is just pretty hard to sign up thousands of new accounts without having fake accounts get in at the same time. there are sophisticated botnets already which flood new PDS instances when possible

there are multiple relays in the network, and it is good to be public/transparent about bumping quotas when possible.

we've been burned a few times with folks asking for a large account limit bump to take advantage of a special launch or migration, and then they just end up with many bot/spam accts

doing a backfill is unrelated to the relay PDS account limit.

what is your PDS instance? how many active human accounts are you hosting?

one thing you could do instead is 'goat repo unpack' to a bunch of JSON files on disk, and then 'goat record create - < ./file.json'

would need some scripting/labor to figure out which records to import, set the rkey, etc