tldr: instead of using tap on a separate platform, I used a cloudflare durable object + jetstream to bring it all under one roof. It does have its limitations though, and I do plan to unpack a lot of that in a blog post
lol 😂 I would be honored! I'll hit you up when I have a draft and we'll see if it fits :)
I think it’s long over due; will put it on my blog idea list!
Thanks to the awesome atproto community advice, docs.surf lives to see another day! Was able to create a closed loop on Cloudflare and cut my usage to practically nothing. Should be stable for months to come!
If you’re using a consistent design pattern you can make a skill that follows it. I did this for a set of uniformed apps I’m making and it works pretty well. Over time you can always refine it and make it more and more specific. github.com/stevedylande...
I did some deeper digging around my billing for Railway which has been the primary cost, and its kinda split between egress and CPU. Currently switching to a durable object on cloudflare that listens to jetstream which should help a bunch.
I might look into that!
Yeah I’m gonna have to start filtering bridgy records at this rate. I’m less annoyed by the noise and more concerned about the network effects; we should be able to handle a lot more.
I’m likely going to look at quickslice since it can use jetstream instead, so might be going back to a local setup if it’s not too intense. I think even then I might have to filter out bridgy records because there’s just so many.
For a while I moved the tap instance to my home server which did work, but eventually the ingress was so high my home internet was getting throttled (kids and wife were not happy lol).
That’s kinda where I ended up. I’m currently running tap on railway and sending webhooks to Cloudflare to handle data processing, but since it’s getting so many records it’s pushing egress costs up.
10000%
A refactor might be in order using tangled.org/slices.netwo...
Thanks for the heads up @goose.art!
Maybe I just need to look into quickslice? first time hearing about it. Link?
That would be really nice!!
I have decent internet, but definitely not gigabit. Basically I started hitting a rate where my family's wifi experience was getting throttled, and for a project like this, wasn't worth it.
Unfortunately is the incoming traffic bandwidth. Tap listening to everything coming from the firehose, so even if you only subscribe for a specific record, the incoming bandwidth is heavy. I even tried to use my home server but started blowing through my ISP bw limits 😅
It's rough out there fam 😅 I think if you only store valid records it helps a little bit with DB size, but doesn't stop the mass firehose of data that comes inbound.
The good news is that it’s MIT open source so if anyone wants to tinker with it, have at it!
tangled.org/stevedylan.d...
Hate to say it, but I might need to stop my indexer for docs.surf. It’s. Fun little project, but the costs are slowly creeping upwards as standard.site continues to grow. Tap listening to every single record just eats up way too much egress.
This is America? 😅😭
マリウス.com
Finally moved my git server to a dedicated VPS, so now you can do this
ssh git.stevedylan.dev
Can we get a atproto bsky clone where we fork the lexicons? I want to talk atproto in a bsky format without the bsky user base lol
If it’s not 8oz, it’s a sign you’re in the wrong shop 🥲
computer desktop showing two terminal windows and a music player
current work machine rice; loving it
Incredible
I definitely thought this was some prompt injection technique lolz
