Tried recreating the service record again, still feel like the service is missing a lot of screenshots.

I think model retraining time draws nigh, RIP my free evenings

👀

I think Truth Social has changed UI a lot recently so I need to retrain the model with that. Same with Twitter and Bluesky. Hoping to do so in the next month or two.

Apologies for how lacklustre model performance has been of late!

I'm honestly not sure, I find it really challenging to debug label propagation issues, wish there was better tooling to investigate them.

The AppView disconnected from the labeller so I had to recreate the service record, which means that anything labelled since that happened and now wasn't properly ingested. I think I've fixed it now though.

Yes, especially! Thank you! 💚

Please report posts that the labeller misses:

"..." menu › Report post › Other › Xblock

I'm retraining the model soon and reports are extremely helpful for improving accuracy!

There's a lot of screenshots happening today so apologies if processing is a bit slower than usual

We have a localised power outage at the moment so services are down for now

We're experiencing some inference downtime due to the relay switchover, hopefully back soon!

Status: Localised network outage, hopefully back soon

Ah! Much appreciated! I updated my Ozone image just now and stuff started working again.

Back up

Autolabelling service outage while I figure out why Ozone keeps dropping 502s

Okay I think we're back

Argh it's not PC maintenance and repair without at least 2 aggravating moments

Service update: Inference is going down while we continue to upgrade our systems

Service update: system upgrade complete

(tho I still wanna upgrade CUDA to 13 at some point)

New GPU has been installed and we're just updating drivers now. Hold onta yer butts

Service update: scheduled downtime for system upgrade about to commence, will update here once complete.

Pretty relieved to realise my AWS credit doesn't expire until September next year (and that I've only used £2k of the £5k from the ATProto grant), that gives me a good few months to get XBlock onto stable local infra. I think if I can get Redis and Ozone back locally it'll be sustainable.

Finally upgrading some infra 🎉

Service update: more network fun.

Continuing to have intermittent connectivity issues with our London datacentre (Aendra's flat)

Service update: we had a localised network outage earlier so it might take awhile for the inference server to catch up with backlog

Service update: I've figured out why Ozone isn't accepting appeals, apparently there was a lexicon change or something which meant we had to explicitly configure it to accept appeals and not just reports.

Appealing a label should work now, sorry that took so long to fix!

There is an issue with appeals not submitting properly, I'm investigating that today. I hoped it would be as simple as restarting Ozone but alas nothing in life is that easy.

Yeah, sorry about that. 💚 There's been a lot of UI drift lately and we really need to train another iteration of the model soon.

Yeah that's what I'm thinking

Open to feedback: our moderation efforts have really slowed down so it's taking longer than it should to action appeals (apologies!).

I'm thinking, should the appeals process be semi-automated? I've refrained because it's somewhat open to abuse but I suspect it would help more than hinder.