Me, sitting at a table in Schiphol airport in Amsterdam, laptop open, about to figure out why MSTDN.DK crashed :D
#MSTDNDK was down for a moment there - seems the PostgreSQL database ran full, and didn't exactly recover ngracefully. Will investigate when I'm not drinking beers in #Schiphol 🙂
Some time this weekend, we'll be upgrading the #Mastodon #PostgreSQL cluster on #MSTDNDK. The cluster is being managed by #CloudNativePG and upgrades have been successfully tested on multiple other clusters - even upgrades directly from version 15 to 18.
While this is an _incredibly_ smooth and […]
Screenshot of the beginning out the script output. Checks are being done before working on a node.
Screenshot of the beginning out the script output. Post-upgrade reboot of node and waiting for it to be ready before lifting the cordon.
I've been a little rough on irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.
I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and […]
[Original post on mstdn.dk]
One of the #MSTDNDK nodes is currently experiencing issues with an #NVMe disk that started reporting #SMART errors. It's still up, and we're talking to #Hetzner support about scheduling a replacement. Once we've agreed on a plan moving forward, the node will be taken down and hardware will be […]
Hård uge. Arbejder og sover. Heldigvis holder @leeleedee styr på #mstdndk imens jeg jamrer og klager! Tak for det! 😊
.. og med de ord vil jeg se dyner! 🤣
The #MSTDNDK Mastodon instance just migrated away from Bitnami's #Redis helm chart and container images to #Valkey.
#Bitnami and Redis both seem to be exiting the open source community, which could prevent us from staying current, meaning always running the latest versions of the software […]
Velkommen til @anduin som - hvis ikke #nypåmastodon - så i hvertfald ny på #mstdndk 🙂
The #Matrix Homeserver loosely associated with #MSTDNDK - https://mtrix.dk/ - is currently undergoing maintenance. We'll probably be dropping #KeyCloak in favor of the new Matrix Authentication Service to simplify the setup. Stay tuned.
#mtrixdk
Kære medborgere på #mstdndk! Glædelig bagjul og godt nytår! Tag fordi I er her, og tak for jeres tålmodighed! Det er en rejse at drive et setup som dette, og engang imellem koster det lidt nedetid at blive klogere. Med fare for a jinxe det, så har det kørt rigtig godt et stykke tid nu, og vi […]
Screenshot of Grafana showing graphs and metrics for the mstdn.dk PostgrSQL cluster. Some red boxes, but not sure how to read that yet.
Monitoring is slowly getting up and running.. now if only I knew how to read all that! ;-)
#mastodon #postgresql #cloudnativepg #grafana #mstdndk
One of the nodes of the Midgaard #Kubernetes cluster running #mstdndk is currently down, and awaiting a disk replacement. Luckily clustering works, and you shouldn't feel a thing ;-)
Resiliency for the win! :-)
Tonight is #Halloween, which also means it's the #birthday of #mstdndk! Three years ago exactly, the domain was registered and we deployed #Mastodon with my home grown Helm chart. A couple of horrible crashes and a lot of experience later, we're still alive and kicking! With backups and […]
#mstdndk and the underlying Kubernetes cluster died a couple of days ago. Working on recovering.
Considering switching the #MinIO backend of #mstdndk to #Garage by #deuxfleurs. 3 replicas on #Kubernetes. Anyone with real life experience and/or tips? :-)
https://garagehq.deuxfleurs.fr/
#mastodon #s3
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the […]
A node crashed today, and unfortunately the remaining three were overburdened with a seemingly spontaneous filesystem check of some rather large volumes, including the ones holding the three database replicas and three media storage replicas for #mstdndk. We're running #Longhorn on #Kubernetes […]
We're experiencing an issue with the media storage backend on #mstdndk today. We'll look into it as soon as possible, which unfortunately isn't for at least a couple of hours, as I have some stuff to take care of in the real world first ;-)
Velkommen til, @fruAnne, @mrfr, @ethan, @madmilo, @jaw, @fynn1, @sindum, @peterhelms og @chang! Nye på #mstdndk, og måske #nypåmastodon? :-)
We'll be updating #mstdndk during the night one of the next days. Hopefully it will be dreadfully boring and you won't even notice, but just in case - if the server goes down, know that it's probably related to the upgrade, and should be fixed shortly.
We'll put out a notice a couple of hours […]
Issues from the expired DNSSEC incident are unfortunately still affecting #mstdndk. We're still temporarily disabling DNSSEC for the affected domains, but missed a few in the first round. We'll try to improve our procedure for actually keeping the damned things updated, but there's a first time […]
Issues with expired DNSSEC keys has caused a bit of an outage on #mstdndk today. DNSSEC has been temporarily disabled and everything should work as the update propagates. Will get it sorted ASAP.
