Click or drag the file here to uploadBrowser is not supported, please use IE10+ or other browsers

Cancel

Codeberg Service Status

Please find recent status messages on our Mastodon account.

We are rebooting and upgrading BIOS and Firmware on the server that hosts mosts services. This will take at most 10 minutes.

We have rebooted this server. We are working on getting pages back online.

Impact:

• Several short interruptions of Forgejo ("Codeberg.org") service in the next 24 hours
• As a consequence, interruptions or transient failures in connected services (e.g. hosted CI/CD, Weblate)
• Potential performance degradation

Status:

• The planned changes have been conducted and the affected servers are up and running again. We're still monitoring the systems in case of follow-up issues.
• The hardware maintenance has continued at 17.55 UTC. Server replacement is ongoing.
• The first round of failover was successfully completed. Disruption was longer than expected due to human error.

Background:

We are replacing 1 of 3 physical servers powering Codeberg operations, upgrading computing capacity and replacing a machine with a broken mainboard. To complete the operation, several services need to be moved across servers, leading to periods of unavailability. Also, computing capacity is temporarily reduced due to the hardware constraints or blocked by backfilling operations (data replication from MariaDB/Galera database and Ceph storage clusters).

We aim to keep the impact as minimal as possible. Unfortunately, our current deployment of Forgejo only supports exactly one primary instance, and each failover / reconfiguration currently involves a downtime of about 1 minute.

Codeberg.org will be down for a few minutes to deploy spam fighting features. 

We are performing scheduled maintenance on Codeberg Pages. While we didn't expect any impact on the live operation, we are currently investigating "502 Bad Gateway" responses on some sites.

21:26 UTC: We have upgraded our last server to Debian Trixie. We are bringing up all services up and monitoring the situation.

20:44 UTC: We are upgrading our last server to Debian Trixie, the upgrade with the other server went very well and had minimal downtime. This servers hosts codeberg.org and will be unavailable during the upgrade.

19:59 UTC: The server upgrade is done we are bringing all services up and monitoring the situation.

We are expecting to upgrade one of our servers to Debian Trixie this evening (CEST). This server hosts Codeberg CI (Woodpecker), Codeberg Translate and Codeberg pages will therefore be unavailable during the upgrade.

It is possible that other services are impacted, our filesystem (Ceph) will be put into maintenance mode during the upgrade.

Update: The cause for the performance degradation was identified to be unprecendented AI crawling on our servers. We have had to do more aggressive proof-of-work challenging for various expensive URLs. A novelty is that AI crawlers not only access links that are actually generated by our web frontend, but they seem to convert the URLs into a format that bypassed existing filter rules. https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/commit/e4aaceeb434a82d2b8c610284006d3d0a3cbe665

We are currently investigating high traffic on Codeberg that has degraded the performance of our database cluster. We are still investigating. Some attempts to remediate the problem did not yield the required results.

We're scheduling a Forgejo upgrade. It will result in an expected downtime of a few minutes.

You can join the upgrade party via  https://lecture.senfcall.de/ott-zml-1vs-qcc if you want to watch the progress.

We are currently performing scheduled maintenance on the email setup.

Please expect

• delay in email notification
• temporary shutdowns of services during reconfiguration
• unexpected issues due to human mistakes

The following services are temporarily degraded:

• Codeberg Pages
• Hosted Woodpecker CI
• Blog, Documentation and Design assets

The reason is a failing SSD on our host Kampenwand.