Anoxinon e.V. @Anoxinon

Continued thread

**Codeberg Status** @codebergstatus · 3d

In the most recent case, it was unfortunately a deadlock issue causing Forgejo to not respond within that timeout. It takes more than 15 seconds to be returned as a deadlock error. Hence, you get to see the timeout error instead of a 500 error (https://codeberg.org/Codeberg/Community/issues/2596).

The timeout was not the issue here, although we have bumped it for POST method requests to give Forgejo extra margin to work with.

Codeberg.org500 server error creating issues/PR, commenting etc.### Comment It had been happening several hours a day for previous 3 days. E.g. when posting a comment a "loading" (sending?) spinner is visible and it spins for several seconds, then a red `! server error: 500` error bar appears and upon page refresh its clear the comment hasn't been added. ...

Continued thread

**Codeberg Status** @codebergstatus · 3d

Codeberg Status @codebergstatus

You might've encountered a 504 error, which means Forgejo did not respond within a specified timeout (15s in most cases).

When we used Caddy, we did not configure timeouts, and it would indefinitely wait for a request to finish. This is not ideal when Forgejo is under high load and gets increasingly more traffic. For HAProxy we have set a max amount of requests and timeouts to prevent (and possibly recover from) Forgejo from a state where it's overloaded and stops responding.

**Codeberg Status** @codebergstatus · 3d *

3d *

Codeberg Status @codebergstatus

It has been in the works for months, but the recent increase of downtime has accelerated work. We've moved from Caddy to HAProxy for codeberg.org traffic.

Starting over with a new reverse proxy has enabled us to make choices that enable us to set up HAProxy to scale with our increasing traffic. For example, traffic is split (when possible, decided via heuristics) over several instances to load balance expensive traffic.

We hope to share more details at a later time about this change.

**Codeberg Status** @codebergstatus · 3d

Codeberg Status @codebergstatus

Please ignore.

Hi Matrix chat!

Continued thread

**Codeberg Status** @codebergstatus · 6d

Codeberg Status @codebergstatus

The maintenance is completed and services are expected to recover soonish (still fighting with the Codeberg e. V. forum).

We're postponing the reboot of the third machine.

Continued thread

**Codeberg Status** @codebergstatus · 6d

Codeberg Status @codebergstatus

Ceph is happy, we're able to bring this machine down finally and are doing so now.

Continued thread

**Codeberg Status** @codebergstatus · 6d *

6d *

Codeberg Status @codebergstatus

We're uncertain about the timeframe of when we can shutdown this machine. Therefore, we're going to bring up services again for the time being, we only require them to be down just before bringing the machine down.

Continued thread

**Codeberg Status** @codebergstatus · 6d

Codeberg Status @codebergstatus

Hi,

It's taking longer than expected. Ceph made a surprise move on us at the last moment by telling us our network configuration is maybe no longer up to date after we replaced servers (not today, but a while ago). This is preventing us from deploying new monitors to keep quorum. We're working on fixing this.

It's _simply_ preventing us from gracefully shutting down, no data is at risk.

**Codeberg Status** @codebergstatus · 6d

Codeberg Status @codebergstatus

We'll be performing a reboot of all our servers this evening.

This means that our services (CI, pages, translate) will be temporarily offline. We invested a lot of effort in the last few months to be able to move Forgejo between servers easily and this should see minimal downtime.

**Codeberg Status** @codebergstatus · May 4

May 4

Codeberg Status @codebergstatus

Codeberg.org appears to be very slow right now, we are trying to resolve this as we speak.

Codeberg StatusHi all, sorry for the lack of updates. We did try to fix the problem shortly afterwards (and seemingly, we did so successfully), but we were alerted a few hours later that this hiccup seemingly occurred again. Thank you to the affected users that helped us notice by reporting this to us.We fixed the problem for now through the classic "turning it off and on again method", but establishing the root cause is still a pending task. ~n

Codeberg StatusHi, we are aware of reports that some users are not able to submit issues/pull requests. We apologize for the hiccup and are working on fixing this problem.

Codeberg StatusWe understand that this was a long wait, but we didn't want to leave it up to chance. So, as an immediate security measure we temporarily disabled our CI. Thank you all for your patience.CI operations (Woodpecker CI, Forgejo Actions) have been restored.

Codeberg StatusHi all, sorry for the long wait.We wanted to preemptively ensure that our servers (especially the ones hosting CI/CD services) were sufficiently shielded against <a href="https://copy.fail" target="_blank" rel="nofollow noopener" translate="no">https://copy.fail</a>; we found no evidence of a compromise.We waited for the greater community to react and provide some additional clarity on what is affected and what isn't (e.g. container engines), we carefully studied the LKML, and waited for upstreams to release official updates and guidance.

Codeberg StatusWe estimate that the recovery process of our CI services may potentially take multiple hours. We ask for your understanding as we are trying to reinforce our services' reliability.

Codeberg StatusWe are expanding our maintenance efforts to include our Forgejo Actions CI as well. Again, we apologize for the inconvenience. We'll try to bring both services back up as soon as possible.

Codeberg StatusOur hosted Woodpecker CI/CD is going down for maintenance.* Builds will be delayed. * Web UI will be inaccessible for some minutes. * Commits pushed during the downtime will not trigger builds.We apologize for the inconvenience.

Codeberg StatusWe're investigating a downtime of our primary instance.

Codeberg StatusCodeberg.org will be upgraded to Forgejo v15 in a few minutes. We expect downtime to be at most a few minutes, according to migration tests in the staging environment.

Codeberg StatusWe have recovered Codeberg pages v2 from an outage rendering websites inaccessible. The newer "git-pages" deployment was not affected.We apologize for the disruption.Yesterday, we deployed a change to our main Forgejo instance that was supposed to not affect the read-only shadow instance used to serve Codeberg Pages' API requests. However, more than 24 hours later, the read-only instance was restarted automatically and the configuration change inadvertently prevented it from coming back up.

Codeberg StatusWe believe we have established the cause and are observing that our services are recovering. We are monitoring the situation. Should we have any further updates to make, we will let you know. Thank you for your patience.

Codeberg StatusHi all, we're aware that Codeberg is down right now and are investigating.

Codeberg StatusWe're up again.

Codeberg StatusWe are aware of reports of Codeberg being down and are investigating. Thank you for your patience.

Codeberg StatusRe-indexing just started.

Codeberg StatusThis evening we will switch Codeberg.org's issue search backend from Bleve to our new OpenSearch cluster. During this transition, search results might be unavailable or inaccurate as every issue is getting re-indexed. We've done a test on a snapshot of the data and then it took roughly 1 hour for re-indexing.

Codeberg StatusWe believe that access has been restored for the time being, we'll continue monitoring the situation for now.

Codeberg StatusGood morning. Since a bit more than an hour ago, multiple people from our side begun working on restoring the availability of our services and will share further updates as they come.Nothing that's impossible, it's all just a bit of a bother.

Codeberg StatusWe were able to restore IPv4 and IPv6 connectivity. Performance is still not optimal, and connections are occasionally dropping. We've taken measures to try and maintain uptime overnight (with a certain probability).Routes protected by iocaine (crawler protection) may return 503 errors. Currently, it appears that these packets are also occasionally being lost.It's 2:30am, and we require a good night's sleep before we can look into this further.

Codeberg StatusHi everyone! We are still fighting with an ongoing DDoS attack against our service on the network level. We do not yet have any ETA on recovery time, as we're still investigating different mitigation strategies.As always, we are very grateful for your support and patience.

Codeberg StatusWe're currently suffering from performance degradation due to heavy traffic. We are still investigating.

Codeberg StatusMaintenance is over. Ceph is still recovering and moving data around for the next hours.We have finally restored our 3rd host that was down for about 2 weeks. Now, we have full redundancy on the Ceph filesystem again.Replacement of the affected server was delayed due to FOSDEM and poor performance of initial host system SSDs (which was unsuitable for operating our database cluster on it).

Codeberg StatusWe're entering Ceph filesystem maintenance. Please expect performance degradation or (in worst case) some downtime.

Codeberg StatusWe are currently struggling with database performance and investigating.Tonight, we shipped a new server to the datacenter and started restoring Galera cluster on the node (since the hardware breakage last week, we were running on 2/3 nodes only).We are not yet sure what is causing the sudden performance drop.

Codeberg StatusGreetings from the emergency response team in trains and buses to <a href="https://social.anoxinon.de/tags/FOSDEM2026" class="mention hashtag" rel="tag">#FOSDEM2026</a>! Looking forward to seeing you soon.

Codeberg StatusAbout 30 minutes ago, we have recreated a runner which is currently processing the backlog of Hosted Woodpecker CI/CD builds.Take a look at details about which jobs have been queued or missed and changes in the environment at <a href="https://codeberg.org/Codeberg/changelog/issues/4#issuecomment-10265260" target="_blank" rel="nofollow noopener" translate="no">https://codeberg.org/Codeberg/changelog/issues/4#issuecomment-10265260</a>

Codeberg StatusWe have restored the Weblate ("Codeberg Translate") instance from backups a few hours ago. There might have been up to an hour of lost work, and we apologize for this. In the next days, we are evaluating how to improve recovery point and time objectives for this service.

Codeberg StatusWe have finally restored access to your Codeberg Pages.Weblate and Hosted Woodpecker CI/CD will unfortunately have to remain offline.We are currently unable to access any of the two boot devices on that host and we are currently evaluating whether we can rescue the latest data or restore from one of the hourly system snapshots (which would involve data loss since the last snapshots).We'll proceed with further investigation and a decision after some sleep.

Codeberg StatusHardware maintenance on the 3rd server (the one that had a broken SSD and was the reason for the maintenance) did not succeed.We have to keep it down and continue debugging remotely.This affects Weblate, Hosted Woodpecker CI/CD and some other services.We apologize for the issue and will work on restoring access to them as soon as possible.

Codeberg StatusDuring the following steps, we did an elevated amount of human errors (likely attributed to increased stress) and had more downtime of Codeberg.org than initially expected.This was due to a configuration error that made the "failover" instance still depend on a service hosted on the primary server, and so the failover also became unavailable when the primary went down for maintenance.It took us a few minutes to realize and correct the mistake.

Recent searches

Search options

Administered by:

Server stats:

Codeberg Status@codebergstatus@social.anoxinon.de

Recent searches

Search options

Administered by:

Server stats:

Codeberg Status@codebergstatus@social.anoxinon.desocial.anoxinon.de

Codeberg Status@codebergstatus@social.anoxinon.de