Frappeframework.com & Other Frappe websites are down

Frappe Framework Bug
1

Hello,

I have been experiencing weird bugs in the last few days and i though there was an update ongoing but it took way longer than expected. Mostly all frappe websites except the forum are not working properly. They were working perfectly but all of a sudden they stopped loading from Egypt. It works only using a VPN. And that problem is happening as well with all of my colleagues in different locations in egypt with different internet provides. So, i wanted to just raise that bug so if there was any update that made that happen, maybe you can guyz look into it

thanks in advance

2

checking it now, the problem still persists

3

I received this email this morning, perhaps related (?):

Recently, we have seen recurring performance and downtime issues on Frappe Cloud.

We have traced these issues to a known bug in MariaDB. This bug is fixed but yet unreleased. We are working on deploying a patched MariaDB version on Frappe Cloud, ahead of the MariaDB release schedule.

Database backups are triggering these issues. So, to avoid them, we have temporarily stopped all database backups.

Do not panic! We have increased the frequency of complete disk snapshots, to compensate for the lack of backups.

Please raise a ticket on our support portal if you need access to your site’s data from these snapshots.

Meanwhile, we are also working on physical database backups instead of logical database backups. Physical backups will also unlock more resources for your sites and servers.

We apologize for the inconvenience. We appreciate you choosing Frappe Cloud.

Thanks,

Frappe Cloud Team

4

Possibly also this (but that’s just me speculating):

posted around the same time as the post from abdallahkorraim preceding yours.

5

am not sure if they are related or not, as i am not a cloud user. However, that database bug may be related. But I believe it is more of a Firewall or DNS bug, since the official websites themselves like the documentations of frappe and erpnext are not working but only in Egypt, they work fine with VPNs

6

From the links in the posting of DCRob, it’s clear that it’s a MariaDB issue, e.g.:
https://jira.mariadb.org/browse/MDEV-32371

I didn’t try to understand it completely, but it reads more like a race condition regarding internal signaling and available memory. So no firewall nor DNS issue, at least not primarily, as it seems to me.

Impact?
If you search for MDEV-32371, you find a google entry, but on the linked page of MariaDB server:

it doesn’t appear, which I consider an anomaly. Could be an attempt to not highlight, e.g. retraction after initial publication.
You can find that this issue duplicates others, e.g.
https://jira.mariadb.org/browse/MDEV-32909 (critical, from november)
and it’s also mentioned here:
https://jira.mariadb.org/browse/MDEV-32174 (relation unclear to me from superficial reading)

I tried to find any related CVE, but didn’t find any.

7

The mariadb issue is unrelated and mostly resolved now (we deployed patched version yesterday)

Can you confirm if it’s just a dns issue by trying any public DNS like google/cisco/cloudflare?

Otherwise I’d suggest contacting your ISP for authoritative answer on why sites are not accessible.

Let us know if you need some information from our side. (DM me on telegram @ankushmnt or email on ankush(at)frappe.io )

1 Like