Frappeframework.com & Other Frappe websites are down

abdallahkorraim · January 24, 2024, 1:34pm

Hello,

I have been experiencing weird bugs in the last few days and i though there was an update ongoing but it took way longer than expected. Mostly all frappe websites except the forum are not working properly. They were working perfectly but all of a sudden they stopped loading from Egypt. It works only using a VPN. And that problem is happening as well with all of my colleagues in different locations in egypt with different internet provides. So, i wanted to just raise that bug so if there was any update that made that happen, maybe you can guyz look into it

thanks in advance

abdallahkorraim · January 25, 2024, 9:10am

checking it now, the problem still persists

DCRob · January 25, 2024, 4:00pm

I received this email this morning, perhaps related (?):

Recently, we have seen recurring performance and downtime issues on Frappe Cloud.

We have traced these issues to a known bug in MariaDB. This bug is fixed but yet unreleased. We are working on deploying a patched MariaDB version on Frappe Cloud, ahead of the MariaDB release schedule.

Database backups are triggering these issues. So, to avoid them, we have temporarily stopped all database backups.

Do not panic! We have increased the frequency of complete disk snapshots, to compensate for the lack of backups.

Please raise a ticket on our support portal if you need access to your site’s data from these snapshots.

Meanwhile, we are also working on physical database backups instead of logical database backups. Physical backups will also unlock more resources for your sites and servers.

We apologize for the inconvenience. We appreciate you choosing Frappe Cloud.

Thanks,

Frappe Cloud Team

Peer · January 25, 2024, 4:10pm

Possibly also this (but that’s just me speculating):

posted around the same time as the post from abdallahkorraim preceding yours.

abdallahkorraim · January 25, 2024, 4:30pm

am not sure if they are related or not, as i am not a cloud user. However, that database bug may be related. But I believe it is more of a Firewall or DNS bug, since the official websites themselves like the documentations of frappe and erpnext are not working but only in Egypt, they work fine with VPNs

Peer · January 25, 2024, 5:16pm

From the links in the posting of DCRob, it’s clear that it’s a MariaDB issue, e.g.:
https://jira.mariadb.org/browse/MDEV-32371

github.com/MariaDB/server

MDEV-32371 Deadlock between buf_page_get_zip() and buf_pool_t::corrupted_evict()

MariaDB:10.6 ← MariaDB:10.6-MDEV-32371

opened 10:19AM - 23 Nov 23 UTC

dr-m

+14 -7

- [x] *The Jira issue number for this PR is: MDEV-32371* ## Description `buf…_page_get_zip()`: Do not wait for the page latch while holding `hash_lock`. If the latch is not available, ensure that any concurrent `buf_pool_t::corrupted_evict()` will be able to acquire the `hash_lock`, and then retry the lookup. If the page was corrupted and evicted, we will finally `goto must_read_page`, retry the read once more, and then report an error. ## How can this PR be tested? I think that this is best tested together with [MDEV-31817](https://jira.mariadb.org/browse/MDEV-31817) #2865, which is included for the purpose of testing. The workload must use `ROW_FORMAT=COMPRESSED` tables, and we might want to use `CMAKE_BUILD_TYPE=RelWithDebInfo`. ## Basing the PR against the correct MariaDB version - [ ] *This is a new feature and the PR is based against the latest MariaDB development branch.* - [x] *This is a bug fix and the PR is based against the earliest maintained branch in which the bug can be reproduced.* ## PR quality check - [x] I checked the [CODING_STANDARDS.md](https://github.com/MariaDB/server/blob/11.0/CODING_STANDARDS.md) file and my PR conforms to this where appropriate. - [x] For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

I didn’t try to understand it completely, but it reads more like a race condition regarding internal signaling and available memory. So no firewall nor DNS issue, at least not primarily, as it seems to me.

Impact?
If you search for MDEV-32371, you find a google entry, but on the linked page of MariaDB server:

it doesn’t appear, which I consider an anomaly. Could be an attempt to not highlight, e.g. retraction after initial publication.
You can find that this issue duplicates others, e.g.
https://jira.mariadb.org/browse/MDEV-32909 (critical, from november)
and it’s also mentioned here:
https://jira.mariadb.org/browse/MDEV-32174 (relation unclear to me from superficial reading)

I tried to find any related CVE, but didn’t find any.

ankush · January 31, 2024, 3:46pm

The mariadb issue is unrelated and mostly resolved now (we deployed patched version yesterday)

Can you confirm if it’s just a dns issue by trying any public DNS like google/cisco/cloudflare?

Otherwise I’d suggest contacting your ISP for authoritative answer on why sites are not accessible.

Let us know if you need some information from our side. (DM me on telegram @ankushmnt or email on ankush(at)frappe.io )