Google drive backups suddenly failing

Hi

I have a multi-tenant system that has been working for a considrable time and suddenly my google-drive backups are failing.

I dont actually do any modifications / addaptions / cusotmisations on these systems.
I just use it. The only big work I did on it is to upgrade it from V13 to V14 at the beginning of august. The intention was to next upgrade from V14 to V15.

I noticed now that my backups have not uploaded for the past few days.

When I open one of these

Traceback (most recent call last):
  File "env/lib/python3.10/site-packages/googleapiclient/discovery.py", line 35, in <module>
    from email.generator import BytesGenerator
  File "/usr/lib/python3.10/email/generator.py", line 17, in <module>
    from email.errors import HeaderWriteError
ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.10/email/errors.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/core/doctype/scheduled_job_type/scheduled_job_type.py", line 117, in execute
    frappe.get_attr(self.method)()
  File "apps/frappe/frappe/__init__.py", line 1609, in get_attr
    return getattr(get_module(modulename), methodname)
  File "apps/frappe/frappe/__init__.py", line 1343, in get_module
    return importlib.import_module(modulename)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "apps/frappe/frappe/integrations/doctype/google_drive/google_drive.py", line 7, in <module>
    from apiclient.http import MediaFileUpload
  File "env/lib/python3.10/site-packages/apiclient/__init__.py", line 8, in <module>
    from googleapiclient import discovery
  File "env/lib/python3.10/site-packages/googleapiclient/discovery.py", line 37, in <module>
    from email.generator import Generator as BytesGenerator
  File "/usr/lib/python3.10/email/generator.py", line 17, in <module>
    from email.errors import HeaderWriteError
ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.10/email/errors.py)

This would suggest an email problem and yet I can send docs via email.

I have re-authorised GOogle drive …

Goto "Google Drive"
Click "Authorise Google Drive Access"

When I click on “Take Backup” it says “Backup is queued”, but no backup is stored
on Google drive

The scheduled backups under “Download Backups” are still done.
Error log list shows no error

My observations are the same for the other sites on this multi tenant system.

ERPNext: v14.70.9 (version-14)

Frappe Framework: v14.77.5 (version-14)

Frappe HR: v14.28.8 (version-14)

I logged in on the backend and looked that the logs in /frappe-bench/logs
schedule.log is empty
Schedule.error.log is empty

There are a few entries in scheduler.log

Note ; I updated to V14 on 1 Jul

5 Jul : 3 errors sugesting that tabSingles doesnt exist ( 1 error for each site )

30 Jul : an error about Unexpected job format last heartbeat.

The only link I can find between the “tabSingles” doesn’t exist error and backups
is that tabSingles seem to store Settings.

The table does exist but I do not see “Google-drive” settings in this table ???
And yet the Google-drive settings are visible via the Desk ???

I dont know where else to look to see what the problem is. Would really
appreciate some assistance / opinions / suggestions

Anyone that can perhaps help with an suggestion ??
I still have my version 13 system but I have to many entries to go back to it.
I will have to back-track 2 months of entries and enter it into the V13 system to go back to a working system.

Take latest pull of HRMS and then check it

Regarding the error you posted, this seems to be a problem in the googleapiclient. Just deploying the latest update should resolve it. (If you don’t use bench update, make sure the python dependencies are updated/reinstalled).

Thank you @rmeyer and @sahilsuthar for your suggestions.

As I explained, what I set out to do is to update from V13 to V14 to V15. I completed
the first step beginning of August. So if I understand your suggestions correctly,
then simply continuing on my upgrade-process is the solution … because in doing
that, I shall ensure that HRMS and python dependencies are updated.

May I thank you both for assisting.

You don’t necessarily need to upgrade to v15 to resolve this error, just to the latest v14 release.

Thank you @rmeyer for your comemnt.
I understand. I did that and the issue is resolved.

I see that the wording on my post could have been better. I just see the upgrade to V15
as the next step after fixing the issue.

Thank you again for your time.