Not Found / Site does not exist - After RPC

Frappe Framework
,
#1

Hi guys,

Whenever I execute a RPC for an argumentless server side script, I get a 200, the script executes successfully, but then the entire system goes down. I have to perform a supervisorctl restart all for it to work again.

Here’s script:

@frappe.whitelist(allow_guest=True, methods=['PUT'])
def update_mdf():
    os.chdir('/home/frappe/frappe-bench/apps/erpnext/erpnext')
    last_start_date = frappe.db.sql("""SELECT MAX(activity_start_date) FROM tabMDF;""")
    df = pd.read_csv('data_files/result-activities.csv')
    df['Activity Start Date'] = pd.to_datetime(df['Activity Start Date'], format="%d/%m/%Y")
    df['Activity End Date'] = pd.to_datetime(df['Activity End Date'], format="%d/%m/%Y")
    df = df[df['Activity Start Date'] >= pd.to_datetime(last_start_date[0][0], format="%Y-%m-%d")]
    df = df[['Activity ID',
             'Activity Name',
             'Activity Description',
             'Activity Type',
             'Activity Start Date',
             'Activity End Date'
             ]]

    df.rename(columns={'Activity ID': 'activity_id',
                       'Activity Name': 'activity_name',
                       'Activity Description': 'activity_description',
                       'Activity Type': 'activity_type',
                       'Activity Start Date': 'activity_start_date',
                       'Activity End Date': 'activity_end_date'}, inplace=True)

    rows = df.to_dict(orient='records')
    for row in rows:
        # Insert if doesn't exists
        try:
            doc = frappe.get_doc({'doctype': 'MDF'} | {**row})
            doc.insert(ignore_permissions=True)
        # Update if exists
        except:
            doc = frappe.get_doc('MDF', row['activity_id'])
            for k, v in row.items():
                doc.__setattr__(k, v)
            doc.save(ignore_permissions=True)

Thanks so much in advance!

Some observations:

  • ERPNext 14
  • If I call this script using bench --site site execute erpnext.calls.update_mdf I don’t get any issues
#2

Did you notice anything in web.error.log or web.log from the frappe-bench/logs directory?

#3

Hi!

I’ve cleared the logs, attempted to call the script again, same error. The logs produced the following:

web.error.log:

/usr/lib/python3.10/functools.py:981: UserWarning: The 'filters_config' hook used to add custom operators is not yet implemented in frappe.db.query engine. Use db_query (frappe.get_list) instead.
  val = self.func(instance)

/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/version/version.py:111: FutureWarning: Comparison of Timestamp with datetime.date is deprecated in order to match the standard library behavior. In a future version these will be considered non-comparable. Use 'ts == pd.Timestamp(date)' or 'ts.date() == date' instead.
  elif old_value != new_value:

Web.log:

Traceback (most recent call last):
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 133, in _verify_signature
    h.verify(data[-32:])
  File "env/lib/python3.10/site-packages/cryptography/hazmat/primitives/hmac.py", line 72, in verify
    ctx.verify(signature)
  File "env/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 85, in verify
    raise InvalidSignature("Signature did not match digest.")
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/utils/password.py", line 216, in decrypt
    return cstr(cipher_suite.decrypt(encode(txt)))
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 90, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 151, in _decrypt_data
    self._verify_signature(data)
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 135, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidToken

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 60, in application
    frappe.api.validate_auth()
  File "apps/frappe/frappe/api.py", line 174, in validate_auth
    validate_auth_via_api_keys(authorization_header)
  File "apps/frappe/frappe/api.py", line 234, in validate_auth_via_api_keys
    validate_api_key_secret(api_key, api_secret, authorization_source)
  File "apps/frappe/frappe/api.py", line 249, in validate_api_key_secret
    doc_secret = frappe.utils.password.get_decrypted_password(doctype, doc, fieldname="api_secret")
  File "apps/frappe/frappe/utils/password.py", line 62, in get_decrypted_password
    return decrypt(result[0][0])
  File "apps/frappe/frappe/utils/password.py", line 219, in decrypt
    frappe.throw(_("Encryption key is invalid! Please check site_config.json"))
  File "apps/frappe/frappe/__init__.py", line 522, in throw
    msgprint(
  File "apps/frappe/frappe/__init__.py", line 490, in msgprint
    _raise_exception()
  File "apps/frappe/frappe/__init__.py", line 442, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: Encryption key is invalid! Please check site_config.json

Traceback (most recent call last):
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 133, in _verify_signature
    h.verify(data[-32:])
  File "env/lib/python3.10/site-packages/cryptography/hazmat/primitives/hmac.py", line 72, in verify
    ctx.verify(signature)
  File "env/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 85, in verify
    raise InvalidSignature("Signature did not match digest.")
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/utils/password.py", line 216, in decrypt
    return cstr(cipher_suite.decrypt(encode(txt)))
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 90, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 151, in _decrypt_data
    self._verify_signature(data)
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 135, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidToken

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 60, in application
    frappe.api.validate_auth()
  File "apps/frappe/frappe/api.py", line 174, in validate_auth
    validate_auth_via_api_keys(authorization_header)
  File "apps/frappe/frappe/api.py", line 234, in validate_auth_via_api_keys
    validate_api_key_secret(api_key, api_secret, authorization_source)
  File "apps/frappe/frappe/api.py", line 249, in validate_api_key_secret
    doc_secret = frappe.utils.password.get_decrypted_password(doctype, doc, fieldname="api_secret")
  File "apps/frappe/frappe/utils/password.py", line 62, in get_decrypted_password
    return decrypt(result[0][0])
  File "apps/frappe/frappe/utils/password.py", line 219, in decrypt
    frappe.throw(_("Encryption key is invalid! Please check site_config.json"))
  File "apps/frappe/frappe/__init__.py", line 522, in throw
    msgprint(
  File "apps/frappe/frappe/__init__.py", line 490, in msgprint
    _raise_exception()
  File "apps/frappe/frappe/__init__.py", line 442, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: Encryption key is invalid! Please check site_config.json

Traceback (most recent call last):
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 133, in _verify_signature
    h.verify(data[-32:])
  File "env/lib/python3.10/site-packages/cryptography/hazmat/primitives/hmac.py", line 72, in verify
    ctx.verify(signature)
  File "env/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 85, in verify
    raise InvalidSignature("Signature did not match digest.")
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/utils/password.py", line 216, in decrypt
    return cstr(cipher_suite.decrypt(encode(txt)))
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 90, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 151, in _decrypt_data
    self._verify_signature(data)
  File "env/lib/python3.10/site-packages/cryptography/fernet.py", line 135, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidToken

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 60, in application
    frappe.api.validate_auth()
  File "apps/frappe/frappe/api.py", line 174, in validate_auth
    validate_auth_via_api_keys(authorization_header)
  File "apps/frappe/frappe/api.py", line 234, in validate_auth_via_api_keys
    validate_api_key_secret(api_key, api_secret, authorization_source)
  File "apps/frappe/frappe/api.py", line 249, in validate_api_key_secret
    doc_secret = frappe.utils.password.get_decrypted_password(doctype, doc, fieldname="api_secret")
  File "apps/frappe/frappe/utils/password.py", line 62, in get_decrypted_password
    return decrypt(result[0][0])
  File "apps/frappe/frappe/utils/password.py", line 219, in decrypt
    frappe.throw(_("Encryption key is invalid! Please check site_config.json"))
  File "apps/frappe/frappe/__init__.py", line 522, in throw
    msgprint(
  File "apps/frappe/frappe/__init__.py", line 490, in msgprint
    _raise_exception()
  File "apps/frappe/frappe/__init__.py", line 442, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: Encryption key is invalid! Please check site_config.json

This instance was a backup restore from a Ubuntu 18, ERP13 into Ubuntu 22, ERP14.

I just tried replacing the encryption key on site_config.json from the old instance + sudo supervisorctl restart all, that seemed to have sorted the encryption error, but calling the script still crashes the connection.

On worker.error.log, however, I’m getting the following:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 109, in <module>
    main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name="bench")
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/scheduler.py", line 189, in start_worker
    start_worker(queue, quiet=quiet, rq_username=rq_username, rq_password=rq_password)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 217, in start_worker
    Worker(queues, name=get_worker_name(queue_name)).work(logging_level=logging_level)
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/rq/worker.py", line 644, in work
    self.register_death()
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/rq/worker.py", line 344, in register_death
    p.execute()
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/client.py", line 4019, in execute
    return execute(conn, stack, raise_on_error)
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/client.py", line 3894, in _execute_transaction
    self.parse_response(connection, '_')
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/client.py", line 3977, in parse_response
    result = Redis.parse_response(
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/connection.py", line 470, in read_response
    self.read_from_socket()
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/connection.py", line 452, in read_from_socket
    raise ConnectionError("Error while reading from socket: %s" %
redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/connection.py", line 427, in read_from_socket
    bufflen = recv_into(self._sock, self._buffer)
  File "/home/frappe/frappe-bench/env/lib/python3.10/site-packages/redis/_compat.py", line 75, in recv_into
    return sock.recv_into(*args, **kwargs)
ConnectionResetError: [Errno 104] Connection reset by peer

Could this be related to due socketio spawn error?

frappe-bench-web:frappe-bench-node-socketio: ERROR (spawn error)

I’ve tried sudo bench setup socketio + restart but I couldn’t get it working.

Thanks!!

#4

anybody? :slightly_smiling_face:

#5

Same problem,
restored db from v14 to v14
the site goes down, and on, and when it’s online I can access it for about 10 seconds then goes off again,
all logs are okay except web.error.log
which is showing

[2023-06-21 15:09:32 +0200] [28994] [INFO] Booting worker with pid: 28994
/usr/lib/python3.10/functools.py:981: UserWarning: The 'filters_config' hook used to add custom operators is not yet implemented in frappe.db.query engine. Use db_query (frappe.get_list) instead.
  val = self.func(instance)
/usr/lib/python3.10/functools.py:981: UserWarning: The 'filters_config' hook used to add custom operators is not yet implemented in frappe.db.query engine. Use db_query (frappe.get_list) instead.
  val = self.func(instance)
/usr/lib/python3.10/functools.py:981: UserWarning: The 'filters_config' hook used to add custom operators is not yet implemented in frappe.db.query engine. Use db_query (frappe.get_list) instead.
  val = self.func(instance)
/usr/lib/python3.10/functools.py:981: UserWarning: The 'filters_config' hook used to add custom operators is not yet implemented in frappe.db.query engine. Use db_query (frappe.get_list) instead.
  val = self.func(instance)
/usr/lib/python3.10/functools.py:981: UserWarning: The 'filters_config' hook used to add custom operators is not yet implemented in frappe.db.query engine. Use db_query (frappe.get_list) instead.
  val = self.func(instance)
/usr/lib/python3.10/functools.py:981: UserWarning: The 'filters_config' hook used to add custom operators is not yet implemented in frappe.db.query engine. Use db_query (frappe.get_list) instead.
  val = self.func(instance)