Executing frappe.patches.v12_0.setup_tags throws pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

aldoblack · March 16, 2022, 3:46pm

I am trying to migrate database from version 11 to version 13. During migration I get this error when Executing frappe.patches.v12_0.setup_tags is running

Traceback (most recent call last):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 91, in execute_patch
    frappe.get_attr(patchmodule.split()[0] + ".execute")()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/patches/v12_0/setup_tags.py", line 15, in execute
    if not frappe.db.count(doctype.name) or not frappe.db.has_column(doctype.name, "_user_tags"):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 882, in count
    count = self.sql("""select count(*)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 152, in sql
    self._cursor.execute(query)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/connections.py", line 692, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/connections.py", line 748, in _read_bytes
    raise err.OperationalError(
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 104, in <module>
    main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 19, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 27, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py", line 309, in migrate
    migrate(
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 69, in migrate
    frappe.modules.patch_handler.run_all(skip_failing)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 41, in run_all
    run_patch(patch)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 30, in run_patch
    if not run_single(patchmodule = patch):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 71, in run_single
    return execute_patch(patchmodule, method, methodargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 97, in execute_patch
    frappe.db.rollback()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 810, in rollback
    self.sql("rollback")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 152, in sql
    self._cursor.execute(query)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/connections.py", line 547, in query
    self._execute_command(COMMAND.COM_QUERY, sql)
  File "/home/frappe/frappe-bench/env/lib/python3.8/site-packages/pymysql/connections.py", line 793, in _execute_command
    raise err.InterfaceError(0, "")
pymysql.err.InterfaceError: (0, '')

Not sure why it fails to connect to the database.

smino · March 16, 2022, 4:16pm

Hi,

It may help to say the operating environment, for example is this a containerized instance? Does bench mariadb work? Does bench show-config have the correct mariadb host?

aldoblack · March 16, 2022, 4:28pm

No. It’s not a containerized instance. It’s standard on Ubuntu 20.04 (LTS) x64.
Yes, bench mariadb works. I can access database.
Yes, bench show-config, works as well.

smino · March 16, 2022, 4:47pm

What command results in the error? Or is it a script?

aldoblack · March 16, 2022, 5:15pm

It’s bench --site <site_name> migrate . The error is thrown on Executing frappe.patches.v12_0.setup_tags in <site_name> (<db_name>)

I took a look at frappe.patches.v12_0.setup_tags.py and did not see any code out of the ordinary.

smino · March 16, 2022, 5:19pm

What does bench version return
Have you tried bench update --patch ?

aldoblack · March 16, 2022, 5:26pm

When I execute bench version I get this:

frappe 13.22.1
erpnext 13.22.1

Going to apps/frappe and executing git branch, it’s pulled from branch version-13. The same goes for apps/erpnext. It’s branch version-13 .

No, I have not tried bench update --patch . On previous versions, every time I have executed this command, it broke everything and I had to restart the update from scratch. I am a little worried it might happen again.

aldoblack · March 16, 2022, 5:55pm

UPDATE: I just executed bench update --patch and it’s the same error. No fix.

smino · March 16, 2022, 9:17pm

Similar problem in this thread, I wonder if @trentmu 's idea of commenting out the patch would get you further along?

aldoblack · March 17, 2022, 2:29pm

Thank you for your time. Hopefully @trentmu replies.

aldoblack · March 17, 2022, 2:41pm

UPDATE: I commented out but there is still another bug. It’s identical as this Move from v11 to 13 fails - #5 by leonp .

The command is Executing erpnext.patches.v13_0.patch_to_fix_reverse_linking_in_additional_salary_encashment_and_incentive in <site_name> (<db_name>)

smino · March 17, 2022, 2:59pm

If what @dj12djdjs found is a bug, then we may have to wait for it to be fixed in order to proceed. It might be worth trying bench migrate with the –skip-failing option to see if that get things along any further.

dj12djdjs · March 17, 2022, 10:43pm

@aldoblack Don’t comment out the patch. The problem is with database. The error is connection lost during query. This means the database has reached it’s query timeout limit. Look in your mariadb config and increase the timeout.

aldoblack · March 18, 2022, 1:11pm

That is a great idea and it worked. But, unfortunately I am having the other error that I mentioned above.

Command:

Executing erpnext.patches.v13_0.patch_to_fix_reverse_linking_in_additional_salary_encashment_and_incentive in <site_name> (db_name)

Error:

...
pymysql.err.OperationalError: (1054, "Unknown column 'additional_salary' in 'field list'")

dj12djdjs · March 18, 2022, 9:55pm

This is a bug. I’ve seen it reported here before
Here is the reference to the github issue, though frappe team hasn’t put any attention to it yet.
I tried looking through the history to see when the field got removed. I could not find the commit that removed it.
https://github.com/frappe/erpnext/issues/29841

akkalame_ereut · April 25, 2023, 7:58pm

Hi mate, i have the same issue and this is my configuration of timeout:
±--------------------------------------±---------+
| Variable_name | Value |
±--------------------------------------±---------+
| connect_timeout | 5 |
| deadlock_timeout_long | 50000000 |
| deadlock_timeout_short | 10000 |
| delayed_insert_timeout | 300 |
| idle_readonly_transaction_timeout | 0 |
| idle_transaction_timeout | 0 |
| idle_write_transaction_timeout | 0 |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 86400 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_slave_kill_conn_timeout | 5 |
| slave_net_timeout | 60 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
±--------------------------------------±---------+

can you talk me any idea?