Should patches be applied on a new site?

Creating a new site, and running bench migrate over the same causes patches as old as v4 to be executed. If someone installs 12.x.x from scratch, should they install all patches from v4 to v11?

Maybe I’m missing something here. Should this behavoiur be deprecated? Or should we have explicit forward compatibility in all the patches (Which seems slightly impossible).

  • I created new-site on v12 and did bench update.
  • I created new-site on v12, switched to develop and did bench update.

I don’t see old patches when I run bench update

cd frappe@9d824cfaf05d:/workspace/development$ cd frappe-bench/
frappe@9d824cfaf05d:/workspace/development/frappe-bench$ bench new-site v12toedge.localhost
MySQL root password: 

Installing frappe...
Updating DocTypes for frappe        : [========================================]
Updating country info               : [========================================]
Set Administrator password: 
Re-enter Administrator password: 
*** Scheduler is disabled ***
frappe@9d824cfaf05d:/workspace/development/frappe-bench$ bench update --no-backup
INFO:bench.app:pulling frappe
$ git pull  upstream version-12
From https://github.com/frappe/frappe
 * branch                version-12 -> FETCH_HEAD
Already up to date.
$ find . -name "*.pyc" -delete
INFO:bench.app:pulling erpnext
$ git pull  upstream version-12
From https://github.com/frappe/erpnext
 * branch                  version-12 -> FETCH_HEAD
Already up to date.
$ find . -name "*.pyc" -delete
Updating Python libraries...
$ /workspace/development/frappe-bench/env/bin/pip install -q -U pip
INFO:bench.app:installing frappe
$ /workspace/development/frappe-bench/env/bin/pip install -q -U -e /workspace/development/frappe-bench/apps/frappe 
INFO:bench.app:installing erpnext
$ /workspace/development/frappe-bench/env/bin/pip install -q -U -e /workspace/development/frappe-bench/apps/erpnext 
Updating node packages...

Installing node dependencies for frappe
$ yarn install
yarn install v1.22.4
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.49s.

Installing node dependencies for erpnext
$ yarn install
yarn install v1.22.4
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.08s.
Patching sites...
Migrating v12toedge.localhost
Updating DocTypes for frappe        : [========================================]
Generating Website Theme Files...
Compiling Python Files...
$ bench build
yarn run v1.22.4
$ FRAPPE_ENV=production node rollup/build.js
Production mode
✔ Built js/moment-bundle.min.js
✔ Built js/libs.min.js

Building frappe assets...

✔ Built js/checkout.min.js
✔ Built js/dialog.min.js
✔ Built js/modules.min.js
✔ Built js/social.min.js
✔ Built js/web_form.min.js
✔ Built js/chat.js
✔ Built js/list.min.js
✔ Built css/frappe-rtl.css
✔ Built css/printview.css
✔ Built css/form.min.css
✔ Built css/report.min.css
✔ Built css/list.min.css
✔ Built frappe/css/email.css
✔ Built css/module.min.css
✔ Built css/frappe-chat-web.css
✔ Built js/bootstrap-4-web.min.js
✔ Built js/barcode_scanner.min.js
✔ Built js/frappe-recorder.min.js
✔ Built js/desk.min.js
✔ Built js/frappe-web.min.js
✔ Built css/web_form.css
✔ Built css/desk.min.css
✔ Built css/frappe-web-b4.css
✔ Built js/control.min.js
✔ Built js/form.min.js
✔ Built js/data_import_tools.min.js
✔ Built js/report.min.js
✨  Done in 33.584s

Building erpnext assets...

✔ Built js/erpnext-web.min.js
✔ Built css/marketplace.css
✔ Built css/erpnext.css
✔ Built js/item-dashboard.min.js
✔ Built css/erpnext-web.css
✔ Built js/erpnext.min.js
✔ Built js/marketplace.min.js
✨  Done in 9.034s
Done in 43.51s.
________________________________________________________________________________
Bench: Deployment tool for Frappe and Frappe Applications (https://frappe.io/bench).
Open source depends on your contributions, so do give back by submitting bug reports, patches and fixes and be a part of the community :)
frappe@9d824cfaf05d:/workspace/development/frappe-bench$ bench switch-to-develop
INFO: Fetching upstream unshallow for frappe
$ git remote set-branches upstream  '*'
$ git fetch --all --unshallow
Fetching upstream
remote: Enumerating objects: 254, done.
remote: Counting objects: 100% (155/155), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 80 (delta 67), reused 62 (delta 49), pack-reused 0
Unpacking objects: 100% (80/80), done.
Switching for frappe
$ git checkout develop
Switched to branch 'develop'
Your branch is behind 'upstream/develop' by 329 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
INFO: Fetching upstream unshallow for erpnext
$ git remote set-branches upstream  '*'
$ git fetch --all --unshallow
Fetching upstream
remote: Enumerating objects: 5253, done.
remote: Counting objects: 100% (3277/3277), done.
remote: Compressing objects: 100% (814/814), done.
remote: Total 2091 (delta 1675), reused 1651 (delta 1259), pack-reused 0
Receiving objects: 100% (2091/2091), 8.53 MiB | 2.01 MiB/s, done.
Resolving deltas: 100% (1675/1675), completed with 441 local objects.
From https://github.com/frappe/erpnext
   f411e754ae..e720882410  develop                     -> upstream/develop
 * [new branch]            mergify/bp/develop/pr-21635 -> upstream/mergify/bp/develop/pr-21635
 + c5f40d9072...178ad9b4d6 sahil28297-patch-1          -> upstream/sahil28297-patch-1  (forced update)
 * [new branch]            sahil28297-patch-2          -> upstream/sahil28297-patch-2
   7ed7c3237a..626585e9f3  v12-pre-release             -> upstream/v12-pre-release
   fdd93cba03..c6466c7d97  version-11-hotfix           -> upstream/version-11-hotfix
   bd4b5da11b..767d17ce2f  version-12-hotfix           -> upstream/version-12-hotfix
 * [new branch]            version-13-beta             -> upstream/version-13-beta
   168babfebc..e946699c3e  version-13-beta-pre-release -> upstream/version-13-beta-pre-release
 * [new tag]               v12.8.0                     -> v12.8.0
 * [new tag]               v13.0.0-beta.1              -> v13.0.0-beta.1
Switching for erpnext
$ git checkout develop
Switched to branch 'develop'
Your branch is behind 'upstream/develop' by 159 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
SUCCESS: Successfully switched branches for: frappe, erpnext
Please run `bench update --patch` to be safe from any differences in database schema
Updating Python libraries...
$ ./env/bin/pip install -q -U pip
INFO:bench.app:installing frappe
$ ./env/bin/pip install -q -U -e ./apps/frappe 
INFO:bench.app:installing erpnext
$ ./env/bin/pip install -q -U -e ./apps/erpnext 
Updating node packages...

Installing node dependencies for frappe
$ yarn install
yarn install v1.22.4
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > bootstrap@4.4.1" has unmet peer dependency "jquery@1.9.1 - 3".
warning " > bootstrap@4.4.1" has unmet peer dependency "popper.js@^1.16.0".
[4/4] Building fresh packages...
Done in 144.05s.

Installing node dependencies for erpnext
$ yarn install
yarn install v1.22.4
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ yarn run snyk-protect
yarn run v1.22.4
$ snyk protect
Successfully applied Snyk patches

Done in 2.38s.
Done in 6.13s.
Traceback (most recent call last):
  File "/home/frappe/.local/bin/bench", line 11, in <module>
    load_entry_point('bench==5.0.0', 'console_scripts', 'bench')()
  File "/home/frappe/.local/lib/python3.7/site-packages/bench/cli.py", line 58, in cli
    bench_command()
  File "/home/frappe/.local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/.local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/frappe/.local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/.local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/.local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/.local/lib/python3.7/site-packages/bench/commands/update.py", line 53, in switch_to_develop
    switch_to_develop(apps=['frappe', 'erpnext'])
  File "/home/frappe/.local/lib/python3.7/site-packages/bench/app.py", line 424, in switch_to_develop
    switch_branch('develop', apps=apps, bench_path=bench_path, upgrade=upgrade)
  File "/home/frappe/.local/lib/python3.7/site-packages/bench/app.py", line 410, in switch_branch
    reload_module(utils)
NameError: name 'utils' is not defined
frappe@9d824cfaf05d:/workspace/development/frappe-bench$ bench update --patch
This update will cause a major version change in Frappe/ERPNext from 12 to 13. 
This would take significant time to migrate and might break custom apps.
Do you want to continue? [y/N]: y
Backing up sites...
Patching sites...
Migrating v12toedge.localhost
Executing execute:frappe.reload_doc('core', 'doctype', 'doctype_action', force=True) #2019-09-23 in v12toedge.localhost (_0bf84a7f929fe3ad)
Success: Done in 0.107s
Executing execute:frappe.reload_doc('core', 'doctype', 'doctype_link', force=True) #2019-09-23 in v12toedge.localhost (_0bf84a7f929fe3ad)
Success: Done in 0.074s
Executing frappe.patches.v12_0.change_existing_dashboard_chart_filters in v12toedge.localhost (_0bf84a7f929fe3ad)
Success: Done in 0.046s
Executing execute:frappe.delete_doc("Test Runner") in v12toedge.localhost (_0bf84a7f929fe3ad)
Success: Done in 0.018s
Executing execute:frappe.db.set_default('desktop:home_page', 'workspace') in v12toedge.localhost (_0bf84a7f929fe3ad)
Success: Done in 0.034s
Executing frappe.patches.v13_0.website_theme_custom_scss in v12toedge.localhost (_0bf84a7f929fe3ad)
Success: Done in 0.275s
Executing frappe.patches.v13_0.set_existing_dashboard_charts_as_public in v12toedge.localhost (_0bf84a7f929fe3ad)
Syntax error in query:

                UPDATE
                        `tabDashboard Chart`
                SET
                        `tabDashboard Chart`.`is_public`=1
                WHERE
                        `tabDashboard Chart`.owner in ('Administrator',)

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/workspace/development/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 25, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/commands/site.py", line 252, in migrate
    migrate(context.verbose, rebuild_website=rebuild_website, skip_failing=skip_failing)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/migrate.py", line 49, in migrate
    frappe.modules.patch_handler.run_all(skip_failing)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 41, in run_all
    run_patch(patch)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 30, in run_patch
    if not run_single(patchmodule = patch):
  File "/workspace/development/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 71, in run_single
    return execute_patch(patchmodule, method, methodargs)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 91, in execute_patch
    frappe.get_attr(patchmodule.split()[0] + ".execute")()
  File "/workspace/development/frappe-bench/apps/frappe/frappe/patches/v13_0/set_existing_dashboard_charts_as_public.py", line 28, in execute
    """.format(users=users)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/database/database.py", line 173, in sql
    self._cursor.execute(query)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
    result = self._query(query)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/cursors.py", line 328, in _query
    conn.query(q)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 6")
frappe@9d824cfaf05d:/workspace/development/frappe-bench$ 

There were other patch errors because of using develop branch. v12 is stable and no migration errors or old patches observed.

1 Like

Maybe i really am missing something here. I guess i’ll update my bench once and things should be fine. Thanks.

I have some experience in this area. Here’s how it works.

  • All the patch files, version 4 to latest, are located in …/apps/<app_name>/<app_name>/patches.
    They are Python scripts. And there are many hundreds of them.

  • There is a SQL table named ‘tabPatch Log’. It contains the names of previously executed patches.

SELECT name, patch, creation
FROM myERPNextDatabase.`tabPatch Log`;

When you execute a bench update, it loops through every patch file. Does this file exist already in tabPatch Log?

If no? Then Bench executes that patch. When complete, it writes a new SQL record in tabPatch Log.
If yes? Then it skips the patch.

If old patches are being executed by mistake, there might be a problem with your SQL table. Make sure you follow all the installation steps precisely, and don’t skip any steps.

1 Like

No wonder an update takes so danged long!

BKM

Yep. Link to the actual code below. Relevant lines are 22 through 41.

  1. A Python List named “executed” contains every patch previously run (fetched from MySQL)
  2. A Python List named “patches” contains the name of every patch since v4.
  3. Loop through “patches” one-at-time. If that patch exists in “executed”, skip it.
  4. Rinse and repeat several hundred times.

Certainly there’s room for performance improvement. By improving the SQL patch table and doing some joins, you could loop through only new patches. Instead of all of them.

https://github.com/frappe/frappe/blob/develop/frappe/modules/patch_handler.py

1 Like