V7 Migration Error

camilopianos · December 2, 2016, 5:47pm

I got this error. Restore was doing well. Please help!

/frappe-bench$ bench --site site2.local migrate
Migrating site2.local
Executing frappe.patches.v7_0.update_auth in site2.local (a46003081c220c38)
Traceback (most recent call last):
File “/usr/lib/python2.7/runpy.py”, line 174, in _run_module_as_main
“main”, fname, loader, pkg_name)
File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 79, in
main()
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 16, in main
click.Group(commands=commands)(prog_name=‘bench’)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/init.py”, line 24, in _func
ret = f(frappe._dict(ctx.obj), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py”, line 209, in migrate
migrate(context.verbose, rebuild_website=rebuild_website)
File “/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py”, line 30, in migrate
frappe.modules.patch_handler.run_all()
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 29, in run_all
if not run_single(patchmodule = patch):
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 63, in run_single
return execute_patch(patchmodule, method, methodargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 83, in execute_patch
frappe.get_attr(patchmodule.split()[0] + “.execute”)()
File “/home/frappe/frappe-bench/apps/frappe/frappe/patches/v7_0/update_auth.py”, line 21, in execute
frappe.reload_doctype(doctype)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 633, in reload_doctype
reload_doc(scrub(db.get_value(“DocType”, doctype, “module”)), “doctype”, scrub(doctype), force=force)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 645, in reload_doc
return frappe.modules.reload_doc(module, dt, dn, force=force)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/utils.py”, line 141, in reload_doc
return import_files(module, dt, dn, force=force)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 17, in import_files
return import_file(module, dt, dn, force=force, pre_process=pre_process)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 22, in import_file
ret = import_file_by_path(path, force, pre_process=pre_process)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 54, in import_file_by_path
import_doc(doc, force=force, data_import=data_import, pre_process=pre_process)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 114, in import_doc
frappe.delete_doc(doc.doctype, doc.name, force=1, ignore_doctypes=ignore, for_reload=True)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 624, in delete_doc
ignore_permissions, flags)
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/delete_doc.py”, line 38, in delete_doc
remove_all(doctype, name)
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/file_manager.py”, line 235, in remove_all
attached_to_doctype=%s and attached_to_name=%s"“”, (dt, dn)):
File “/home/frappe/frappe-bench/apps/frappe/frappe/database.py”, line 205, in sql_list
return [r[0] for r in self.sql(query, values, debug=debug)]
File “/home/frappe/frappe-bench/apps/frappe/frappe/database.py”, line 137, in sql
self._cursor.execute(query, values)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/cursors.py”, line 205, in execute
self.errorhandler(self, exc, value)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/connections.py”, line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1146, “Table ‘a46003081c220c38.tabFile’ doesn’t exist”)

clarkej · December 2, 2016, 11:29pm

I am not familiar with migrate. But print statements would debug the identify and path of the file that is the source of the problem. For eg. add this to /home/frappe/frappe-bench/apps/frappe/frappe/model/delete_doc.py", line 38

            # delete attachments                                                                                                  
            print doctype + '  ' + name
            remove_all(doctype, name)

Rerun the migration and report back what you find out…

camilopianos · December 2, 2016, 11:56pm

thank you clarkej. I did the add and run migrate again:Exact the same results…

Question: on v6, updates are automatic? I was thinking, before I backup the system, should I update manualy?
I’m not a programmer! Very little idea about.

clarkej · December 3, 2016, 4:17pm

Same print output to console? Then your delete_doc.py edit did not recompile and run as a new .pyc

Your file timestamps should confirm this. For example in my case:

frappe@erpnext:~/frappe-bench$ find . -name delete_doc.* | xargs ls -al
-rw-rw-r-- 1 frappe frappe  9409 Oct 12 14:08 ./apps/frappe/frappe/model/delete_doc.py
-rw-rw-r-- 1 frappe frappe 10261 Nov  6 11:29 ./apps/frappe/frappe/model/delete_doc.pyc

So delete your stale old pyc:

frappe@erpnext:~/frappe-bench$ rm ./apps/frappe/frappe/model/delete_doc.pyc

Rerun the migrate now should recompile the py to run the new pyc for the expected print out.

‘Question: on v6, updates are automatic? I was thinking, before I backup the system, should I update manualy?’

I can’t say what you aim to do. Yes, always backup your data before you migrate or update. You need to make sense of fundamental bench commands.
These two affect database state ie ‘an sql snapshot of the data at a point in time’

backup 
restore

These affect both the code and also the database state:

update
migrate

Your traceback and exceptions may mean the database (ie the data model schema and contents) and Python code (that operate or function on the data) may be ‘out of sync’ ie their versions mismatch or are inconsistent with eachother.

camilopianos · December 5, 2016, 11:09am

Thank you klarkej! but same output to console.
I grant frappe access to db in /var/lib/mysql/a46003081c220c38
Nothing change. Same output. How could I check for inconsistencies?
I appreciate your help a lot!

clarkej · December 5, 2016, 3:45pm

Sorry my error Camilo, I see now that I assumed wrong.

I ran a test just now with this learning - after my edit to a .py, I deleted the stale .pyc but like you
no sign of my expected change output to the console

See if ‘bench clear-cache’ solves your no print output problem…

rmehta · December 5, 2016, 3:46pm

try

bench --site [sitename] reload-doctype File

and then bench update again

camilopianos · December 5, 2016, 5:45pm

I get this rmehta. Any ideas? Some more info:
Ubuntu 16.04 on VM. Fresh install of v7. Thank you!

frappe@erp:~/frappe-bench$ bench --site site2.local reload-doctype File
Traceback (most recent call last):
File “/usr/lib/python2.7/runpy.py”, line 174, in _run_module_as_main
“main”, fname, loader, pkg_name)
File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 79, in
main()
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 16, in main
click.Group(commands=commands)(prog_name=‘bench’)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/init.py”, line 24, in func
ret = f(frappe.dict(ctx.obj), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py”, line 252, in reload_doctype
frappe.reload_doctype(doctype, force=context.force)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 633, in reload_doctype
reload_doc(scrub(db.get_value(“DocType”, doctype, “module”)), “doctype”, scrub(doctype), force=force)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 658, in scrub
return txt.replace(’ ‘,’‘).replace(’-', '').lower()
AttributeError: ‘NoneType’ object has no attribute ‘replace’

rmehta · December 5, 2016, 6:14pm

try

bench --site sitename reload-doc DocType File

camilopianos · December 5, 2016, 7:31pm

bench --site site2.local reload-doc DocType File
File =path to a File? because I get this:

Usage: bench reload-doc [OPTIONS] MODULE DOCTYPE DOCNAME

Error: Missing argument “docname”.

Very sorry: _mysql_exceptions.ProgrammingError: (1146, “Table ‘a46003081c220c38.tabFile’ doesn’t exist”)
frappe has access to DB a46003081c220c38 with 775…
Any ideas?

clarkej · December 6, 2016, 1:40am

ok I grok where rmehta is going with this…reload expects three arguments so try this:

frappe@erpnext:~/frappe-bench$ bench reload-doc Core DocType File

As a cross check, try this before and after you run the above command:

frappe@erpnext:~/frappe-bench$ bench mysql

MariaDB [d56cb677eaab3383]> desc tabFile;

MariaDB [d56cb677eaab3383]> exit
Bye
frappe@erpnext:~/frappe-bench$'exit' to exit Mysql and return to the bench prompt.

You wrote ‘frappe has access to DB a46003081c220c38 with 775’

FYI my file system permissions are:

frappe@erpnext:~/frappe-bench$ ls -al /var/lib/mysql/
total 1151576
drwxr-xr-x  5 mysql mysql      4096 Dec  2 06:53 .
drwxr-xr-x 51 root  root       4096 Nov 18 10:50 ..
-rw-rw----  1 mysql mysql     16384 Nov 21 06:14 aria_log.00000001
-rw-rw----  1 mysql mysql        52 Nov 21 06:14 aria_log_control
drwx------  2 mysql mysql     36864 Nov 24 07:31 d56cb677eaab3383

and

frappe@erpnext:~/frappe-bench$ sudo find /var/lib/mysql -name tabFile.* | xargs sudo ls -al
-rw-rw---- 1 mysql mysql 14803 Nov  3 16:36 /var/lib/mysql/d56cb677eaab3383/tabFile.frm
-rw-rw---- 1 mysql mysql 90112 Nov 22 14:07 /var/lib/mysql/d56cb677eaab3383/tabFile.ibd

camilopianos · December 6, 2016, 10:15pm

Something new after bench reload and desc tabFile.
Should I commit and please how, since I’m not a git expert…love it already but need time…
Thank you clarkej!

Updating 2f8fc3e…dd53ed0
error: Your local changes to the following files would be overwritten by merge:
frappe/init.py
frappe/core/doctype/error_log/error_log.py
frappe/core/doctype/report/report.py
frappe/core/doctype/translation/translation.json
frappe/core/report/document_share_report/document_share_report.json
frappe/desk/query_report.py
frappe/desk/reportview.py
frappe/email/doctype/auto_email_report/auto_email_report.py
frappe/model/db_query.py
frappe/model/delete_doc.py
frappe/model/rename_doc.py
frappe/patches.txt
frappe/patches/v7_0/add_communication_in_doc.py
frappe/public/js/frappe/model/create_new.js
frappe/public/js/frappe/views/reports/reportview.js
Please, commit your changes or stash them before you can merge.
Aborting
Traceback (most recent call last):
File “/usr/local/bin/bench”, line 11, in
load_entry_point(‘bench’, ‘console_scripts’, ‘bench’)()
File “/home/frappe/.bench/bench/cli.py”, line 40, in cli
bench_command()
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/home/frappe/.bench/bench/commands/update.py”, line 62, in update
_update(pull, patch, build, bench, auto, restart_supervisor, requirements, no_backup, upgrade, force=force, reset=reset)
File “/home/frappe/.bench/bench/commands/update.py”, line 78, in _update
pull_all_apps(bench_path=bench_path, reset=reset)
File “/home/frappe/.bench/bench/app.py”, line 146, in pull_all_apps
remote=remote, branch=get_current_branch(app, bench_path=bench_path)), cwd=app_dir)
File “/home/frappe/.bench/bench/utils.py”, line 127, in exec_cmd
raise CommandFailedError(cmd)
bench.utils.CommandFailedError: git pull upstream master

clarkej · December 7, 2016, 2:35pm

Well done! Yes learning takes time and is the basic currency of this forum.

Now if you follow the pointers in your merge log you are on your way here

clarkej · December 7, 2016, 3:25pm

I just stumbled on this and it’s perfect for your case

Got 15 minutes and want to learn Git?
https://try.github.io/levels/1/challenges/1

rmehta · December 9, 2016, 6:17am

You have local changes in your repo

camilopianos · December 12, 2016, 9:14am

Please help!
how to handle this message:
Executing frappe.patches.v6_2.rename_backup_manager in site1.local (1bd3e0294da19198)
/home/frappe/frappe-bench/apps/frappe/frappe/integrations/doctype/dropbox_backup/dropbox_backup.json missing

Folder dropbox_backup and .json are not there… Something went wrong with the installation? Module not installed?

I would appreciate since I have not a clue and looking and at the forum I don’t found any solution.

Thanks for your advice and help!
Now erpnext and frappe, git status: working directory clean. I did it with git stash. Are there other repositotries to consider? Still getting following error after bench migrate:
Migrating site1.local
Executing frappe.patches.v6_2.rename_backup_manager in site1.local (1bd3e0294da19198)
/home/frappe/frappe-bench/apps/frappe/frappe/integrations/doctype/dropbox_backup/dropbox_backup.json missing
Module import failed for Dropbox Backup (frappe.core.doctype.dropbox_backup.dropbox_backup)
Traceback (most recent call last):
File “/usr/lib/python2.7/runpy.py”, line 174, in _run_module_as_main
“main”, fname, loader, pkg_name)
File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 79, in
main()
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 16, in main
click.Group(commands=commands)(prog_name=‘bench’)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/init.py”, line 24, in _func
ret = f(frappe._dict(ctx.obj), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py”, line 209, in migrate
migrate(context.verbose, rebuild_website=rebuild_website)
File “/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py”, line 30, in migrate
frappe.modules.patch_handler.run_all()
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 29, in run_all
if not run_single(patchmodule = patch):
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 63, in run_single
return execute_patch(patchmodule, method, methodargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 83, in execute_patch
frappe.get_attr(patchmodule.split()[0] + “.execute”)()
File “/home/frappe/frappe-bench/apps/frappe/frappe/patches/v6_2/rename_backup_manager.py”, line 7, in execute
dropbox_backup = frappe.get_doc(“Dropbox Backup”, “Dropbox Backup”)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 589, in get_doc
return frappe.model.document.get_doc(arg1, arg2)
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py”, line 45, in get_doc
controller = get_controller(doctype)
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/base_document.py”, line 34, in get_controller
module = load_doctype_module(doctype, module_name)
File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/utils.py”, line 171, in load_doctype_module
doctype_python_modules[key] = frappe.get_module(module_name)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 654, in get_module
return importlib.import_module(modulename)
File “/usr/lib/python2.7/importlib/init.py”, line 37, in import_module
import(name)
ImportError: No module named dropbox_backup.dropbox_backup

clarkej · December 16, 2016, 12:41am

Before you do anything first recognize what it is you wish to do…

Once you identify that these should help get you there:

‘list the installed modules’
frappe@erpnext:~/frappe-bench$ sudo pip list

frappe@erpnext:~/frappe-bench$ sudo pip install dropbox_backup

frappe@erpnext:~/frappe-bench$ sudo pip list | grep dropbox-backup
dropbox-backup (0.1.6)

In my case Dropbox Settings is not set and the module was not installed.

I also found this ImportError: No module named bleach

camilopianos · December 16, 2016, 4:15pm

Thank you so much for your help!
Please allow me some questions.

at the time of installation I got an error and may be this was why something went wrong:

The directory ‘/home/cami/.cache/pip/http’ or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.

I asume now it is a matter of permissions…
from your message I see that frappe user has sudo privileges.

Should I grant frappe sudo privileges and evetualy install dropbox_backup as frappe with sudo if not listed in pip list?

Now I’m using bench as frappe user in a production enviroment but I could not access as user frappe in other instance for development (just in order to learn about develompment): No passwd entry for user ‘frappe’

I assume frappe user is not created in development. Is this wright? Or should I create a frappe user for the installation?

I appreciate a lot!

camilopianos · December 17, 2016, 10:53pm

I installed dropbox-backup (0.1.6) but the ImportError remains: No module named dropbox_backup.dropbox_backup

Also following message apears after
pip list | grep dropbox-backup

The directory ‘/home/frappe/.cache/pip/http’ or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.

But /home owner is frappe:
drwxr-xr-x 4 frappe frappe 4096 Dec 16 21:34 home

Please HELP!