Stuck on "Sorry! We will be back soon.", after Verifying

[xickomesquita]

Hi,

I had an issue, stuck on Verifying…, after trying to use the App (which I can’t use because the login doesn’t happen).

After running a bench restart (ok), and a bench update, I’ve got the following error:

Traceback (most recent call last):
File “/usr/lib/python2.7/logging/init.py”, line 885, in emit
self.flush()
File “/usr/lib/python2.7/logging/init.py”, line 845, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file utils.py, line 255
INFO:bench.utils:updating bench
Traceback (most recent call last):
File “/usr/lib/python2.7/logging/init.py”, line 885, in emit
self.flush()
File “/usr/lib/python2.7/logging/init.py”, line 845, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file utils.py, line 148
INFO:bench.utils:git pull
remote: Counting objects: 84, done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 84 (delta 61), reused 69 (delta 46), pack-reused 0
error: file write error (No space left on device)
fatal: unable to write sha1 file
fatal: unpack-objects failed
Traceback (most recent call last):
File “/usr/local/bin/bench”, line 11, in
load_entry_point(‘bench’, ‘console_scripts’, ‘bench’)()
File “/home/frappe/.bench/bench/cli.py”, line 40, in cli
bench_command()
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/frappe/.bench/bench/commands/update.py”, line 36, in update
update_bench()
File “/home/frappe/.bench/bench/utils.py”, line 260, in update_bench
exec_cmd(“git pull”, cwd=cwd)
File “/home/frappe/.bench/bench/utils.py”, line 159, in exec_cmd
raise CommandFailedError(cmd)
bench.utils.CommandFailedError: git pull

sudo supervisorctl restart all is ok

When I run bench clear-cache, I’ve got:

Traceback (most recent call last):
File “/usr/lib/python2.7/runpy.py”, line 174, in _run_module_as_main
“main”, fname, loader, pkg_name)
File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 94, in
main()
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 18, in main
click.Group(commands=commands)(prog_name=‘bench’)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/init.py”, line 24, in _func
ret = f(frappe._dict(ctx.obj), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/utils.py”, line 41, in clear_cache
frappe.clear_cache()
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 510, in clear_cache
frappe.sessions.clear_cache()
File “/home/frappe/frappe-bench/apps/frappe/frappe/sessions.py”, line 50, in clear_cache
clear_notifications(user)
File “/home/frappe/frappe-bench/apps/frappe/frappe/desk/notifications.py”, line 163, in clear_notifications
config = get_notification_config()
File “/home/frappe/frappe-bench/apps/frappe/frappe/desk/notifications.py”, line 220, in get_notification_config
return frappe.cache().get_value(“notification_config”, _get)
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py”, line 72, in get_value
val = generator()
File “/home/frappe/frappe-bench/apps/frappe/frappe/desk/notifications.py”, line 214, in _get
nc = frappe.get_attr(notification_config)()
File “/home/frappe/frappe-bench/apps/erpnext/erpnext/startup/notifications.py”, line 77, in get_notification_config
fields= [“name”], filters = {“name”: (“not in”, doctype), ‘is_submittable’: 1}):
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 1177, in get_all
return get_list(doctype, *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 1150, in get_list
return frappe.model.db_query.DatabaseQuery(doctype).execute(None, *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py”, line 88, in execute
result = self.build_and_run()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py”, line 100, in build_and_run
args = self.prepare_args()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py”, line 118, in prepare_args
self.set_optional_columns()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py”, line 257, in set_optional_columns
columns = frappe.db.get_table_columns(self.doctype)
File “/home/frappe/frappe-bench/apps/frappe/frappe/database.py”, line 849, in get_table_columns
return self.get_db_table_columns(‘tab’ + doctype)
File “/home/frappe/frappe-bench/apps/frappe/frappe/database.py”, line 845, in get_db_table_columns
return [r[0] for r in self.sql(“DESC %s” % table)]
File “/home/frappe/frappe-bench/apps/frappe/frappe/database.py”, line 176, in sql
self._cursor.execute(query)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/cursors.py”, line 170, in execute
result = self._query(query)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/cursors.py”, line 328, in _query
conn.query(q)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/connections.py”, line 516, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/connections.py”, line 727, in _read_query_result
result.read()
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/connections.py”, line 1066, in read
first_packet = self.connection._read_packet()
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/connections.py”, line 683, in _read_packet
packet.check_error()
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/protocol.py”, line 220, in check_error
err.raise_mysql_exception(self._data)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql/err.py”, line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1021, u’Disk full (/tmp/#sql_dd9_0.MAI); waiting for someone to free some space… (errno: 28 “No space left on device”)')

So… How can I access /tmp folder?
Anybody have some suggestion of how to solve it?

[clarkej]

@xickomesquita

error: file write error (No space left on device)
fatal: unable to write sha1 file
fatal: unpack-objects failed

There’s your problem

[xickomesquita]

I understand…

Can you give me some tips of how can I solve it?

Thanks!

[clarkej]

@xickomesquita

So… How can I access /tmp folder?

ok my guess is you want to delete files here to free up space.

What is the problem that you can’t access this?

[Julian_Robbins]

Do a

df -h

To see how much space you have free. Sometimes Los caches etc grow immensely and you may have to delete the offending files first.

[xickomesquita]

I’m trying to follow, consider my noob linux level, a lot of post I’ve seeing on the net.

My reality at now is:

ubuntu@ip-172-31-86-84:/$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
tmpfs 100M 3.3M 96M 4% /run
/dev/xvda1 7.7G 7.6G 110M 99% /
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/loop0 17M 17M 0 100% /snap/amazon-ssm-agent/734
/dev/loop1 88M 88M 0 100% /snap/core/5548
/dev/loop2 88M 88M 0 100% /snap/core/5328
/dev/loop3 13M 13M 0 100% /snap/amazon-ssm-agent/295
/dev/loop4 13M 13M 0 100% /snap/amazon-ssm-agent/495
/dev/loop5 87M 87M 0 100% /snap/core/5145
tmpfs 100M 0 100M 0% /run/user/1000

ubuntu@ip-172-31-86-84:/$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 124813 322 124491 1% /dev
tmpfs 126814 440 126374 1% /run
/dev/xvda1 1024000 173452 850548 17% /
tmpfs 126814 1 126813 1% /dev/shm
tmpfs 126814 4 126810 1% /run/lock
tmpfs 126814 16 126798 1% /sys/fs/cgroup
/dev/loop0 15 15 0 100% /snap/amazon-ssm-agent/734
/dev/loop1 12783 12783 0 100% /snap/core/5548
/dev/loop2 12860 12860 0 100% /snap/core/5328
/dev/loop3 13 13 0 100% /snap/amazon-ssm-agent/295
/dev/loop4 13 13 0 100% /snap/amazon-ssm-agent/495
/dev/loop5 12847 12847 0 100% /snap/core/5145
tmpfs 126814 4 126810 1% /run/user/1000

I don’t know what to do… Can you help?

[Julian_Robbins]

Hi

You have a few problems here. Firstly you need to clear space in your root partition. Google the command on CMD line to find really big files to see where you can save space. You also have your /snap partitions 100% full too so there may well need getting more space on these

[xickomesquita]

I’m sorry… I’ve spent all day trying to do this simple step… seek and delete the huge files, but I’m a windows man, trying to do this on AWS…

I know that I still have to fix something, but I’m starting to be afraid of ruin all the things.

I went to AWS EC2, and double the size of xvda1:

root@ip-172-31-86-84:/home/frappe/frappe-bench# df -h
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
tmpfs 100M 3.3M 96M 4% /run
/dev/xvda1 16G 7.4G 8.1G 48% /
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
/dev/loop0 88M 88M 0 100% /snap/core/5328
/dev/loop1 88M 88M 0 100% /snap/core/5548
/dev/loop2 17M 17M 0 100% /snap/amazon-ssm-agent/734
/dev/loop3 87M 87M 0 100% /snap/core/5145
/dev/loop4 13M 13M 0 100% /snap/amazon-ssm-agent/495
/dev/loop5 13M 13M 0 100% /snap/amazon-ssm-agent/295
tmpfs 100M 0 100M 0% /run/user/1000

I’m still seeing the “sorry, its…” page. I did the backups, have made some customizations on Doctypes…

I really don’t know what to do.

[xickomesquita]

I’m now trying to solve in a diferent way… Trying to backup and restore in another server… But, it doesn’t work also…

Back to the files:

I can see some git files and, looking close the mysql folder…

But, still stuck on what file I need to delete… By the way, I increased the size, and, now I have space to load things… So, I don’t know if there is something that I need to do after solve this mysterious size issue…

[trentmu]

The mysql data files are (normally) stored at /var/lib/mysql, which, from your picture, is in the red.
Also, you /home/frappe-bench subdir is full/red.
8GB is not enough space for what you are trying to achieve, so you need to change the partition sizes and/or capacity.
If you are new to linux, I recommend trying a single partition / instead of fixed sizes for /, /home, /var, tmp etc.
A minimalist approach would be
/boot ~200MB
swap ~512MB

The best way to experiment with linux is to create a few VM’s using something like VirtualBox - play around with one or 2 and see how it all works to determine the best option for when you work on your live AWS machine
/ the rest

[xickomesquita]

@trentmu

ou need to change the partition sizes and/or capacity.

how much space do you think I’ll gonna need?

[trentmu]

16-20GB would be a safe start point. 40GB would be better for future expansion.
I have installed a VM with only 12GB of disk, but it quickly ran into problems

ERPNext hardware requirements

Hello, I am not expert in security, but I am trying to give you answer based on my experience as End User. 1- What are the required HW specs required for the Machines (App, DB, Front end server) ? Ans: Ubuntu/Cent OS installed with minimum 4GB RAM and 40GB hard disk. You need more space if you are using large file attachment. 2- What is the SW required for servers as well ? Ans: Frappe take care of all dependency 3- What is Backup methodology that ERP is following ? Ans: O…

[xickomesquita]

OK. Thanks…

As you can see, I already increased the size of partition:

… but, I’m still stuck.

I have restarted all the services, and tryed to bench update:

From GitHub - frappe/frappe: Low code web framework for real world applications, in Python and Javascript

• branch master → FETCH_HEAD
  Already up-to-date.
  INFO:bench.utils:find . -name “*.pyc” -delete
  INFO:bench.app:pulling frappe_io
  INFO:bench.utils:git pull upstream master
  error: cannot open .git/FETCH_HEAD: Permission denied

Traceback (most recent call last):
File “/usr/local/bin/bench”, line 11, in
load_entry_point(‘bench’, ‘console_scripts’, ‘bench’)()
File “/home/frappe/.bench/bench/cli.py”, line 40, in cli
bench_command()
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/frappe/.bench/bench/commands/update.py”, line 60, in update
_update(pull, patch, build, bench, auto, restart_supervisor, restart_systemd, requirements, no_backup, force=force, reset=reset)
File “/home/frappe/.bench/bench/commands/update.py”, line 73, in _update
pull_all_apps(bench_path=bench_path, reset=reset)
File “/home/frappe/.bench/bench/app.py”, line 257, in pull_all_apps
remote=remote, branch=get_current_branch(app, bench_path=bench_path)), cwd=app_dir)
File “/home/frappe/.bench/bench/utils.py”, line 159, in exec_cmd
raise CommandFailedError(cmd)
bench.utils.CommandFailedError: git pull upstream master

But I can’t update…

I did also git in the apps/*.

[trentmu]

Your console prompt indicates that you are logged in as user “ubuntu”. This user might not have permission in the /home/frappe subdirectory.

su frappe
cd ~/frappe-bench
bench update --reset

Will probably/hopefully work

[xickomesquita]

unfortunately not… I’ve got the same:

frappe@ip-172-31-86-84:~/frappe-bench$ bench update --reset
INFO:bench.utils:updating bench
INFO:bench.utils:git pull
Already up-to-date.
INFO:bench.utils:./env/bin/pip install Pillow
Requirement already satisfied: Pillow in ./env/lib/python2.7/site-packages
You are using pip version 9.0.3, however version 18.0 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
INFO:bench.app:pulling frappe
INFO:bench.utils:git fetch --all
Fetching upstream
INFO:bench.utils:git reset --hard upstream/master
HEAD is now at 9d74acf Merge branch ‘hotfix’
INFO:bench.utils:find . -name “*.pyc” -delete
INFO:bench.app:pulling frappe_io
INFO:bench.utils:git fetch --all
error: cannot open .git/FETCH_HEAD: Permission denied

Traceback (most recent call last):
File “/usr/local/bin/bench”, line 11, in
load_entry_point(‘bench’, ‘console_scripts’, ‘bench’)()
File “/home/frappe/.bench/bench/cli.py”, line 40, in cli
bench_command()
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/dist-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/frappe/.bench/bench/commands/update.py”, line 60, in update
_update(pull, patch, build, bench, auto, restart_supervisor, restart_systemd, requirements, no_backup, force=force, reset=reset)
File “/home/frappe/.bench/bench/commands/update.py”, line 73, in _update
pull_all_apps(bench_path=bench_path, reset=reset)
File “/home/frappe/.bench/bench/app.py”, line 252, in pull_all_apps
exec_cmd(“git fetch --all”, cwd=app_dir)
File “/home/frappe/.bench/bench/utils.py”, line 159, in exec_cmd
raise CommandFailedError(cmd)
bench.utils.CommandFailedError: git fetch --all

[trentmu]

If you don’t have any data in your system yet, can I recommend that you do a reinstall from scratch?

[xickomesquita]

when I try to git pull on frappe_io, using ‘git pull’, I got:

frappe@ip-172-31-86-84:~/frappe-bench/apps/frappe_io$ git pull
error: cannot open .git/FETCH_HEAD: Permission denied

when I use ‘sudo git pull’:

frappe@ip-172-31-86-84:~/frappe-bench/apps/frappe_io$ sudo git pull
[sudo] password for frappe:
Sorry, user frappe is not allowed to execute ‘/usr/bin/git pull’ as root on ip-172-31-86-84.ec2.internal.

Maybe this file is causing the problem…

[trentmu]

When you created user frappe, did you give it sudoers permission?

You might be better off doing a manual install using this

Guide: Manual Install ERPNext on Ubuntu 16.xx & Debian v8 & 9

Hi All, After doing a clean manual install for Ubuntu 16.xx & Debian v8 & 9 (Specifically Debian Stretch) to understand what some of the issues/errors that we all get, following is what I ended up with that solves some of issues that come up. If anyone comes across any issues and resolves them please add them to the steps here. This is not final and I expect it can be refined further. NOTE: This of course could vary for everyone. But a few things to take into consideration Clean install on…

[xickomesquita]

yes… It was running, but… after trying to use the Android app, it was stuck on login page… After, I tried to use the site login… and then, it was loading, on “Verifying…” status… and then, out of service…

[trentmu]

Sorry - I have no experience/knowledge about the android app