Internal server error on Production server

EnSeal · April 6, 2017, 10:53pm

Hello Team

I am have an internal server error message and below is the web,log message… Please how do i resolve. Its a production environment

Module import failed for Page (frappe.core.doctype.page.page_dashboard)
Module import failed for Page (frappe.core.doctype.page.page_dashboard)
Module import failed for Page (frappe.core.doctype.page.page_dashboard)
Module import failed for User (frappe.core.doctype.user.user_dashboard)
Traceback (most recent call last):
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/user/user.py”, line 205, in send_password_notification
self.send_welcome_mail_to_user()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/user/user.py”, line 265, in send_welcome_mail_to_user
{“link”: link, “site_url”: get_url()})
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/user/user.py”, line 294, in send_login_mail
delayed=(not now) if now!=None else self.flags.delay_emails, retry=3)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 404, in sendmail
cc=cc, message_id=message_id, in_reply_to=in_reply_to, retry=retry)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/init.py”, line 21, in sendmail
reply_to=reply_to, cc=cc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/email_body.py”, line 17, in get_email
emailobj = EMail(sender, recipients, subject, reply_to=reply_to, cc=cc, email_account=email_account)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/email_body.py”, line 61, in init
self.email_account = email_account or get_outgoing_email_account()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/smtp.py”, line 66, in get_outgoing_email_account
frappe.OutgoingEmailError)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 300, in throw
msgprint(msg, raise_exception=exc, title=title, indicator=‘red’)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 293, in msgprint
_raise_exception()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 266, in _raise_exception
raise raise_exception, encode(msg)
OutgoingEmailError: Please setup default Email Account from Setup > Email > Email Account

Module import failed for Web Page (frappe.website.doctype.web_page.web_page_dashboard)
Module import failed for Website Script (frappe.website.doctype.website_script.website_script_dashboard)
Module import failed for Customize Form (frappe.custom.doctype.customize_form.customize_form_dashboard)
Module import failed for Page (frappe.core.doctype.page.page_dashboard)
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist

aaimaa · April 6, 2017, 11:00pm

Now, I’m sure it should be Production Server Issues. I also created thread just before then you raised concern. "Internal Server Error" while installing Dropbox

*FOLLOWING

EnSeal · April 7, 2017, 6:33pm

@aaimaa Have you made any headway on this issue. I seem not to be able to resolve this issue. Its a Production platform and would need an urgent help from anyone right now

aaimaa · April 7, 2017, 7:10pm

Yes, I had resolved my issue. Running perfectly with production as well. I think you should also check first your error-log and if it was issue of database connectivity. Should also try this possible solution.

Please go through complete thread: "Internal Server Error" while installing Dropbox

If still exist your issue, reply with error log.

EnSeal · April 7, 2017, 7:39pm

Thanks for your prompt response @aaimaa I have tried several suggestions on different threads but no luck

Below is my error log

Module import failed for Page (frappe.core.doctype.page.page_dashboard)
Module import failed for Page (frappe.core.doctype.page.page_dashboard)
Module import failed for Page (frappe.core.doctype.page.page_dashboard)
Module import failed for User (frappe.core.doctype.user.user_dashboard)
Traceback (most recent call last):
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/user/user.py”, line 205, in send_password_notification
self.send_welcome_mail_to_user()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/user/user.py”, line 265, in send_welcome_mail_to_user
{“link”: link, “site_url”: get_url()})
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/user/user.py”, line 294, in send_login_mail
delayed=(not now) if now!=None else self.flags.delay_emails, retry=3)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 404, in sendmail
cc=cc, message_id=message_id, in_reply_to=in_reply_to, retry=retry)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/init.py”, line 21, in sendmail
reply_to=reply_to, cc=cc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/email_body.py”, line 17, in get_email
emailobj = EMail(sender, recipients, subject, reply_to=reply_to, cc=cc, email_account=email_account)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/email_body.py”, line 61, in init
self.email_account = email_account or get_outgoing_email_account()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/email/smtp.py”, line 66, in get_outgoing_email_account
frappe.OutgoingEmailError)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 300, in throw
msgprint(msg, raise_exception=exc, title=title, indicator=‘red’)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 293, in msgprint
raiseexception()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/init.py”, line 266, in raiseexception
raise raise_exception, encode(msg)
OutgoingEmailError: Please setup default Email Account from Setup > Email > Email Account

Module import failed for Web Page (frappe.website.doctype.web_page.web_page_dashboard)
Module import failed for Website Script (frappe.website.doctype.website_script.website_script_dashboard)
Module import failed for Customize Form (frappe.custom.doctype.customize_form.customize_form_dashboard)
Module import failed for Page (frappe.core.doctype.page.page_dashboard)
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist
107.23.87.100 does not exist

aaimaa · April 7, 2017, 10:31pm

Try running bench update if still not solved. Wait for comments of Leaders or should raise voice at Github Issues to get support asap.

olamide_shodunke · April 8, 2017, 8:36pm

@rmehta sorry to bother you but can you help with this? A production system is down

EnSeal · April 10, 2017, 10:23am

@sagar trust you are doing very well. Please can you assist with resolving issue above.

sagar · April 10, 2017, 12:02pm

Hi @EnSeal,
please check you erpnext & frappe branch is same.
if it is same please update and check the status of supervisor is working or not.

Thanks.

EnSeal · April 10, 2017, 2:17pm

thanks for coming through @sagar my erpnext & frappe bench are same and my supervisor status seems fine. see output below

frappe-bench-redis:frappe-bench-redis-cache RUNNING pid 10296, uptime 4:03:16
frappe-bench-redis:frappe-bench-redis-queue RUNNING pid 10295, uptime 4:03:16
frappe-bench-redis:frappe-bench-redis-socketio RUNNING pid 10297, uptime 4:03:16
frappe-bench-web:frappe-bench-frappe-web RUNNING pid 10293, uptime 4:03:16
frappe-bench-web:frappe-bench-node-socketio RUNNING pid 10294, uptime 4:03:16
frappe-bench-workers:frappe-bench-frappe-default-worker-0 RUNNING pid 10289, uptime 4:03:16
frappe-bench-workers:frappe-bench-frappe-long-worker-0 RUNNING pid 10291, uptime 4:03:16
frappe-bench-workers:frappe-bench-frappe-schedule RUNNING pid 10290, uptime 4:03:16
frappe-bench-workers:frappe-bench-frappe-short-worker-0 RUNNING pid 10292, uptime 4:03:16

clarkej · April 10, 2017, 4:19pm

Assuming the website setting option exists, maybe turn off ‘send_welcome_mail_to_user’ !?

rmehta · April 12, 2017, 11:25am

@EnSeal can you run bench update again and let us know if you see any error

For internal server error see the gunicorn log at frappe-bench/logs/frappe.log

noetico · July 12, 2017, 11:52pm

@olamide_shodunke; upgrading erpnext can be a task in itself and from experience, any hit and miss and you will end up with internal server error or 503, with Updating… so, heed this advice; dont just directly update your production server. If you have the time, make a clone of the disk and do the upgrade on a demo system; log your steps and when you succeed, repeat on server; even with this, keep a clean backup. Updates/upgrades can fail for thing as simple as a new Doctype added, like I saw one recently where it failed with error Doctype Domain not found; the upgrade/update script couldnt create this doctype for me!!

noetico · November 24, 2017, 9:47am

Have you tried bench update --reset? That command seems to work very well to solve issues. Also post the error; maybe a patch not getting applied properly?

natnael · November 24, 2017, 9:55am

@noetico thank you for the reply. I just tried bench update --reset and it results in:

INFO:bench.utils:updating bench
INFO:bench.utils:git pull
Updating 9c15073…4e896b5
error: Your local changes to the following files would be overwritten by merge:
README.md
bench/cli.py
bench/commands/install.py
bench/commands/setup.py
Please, commit your changes or stash them before you can merge.
Aborting
Traceback (most recent call last):
File “/bin/bench”, line 11, in
load_entry_point(‘bench’, ‘console_scripts’, ‘bench’)()
File “/home/frappe/.bench/bench/cli.py”, line 40, in cli
bench_command()
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/frappe/.bench/bench/commands/update.py”, line 34, in update
update_bench()
File “/home/frappe/.bench/bench/utils.py”, line 256, in update_bench
exec_cmd(“git pull”, cwd=cwd)
File “/home/frappe/.bench/bench/utils.py”, line 140, in exec_cmd
raise CommandFailedError(cmd)
bench.utils.CommandFailedError: git pull

noetico · November 24, 2017, 10:15am

Ok; try doing git stash on the /apps/frappe and apps/erpnext folders. Then do git pull against these folders. After that, head to the frappe-bench folder and do bench update --reset. Let’s see how that goes.

noetico · November 24, 2017, 10:38am

You may also try git reset --hard against those folders as well (apps/frappe and apps/erpnext).

natnael · November 24, 2017, 11:58am

Sorry for replying so late and thank your for your quick reply. I followed the steps you just mentioned(in both of your previous posts) and did bench update --reset but it outputs the same thing as the previous one:

[dagm@clouderp frappe]$ pwd
/home/frappe/frappe-bench/apps/frappe
[dagm@clouderp frappe]$ git stash
No local changes to save
[dagm@clouderp frappe]$ cd … && cd erpnext
[dagm@clouderp erpnext]$ pwd
/home/frappe/frappe-bench/apps/erpnext
[dagm@clouderp erpnext]$ git stash
No local changes to save
[dagm@clouderp erpnext]$ git pull
fatal: Unable to create ‘/home/frappe/frappe-bench/apps/erpnext/.git/ORIG_HEAD.lock’: Permission denied
[dagm@clouderp erpnext]$ sudo git pull
Already up-to-date.
[dagm@clouderp erpnext]$ cd …
[dagm@clouderp apps]$ cd frappe/
[dagm@clouderp frappe]$ git pull
fatal: Unable to create ‘/home/frappe/frappe-bench/apps/frappe/.git/ORIG_HEAD.lock’: Permission denied
[dagm@clouderp frappe]$ sudo git pull
Already up-to-date.
[dagm@clouderp frappe]$ cd …
[dagm@clouderp apps]$ cd …
[dagm@clouderp frappe-bench]$ sudo bench update --reset

INFO:bench.utils:updating bench
INFO:bench.utils:git pull
Updating 9c15073…4e896b5
error: Your local changes to the following files would be overwritten by merge:
README.md
bench/cli.py
bench/commands/install.py
bench/commands/setup.py
Please, commit your changes or stash them before you can merge.
Aborting
Traceback (most recent call last):
File “/bin/bench”, line 11, in
load_entry_point(‘bench’, ‘console_scripts’, ‘bench’)()
File “/home/frappe/.bench/bench/cli.py”, line 40, in cli
bench_command()
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/lib64/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/frappe/.bench/bench/commands/update.py”, line 34, in update
update_bench()
File “/home/frappe/.bench/bench/utils.py”, line 256, in update_bench
exec_cmd(“git pull”, cwd=cwd)
File “/home/frappe/.bench/bench/utils.py”, line 140, in exec_cmd
raise CommandFailedError(cmd)
bench.utils.CommandFailedError: git pull

The same goes for the git reset --hard command against the frappe and erpnext folders.

noetico · November 24, 2017, 1:24pm

Do you have a bench-repo folder? If you do, git stash, git fetch then git pull

natnael · November 24, 2017, 1:35pm

Yes I do have a bench-repo folder. I did a git stash, git fetch and git pull on the folder:

[dagm@clouderp frappe]$ cd .bench
[dagm@clouderp .bench]$ sudo git stash
Saved working directory and index state WIP on master: 9c15073 Merge pull request #495 from saurabh6790/ansible_version_fix
HEAD is now at 9c15073 Merge pull request #495 from saurabh6790/ansible_version_fix
[dagm@clouderp .bench]$ sudo git fetch
[dagm@clouderp .bench]$ sudo git pull
Updating 9c15073…4e896b5
Fast-forward
README.md | 2 ±
bench/cli.py | 2 ±
bench/commands/install.py | 6 ++++±
bench/commands/setup.py | 10 ++++++++±
playbooks/prerequisites/roles/fail2ban/defaults/main.yml | 5 +++±
…/roles/fail2ban/tasks/configure_nginx_jail.yml | 11 +++++++++++
…/roles/fail2ban/templates/nginx-proxy-jail.conf.j2 | 5 ++±-
playbooks/prerequisites/setup_nginx_proxy_jail.yml | 2 ++
8 files changed, 36 insertions(+), 7 deletions(-)
create mode 100644 playbooks/prerequisites/roles/fail2ban/tasks/configure_nginx_jail.yml
create mode 100644 playbooks/prerequisites/setup_nginx_proxy_jail.yml