Pages not loading after moving to production

Shaun · January 4, 2023, 8:26am

Hi -

I’ve just moved from dev to production following the steps below.

I’ve used these steps before successfully, however this time I’m getting an issue of pages not loading. As you can see “Refreshing…” is all the page says. Workspaces also hang and do not load the shortcuts.

Refreshing the page manually can sometimes kick start the load and get it done, however not always.

Any suggestions as to what’s going on here would be very much appreciated.


## PRODUCTION SETUP

bench --site rv.erpnext.local enable-scheduler
bench --site rv.erpnext.local add-to-hosts
bench --site rv.erpnext.local set-maintenance-mode off
bench --site rv.erpnext.local scheduler resume

sudo snap install --classic certbot
sudo bench setup production ubuntu
bench setup supervisor
sudo ln -s /home/bench/frappe/config/supervisor.conf /etc/supervisor/conf.d/frappe.conf
sudo bench setup sudoers $(whoami)

## STOP BENCH (ctrl+c)

sudo service supervisor stop
sudo service nginx stop
sudo service supervisor start
sudo service nginx start

cd /home/bench/frappe
sudo bench setup production ubuntu

bench --site rv.erpnext.local clear-website-cache

PyJumper · January 5, 2023, 4:30am

hellol did you also build & migrate your site ?

Shaun · January 13, 2023, 3:24am

Hi @PyJumper, I did migrate the site. Could you explain what you mean by ‘build’?

I followed these instructions for switching to production.

PyJumper · January 13, 2023, 5:05am

by doing this command:
bench build

you can also check your browser console, maybe you can get useful info for debugging

Shaun · January 13, 2023, 5:29am

Ok cool, thanks for the suggestion. Unfortunately that didn’t fix the issue

I get this console error when the home screen hangs (second picture from original post). Does this mean anything to anyone?

workspace.js:309 Uncaught (in promise) {readyState: 0, getResponseHeader: ƒ, getAllResponseHeaders: ƒ, setRequestHeader: ƒ, overrideMimeType: ƒ, …}abort: ƒ (W1)always: ƒ ()catch: ƒ (A)done: ƒ ()fail: ƒ ()getAllResponseHeaders: ƒ ()getResponseHeader: ƒ (W1)overrideMimeType: ƒ (W1)pipe: ƒ ()progress: ƒ ()promise: ƒ (A)readyState: 0responseJSON: undefinedsetRequestHeader: ƒ (W1,Q1)state: ƒ ()status: 0statusCode: ƒ (W1)statusText: "error"then: ƒ (A,z,B)[[Prototype]]: Object
show_page @ workspace.js:309
await in show_page (async)
show @ workspace.js:209
(anonymous) @ workspace.js:15
dispatch @ jquery.js:5430
c1.handle @ jquery.js:5234
trigger @ jquery.js:8719
(anonymous) @ jquery.js:8797
each @ jquery.js:385
each @ jquery.js:207
trigger @ jquery.js:8796
change_to @ container.js:79
(anonymous) @ pageview.js:52
with_page @ pageview.js:13
(anonymous) @ pageview.js:45
with_doctype @ model.js:127
show @ pageview.js:44
render_page @ router.js:231
render @ router.js:205
route @ router.js:115
push_state @ router.js:378
Promise.finally.frappe.route_flags @ router.js:274
set_route @ router.js:268
frappe.set_route @ router.js:449
t @ router.js:37
(anonymous) @ router.js:69
dispatch @ jquery.js:5430
c1.handle @ jquery.js:5234

PyJumper · January 13, 2023, 5:43am

if you do the following what is the output
sudo supervisorctl status all

Shaun · January 13, 2023, 6:09am

yikes. lots of fatal status

frappe-redis:frappe-redis-cache                 FATAL     Exited too quickly (process log may have details)
frappe-redis:frappe-redis-queue                 FATAL     Exited too quickly (process log may have details)
frappe-redis:frappe-redis-socketio              FATAL     Exited too quickly (process log may have details)
frappe-web:frappe-frappe-web                    RUNNING   pid 19556, uptime 0:00:03
frappe-web:frappe-node-socketio                 FATAL     Exited too quickly (process log may have details)
frappe-workers:frappe-frappe-default-worker-0   RUNNING   pid 19236, uptime 0:00:40
frappe-workers:frappe-frappe-long-worker-0      RUNNING   pid 19238, uptime 0:00:40
frappe-workers:frappe-frappe-schedule           RUNNING   pid 19235, uptime 0:00:40
frappe-workers:frappe-frappe-short-worker-0     RUNNING   pid 19237, uptime 0:00:40

Update: after a reboot I was able to get all of these up and running. Unfortunately I’m still having the page refresh issue described in my original post

frappe-redis:frappe-redis-cache                 RUNNING   pid 855, uptime 0:03:42
frappe-redis:frappe-redis-queue                 RUNNING   pid 856, uptime 0:03:42
frappe-redis:frappe-redis-socketio              RUNNING   pid 857, uptime 0:03:42
frappe-web:frappe-frappe-web                    RUNNING   pid 858, uptime 0:03:42
frappe-web:frappe-node-socketio                 RUNNING   pid 865, uptime 0:03:42
frappe-workers:frappe-frappe-default-worker-0   RUNNING   pid 869, uptime 0:03:42
frappe-workers:frappe-frappe-long-worker-0      RUNNING   pid 871, uptime 0:03:42
frappe-workers:frappe-frappe-schedule           RUNNING   pid 868, uptime 0:03:42
frappe-workers:frappe-frappe-short-worker-0     RUNNING   pid 870, uptime 0:03:42

PyJumper · January 14, 2023, 6:39am

strange, maybe you need to check bench logs also check if you have any nginx permissions errors,

sudo cat /var/log/nginx/error.log

also you can try bench setup requirements then bench setup production

Shaun · January 14, 2023, 9:51pm

Thanks for the info @PyJumper. I’ve got LOADS of these errors in the logs.

I’ve successfully ran bench setup requirements and bench setup production. Neither seemed to fix the performance.

Does this mean anything to you? They’re basically all the same, apart from “referrer”, which changes depending on the page I’m on.

2023/01/14 06:47:00 [error] 17186#17186: *509 connect() failed (111: Unknown error) while connecting to upstream, client: 103.137.12.204, server: rv.erpnext.local, request: "GET /socket.io/?EIO=4&transport=polling&t=OMl21OT HTTP/1.1", upstream: "http://127.0.0.1:9000/socket.io/?EIO=4&transport=polling&t=OMl21OT", host: "13.238.182.173", referrer: "http://13.238.182.173/app/home"

Strangely (and a little concerning), I’ve tried fresh installs of frappe/ERPNext and get the same result every time. Could these errors have anything to do with my Amazon EC2 instance? When accessing the production site, I’m simply going to the servers public IPv4 address.

Shaun · January 16, 2023, 12:42am

In addition to the nginx error. This error shows up consistently in the console when a page or report/list fails to load:

jquery.js:10109          POST http://xxxx/api/method/frappe.desk.reportview.get net::ERR_CONNECTION_RESET
send @ jquery.js:10109
ajax @ jquery.js:9690
frappe.request.call @ request.js:253
frappe.call @ request.js:103
refresh @ base_list.js:487
refresh @ list_view.js:280
(anonymous) @ utils.js:862
(anonymous) @ base_list.js:247
dispatch @ jquery.js:5430
c1.handle @ jquery.js:5234

I’m quite concerned about this ‘refreshing’ issue. I don’t want to continue with development until I’m sure that I can successfully switch to a production environment. Any help would be greatly appreciated.

UPDATE: I’ve discovered that this issue is ONLY occurring on my work PC. I just logged into the production environment on my phone and laptop… and there were no console errors, no refresh/page loading issues.

I wish I knew what was going on here, I get the same console error across many browsers (chrome, brave, firefox, edge) on my work PC.

Still keen to hear suggestions if anyone out there has any.

PyJumper · January 16, 2023, 1:10pm

maybe the firewall, anti virus, network config or proxy config, i would suggest to make ubuntu usb bootable and test your pc in “try mode” without installation and check if the problem still exist when you login to your remote site

Shaun · January 16, 2023, 9:26pm

I appreciate all the suggestions @PyJumper, thanks for the assistance mate!