Troubleshooting slow ERPNext instance

Hello everyone-- recently I’ve been having troubles with the speed of my self-hosted ERPNext installation, from loading pages to processing logins to just about any function. I have been reading around other forum posts but no luck.

Operating System /cat/*elease

CentOS Linux release 7.5.1804 (Core)
NAME=“CentOS Linux”
VERSION=“7 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“7”
PRETTY_NAME=“CentOS Linux 7 (Core)”
ANSI_COLOR=“0;31”
[…]

CENTOS_MANTISBT_PROJECT=“CentOS-7”
CENTOS_MANTISBT_PROJECT_VERSION=“7”
REDHAT_SUPPORT_PRODUCT=“centos”
REDHAT_SUPPORT_PRODUCT_VERSION=“7”

CentOS Linux release 7.5.1804 (Core)
CentOS Linux release 7.5.1804 (Core)

CPU cat /proc/cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version (cpu64-rhel6)
stepping : 3
microcode : 0x1
cpu MHz : 1999.999
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl pni cx16 hypervisor lahf_lm
bogomips : 3999.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version (cpu64-rhel6)
stepping : 3
microcode : 0x1
cpu MHz : 1999.999
cache size : 4096 KB
physical id : 1
siblings : 1
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl pni cx16 hypervisor lahf_lm
bogomips : 3999.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version (cpu64-rhel6)
stepping : 3
microcode : 0x1
cpu MHz : 1999.999
cache size : 4096 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl pni cx16 hypervisor lahf_lm
bogomips : 3999.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version (cpu64-rhel6)
stepping : 3
microcode : 0x1
cpu MHz : 1999.999
cache size : 4096 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl pni cx16 hypervisor lahf_lm
bogomips : 3999.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version (cpu64-rhel6)
stepping : 3
microcode : 0x1
cpu MHz : 1999.999
cache size : 4096 KB
physical id : 4
siblings : 1
core id : 0
cpu cores : 1
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm nopl pni cx16 hypervisor lahf_lm
bogomips : 3999.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

Memory cat /proc/meminfo

MemTotal: 5945036 kB
MemFree: 2654264 kB
MemAvailable: 3498624 kB
Buffers: 93692 kB
Cached: 923608 kB
SwapCached: 0 kB
Active: 2441552 kB
Inactive: 622432 kB
Active(anon): 2047560 kB
Inactive(anon): 13700 kB
Active(file): 393992 kB
Inactive(file): 608732 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 6291452 kB
SwapFree: 6291452 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 2046824 kB
Mapped: 53948 kB
Shmem: 14580 kB
Slab: 139392 kB
SReclaimable: 105756 kB
SUnreclaim: 33636 kB
KernelStack: 3472 kB
PageTables: 18728 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 9263968 kB
Committed_AS: 7115092 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 27428 kB
VmallocChunk: 34359703292 kB
HardwareCorrupted: 0 kB
AnonHugePages: 901120 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 79860 kB
DirectMap2M: 6211584 kB

Speedtest

Testing from QuadraNet (155.94.164.195)…
Retrieving speedtest.net server list…
Selecting best server based on ping…
Hosted by AT&T (Los Angeles, CA) [1.30 km]: 12.49 ms
Testing download speed…
Download: 84.84 Mbit/s
Testing upload speed…
Upload: 133.95 Mbit/s

Disk Benchmark dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 12.4668 s, 86.1 MB/s

MySQL config /etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

[…]

innodb-file-format=barracuda
innodb-file-per-table=1
innodb-large-prefix=1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
innodb_buffer_pool_size = 4G
innodb_buffer_pool_instances = 6
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
innodb_thread_concurrency = 0
innodb_read_io_threads = 8
innodb_write_io_threads = 8

[mysql]
default-character-set = utf8mb4

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

include all files from the config directory

!includedir /etc/my.cnf.d

Live server erp.frc4079.org

Usually if everything is slow then it is the network or the internet which is a suspect. Try from a different country. Will establish a clear case!

What do you mean from a different country? My server is is LA, which is the closest to where I am, in Southern California.

What’s in your slow query log?

Where i live if i drive > 1hr, I need to cross a country border. So for you a different county or a state? The point i am making is try from different places and see if it makes a different. I moved a server from FRANKFURT (Germany) to MUMBAI (India). Both on AWS. And made a huge difference to the user XP.

BTW: i signed up several users with dummy emails on the ERP instance you have put up the link. Seemed ok to me. <5 seconds.

Slow performance Depends upon Different Factors.

  1. Number of Concurrent Users
  2. Size of the Database
  3. Gunicorn Workers Configured and Running
  4. Server CPU and Memory
  5. mySQL Configuration according to RAM

if you know all the above answer, you’ll have idea where the issue is.

1 Like