I am using the digital ocean droplet for 1GB,
I have not been using ErpNext for a while, and kept it running on a droplet,
when I am trying now, it keeps getting hung/timeout, and when I login, I see the system is gets into low memory and constantly throwing fork: Cannot allocate memory
Once I restart, its okay for about 30 mins, then goes into bad state again.
Here is the usage just after reboot of the machine, and login and navigating erpnext,
root# ps --sort -rss -eo rss,pid,command | head
RSS PID COMMAND
189344 1607 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
88864 2016 /home/frappe/frappe-bench/env/bin/python /home/frappe/frappe-bench/env/bin/gunicorn -b 127.0.0.1:8000 -w 1 -t 120 frappe.app:application --preload
40144 1918 /home/frappe/frappe-bench/env/bin/python /home/frappe/frappe-bench/env/bin/gunicorn -b 127.0.0.1:8000 -w 1 -t 120 frappe.app:application --preload
37956 1936 /usr/bin/node /home/frappe/frappe-bench/apps/frappe/socketio.js
33704 1882 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe worker --queue default
33644 1885 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe schedule
33492 1901 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe worker --queue short
33488 1895 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe worker --queue long
23380 1947 /usr/bin/redis-server 127.0.0.1:13000
After about 20 mins of leaving the server running, the memory used increases drastically,
root# ps --sort -rss -eo rss,pid,command | head
RSS PID COMMAND
339852 1608 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
205048 1942 /usr/bin/redis-server 127.0.0.1:11000
45296 1948 /usr/bin/redis-server 127.0.0.1:13000
40320 2165 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe worker --queue short
33728 1919 /home/frappe/frappe-bench/env/bin/python /home/frappe/frappe-bench/env/bin/gunicorn -b 127.0.0.1:8000 -w 1 -t 120 frappe.app:application --preload
32108 8687 /home/frappe/frappe-bench/env/bin/python /home/frappe/frappe-bench/env/bin/gunicorn -b 127.0.0.1:8000 -w 1 -t 120 frappe.app:application --preload
30776 3067 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe worker --queue default
28756 1884 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe schedule
26180 1879 /home/frappe/frappe-bench/env/bin/python -m frappe.utils.bench_helper frappe worker --queue default
Looking at the config mentioned in other thread, I see low normal configuration of mariadb and redis cache memory limits, (removed comments)
cat /etc/mysql/my.cnf
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
bind-address = 127.0.0.1
max_connections = 100
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
myisam_recover_options = BACKUP
key_buffer_size = 128M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
query_cache_limit = 128K
query_cache_size = 64M
log_warnings = 2
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
log_slow_verbosity = query_plan
log_bin = /var/log/mysql/mariadb-bin
log_bin_index = /var/log/mysql/mariadb-bin.index
expire_logs_days = 10
max_binlog_size = 100M
default_storage_engine = InnoDB
innodb_buffer_pool_size = 256M
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
[galera]
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/
root# cat /home/frappe/frappe-bench/config/redis_cache.conf
dbfilename redis_cache.rdb
dir /home/frappe/frappe-bench/config/pids
pidfile /home/frappe/frappe-bench/config/pids/redis_cache.pid
bind 127.0.0.1
port 13000
maxmemory 50mb
maxmemory-policy allkeys-lru
appendonly no
save “”
Can anyone help me identifying what might be causing this issue.
I am new to linux, so I am not sure how to debug this issue.