Memory and SWAP 100%

[fas143]

Hi,

Memory and SWAP are full at everytime. I have done bench restart and mariadb restart.
After few times it will be okay, then again frappe scheduler will eat whole memory and swap.

ERPnext is too slow to process single transaction.

HTOP

[rtdany10]

Check your email queue and see if there are a lot of unsent emails. If yes, try suspending emails and check if the system speeds up. I had similar issue and it was a badly configured notification that generated huge amount of emails every day. Email send is a resource hungry process.

[Muzzy]

Faced similar situation. Its not ERPNext but MariaDB issue. On backup it takes the RAM but does not release it. There are many posts across the net with similar issue but no resolution.

Full RAM usage on 4 core, 8 GB RAM server with only one retail outlet with one POS machine

Hi, have an instance of 4 core Intel Xeon, 8 GB RAM server. It is running only one site with one retail outlet. Total invoices as of now is 192,721. Majority (about 97%) is from POS. On an average there are 4-5 pos invoices to be synced every 5 minutes. One store with one POS only. I have tried everything to bring down RAM usage. Moved db to another VPS thinking that previous VPS has faulty hardware. Sometimes cant even update without rebooting as that is the only time the RAM goes down. Once s…

[fas143]

@Muzzy
I think that was the same issue. Once we give the backup, mysql not releasing RAM after completing backup.

I tried to check the bottle neck, there no jobs pending and disabled emails and deleted emails queues.

Still the same issue.

[tus1688]

Coming from Muzzy’s post do you also use ubuntu? I wonder if rhel derivatives face same issue. Thanks in advance

[Muzzy]

Tested it on Ubuntu 18 and 20. Both facing similar issue. It happened yesterday on a Ubuntu 20 server which had original db. Hadn’t done db restoration on it.

[tus1688]

@Muzzy it is really weird, I hope things turn around for you

[rahy]

Just to inform. I’m on version-12 and CentOS (now migrated to AlmaLinux). I also backed up and restored sites from old server to new server.
But I didn’t experience this memory issue.

[fas143]

@rahy
It was running perfectly fine with version 12 and Ubuntu 16.04. I have been doing db and file back / restore in that.

After upgrading ubuntu 18.04 with erp 13, I started noticing about memory.

[Muzzy]

Try this and let us know if works

Full RAM usage on 4 core, 8 GB RAM server with only one retail outlet with one POS machine

Well I tried many stuff. Finally tried are these steps 1 apt-get update 2 apt-get install -y libjemalloc-dev 3 in /etc/systemd/system/mariadb.service.d/ edit any .conf in this directory by adding information at the end-line. LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 4 run systemctl daemon-reload 5 run sudo systemctl restart mysql.service 6 run sudo reboot I have found some respite on a demo server. Will be testing on production server.

[fas143]

@Muzzy
Its same. no difference.

[fas143]

@Muzzy
Any other solution to work out

[fkardame]

No solution yet. It is an upstream pkg issue with mariadb.

Currently we have a temporary fix to restart sql every night which is not a good solution but it does the work.

[fas143]

Sort out the issue by increasing Server RAM from 8GB to 16GB.

[pioneerpathan]

Actually, there is the same issue even 64GB in my VM. So, actually increasing RAM is not a solution. It is an upstream package of mariadb should be handled and solved.

[Praveenkumar]

@pioneerpathan Facing same issue here, How to resolve this?

[pioneerpathan]

Tell me how much RAM, CPU and SWAP allocation do you have?

[Praveenkumar]

@pioneerpathan

Actually, there is the same issue even 64GB in my VM. So, actually increasing RAM is not a solution. It is an upstream package of mariadb should be handled and solved.

32GB RAM

[Praveenkumar]

[pioneerpathan]

1. Stop Swap Thrashing (Critical). First of all, SWAP shouldn’t allow to go 100% utilised otherwise serve would go down. Hence, first of all restrict SWAP utlisation:

sudo sysctl vm.swappiness=10

Make permanent. Open:

sudo nano /etc/sysctl.conf

Add following two lines in that file:

vm.swappiness = 10
vm.vfs_cache_pressure = 50

Apply:

sudo sysctl -p

2. Tune MySQL properly. innodb_buffer_pool_size should be between 12G to 16G maximum but not more than that. You have 31 G RAM - 12 or 16 then server would have at least 18-16 G for other equally important processes like NGINX, Redis, Supervisor and python. Your mysql innodb config should look something like below:

   default_storage_engine          = InnoDB
   innodb_buffer_pool_size         = 12G
   innodb_log_buffer_size          = 256M
   innodb_log_file_size            = 1G
   innodb_buffer_pool_instances    = 8
   innodb_flush_method             = O_DIRECT
   innodb_flush_log_at_trx_commit  = 2
   

you may increase innodb_buffer_pool_size upto 16G if require.

3. Restrict MySQL memory uses by mariadb util:

sudo systemctl edit mariadb

Add following lines in specific allowed area between two commented section:

[Service]
MemoryAccounting=yes
MemoryMax=14G
MemoryHigh=15G

sudo systemctl daemon-reexec
sudo systemctl restart mariadb

4. Reduce gunicorn works. By default it may be 33 but that many does not require. Set it 6-10.
5. Install EarlyOOM (Prevents total freeze) (OOM tuning):

Install:

sudo apt install earlyoom -y
sudo systemctl enable --now earlyoom

EarlyOOM tuning (last line of defense)

Edit:

sudo nano /etc/default/earlyoom

Set:

EARLYOOM_ARGS=“-r 3600 -m 5 -s 10”

Restart:

sudo systemctl restart earlyoom

EarlyOOM kills bad processes early instead of letting the whole server freeze.
EarlyOOM doesn’t restrict memory — it prevents total collapse.
EarlyOOM saves your server before it becomes unresponsive.