You should increase your background workers and gunicorn workers. I have beefy multiple VPS, and I started messing with performance tuning for my ERPNext instances, and editing those after increasing innodb cache size really helps. My emails are sent in an instant.
Just my experience though, your mileage may vary 
Here’s an old thread discussing about performance: What is the difference between background_workers and gunicorn_workers