Hi @revant_one,
Please, is there a step by step guide on how to migrate a Frappe setup from using a local redis server connection (for queue, socketio etc) to AWS ElastiCache for Redis?
I want to setup an ElastiCache cluster but I need to know what needs to be done when moving new and existing sites to the ElastiCache Redis server.
Thanks.
set the keys redis_cache, redis_queue and redis_socketio in common_site_config.json to point to external redis hosts.
redis is used as in memory cache. keeping it on same server will be faster than external host.
Ok Thanks. I was able to successfully set it up. We intend to use an AWS Auto scaling group for our EC2 instances running Frappe so we needed to have a central server for Redis (ElastiCache), MariaDB (RDS) and file storage (S3) so that each replica in the auto scaling group can be in sync when running concurrently.
I would like to know more about the work you have done.
We have an architecture where we create new sites for different clients. We usually create new VMs (i.e EC2 instances) and install Frappe, ERPNext and our custom apps. To avoid a single point of failure we limited the number of sites that can be created on a single bench.
To have as many sites as possible on a single VM and allow auto-scaling to handle traffic and CPU spikes, we decided to run our servers behind a load balancer on an auto-scaling group. This would require building an image that has been installed with Frappe, ERPNext and our custom apps. The image when started by an auto-scaling group would mount a network filesystem (AWS EFS) shared by other VMs in the same auto-scaling group and all VMs would be connected to the same database (AWS RDS) database and Redis server (AWS ElastiCache ). This allows scaling to be seamless on handling traffic and CPU fluctuations and elastic disk space provided by the NFS volumes.
For the image, does it include the existing sites?
How if there is a new site created?
Is it included to the new auto-scaling created server or to the old one?
What type of load balancer are you using?
When new sites are created, their site directory exists on the network filesystem volume that will be mounted on any new server created by the auto-scaling group. I used AWS ELB for load balancing.
So the EFS is mounted as /home?
It doesn’t have to be /home but it is mounted as the frappe bench directory that contains the app, site, and static assets directories.
Try not to use volume for apps directory and let them be part of image.
sites directory can be part of volumes.
In case of containers, images have app code and sites are stored in volumes.
How to separate apps and sites (not on same bench)?
For not using containers.
Thanks
Yes. In terms of CI/CD, you can either make the apps part of the image which means every change pushed to the pipeline would build a new image using AWS CodeBuild or you can make apps part of the volume if you prefer to update code using AWS CodeDeploy tool.
I’ll not recommend this as it adds unnecessary IO on network volume.