While using above implementation, we found out both servers were using different cache, causing differences in results with functions using redis cache(frappe.cache).
To fix it, we did a few tweaks to point secondary server to use redis of primary server. To do that,
- Changed bind_address in redis_cache and redis_socketio to lan_ip(in our case, we did 0.0.0.0) instead of 127.0.0.1 on primary server. (make sure not to make it 0.0.0.0 on a server having a public ip, use lan ethernet ip instead).
- Changed redis_cache and redis_socketio hosts in common_site_config in secondary server to point to master server.
"redis_cache": "redis://localhost:13000",
"redis_socketio": "redis://localhost:12000",
to
"redis_cache": "redis://master_server_ip:13000",
"redis_socketio": "redis://master_server_ip:12000",
This way, we got all servers to use single redis, hence resolved the problem of wrong caching.
Taking forward what we were doing, we did a few more tweaks to increase no of workers, which spread across multiple servers as well. Following changes were done.
- Change bind address in config files of redis_queue as well to lan_ip or 0.0.0.0.
- Changed common site_config in secondary server to point to master server’s ip
"redis_queue": "redis://localhost:11000",
to
"redis_queue": "redis://master_server_ip:11000",
Along with this, we had to disable the scheduler on secondary server by removing frappe-schedule entry from supervisor.conf. Following this, reloading and restarting supervisor did the trick.
Removed below entry from supervisor.conf in secondary server.
[program:frappe-bench-frappe-schedule]
command=/usr/local/bin/bench schedule
priority=3
autostart=true
autorestart=true
stdout_logfile=/part2/workspace/pr/frappe-bench/logs/schedule.log
stderr_logfile=/part2/workspace/pr/frappe-bench/logs/schedule.error.log
user=himanshu
directory=/part2/workspace/pr/frappe-bench
Edit: Now after this, we came across third issue. Keeping both instance’s corresponding site folders synced with each other, so that changes in site config folder, as well as any updated files created could be synced to secondary.
For this, rsync was the solution. We set up a cron job of syncing sites folder using rsync by adding following entry to crontab of secondary server.
* * * * * rsync -rakv frappe@master_server_ip:~/frappe-bench/sites/my_site_folder ~/frappe-bench/sites/ >> ~/frappe-bench/logs/rsync.log
Now any changes done on primary site_configs are reflected in secondary in a minute (very acceptable for planned deployments)