Automated continuous master/slave streaming backup

ERPNext Customize ERPNext
29

Hi:
I can provide 2 virtual servers too (in our own premises).

1 Like
30

I’m in Eastern Ontario, not far from Montreal.

You?

31

The criteria for now is …

  • Ubuntu
  • ERPNext V13

… in two devices, a master and a slave.

Great to have your support!

32

Awesome! Just hit me up when we’re ready to go and I’ll have them up in no time. I already have the images set so it will take just a few minutes to spin them up and get them running once we’re ready for the tests.

2 Likes
33

Yeah! That works for sure for testing purposes. It’ll work between virtual machines in the same host. It’ll also work between a local virtual machine as slave and a remote VPS somewhere in the cloud as master because the slaves do all the work. Apart from opening firewall ports and authorising slave users, the role of the master is entirely passive.

However, if you have numerous local machines you might look into Galera clustering as a superior alternative. The kind of replication I’m using here is for fragile, unreliable connections. Galera is optimised for high speed LAN clustering.

34

Bursa, Türkiye. Not far from Istanbul :slight_smile:

35

@TurkerTunali Oh! Amazing. Yeah, I am sure we could have some very interesting conversations.

1 Like
36

Hey everyone!

Here it is :

3 Likes
37

Excellent! I will test it as soon as I have some free time and give you feedback.

Thanks!

1 Like
38

Hey everyone,

I can confirm that these scripts now also work with Version 14.

Please do do some testing on a pair of disposable installations. I believe you will like it.

3 Likes
39

This will be a very useful feature. Certainly interested. Plz let us know the Public Github Repo Link.
Thanks.

40
1 Like
41

To date I have had no feedback from anyone. :face_with_raised_eyebrow:

Please be aware that I will happily sort out any difficulties you may encounter.

It is quite likely that I have allowed dependencies on my own installations to creep into the toolkit’s functionality. In order to solve those issues I need to see the results of running the scripts in other environments, so I would be grateful to anyone who is willing to collaborate a bit to clear those up.

1 Like
42

@MartinHBramwell I did promise to help test and provide feedback but I have been out of commission due to other commitments. I will try and test it out this weekend and provide feedback.

Thank you very much for your efforts.

43

Thank you for letting me know.

Appreciated!

44

I just now uploaded some changes.

The main improvement is protection of the slave site’s Social Login credentials.

A normal database backup and restore expects the restored site to be the same as the backed up site. Since replication necessarily requires the slave site to have a different domain name, while Social Logins require an exact amtch to the domain name, I provided a restore tool that replaces references to the master site URL with references to the slave site URL.

That tool did not include updating the Social Login credentials. Now it does.

You merely need to have a file named socials_google.json in the same directory as your site_config.json

admin@denh:~$ cat ~/frappe-bench-DENH/sites/dev.erpnext.host/socials_google.json 
{
    "name": "Google",
    "enable_social_login": 1,
    "social_login_provider": "Google",
    "client_id": "{{ClientID}}",
    "provider_name": "Google",
    "client_secret": "{{ClientSecret}}",
    "base_url": "https://www.googleapis.com",
    "authorize_url": "https://accounts.google.com/o/oauth2/auth",
    "access_token_url": "https://accounts.google.com/o/oauth2/token",
    "redirect_url": "/api/method/frappe.www.login.login_via_google",
    "api_endpoint": "oauth2/v2/userinfo",
    "auth_url_data": "{\"scope\": \"https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email\", \"response_type\": \"code\"}",
    "doctype": "Social Login Key"
}
admin@denh:~$

(Albeit with the correct replacements for the variables {{ClientID}} & {{ClientSecret}} )

@TurkerTunali I do hope you and those close to you are safe and secure.

45

V14 supported now, as you requested

6 Likes
46

We are OK. Thank you. It is really hard to do stuff. I hope next week will be better.

47

Will attempt and update, thank you

1 Like
48

This past week, my VPS hosting service had some sort of maintenance hiccup. The slave server stopped pulling changes from the master, and showed it was missing 4 hours of data changes.

I rebooted the slave, logged back into ERPNext and found that the missing deltas had already all been applied and the two databases were synchronised again.

I gotta say, after trying initially to see if there would be real interest in this toolkit, getting many positive responses and, consequently, doing all work, it is very disappointing that still no one has even tried it.