Bench migrate too slow

Install / Update Setup
1

Hi everyone, I have an Issue with bench migrate, every time that I did migrate my screen stays about 15 minutes this way, in every bench migrate I need wait this time.
Obs: When I install erpnext or did bench update happens the same thing.

migrate.jpeg1280×720 88.4 KB

If someone know how to fix it, I appreciate, thanks for any help.

2

Make sure you keep the redis cache on when you migrate! (run bench start on another tab)

Could the issue be in your custom app?

1 Like
3

ok @rmehta, thanks for help.

4

Same issue with me it takes 6-7 hours to complete bench migrate command
on test server I have less data so it completes under 5-10 mins but on production I have lots of data so it takes 6-7 hours

It fires same query again and again on bench migrate

SELECT * FROM `tabUser Email` WHERE `parent`='test@gmail.com' AND `parenttype`='User' AND `parentfield`='user_emails' ORDER BY `idx` ASC
Execution time: 0.00 sec
SELECT * FROM `tabBlock Module` WHERE `parent`='test@gmail.com' AND `parenttype`='User' AND `parentfield`='block_modules' ORDER BY `idx` ASC
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1
Execution time: 0.00 sec
SELECT `module`,`custom` FROM `tabDocType` WHERE `name`='Block Module' ORDER BY `modified` DESC LIMIT 1

any suggestion?

5

I solved my issue by remove role and role profile file from fixtures, it was looping through all users and roles and I have near about 8K users

1 Like
6

This should fix role fixture migration performance issue: perf(migrate): Only re-evaluate users which are potentially changed by ankush · Pull Request #23820 · frappe/frappe · GitHub

And role profile: perf(migrate): update only affected users via role profile by ankush · Pull Request #23821 · frappe/frappe · GitHub

2 Likes