[Important] Bench version 2 - see this before you run bench update

anand · March 22, 2016, 11:52am

Hi everyone,

We have merged bench v2 into master.

Things can break.
So, if you want to continue without disruption, you can change the bench branch to v1.x.

Troubleshooting

If the update breaks at: bench.utils.CommandFailedError: sudo supervisorctl restart frappe-bench-processes:, update again. A fix has been pushed.
If the system keeps asking for sudo password for a user, you will need to go to your frappe-bench folder and run sudo bench setup sudoers USER. Replace USER with frappe / ubuntu or whichever user you used to install erpnext.

If you see an error like below, but if you see that the processes stopped and restarted, you can ignore this line.

error: <class 'xmlrpclib.Fault'>, <Fault 10: 'BAD_NAME: frappe-bench-processes'>: file: /usr/lib/python2.7/xmlrpclib.py line: 794

If you still want to get rid of the above message, you will have to regenerate all your configuration files by going to the frappe-bench folder and running these commands:

mv config config-bak
mkdir -p config/pids
sudo supervisorctl stop all
sudo service nginx stop
bench setup config
bench setup redis
bench setup supervisor
bench setup nginx
bench setup procfile
sudo bench setup sudoers frappe # or the user that you used to install erpnext
sudo nginx -t  # check if this command shows any error 
sudo service nginx reload    
sudo supervisorctl reread
sudo supervisorctl update

If after doing this, you get the following error in sudo supervisorctl status:

    frappe-bench-redis:frappe-bench-redis-queue FATAL      Exited too quickly (process log may have details)

then run:

    sudo update-rc.d redis-server disable
    sudo service redis-server stop
    sudo supervisorctl start all

If you get the error pkg_resources.DistributionNotFound: bench==0.92, try:

    # go back one level outsite frappe-bench folder
    cd ..
    sudo pip uninstall bench
    sudo pip install -e bench-repo

Thank you @Kody_Baker, @revant_one, @Ben_Cornwell_Mott for your persistence

What’s new?

frappe-bench/config.json deprecated and merged into frappe-bench/sites/common_site_config.json. This will be done automatically on bench update
Some keys in site_config.json / common_site_config.json have changed, which will be changed automatically on bench update.
- celery_broker → redis_queue with default as redis://localhost:6379
- async_redis_server → redis_socketio with default as redis://localhost:12311
- cache_redis_server → redis_cache with default as redis://localhost:11311

File names in frappe-bench/config have changed. If you get any error about config, run:

 bench setup redis
 bench setup procfile

 # if production setup
 bench setup supervisor
 bench setup nginx

Multi-bench setup
- Assumes all benches are installed in parallel, i.e. some-folder/frappe-bench-1, some-folder/frappe-bench-2, etc.
- On bench init, loops through all benches’ configuration and sets unique ports for webserver, socketio and various redis-server
- bench setup production creates symlinks of nginx and supervisor with the bench’s name
Other improvements
- A new installer using ansible (WIP) - for now creates development setup
  (production setup yet to be implemented)
- Test cases for bench and refactored code
- Nginx https config auto-redirects http:// to https://

What do you think? I am sure some of you must have questions.

alex_melkoff · March 22, 2016, 4:52pm

@anand, thank you for the announcement! I’ll give it a shot tomorrow.

I have a question about #2. I can’t find any information about redis usage in Frappe, could you please clarify why 3 instances of redis are needed instead of one? Or at least point me at some article.

anand · March 22, 2016, 5:02pm

redis-queue - celery queuing
redis-cache - caching
redis-socketio - pub/sub for realtime using socketio

We are using 3 instances because each redis instance can utilise separate cores of the processor and thus give better throughput. You can use a single redis but partition it in different databases, like redis://localhost:6379/0, redis://localhost:6379/1 and redis://localhost:6379/2, (or use the same redis, which I think will work without issues too).

Frankly, I am not a redis expert. So if you think this is a bad design, we could surely re-evaluate

alex_melkoff · March 22, 2016, 5:34pm

I’m not a redis expert either, but I think it was a good decision in that case.

However there is a trend of using more machines with average hardware instead of one big machine with lots of CPUs and RAM. ERPNext is already pretty big system for one machine and I think some manual is required to cover complex deployment cases.

anand · March 22, 2016, 5:36pm

@alex_melkoff monolithic architecture has been working for us. One of our servers has like 4000 sites. It keeps things simple.

alex_melkoff · March 22, 2016, 5:46pm

@anand, wow, that’s a lot of sites… But what about highload? How many simultaneous users can such a system handle?

anand · March 22, 2016, 5:48pm

@alex_melkoff we don’t have stats on that yet. But in the coming months, we will be capping it to 1000 users a server just for safety. Still that is a big number and turns out to be cheaper and easier to manage. And it is so easy to troubleshoot too

vrms · March 23, 2016, 4:37am

I assume as long as you just do bench update nothing is going to happen apart from updating ERPNext and frappe (using the existing bench), right?

In order to pull the new bench v2, you’d have to specifically perform a git pull inside of the folder where the bench is installed. Is that so?

anand · March 23, 2016, 7:14am

@vrms bench update also updates bench code. Since I can’t possible test it for all scenarios, things could break. So if you don’t want to take even a bit of risk, you should change bench’s branch to v1.x. Or try it out in your local/test setup first.

vrms · March 23, 2016, 8:23am

@anand just to clarify and out of interest

which of the branches in your repo is the v2? The new-install branch?
when precisely are you going to merge? Will there be an extra announcement, so anybody knows, when to be extra cautious with a bench update?

anand · March 23, 2016, 9:56am

Merged

Kody_Baker · March 23, 2016, 5:52pm

Things broke for our installation. Want this posted to github rather than this thread?

Updates everything, but then at the end of setup after writing the ./assets/, it hits this error:

Wrote ./assets/css/module.min.css - 2k
error: <class 'xmlrpclib.Fault'>, <Fault 10: 'BAD_NAME: frappe-bench-processes'>: file: /usr/lib/python2.7/xmlrpclib.py line: 794
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 9, in <module>
    load_entry_point('bench==0.1', 'console_scripts', 'bench')()
  File "/home/ubuntu/bench-repo/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/bench-repo/bench/commands/update.py", line 60, in update
    _update(pull, patch, build, bench, auto, restart_supervisor, requirements, no_backup, upgrade, force=force)
  File "/home/ubuntu/bench-repo/bench/commands/update.py", line 96, in _update
    restart_supervisor_processes(bench=bench_path)
  File "/home/ubuntu/bench-repo/bench/utils.py", line 267, in restart_supervisor_processes
    exec_cmd(cmd, cwd=bench)
  File "/home/ubuntu/bench-repo/bench/utils.py", line 99, in exec_cmd
    raise CommandFailedError(cmd)
bench.utils.CommandFailedError: sudo supervisorctl restart frappe-bench-processes:

For reference: supervisorctl status

frappe:frappe-web                RUNNING    pid 18572, uptime 1 day, 0:55:17
frappe:frappe-worker             RUNNING    pid 18571, uptime 1 day, 0:55:17
frappe:frappe-workerbeat         RUNNING    pid 18580, uptime 1 day, 0:55:16
node-socketio                    RUNNING    pid 1634, uptime 6 days, 0:08:53
redis-async-broker               RUNNING    pid 1633, uptime 6 days, 0:08:53
redis-cache                      RUNNING    pid 1648, uptime 6 days, 0:08:53

anand · March 23, 2016, 6:00pm

Ok let me see if I can make this backward compatible. Give me 5 mins.

anand · March 23, 2016, 6:19pm

@Kody_Baker can you try running update again? I’ve pushed a fix.

In your case, you might need to run cd /home/ubuntu/frappe-bench && setup sudoers ubuntu if it keeps asking you the sudo password for ubuntu user

Kody_Baker · March 23, 2016, 6:25pm

Looks like we’re 99% fixed.
No more traceback, but the error persists.

Wrote ./assets/css/module.min.css - 2k
error: <class 'xmlrpclib.Fault'>, <Fault 10: 'BAD_NAME: frappe-bench-processes'>: file: /usr/lib/python2.7/xmlrpclib.py line: 794
frappe-web: stopped
frappe-workerbeat: stopped
frappe-worker: stopped
frappe-web: started
frappe-workerbeat: started
frappe-worker: started

anand · March 23, 2016, 6:28pm

You can ignore that error till you move to the new configuration. For new configuration you will have to regenerate all config files + nginx config.

Kody_Baker · March 23, 2016, 6:29pm

How do I move to new config?
bench update migrate?

anand · March 23, 2016, 6:38pm

@Kody_Baker see point 4 in troubleshooting in the first post

Kody_Baker · March 23, 2016, 7:14pm

Worked. New error now at the end:

...Wrote ./assets/css/module.min.css - 2k
frappe-bench-frappe-async-worker: stopped
frappe-bench-frappe-longjob-worker: stopped
frappe-bench-frappe-workerbeat: stopped
frappe-bench-frappe-web: stopped
frappe-bench-frappe-worker: stopped
frappe-bench-frappe-async-worker: started
frappe-bench-frappe-longjob-worker: started
frappe-bench-frappe-workerbeat: started
frappe-bench-frappe-web: started
frappe-bench-frappe-worker: started
frappe-bench-node-socketio: ERROR (abnormal termination)