What is the proper configuration for a server with 3000 users?

The supervisor.conf is in the conf directory under bench, along with nginx.conf, etc.

Maybe you mean config directory?

Yes. Sorry. I mean config

Actually this is the symlink to the conf file in config, right?
So opening it is the same?

Hi folks

This what happened today

Setup:

8 core
31gb memory

4 background workers
17 gunicorn workers

Users:
Around 1700-2000

We haven’t tried this yet.

Just wondering whether 2000 users in 1 site is the same burden to the server as 1 user in each of 2000 sites (or 10 users in 200 sites) in the same bench…

Maybe the only party here has this experience is frappe.com :slight_smile:

1 Like

The “burden” does not necessarily have something to do with ERPNext. It has something to do with overcoming python’s GIL. To overcome this, ERPNext uses werkzeug, gunicorn, and nginx. Flask and Django do the same.

You may have a very powerful server but if your gunicorn worker is set to 1, you will end up with a very slow and unreliable server.

You may have a not so powerful server like 2 core, and you set your gunicorn workers to 5 and threads to 10, and also deal with gunicorn’s heartbeat issue in Ubuntu, and you will have a fast ERPNext server serving many users. One benefit of using thread is there will no longer be a worker timeout)

Improvements have been made in werkzeug and gunicorn, and it may be safe to use async gevent workers in gunicorn

To understand gunicorn better:

  1. Workers only (4 workers)

It is like having 4 rooms, and each user can only stay in that room for a number of time-out seconds and then you have to get out. If you are not done within the the time, you get timed out. (Additional problem: vulnerable to DDOS)

  1. Workers and Threads (4 workers 10 threads)

You have 4 rooms with 10 cubicles. A thread does not time out. So, each user may stay in the cubicle as long as you like. (Advantage: No time out, More resistant to DDOS)

The issue with ERPNext gunicorn’s configuration is, it only uses gunicorn workers but does not use threads. This is a serious flaw because ERPNext is I/O bound.

Additional Note: If you use threads, you can try erasing the --preload setting in ERPNext’s gunicorn.

Reference: https://docs.gunicorn.org/en/stable/design.html#:~:text=Gunicorn%20should%20only%20need%204,load%20balancing%20when%20handling%20requests.

6 Likes

I’m using CentOS 8, ERPNExt v12.
And this is in my supervisor.conf -w 4 -t 120 .
It uses thread, right?
I didn’t change anything since installation regarding the -t parameter. So I guess bench setup supervisor has included thread setting?

-t is for timeout, --threads is for thread,

❯ ./env/bin/gunicorn --help
usage: gunicorn [OPTIONS] [APP_MODULE]

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -c CONFIG, --config CONFIG
                        The Gunicorn config file. [None]
  -b ADDRESS, --bind ADDRESS
                        The socket to bind. [['127.0.0.1:8000']]
  --backlog INT         The maximum number of pending connections. [2048]
  -w INT, --workers INT
                        The number of worker processes for handling requests.
                        [1]
  -k STRING, --worker-class STRING
                        The type of workers to use. [sync]
  --threads INT         The number of worker threads for handling requests.
                        [1]
  --worker-connections INT
                        The maximum number of simultaneous clients. [1000]
  --max-requests INT    The maximum number of requests a worker will process
                        before restarting. [0]
  --max-requests-jitter INT
                        The maximum jitter to add to the *max_requests*
                        setting. [0]
  -t INT, --timeout INT
                        Workers silent for more than this many seconds are
                        killed and restarted. [30]
  --graceful-timeout INT
                        Timeout for graceful workers restart. [30]
  --keep-alive INT      The number of seconds to wait for requests on a Keep-
                        Alive connection. [2]
  --limit-request-line INT
                        The maximum size of HTTP request line in bytes. [4094]
  --limit-request-fields INT
                        Limit the number of HTTP headers fields in a request.
                        [100]
  --limit-request-field_size INT
                        Limit the allowed size of an HTTP request header
                        field. [8190]
  --reload              Restart workers when code changes. [False]
  --reload-engine STRING
                        The implementation that should be used to power
                        :ref:`reload`. [auto]
  --reload-extra-file FILES
                        Extends :ref:`reload` option to also watch and reload
                        on additional files [[]]
  --spew                Install a trace function that spews every line
                        executed by the server. [False]
  --check-config        Check the configuration. [False]
  --preload             Load application code before the worker processes are
                        forked. [False]
  --no-sendfile         Disables the use of ``sendfile()``. [None]
  --reuse-port          Set the ``SO_REUSEPORT`` flag on the listening socket.
                        [False]
  --chdir CHDIR         Chdir to specified directory before apps loading.
                        [/home/revant/frappe-bench]
  -D, --daemon          Daemonize the Gunicorn process. [False]
  -e ENV, --env ENV     Set environment variable (key=value). [[]]
  -p FILE, --pid FILE   A filename to use for the PID file. [None]
  --worker-tmp-dir DIR  A directory to use for the worker heartbeat temporary
                        file. [None]
  -u USER, --user USER  Switch worker processes to run as this user. [1000]
  -g GROUP, --group GROUP
                        Switch worker process to run as this group. [1000]
  -m INT, --umask INT   A bit mask for the file mode on files written by
                        Gunicorn. [0]
  --initgroups          If true, set the worker process's group access list
                        with all of the [False]
  --forwarded-allow-ips STRING
                        Front-end's IPs from which allowed to handle set
                        secure headers. [127.0.0.1]
  --access-logfile FILE
                        The Access log file to write to. [None]
  --disable-redirect-access-to-syslog
                        Disable redirect access logs to syslog. [False]
  --access-logformat STRING
                        The access log format. [%(h)s %(l)s %(u)s %(t)s
                        "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"]
  --error-logfile FILE, --log-file FILE
                        The Error log file to write to. [-]
  --log-level LEVEL     The granularity of Error log outputs. [info]
  --capture-output      Redirect stdout/stderr to specified file in
                        :ref:`errorlog`. [False]
  --logger-class STRING
                        The logger you want to use to log events in Gunicorn.
                        [gunicorn.glogging.Logger]
  --log-config FILE     The log config file to use. [None]
  --log-config-dict LOGCONFIG_DICT
                        The log config dictionary to use, using the standard
                        Python [{}]
  --log-syslog-to SYSLOG_ADDR
                        Address to send syslog messages. [udp://localhost:514]
  --log-syslog          Send *Gunicorn* logs to syslog. [False]
  --log-syslog-prefix SYSLOG_PREFIX
                        Makes Gunicorn use the parameter as program-name in
                        the syslog entries. [None]
  --log-syslog-facility SYSLOG_FACILITY
                        Syslog facility name [user]
  -R, --enable-stdio-inheritance
                        Enable stdio inheritance. [False]
  --statsd-host STATSD_ADDR
                        ``host:port`` of the statsd server to log to. [None]
  --statsd-prefix STATSD_PREFIX
                        Prefix to use when emitting statsd metrics (a trailing
                        ``.`` is added, []
  -n STRING, --name STRING
                        A base to use with setproctitle for process naming.
                        [None]
  --pythonpath STRING   A comma-separated list of directories to add to the
                        Python path. [None]
  --paste STRING, --paster STRING
                        Load a PasteDeploy config file. The argument may
                        contain a ``#`` [None]
  --proxy-protocol      Enable detect PROXY protocol (PROXY mode). [False]
  --proxy-allow-from PROXY_ALLOW_IPS
                        Front-end's IPs from which allowed accept proxy
                        requests (comma separate). [127.0.0.1]
  --keyfile FILE        SSL key file [None]
  --certfile FILE       SSL certificate file [None]
  --ssl-version SSL_VERSION
                        SSL version to use (see stdlib ssl module's)
                        [_SSLMethod.PROTOCOL_TLS]
  --cert-reqs CERT_REQS
                        Whether client certificate is required (see stdlib ssl
                        module's) [VerifyMode.CERT_NONE]
  --ca-certs FILE       CA certificates file [None]
  --suppress-ragged-eofs
                        Suppress ragged EOFs (see stdlib ssl module's) [True]
  --do-handshake-on-connect
                        Whether to perform SSL handshake on socket connect
                        (see stdlib ssl module's) [False]
  --ciphers CIPHERS     Ciphers to use (see stdlib ssl module's) [TLSv1]
  --paste-global CONF   Set a PasteDeploy global config variable in
                        ``key=value`` form. [[]]
  --strip-header-spaces
                        Strip spaces present between the header name and the
                        the ``:``. [False]
1 Like

@gavindsouza guess this needs to be taken care of in the bench when generating supervisor.conf.

-w 4 --threads 10 -t 120

Does anyone from the Frappe team have any comment on this? I don’t think it’s a good idea for users to have to edit config files directly…

Thanks

For docker WORKER_CLASS environment variable is configurable. It defaults to gthread. You can use gevent as well. If default like native bench setup is needed use sync

2 Likes