Bench fails to start after a bench update

1

Hey folks,

This is what I’m getting, I moved from develop to master and back to develop…

12:57:41 system           | redis_socketio.1 started (pid=1036)
12:57:41 redis_socketio.1 | 1043:C 21 Mar 12:57:41.582 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
12:57:41 redis_socketio.1 | 1043:C 21 Mar 12:57:41.582 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=1043, just started
12:57:41 redis_socketio.1 | 1043:C 21 Mar 12:57:41.582 # Configuration loaded
12:57:41 redis_socketio.1 | 1043:M 21 Mar 12:57:41.582 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:57:41 redis_socketio.1 | 1043:M 21 Mar 12:57:41.583 * Running mode=standalone, port=12000.
12:57:41 redis_socketio.1 | 1043:M 21 Mar 12:57:41.583 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
12:57:41 redis_socketio.1 | 1043:M 21 Mar 12:57:41.583 # Server initialized
12:57:41 redis_socketio.1 | 1043:M 21 Mar 12:57:41.584 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
12:57:41 redis_socketio.1 | 1043:M 21 Mar 12:57:41.584 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
12:57:41 redis_socketio.1 | 1043:M 21 Mar 12:57:41.584 * Ready to accept connections
12:57:41 system           | worker_long.1 started (pid=1047)
12:57:41 system           | watch.1 started (pid=1048)
12:57:41 system           | redis_queue.1 started (pid=1051)
12:57:41 redis_queue.1    | 1054:C 21 Mar 12:57:41.606 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
12:57:41 redis_queue.1    | 1054:C 21 Mar 12:57:41.606 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=1054, just started
12:57:41 redis_queue.1    | 1054:C 21 Mar 12:57:41.606 # Configuration loaded
12:57:41 redis_queue.1    | 1054:M 21 Mar 12:57:41.606 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:57:41 redis_queue.1    | 1054:M 21 Mar 12:57:41.607 * Running mode=standalone, port=11000.
12:57:41 redis_queue.1    | 1054:M 21 Mar 12:57:41.607 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
12:57:41 redis_queue.1    | 1054:M 21 Mar 12:57:41.607 # Server initialized
12:57:41 redis_queue.1    | 1054:M 21 Mar 12:57:41.607 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
12:57:41 redis_queue.1    | 1054:M 21 Mar 12:57:41.607 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
12:57:41 redis_queue.1    | 1054:M 21 Mar 12:57:41.607 * Ready to accept connections
12:57:41 system           | web.1 started (pid=1053)
12:57:41 system           | schedule.1 started (pid=1059)
12:57:41 system           | worker_default.1 started (pid=1060)
12:57:41 system           | socketio.1 started (pid=1061)
12:57:41 system           | redis_cache.1 started (pid=1063)
12:57:41 redis_cache.1    | 1072:C 21 Mar 12:57:41.626 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
12:57:41 redis_cache.1    | 1072:C 21 Mar 12:57:41.626 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=1072, just started
12:57:41 redis_cache.1    | 1072:C 21 Mar 12:57:41.626 # Configuration loaded
12:57:41 redis_cache.1    | 1072:M 21 Mar 12:57:41.626 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:57:41 redis_cache.1    | 1072:M 21 Mar 12:57:41.627 * Running mode=standalone, port=13000.
12:57:41 redis_cache.1    | 1072:M 21 Mar 12:57:41.627 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
12:57:41 redis_cache.1    | 1072:M 21 Mar 12:57:41.627 # Server initialized
12:57:41 redis_cache.1    | 1072:M 21 Mar 12:57:41.627 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
12:57:41 redis_cache.1    | 1072:M 21 Mar 12:57:41.627 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
12:57:41 redis_cache.1    | 1072:M 21 Mar 12:57:41.627 * Ready to accept connections
12:57:41 system           | worker_short.1 started (pid=1066)
12:57:42 socketio.1       | listening on *: 9000
12:57:44 worker_long.1    | 12:57:44 RQ worker u'rq:worker:erpnext.1049.long' started, version 0.10.0
12:57:44 worker_long.1    | 12:57:44 *** Listening on long...
12:57:44 worker_long.1    | 12:57:44 Cleaning registries for queue: long
12:57:45 worker_short.1   | 12:57:45 RQ worker u'rq:worker:erpnext.1075.short' started, version 0.10.0
12:57:45 worker_short.1   | 12:57:45 *** Listening on short...
12:57:45 worker_short.1   | 12:57:45 Cleaning registries for queue: short
12:57:45 web.1            |  * Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
12:57:45 worker_default.1 | 12:57:45 RQ worker u'rq:worker:erpnext.1064.default' started, version 0.10.0
12:57:45 worker_default.1 | 12:57:45 *** Listening on default...
12:57:45 worker_default.1 | 12:57:45 Cleaning registries for queue: default
12:57:45 web.1            |  * Restarting with inotify reloader
12:57:45 web.1            |  * Debugger is active!
12:57:45 web.1            |  * Debugger PIN: 836-062-199
12:57:45 watch.1          | compiling avatar.less
12:57:45 watch.1          | compiling calendar.less
12:57:45 watch.1          | compiling common.less
12:57:45 watch.1          | compiling controls.less
12:57:45 watch.1          | compiling desk.less
12:57:45 watch.1          | compiling desktop.less
12:57:45 watch.1          | compiling docs.less
12:57:45 watch.1          | compiling email.less
12:57:45 watch.1          | compiling form.less
12:57:45 watch.1          | compiling form_grid.less
12:57:45 watch.1          | compiling gantt.less
12:57:45 watch.1          | compiling indicator.less
12:57:45 watch.1          | compiling kanban.less
12:57:45 watch.1          | compiling list.less
12:57:45 watch.1          | compiling mixins.less
12:57:45 watch.1          | compiling mobile.less
12:57:45 watch.1          | compiling module.less
12:57:45 watch.1          | compiling navbar.less
12:57:45 watch.1          | compiling page.less
12:57:45 watch.1          | compiling report.less
12:57:45 watch.1          | compiling role_editor.less
12:57:45 watch.1          | compiling sidebar.less
12:57:45 watch.1          | compiling slickgrid.less
12:57:45 watch.1          | compiling tags.less
12:57:45 watch.1          | compiling tree.less
12:57:45 watch.1          | compiling website.less
12:57:45 watch.1          | compiling email.less
12:57:45 watch.1          | compiling erpnext.less
12:57:45 watch.1          | compiling hub.less
12:57:45 watch.1          | compiling pos.less
12:57:45 watch.1          | compiling website.less
12:57:45 watch.1          | file watching on *: 6787
12:57:45 watch.1          | Error compiling  erpnext.less
12:57:45 watch.1          | { [Error: 'variables.less' wasn't found. Tried - variables.less,/home/frappe/frappe-bench/apps/erpnext/erpnext/public/less/variables.less,/home/frappe/frappe-bench/sites/node_modules/variables.less,variables.less]
12:57:45 watch.1          |   message: '\'variables.less\' wasn\'t found. Tried - variables.less,/home/frappe/frappe-bench/apps/erpnext/erpnext/public/less/variables.less,/home/frappe/frappe-bench/sites/node_modules/variables.less,variables.less',
12:57:45 watch.1          |   stack: undefined,
12:57:45 watch.1          |   type: 'File',
12:57:45 watch.1          |   filename: 'erpnext.less',
12:57:45 watch.1          |   index: 0,
12:57:45 watch.1          |   line: 1,
12:57:45 watch.1          |   column: 0,
12:57:45 watch.1          |   callLine: NaN,
12:57:45 watch.1          |   callExtract: undefined,
12:57:45 watch.1          |   extract: [ undefined, '@import "variables.less";', '' ] }
12:57:46 watch.1          | Less files compiled
12:57:46 watch.1          | Wrote css/frappe-web.css - 65.43 KB
12:57:46 watch.1          | Wrote js/frappe-web.min.js - 135.24 KB
12:57:46 watch.1          | Wrote js/control.min.js - 81.34 KB
12:57:46 watch.1          | Wrote js/dialog.min.js - 124.09 KB
12:57:46 watch.1          | Wrote css/desk.min.css - 311 KB
12:57:46 watch.1          | Wrote css/frappe-rtl.css - 32.73 KB
12:57:46 watch.1          | Wrote js/libs.min.js - 1.12 MB
12:57:46 watch.1          | Wrote js/desk.min.js - 477.83 KB
12:57:46 watch.1          | Wrote css/module.min.css - 2.16 KB
12:57:46 watch.1          | Wrote css/form.min.css - 4.55 KB
12:57:46 watch.1          | Wrote js/form.min.js - 205.68 KB
12:57:46 watch.1          | Wrote css/list.min.css - 13.67 KB
12:57:46 watch.1          | Wrote js/list.min.js - 161.76 KB
12:57:46 watch.1          | Wrote css/report.min.css - 8.31 KB
12:57:46 watch.1          | Wrote js/report.min.js - 262.94 KB
12:57:46 watch.1          | Wrote js/web_form.min.js - 248.66 KB
12:57:46 watch.1          | Wrote css/web_form.css - 24.69 KB
12:57:46 watch.1          | Wrote js/print_format_v3.min.js - 25.87 KB
12:57:46 watch.1          | Wrote css/erpnext.css - 6.41 KB
12:57:46 watch.1          | Wrote js/erpnext-web.min.js - 4.23 KB
12:57:46 watch.1          | Wrote js/erpnext.min.js - 172.32 KB
12:57:46 watch.1          | Wrote js/item-dashboard.min.js - 8.41 KB
12:57:48 watch.1          | events.js:163
12:57:48 watch.1          |       throw er; // Unhandled 'error' event
12:57:48 watch.1          |       ^
12:57:48 watch.1          |
12:57:48 watch.1          | Error: watch /home/frappe/frappe-bench/apps/frappe/frappe/integrations/doctype/oauth_bearer_token ENOSPC
12:57:48 watch.1          |     at exports._errnoException (util.js:1050:11)
12:57:48 watch.1          |     at FSWatcher.start (fs.js:1398:19)
12:57:48 watch.1          |     at Object.fs.watch (fs.js:1424:11)
12:57:48 watch.1          |     at createFsWatchInstance (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:37:15)
12:57:48 watch.1          |     at setFsWatchListener (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:80:15)
12:57:48 watch.1          |     at FSWatcher.NodeFsHandler._watchWithNodeFs (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:228:14)
12:57:48 watch.1          |     at FSWatcher.NodeFsHandler._handleDir (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:407:19)
12:57:48 watch.1          |     at FSWatcher.<anonymous> (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:455:19)
12:57:48 watch.1          |     at FSWatcher.<anonymous> (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:460:16)
12:57:48 watch.1          |     at FSReqWrap.oncomplete (fs.js:114:15)
12:57:48 system           | watch.1 stopped (rc=0)
12:57:48 system           | sending SIGTERM to redis_socketio.1 (pid 1036)
12:57:48 system           | sending SIGTERM to worker_long.1 (pid 1047)
12:57:48 system           | sending SIGTERM to redis_queue.1 (pid 1051)
12:57:48 system           | sending SIGTERM to web.1 (pid 1053)
12:57:48 system           | sending SIGTERM to schedule.1 (pid 1059)
12:57:48 system           | sending SIGTERM to worker_default.1 (pid 1060)
12:57:48 system           | sending SIGTERM to socketio.1 (pid 1061)
12:57:48 system           | sending SIGTERM to redis_cache.1 (pid 1063)
12:57:48 system           | sending SIGTERM to worker_short.1 (pid 1066)
12:57:48 system           | socketio.1 stopped (rc=-15)
12:57:48 worker_short.1   | 12:57:48 Warm shut down requested
12:57:48 redis_cache.1    | 1072:signal-handler (1521637068) Received SIGTERM scheduling shutdown...
12:57:48 redis_socketio.1 | 1043:signal-handler (1521637068) Received SIGTERM scheduling shutdown...
12:57:48 worker_long.1    | 12:57:48 Warm shut down requested
12:57:48 worker_default.1 | 12:57:48 Warm shut down requested
12:57:48 system           | schedule.1 stopped (rc=-15)
12:57:48 redis_queue.1    | 1054:signal-handler (1521637068) Received SIGTERM scheduling shutdown...
12:57:48 redis_socketio.1 | 1043:M 21 Mar 12:57:48.509 # User requested shutdown...
12:57:48 redis_socketio.1 | 1043:M 21 Mar 12:57:48.509 * Removing the pid file.
12:57:48 redis_socketio.1 | 1043:M 21 Mar 12:57:48.509 # Redis is now ready to exit, bye bye...
12:57:48 system           | redis_socketio.1 stopped (rc=-15)
12:57:48 redis_queue.1    | 1054:M 21 Mar 12:57:48.527 # User requested shutdown...
12:57:48 redis_queue.1    | 1054:M 21 Mar 12:57:48.527 * Removing the pid file.
12:57:48 redis_queue.1    | 1054:M 21 Mar 12:57:48.527 # Redis is now ready to exit, bye bye...
12:57:48 system           | redis_queue.1 stopped (rc=-15)
12:57:48 system           | worker_long.1 stopped (rc=-15)
12:57:48 redis_cache.1    | 1072:M 21 Mar 12:57:48.539 # User requested shutdown...
12:57:48 redis_cache.1    | 1072:M 21 Mar 12:57:48.539 * Removing the pid file.
12:57:48 redis_cache.1    | 1072:M 21 Mar 12:57:48.539 # Redis is now ready to exit, bye bye...
12:57:48 system           | redis_cache.1 stopped (rc=-15)
12:57:48 system           | worker_default.1 stopped (rc=-15)
12:57:48 system           | worker_short.1 stopped (rc=-15)
12:57:48 system           | web.1 stopped (rc=-15)
2

Try executing

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Ref: Node.js: what is ENOSPC error and how to solve? - Stack Overflow

1 Like
3

I had already done that, my sysctl.conf has the following additional entries:

vm.overcommit_memory = 1
fs.inotify.max_user_watches=524288

The above trace is actually after those changes. I’m running in a Virtual container (OpenVZ).

4

I don’t see the problem with the VM since it is 2Gb RAM, with 400 Gb HDD, in any case those are warnings and not errors. Can I reinstall redis through bench?

5

This is what I get at bench start, any idea how can I bring back my dead system?

15:26:56 watch.1          | compiling pos.less
15:26:56 watch.1          | compiling website.less
15:26:56 watch.1          | file watching on *: 6787
15:26:56 watch.1          | Error compiling  erpnext.less
15:26:56 watch.1          | { [Error: 'variables.less' wasn't found. Tried - variables.less,/home/frappe/frappe-bench/apps/erpnext/erpnext/public/less/variables.less,/home/frappe/frappe-bench/sites/node_modules/variables.less,variables.less]
15:26:56 watch.1          |   message: '\'variables.less\' wasn\'t found. Tried - variables.less,/home/frappe/frappe-bench/apps/erpnext/erpnext/public/less/variables.less,/home/frappe/frappe-bench/sites/node_modules/variables.less,variables.less',
15:26:56 watch.1          |   stack: undefined,
15:26:56 watch.1          |   type: 'File',
15:26:56 watch.1          |   filename: 'erpnext.less',
15:26:56 watch.1          |   index: 0,
15:26:56 watch.1          |   line: 1,
15:26:56 watch.1          |   column: 0,
15:26:56 watch.1          |   callLine: NaN,
15:26:56 watch.1          |   callExtract: undefined,
15:26:56 watch.1          |   extract: [ undefined, '@import "variables.less";', '' ] }
15:26:56 watch.1          | Less files compiled
15:26:56 watch.1          | Wrote css/frappe-web.css - 65.43 KB
15:26:56 watch.1          | Wrote js/frappe-web.min.js - 135.24 KB
15:26:56 watch.1          | Wrote js/control.min.js - 81.34 KB
15:26:56 watch.1          | Wrote js/dialog.min.js - 124.09 KB
15:26:56 watch.1          | Wrote css/desk.min.css - 311 KB
15:26:56 watch.1          | Wrote css/frappe-rtl.css - 32.73 KB
15:26:56 watch.1          | Wrote js/libs.min.js - 1.12 MB
15:26:56 watch.1          | Wrote js/desk.min.js - 477.83 KB
15:26:56 watch.1          | Wrote css/module.min.css - 2.16 KB
15:26:56 watch.1          | Wrote css/form.min.css - 4.55 KB
15:26:56 watch.1          | Wrote js/form.min.js - 205.68 KB
15:26:56 watch.1          | Wrote css/list.min.css - 13.67 KB
15:26:56 watch.1          | Wrote js/list.min.js - 161.76 KB
15:26:56 watch.1          | Wrote css/report.min.css - 8.31 KB
15:26:56 watch.1          | Wrote js/report.min.js - 262.94 KB
15:26:56 watch.1          | Wrote js/web_form.min.js - 248.66 KB
15:26:56 watch.1          | Wrote css/web_form.css - 24.69 KB
15:26:56 watch.1          | Wrote js/print_format_v3.min.js - 25.87 KB
15:26:56 watch.1          | Wrote css/erpnext.css - 6.41 KB
15:26:56 watch.1          | Wrote js/erpnext-web.min.js - 4.23 KB
15:26:56 watch.1          | Wrote js/erpnext.min.js - 172.32 KB
15:26:56 watch.1          | Wrote js/item-dashboard.min.js - 8.41 KB
15:26:58 watch.1          | events.js:163
15:26:58 watch.1          |       throw er; // Unhandled 'error' event
15:26:58 watch.1          |       ^
15:26:58 watch.1          |
15:26:58 watch.1          | Error: watch /home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/docfield ENOSPC
15:26:58 watch.1          |     at exports._errnoException (util.js:1050:11)
15:26:58 watch.1          |     at FSWatcher.start (fs.js:1398:19)
15:26:58 watch.1          |     at Object.fs.watch (fs.js:1424:11)
15:26:58 watch.1          |     at createFsWatchInstance (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:37:15)
15:26:58 watch.1          |     at setFsWatchListener (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:80:15)
15:26:58 watch.1          |     at FSWatcher.NodeFsHandler._watchWithNodeFs (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:228:14)
15:26:58 watch.1          |     at FSWatcher.NodeFsHandler._handleDir (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:407:19)
15:26:58 watch.1          |     at FSWatcher.<anonymous> (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:455:19)
15:26:58 watch.1          |     at FSWatcher.<anonymous> (/home/frappe/frappe-bench/node_modules/chokidar/lib/nodefs-handler.js:460:16)
15:26:58 watch.1          |     at FSReqWrap.oncomplete (fs.js:114:15)
15:26:58 system           | watch.1 stopped (rc=0)
15:26:58 system           | sending SIGTERM to redis_socketio.1 (pid 5800)
15:26:58 system           | sending SIGTERM to worker_long.1 (pid 5814)
15:26:58 system           | sending SIGTERM to redis_queue.1 (pid 5817)
15:26:58 system           | sending SIGTERM to web.1 (pid 5834)
15:26:58 system           | sending SIGTERM to schedule.1 (pid 5835)
15:26:58 system           | sending SIGTERM to worker_default.1 (pid 5836)
15:26:58 system           | sending SIGTERM to socketio.1 (pid 5841)
15:26:58 system           | sending SIGTERM to redis_cache.1 (pid 5815)
6

Pardon me to ask this again. But did you run

sudo sysctl -p

after making the changes? Since the error log clearly states it’s an ENOSPC error, which is always resolved by making the given changes.

7

This is what sysctl tells me:

frappe@erpnext:~/frappe-bench$ sudo sysctl -p
sysctl: setting key "vm.overcommit_memory": Read-only file system
sysctl: setting key "fs.inotify.max_user_watches": Read-only file system
8

Which means it is failing!
I think it is an drawback of using OpenVZ, or rather any container based system

Ref:

9

Yes a permissions issue with the container setup it seems…

10

Its not a real permissions issue. Requiring to play with such parameters in the kernel of the host or guest OS is not correct. I don’t see lack of resources in the VM that we’re using in our testing, we can easily crank them up, but requiring to play with kernel parameters is not something that leads to a safe/stable system. To hack this I can easily setup host parameters, which will obviously have unknown and unpredictable errors in the other guests which are currently being hosted on the hypervisor. However, one can setup those parameters and in the upstream run the guest in a privileged container. This is unacceptable solution for an enterprise ERP. This appears to be coming from either Rdis or bench/docker and in my humble opinion, its a major issue and in my opinion a serious bug in the implementation.

11

I do agree with you and will leave you to your devices!

1 Like
12

Hi CodingCoffee, thanks for getting back to me. Actually, I disagree with your statement that this is a drawback of using OpenVZ, it may be so, but an enterprise ERP should not depend on, and better still, not require, any kernel modifications to make it run. Like I said below, we can increase the system resources at will, the current VM is not even taxed upto 2% of its capacity and therefore, it does not make sense to make kernel changes. Its not good even as an intermediate solution.

And I really need a solution since we’ve got a system that is being tried out in the company to give credentials to an OSS experiment and its efficacy and stability in running our operations before we make serious inroads into it or major investments.

13

I think we’ll have to dive into workings of Linux for this one,

What is ENOSPC?
It stands for Error No space on device. But you do have space on your Virtual Machine,so what is the problem? The thing is ENOSPC is actually referring to lack of inodes.

What are inodes?

The inode is a data structure in a Unix-style file system that describes a filesystem object such as a file or a directory. Each inode stores the attributes and disk block location(s) of the object’s data. Filesystem object attributes may include metadata (times of last change, access, modification), as well as owner and permission data.

Refs:
http://www.noah.org/wiki/ENOSPC_error,yet%27df%27_shows_available_space
https://en.wikipedia.org/wiki/Inode

ERPNext is a full fledged ERP. Our tagline itself says “ERP for Everybody”. ERPNext supports multiple domains including, but not limited to accounting, inventory, manufacturing, CRM, sales, purchase, distribution, services, retail, project management, education, healthcare and non profit which bascically translates to a large code base. And regardless of which domain you need or want to use, you’ll get the entire deal.

Now it is due to this that the system needs a whole lot of inodes, which means modifying the kernel parameters. But I don’t see it to be that big of a deal.

Now to give you a solution:
Maybe you can change the parameters on the host system, which I doubt you’ll do, so maybe try it out by installing in a non container-based VPS !?

3 Likes
14

Correct, I’m not going to change the hypervisor params. I’m somewhat familiar with Linux, being one of the first guys to have worked on it in circa 1991-1997. There are other solutions that could make it redundant to use something like inodes in the kernel space. But thats going off topic and thats for you folks to decide…

I was indeed thinking of moving our implementation out of an OpenVZ into a VM, without losing data.

15

try this
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Thanks

1 Like
16

@nash if you could suggest anything that would be great !

That would be awesome!! :smile: