I noticed that “Scheduler” container is eating ~2GB of RAM!!
I just restarted the container and the RAM consumption went down to less than 60MB.
Everything was working fine in both cases - before and after the restart of that container.
I am trying to run:
docker logs {project-name}-scheduler-1
Yet, no output there.
That is very serious as growing RAM usage to 2 GB by one container of 10 that comprise a Frappe setup is huge.
How can I know the reason for this?
Maybe if someone can help me to figure out how to get Logs recorded there… or if there are records, to show these log records.
If this is natural and the scheduler needs that amount of RAM no one is indicating that.
Also, there are no logs to check, is this some jobs running the container and expanding the RAM usage unnecessarily?
Let me call it: situation instead of: problem.
I would assume the DB will be the biggest RAM consumer in a Frappe Instance. Or the Backend container. The Scheduler just runs jobs. It can grow RAM usage for a job with big data, but that should go down after the job completes.
Why to have RAM consumption at 50%, with Scheduler containers taking the lions share of that instead of having a 20% utilization?
What I am doing now, is to find out if it will grow RAM usage beyond 2.2 GB.
I know that at some point in time (many years ago) the Linux Kernel started using RAM for disk access caching, just opportunistically because if it’s unused, it can as well be used for speeding up things.
I’m not sure that this was widely advertised (and if it’s still happening), and if this counts as “used” RAM. Because there can be “used”, and “used but freeable if needed”.
You say the Scheduler just runs jobs. Maybe, but did you ever look at the scheduler’s logs? There can be a lot of stuff going on. Running jobs entails pushing job data into some sort of pipeline and assuring results get where they need to get. Depending on what and how this is done, this could imply lots of data and also caching, indexing, etc. Running scheduler jobs generally takes place for long-running tasks which cannot be done while serving a request, so they get pushed to be run at other times. So these are jobs which typically need more resources than normal reactions of an instance.
It’s easy to make assumptions about things, although things in reality are different.
I don’t say that what you observe is not problematic, it could well be.
But don’t rush to conclusions too quickly either.
@knimer I have seen the same behaviors. Schedulers consuming a huge amount of memory until they restart. Then back to a normal reasonable level (30-50 megabytes)
You are right to be concerned: 2GB usage for a scheduler is insane.
Here’s my suspicion: as the scheduler writes to stdout and stderr, those text strings remain in memory (even if they’re also being written to disk.). When you restart, stdout/stderr are cleared.
For what it’s worth, in my very small deployment, scheduler on docker is sitting at 61MB RAM usage. But schedulers will vary, depending on what they are running. Even so, it’s hard to imagine any deployment using 2GB when functioning “as designed”. Maybe jobs are crashing, getting stuck, or running concurrently, etc. Have you done any customization to the jobs? Maybe a selective restore of the scheduled job type list table(s) on a troubleshooting instance could help isolate things.
EDIT: I also have no output running this from the project folder: docker compose logs scheduler It’s the only container in the whole stack that gives me no logs, so maybe that is expected.
ANOTHER EDIT: Have you checked the scheduled job logs in frappe? Search in the awesome bar: scheduled job log list
There are thousands of it.
I scrolled down to 2500 and got 100% completed status. So, it does not look like crashing jobs.
Yet, few days ago… I started “restarting” the scheduler container every night. Will stop that, which will get the memory usage to grow back and then will check these logs again and get back here.
Yes, the Scheduler container missing logs matter is interesting!