ERPNext on Kubernetes - Intermittent Slow API Response Despite Healthy MySQL & Redis

Frappe Framework
1

I am running ERPNext on Kubernetes with an external MySQL server and Redis. I have monitored MySQL performance metrics, and there are no signs of issues.

To further investigate, I integrated OpenTelemetry for tracing. As shown in the attached screenshot, Redis and MySQL performance appear to be fine. However, I’m experiencing intermittent slow responses when calling a single backend endpoint.

For example, in Jaeger traces, a simple login request sometimes takes an unexpectedly long time, despite no apparent database or Redis bottlenecks.

What else can I check? Could cache expiration be causing slow page loads when the cache is invalidated? Any insights or debugging tips would be appreciated.

get
get1280×405 52.3 KB

login
login1280×405 48.3 KB

2

Hi,

See if this thread is helpful:

https://discuss.frappe.io/t/erp-schedule-slow-in-kubernetes-and-fast-in-docker/140630/3

3

Actually I also saw that. And tried everything on that thread.

4

Where are the gunicorn traces? try calling few endpoints and check traces.

demo of gunicorn traces: otel collector demo · Issue #1573 · frappe/frappe_docker · GitHub

gunicorn command changes,

OTEL_EXPORTER_OTLP_TRACES_INSECURE=true OTEL_EXPORTER_OTLP_ENDPOINT=jaeger:4317 OTEL_RESOURCE_ATTRIBUTES=service.name=client-sites-gunicorn /home/frappe/frappe-bench/env/bin/opentelemetry-instrument \
    /home/frappe/frappe-bench/env/bin/gunicorn \
    --chdir=/home/frappe/frappe-bench/sites \
    --bind=0.0.0.0:8000 \
    --threads=4 \
    --workers=2 \
    --log-level=debug \
    --worker-class=gthread \
    --worker-tmp-dir=/dev/shm \
    --timeout=120 \
    --preload \
    frappe_otel.app:application

Add HPA if you can see the cpu going up for gunicorn pods when service is slow.

5

Total doesn’t add up to 7.55, check the otel auto-instrumentation and config. I shared example trace for specific endpoint on github issue.

6

I have a dedicated application for telemetry and have configured OpenTelemetry accordingly.

The main issue I’m facing is that when no one has used the app for a while, the first page load takes an extremely long time. Additionally, I suspect that when the cache is invalidated, page loads become significantly slower.

From the OpenTelemetry traces, Redis and MySQL respond within milliseconds, yet the overall page load time remains high.

Our Kubernetes cluster performs well in other applications across the company, and this issue occurs only in ERPNext. Additionally, we experience the same problem in the test environment.

What could be causing this? Are there specific areas I should investigate to optimize the initial load time?

I am adding a screenshot from Jaeger.

image
image1728×799 74.8 KB

7

Can you get what traces are taking 8+ sec?

I can see it’s 8 sec. What is taking 8 sec? Which db query or redis query or jinja rendering?

I’ve seen frappe apps (no erpnext) at hyperscale. The team that made frappe apps made sure no request is taking more than 3sec. It was done using traces and basics discussed by Ankush during v14 release. (youtube video available)

It is important to know internal traces to decide if code/queries need changes or setup/configuration needs to be improved.

8

After really long debugging sessions we found out that we are having latencies between our kubernetes nodes. So after fixing network problems our instance started work really fast ( arround 10x more).

Thanks a lot for the support!

1 Like