Assets stop loading after some days in dokploy

[kaksarkarr]

Hello, I have ERPNext installed using the templates inside Dokploy. The instance works perfectly at first, but after about 5–15 days, all the assets (CSS and JS) suddenly stop working. This has happened three times now. After installation, everything runs fine for a while, then the assets fail to load, and I have to reinstall which is very frustrating.

Here are my observations:

The website requests these asset files:

website.bundle.NSZNBQCJ.css  
login.bundle.IGORFKYS.css  
frappe-web.bundle.4QVP5SVA.js  
frappe-web.bundle.4QVP5SVA.js

However, the assets.json file points to completely different file names.

frappe@31efd83389d8:/$ cat /home/frappe/frappe-bench/sites/assets/assets.json 
{
    "billing.bundle.js": "/assets/frappe/dist/js/billing.bundle.QQE2RPCA.js",
    "bootstrap-4-web.bundle.js": "/assets/frappe/dist/js/bootstrap-4-web.bundle.FOZOVELL.js",
    "controls.bundle.js": "/assets/frappe/dist/js/controls.bundle.NZC3GXOU.js",
    "data_import_tools.bundle.js": "/assets/frappe/dist/js/data_import_tools.bundle.IGMLQRF6.js",
    "desk.bundle.js": "/assets/frappe/dist/js/desk.bundle.IMPJIRYH.js",
    "dialog.bundle.js": "/assets/frappe/dist/js/dialog.bundle.YHWHYNFQ.js",
    "form.bundle.js": "/assets/frappe/dist/js/form.bundle.7PHQTBOA.js",
    "frappe-web.bundle.js": "/assets/frappe/dist/js/frappe-web.bundle.5RBX33XN.js",
    "libs.bundle.js": "/assets/frappe/dist/js/libs.bundle.TIV7ZGVY.js",
    "list.bundle.js": "/assets/frappe/dist/js/list.bundle.ET452RHV.js",
    "logtypes.bundle.js": "/assets/frappe/dist/js/logtypes.bundle.EKN7LWKW.js",
    "onboarding_tours.bundle.js": "/assets/frappe/dist/js/onboarding_tours.bundle.RAUR6X4Z.js",
    "report.bundle.js": "/assets/frappe/dist/js/report.bundle.AMKILTJH.js",
    "sentry.bundle.js": "/assets/frappe/dist/js/sentry.bundle.AX44GVWW.js",
    "telemetry.bundle.js": "/assets/frappe/dist/js/telemetry.bundle.LKEZCADB.js",
    "user_profile_controller.bundle.js": "/assets/frappe/dist/js/user_profile_controller.bundle.TPZWXYWN.js",
    "video_player.bundle.js": "/assets/frappe/dist/js/video_player.bundle.DUYYLSFO.js",
    "web_form.bundle.js": "/assets/frappe/dist/js/web_form.bundle.5DVA6C7G.js",
    "form_builder.bundle.js": "/assets/frappe/dist/js/form_builder.bundle.CYK52TRC.js",
    "print_format_builder.bundle.js": "/assets/frappe/dist/js/print_format_builder.bundle.3YCNST3U.js",
    "workflow_builder.bundle.js": "/assets/frappe/dist/js/workflow_builder.bundle.3GDVTW3J.js",
    "build_events.bundle.js": "/assets/frappe/dist/js/build_events.bundle.BCX32TWL.js",
    "file_uploader.bundle.js": "/assets/frappe/dist/js/file_uploader.bundle.IO2BUJDB.js",
    "kanban_board.bundle.js": "/assets/frappe/dist/js/kanban_board.bundle.FBBHCLAN.js",
    "desk.bundle.css": "/assets/frappe/dist/css/desk.bundle.YEWO6UTN.css",
    "email.bundle.css": "/assets/frappe/dist/css/email.bundle.NAP7GUIG.css",
    "login.bundle.css": "/assets/frappe/dist/css/login.bundle.EEV2YCSU.css",
    "print.bundle.css": "/assets/frappe/dist/css/print.bundle.5UWPJEM3.css",
    "print_format.bundle.css": "/assets/frappe/dist/css/print_format.bundle.NVW3TB2Z.css",
    "report.bundle.css": "/assets/frappe/dist/css/report.bundle.GC4I5GYW.css",
    "web_form.bundle.css": "/assets/frappe/dist/css/web_form.bundle.7LGJLN7Y.css",
    "website.bundle.css": "/assets/frappe/dist/css/website.bundle.QFVSUF2X.css",
    "bank-reconciliation-tool.bundle.js": "/assets/erpnext/dist/js/bank-reconciliation-tool.bundle.PQSY2TRN.js",
    "erpnext-web.bundle.js": "/assets/erpnext/dist/js/erpnext-web.bundle.J4A2DQB4.js",
    "erpnext.bundle.js": "/assets/erpnext/dist/js/erpnext.bundle.F4VLHMIR.js",
    "item-dashboard.bundle.js": "/assets/erpnext/dist/js/item-dashboard.bundle.33GEOV2J.js",
    "point-of-sale.bundle.js": "/assets/erpnext/dist/js/point-of-sale.bundle.JTJY4AB2.js",
    "bom_configurator.bundle.js": "/assets/erpnext/dist/js/bom_configurator.bundle.VB3HBFNX.js",
    "erpnext-web.bundle.css": "/assets/erpnext/dist/css/erpnext-web.bundle.6B3MDDZU.css",
    "erpnext.bundle.css": "/assets/erpnext/dist/css/erpnext.bundle.JQ7CB7T3.css",
    "erpnext_email.bundle.css": "/assets/erpnext/dist/css/erpnext_email.bundle.47GDA6MO.css"
}

Also, the /assets directory actually contains the files listed in assets.json, but the website still tries to load the wrong ones.

This is the compose file from the template

x-custom-image: &custom_image
  image: ${IMAGE_NAME:-docker.io/frappe/erpnext}:${VERSION:-version-15}
  pull_policy: ${PULL_POLICY:-always}
  deploy:
    restart_policy:
      condition: always

services:
  backend:
    <<: *custom_image
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network
    healthcheck:
      test:
        - CMD
        - wait-for-it
        - '0.0.0.0:8000'
      interval: 2s
      timeout: 10s
      retries: 30

  frontend:
    <<: *custom_image
    command:
      - nginx-entrypoint.sh
    depends_on:
      backend:
        condition: service_started
        required: true
      websocket:
        condition: service_started
        required: true
    environment:
      BACKEND: backend:8000
      FRAPPE_SITE_NAME_HEADER: ${FRAPPE_SITE_NAME_HEADER:-$$host}
      SOCKETIO: websocket:9000
      UPSTREAM_REAL_IP_ADDRESS: 127.0.0.1
      UPSTREAM_REAL_IP_HEADER: X-Forwarded-For
      UPSTREAM_REAL_IP_RECURSIVE: "off"
    volumes:
      - sites:/home/frappe/frappe-bench/sites

    networks:
      - bench-network
    
    healthcheck:
      test:
        - CMD
        - wait-for-it
        - '0.0.0.0:8080'
      interval: 2s
      timeout: 30s
      retries: 30

  queue-default:
    <<: *custom_image
    command:
      - bench
      - worker
      - --queue
      - default
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network
    healthcheck:
      test:
        - CMD
        - wait-for-it
        - 'redis-queue:6379'
      interval: 2s
      timeout: 10s
      retries: 30
    depends_on:
      configurator:
        condition: service_completed_successfully
        required: true

  queue-long:
    <<: *custom_image
    command:
      - bench
      - worker
      - --queue
      - long
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network
    healthcheck:
      test:
        - CMD
        - wait-for-it
        - 'redis-queue:6379'
      interval: 2s
      timeout: 10s
      retries: 30
    depends_on:
      configurator:
        condition: service_completed_successfully
        required: true

  queue-short:
    <<: *custom_image
    command:
      - bench
      - worker
      - --queue
      - short
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network
    healthcheck:
      test:
        - CMD
        - wait-for-it
        - 'redis-queue:6379'
      interval: 2s
      timeout: 10s
      retries: 30
    depends_on:
      configurator:
        condition: service_completed_successfully
        required: true

  scheduler:
    <<: *custom_image
    healthcheck:
      test:
        - CMD
        - wait-for-it
        - 'redis-queue:6379'
      interval: 2s
      timeout: 10s
      retries: 30
    command:
      - bench
      - schedule
    depends_on:
      configurator:
        condition: service_completed_successfully
        required: true
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network

  websocket:
    <<: *custom_image
    healthcheck:
      test:
        - CMD
        - wait-for-it
        - '0.0.0.0:9000'
      interval: 2s
      timeout: 10s
      retries: 30
    command:
      - node
      - /home/frappe/frappe-bench/apps/frappe/socketio.js
    depends_on:
      configurator:
        condition: service_completed_successfully
        required: true
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network

  configurator:
    <<: *custom_image
    deploy:
      mode: replicated
      replicas: ${CONFIGURE:-0}
      restart_policy:
        condition: none
    entrypoint: ["bash", "-c"]
    command:
      - >
        [[ $${REGENERATE_APPS_TXT} == "1" ]] && ls -1 apps > sites/apps.txt;
        [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".db_host // empty"` ]] && exit 0;
        bench set-config -g db_host $$DB_HOST;
        bench set-config -gp db_port $$DB_PORT;
        bench set-config -g redis_cache "redis://$$REDIS_CACHE";
        bench set-config -g redis_queue "redis://$$REDIS_QUEUE";
        bench set-config -g redis_socketio "redis://$$REDIS_QUEUE";
        bench set-config -gp socketio_port $$SOCKETIO_PORT;
    environment:
      DB_HOST: "${DB_HOST:-db}"
      DB_PORT: "3306"
      REDIS_CACHE: redis-cache:6379
      REDIS_QUEUE: redis-queue:6379
      SOCKETIO_PORT: "9000"
      REGENERATE_APPS_TXT: "${REGENERATE_APPS_TXT:-0}"
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network

  create-site:
    <<: *custom_image
    deploy:
      mode: replicated
      replicas: ${CREATE_SITE:-0}
      restart_policy:
        condition: none
    entrypoint: ["bash", "-c"]
    command:
      - >
        wait-for-it -t 120 $$DB_HOST:$$DB_PORT;
        wait-for-it -t 120 redis-cache:6379;
        wait-for-it -t 120 redis-queue:6379;
        export start=`date +%s`;
        until [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".db_host // empty"` ]] && \
          [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".redis_cache // empty"` ]] && \
          [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".redis_queue // empty"` ]];
        do
          echo "Waiting for sites/common_site_config.json to be created";
          sleep 5;
          if (( `date +%s`-start > 120 )); then
            echo "could not find sites/common_site_config.json with required keys";
            exit 1
          fi
        done;
        echo "sites/common_site_config.json found";
        [[ -d "sites/${SITE_NAME}" ]] && echo "${SITE_NAME} already exists" && exit 0;
        bench new-site --mariadb-user-host-login-scope='%' --admin-password=$${ADMIN_PASSWORD} --db-root-username=root --db-root-password=$${DB_ROOT_PASSWORD} $${INSTALL_APP_ARGS} $${SITE_NAME};
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    environment:
      SITE_NAME: ${SITE_NAME}
      ADMIN_PASSWORD: ${ADMIN_PASSWORD}
      DB_HOST: ${DB_HOST:-db}
      DB_PORT: "${DB_PORT:-3306}"
      DB_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      INSTALL_APP_ARGS: ${INSTALL_APP_ARGS}
    networks:
      - bench-network

  migration:
    <<: *custom_image
    deploy:
      mode: replicated
      replicas: ${MIGRATE:-0}
      restart_policy:
        condition: none
    entrypoint: ["bash", "-c"]
    command:
      - >
        curl -f http://${SITE_NAME}:8080/api/method/ping || echo "Site busy" && exit 0;
        bench --site all set-config -p maintenance_mode 1;
        bench --site all set-config -p pause_scheduler 1;
        bench --site all migrate;
        bench --site all set-config -p maintenance_mode 0;
        bench --site all set-config -p pause_scheduler 0;
    volumes:
      - sites:/home/frappe/frappe-bench/sites
    networks:
      - bench-network

  db:
    image: mariadb:10.6
    deploy:
      mode: replicated
      replicas: ${ENABLE_DB:-0}
      restart_policy:
        condition: always
    healthcheck:
      test: mysqladmin ping -h localhost --password=${DB_ROOT_PASSWORD}
      interval: 1s
      retries: 20
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_unicode_ci
      - --skip-character-set-client-handshake
      - --skip-innodb-read-only-compressed
    environment:
      - MYSQL_ROOT_PASSWORD=${DB_ROOT_PASSWORD}
      - MARIADB_ROOT_PASSWORD=${DB_ROOT_PASSWORD}
    volumes:
      - db-data:/var/lib/mysql
    networks:
      - bench-network

  redis-cache:
    deploy:
      restart_policy:
        condition: always
    image: redis:6.2-alpine
    networks:
      - bench-network
    healthcheck:
      test:
        - CMD
        - redis-cli
        - ping
      interval: 5s
      timeout: 5s
      retries: 3

  redis-queue:
    deploy:
      restart_policy:
        condition: always
    image: redis:6.2-alpine
    volumes:
      - redis-queue-data:/data
    networks:
      - bench-network
    healthcheck:
      test:
        - CMD
        - redis-cli
        - ping
      interval: 5s
      timeout: 5s
      retries: 3

  redis-socketio:
    deploy:
      restart_policy:
        condition: always
    image: redis:6.2-alpine
    volumes:
      - redis-socketio-data:/data
    networks:
      - bench-network
    healthcheck:
      test:
        - CMD
        - redis-cli
        - ping
      interval: 5s
      timeout: 5s
      retries: 3

volumes:
  db-data:
  redis-queue-data:
  redis-socketio-data:
  sites:
    driver_opts:
      type: "${SITE_VOLUME_TYPE}"
      o: "${SITE_VOLUME_OPTS}"
      device: "${SITE_VOLUME_DEV}"

networks:
  bench-network:

Things I’ve already tried:

1. bench build
2. bench restart
3. bench clear-cache
4. bench clear-website-cache
5. Verified file permissions belong to the frappe user
6. Cleared and disabled Redis cache
7. Removed Redis cache volume
8. Restarted the machine

None of these helped.

I’ve checked every related discussion on the forums, but I still can’t find the root cause. I need to understand why this keeps happening and how to apply a permanent fix. Thanks.

[kaksarkarr]

Anyone?

[revant_one]

Check this

Crashing frontend

Only option is to clear redis cache. Or run the specific command that will clear a key from redis. Easiest for devops is to have redis container not mount volume and delete container on upgrade. Orchestrator will recreate new with fresh volume. WARNING: Easy delete will work until some developer decides to start using cache as database and we cannot clear cache anymore. Then we’ll have to clear the specific key from cache using redis-cli. The “bench update” does everything in a vm where redi…

Further, the assets*.json and all the dist comes from image. If assets files are messed up even after clearing cache then delete all files from assets, clear cache and recreate containers.

[kaksarkarr]

I’ve done all the steps and cleared everything from the Redis cache, but as soon as I refresh the browser, the Redis cache gets repopulated with the wrong file references.

Here’s the current situation:

• Browser requests (404 errors):
  website.bundle.S3WQQPTV.css
erpnext-web.bundle.O7J2JAJ4.css

• In assets.json:
  website.bundle.2ZF634LL.css
erpnext-web.bundle.3PAVTJGE.css

• On disk at /frappe-bench/sites/assets/frappe/dist/css:
  Files match the ones in assets.json —
  website.bundle.2ZF634LL.css
erpnext-web.bundle.3PAVTJGE.css

• In Redis under assets_json:
  It still lists the outdated files —
  website.bundle.S3WQQPTV.css
erpnext-web.bundle.O7J2JAJ4.css

Even after clearing the Redis cache and deleting everything from the assets folder, then rebuilding with
bench rebuild --force --hard-link,
the cache gets refilled with the same incorrect entries right after a browser refresh.