Kubernetes Helm on GKE

nanacnote · March 23, 2021, 7:54am

Can someone who has successfully deployed erpnext on GKE using the helm charts shed some more light on the extra steps they had to do?

I have provisioned an nfs
I have correctly pointed to a mariadb instance
I have set up an nginx controller
I have setup up an ingress resource for nginx

However when I examine the erpnext pod it says waiting for connection on 0.0.0.0:8000

And all the worker pods are crashlooping.

This happens for erpnext helm chart version 2.1.5

However when I try version 2.1.2 I end up with the sorry this is a problem on our side screen.

Thank you so much.

revant_one · March 23, 2021, 9:35am

Recently volume for logs was added. Did you provide pvc/storageclass for logs?

Can you share logs of failing worker pods?

the erpnext pod has 2 containers. Check log for erpnext-python container

Also provide output for kubectl describe pod ...

After frappe 12.16.2 got released yesterday helm chart was bumped to 2.1.6 with frappe version changed. Nothing from templates changed.

nanacnote · March 23, 2021, 11:08am

thanks for getting back to me so quickly so i have added a link to the screen shots below since i was not allowed to attached them directly here. Many thanks once again.

https://pdfhost.io/v/zVOW6vejZ_Screenshot_20210323_at_094306convertedpdf.pdf[Screenshots](https://pdfhost.io/v/zVOW6vejZ_Screenshot_20210323_at_094306convertedpdf.pdf)

revant_one · March 23, 2021, 11:13am

Redis used by frappe has no authentication enabled.

If you’re installing separate redis chart then set it to allow connection without auth or just use the deployments that are part of frappe/erpnext helm chart.

nanacnote · March 23, 2021, 11:35am

Thanks but I can confirm I am not using any external redis … I am using the helm chart from frappe/erpnext where how can I go about making this changes

revant_one · March 23, 2021, 11:41am

what is redis.extraEnv[] used for in values, erpnext.yml?

The error shows NOAUTH Authentication required. That means redis needs authentication.

Frappe does not have any config to pass auth to redis.

nanacnote · March 23, 2021, 12:02pm

i have replaced this with

redis:
  extraEnv:
    # - name: REDIS_PASSWORD
    #   value: "super_secret_password"
    - name: ALLOW_EMPTY_PASSWORD
      value: "yes"

and it is all loaded now so from the docs it says ALLOW_EMPTY_PASSWORD is set by default but this does not seem to be the case it needs to be explicitly added in custom values.

However i am now met with the sorry we will be back soon screen i believe this has got to do with site job or something. Can you briefly give me a high level overview of the architecture if you can. Sorry for being so slow but i have been trying to read the code itself but its a lot and i am on a deadline. Thank you for your support today much appreciated.

revant_one · March 23, 2021, 1:28pm

what is the log for erpnext-python container from erpnext-erpnext pod? The pod with 2 containers.

Have you created the create-site job? https://helm.erpnext.com/kubernetes-resources/create-new-site-job

It creates the first site, name the site what the ingress host is. example if ingress host is erp.mysite.com create site called erp.mysite.com. it should be FQDN.

Architecture:

templates install bench like env.

3 workers
3 redis
1 socketio
1 scheduler
1 nginx reverse proxy with static assets + 1 gunicorn worker as 1 pod.

Installing helm chart on it’s own doesn’t create the first site.

First site is created manually by creating the create-site job.

nanacnote · March 23, 2021, 2:58pm

thanks for the reply but i have done this but the pod is still pending i think i do not understand how the whole frappe/erpnext system works. if you have time can you explain what creating a site job means what container images to use for the site job what volume claims to use. There is not much in the link you provided i think this will help others who come after me and if i am able to get my setup working i am planning to contribute by recording a video on how to deploy erpnext to google k8s.

nanacnote · March 23, 2021, 3:07pm

apiVersion: batch/v1
kind: Job
metadata:
  name: test-site
spec:
  backoffLimit: 1
  template:
    spec:
      securityContext:
        supplementalGroups: [1000]
      containers:
        - name: create-site
          image: frappe/erpnext-worker:latest
          args: ["new"]
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: sites-dir
              mountPath: /home/frappe/frappe-bench/sites
          env:
            - name: "SITE_NAME"
              value: "erpnext.hiramlabs.com"
            - name: "DB_ROOT_USER"
              value: root
            - name: "MYSQL_ROOT_PASSWORD"
              valueFrom:
                secretKeyRef:
                  key: password
                  name: mariadb-root-password
            - name: "ADMIN_PASSWORD"
              value: "p4ssw0rd"
            - name: "INSTALL_APPS"
              value: "erpnext"
      restartPolicy: Never
      volumes:
        - name: sites-dir
          persistentVolumeClaim:
            claimName: erpnext
            readOnly: false

ERROR
2021-03-23T14:53:58.399124814Z

Traceback (most recent call last): File "/home/frappe/frappe-bench/commands/new.py", line 118, in <module> main() File "/home/frappe/frappe-bench/commands/new.py", line 70, in main db_port=db_port, File "/home/frappe/frappe-bench/apps/frappe/frappe/installer.py", line 72, in _new_site no_mariadb_socket=no_mariadb_socket, File "/home/frappe/frappe-bench/apps/frappe/frappe/installer.py", line 101, in install_db make_conf(db_name, site_config=site_config, db_password=db_password, db_type=db_type, db_host=db_host, db_port=db_port) File "/home/frappe/frappe-bench/apps/frappe/frappe/installer.py", line 321, in make_conf frappe.init(site, sites_path=sites_path) File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 199, in init setup_module_map() File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1091, in setup_module_map _cache.set_value("module_app", local.module_app) File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 48, in set_value self.set(key, pickle.dumps(val)) File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/client.py", line 1801, in set return self.execute_command('SET', *pieces) File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/client.py", line 901, in execute_command return self.parse_response(conn, command_name, **options) File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/client.py", line 915, in parse_response response = connection.read_response() File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/connection.py", line 756, in read_response raise response redis.exceptions.ResponseError: DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.

revant_one · March 24, 2021, 12:37am

Something wrong with redis setup or configuration.

revant_one · March 24, 2021, 2:10am

https://github.com/frappe/helm/issues/72

revant_one · March 24, 2021, 2:49am

try the helm chart version 2.1.7

nanacnote · March 24, 2021, 3:13am

Thanks a lot i have upgraded to that and it seems to work i have got a different problem which is unrelated to erpnext chart there is some inconsistence between my mariadb auth variables and the one used in the worker manifest. i am reconciling this and i will report back asap with status.