Kubernetes setup, error when submit sales invoice

we are working on a kubernetes setup for the last three months, it was working smoothly.

Today I receive the first error message when I try to submit a sales invoice, it through two error messages as below:

first message.

Message
Internal Server Error

second message.

Server Error
OSError: [Errno 30] Read-only file system: b'./erp.mydomain.com/public/files/QRCode-e535c.png'

the second message with (copy error to clipboard)
and return this log:

### App Versions

{
“erpnext”: “13.40.1”,
“frappe”: “13.41.6”,
“posawesome”: “3.4.0”,
“wiki”: “0.0.1”
}

### Route

Form/Sales Invoice/ACC-SINV-IBIAUE-2023-00008

### Trackeback

Traceback (most recent call last):
File “apps/frappe/frappe/app.py”, line 69, in application
response = frappe.api.handle()
File “apps/frappe/frappe/api.py”, line 55, in handle
return frappe.handler.handle()
File “apps/frappe/frappe/handler.py”, line 38, in handle
data = execute_cmd(cmd)
File “apps/frappe/frappe/handler.py”, line 76, in execute_cmd
return frappe.call(method, **frappe.form_dict)
File “apps/frappe/frappe/init.py”, line 1457, in call
return fn(*args, **newargs)
File “apps/frappe/frappe/desk/form/save.py”, line 22, in savedocs
doc.submit()
File “apps/frappe/frappe/model/document.py”, line 1018, in submit
return self._submit()
File “apps/frappe/frappe/model/document.py”, line 1007, in _submit
return self.save()
File “apps/frappe/frappe/model/document.py”, line 310, in save
return self._save(*args, **kwargs)
File “apps/frappe/frappe/model/document.py”, line 364, in _save
self.run_post_save_methods()
File “apps/frappe/frappe/model/document.py”, line 1088, in run_post_save_methods
self.run_method(“on_submit”)
File “apps/frappe/frappe/model/document.py”, line 941, in run_method
out = Document.hook(fn)(self, *args, **kwargs)
File “apps/frappe/frappe/model/document.py”, line 1261, in composer
return composed(self, method, *args, **kwargs)
File “apps/frappe/frappe/model/document.py”, line 1245, in runner
add_to_return_value(self, f(self, method, *args, **kwargs))
File “apps/erpnext/erpnext/regional/saudi_arabia/utils.py”, line 130, in create_qr_code
_file.save()
File “apps/frappe/frappe/model/document.py”, line 310, in save
return self._save(*args, **kwargs)
File “apps/frappe/frappe/model/document.py”, line 332, in _save
return self.insert()
File “apps/frappe/frappe/model/document.py”, line 254, in insert
self.run_method(“before_insert”)
File “apps/frappe/frappe/model/document.py”, line 941, in run_method
out = Document.hook(fn)(self, *args, **kwargs)
File “apps/frappe/frappe/model/document.py”, line 1261, in composer
return composed(self, method, *args, **kwargs)
File “apps/frappe/frappe/model/document.py”, line 1243, in runner
add_to_return_value(self, fn(self, *args, **kwargs))
File “apps/frappe/frappe/model/document.py”, line 938, in fn
return method_object(*args, **kwargs)
File “apps/frappe/frappe/core/doctype/file/file.py”, line 88, in before_insert
self.save_file(content=self.content, decode=self.decode)
File “apps/frappe/frappe/core/doctype/file/file.py”, line 523, in save_file
return self.save_file_on_filesystem()
File “apps/frappe/frappe/core/doctype/file/file.py”, line 526, in save_file_on_filesystem
fpath = self.write_file()
File “apps/frappe/frappe/core/doctype/file/file.py”, line 465, in write_file
with open(os.path.join(file_path.encode(“utf-8”), self.file_name.encode(“utf-8”)), “wb+”) as f:
OSError: [Errno 30] Read-only file system: b’./erp.mydomain.com/public/files/QRCode-e535c.png’

### Request Data

{
“type”: “POST”,
“args”: {
“doc”: “{"name":"ACC-SINV-IBIAUE-2023-00008","owner":"#############@gmail.com","creation":"2023-02-09 03:23:01.529200","modified":"2023-02-09 03:23:01.529200","modified_by":"##########@gmail.com","idx":0,"docstatus":0,"title":"الشركة السعودية الخليجية لحماية البيئة ","naming_series":"ACC-SINV-.abbr.-.YYYY.-","customer":"الشركة السعودية الخليجية لحماية البيئة","customer_name":"الشركة السعودية الخليجية لحماية البيئة ","customer_name_in_arabic":"الشركة السعودية الخليجية لحماية البيئة ","tax_id":"300178069200003","posa_is_printed":0,"is_pos":0,"is_consolidated":0,"is_return":0,"is_debit_note":0,"update_billed_amount_in_sales_order":0,"company":"Ibrahim Bander Ibrahim Aseeri Uniform Est.","abbr":"IBIAUE","company_tax_id":"##########","posting_date":"2023-02-09","posting_time":"3:23:1.695588","set_posting_time":0,"due_date":"2023-02-09","po_no":"","territory":"Saudi Arabia","shipping_address_name":"","company_trn":"311435271700003","currency":"SAR","conversion_rate":1,"selling_price_list":"البيع القياسية","price_list_currency":"SAR","plc_conversion_rate":1,"ignore_pricing_rule":0,"update_stock":0,"total_billing_amount":0,"total_billing_hours":0,"total_qty":18,"base_total":918,"base_net_total":918,"total_net_weight":0,"total":918,"net_total":918,"taxes_and_charges":"KSA VAT 15% - IBIAUE","posa_delivery_charges_rate":0,"tax_category":"","other_charges_calculation":"<div class=\"tax-break-up\" style=\"overflow-x: auto;\">\n\t<table class=\"table table-bordered table-hover\">\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t<th class=\"text-left\">Item\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t<th class=\"text-right\">Taxable Amount\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t<th class=\"text-right\">VAT 15% @ 15.0\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\tيونيفورم عمال\n\t\t\t\t\t<td class=\"text-right\">\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tر.س 918.00\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t<td class=\"text-right\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t(15.0%)\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tر.س 137.70\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n","base_total_taxes_and_charges":137.7,"total_taxes_and_charges":137.7,"loyalty_points":0,"loyalty_amount":0,"redeem_loyalty_points":0,"apply_discount_on":"Grand Total","is_cash_or_non_trade_discount":0,"base_discount_amount":0,"additional_discount_percentage":0,"discount_amount":0,"base_grand_total":1055.7,"base_rounding_adjustment":0.3,"base_rounded_total":1056,"base_in_words":"","grand_total":1055.7,"rounding_adjustment":0.3,"rounded_total":1056,"in_words":"","total_advance":0,"outstanding_amount":1056,"disable_rounded_total":0,"write_off_amount":0,"base_write_off_amount":0,"write_off_outstanding_amount_automatically":0,"allocate_advances_automatically":0,"ignore_default_payment_terms_template":0,"base_paid_amount":0,"paid_amount":0,"base_change_amount":0,"change_amount":0,"letter_head":"1","group_same_items":0,"vat_emirate":"","tourist_tax_return":0,"language":"en","status":"Draft","customer_group":"تجاري","is_internal_customer":0,"is_discounted":0,"debit_to":"1310 - Debtors - IBIAUE","party_account_currency":"SAR","is_opening":"No","c_form_applicable":"No","remarks":"No Remarks","amount_eligible_for_commission":918,"commission_rate":0,"total_commission":0,"against_income_account":"4110 - Sales - IBIAUE","doctype":"Sales Invoice","items":[{"name":"f874df58e5","owner":"##########@gmail.com","creation":"2023-02-09 03:23:01.529200","modified":"2023-02-09 03:23:01.529200","modified_by":"##########@gmail.com","parent":"ACC-SINV-IBIAUE-2023-00008","parentfield":"items","parenttype":"Sales Invoice","idx":1,"docstatus":0,"item_code":"يونيفورم عمال","item_name":"يونيفورم عمال","description":"يونيفورم عمال","tax_rate":15,"tax_amount":137.7,"total_amount":1055.7,"is_zero_rated":0,"is_exempt":0,"item_group":"المنتجات","image":"","qty":18,"stock_uom":"Nos","uom":"Nos","conversion_factor":1,"stock_qty":18,"price_list_rate":51,"base_price_list_rate":51,"margin_type":"","margin_rate_or_amount":0,"rate_with_margin":0,"discount_percentage":0,"discount_amount":0,"base_rate_with_margin":0,"rate":51,"amount":918,"base_rate":51,"base_amount":918,"posa_offer_applied":0,"posa_is_offer":0,"stock_uom_rate":51,"is_free_item":0,"grant_commission":1,"net_rate":51,"net_amount":918,"base_net_rate":51,"base_net_amount":918,"delivered_by_supplier":0,"income_account":"4110 - Sales - IBIAUE","is_fixed_asset":0,"expense_account":"5111 - Cost of Goods Sold - IBIAUE","enable_deferred_revenue":0,"weight_per_unit":0,"total_weight":0,"warehouse":"All Warehouses - IBIAUE","incoming_rate":120,"allow_zero_valuation_rate":0,"item_tax_rate":"{}","actual_batch_qty":0,"actual_qty":0,"delivered_qty":0,"cost_center":"Main - IBIAUE","page_break":0,"doctype":"Sales Invoice Item"}],"pricing_rules":[],"posa_offers":[],"posa_coupons":[],"packed_items":[],"timesheets":[],"taxes":[{"name":"e74d532ac5","owner":"##########@gmail.com","creation":"2023-02-09 03:23:01.529200","modified":"2023-02-09 03:23:01.529200","modified_by":"###########@gmail.com","parent":"ACC-SINV-IBIAUE-2023-00008","parentfield":"taxes","parenttype":"Sales Invoice","idx":1,"docstatus":0,"charge_type":"On Net Total","account_head":"VAT 15% - IBIAUE","description":"VAT 15% @ 15.0","included_in_print_rate":0,"included_in_paid_amount":0,"cost_center":"Main - IBIAUE","rate":15,"tax_amount":137.7,"total":1055.7,"tax_amount_after_discount_amount":137.7,"base_tax_amount":137.7,"base_total":1055.7,"base_tax_amount_after_discount_amount":137.7,"item_wise_tax_detail":"{\"يونيفورم عمال\":[15,137.7]}","dont_recompute_tax":0,"doctype":"Sales Taxes and Charges"}],"advances":[],"payment_schedule":[{"name":"bb39adf620","creation":"2023-02-09 03:23:01.731624","modified":"2023-02-09 03:23:01.731624","modified_by":"##########@gmail.com","parent":"ACC-SINV-IBIAUE-2023-00008","parentfield":"payment_schedule","parenttype":"Sales Invoice","idx":1,"docstatus":0,"due_date":"2023-02-09","invoice_portion":100,"discount":0,"payment_amount":1056,"outstanding":1056,"paid_amount":0,"discounted_amount":0,"base_payment_amount":1056,"doctype":"Payment Schedule"}],"payments":[],"sales_team":[],"__onload":{"make_payment_via_journal_entry":0},"__last_sync_on":"2023-02-09T02:17:20.002Z"}”,
“action”: “Submit”
},
“btn”: {
“jQuery2240240066131694238741”: {
“events”: {
“click”: [
{
“type”: “click”,
“origType”: “click”,
“guid”: 361,
“namespace”: “”
}
]
}
}
},
“freeze”: true,
“headers”: {},
“error_handlers”: {},
“url”: “/api/method/frappe.desk.form.save.savedocs”
}

### Response Data

{
“exception”: “OSError: [Errno 30] Read-only file system: b’./erp.mydomain.com/public/files/QRCode-e535c.png’”
}

it is challenging to troubleshoot erpnext inside kubernetes.

any help in this regards is appreciated,

know we can’t issue any sales invoice all invoices saved as drafts, and when try to submit invoices it through an error message,

any help is appreciated.

@revant_one my Kubernetes guru I would appreciate any help to solve this problem.

check permission for sites volume.

what version of helm chart are you using?

in latest chart there is a job that can fix volume permission that you can enable and run.

Hi Revant,

Thanks for your reply,
we are using rancher as our Kubenetes provider:

Provider: RKE1
Kubernetes Version: v1.21.13
Created: 227 days ago

as for the volume permission we are using the in-cluster nfs :

sgt-erpnext	        sgt	Bound	pvc-7e137a40-3d73-485b-a6a6-e09d9d47ca7a	8Gi	RWX	nfs	Filesystem

sgt-erpnext-logs	sgt	Bound	pvc-91b8779e-8d67-4fdb-8405-099699f487db	8Gi	RWX	nfs	Filesystem

as for the helm chart :

labels:
    app.kubernetes.io/instance: sgt
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: erpnext
    app.kubernetes.io/version: v13.34.1
    helm.sh/chart: erpnext-4.0.32

and yes we have the job that fix the volume permission in the chart

jobs:
  volumePermissions:
    enabled: true
    backoffLimit: 0
    resources: {}
    nodeSelector: {}
    tolerations: []
    affinity: {}

ksnip_20230209-142053

thanks for your time and the reply.

I appreciate all the help in this regards

Hi @revant_one

the issue is not solved, and would appreciate any help in this regards.

I can’t help you want further.

share access if you can.