Bench migrate failed - Alter table `tabPayment Entry`

I have docker setup with a custom image with the following applications installed:
builder
crm
erpnext
expense_request
frappe
helpdesk
hrms
payments
wiki

I just updated my customer image to erpnext version 15.65.1 however, i am getting a migration error when trying to ALTER TABLE tabPayment Entry.

full stack trace:

Failed to alter schema using query: ALTER TABLE `tabPayment Entry` MODIFY `received_amount_after_tax` decimal(21,9) not null default 0, MODIFY `total_allocated_amount` decimal(21,9) not null default 0, MODIFY `base_received_amount` decimal(21,9) not null default 0, MODIFY `base_paid_amount_after_tax` decimal(21,9) not null default 0, MODIFY `reference_date` date default 'posting_date', MODIFY `unallocated_amount` decimal(21,9) not null default 0, MODIFY `base_paid_amount` decimal(21,9) not null default 0, MODIFY `paid_to_account_balance` decimal(21,9) not null default 0, MODIFY `total_taxes_and_charges` decimal(21,9) not null default 0, MODIFY `party_balance` decimal(21,9) not null default 0, MODIFY `paid_amount` decimal(21,9) not null default 0, MODIFY `base_received_amount_after_tax` decimal(21,9) not null default 0, MODIFY `difference_amount` decimal(21,9) not null default 0, MODIFY `received_amount` decimal(21,9) not null default 0, MODIFY `source_exchange_rate` decimal(21,9) not null default 0, MODIFY `paid_amount_after_tax` decimal(21,9) not null default 0, MODIFY `paid_from_account_balance` decimal(21,9) not null default 0, MODIFY `target_exchange_rate` decimal(21,9) not null default 0, MODIFY `base_total_allocated_amount` decimal(21,9) not null default 0, MODIFY `base_total_taxes_and_charges` decimal(21,9) not null default 0


There was an issue while migrating the DocType: Payment Entry

Queued rebuilding of search index for erpfin.simplementix.com

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 114, in <module>
    main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 20, in main
    click.Group(commands=commands)(prog_name="bench")
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 29, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py", line 684, in migrate
    ).run(site=site)
      ^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 188, in run
    self.run_schema_updates()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 52, in wrapper
    raise e
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 44, in wrapper
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 120, in run_schema_updates
    frappe.model.sync.sync_all()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/sync.py", line 43, in sync_all
    sync_for(app, force, reset_permissions=reset_permissions)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/sync.py", line 111, in sync_for
    import_file_by_path(
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py", line 146, in import_file_by_path
    import_doc(
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py", line 239, in import_doc
    doc.insert()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 334, in insert
    self.run_post_save_methods()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1173, in run_post_save_methods
    self.run_method("on_update")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1007, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1367, in composer
    return composed(self, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1349, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1004, in fn
    return method_object(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/doctype/doctype.py", line 514, in on_update
    raise e
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/doctype/doctype.py", line 511, in on_update
    frappe.db.updatedb(self.name, Meta(self))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/mariadb/database.py", line 463, in updatedb
    db_table.sync()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/schema.py", line 46, in sync
    self.alter()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/mariadb/schema.py", line 111, in alter
    frappe.db.sql_ddl(query)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 413, in sql_ddl
    self.sql(query, debug=debug)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 230, in sql
    self._cursor.execute(query, values)
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute
    result = self._query(query)
             ^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/connections.py", line 563, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/connections.py", line 825, in _read_query_result
    result.read()
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/connections.py", line 1199, in read
    first_packet = self.connection._read_packet()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/connections.py", line 775, in _read_packet
    packet.raise_for_error()
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/protocol.py", line 219, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/frappe/frappe-bench/env/lib/python3.11/site-packages/pymysql/err.py", line 150, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1067, "Invalid default value for 'reference_date'")

Any and all help is appreciated. Please let me know if I can provide additional information.

After looking at the stack trace, I noticed the last line "Invalid default value for 'reference_date'". How can i update this now that I cant access my site? getting Status: 503 error when I open the site.

Is there a way to update this manually via CLI?

I turned off maintenance mode and then updated the payment entry field that i updated the default value on and that fixed the migration issue.