the Custom app being referred to in such cases is a basic skeleton app provided which is to help you get started. ERPNext is also an app, but it has various doctypes and reports and pages made by the community. Now if you wish to add some new doctype or report, you add those to your app’s modules.
https://frappe.io/docs/user/en/tutorial/app
https://frappe.io/docs/user/en/tutorial/new-app
Actually it is BOTH.
Any custom changes you make to the default doctypes such as creating custom fields, adding custom scripts, creating new doctypes are supposed to saved in your custom app. This helps preserve your customizations. You can do so via Fixtures for the customizations. Custom Doctypes should be created with is_custom unchecked and Module should be the one you defined for your custom app.
Both add to the doctype behavior via the front-end. However, one thing to note. People who use ERPNext as a SaaS and do not have control on the machine, Custom Script is their best bet. For people running ERPNext on their own servers, and have admin access, can add more JS functionality by writing JS in their custom app and calling them for specified doctypes on specified doc_events as mentioned in their app’s hooks.py.
It is not recommended to change any of the default files that come from Github. Why ? Because those changes create conflicts on updates.