Error During Buk Import. Bug? Wrong Template?

ebsjbulcher · January 19, 2021, 5:29pm

I have 60K+ records to import, so using the new importer is not much of an option for me. I thought I’d just try uploading a csv file to the server and using bench import-csv or bench data-import to upload the records instead.

Here is the command I am using:
bench --site site1.local import-csv ../Contacts.csv

And here is the error I am getting:
frappe.exceptions.ValidationError: Please do not change the rows above Start entering data below this line

To be clear, I am simply uploading a CSV file with the headers I get from the Data Import tool in ERPNext. There is no line that says “Please do not change the rows…”; the file starts with the headers rows. I keep seeing references to more complicated CSV files for importing data around the forums and in videos, but I have no idea where they come from or even if I should be using them.

I’m on 12.16.2.

Any help would be appreciated! Thanks.

kennethsequeira · January 19, 2021, 6:32pm

Seems to be a mismatch with the downloaded template and the data import being executed.

Can you share a screenshot of the template which is being used?

ebsjbulcher · January 19, 2021, 7:00pm

There’s nothing to screenshot really - as I said, it’s jut a row of headers I got from the data import tool. That’s all I get from the GUI - that single row of headers, which is why I’m stumped.

I can use the template to upload a limited number of rows from my data via ERPNext’s import GUI. The full data set doesn’t work with the import-csv command. I haven’t tried it with a limited data set, but the error doesn’t give me any reason to think it would work differently.

f_deryckel · January 20, 2021, 7:27am

Why don’t you use the import template while doing your bulk import?

ebsjbulcher · January 20, 2021, 4:49pm

That is the only template I can find in the GUI. Hence my question in the title - “Wrong Template?” Is there another template hidden somewhere?

kennethsequeira · January 20, 2021, 6:12pm

There’s a Data Import Legacy in ERPNext as well, an import tool with a slightly different template.

From the code for version 12 I believe it’s still referencing the legacy tool when running bench import-csv.

github.com

frappe/frappe/blob/81df6fc92a4004df327d4c90a787c2a9ee565048/frappe/commands/utils.py#L302


      
          	for site in context.sites:
          		try:
          			frappe.init(site=site)
          			frappe.connect()
          			import_doc(path, overwrite=context.force)
          		finally:
          			frappe.destroy()
          	if not context.sites:
          		raise SiteNotSpecifiedError
          
          
@click.command('import-csv')
          @click.argument('path')
          @click.option('--only-insert', default=False, is_flag=True, help='Do not overwrite existing records')
          @click.option('--submit-after-import', default=False, is_flag=True, help='Submit document after importing it')
          @click.option('--ignore-encoding-errors', default=False, is_flag=True, help='Ignore encoding errors while coverting to unicode')
          @click.option('--no-email', default=True, is_flag=True, help='Send email if applicable')
          @pass_context
          def import_csv(context, path, only_insert=False, submit_after_import=False, ignore_encoding_errors=False, no_email=True):
          	"Import CSV using data import"
          	from frappe.core.doctype.data_import_legacy import importer
          	from frappe.utils.csvutils import read_csv_content

Here’s the error message you’re getting:

github.com

frappe/frappe/blob/57cc556de61c52f8d0600aeaae657bdf1ded8fbe/frappe/core/doctype/data_import_legacy/importer.py#L76


      
          			from_data_import = params.get('from_data_import')
          		if not params.get('skip_errors'):
          			skip_errors = params.get('skip_errors')
          
          
	frappe.flags.in_import = True
          	frappe.flags.mute_emails = no_email
          
          
	def get_data_keys_definition():
          		return get_data_keys()
          
          
	def bad_template():
          		frappe.throw(_("Please do not change the rows above {0}").format(get_data_keys_definition().data_separator))
          
          
	def check_data_length():
          		if not data:
          			frappe.throw(_("No data found in the file. Please reattach the new file with data."))
          
          
	def get_start_row():
          		for i, row in enumerate(rows):
          			if row and row[0]==get_data_keys_definition().data_separator:
          				return i+1

ebsjbulcher · January 20, 2021, 6:19pm

It sounds like bulk import is quasi-broken in 12 if there’s no where to get the template files from the interface. What is the/is there a recommended way to proceed?

kennethsequeira · January 20, 2021, 6:21pm

I think you can still get the older template in version 12. Search for Data Import Legacy in the awesombar (Ctrl/Cmd + G) and select the doctype. You’ll get the option to export the template.

Copy over the data from your current file to this newer one and run the import again and check.

ebsjbulcher · January 20, 2021, 6:23pm

Ah, got it. Thanks Kenneth! That’s one more hurdle down!

gunic0rn · May 14, 2022, 11:03pm

I am on v13. What has worked for me is as follows:

Replace the import-csv command with the data-import command, as shown below:

bench --site <SITE_NAME> data-import --file <PATH_TO_CSV> --doctype <DOCTYPE> --type <Insert|Update>

The import-csv command expects a Template that adheres to the deprecated Data Import Legacy format. The data-import command expects the current Data Import template format.

You can find reference to the data-import command here:

https://github.com/frappe/frappe/blob/6443ae571d858918567f5ec518e06765cea62c53/frappe/commands/utils.py#L461