New Data Import

clarkej · October 23, 2019, 11:42pm

What tests have you tried with your data set? First create an Export template, the video on post #1 refers to Export All Records - the screenshots suggest Variants and Pricing can be selected for export.

Let us know how that works?

Raymond_Liew · January 20, 2020, 9:05am

Encountered this error in v12.4.0:

{'method_name': 'start_import', 'log': <function log at 0x7f91494217b8>, 'retry': 0, 'is_async': True, 'user': 'Administrator', 'kwargs': {'data_import': 'DocType Import on 2020-01-20 16:57:00.025778'}, 'job_name': 'DocType Import on 2020-01-20 16:57:00.025778', 'event': 'data_import', 'method': <function start_import at 0x7f915826a6a8>, 'site': 'site1.local'}
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 99, in execute_job
    method(**kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import_beta/data_import_beta.py", line 68, in start_import
    return i.import_data()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 403, in import_data
    payloads = self.get_payloads_for_import()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 529, in get_payloads_for_import
    doc, rows, data = self.parse_next_row_for_import(data)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 709, in parse_next_row_for_import
    doc = parse_doc(doctype, docfields, values, row_number)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 625, in parse_doc
    doc[df.fieldname] = self.parse_value(value, df)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 340, in parse_value
    if df.fieldtype == "Check" and value.lower().strip() in ["t", "f", "true", "false"]:
AttributeError: 'int' object has no attribute 'lower'

.

{'method_name': 'start_import', 'log': <function log at 0x7f91494217b8>, 'retry': 0, 'is_async': True, 'user': 'Administrator', 'kwargs': {'data_import': 'DocType Import on 2020-01-20 16:42:33.764109'}, 'job_name': 'DocType Import on 2020-01-20 16:42:33.764109', 'event': 'data_import', 'method': <function start_import at 0x7f915826a6a8>, 'site': 'site1.local'}
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 99, in execute_job
    method(**kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import_beta/data_import_beta.py", line 68, in start_import
    return i.import_data()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 403, in import_data
    payloads = self.get_payloads_for_import()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 529, in get_payloads_for_import
    doc, rows, data = self.parse_next_row_for_import(data)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 709, in parse_next_row_for_import
    doc = parse_doc(doctype, docfields, values, row_number)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 625, in parse_doc
    doc[df.fieldname] = self.parse_value(value, df)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py", line 340, in parse_value
    if df.fieldtype == "Check" and value.lower().strip() in ["t", "f", "true", "false"]:
AttributeError: 'int' object has no attribute 'lower'

Muzzy · January 20, 2020, 2:07pm

Looks like you have added text in place of integer (number). Check your sheet.

flexy2ky · January 21, 2020, 9:47am

I have been getting the following error when i use the new data import tool:

Traceback (most recent call last):
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 461, in import_data
doc = self.process_doc(doc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 752, in process_doc
return self.insert_record(doc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 763, in insert_record
new_doc.insert()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py”, line 251, in insert
d.db_insert()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/model/base_document.py”, line 324, in db_insert
), list(d.values()))
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/database/database.py”, line 156, in sql
self._cursor.execute(query, values)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 170, in execute
result = self._query(query)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 328, in _query
conn.query(q)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 732, in _read_query_result
result.read()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 1075, in read
first_packet = self.connection._read_packet()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 684, in _read_packet
packet.check_error()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/protocol.py”, line 220, in check_error
err.raise_mysql_exception(self._data)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/err.py”, line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1292, “Incorrect datetime value: ‘05-01-2020 08:00:00’ for column _0fa56694735f44cb.tabTimesheet Detail.from_time at row 1”)

When i use the old data import tool with exact same data import is completed successfully.

flexy2ky · January 29, 2020, 7:29am

@netchampfaris any chance you could take a look at the error with the new data import tool? it keeps throwing a date format error:

Traceback (most recent call last):
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 461, in import_data
doc = self.process_doc(doc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 752, in process_doc
return self.insert_record(doc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 763, in insert_record
new_doc.insert()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py”, line 244, in insert
self.db_insert()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/model/base_document.py”, line 324, in db_insert
), list(d.values()))
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/database/database.py”, line 156, in sql
self._cursor.execute(query, values)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 170, in execute
result = self._query(query)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 328, in _query
conn.query(q)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 732, in _read_query_result
result.read()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 1075, in read
first_packet = self.connection._read_packet()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 684, in _read_packet
packet.check_error()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/protocol.py”, line 220, in check_error
err.raise_mysql_exception(self._data)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/err.py”, line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1292, “Incorrect date value: ‘28/02/2020’ for column _0fa56694735f44cb.tabAdditional Salary.payroll_date at row 1”)

Even when the date is entered as DD-MM-YYYY which is the standard format it either changes the format to DD/MM/YYY or still says the throws the same error on the standard format as seen below:

Traceback (most recent call last):
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 461, in import_data
doc = self.process_doc(doc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 752, in process_doc
return self.insert_record(doc)
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/core/doctype/data_import/importer_new.py”, line 763, in insert_record
new_doc.insert()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py”, line 244, in insert
self.db_insert()
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/model/base_document.py”, line 324, in db_insert
), list(d.values()))
File “/home/ubuntu/frappe-bench/apps/frappe/frappe/database/database.py”, line 156, in sql
self._cursor.execute(query, values)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 170, in execute
result = self._query(query)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 328, in _query
conn.query(q)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 732, in _read_query_result
result.read()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 1075, in read
first_packet = self.connection._read_packet()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 684, in _read_packet
packet.check_error()
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/protocol.py”, line 220, in check_error
err.raise_mysql_exception(self._data)
File “/home/ubuntu/frappe-bench/env/lib/python3.6/site-packages/pymysql/err.py”, line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1292, “Incorrect date value: ‘30-01-2020’ for column _0fa56694735f44cb.tabAdditional Salary.payroll_date at row 1”)

It’s a much simpler tool to use for data import but this date error is a stumbling block.

Smit_Vora · April 23, 2020, 5:37pm

Does it remember Mapping?

There should be check box to ask save mapping of existing excel sheet. So that those who import same sheet frequently and directy use output from other system more quickly.

szufisher · April 24, 2020, 3:29am

yes, this is badly needed feature

toceguera · May 14, 2020, 4:57am

Awesome update and a much improved template.

I have a basic UI feature request and a question about a related task.

I understand how the export feature is accomplished with the “Template Download, with Records” process. However, it would be nice to actually label it as an export for future users, because it isn’t very intuitive. Or maybe just an explanation of how to export in text somewhere on the page.
E.G If I type “Import” in the awesome bar, I can see the old and new options for importing , however if I type “Export” into the bar, I only see the old option in “Settings”
My question, does anyone on this chain have an idea of how to export an Exploded BOM? I can get a BOM list but not an individual BOM with all the related “Items” and their attributes listed. If you do know how to accomplish this, you will be the object of a great amount of goodwill from around the world!

Thanks All for your efforts!

Karl · July 2, 2020, 5:47am

This is a great tool for importing data, is there a way i can import link data?
For example, i was able to import my supplier list but also want to import the associated contacts and addresses associated with the suppliers.
I can import these lists but can’t get the links to the supplier to work.
thanks.

kennethsequeira · July 2, 2020, 10:38am

You will need to import the supplier, address list, and contact list separately.

Step 1: Import all suppliers

Step 2: When exporting the template for Contact and Address lists, be sure to download the Links section.

Step 3: In the import template, add the links of the document you want that contact/address to be connected to. On import the addresses and contacts imported will be linked with the supplier.

PS: it is always best to create one record manually, export that and follow the format for importing newer records.

Karl · July 2, 2020, 11:06am

thanks. my mistake was i didn’t have the empty column of “Link ID” in the address and contact lists, once I added this and left it empty the import worked.
thanks again.

fishter · July 9, 2020, 12:04pm

I’m importing some BOM. Annoyingly, the Item column is not automatically recognised. I have to set the mapping manually each time.

Item is in the first column of my csv file, it should be getting matched with “Item” in the BOM master.