New Data Import

Hi Community,

We recently merged New Data Import into the latest develop branch of frappe.

The goal of the new Data Import was to move away from the rigid template we had before and move to a simpler template format which is more forgiving. It is also supposed to give better warnings to the user so that the task of importing data is simpler than before.

The new template looks something like this:
image

Features:

  • Ability to map columns that are not identified automatically manually.
  • Link field dependencies are automatically created if there is a single mandatory field.
    • For e.g Item Group while importing Item
  • If the autoname is based on a field, export only one ID column
    • For e.g ID and Item Code are the same in Item, so they are merged into one column ID (Item Code) so that the user is not confused
  • Ignore empty rows and columns in the template
  • Supported file types: .csv, .xlsx and .xls
  • Parse 18 different types of date formats automatically
  • Show a preview of the first 10 rows from the template
  • Export errored rows into a separate CSV
  • Child records are identified if the corresponding parent value cells are blank or have the same values as the previous row
  • Show Data Import Link in timeline if the records were updated using Data Import.
  • Show estimated time remaining when importing large number of records.
  • New command data-import
    • bench --site sitename data-import --doctype Item --file /path/to/file.csv

Download Template

new-data-import-exporter

Import Data

This is still in Beta and it will live alongside the old Data Import until it is tested thoroughly.

To test the new Data Import:

  1. Make sure you are on the latest develop of frappe
  2. Type Data Import Beta in the awesomebar
  3. Create a new Data Import and test it

Please provide feedback and report any bugs here.

30 Likes

Very nice improvement. just one question, do we have condition based export? like date range etc…

if yes that will be great. in some cases, the data is huge and exporting all data is not possible. condition based export will be great. even if we can add this export through bench command’s will be fine.

1 Like

I would like to suggest supporting upload one single excel files with multiple sheets each map to one doctype. the import program handles sequence of import accordingly.

3 Likes

This is already covered under current Data Export where you can export via filters:

Can’t comment if data export is being re-written as well.

2 Likes

Export is possible currently:

Make sure you have enabled the option via role permission manager:

1 Like

Indeed you are right. Enabling export in role permissions did the trick. Thanks for the tip. This solves @adnan and my problem esentially.

Thanks @kennethsequeira, i didnt spend much time on v12. still struggling with v10.
Yes it’s a very nice feature.

Regards,
Adnan

The export feature was added in version 11 and I genuinely think many are still unaware about this :sweat_smile:

2 Likes

Is there a way to export attachments on some documents?

its difficult to update item master in v11 even not able to export all data through erp
kindly do something in this

@Harshal_nirmal You can export Data in v11, through Data export tool , it even has filters

@Anthonys if i wanted to update all item i cannot do through the erp system i have number of item group i cannot filter each and every group export it in excel

@Harshal_nirmal What do you mean by update all items ?? update what exactly??
If you want to update too many fields for example
just export all of them from data export , and change what you want to change then from data import tool import back the excel file while selecting update records.

To export via excel for a particular item group, go data export and select item doctype.
In filters select what filters you need you can add multiple filters and export the file.

you selected only one item group , we have lots of item group in item master so if i want to update all group of item master i cannot export it its show 504 error

just 1 take example if i wanted to change default warehouse multiple item group of item . so first i need to filter item group second export ok then update that group of item , here not 1 item group there is no of item group master its not possible to export and update according this case
here we cannot update item in one stoke its unnecessary create duplication of work.

in this earlier v10 we can easily update item master but v11 its not good system friendly medium scale

Can this tool import more number of records as compared to the past ?

Very nice improvement.
one question,
I want to import Item with Variant and Pricing as I attached in the screenshot.

is it any way to add this.

Number of records is not limited in anyway. The task is run in a background job.

1 Like