Upload 1000000 customer records using csv

1
Hi All,

We have make a functionality in which, when we upload a csv of customer records from data import tool it create customer and sales order respectively. Now our requirement is to upload a csv which has 1000000 (Around Ten Lakh) customer records. It is not possible through data import tool, is there any other way so i can create both customer and sales order respctively for 1000000 records.

Regards
Rohit

–

Note:

 

If you are posting an issue,

  1. We should be able to replicate it at our end. So please give us as much information as you can. Please see it from the point of view of the person receiving the communication.
  2. Paste your code at http://pastebin.com or http://gist.github.com and send only the URL via email
  3. For sending images, use http://imgur.com or other similar services. Do not send images as attachments. Links are good. Same goes for any file you are going to send.

     

    End of Note

    —

    You received this message because you are subscribed to the Google Groups "ERPNext Developer Forum" group.

    To unsubscribe from this group and stop receiving emails from it, send an email to erpnext-developer-forum+un…@googlegroups.com.

    To view this discussion on the web visit https://groups.google.com/d/msgid/erpnext-developer-forum/e8aa458c-313e-4f53-a7f1-3b5f881d90be%40googlegroups.com.

    For more options, visit https://groups.google.com/d/optout.
2
Hi Rohit,

The limitation of 5000 records was put into place so that a lot of uncommitted changes do not pile up in mysql and cause it to crash.

To overcome this, you should break the million records into batches of 5000 and try to insert them batch wise. In ver 4, you could use celery to create queued tasks.

Thanks,
Anand.


On Mon, Mar 10, 2014 at 9:42 PM, Rohit Waghchaure <ro...@indictranstech.com> wrote:
Hi All,

We have make a functionality in which, when we upload a csv of customer records from data import tool it create customer and sales order respectively. Now our requirement is to upload a csv which has 1000000 (Around Ten Lakh) customer records. It is not possible through data import tool, is there any other way so i can create both customer and sales order respctively for 1000000 records.

Regards
Rohit

–

Note:



If you are posting an issue,

  1. We should be able to replicate it at our end. So please give us as much information as you can. Please see it from the point of view of the person receiving the communication.
  2. Paste your code at http://pastebin.com or http://gist.github.com and send only the URL via email
  3. For sending images, use http://imgur.com or other similar services. Do not send images as attachments. Links are good. Same goes for any file you are going to send.



    End of Note

    —

    You received this message because you are subscribed to the Google Groups "ERPNext Developer Forum" group.

    To unsubscribe from this group and stop receiving emails from it, send an email to erpnext-developer-forum+un…@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/erpnext-developer-forum/e8aa458c-313e-4f53-a7f1-3b5f881d90be%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


–

Note:

 

If you are posting an issue,

  1. We should be able to replicate it at our end. So please give us as much information as you can. Please see it from the point of view of the person receiving the communication.
  2. Paste your code at http://pastebin.com or http://gist.github.com and send only the URL via email
  3. For sending images, use http://imgur.com or other similar services. Do not send images as attachments. Links are good. Same goes for any file you are going to send.

     

    End of Note

    —

    You received this message because you are subscribed to the Google Groups "ERPNext Developer Forum" group.

    To unsubscribe from this group and stop receiving emails from it, send an email to erpnext-developer-forum+un…@googlegroups.com.

    To view this discussion on the web visit https://groups.google.com/d/msgid/erpnext-developer-forum/CAAQzbF32CcGfPo93y_%3Dp4g9%2BDJQVta6saYmCtUL2XcpYM4JutA%40mail.gmail.com.

    For more options, visit https://groups.google.com/d/optout.