Anand,
after checkouting your changes I have another error when submitting Stock Reco:
Traceback
(innermost last):
File “lib/py/webnotes/widgets/form/save.py”, line 40, in savedocs
getattr(doclist, action.lower())()
File “lib/py/webnotes/model/doclist.py”, line 241, in save
self.run_method(‘validate’)
File “lib/py/webnotes/model/doclist.py”, line 197, in run_method
getattr(self.obj, method)()
File
“erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”,
line 102, in validate
self.get_reconciliation_data()
File
“erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”,
line 74, in get_reconciliation_data
self.convert_into_list(data)
File
“erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”,
line 59, in convert_into_list
self.validate_item(s[0], count)
File
“erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”,
line 79, in validate_item
det = sql(“select item_code, has_serial_no from tabItem where name
= ‘%s’”% cstr(item), as_dict = 1)
File “lib/py/webnotes/db.py”, line 169, in sql
raise e
ProgrammingError: (1064, “You have an error in your SQL syntax; check
the manual that corresponds to your MySQL server version for the right
syntax to use near ‘Item Code’‘’ at line 1”)
Traceback (innermost last):
File “lib/py/webnotes/handler.py”, line 224, in handle
execute_cmd(cmd)
File “lib/py/webnotes/handler.py”, line 258, in execute_cmd
ret = method()
File “lib/py/webnotes/widgets/form/save.py”, line 54, in savedocs
raise e
ProgrammingError: (1064, “You have an error in your SQL syntax; check
the manual that corresponds to your MySQL server version for the right
syntax to use near ‘Item Code’‘’ at line 1”)
On Monday, March 26, 2012 9:16:23 AM UTC+4, Anand Doshi wrote:
Hi everyone,As Rushabh said, the current code is incorrect:
self.data.append([d.encode(“ascii”) for d in s])
We just need to do this:
self.data.append(s)
encode causes unicode string to be converted into str string. i.e. byte string.
decode causes str sting to be converted into unicode string.
No need to encode a ‘str’ type, as it is already encoded. It will
throw UnicodeDecodeError if it finds that ‘str’ is already encoded.
Now, while writing the file: since while importing, the file is
already read as utf-8 encoded, i.e. as a byte string, doing this:
file_manager.write_file(fname[1], out.encode(‘utf-8’))
will cause a UnicodeDecodeError, since the out variable is of ‘str’ type.
If it is already encoded, and if we try to encode it in another
encoding, it casues UnicodeDecodeError. Hence,
file_manager.write_file(fname[1], out)
suffices.
@Anton:
The file with the single record, which you sent, was not encoded as utf-8.
It was showing a “?001” instead of “П001”
I replaced the item code with “П001” and saved it as a utf-8 encoded csv file.
I added this item in my test database. And then did the stock
reconciliation. It worked perfectly… I am attaching the output file
with this email.
In summary:
So the correct code is:
In stock_reconciliation.py in def convert_into_list(self, data): line 62
use
self.data.append(s)
instead of
self.data.append([d.encode(“ascii”) for d in s])
in file_manager.write_file(fname[1], out): line 188
use as it is
file_manager.write_file(fname[1], out)
In warehouse.py in get_bin: line 42:
use
bin = sql(“select name from tabBin where item_code = %s and <br> warehouse = %s”, (item_code, self.doc.name))
instead of
bin = sql(“select name from tabBin where item_code = ‘%s’ and <br> warehouse = ‘%s’” % (item_code, self.doc.name))
This solved the whole issue, when the csv file is encoded in ‘utf-8’
for non english languages. I tested this with the csv provided by
Anton and it worked…
I will push this release in master branch and you can try this out.
Thanks,
Anand.
On Mon, Mar 26, 2012 at 10:00 AM, Pratik Vyas <pd…@gmail.com> wrote:
>
>
> On 26-Mar-2012, at 9:41 AM, Rushabh Mehta wrote:
>
> > Great job guys :)…
> >
> > For my understanding - did you try to remove both the “decode” and “encode” to see if it worked?
> >
> That depends on what you plan to do between reading and writing. 
> To make sense of out of you the utf-8 you read, you’ll have to decode it otherwise what you see is raw codepoints.
>
> So this is what you do,
> * Read a utf-8 encoded file., eg ‘\xd0\x9f001\n’
> * Decode it to Unicode. So you have u’\u041f001\n’
> * Do stuff with it.
> * To write it back as utf-8, you need to encode it as utf-8
>
>
> > best,
> > Rushabh
> >
> > On Mon, Mar 26, 2012 at 12:11 AM, Pratik Vyas <pd…@erpnext.com> wrote:
> > Great.
> > Item codes starting with ‘pi’ sound ultra cool to me
> >
> >
> > On 25-Mar-2012, at 11:56 PM, AntonK wrote:
> >
> > > It works now, thank you,
> > > pulled request
> > >
> > > On Sunday, March 25, 2012 9:57:46 PM UTC+4, Pratik Vyas wrote:
> > > Great!
> > > erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py
> > >
> > > Line#189
> > >
> > > replace
> > >
> > > file_manager.write_file(fname[1], out)
> > >
> > > by
> > >
> > > file_manager.write_file(fname[1], out.encode(‘utf-8’))
> > >
> > > That should bring balance to the Universe.
> > >
> > > BTW, when you get all this working, please send a pull request / patch
> > >
> > > On 25-Mar-2012, at 11:21 PM, AntonK wrote:
> > >
> > > > Pratik,
> > > > Reading step now works I guess, but new problem is in write back to file stage:
> > > >
> > > > Traceback (innermost last): File “lib/py/webnotes/widgets/form/save.py”, line 40, in savedocs getattr(doclist, action.lower())() File “lib/py/webnotes/model/doclist.py”, line 254, in submit self.run_method(‘on_submit’) File “lib/py/webnotes/model/doclist.py”, line 197, in run_method getattr(self.obj, method)() File “erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”, line 197, in on_submit self.do_stock_reco(is_submit = 1) File “erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”, line 170, in do_stock_reco self.add_data_in_CSV(qty_diff, rate_diff) File “erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”, line 189, in add_data_in_CSV file_manager.write_file(fname[1], out) File “lib/py/webnotes/utils/file_manager.py”, line 214, in write_file file.write(content) UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u041f’ in position 79: ordinal not in range(128)
> > > >
> > > > On Sunday, March 25, 2012 9:15:24 PM UTC+4, Pratik Vyas wrote:
> > > >
> > > > –
> > > > Pratik Vyas
> > > > Twitter : @pdvyas
> > > > ERPNext - World’s most affordable ERP - (Rs 299 / $7 per user per month)
> > > > Website: https://www.erpnext.com
> > > >
> > > > On 25-Mar-2012, at 10:05 PM, Pratik Vyas wrote:
> > > >
> > > > > Hi all,
> > > > > Thinking wild here.
> > > > > Do we have,
> > > > >
> > > > > # -- coding: utf-8 --
> > > > >
> > > >
> > > > This comment is not required, pardon my brain damage due to exams.
> > > >
> > > > Anton,
> > > > Can you try replacing
> > > >
> > > > [d.encode(“ascii”) for d in s]
> > > >
> > > > with
> > > >
> > > > [d.decode(“utf-8”) for d in s]
> > > >
> > > > in erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py (line#63)?
> > > >
> > > > This works for me https://gist.github.com/2198372
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > –
> > > You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> > > To view this discussion on the web visit https://groups.google.com/d/msg/wnframework/-/VmAyVN2bHR0J.
> > > To post to this group, send email to wn…@googlegroups.com.
> > > To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> > > For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
> >
> >
> >
> > –
> > Pratik Vyas
> > Twitter : @pdvyas
> >
> > ERPNext - World’s most affordable ERP - (Rs 299 / $7 per user per month)
> > Website: https://www.erpnext.com
> >
> > –
> > You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> > To post to this group, send email to wn…@googlegroups.com.
> > To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> > For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
> >
> >
> >
> > –
> > You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> > To post to this group, send email to wn…@googlegroups.com.
> > To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> > For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
>
> –
> You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> To post to this group, send email to wn…@googlegroups.com.
> To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
>
On Monday, March 26, 2012 9:16:23 AM UTC+4, Anand Doshi wrote:
Hi everyone,As Rushabh said, the current code is incorrect:
self.data.append([d.encode(“ascii”) for d in s])
We just need to do this:
self.data.append(s)
encode causes unicode string to be converted into str string. i.e. byte string.
decode causes str sting to be converted into unicode string.
No need to encode a ‘str’ type, as it is already encoded. It will
throw UnicodeDecodeError if it finds that ‘str’ is already encoded.
Now, while writing the file: since while importing, the file is
already read as utf-8 encoded, i.e. as a byte string, doing this:
file_manager.write_file(fname[1], out.encode(‘utf-8’))
will cause a UnicodeDecodeError, since the out variable is of ‘str’ type.
If it is already encoded, and if we try to encode it in another
encoding, it casues UnicodeDecodeError. Hence,
file_manager.write_file(fname[1], out)
suffices.
@Anton:
The file with the single record, which you sent, was not encoded as utf-8.
It was showing a “?001” instead of “П001”
I replaced the item code with “П001” and saved it as a utf-8 encoded csv file.
I added this item in my test database. And then did the stock
reconciliation. It worked perfectly… I am attaching the output file
with this email.
In summary:
So the correct code is:
In stock_reconciliation.py in def convert_into_list(self, data): line 62
use
self.data.append(s)
instead of
self.data.append([d.encode(“ascii”) for d in s])
in file_manager.write_file(fname[1], out): line 188
use as it is
file_manager.write_file(fname[1], out)
In warehouse.py in get_bin: line 42:
use
bin = sql(“select name from tabBin where item_code = %s and <br> warehouse = %s”, (item_code, self.doc.name))
instead of
bin = sql(“select name from tabBin where item_code = ‘%s’ and <br> warehouse = ‘%s’” % (item_code, self.doc.name))
This solved the whole issue, when the csv file is encoded in ‘utf-8’
for non english languages. I tested this with the csv provided by
Anton and it worked…
I will push this release in master branch and you can try this out.
Thanks,
Anand.
On Mon, Mar 26, 2012 at 10:00 AM, Pratik Vyas <pd…@gmail.com> wrote:
>
>
> On 26-Mar-2012, at 9:41 AM, Rushabh Mehta wrote:
>
> > Great job guys :)…
> >
> > For my understanding - did you try to remove both the “decode” and “encode” to see if it worked?
> >
> That depends on what you plan to do between reading and writing.
> To make sense of out of you the utf-8 you read, you’ll have to decode it otherwise what you see is raw codepoints.
>
> So this is what you do,
> * Read a utf-8 encoded file., eg ‘\xd0\x9f001\n’
> * Decode it to Unicode. So you have u’\u041f001\n’
> * Do stuff with it.
> * To write it back as utf-8, you need to encode it as utf-8
>
>
> > best,
> > Rushabh
> >
> > On Mon, Mar 26, 2012 at 12:11 AM, Pratik Vyas <pd…@erpnext.com> wrote:
> > Great.
> > Item codes starting with ‘pi’ sound ultra cool to me
> >
> >
> > On 25-Mar-2012, at 11:56 PM, AntonK wrote:
> >
> > > It works now, thank you,
> > > pulled request
> > >
> > > On Sunday, March 25, 2012 9:57:46 PM UTC+4, Pratik Vyas wrote:
> > > Great!
> > > erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py
> > >
> > > Line#189
> > >
> > > replace
> > >
> > > file_manager.write_file(fname[1], out)
> > >
> > > by
> > >
> > > file_manager.write_file(fname[1], out.encode(‘utf-8’))
> > >
> > > That should bring balance to the Universe.
> > >
> > > BTW, when you get all this working, please send a pull request / patch
> > >
> > > On 25-Mar-2012, at 11:21 PM, AntonK wrote:
> > >
> > > > Pratik,
> > > > Reading step now works I guess, but new problem is in write back to file stage:
> > > >
> > > > Traceback (innermost last): File “lib/py/webnotes/widgets/form/save.py”, line 40, in savedocs getattr(doclist, action.lower())() File “lib/py/webnotes/model/doclist.py”, line 254, in submit self.run_method(‘on_submit’) File “lib/py/webnotes/model/doclist.py”, line 197, in run_method getattr(self.obj, method)() File “erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”, line 197, in on_submit self.do_stock_reco(is_submit = 1) File “erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”, line 170, in do_stock_reco self.add_data_in_CSV(qty_diff, rate_diff) File “erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py”, line 189, in add_data_in_CSV file_manager.write_file(fname[1], out) File “lib/py/webnotes/utils/file_manager.py”, line 214, in write_file file.write(content) UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u041f’ in position 79: ordinal not in range(128)
> > > >
> > > > On Sunday, March 25, 2012 9:15:24 PM UTC+4, Pratik Vyas wrote:
> > > >
> > > > –
> > > > Pratik Vyas
> > > > Twitter : @pdvyas
> > > > ERPNext - World’s most affordable ERP - (Rs 299 / $7 per user per month)
> > > > Website: https://www.erpnext.com
> > > >
> > > > On 25-Mar-2012, at 10:05 PM, Pratik Vyas wrote:
> > > >
> > > > > Hi all,
> > > > > Thinking wild here.
> > > > > Do we have,
> > > > >
> > > > > # -- coding: utf-8 --
> > > > >
> > > >
> > > > This comment is not required, pardon my brain damage due to exams.
> > > >
> > > > Anton,
> > > > Can you try replacing
> > > >
> > > > [d.encode(“ascii”) for d in s]
> > > >
> > > > with
> > > >
> > > > [d.decode(“utf-8”) for d in s]
> > > >
> > > > in erpnext/stock/doctype/stock_reconciliation/stock_reconciliation.py (line#63)?
> > > >
> > > > This works for me https://gist.github.com/2198372
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > –
> > > You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> > > To view this discussion on the web visit https://groups.google.com/d/msg/wnframework/-/VmAyVN2bHR0J.
> > > To post to this group, send email to wn…@googlegroups.com.
> > > To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> > > For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
> >
> >
> >
> > –
> > Pratik Vyas
> > Twitter : @pdvyas
> >
> > ERPNext - World’s most affordable ERP - (Rs 299 / $7 per user per month)
> > Website: https://www.erpnext.com
> >
> > –
> > You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> > To post to this group, send email to wn…@googlegroups.com.
> > To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> > For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
> >
> >
> >
> > –
> > You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> > To post to this group, send email to wn…@googlegroups.com.
> > To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> > For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
>
> –
> You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
> To post to this group, send email to wn…@googlegroups.com.
> To unsubscribe from this group, send email to wnframework+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.
>
–
You received this message because you are subscribed to the Google Groups “ERPNext Technical Forum” group.
To view this discussion on the web visit https://groups.google.com/d/msg/wnframework/-/KPOOUIWAbgwJ.
To post to this group, send email to wn...@googlegroups.com.
To unsubscribe from this group, send email to wnframework+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/wnframework?hl=en.