ERPNext version 9 upgrade test in high data live environment

[System19]

Agreed its great to hear everyones diverse situations but to put it succently at post 84 of 120

Forget about about offline/online speed etc. Thats all about to become irrelevant with the native React apps and ERPNext desktop.
All physical barcodes can be scanned and printed with no issue at minimal cost.
Items can be divided for the cashier by product group / barcode and alphabetical search.

For a fast small supermarket or food place what has to be done is to work out the most efficient use of data TX and DB overhead that we can get away with on each transaction/sale.
We need a DBA who can look at the current ERPnext POS transaction process/API… Who will then probably have to mod it a bit… Then we utilise the many advantages of native separate POS apps.
Slightly slower retail stores would probably use the built in ERPnext POS as it has more printing / payment and customer features.
Separating the POS module is a good idea I think. Just need to find Funds for DBA and React Dev with a strong knowledge of frappe…

[noetico]

Ok; sure. Damaged barcodes are not IT problem; but‎ previously, when they couldn’t scan they just punched in the barcodes! Now doing that; most times throws in unwanted items into the cart. Again, I restate; this has not ever been an issue for over 15 installations I’ve ben involved with until now, last Sunday. So something has changed about the pos search bar! I got the same complaint just this morning! It’s something I’ve never got for about 3 years now.

[noetico]

Great stuff!

[Sami_Tayara]

Hi BKM. I like to contribute in your POS Module Volunteer work. I think POS is critical component and it has to be optimized from both usability and security sides.

I have experience in high-traffic high-value retail and I was in charge of ensuring security and theft/loss prevention too. All the extreme use cases have to be considered, not just idealistic world scenarios. Because in the real world fraudsters are lurking everywhere among your customers and employees. The POS at large supermarkets have already mastered these problems as you see no items delayed or captured incorrectly, and when exceptions happen the cashier has to call a supervisor to handle.

[noetico]

Awesome input! Thanks a lot.

[noetico]

UPDATE ON PERFORMANCE

Hello People; I paid a visit to the shop this morning. So we’ve achieved print security by merging the print and new to one button and enabling silent print; essentially no room for multiple copies except in the rare case of an exceptionally technical cashier who can turn off silent print; for now; we are super good with printing.

On performance, invoice submission is still fairly good, 7 to 12 seconds, a few longer.

CRITICAL ISSUES (I confirmed for myself)
Some of these I said were impossible when I got calls yesterday
1; Cashier is scanning items and suddenly the screen freezes for say 5 seconds and when it returns the sale is SUBMITTED automatically, no tender screen, no submit confirmation! This means remaining items have to be done on a new invoice. this clearly is a bug, I witnessed it first hand.
2. Many times, at random, the + and - signs to increase quantity do not work, setting an item to zero does not delete it, the pos page needs to be reloaded to make progress, essentially; she restarts scanning customer’s items, serious issue!
3. The total for scanned items may not update on adding or removing items, page needs to be reloaded.
Has anyone noticed these? @olamide_shodunke @ganas @bkm @System19 @Sami_Tayara @LifeP

[ganas]

can you document the behaviours with screen shots and how to reproduce them in a github issue so it can be looked at

[noetico]

hmmm, will be hard to get screen shots, except I go stay a few hours there, its random; but I have seen for myself though. Also got a call today, seems like some batch process runs on schedule in the framework; they witness a hang up of 2 to 5 minutes and then things spring to normal speed. are there scheduled tasks that may run during business hours?

[noetico]

This may be the most troubling thing of all that I have seen and heard from this new version. As we make progress, I am deeply regretting this ugrade how can 1 item enter the pos in 2 lines??? @ganas @bkm please observe. This is from the same shop we’ve been following. They call me with these issues, today I got them to send one to me.

Take a look at the last 2 items, the pos has put in the same item twice, not quantity of 2, but as 2 lines when 1 was expected. and it has updated the total to 10k+ instead of 7k+!! I may head back to the previous version and update their stock balances, this has been a major embarrassment, it appears this online pos was built due to pressure on the team and they were not given time. Its not production ready, simple.
Please have a look, I have updated this topic on GitHub also: Possible Critical bugs on ERPNext V9 Online POS · Issue #11453 · frappe/erpnext · GitHub

[bkm]

@noetico

I have been trying now for 2 days to replicate the issues you have brought to us in this thread. I have NOT been able to get any of them to replicate in my two test systems. I have a cloud test system and a local box. Both are Debian 8 servers with fresh installs of the latest v9 ERPNext.

Has anyone else been able to duplicate any of the errors Chidi is showing us here? If you have, please add your steps here so we can try to track them down.

If we are unable to replicate these errors, I might suspect that something went wrong during your multiple upgrade steps to get from version 6 to version 9. That is why I have spent several silent days trying to get any of the errors you speak of to occur in any of my testing systems. I am just not able to make the same errors.

So I am now asking if anyone else can replicate these?

Thank you,

BKM

[noetico]

@bkm Have you checked the github issues I posted on these issue?
‎Possible Critical bugs on ERPNext V9 Online POS · Issue #11453 · frappe/erpnext · GitHub

I also posted an invoice sample sent; where the same item can be entered twice, as separate line items. I also argued these things; but I have seen most for myself. As I get samples; i’ll post. You may not be able to replicate easily; this is a very rapid and volume intensive environment, unlike your sample one, myself I can’t replicate! But once the traffic spikes then you see some little issues.

Now; I also saw this earlier and again today; certain items will not be available when scanned, it comes back with no data on pos. When you check in item master (by barcode) it comes up. Now, ‎to solve this, we rename the item, its prevalent for items where there’s a symbol especially %, ', &, so I suppose there’s some code interference with these symbols at pos; I think the wildcard search feature. On removing these symbols, the item becomes instantly available.

Note also; that these happen once traffic peaks in the evenings! Honestly; its hard to replicate on a simple setup; so that’s why I try to make long, detailed explanations. So let’s look at what needs optimization when there a lot of queries. ‎

[noetico]

Did you see the invoice posted above? Its a pdf print of an invoice with the issue of possibly entering one item in 2 lines. At pos this has never been possible. All these updates we post are not to make the system look inefficient, we want it better and more able to handle bigger businesses, its a good thing we have this thread, the developers should also be very interested in looking at what parts of the new online pos may be giving such issues.

I dont think its due to the update as well, the patches skipped have nothing to do with this, mainly the desktop icon patches, these should have no impact on pos.

[noetico]

Lets not forget a few original points

1. We were on v6 for over 15 shops in like 2 years now and until this store grew out of hand we have received NO COMPLAIN ABOUT POS, NONE! Over 15 installations (one v4).
2. This whole thing was a performance based complaint where invoices and transactions took a long time to submit and also pos sometimes took a long time to pull in scanned items, no issues with data reliability or integrity at pos… never, not one. And within these 15 separate installations I can say there are over 3000 daily invoices cummulative, no issues with selling; its the most dependable pos I know. Simple.
3. These came up just after the upgrade and happen at least 7 times per day, dont forget that the pos, although an upgrade is basically the same, so very low learning curve, the older version like I said; took some cashiers 2 to 3 minutes to learn and start selling easily, same with this, its easy as well, so after initial introduction they just started selling and I remember they posted over 600 invoices that day and amongst them were some of these issues.

All I’m saying is, we just need to optimize ok. Its a new version, but lets see what we can fix by looking at the code and maybe the devs can see a few possible bug sources and patch. I know the client will not pay and dev time is money, but these older clients are our future for better things as a community, if they speak well, we get bigger, richer, more generous clients.

[noetico]

@ganas @bkm @System19 here’s something I just received right after sending the last update. I have also brought this up before. Items may show no data on pos but available in item master and every other part of the system. I realized its something with special characters like ’ % or &. renaming the item and removing this solves the problem, its just an example, so a well known item which was sold frequently, suddenly shows no data at pos. I understand the special characters may be interfering with the code, but this was never a problem.

[noetico]

more coming in… unwanted items and same item entering in different lines, unplanned, even if unplanned they should accumulate quantity on 1 line. It appears searching for an item has the potential of making the pos enter the item as unique, this happens for searched items mostly.

[Not_a_countant]

Hi Noetico,
Not sure if it is possible. Note that you have the latest version
(1) download the source code from your production
(2) download the source code from GITHUB.

compare the two. If you see differences, I recommend backing up the database, install fresh from GIT and restore your database.

My guess is that it might fix some of the inconsistencies that happened during your updates.

Also did you customise your code? Maybe you might have to customize it in the new version.

That might already put you on a path that has a better direction. Right now, no one knows what is going on

Good luck!

[noetico]

Ok, I’ll do as you have advised. although this started happening from the first day we installed, and correct me Sir; this online pos is a new module found in the selling>page>pos folder, this was added fresh; this was previously under the accounts>doctype>sales_invoice>pos folder in v6 (seems the offline is still there), so this was pulled in fresh as a new module.

While updating I also used bench update --reset so I think that has discarded any edits in the core right?

for edits, the only edit is where we made the pos send to print and generate a new invoice with one click, as follows:

[bkm]

I am sorry, and I know this is going to sound cold. It is not meant to be.

This is beginning to look like a one person rant. I cannot replicate anything you have presented except the occasional odd character in an item name that confuses the POS. I have also not seen anyone else report experiencing any of the same issues either.

When taking all of this into consideration, I have to suspect your particular installations.

I know that you have many separate installations, but they all went through the same upgrade process from version 6 to version 9. You are the only implementer to have ever reported even attempting such a large single jump in versions, and several of your previous posts indicate you had several trouble points during the upgrade processes that forced you to make changes along the way.

From an unbiased observer perspective, I have to say this really does not add up.
From a POS group perspective, I must also say that we cannot fix things we cannot replicate. We cannot prioritize the issues you bring up unless we can find a cause for them.

I know this is really NOT what you want to hear, but in order to better find the real problems we MUST be able to duplicate the errors in other independent systems in order to take your particular systems out of the list of possible causes.

I have already put 2 working days into trying to duplicate any of the errors you have reported. The only one that I could “sort of” duplicate was when an item name had a non alpha-num character in it. Even then, I could still get it to show up in POS with correct inventory numbers.

During my testing, and on a local box, I lowered the amount of memory to only 4GB, forced the CPU to also run another process in background to eat up CPU cycles, and simultaneously started a large import function in erpnext on another terminal while I was using the POS system. Aside from a minor delay in response time (3-4 seconds) I could not make anything break. I was also only using a 100mb ethernet connection.

Unless we can find other systems and other users where these errors can occur, I don’t have anything I can suspect as the cause except possibly something having gone wrong during your updates.

I really hope that one of the developers reads through all of this and has one of those instant wow moments where they realize what exactly is going on in your setup, then explains it here with possible solutions.

Unfortunately, there do not appear to be any developers interested in the POS module as we do not have any volunteers to work on it.

Chidi, please understand that I get the fact you are frustrated and that you have a unique perspective on the POS module at this time. However, I have no idea how to help because I cannot find a way to replicate your issues. Only when we have that, can we get someone to look through the code for the triggers.

BKM

[ganas]

I have to agree with the guys, I cannot replicate your issues.

Keep in mind, you commented out and skipped dozens of batches when u updated and I think they are coming to hunt you back.

So I think you best next action is to backup your database and restore it on clean install of v9

[noetico]

One person rant; no problem at all, I’ll take that with a smile. **Actually I have just 2 I have upgraded from 6 to 9, same vm image, just 2.
read carefully: I took the image of the big store because they have most of the items a standard supermarket has, I cleaned out the transactions except master data, items, prices, and installed at this other shop, when I did the upgrade at the big shop, I repeated at the small one, 6 to 9.
no complaints, no calls from that small shop, they have very few customers daily, 50 to 100, 1 cashier, 1 inventory pc.
The issue is only with this big shop with huge data, strange things happening; and how can you replicate when you dont have that type of environment and the data? with over 7 cashiers attending to over 30 people at once, waiting in queues. but its all good, if no one’s interested and the team has no interest and it is now a ‘rant’ no problem at all, I rest my case.