Dashboard issue heatmap in v9

bghayad · March 30, 2018, 6:20pm

Can you please upgrade to the updated version as in the below link and advise if it is working?
From my side I tried and it is not working.
https://github.com/frappe/erpnext/issues/12387
https://github.com/frappe/frappe/pull/5318

Regards
Bilal

mtajammul · April 1, 2018, 7:42am

I have upgraded today to:

ERPNext: v10.1.18 (master)
Frappe Framework: v10.1.16 (master)

But heat chart is still not working.

bghayad · April 1, 2018, 6:28pm

Hello @mtajammul
Are you using master or develop branch?
Regards
Bilal

mtajammul · April 2, 2018, 6:35am

I have update Mater branch.
When master will be in the merge?

bghayad · April 3, 2018, 4:48pm

Hi @lasalesi
Did you try the last update at develop branch?
Regards
Bilal

joelios · April 4, 2018, 8:26am

Hi @bghayad @mtajammul @lasalesi

I was able to fix this issue and also
https://github.com/frappe/frappe/issues/5376

There were two causes:

The dict key was not an integer, it was a float
The timestamp was shifted 79200 into the future

The Pull Request #5377 is pending for merge into master branch…
https://github.com/frappe/frappe/pull/5377

Edit: This PR solves the problem that the marks on the heatmap are not displayed. The possibility to change the style of the heatmap is not fulfilled…

bghayad · April 4, 2018, 12:08pm

Hello @joelios
Thank you
My current version is:
ERPNext: v10.1.18 (master)
Frappe Framework: v10.1.16 (master)

And I did the changes that mentioned in this link:
https://github.com/frappe/frappe/pull/5372/commits/8af5281123e191dd6081fdf06341be23a71459ef

But still the marks are not appear in the heatmap.
I did restart after the changes but the same problem.

So, please advise if you were trying this in develop branch and on which version of erpnext and frappe?
And if I am missing something, please advise me.

Please note that I created new orders and sales invoices for the customer (to confirm if it is showing at least new events), but it did not show them.

Regards
Bilal

joelios · April 4, 2018, 12:16pm

Hi @bghayad

I use the same versions like you:
ERPNext: v10.1.18 (master)
Frappe Framework: v10.1.16 (master)

Please notice that the transactions, shown in the heatmap, are diffrent from doctype to doctype.
For e.g. in the doctype “Customer”, the created Sales Orders and/or Sales Invoices will not be showed in the heatmap!

In the doctype “Customer”, only the quantity of comments per customer will be shown!
So, make a comment in your customer record, and you will see the heatmap (if my fix is included) will show a green mark on the 4th of april

If you have included my fix, but nevertheless a green mark will be shown, make shure you have run:

$ bench build

after inserting my fix.

EDIT:
some e.g.:
Customer:

Item:

Project:

and so on…

If you want to tell me now that the content (transactions) of the heatmap is sometimes not very meaningful; I agree with you completely.

But you have to understand, I did not improve the function in and of itself, but I made sure that the function works properly in its original form again

bghayad · April 4, 2018, 1:35pm

Hello @joelios
First of all, the only way that heatmap marks to appear is to go for Tools, Activity and from there to select the customer and reaching to the customer document, then I can see the green marks. If I go direct to the customer from the CRM and selecting the customer, then I can not see the marks at the heatmap.

Second thing: The marks that I can see it (of course if I reach for the customer from the Activity) is not related for comments for sure because they are in a dates that no comments added at all for the customer.

Coming back to the nature of the marks that are appearing in the heatmap:

It is mentioning that it is against the customer transactions, see timeline below for details. So what are these below timeline? It is the transactions that are mentioned in the dashboard which are the sales invoice, the sales order, … etc. That is what I can understand, otherwise: why it is mentioned see
timeline below for more details? What are these transactions below?

Regards
Bilal

joelios · April 4, 2018, 3:02pm

Hi @bghayad

So, let’s make it clear. I’ve just put on a new VM and pulled the latest versions of frappe and ERPNext:
grafik

next i added my javascript fix and run

$ bench build

Now I’m ready to go to your points:

I can open the customer record normally via the CRM, and the heatmap is displayed correctly, including marks. I would like to show it to you with a video, but unfortunaly i cant upload it because its to big.
Do you have any errors in your console? Did you implement my solution correctly and run bench build?

Guaranteed refer the marks at the customer to comments. Here (see line 398 till 414) is the corresponding python code that generates the data for the heatmap of doctype customer.

this is “below timeline”:

That would be very useful! Unfortunately, it is definitely not.

EDIT:
I completely agree with you that it is an absolute nonsense to show the comments in the customer’s heatmap instead of the linked documents. I will work out another pullrequest, so that in the future the linked documents will be displayed in the customer’s heatmap. At the moment I can only repeat that the function is “as bad as it is”. But, at least, now it works

joelios · April 4, 2018, 4:26pm

@bghayad
I’ve created (and tested) an example for you, so that you can expand the customer’s heatmap arbitrarily.

Example “Customer’s heatmap based on linked Sales Orders”:

Remove the below bold marked in line 12 of customer.py
from erpnext.accounts.party import validate_party_accounts, get_dashboard_info , get_timeline_data # keep this
Add the function below in line 181 of customer.py

def get_timeline_data(doctype, name):
        from six import iteritems
        from frappe.utils import (cint, cstr, flt, formatdate, get_timestamp, getdate, now_datetime, random_string, strip)
        out = {}
        items = dict(frappe.db.sql('''select transaction_date, count(*)
                from `tabSales Order` where customer_name=%s
                and transaction_date > date_sub(curdate(), interval 1 year)
                group by transaction_date''', name))

        for date, count in iteritems(items):
            timestamp = get_timestamp(date)
            out.update({timestamp: count})

        return out

Result:
My Sales Orders of customer “test”:

The heatmap of customer “test”:

Regards
Joelios

lasalesi · April 4, 2018, 6:28pm

Hi @joelios,

very nice, this is exactly how I would expect the heatmap to work. Thanks! PR?

joelios · April 4, 2018, 6:34pm

Is under construction

At the moment all customer transactions (SO, SINV, DN, quotation and opportunity) will be trackt by the heatmap, the other doctypes are in progress

bghayad · April 4, 2018, 10:06pm

Hello @joelios
Great efforts.
But I am still facing the following troubles:

For example, now the date is 5th of April and I do not see this day in the heatmap. It is only showing 4th of April.
At Customer, I am not able to see any mark, but if you are talking about only comments, I do not have old comments to be displayed.
I am able to see marks at item heatmap. But I did one transaction at the Stock Ledger and did not see any mark on the heatmap (I did the activity now, and the date now is 5th of April, and 5th of April is not appear at the heatmap).
At Tools → Activity: I am able to see marks for previous days activities but I am not able to see marks of today activity and I am not able to see the square of 5th April.

Now the time is 1 AM and the date is 5th April. I do not see 5th April square on the heatmap but I can see 4th April square. I did one comment for customer at 4th April but did not see any mark on the heatmap of the customer (and I do not see any marks for the customer heatmap because I do not have old comments). Also, I checked the time at the server, it is 1 AM and the date is 5th April.

About Student (in education module): I do not see any mark in the heatmap for the student and as you know that student heatmap should display the marks based on the student attendance.

About the changes that I did:
I did the changes for the 2 files that you mentioned (13 additions and 2 deletions).

At the browser console, I do not see any error.

Appreciate your kindly reply and help and I am happy that we are proceeding forward specially after this long time without reaching for any result.

The current versions after my last upgrade (and of course, I did not your changes for these 2 files after I finished from the upgrade):

ERPNext: v10.1.19 (master)
Frappe Framework: v10.1.17 (master)

Regards
Bilal

mtajammul · April 5, 2018, 6:19am

Hi joelios,
and how fix heat map issue with student attendance?

lasalesi · April 5, 2018, 8:19am

Hi all,

the version from @joelios worked smothely in my test:

Configurable style:

And continuous:

Nice work. Please merge.

joelios · April 5, 2018, 8:38am

Hi All

I’m currently working on a pull request with which the following features will be implemented and the following bugs will be fixed:

Feature:

Configurable style of heatmap (continuous or discret) in Global Defaults
The heatmap reflects all transactions against the linked records in the corresponding dashboard

Fixes:

The heatmap will show all 7 days of a week, not just 6
The marks on the heatmap will be shown

Today, my feature/fix branch of Frappe and ERPNext (Branch: joelios-heatmap), reflect all features and fixes above. But at the moment only the heatmap of the doctype Customer will reflect all transactions of linked records in the doctype. I will create an pull request of this stand to merge it into master.

Then i will work out another pull request to update all other heatmaps (not just customer), to merge it in a feature step into master.
To know all doctypes, which has a heatmap and should reflect all linked transactions from the corresponding dashboard, i need your help! You have to tell me all affected doctype of what i have planned. For this, i will open e new, clean, topic in this forum.
As soon as i have all inputs from you and coded all this, i will create the second pull request to improve all other doctypes except customer.
@rmehta What do you think about my plans? Do you agree with it?

@lasalesi thanks for your feedback! Nice to know.

Edit:
Frappe Pull request
ERPNext Pull Request

joelios · April 5, 2018, 10:03am

For further discussions and or inputs please see:

bghayad · April 5, 2018, 10:03am

Hello @joelios

You are doing a great job and thanks a lot for you. This is wonderful.
From my side: at this time, it is 1 PM and date is 5th April, I am able to see the square at the heatmap for day 5th April, but I am not able to see any marks related to yesterday or today activities.

For student attendance (education module): it seems that the marks are not working at the heatmap totally.

What could be the reason for my troubles with the marks (that I am not able to see recently marks, I mean for last few days “recently days”)?

Regards
Bilal

joelios · April 5, 2018, 10:08am

Hi @bghayad

I had a little mistake in my time shift correction.
Please try my lattest pushes on github, now it should work propperly.
Frappe → Branch: joelios-heatmap
ERPNext-> Branch: joelios-heatmap

But please pull both of me, frappe as well as erpnext.
And dont forget to run

$ bench build
$ bench migrate
$ bench restart

after pull

cheers