Dashboard issue heatmap in v9

joelios · April 4, 2018, 6:34pm

Is under construction

At the moment all customer transactions (SO, SINV, DN, quotation and opportunity) will be trackt by the heatmap, the other doctypes are in progress

bghayad · April 4, 2018, 10:06pm

Hello @joelios
Great efforts.
But I am still facing the following troubles:

For example, now the date is 5th of April and I do not see this day in the heatmap. It is only showing 4th of April.
At Customer, I am not able to see any mark, but if you are talking about only comments, I do not have old comments to be displayed.
I am able to see marks at item heatmap. But I did one transaction at the Stock Ledger and did not see any mark on the heatmap (I did the activity now, and the date now is 5th of April, and 5th of April is not appear at the heatmap).
At Tools → Activity: I am able to see marks for previous days activities but I am not able to see marks of today activity and I am not able to see the square of 5th April.

Now the time is 1 AM and the date is 5th April. I do not see 5th April square on the heatmap but I can see 4th April square. I did one comment for customer at 4th April but did not see any mark on the heatmap of the customer (and I do not see any marks for the customer heatmap because I do not have old comments). Also, I checked the time at the server, it is 1 AM and the date is 5th April.

About Student (in education module): I do not see any mark in the heatmap for the student and as you know that student heatmap should display the marks based on the student attendance.

About the changes that I did:
I did the changes for the 2 files that you mentioned (13 additions and 2 deletions).

At the browser console, I do not see any error.

Appreciate your kindly reply and help and I am happy that we are proceeding forward specially after this long time without reaching for any result.

The current versions after my last upgrade (and of course, I did not your changes for these 2 files after I finished from the upgrade):

ERPNext: v10.1.19 (master)
Frappe Framework: v10.1.17 (master)

Regards
Bilal

mtajammul · April 5, 2018, 6:19am

Hi joelios,
and how fix heat map issue with student attendance?

lasalesi · April 5, 2018, 8:19am

Hi all,

the version from @joelios worked smothely in my test:

Configurable style:

And continuous:

Nice work. Please merge.

joelios · April 5, 2018, 8:38am

Hi All

I’m currently working on a pull request with which the following features will be implemented and the following bugs will be fixed:

Feature:

Configurable style of heatmap (continuous or discret) in Global Defaults
The heatmap reflects all transactions against the linked records in the corresponding dashboard

Fixes:

The heatmap will show all 7 days of a week, not just 6
The marks on the heatmap will be shown

Today, my feature/fix branch of Frappe and ERPNext (Branch: joelios-heatmap), reflect all features and fixes above. But at the moment only the heatmap of the doctype Customer will reflect all transactions of linked records in the doctype. I will create an pull request of this stand to merge it into master.

Then i will work out another pull request to update all other heatmaps (not just customer), to merge it in a feature step into master.
To know all doctypes, which has a heatmap and should reflect all linked transactions from the corresponding dashboard, i need your help! You have to tell me all affected doctype of what i have planned. For this, i will open e new, clean, topic in this forum.
As soon as i have all inputs from you and coded all this, i will create the second pull request to improve all other doctypes except customer.
@rmehta What do you think about my plans? Do you agree with it?

@lasalesi thanks for your feedback! Nice to know.

Edit:
Frappe Pull request
ERPNext Pull Request

joelios · April 5, 2018, 10:03am

For further discussions and or inputs please see:

bghayad · April 5, 2018, 10:03am

Hello @joelios

You are doing a great job and thanks a lot for you. This is wonderful.
From my side: at this time, it is 1 PM and date is 5th April, I am able to see the square at the heatmap for day 5th April, but I am not able to see any marks related to yesterday or today activities.

For student attendance (education module): it seems that the marks are not working at the heatmap totally.

What could be the reason for my troubles with the marks (that I am not able to see recently marks, I mean for last few days “recently days”)?

Regards
Bilal

joelios · April 5, 2018, 10:08am

Hi @bghayad

I had a little mistake in my time shift correction.
Please try my lattest pushes on github, now it should work propperly.
Frappe → Branch: joelios-heatmap
ERPNext-> Branch: joelios-heatmap

But please pull both of me, frappe as well as erpnext.
And dont forget to run

$ bench build
$ bench migrate
$ bench restart

after pull

cheers

bghayad · April 5, 2018, 1:33pm

Yes @joelios
Wonderful.
Every thing is working fine. BRAVO.
But still Student heatmap (and I know that still you did not do any thing with it) is not working properly. Actually it is showing the marks on the heatmap but in wrong date.

I need to add one more thing for your previous post, it is required to do the following command:

$ bench migrate

Because you need to create the field Heatmap Style in the database.

I have one point out of this post:
Do you have activity regarding grid? Because we would if it possible to resolve the list view problem (as we need to be able to add more fields and to have the horizontal scroll bar to navigate left and right).

Regards
Bilal

joelios · April 5, 2018, 1:50pm

I’m glad to hear that it works for you too!

Yes, sorry, you’re absolut right!

I will look at the heatmap of student in the near future. In the new topic you see my backlog and where I am currently.

As soon as the heatmap topic is finished, I can look at this.

cheers

joelios · April 6, 2018, 7:31am

Hi @bghayad

I’ve finaly reworked the student heatmap also.
If you pull my latest commit, the student heatmap should work correct

Cheers!

bghayad · April 6, 2018, 12:37pm

Hello @joelios
Great full. Amazing.
It came after long long waiting time.

So it needs pull request to be merged.

Regards
Bilal

joelios · April 6, 2018, 12:41pm

Pull requests are placed. They waiting to be merged

bghayad · April 6, 2018, 12:51pm

Thanks a lot for you @joelios

I was need to ask you about grid and if you can help to resolve the list view problem.
But first of all: what kind of activity you have about grid?
Second: Can we resolve the list view problem: because we would if it possible to resolve the list view problem (as we need to be able to add more fields and to have a horizontal scroll bar to navigate left and right).

Regards
Bilal

joelios · April 10, 2018, 9:38am

Hi @bghayad

With changes on list_render.js, it is possible to add mor fields to listview:

but adding a horizontal scrollbar would involve significant changes to list_render.js!
My question; Do you really need to force more fields in the listview? you can solve the same also over a report …?

Cheers

bghayad · April 10, 2018, 11:52am

Hello @joelios
Wonderful and amazing. Thanks a lot for you on this big effort and great work.
I will try list_render.js as this will resolve maybe 80% of the problem.
Having horizontal scroll bar to navigate left and right for more columns is giving powerful to manage the business because the user will have a look for all the records with very enough information to decide which record is need to work on it. This will prevent from the opposite case where user will need to go through each record and check if this record is the needed or not and coming back to list and repeat this work.

One more feature is also important to have strength while working with the records:
Inside the document, there should be Next and Back button to be able to go for the next record or previous record without coming back to the list (of course this to be within the filtered records).

But let me tell you that you helped us a lot by directing us to list_render.js and as I told you, this resolved 80% of the problem. I will go for the posts that we placed about this problem and direct them for this link.

Regards
Bilal

joelios · April 10, 2018, 11:59am

Youre welcome!

The code youre looking for is in line 204 (if (me.total_colspans < 12)):

	docfields_in_list_view.forEach(function (d) {
		if (overridden.includes(d.fieldname) || d.fieldname === me.title_field) {
			return;
		}
		if (me.total_colspans < 12) {
			me.add_column(d);
		}
	});

and line 249:

		// Limit number of columns to 4
		this.columns = this.columns.slice(0, 4);

best regards
Joelios

bghayad · April 15, 2018, 10:39pm

Hello @joelios

I would to notify about heatmap for education, certainly for student attendance.
I am getting the following error when I add attendance for the student and coming to student document which contains the heatmap:

Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/app.py”, line 62, in application
response = frappe.handler.handle()
File “/home/frappe/frappe-bench/apps/frappe/frappe/handler.py”, line 22, in handle
data = execute_cmd(cmd)
File “/home/frappe/frappe-bench/apps/frappe/frappe/handler.py”, line 53, in execute_cmd
return frappe.call(method, **frappe.form_dict)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 939, in call
return fn(*args, **newargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/desk/notifications.py”, line 274, in get_open_count
out[‘timeline_data’] = module.get_timeline_data(doctype, name)
File “/home/frappe/frappe-bench/apps/erpnext/erpnext/education/doctype/student/student.py”, line 63, in get_timeline_data
timestamp = get_timestamp(date)
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/data.py”, line 126, in get_timestamp
return time.mktime(getdate(date).timetuple())
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/data.py”, line 42, in getdate
return parser.parse(string_date).date()
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 1182, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 556, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 675, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 192, in split
return list(cls(s))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 61, in init
‘{itype}’.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not int

I think that the problem maybe related to the unix_timestamp as the following line is causing the error:

timestamp = get_timestamp(date)

Again, it is happening only with student attendance transactions.

Regards
Bilal

bghayad · April 15, 2018, 11:01pm

Hello @joelios

I am sorry, for education: I discovered that I did not do the frappe changes. I will re fix and come back to you.

Regards
Bilal

bghayad · April 16, 2018, 2:32pm

Hello @joelios
I confirmed the problem that is related to the education heatmap that you proposed, but it is only related to the attendance and it seems that it is related to the unix_timestamp at the query which cause error that is related to the timestamp = get_timestamp(date) as in the following traceback:

Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/app.py”, line 62, in application
response = frappe.handler.handle()
File “/home/frappe/frappe-bench/apps/frappe/frappe/handler.py”, line 22, in handle
data = execute_cmd(cmd)
File “/home/frappe/frappe-bench/apps/frappe/frappe/handler.py”, line 53, in execute_cmd
return frappe.call(method, **frappe.form_dict)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 939, in call
return fn(*args, **newargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/desk/notifications.py”, line 274, in get_open_count
out[‘timeline_data’] = module.get_timeline_data(doctype, name)
File “/home/frappe/frappe-bench/apps/erpnext/erpnext/education/doctype/student/student.py”, line 63, in get_timeline_data
timestamp = get_timestamp(date)
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/data.py”, line 126, in get_timestamp
return time.mktime(getdate(date).timetuple())
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/data.py”, line 42, in getdate
return parser.parse(string_date).date()
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 1182, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 556, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 675, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 192, in split
return list(cls(s))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/dateutil/parser.py”, line 61, in init
’{itype}’.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not int

Can you please help?
Regards
Bilal