I am being killed by the "Don't Panic It's Us" monster

Frappe Framework
#1

I have created a whitelisted API method that calls frappe.db.sql in a loop.

It doesn’t matter if I call the query with:

  • limit 1000 and offset 0, 1000, 2000 etc. or
  • limit 5000 and offset 0, 5000, 10000 etc …

… the API call returns “Sorry! We will be back soon. Don’t panic.” at a random point between 10000 & 20000 rows.

No result row exceeds 2000 bytes, so the result sets are really unlikely to exceed 5MB.

I tried with virtual machine memory of 4GB …

erpdev@djst:~/frappe-bench-DJST/apps/erpnext$ free -h
              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       2.2Gi       1.0Gi       1.0Mi       635Mi       1.4Gi
Swap:         3.8Gi          0B       3.8Gi

… then jacked it up to 6GB …

erpdev@djst:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          5.8Gi       2.0Gi       3.2Gi       1.0Mi       597Mi       3.6Gi
Swap:         3.8Gi          0B       3.8Gi
erpdev@djst:~$

… and tried again. Available memory seems to makee no difference.

Logs from the bench show nothing out of the ordinary … just the reloading of the edited module ‘testAppBatchItems.py’ :

12:38:07 watch.1          | Rebuilding item-dashboard.min.js
12:38:08 web.1            |  * Detected change in '/home/erpdev/frappe-bench-DJST/apps/testapp/testapp/testapp/doctype/testapp/queries/testAppBatchItems.py', reloading
12:38:10 web.1            |  * Restarting with inotify reloader
12:38:10 web.1            |  * Debugger is active!
12:38:10 web.1            |  * Debugger PIN: 227-261-400
12:38:14 web.1            |  * Detected change in '/home/erpdev/frappe-bench-DJST/apps/testapp/testapp/testapp/doctype/testapp/queries/__pycache__/testAppBatchItems.cpython-38.pyc', reloading
12:38:16 web.1            |  * Restarting with inotify reloader
12:38:17 web.1            |  * Debugger is active!
12:38:17 web.1            |  * Debugger PIN: 227-261-400
12:38:39 web.1            | 127.0.0.1 - - [23/Feb/2021 12:38:39] "POST /api/method/testapp.testapp.doctype.testapp.cleanser.clean HTTP/1.0" 200 -

Questions

  • How can I track down the root cause of this?
  • Are there any documented conditions under which that message is returned?
#2

Is this docker?
Seems like a limit issue but not from the host itself(or maybe yes, I don’t know).

Can you do a script execution with “event listener breakpoints” on “mouse” and step by step check the network and console and sources inside desk to identify this issue? Some errors should pop.

#3

I appreciate your help, but I have made a discovery.

It seems I have been calling the API whitelist method after saving edited code, but before Python has had time to compile the change, create the new *.pyc file and link it into the Frappe code base (or whatever it is that actually goes on in there).

My workflow involves using inotifywait and rsync to first ftp changed files to the development server and second to run a curl command that calls the API.

This has been working fine for a long time, but I had all the code in one Python file.

My problems began when I refactored into smaller files needing to be imported into the main one. I failed to take into account the considerable extra time that that obviously requires.

UPDATE :: 2021/03/15

I have found and solved another cause for the “Don’t Panic” critter. See post below.

1 Like
#4

I’d make some fun on that but my guess is that you’re already full:rofl::rofl::rofl::rofl:.
Had a same’ish problem a while ago.

#6

Tell me!

I could use some fun. I spend so much time on the computer that my pandemic paunch scarcely lets me reach the keyboard anymore.

#7

Had a similar issue with jsignature, but now is fine. :sweat_smile:
image
Them “midgets” that get us carried away are much trollers such time lost.

#8

Here’s another solution, for a different root cause.

Long duration whitelisted methods called through the REST API will time out and throw the “Don’t Panic” page. Here’s how I fixed that.

Here’s the call to the whitelisted method:

install.sh

source  envars.sh;
declare ENDPOINT="${PRCTL}://${TARGET_HOST}/api/method/myApp.myApp.doctype.myApp.myApp";

echo -e "---------------------------------------------------------\n\n"
echo -e "Calling ${ENDPOINT}\n------\n";

declare LOG="/dev/shm/install_myApp.html";
declare AUTHZ="Authorization: token ${KEYS}";

declare EP_NAME="";
EP_NAME="installMyApp";
# EP_NAME="queueInstallMyApp";

curl -s -L -X POST "${ENDPOINT}.${EP_NAME}" \
-H "${AUTHZ}" \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'company=My Company' > ${LOG}
jq -r '.message' ${LOG};

Here’s the Python file with two whitelisted methods. The first fails. The second runs to completion:

myApp.py

# ----------------------------
# This times out
@frappe.whitelist()
def installMyApp(company):
  return install_MyApp(company)

# ----------------------------
# This does not time out
@frappe.whitelist()
def queueInstallMyApp(company):
  frappe.enqueue('myApp.myApp.doctype.myApp.myApp.install_MyApp',
    company=company,
    is_async=True,
    timeout=60000
  )
  return "Enqueued"