We upgraded to v16 and have noticed that all real time notifications have stopped working. I spent a good chunk of time today trying to debug but can’t figure out the problem. Here is kind of a run down at what I have looked at:
Redis Is Running
I have redis running on port 12000. I tried changing the IP binding from 127.0.0.1 to 0.0.0.0. Played around with other config options in the config file to no avail. But I can confirm it’s running and I can connect to it. When I do redis-cli -p 12000 PUBSUB CHANNELS, it returns “(empty array)”. If I do redis-cli -p 12000 MONITOR it initially returns OK but nothing beyond that.
Site Configuration
In file sites/common_site_config.json, I confirmed that parameter “redis_socketio” is set to “redis://127.0.0.1:12000”. I saw in a different forum post about appending “?protocol=3” at the end which I tried but no luck. Nonetheless, it appears to be configured correctly.
SocketIO
The socketio.js program is running. In version 15 I was originally running it as /usr/bin/node /work/frappe/apps/frappe/socketio.js --redis-url redis://127.0.0.1:12000but ChatGPT told me to remove the command line option and instead set environmental variable REDIS_SOCKETIO to “redis://127.0.0.1:12000”. The program is running and the logs don’t show any errors either way I have it but I do notice in the standard log it only shows the following:
Realtime service listening on: ws://0.0.0.0:9000
Realtime service listening on: ws://0.0.0.0:9000
Realtime service listening on: ws://0.0.0.0:9000
Realtime service listening on: ws://0.0.0.0:9000
Realtime service listening on: ws://0.0.0.0:9000
Nothing about being connected to Redis.
Debugging
Using wscat, I am able to connect to the websocket and get a response which seems to indicate that at least the connection between the public internet and SocketIO is sound.
username@hostname:~$ wscat -c wss://my.frappe.server.com/socket.io/?EIO=4\&transport=websocket
Connected (press CTRL+C to quit)
< 0{"sid":"asGWzYvHIpUhhxQCAAAE","upgrades":[],"pingInterval":25000,"pingTimeout":20000,"maxPayload":1000000}
On my Frappe site I added the following JavaScript to try and help debug:
frappe.realtime.on("systemwide.broadcast", function(data) {
// data will be whatever you send from Python
frappe.show_alert({
message: __("Broadcast message: " + data.message),
indicator: "blue"
});
console.log("Realtime broadcast received:", data);
});
console.log("Realtime should be in place!");
And then I both tried running python code from bench console and going into redis directly to try and trigger but no luck.
Python:
import frappe
frappe.publish_realtime(event="systemwide.broadcast",message={"message": "Hello World"}, room="socketio#global")
Redis Console:
$ redis-cli -p 12000
PUBLISH socketio#global '{"event":"systemwide.broadcast","message":{"message":"Hello World!"}}'
I’m running version 16.2.0. Has anyone else encountered this and/or have any other suggestions of what I may be missing here?
