You are right, It works when we internal IP, but partially. The containers are getting identified. But portainer is finding difficulty in getting the volumes, due to which, logs and console are not accessible.
Further portainer becomes very slow as soon as we add a swarm node. With single manager node it works blazingly fast.
There is some serious configuration issue with respect to networks and firewall or even the way I had setup the volumes.
Anyone was successful setting up a cluster for the VMs in GCP can help here with steps or articles.
Your database also need to be labeled and scheduled on same node everytime to access same volumes. Basically anything that has volume either need to be handled via nfs or lock it to same node.
Haven’t used multi-managers. Kubernetes can be re-thought if control-plane HA is needed.
I have achieved 1 manager, 2 worker,
NFS or Managed NFS for sites volume,
DBaaS or node locked stack or separate VM for DB.
Beyond that I’ll keep adding workers.
If I need anything more complex I’ll move to K8s.