Connection from web refused after successful install

zenny · December 29, 2017, 6:43pm

@rmehta Having the same issue both when installed on ubuntu 16.04 and centos7 with frappe_docker as well as vanilla easy install on centos7. external_ip:8000 is not reachable.

In both of the above cases the host cannot connect to frappe/erpnext though the required ports are DNATed okay.

# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
PREROUTING_direct  all  --  0.0.0.0/0            0.0.0.0/0           
PREROUTING_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
PREROUTING_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
MASQUERADE  all  --  172.18.0.0/16        0.0.0.0/0           
POSTROUTING_direct  all  --  0.0.0.0/0            0.0.0.0/0           
POSTROUTING_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
POSTROUTING_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:8080
MASQUERADE  tcp  --  172.18.0.4           172.18.0.4           tcp dpt:3306
MASQUERADE  tcp  --  172.18.0.6           172.18.0.6           tcp dpt:9000
MASQUERADE  tcp  --  172.18.0.6           172.18.0.6           tcp dpt:8000
MASQUERADE  tcp  --  172.18.0.6           172.18.0.6           tcp dpt:6787
MASQUERADE  tcp  --  172.17.0.3           172.17.0.3           tcp dpt:22

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 to:172.17.0.2:8080
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:3307 to:172.18.0.4:3306
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:9000 to:172.18.0.6:9000
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8000 to:172.18.0.6:8000
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:6787 to:172.18.0.6:6787
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2022 to:172.17.0.3:22

Chain OUTPUT_direct (1 references)
target     prot opt source               destination         

Chain POSTROUTING_ZONES (1 references)
target     prot opt source               destination         
POST_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
POST_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain POSTROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain POSTROUTING_direct (1 references)
target     prot opt source               destination         

Chain POST_public (2 references)
target     prot opt source               destination         
POST_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
POST_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
POST_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain POST_public_allow (1 references)
target     prot opt source               destination         

Chain POST_public_deny (1 references)
target     prot opt source               destination         

Chain POST_public_log (1 references)
target     prot opt source               destination         

Chain PREROUTING_ZONES (1 references)
target     prot opt source               destination         
PRE_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
PRE_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain PREROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain PREROUTING_direct (1 references)
target     prot opt source               destination         

Chain PRE_public (2 references)
target     prot opt source               destination         
PRE_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
PRE_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
PRE_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain PRE_public_allow (1 references)
target     prot opt source               destination         

Chain PRE_public_deny (1 references)
target     prot opt source               destination         

Chain PRE_public_log (1 references)
target     prot opt source               destination

Containers connected to default docker0 bridge are reachable, but not the frappedocker_default brige (frappedocker_default network cannot be reached from outside · Issue #27 · frappe/frappe_docker · GitHub)

# brctl show
bridge name	bridge id		STP enabled	interfaces
br-6e05216c4e39		8000.0242206fd64c	no		veth04de492
							veth7d52632
							veth94a19ba
							vethe69056f
							vetheb4465e
docker0		8000.0242f8ff1cc7	no		veth531a3b4
							veth73e5e4c

All necessary docker instances are running as it should:

# docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                 PORTS                                                                    NAMES
c74dfdfc6c0f        jdeathe/centos-ssh:centos-7   "/usr/bin/supervisor…"   9 hours ago         Up 6 hours (healthy)   0.0.0.0:2022->22/tcp                                                     centos-frappe
35beecebafd6        rancher/server:preview        "/usr/bin/entry /usr…"   22 hours ago        Up 10 hours            3306/tcp, 0.0.0.0:8080->8080/tcp                                         naughty_visvesvaraya
ef89f33b1729        frappedocker_frappe           "/bin/bash"              23 hours ago        Up 10 hours            0.0.0.0:6787->6787/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:9000->9000/tcp   frappe
3474f6b5113e        redis:alpine                  "docker-entrypoint.s…"   23 hours ago        Up 10 hours            6379/tcp                                                                 redis-socketio
cf7982c42492        mariadb                       "docker-entrypoint.s…"   23 hours ago        Up 10 hours            0.0.0.0:3307->3306/tcp                                                   mariadb
317dfd665edb        redis:alpine                  "docker-entrypoint.s…"   23 hours ago        Up 10 hours            6379/tcp                                                                 redis-queue
99646f3d70a0        redis:alpine                  "docker-entrypoint.s…"   23 hours ago        Up 10 hours            6379/tcp                                                                 redis-cache

clarkej · December 29, 2017, 11:10pm

Your dnat and network bridge plumbing plus docker config and running process process listings I will leave to you - I do not know what runs inside what in the case of a Docker container…

But now you must prove network connectivity and that the required ports are open as expected.

To validate that, ssh into your container and/or vm instances to run for eg this:

frappe@citron:~/frappe-bench$ nc -vv localhost 8000
nc: connect to localhost port 8000 (tcp) failed: Connection refused
Connection to localhost 8000 port [tcp/*] succeeded!

hope this helps

zenny · December 30, 2017, 6:32pm

@clarkej The iptables output from the host as well as the ‘docker ps’ output clearly shows that the ports are open and DNATed as needed.

However there are more than one tool to test including telnet. If I create any docker container using the default docker0 bridge, the connection is okay, but not with frappedocker_default bridge. Listed all my ordeals at frappedocker_default network cannot be reached from outside · Issue #27 · frappe/frappe_docker · GitHub, Multiple issue with rendering, updating and so on · Issue #25 · frappe/frappe_docker · GitHub and ./dbench init errors on Centos7 host · Issue #28 · frappe/frappe_docker · GitHub, but didn’t witness any traction there.

Anyway, Happy New Year 2018.

Cheers,

clarkej · December 30, 2017, 6:42pm

So you are simply reporting - I assumed you were actually diagnosing!

Thanks you too Zenny! -25 centrigrade plus windchill here today makes -35 or so brrr

zenny · December 30, 2017, 6:45pm

Nope, I am trying to deploy without much success.

Instead of nc I used telnet and got connection problem. I wish you read the long threads I created over github.

-35! Seems like you are based inside Arctic circle?! Keep yuourself warm.

clarkej · December 30, 2017, 7:09pm

Most of us hug our southern border but Canada is called the frozen north for good reason.

“I wish you read the long threads”

Not my idea of fun - for me a case of tldnr.

Enjoy your poking around under the hood - one learns lots but you must be hungry to learn…

zenny · December 31, 2017, 11:52am

Here comes the output from the local frappe instance

root@frappe:~# nc -vv localhost 8000
localhost [127.0.0.1] 8000 (?) open

And this is what it looks from outside:

$ nc -vv 192.168.10.111 8000
Connection to 192.168.10.111 8000 port [tcp/*] succeeded!

This makes obvious that the ports are open, yet I just get a blank page when accessed to http://192.168.10.111:8000.

Therefore it certainly is not a networking issue, but a frappe/erpnext issue, imho.

UPDATE: I tried to do everything manually (‘bench start’) in a vanilla debian 9 server, all necessary components runs fine, yet when I try to access external_ip:8000, the console outputs as follows:

12:10:59 web.1 | 192.168.10.111 does not exist
12:10:59 web.1 | ----------------------------------------
12:10:59 web.1 | Exception happened during processing of request from (‘192.168.10.3’, 52718)
12:10:59 web.1 | Traceback (most recent call last):
12:10:59 web.1 | File “/usr/lib/python2.7/SocketServer.py”, line 596, in process_request_thread
12:10:59 web.1 | self.finish_request(request, client_address)
12:10:59 web.1 | File “/usr/lib/python2.7/SocketServer.py”, line 331, in finish_request
12:10:59 web.1 | self.RequestHandlerClass(request, client_address, self) 12:10:59 web.1 | File “/usr/lib/python2.7/SocketServer.py”, line 652, in init
12:10:59 web.1 | self.handle()
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/serving.
py”, line 290, in handle
12:10:59 web.1 | rv = BaseHTTPRequestHandler.handle(self) 12:10:59 web.1 | File “/usr/lib/python2.7/BaseHTTPServer.py”, line 340, in handle
12:10:59 web.1 | self.handle_one_request()
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/serving.py”, line 325, in handle_one_request
12:10:59 web.1 | return self.run_wsgi()
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/serving.py”, line 267, in run_wsgi
12:10:59 web.1 | execute(self.server.app)
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/serving.py”, line 257, in execute
12:10:59 web.1 | for data in application_iter:
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/debug/init.py”, line 284, in debug_application
12:10:59 web.1 | app_iter = self.app(environ, start_response)
12:10:59 web.1 | File “/home/frappe/frappe-bench/apps/frappe/frappe/middlewares.py”, line 15, in call
12:10:59 web.1 | return super(StaticDataMiddleware, self).call(environ, start_response)
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/wsgi.py”, line 609, in call
12:10:59 web.1 | return self.app(environ, start_response)
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/wsgi.py”, line 609, in call
12:10:59 web.1 | return self.app(environ, start_response)
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/local.py”, line 228, in application
12:10:59 web.1 | return ClosingIterator(app(environ, start_response), self.cleanup)
12:10:59 web.1 | File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeug/wrappers
.py”, line 302, in application
12:10:59 web.1 | return f(*args[:-2] + (request,))(*args[-2:])
12:10:59 web.1 | File “/home/frappe/frappe-bench/apps/frappe/frappe/app.py”, line 59, in application
12:10:59 web.1 | init_request(request)
12:10:59 web.1 | File “/home/frappe/frappe-bench/apps/frappe/frappe/app.py”, line 110, in init_request
12:10:59 web.1 | frappe.init(site=site, sites_path=_sites_path) 12:10:59 web.1 | File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 135, in init
12:10:59 web.1 | local.conf = dict(get_site_config())
12:10:59 web.1 | File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 192, in get_site
config
12:10:59 web.1 | sys.exit(1)
12:10:59 web.1 | SystemExit: 1
12:10:59 web.1 | ----------------------------------------

At 192.168.10.111:80 the nginx runs and I can get the default index page.

I have also tried to turn of the dns_multitenant as @rmehta suggested in this thread ( Unable to access the ERPNEXT page after easy install - #2 by rmehta) without success. The new sites were created manually in this update.