Security Breach on erpnext.com

Install / Update Security
1

There was recent post on the discuss forum indicating that some of our customers on the Frappe Cloud were getting spam emails from an entity claiming to be a verified ERPNext vendor.

Now this seemed to be a very weird issue since neither Frappe, nor the Foundation shares or sells email ids to 3rd parties. Neither is any information collected if you use the easy install script or the official VM. This led us to believe there could’ve been a breach on eprnext.com.

To our dismay this turned out to be true.

After looking at the original source of the email sent by the bad actor, we were able to locate the IP address from which the emails were being sent. Here is the link to the original source of the email.

An ERPNext instance was hosted on the IP address and, emails were being sent via it.

Next we started looking into our nginx access logs, for any requests coming in from the aforementioned IP address, and we realized there were quite a few requests coming in from there. All the requests coming in from that IP were constantly snooping on newly created Communications and Account Requests.

bash-4.2# grep -nRH "167.99.175.247" .
./access.log-20180930:11742:167.99.175.247 - - [29/Sep/2018:12:37:47 +0530] "POST / HTTP/1.1" 200 70 "-" "python-requests/2.19.1" "-"
./access.log-20180930:11743:167.99.175.247 - - [29/Sep/2018:12:37:48 +0530] "GET /api/resource/Account%20Request?limit_start=0&fields=%22%2A%22&limit_page_length=1000 HTTP/1.1" 200 59892 "-" "python-requests/2.19.1" "-"
./access.log-20180930:11744:167.99.175.247 - - [29/Sep/2018:12:37:49 +0530] "GET /api/resource/Communication?limit_start=0&fields=%22%2A%22&filters=%5B%5B%22Communication%22%2C+%22communication_type%22%2C+%22%3D%22%2C+%22Communication%22%5D%2C+%5B%22Communication%22%2C+%22sender_full_name%22%2C+%22%3D%22%2C+%22Guest%22%5D%5D&limit_page_length=1000 HTTP/1.1" 200 98044 "-" "python-requests/2.19.1" "-"
./access.log-20180930:31781:167.99.175.247 - - [30/Sep/2018:00:00:50 +0530] "POST / HTTP/1.1" 200 70 "-" "python-requests/2.19.1" "-"
./access.log-20180930:31782:167.99.175.247 - - [30/Sep/2018:00:00:50 +0530] "GET /api/resource/Account%20Request?limit_start=0&limit_page_length=1000&fields=%22%2A%22 HTTP/1.1" 200 59746 "-" "python-requests/2.19.1" "-"
./access.log-20180930:31783:167.99.175.247 - - [30/Sep/2018:00:00:52 +0530] "GET /api/resource/Communication?limit_start=0&limit_page_length=1000&filters=%5B%5B%22Communication%22%2C+%22communication_type%22%2C+%22%3D%22%2C+%22Communication%22%5D%2C+%5B%22Communication%22%2C+%22sender_full_name%22%2C+%22%3D%22%2C+%22Guest%22%5D%5D&fields=%22%2A%22 HTTP/1.1" 200 98454 "-" "python-requests/2.19.1" "-"

Since the Frappe Framework stores the IP address of the sessions created by the user, we started looking into it. We soon realized that the password of one of the users on ERPNext was compromised.

{u'device': u'desktop',
  u'sessiondata': u"{u'session_expiry': u'12:00:00', u'session_country': {u'geoname_id': 6252001, u'iso_code': u'US', u'names': {u'ru': u'\\u0421\\u0428\\u0410', u'fr': u'\\xc9tats-Unis', u'en': u'United States', u'de': u'USA', u'zh-CN': u'\\u7f8e\\u56fd', u'pt-BR': u'Estados Unidos', u'ja': u'\\u30a2\\u30e1\\u30ea\\u30ab\\u5408\\u8846\\u56fd', u'es': u'Estados Unidos'}}, u'last_updated': '2018-09-30 00:00:50.954194', u'lang': u'en', u'user_type': u'System User', 'user': u'********@erpnext.com', u'full_name': None, u'device': u'desktop', 'session_ip': u'167.99.175.247'}

We immediately reset all user passwords and restricted admin and server access to a select few. Withing hours of doing this the ERPNext instance hosted on the IP address of the bad actor was taken down.

The IP address in question belonged to a Digital Ocean droplet based in California. We have already emailed Digital Ocean about the breach, but are yet to hear back from them.

We can confirm that none of our servers, or the ERPNext instances of any of our customers were breached, in the scenario. The only access the bad actor had to, were the Communications and Account Requests, of our cloud. We believe this is where he was harvesting the email addresses of our customers.

Further steps we’ve taken to prevent any further such breaches:

  1. Enforced level 4 Password Policy within the organization.
  2. Limited Incorrect Logins to 3, after which there will be a cool down period for attempting to login again.
  3. A new feature is in works which triggers an email if a new session is created by a new from a different browser.

We would also like the community to learn from our mistakes and didn’t want them to face the same problems we faced, hence we have written a patch to enforce and enable better security standards by default in Frappe and ERPNext.

Going forward security will be something which will be taken very seriously at Frappe, rest assured we’ve taken the necessary steps to assure such a security incident does not occur again.

Finally, we would like to apologize again to the community for the security breach.

6 Likes
3

Enforce 2FA, ideally through a FIDO U2F token

Anything less and you’re still leaving yourself open to numerous targeted attacks.

5 Likes
4

Is it possible to have a message or some kind of forced pop up or screen which forces the user to change the password once the password policy is enabled or changed in the settings?

2FA is feature is excellent, I have come across firms / users not using it as the users have no email id or the firm has no SMS gateway.

Forcing them to change upon login is good feature to have

1 Like
5

So given that passwords are not really stored, only the hashes are, there’s no way to workback if a user’s password is weak or not just by looking at the hash. So just showing the popup to everyone, forcing all of them to change their password doesn’t seem like a good idea!

6

I understand I was not talking about analyzing existing password looking for a weak password.

What I intended was something like if an organization using ERPNext decide to have a strict password policy, which they did not have earlier and decide to use the password policy feature in system settings. At present the users can continue to use the software with their existing password nothing changes, most users wont even realize that a policy is in place unless they hit forgot password or try to change their password.

What I was suggesting was like how we have the option to “Logout from all devices while changing Password” while changing the password in my settings, we have an option to alert users upon the next login have a message, “password policy has changed please change the password” or force users like most web-based banking application to change the password upon login.

2 Likes
7

@bibinqcs sounds like a good security feature addon. Maybe start a new thread on the forum discussing it or open an issue on GitHub or you’re free to contribute also ! :smiley:

1 Like
8

Yea, Sure :smiley: