Custom App's JS is running even if the app is not installed

I have a custom app installed on site 1 and a second site with just ERPNext on the same bench. But when I run site 2 I see console logs from the custom app. And it seems the JS is loaded there