Disable automatic contact adding


We have some contacts linked to our customers to store a contact that shall receive invoices etc. Unfortunately this contact gets added pretty much everywhere for no reason. How can this be disabled, so accountant’s name isn’t plastered everywhere over invoices, delivery notes etc.?