Perhaps you should insert Custom Link field in the Timesheet of Department. When the User is entering Timesheet, he/she should also select Department it belongs to.
Once this information is available, you can restrict the Manager based on the value in the Department field.
For the Users, you can apply Owner permission on Timesheets.