Did you leverage the ownership permission as well.
Turning off the "View Records Owned By Others" permission is a very powerful way to limit access to only a select set of records. Then you can either leverage the "owner" or "additional owners" settings at the bottom of the item details tab to control exactly what they see.
And as you said - change visibility of tabs and even hide sections on the item details tab!
Director of Product Lifecycle Management
Some more info on the bit Brian mentioned about restricting access to a certain section can be found here - might come in handy for other purposes as you already a separate group set up.
Welcome to the forum!
PLM Product Manager