Side info: When you open a file with the specify worksets option and change your mind (ie you do not want to open the file anymore) you cannot cancel the operation as the cancel button suggests but you simply cancel the workset selection and continue with none at all. I admit I was annoyed with this but now often use this bug feature to check on files that will not otherwise open any more.
I figured that even if I choose not to load even a single workset, revit still checks every single linked file and even complains about dwgs having too large coordinates or an entity it cannot handle when I explicitly did not want this loaded.
So far every revit model I linked into others loaded all worksets that were set to 'visible in all views' and not only the last one used. I will pay more attention to see whether this is not the case. If I run into this again, I will remember this idea/bug report.