I am not building and supplying the path to LoadFrom because the code that calls LoadFrom is in a third party library. Luckily in this case I have the source code that I can modify, so it is not a huge deal now that I understand what is going on.
This is why it threw me for such a loop, I did not even know about or ever used LoadFrom. If I was going to build something extensible that dynamically loaded assemblies I would try use MEF which may or may not have avoided all this.
My main concern is that everytime I update that library this same issue is going to come up if I can't convince them to change their code base. If I set the CurrentDirectory then this should allow me to upgrade the library as it is released without having to worry about changing this code every time.
The library is based around Entity Framework using a generic repository. The mapping files are for linking POCO objects to the SQL server data store.