Personally I try not to import libraries. I have created a few personal libraries divided by type of component and the only imported geometry in any of them is a step file for the 3D package if I was able to download one from the manufacturer, SnapEDA, or any of the other ECAD library sites. I'm not sure I truly understand the issue you're having because your approach isn't how I usually go about things. A screencast would make it easier to visualize the problems you're experiencing.
My general library process...
Once you have the footprint in your library. Right-click>"Create Package from footprint". Once in the package editor, you have to save the document first, but then you can navigate in the data panel to a 3D model of your component. Right-click the component and "Insert into current design".
From there if you need to change the appearance or color of the component, you can break the link to the original file and essentially create a localized copy of the model inside your 3D package file. Then save and click finish to add the package to your library.
After that, you can assemble a "device" in your library where you place the symbol, add the package, and connect the pins to the footprint to give you a complete library component.
Hope this helps.
Steve Ritter
Manufacturing Engineer
AutoCAD/Draftsight
Inventor/Solidworks
Fusion 360