Hello @pstormarkG9SGY,
I hope this message finds you well. Selecting ports is currently a pain. If you want to delete a port, activate the delete command and then hold ctrl and left click where the port and the module outline meet. It's touchy but you'll be able to delete it after a few times.
I'll add this thread to the existing enhancement request about this issue.
Let me know if there's anything else I can do for you.
Best Regards,

Jorge Garcia
Product Support Specialist for Fusion 360 and EAGLE
Kudos are much appreciated if the information I have shared is helpful to you and/or others.
Did this resolve your issue? Please accept it "As a Solution" so others may benefit from it.