I have quickly thrown together a similar scenario with my custom connection and a simple frame with the PFC in multiple orientation which is similar to what I require for my current structure.
You can see that the connection I have created gets imputed in incorrect orientations in a few places. I have read that this can be due to the method I initially use to lay down the members in the first place and that its often requires to make multiple custom connections of the same connection to cover all the orientation possibilities and I have tried that on my project and found I was up to 4 templates for the same connection and still getting the connection being places in space when being placed. It was also a very counter productive process as I not only needed to create multiple templates for the same connection but in order to find which of the many templates I had created will work for the orientation was a process of trial and error of placing them all in one by one, deleting and repeating until I find the right combination.
Very time consuming when there are many instances of the same connection. I also don't see why it should be so orientation dependent when you initially lay the member down, as orientation doesn't matter standard connections from the library are used.
Whether or not it makes a difference, the process I used to start the project was by bringing in a wireframe of the structure I created in Inventor over to Advance Steel then using the beam on line command to place all my members.