Hi @Anonymous,
I'm glad you reported this to us - you've identified a valid and reproducible defect in Maya 2017 Update 3. And yes, this is a regression that isn't reproducible in Maya 2017 Update 2 or earlier. Thankfully there's an easy workaround solution for you in the meantime, while we fix this bug on our end.
And so what's the bug? Put simply, running a "Method: Legacy" Unfold operation after selecting all UVs for a UV shell will produce bad results. For some reason, Maya isn't recognizing that by selecting all UVs for a UV shell, you're essentially selecting the entire UV shell ... and therefore producing incorrect results while unfolding that selection. Again, this only happens for the legacy algorithm for Unfold, not the new Unfold3D method. However, if you select the UV shell in the UV Shell selection mask, you'll get the results you're looking for with the legacy unfold method. For example:
1. Create a poly cylinder primitive
2. Select the UV shell corresponding to the side of the cylinder
3. Project a planar mapping
4. Select an edge and Cut UVs
5. Now instead of selecting all the UVs while in UV selection mask, switch to the UV Shell selection mask (in the UV Toolkit or RMB marking menu) and select the UV shell
6. Unfold using Legacy method and all the settings you described in your screenshot above
RESULT: You get a proper horizontally-constrained unfold.
Again, this is a defect and not an intentional change in behavior. I've logged it on our end and we'll try and get a fix out in an upcoming release.
So yeah ... good catch! Thanks again and if you have any further questions or any other problems to report, let me know. And if the workaround works for you in the meantime, feel free to click "Accept as solution".
Trevor Adams
Maya Design Team Manager, Mudbox Designer, Maya Modeling Designer
Twitter | LinkedIn