"Almost always, I use a different plot size for each layout."
An architect just can't work that way. This newsgroup is about AutoCAD Architecture. Architectural projects must be documented as many different views which are distributed in the field for construction as a single printed set. Having different sized sheets bound together would be a mess.
That said, you should create a sheet template file with a layout for each size and plot setting you want to use regularly. When creating a new layout, use "from template". It doesn't make sense, however, to use the same layout for several sheet sizes. Only one size will be to scale. Other sizes will end up not to scale. This also doesn't make much sense as plots should generally be made to pdf, period. Then, the PDF file could be print however it is needed to a physical printer, after being checked.
The pagesetup command is much more versatile than you understand. See the image below:

Architect, Registered NC, VA, SC, & GA.