Yes, it seems to be pretty difficult to make custom reports work right now, atleast without any proper guide, or how/to examples.
Somehow after numerous trial/errors i got my custom FileTransmittal.rdlc file to work with my own custom UDP:s and pictures etc.
I am not sure what fixed it in the end, i added some items that the error was pointing to manually to the XML file (rdlc) and played around with the encoding, in the end it worked.
I selected the desired UDP:s and system properties using ReportTemplateAuthoringUtility.exe, and then opened the rdlc file with Microsoft Visual Studio Community 2015 (which has SQL Server data tools installed), i edited the texts, changed the picture etc. and added fields, deleted some fields, then saved it from Visual studio.
One strange thing about my custom reports is that it looks different depending on where it is run from in the Vault. For example if i run a FileTransmittal.rdlc from the "Pack and go"-command, it will work as expected, but when i run the same report .rdlc file from "File -> Report", or from "Find -> Report..." the same report will have some of its data fields resetted to the original value. Fields resetted are the report name right below the Autodesk logo, and the field names after that, (Generated by, Date, Source, Action, Destination).
I had translated those fields in my custom report, but the translated texts are visible only when i run the report through "pack and go"-command, but not when run through "File -> Report", or from "Find -> Report..."
I did not find anything in the actual .rdlc file that would overwrite those values, maybe these are somehow overwritten from the Vault when the report is run?