Inventor might not "care" about non-clearance contact, but in the real world there must be allowance for manufacturing tolerances.
The largest diameter must be smaller than the smallest slot when accounting for normal manufacturing tolerance. The side of the cylinder touching might not be as critical depending on the rest of the design.
It is good practice to design up-front with the necessary clearances in mind.
It is easier to manufacture cylinders than it is to manufacture "slots", therefore I recommend making the cylindrical feature smaller with tight tolerance and the slot larger with loose tolerance. Tight and loose tolerance are relative terms - important considerations of manufacturing process and cost (and of course - the function of the parts in the assembly). In general, use the loosest tolerances possible with appropriate dimensions for clearance that will result in an assembly that functions as needed.

Now that I have experimented with your assembly - I see why you offset this to one side.

-----------------------------------------------------------------------------------------
Autodesk Inventor 2019 Certified Professional
Autodesk AutoCAD 2013 Certified Professional
Certified SolidWorks Professional