The easiest way to see what is causing the issue is to switch from object mode rotation to gimbal mode. You'll see that even though the x object axis is aligned to the direction you want to roll, the "real" axis is probably still aligned to the world.
So the solution will depend on what kind of rig your ball has. If it has a control that lets you move the whole rig at once then you can rotate that control so the rolling control is aligned with diagonal you want to roll on. I'm assuming your ball rig works the same as the ones I've seen before but if that answer doesn't make sense could you post a picture of the rig so I can see what controls you have to work with?
