I finally got it. I had to get out of styles and actually Edit the Hole Note itself and uncheck the Part Units there. You would of thought that if I picked a style that had that unchecked that it would have worked, but apparently the Hole Note itself over ruled it.