So, you have a drawing with 1 unit = 1 mm and you are drawing "full size" in model space. You make a layout view current and add a viewport, and you can see your drawing initially, but then you change the scale of the viewport and the drawing disappears. Does everything disappear (Walls, Doors, Windows, and notes/annotation) or does just the annotation disappear?
If the latter, did you set the drawing scale to 1:100 before creating the notes and annotation? If not, and if you are using annotative content, and the viewport is set to display the current scale only, then the annotation will not be visible. You will want to add the 1:100 scale to the annotative content that you want to see in the viewport.