You want to create a hole in a viewport either to mask the area underneath or to use a second viewport to show a detail.
To do this, create two polylines. One should represent the overall viewport and one the area you want to cut out. If you want to show a detail in the masked area, make a copy of the polyline you're using for the interior viewport. Use the REGION command to create a region out of the lager polyline and the copy of the smaller one. Use the SUBTRACT command to subtract the smaller region from the larger one. You will now have a region with a hole in it.
Create a viewport from the resultant region. It will also have a hole in it. To show a detail in the area of the hole, create a viewport from the remaining polyline. That will fill in the hole with a new viewport.
If you want the cut out area to be circular, create a polygon with 20 to 50 sides, depending on how smooth you want it to look. You can use the resultant polyline to create the hole in the region and a copy as a viewport. The "circle" may not have to be too smoth if you're going to cover it with a circular border