I believe the solution to this problem is quite simple based upon my understanding of the constraints.
Constraints:
1. Must be the maximum area rectangle.
2. The rectangle must be within the white polyline.
These constraints are quite simple.
Now comes the assumptions on the problem statement. Hopefully my assumptions are valid and correct. After looking at the examples provided, it appears that at a minimum, two opposing corners are always touching the polyline. Additionally, it appears that the maximum rectangle always covers the centroid of the polyline.
Assumptions
1. Two opposing corners are always touching the polyline.
2. No padding has been applied to the polyline (this wouldn't complicate much, ignored for simplicity).
3. The rectangle encompasses the centroid.
Code for this can be found here.
Lee Mac's polygon centroid
Lee Mac's geometric functions
A central limitation to the polygon centroid is that it requires straight lines for the polyline. I believe this can be modified to accommodate other types of segments like arcs.
If you have a polyline with arc segments the following code can be used to create information on this polyline:
Lee Mac's polyinfo
It has the code you will need to create a list of points out of the polyline.
Now, for the algorithm:
1. Convert the polyline into a series of points in a list.
a) The points are generated based on a precision parameter e.g. 5000 points
2. Use another precision parameter for the rotation of the rectangle e.g. 360 (1 degree precision)
3. Three nested loops
a) This is not required as it will create duplicate calculations, and you can create a workaround for a reduction in the big O. The nested point loops ignoring the rotation loop is O(n^2). Where as a clever form of these loops is O(n).
(foreach point1 pointList
(foreach point2 pointList
(repeat 360
; process here
)
)
)
4. Calculate the rectangle area, check the following:
a) is the calculated rectangle within the white line? use the centroid as a reference point. This may require a function that does more loops to verify if a line on the rectangle does not intersect the polyline, but does not count the end points of the rectangle onto the polyline.
b) If the rectangle area is larger than the maximum found area, then store the points in variables.
After this process, you're done!
You should have the maximum found area within the precision parameters you have listed.