Verified Photomontage alignment using surveyed reference points

[scottm]

Hi everyone,

I'm new to 3DS Max workflow and I'm struggling with a model that I have to align to surveyed reference points in order to produce a verified photomontage in Max 2017.

The image will have an approximate field of view of 70-degrees. This will be made up of several individual images stitched together to make a panorama.

Question 1) Am I right in thinking that I will have to set up a camera and match its view to each frame/photo that will be needed to make the pano?

Question 2) I've set up a single camera at the exact location/elevation where I took my photographs. I'm trying to match that to a single photo and align reference points that have been surveyed that appear in the photo and that I have placed into my model (so that they appear in the camera viewport.

I am unsure how I move the reference points in order to get them correctly positioned over my photograph which is positioned on a plane.

The only way I can see is to move either the plane or the camera itself using the XYZ sliders. However I am certain I need to keep the plane perpendicular to the camera/target which limits any movement of the plane. I am then finding it very difficult to get all cross-hairs within 3DS Max over the corresponding points when keeping the plane perpendicular.

Am I missing something or is there a tool I don't know about that allows you to selects points and proportion a photograph to match them exactly?

Many thanks for your help,   Guy

[Alfred.DeFlaminis]

Hello @scottm and welcome to the community!

Boy... this is a tough one.  I have a bunch of questions for you so I can understand this a bit more.  I've made pano HDRI's in the past for reflections, but I always had regular backplates to use them on.  Rendering or comping into the pano wasn't something I've done before, so I'm a bit outside of my comfort zone with this question and I may need to learn some about it myself in order to help you.  (Full disclosure.)  So, first... a few questions. 

1) Which tool is creating those white lines?  Is it somehow the Perspective Match Utility?  If so, how did you get the red/green/blue to not be there? 

2) How did you decide that the bottom left/right lines should be where they are?  Would it make more sense to line them up with the glass roof on the other side of the rotary?   Same question for the lines pointing toward the cars... how did you decide that was the correct angle?  It seems like the problem is that all those reference points (grate in road, bar on railing, etc.) are not 90 degrees from eachother, they are all rotated at various angles in space which confuses the calculation.  (Based on what I've done in the past.)

3) What is the end goal?  Or, what does the end result look and act like? 

Thanks for answering those questions and best regards,

[scottm]

Hi Alfred,

Thanks for your response,

I'll answer your questions in turn;

1) The white cross-hairs were produced in Autocad by the surveyors as part of the data supplied after their site survey. They depict the reference points I proposed to them in the top panorama with the small circles. In the Autocad dwg, they're in 3D so possess X,Y and Z details for the exact locations (the X and Y being British National Grid 6-figure grid-references in metres and the Z being the height in metres AOD - Above Ordnance Datum). These were brought into 3DS Max within a single dwg that had the reference points and camera location plotted. Once in 3DSMax I rotated them through 90-degrees so they appeared as much as possible square-on to the camera in order that I could see them and hopefully align them to the corresponding feature in the photo

The two closest cross-hairs appear larger and intersect with each other which is a little confusing.

2) Perspective Match wasn't used here. As mentioned the white lines are a little confusing. It's more the centre/intersection of the cross-hairs that I'm using. It just seems that once brought into 3DS Max, the ones that are close to the camera appear larger and extend far more than required.

The idea is that my model (which is a proposed multi-storey building yet to be constructed and to be positioned behind the church tower) is accurately positioned within the 3D space, based upon the positions dictated by the architects plans. Once that is achieved I then need to produce a photomontage made up of 3-10 individual photographs stitched together into a 70  - 180degree panorama with the model correctly positioned within the centre of the panorama.

I hoped that if I plotted the surveyed reference points in 3DS Max, I could then move the photograph in relation to the 3D reference points and get them all to sit 'on top' of their correct position within the photograph. Once achieved this would be verifiable proof that the model is located accurately within the photograph in terms of both position and scale.

My problem how I see it is two-fold. Firstly, I need to align my model against the photograph points and secondly, because my final stitched pano is made up of several individual photos I believe I have to render out an equal number of camera views that match the individual photos (i.e. the camera/lens combo that took the photos has a HFoV of 39.6degree so I render out as many 39.6degree views as needed to make up my desired HFoV) - Is this correct?

Hope this help clarify what i'm trying to achieve Alfred.

Many thanks once again.

[Alfred.DeFlaminis]

Hello @scottm,

I see, thank you for the detailed explanation.  I'm still feeling this out, I've done lots of viz renders but never like this.  Doesn't mean that it's wrong or anything just new/different.  

1) I wonder if the Camera Match Utility would be good here instead of the perspective match.  It deals with points as you are currently using and may be more relevant. Here is a link.  I would use this on unstitched BG images. That should make sure your camera is aligned properly to each image at the very least, unless there is significant lens distortion.  Which renderer are you using?  Was each photo taken from the exact same position?  Did you get a full 360 degree set of photos, or just the 70-180 degrees? 

2) Some pano images are just wide screen images (photoshop photomerge) while in 3D a pano usually has a spherical warping. Example here.  Which are you after?  I'd think the type of pano you are intending to end up with would have a bearing on your camera setup.  If you are going after the photoshop 'wide image' version, you really only need 1 render.  If you are going after the spherical, you could still get by with 1 render but it would be great if you had the entire 360 view to use for the backplate.  (I think... this is still conceptual for me.) 

Best Regards,

[dbhinnant]

You can try the camera match utility as Alfred suggests.  In my experience this process is hit or miss.  It will often give you a good starting point but manual camera adjustments are usually necessary.  This is especially true in panoramic photos.  I have always worked with the stitched pano and never tried the individual images.

The reference points you have chosen in the posted image are probably a bit too concentrated in the right side of the image.  The tool works best when you have points distributed in all areas of the photo.  A point or two on the left would help.  Optimally you want to have points in the foreground, background, left, right and at least one for height.

After you place your cam point helpers on your surveyed reference points you will use the camera match utility to pick the points in your background image.  This rarely give me a perfect match.  What I do is experiment with each point picking in the background a bit further away in the direction I need to go.  (Hope that makes sense?)  In the end I try to split the difference between all points to get the overall match as close as I can.

Often it is necessary to alternately select the camera and the target and tweak them manually to get a better match.

Hope this helps.  Good luck.

[Alfred.DeFlaminis]

Hello @scottm,

I just wanted to follow up here, any progress on your issue?

Best Regards,

[scottm]

Hi Alfred & dbhinnant,

Many thanks for your replies.

Sorry for the delay getting back. I was out on site again yesterday.

I'm going to have a go with a stitched panorama and Camera Match utility today so will let you know how I get on

[scottm]

Afternoon gentlemen,

I've tried, unsuccessfully, several times this afternoon to try and use the Camera Match facility using the instructions contained within Alfred's link.

I'm getting an error message that doesn't really help/guide me as to what is wrong.

The top image shows my 120degree panorama set as the background. The image is 17850px by 5804px

I created cam points using the Keyboard Entry Rollout so that I could enter the exact XYZ positions supplied by our surveyors.

My first problem was that sometimes when I came to 'Camera Match' the small red crosses wouldn't appear when I clicked on the photo where the cam point should be. Sometimes the crosses would be there, other times they wouldn't.

My next problem is that because the background image is fitting to the viewport window (i.e. I can't seem to be able to zoom in on features within it), I don't think the cam points are where they should be. Is there any way to zoom in on the background (such as in the lower image), when you are positioning your cam points? If I could do this I could shrink the image size down, which I don't think is helping my cause.

Thank you once again for any help.

[Alfred.DeFlaminis]

Hello @scottm, thanks for the update here!  I believe the error message is due to point placement, so before we look into that I wanted to make a small video addressing the other things in your post.  I think it will help and possibly alleviate the error message in the process.  Please let me know if you have any questions, and thank you.  Have a great weekend.  

Best Regards,

[scottm]

Hi Alfred,

Thanks very much for that. The 2D Pan/Zoom feature is just what I was after.

Not sure if I'm doing anything wrong but when I zoom in and place my points for each of the locations in the 'Camera Match' utility, they 'float' when I pan and zoom about the screen (i.e. they don't remain in the locations on top of the photo features I place them on - see below video which is zoomed in on the conical top of the cylindrical structure in the foreground left hand side of my pano).

Surely this would mean that when I pan and zoom about the screen and place my other points they'll not be spaced where they should be in relation to the each other and the photo image?

[Alfred.DeFlaminis]

Hello @scottm,

I see what you mean.  The CamPoints can be initially placed with 2d Pan and Zoom mode (in 3d space), but the red/green dots specifically seem to use screen space so zooming does cause an issue there.  My suggestion is to use 2D pan and Zoom for the initial points (so you know they are perfect) and then use the full unzoomed non-2D Pan and Zoom mode for the "use this point" ticks.  (Red/Green.)  Since they use spinners for placement it's a bit easier to place those red/green dots without the zoom in potential.  

Best Regards,

[scottm]

Hi Alfred,

In my cases I know the initial points are perfect as they are gathered using a Leica Total Station so have accuracy down to a few mm. I just manually enter the co-ordinates supplied by the surveyors using the keyboard entry.

It's then when I need to pick that point out on the photo that I need a zooming feature so that I can place the corresponding mark with pin-point accuracy (assuming the software needs it to be pin-point??).

Either way, I'm still getting the errors.

Do we know what it causing that?

Thank you.

[Alfred.DeFlaminis]

Hello @scottm,

I see.  The red and green crosses just need to be close.  They are used to match the 2d image to the 3d points to map out the camera distortion or lens barreling.  I've never used it on a pano before and I suspect it may be because the image is on it's way to becoming a spherical pano that it is failing to compute the matrix. (I know it would confuse the perspective match tool.)  I wonder, would it be helpful if I were to create a private folder for you to put your file and background images in to do some testing?  I don't need your model/building, just the scene and the background images.  (NDA safe.)

I've done this with stills before but never a panoramic style image so it may be that it's too confusing for the camera match.  I'm in experimental territory with the stitched images, I've not used this process before.  I wonder if @dbhinnant has any suggestions on this process with the pre-stitched images for the bg.  

If it was a full 360 pano it could be converted to a cross or something along those lines but with a partial pano it's a tough scenario.  That's why I was thinking doing a match to each image early on.  I think what I might try is to only use the part of the background that the render will appear in, which should minimize the spherical distortion and then comp the render into the 180 pano image at the end.  Again, I'm a bit outside of what I have done in the past so I'm mostly talking theoretical.  

I supposed that another approach would be to work on the unstitched single background image, get the render into it and comped, and then use that with the other images to create your pano at the end.  This would allow you to work on a mostly flat background but still give you the end result you are looking for.  (Am I making sense?) 

Best Regards,

[dbhinnant]

The best advice I can give is to forget about getting it perfect.  That's not going to happen.  Especially with panoramas.  Attached is a two image pano showing the cam points.  As you can see they are not perfect but are all pretty close.  The one on the far right is off quite a bit but you have to make compromises.

The most important thing is to have points distributed throughout the image.  When I get that error I try to use different points.  Start out with only 5.  4 that define the four corners of the ground plane and one that defines height.  If those points don't work, you can substitute points one at a time until you can create a camera.  Once you create a camera you can select it and modify the match by moving or adding points.

I only use the 2D pan and zoom to check a match as it's often hard to tell if things in the distance are lined up.

[Alfred.DeFlaminis]

I wonder if the pano stitch method is part of the problem here.  @dbhinnant seems to have a photoshop style pano that isn't spherically warped, but the image @scottm posted has some of the spherical warp in it.  I'd imagine that since the Max camera doesn't have a 360 degree lens that it is confused.  Even a macro 180 lens might cause problems.   Just sort of thinking out loud... 

Best Regards,

[dbhinnant]

My image does have some distortion.  Notice the curvature of the road in the foreground while the surveyed line is straight.  The distortion in the OP's image is much worse which will cause issues.  Maybe try splitting it into two matches?

[Alfred.DeFlaminis]

You're right, it's more subtle but it is there.  

Best Regards,

[scottm]

Morning Gentlemen,

Thank you for your responses and sorry for not being back sooner. I was out on site again last week so away from an internet connection.

My original image is probably not the best example to be fair as the images were taken in a portrait orientation which will increase the errors (more images being stitched).

Perhaps rather than using the surveyed points, I'll mark my own points on the image so that these can be moved if the match fails.

I'm happy to send over files if that helps Alfred but it looks as though dbhinnant has done this a lot more than me and has a system in place that should help.

I'll give it a go today on some recent landscape orientation images I captured last week.

[Alfred.DeFlaminis]

Hello @scottm,

Thanks for the update!  I'm not trying to pressure you or anything, just want you to know I am happy to take a look if you'd like.  Let's see how your test goes.  This is a usage case that I am not familiar with so there is a bit of touch-and-go with it, but I am curious.  Thanks for keeping me posted.

Best Regards,

[scottm]

Hmmm. Had a think about this.

The only problem I can foresee with dbhinnant's method is that you have to know the 3D position of the points you try before you can replace them or alter them if the error prevents you 'from passing go'.

From the images I photograph, I identify in advance the cam/reference points and these are sent to the surveyors who also then attend site and gather the precise grid reference locations during a site survey. If 3DS Max doesn't accept these (which it isn't so far)  then I have no accurate way of identifying further points from my scene apart from hoping features exist in the view that appear on my base mapping (meaning I can get their XYZ location from CAD/Civils). A bit hit & miss.

Due to time/cost constraints on my workflow my only option appears to be to place objects (cylinders/cubes) in my scene on known features (corners of buildings or telegraph poles etc) and manually place, then align the camera to them as best I can.