Stereo Camera - With Head Tracking...?

[darawork]

Hi,

I downloaded the iRay+ 90 Day Trial today from NVidia, It adds a new rendering Engine iRay+ in 3DS MAX (in addition to the build-in iRay engine that comes with recent MAX Versions).

With iRay+ the user can change any selected camera rendering view from 'Planar' to 'Spherical' or 'Cylindrical'. In addition to this the user can also check the box for 'Stereo' L / R renderings (L / R renderings render separately, not at the same time unfortunately).

So with 'Spherical' you get a 360 Degree rendering that can be used in Google Cardboard, or Click-Drag-To-Rotate images.

And with 'Stereo' renderings you get perceptive 'Depth' in a still image.

I'm wondering if there is any way to combine the two types, into a Stereoscopic 360 image with depth AND head tracking.

Any time I look on Google for the answer I end up down a rabbit hole of overly complex, expensive and lengthy answers.

Is there an easier way to achieve this effect with a cheap VR viewer like Google Cardboard, and a limited knowledge of everything technical?

[darawork]

Hi,

I downloaded the iRay+ 90 Day Trial today from NVidia, It adds a new rendering Engine iRay+ in 3DS MAX (in addition to the build-in iRay engine that comes with recent MAX Versions).

With iRay+ the user can change any selected camera rendering view from 'Planar' to 'Spherical' or 'Cylindrical'. In addition to this the user can also check the box for 'Stereo' L / R renderings (L / R renderings render separately, not at the same time unfortunately).

So with 'Spherical' you get a 360 Degree rendering that can be used in Google Cardboard, or Click-Drag-To-Rotate images.

And with 'Stereo' renderings you get perceptive 'Depth' in a still image.

I'm wondering if there is any way to combine the two types, into a Stereoscopic 360 image with depth AND head tracking.

Any time I look on Google for the answer I end up down a rabbit hole of overly complex, expensive and lengthy answers.

Is there an easier way to achieve this effect with a cheap VR viewer like Google Cardboard, and a limited knowledge of everything technical?

[spacefrog_]

You have to render with equirectangular projection for the left and the right eye. If iRay does'nt provide this projection type you wan't have much luck.( There's also cubic projection that can be used but i leave that out here )

If you would combine the left and the right eye standard spherical/cylindrical images and display them with your cardboard VR viewer, you will experience some 3d depth while looking in the exact scene camera direction, as only there the eye separation is kept intact. The further away the image pixels are from that exact axis ( usually the image center ) the less eye separation is maintained, thus the less the 3d depth effect. This even goes as far as reversing left<>right eye  when looking in the opposite direction ( ala looking behind, in the opposite of your original scene camera's look direction)

Essentially the equirectangular projection renders the two cameras as if they were mounted on a rig which rotates around the common center vs. rotating each camera individually in the standard sphericall/cylindrcal projection. It takes some thinking to apprehend this, but in the end it's quite logical

Here's some more docs, but you'll find more on google

http://paulbourke.net/stereographics/stereopanoramic/

IF you use Mental Ray, you can use DomeMaster3d which provides a MR camera shader with latLong ( = equirectangular ) stereo projection and use Over/Under arrangement ( Left eye pic top half of the rendered image, right eye bottom half ), latest Corona and Vray have that projection modes + the arrangement in one image already built in . So i would guess that iRay+ should have it too, but what do i know ?

BTW: just in case , here is a quick test ( Sponza atrium ) quickly rendered with MR and using the DomeMaster3d camera shader

[spacefrog_]

You have to render with equirectangular projection for the left and the right eye. If iRay does'nt provide this projection type you wan't have much luck.( There's also cubic projection that can be used but i leave that out here )

If you would combine the left and the right eye standard spherical/cylindrical images and display them with your cardboard VR viewer, you will experience some 3d depth while looking in the exact scene camera direction, as only there the eye separation is kept intact. The further away the image pixels are from that exact axis ( usually the image center ) the less eye separation is maintained, thus the less the 3d depth effect. This even goes as far as reversing left<>right eye  when looking in the opposite direction ( ala looking behind, in the opposite of your original scene camera's look direction)

Essentially the equirectangular projection renders the two cameras as if they were mounted on a rig which rotates around the common center vs. rotating each camera individually in the standard sphericall/cylindrcal projection. It takes some thinking to apprehend this, but in the end it's quite logical

Here's some more docs, but you'll find more on google

http://paulbourke.net/stereographics/stereopanoramic/

IF you use Mental Ray, you can use DomeMaster3d which provides a MR camera shader with latLong ( = equirectangular ) stereo projection and use Over/Under arrangement ( Left eye pic top half of the rendered image, right eye bottom half ), latest Corona and Vray have that projection modes + the arrangement in one image already built in . So i would guess that iRay+ should have it too, but what do i know ?

BTW: just in case , here is a quick test ( Sponza atrium ) quickly rendered with MR and using the DomeMaster3d camera shader

[spacefrog_]

BTW:

how do i get the iRay+ demo ? i entered my email and now they talk about getting in touch with me to discuss my needs ? Could they make this even more cumversome ? I just want to test the iRay+ renderer ....

Edit:

Ahh i think iRay+ does'nt exist even any longer ( Iray+ was the Lightworks name i assume) , so i found the trial download link here

http://www.nvidia.com/object/iray-plugin-trial.html

Currently downloading the Beta from Nvidia so i have to wait till the DL finished and i was able to test .

[spacefrog_]

BTW:

how do i get the iRay+ demo ? i entered my email and now they talk about getting in touch with me to discuss my needs ? Could they make this even more cumversome ? I just want to test the iRay+ renderer ....

Edit:

Ahh i think iRay+ does'nt exist even any longer ( Iray+ was the Lightworks name i assume) , so i found the trial download link here

http://www.nvidia.com/object/iray-plugin-trial.html

Currently downloading the Beta from Nvidia so i have to wait till the DL finished and i was able to test .

[spacefrog_]

Okay - after some brief tests i can confirm that iRay+ perfectly supports the equirectangular projection + the automatic over/under or left/right arrangement into one image. ( So you don't even have to juggle with stereo cameras )

Simply enable "Spherical" lens and "stereo" with "both eyes above/below" arrangment you will get an image with left and right eye in over/under arrangement like my example above. Those can be loaded in a cardboard VR image viewer which supports that format

There is also cylindrical projection supported, which does'nt waste as much pixels on the north/south poles like the spherical , but again the viewer should support this.

Here's the output of this quick iRay test ( the sponza scene now is   textured and with additional object ). I hope you don't mind those off colors ...

[spacefrog_]

Okay - after some brief tests i can confirm that iRay+ perfectly supports the equirectangular projection + the automatic over/under or left/right arrangement into one image. ( So you don't even have to juggle with stereo cameras )

Simply enable "Spherical" lens and "stereo" with "both eyes above/below" arrangment you will get an image with left and right eye in over/under arrangement like my example above. Those can be loaded in a cardboard VR image viewer which supports that format

There is also cylindrical projection supported, which does'nt waste as much pixels on the north/south poles like the spherical , but again the viewer should support this.

Here's the output of this quick iRay test ( the sponza scene now is   textured and with additional object ). I hope you don't mind those off colors ...

[spacefrog_]

And here's a small cardboard Pano test project with some of my rendered images ( as i just develop a Pano viewer so the whole topic hits right home with me ). The project is developed in Unity , you can click through the various pano spheres with your card board magnetic button ( hope it has one )

Simply copy the apk onto your phone's sdcard and you should be ready to install it from there

PanoSphere_Tests.apk for Google Cardboard

The first Pano Image is the iRay rendered Sponza szene, which is pretty bad quality ( because of the time restrictions ) and this hurts the viewing experience pretty badly. The otehr Panos are of better quality ... ( again simply click using the cardboard touch/magnet button )

[spacefrog_]

And here's a small cardboard Pano test project with some of my rendered images ( as i just develop a Pano viewer so the whole topic hits right home with me ). The project is developed in Unity , you can click through the various pano spheres with your card board magnetic button ( hope it has one )

Simply copy the apk onto your phone's sdcard and you should be ready to install it from there

PanoSphere_Tests.apk for Google Cardboard

The first Pano Image is the iRay rendered Sponza szene, which is pretty bad quality ( because of the time restrictions ) and this hurts the viewing experience pretty badly. The otehr Panos are of better quality ... ( again simply click using the cardboard touch/magnet button )

[darawork]

Hi,

Thanks for all the help, and the link to the .apk

"Those can be loaded in a cardboard VR image viewer which supports that format"... yeah, I was looking for one of these viewers. I can make the equirectangular renders alright (In iRay and MR panorama export, and with the iRay plugin). I was just having trouble getting my head around viewing both L/R equirectangular renderings at the same time. One is easy, but I couldn't work out two. Here are a couple of single image 360s:

https://vrapp.co/darawork/tour 

[darawork]

Hi,

Thanks for all the help, and the link to the .apk

"Those can be loaded in a cardboard VR image viewer which supports that format"... yeah, I was looking for one of these viewers. I can make the equirectangular renders alright (In iRay and MR panorama export, and with the iRay plugin). I was just having trouble getting my head around viewing both L/R equirectangular renderings at the same time. One is easy, but I couldn't work out two. Here are a couple of single image 360s:

https://vrapp.co/darawork/tour 

[spacefrog_]

Hey - VRApp looks good ( and your panos too )

Here's another great viewer for Android: this one can read images right from your phone gallery/SD card

There's a whole online portal behind this, which allows to create whole Pano tours ( currently in beta )

But the viewer alone works without any registration etc...

The Andrroid Viewer app

https://play.google.com/store/apps/details?id=co.theConstruct.theViewer.CBD

The Online Portal + Tour editor etc.. ( after beta registration )

http://theconstruct.co/

[spacefrog_]

Hey - VRApp looks good ( and your panos too )

Here's another great viewer for Android: this one can read images right from your phone gallery/SD card

There's a whole online portal behind this, which allows to create whole Pano tours ( currently in beta )

But the viewer alone works without any registration etc...

The Andrroid Viewer app

https://play.google.com/store/apps/details?id=co.theConstruct.theViewer.CBD

The Online Portal + Tour editor etc.. ( after beta registration )

http://theconstruct.co/

[darawork]

I installed the .apk, very cool. This is the exact sort of app I'm looking for. I have a set of L/R Equirectangular renderings I've made here that I'd like to load up into an app like this. Are you going to make it commercially available? Or is there something similar I could use?

:edit: Oh, cool, I'll try that link. Thanks 🙂

[darawork]

I installed the .apk, very cool. This is the exact sort of app I'm looking for. I have a set of L/R Equirectangular renderings I've made here that I'd like to load up into an app like this. Are you going to make it commercially available? Or is there something similar I could use?

:edit: Oh, cool, I'll try that link. Thanks 🙂

[darawork]

Spacefrog, you mentioned "Both eyes Above/Below". Does this appear on your iRay plugin dialogue box? I can't find it on mine. I'm using "Iray for 3ds Max - BETA 1.3.0-116812 (3ds Max 2015/2016/2017)"

Is that the one you're using too? I'm trying it out in MAX 2016.

[darawork]

Spacefrog, you mentioned "Both eyes Above/Below". Does this appear on your iRay plugin dialogue box? I can't find it on mine. I'm using "Iray for 3ds Max - BETA 1.3.0-116812 (3ds Max 2015/2016/2017)"

Is that the one you're using too? I'm trying it out in MAX 2016.

[spacefrog_]

Yes i use that version too ( though the file name i downloaded reads IFMAX_2017-Beta-1.3.0-116812.zip for some reason)

I got that version from the Nvidia Advanced rendering forum, and it was released in the iRay for 3ds Max ->iray for 3ds Max - Beta 1.3.0 (3ds Max 2017/2016/2015) with Pascal GPU support top sticky thread ( on 5th. nov )

I'm on Max 2017 an my rollouts clearly show the Both Eyes options

[spacefrog_]

Yes i use that version too ( though the file name i downloaded reads IFMAX_2017-Beta-1.3.0-116812.zip for some reason)

I got that version from the Nvidia Advanced rendering forum, and it was released in the iRay for 3ds Max ->iray for 3ds Max - Beta 1.3.0 (3ds Max 2017/2016/2015) with Pascal GPU support top sticky thread ( on 5th. nov )

I'm on Max 2017 an my rollouts clearly show the Both Eyes options

[darawork]

Ah, I've got 1.2.115234

They must have added that in, since a few days ago (either that or it's a MAX 2017 exclusive setting).

I've not been using MAX2017 much, the scene conversion is terrible for old scenes,
Something the iRay plugin seems to have gotten spot on.

[darawork]

Ah, I've got 1.2.115234

They must have added that in, since a few days ago (either that or it's a MAX 2017 exclusive setting).

I've not been using MAX2017 much, the scene conversion is terrible for old scenes,
Something the iRay plugin seems to have gotten spot on.

[darawork]

Hi again,

one last question... what resolution do you normally do your spherical renderings at?

[darawork]

Hi again,

one last question... what resolution do you normally do your spherical renderings at?

[spacefrog_]

"Normally" is relative: i just jumped onto the whole VR train just a few weeks ago, and mainly did  test renderings until now

For the Cardboard 4096x4096 combined works pretty great on my phone - despite it's age  ( with the top/bottom arrangement ( so 4096x2048 per eye )

I'm not sure how the average current android phone would handle textures larger than that ( would be 8192 as next step)

Panos using the GearVR cubemap layout and targeting the GearVR can have up to 18432×1536, which is insane horizontal resolution, but i thing the phone splits that into separated textures internally. If you target the Vive or Oculus, i bet you would be able to use 16k x 16k textures ...

[spacefrog_]

"Normally" is relative: i just jumped onto the whole VR train just a few weeks ago, and mainly did  test renderings until now

For the Cardboard 4096x4096 combined works pretty great on my phone - despite it's age  ( with the top/bottom arrangement ( so 4096x2048 per eye )

I'm not sure how the average current android phone would handle textures larger than that ( would be 8192 as next step)

Panos using the GearVR cubemap layout and targeting the GearVR can have up to 18432×1536, which is insane horizontal resolution, but i thing the phone splits that into separated textures internally. If you target the Vive or Oculus, i bet you would be able to use 16k x 16k textures ...