Auto TX - tx files naming including colorspace etc.

[am_wilkins]

Hello,

I know there has been some recent changes to "Auto_TX" however I'm seeing some slow-downs likely related to texture processing. (as we've experienced in the past)

In attempting to pin-point the issue—noticed that the "Auto_TX" default name is very long, contains two extensions and I was wondering if the "Use Existing TX" feature will still correctly detect the TX file?

Example:
TextureName_BaseColor.1001.exr

Get's a tx file (via auto tx) called:
TextureName_BaseColor.1001_ACES - ACEScg_ACEScg.exr.tx


Regards,
Amwilkins

--

Houdini Core Version 19.5.534
Arnold Core: 7.2.1.0

HotA: 6.2.1.0

CPU: AMD Ryzen TR 3990x

RAM: 128GB
GPU: NVIDIA RTX A4000
OS: Windows

[Stephen.Blair]

Use Existing TX will still correctly find the tx file.

Auto tx and Use Existing Tx are now handled by Arnold itself, not by the plugin.

[am_wilkins]

Okay cool, thanks for the info.

I wanted to just directly path the test scene to the TX files (python script swapping .exr to .tx), but this extra naming convention makes this considerably more tricky.
ie. first thought was this issue again.
https://forums.autodesk.com/t5/arnold-for-houdini-forum/logs-textureresolveuseexistingtx/m-p/1104523...

Anyways, will continue to pin-point the IPR rendering slow-downs (only 1-2 threads being used and textures taking forever to load into memory) and post a new forum topic if issue is on-going.

[thiago.ize]

Is this slowdown due to Arnold generating the new tx files on your very first render (this is expected) or it happens on every render even once .tx files already exist (shouldn't happen)?

As usual, log file with stats at the end might show what is happening.

[am_wilkins]

Hi Thiago,

No, the TX files are pre-generated and exist already on the server. So Arnold definitely isn't re-creating them.

I believe the issue is related to that older post regarding the "TextureResolveUseExistingTx " taking an extensive chunk of the time.

In a new log file I've just generated, this is from a smaller selection of assets in my Solaris test scene, I'm seeing the same slow-down with the TextureResolveUseExistingTx.

00:01:10 2031MB | -----------------------------------------------------------------------------------
00:01:10 2031MB | top session self-times by category
00:01:10 2031MB | TextureResolveUseExistingTx 0:19.94 (32.47%)
00:01:10 2031MB | thread blocked 0:17.13 (27.90%)
00:01:10 2031MB | BVH::intersect 0:05.61 ( 9.14%)
00:01:10 2031MB | root 0:02.51 ( 4.10%)

I won't post this whole section, but it's listing many individual textures and the same, TX issue below that.
(I've chanted the texture name to be generic)
00:01:10 2031MB | -----------------------------------------------------------------------------------
00:01:10 2031MB | top session self-times by node
00:01:10 2031MB | image:/GroundA_mtl/image2 0:06.66 (10.85%)
00:01:10 2031MB | TextureResolveUseExistingTx 0:03.03 ( 4.95%)
00:01:10 2031MB | thread blocked 0:03.03 ( 4.94%)
00:01:10 2031MB | AiTextureHandleAccess 0:00.49 ( 0.80%)
00:01:10 2031MB | image:/GroundA_mtl/image3 0:06.62 (10.79%)
00:01:10 2031MB | TextureResolveUseExistingTx 0:03.28 ( 5.35%)
00:01:10 2031MB | thread blocked 0:02.73 ( 4.46%)
00:01:10 2031MB | AiTextureHandleAccess 0:00.51 ( 0.83%)
00:01:10 2031MB | image:/GroundA_mtl/image1 0:06.61 (10.78%)
00:01:10 2031MB | TextureResolveUseExistingTx 0:03.19 ( 5.21%)
00:01:10 2031MB | thread blocked 0:02.82 ( 4.60%)
00:01:10 2031MB | AiTextureHandleAccess 0:00.51 ( 0.84%)
00:01:10 2031MB | standard_surface:/GroundA_sts 0:03.12 ( 5.09%)
00:01:10 2031MB | surface closure 0:00.98 ( 1.60%)
00:01:10 2031MB | AiLightsTrace 0:00.96 ( 1.57%)
00:01:10 2031MB | ray traversal+intersection 0:00.45 ( 0.74%)
00:01:10 2031MB | image:/Roof_mtl/image3 0:02.95 ( 4.81%)
00:01:10 2031MB | TextureResolveUseExistingTx 0:01.64 ( 2.67%)


I can provide more testing or logs tomorrow if required, but I'm pretty as before not using the "Use Existing TX" feature and pathing straight to the TX files could likely solve the issue?

-
EDIT:

I ran a test where I manually pathed some of the textures directly to the .tx file and am seeing a big speed-up already.

Is there anyway to prevent this slow-down?

We're talking x3 texture maps with x3 UDIM each, all being under 15mb or less. (For that ground texture in above logs) I'm not expecting this massive increase on texture processing times using the native/default Arnold expected workflow. ie. path to EXR files and Arnold does the rest.

All the best,

Amwilkins

[andrew.sidwell]

Hi,

There is a ticket, ARNOLD-13116, for investigating any performance issues with the AutoTX workflow/using existing TX files.

Andrew

[am_wilkins]

Hello,

Okay cool, I'm glad there is a ticket up. Thanks for letting me know.

I posted about this issue over a year ago, not great that's its still a problem as this as the default behavior of Arnold. I do wonder how many clients are getting performance issues without pathing directly to the TX files.

All the best,

Amwilkins

[Stephen.Blair]

---

@am_wilkins wrote:

I do wonder how many clients are getting performance issues without pathing directly to the TX files.

All the best,

Amwilkins

---

Nobody is contacting support about it...

[am_wilkins]

This is interesting.
Do you think it could be hardware related, since my CPU is 128 threads or something?

I'm currently waiting for some publishing tool changes, so we can re-publish all our assets pathed directly to the TX, but in some quick local testing I saw an instant speed-up. (Felt like about a 1000% difference in the first IPR passes)

Will report back after some additional testing, but is there anything I can do to help figure this out - let me know!


amwilkins

[am_wilkins]

Hi @Stephen.Blair 

I will send you an email containing a video example of the slowdown via the arnold support email.

Maybe we can then get to the bottom of this issue.

All the best,

Amwilkins

[thiago.ize]

We have seen a half minute long delay to resolve the textures at startup, but that was with a scene with many thousands of textures. If you're getting a minute delay for a scene that contains just 9 files (3 textures, each with 3 udims, I believe you said), that's unexpected. If it's really just these 9 texture files, can you post an example scene that shows this, perhaps three textured quads?

If a simple scene doesn't show it, do you have any idea what kind of added complexity you need to trigger this? Maybe you have thousands or millions of images nodes that all point to the same 3 textures?

[am_wilkins]

Hello Thiago,

This scene is very simple, just containing the one house asset and a dome light.

• There are 27 texture maps in total including UDIMS, however most are quite low resolution in this example.
• The file size of all the EXR files is 52.2mb
• The file size of all the TX files is 77.7mb
• Of the UDIM textures are only a maximum of 2 tiles. (Example: 1001, 1031)
• The maps are BaseColor, Roughness and Normal
• There are no other shading nodes other than the texture image nodes, normal map, standard surface and material out for each shader.
• The slow-down appears with both manually and "auto-tx" created TX files.

I will try to provide an example scene and another log file via the support email.

Of course the big issue is working locally in scene in IPR which is likely even worse, so I may provide a "Progressive" render log to maybe show any additional slowdown due to that.

do you have any idea what kind of added complexity you need to trigger this?
Not much complexity appears necessary, the same issue has happened across 4 projects (across both Arnold and Houdini versions. In "old" houdini and Solaris) and only required texture maps pathed directly to EXR.

Maybe you have thousands or millions of images nodes that all point to the same 3 textures?
On projects, we may reach thousands of image nodes but never all pointing to the same image nodes. In the test example all image nodes are pointed to separate EXR or TX files and there is only 18 image nodes and 27 texture files.

Thanks for assisting so far,

Amwilkins

[am_wilkins]

Good day,
@Stephen.Blair @thiago.ize 

I believe I've solved the "TextureResolveUseExistingTx" problem!

It's when a "Channel Reference" is used on the texture Arnold Image node.
Without the channel reference we get no slow-down and nothing in the log and with the channel reference we get:

TextureResolveUseExistingTx                                        0:11.20 (24.62%) 
thread blocked  

Showing that there is an issue and the slow-down appears.

It's worth noting that this is still an Arnold issue I believe, cause when testing directly pathed texture files to the .TX files and with channel references—that also solves the problem.

So it's how Arnold is handling the switch out with channel refs in the mix.

ie. only when paths are to the .EXR and have channel references.

We need to do one more round of testing, but in all my tests this solved the issue.

Will shout if more findings appear.

All the best,

Amwilkins

[Stephen.Blair]

---

@am_wilkins wrote:

Good day,
@Stephen.Blair @thiago.ize 

I believe I've solved the "TextureResolveUseExistingTx" problem!

It's when a "Channel Reference" is used on the texture Arnold Image node.
Without the channel reference we get no slow-down and nothing in the log and with the channel reference we get:

TextureResolveUseExistingTx                                        0:11.20 (24.62%) 
thread blocked  

Showing that there is an issue and the slow-down appears.

It's worth noting that this is still an Arnold issue I believe, cause when testing directly pathed texture files to the .TX files and with channel references—that also solves the problem.

So it's how Arnold is handling the switch out with channel refs in the mix.

ie. only when paths are to the .EXR and have channel references.

We need to do one more round of testing, but in all my tests this solved the issue.

Will shout if more findings appear.

All the best,

Amwilkins

---

So that's a ch expression on some parameter of an image?

What does that bring in? The whole path and filename, just the filename?