Slow Scene Creation Time using AWS Portal

[tjvollB4GZV]

We are seeing a major discrepancy between on-prem vs ec2 instance render times and it appears to boil down to the scene creation time.  I'm hoping for some insights on how to diagnose where the bottleneck is being introduced.

htoa-5.6.1.1.wip_r0a29cb9_houdini-18.5.532

Attached are the logs for comparison. 

It's evident that there are some optimizations we should be making with the assets.  However, is there a suggestion on where to begin?  We are using the Deadline's AWS Portal mechanism which includes an Asset Server with an S3 backed cache.  One theory is that the missing/broken files are adding a prohibitively high amount of i/o to the process.  Our Cloudwatch usage also spikes during these renders.  That is where our internal investigation is heading. 

Any insights are most welcome.

Regards,

Tom

[thiago.ize]

Unfortunately the log file doesn't have much info on what is happening. It's just a 1 hour gap, presumably due to reading in all the assets. I think newer versions of htoa might have more profile logging which could maybe point out a little better what is going on, but even then I wouldn't hold out too much hope that it'd solve this. [Edit: this is on the roadmap but hasn't yet been done]

I would suggest trying to simplify the scene to see if you can discover what is causing the slowdown. Like you said, does removing missing files fix it? Profiling tools and strace might also be able to help if you know how to use them. Unfortunately I don't have experience with AWS so I can't give guidance on how best to configure and use it.