Render Failing - MSG = "COLOUR : Initializing"

[Anonymous]

Uggg... this is my worst nightmare today.

I am on an older machine and I am not on support.  [Smoke Advanced 2013.2.0]

All of last week I have been working on some fairly major comp shots, which I haven't done in a long time.  Anyways after getting the hang of things last week I set the machine to render my BFX on friday evening and left it over the weekend.  When I came in this morning it had not rendered and the cacheing was acting a little weird.  After several reboots the cache seems to be working again, but it crashes whenever I go to render either of my comps (nearly identical setups).

The machine freezes while rendering the first frame and the message in the lower left says...

COLOUR : Initialising

and that's it, it will not respond after this point.

I was able to find this same message in the most recent log file smoke20132_lustre_app.log

[notice] 2422862544 messageAccumulator.C:291 02/17/14:13:19:25.081 Rendering: Batch FX - frame 0 of 118 - 00:00.
[notice] 1353996608 CoThreadPthread.C:200 02/17/14:13:19:25.164 Registered thread 'MepDispatcher' [ 1353996608 ]
[notice] 1087641920 CoMsgLogger.cpp:211 02/17/14:13:19:33.860 Last entry repeated 6 more times.
[notice] 2422862544 messageAccumulator.C:291 02/17/14:13:19:33.860 COLOUR : Initialising.
[notice] 1353996608 CoThreadPthread.C:200 02/17/14:13:19:34.469 Registered thread 'MepDispatcher' [ 1353996608 ]

I can rebuild my comp from my BFX saved setups, but as soon as I try to render I get the same issue.

Has anyone out there seen this before?  Is it a Colour corrector node somewhere that is unhappy?

Any insights would be appreciated hugely!

Cheers & thanks,

_Chris

Oh yeah, I almost forgot, I can't seem to find the release notes documentation on the service packs for 2013.2 anywhere.  I keep getting 404 pages on the autodesk site.  Anyone seen those around?  If it is a known bug I would like to pressure the money people here to finally give me the time to install the service packs, or better yet, get back on support!

[Robert.Adam]

Hi Chris.

Any messages in smoke's shell log from around the same time?

What's the MemoryApplication keyword set to in Smoke's init.cfg file?

The Color message just happens to be the last message displayed in the Application's message center. I don't believe it's related to the issue you're having.

Here are the 2013 Ext 2 release notes: http://images.autodesk.com/adsk/files/flame2013ext2_releasenotes.pdf

There's no known bug similar to the problem you're describing.

BTW: apologies for the issues with finding documentation on the Autodesk.com site. The site went through a big overhaul last weekend and we're still working on making sure all pages are online.

Regards,

Robert.

[Anonymous]

Hi Robert, thanks very much for your reply.

The shell and shell log didn't have anything related to the freeze in them unfortunately, only the app.log.

The MemoryApplication keyword is commented out (#).

I am running a Z800 with 12GB, what would be a good number to set it at?

Thankyou for the release notes, do you have a link to the release notes for the last service pack for 2013.2?  (was it SP2 or SP3?)

I am hoping to convince my superiors to give me some time to download & install the latest service pack before I do any more of these rather complex vfx shots.

Thankyou very much for your help!

Cheers,

_Chris

[Robert.Adam]

Hi Chris.

Here are the SP3 release notes: http://images.autodesk.com/adsk/files/flame2013ext2_sp3_releasenotes.pdf

Try uncommenting the memory keyword and set it to 8000 and see if that makes a difference.

If that doesn't fix things and you have another Z800 with the same graphics card model, it might be worth trying to swap graphics cards to rule out graphics card hw issues as a possible cause.

Your Z800 could also use a bit more RAM if you're working on large projects.

Regards,

Robert.

[Anonymous]

Hi Robert,

Surprisingly, I do have another Q6000 that I might try swapping in when things cool down a little.  For now I have been able to break my composite apart into separate BFX.

I tried the setting the memory keyword to 8000 and ended up with some new errors in the shell (I've cut and pasted these below).  So I tried again with 7000 and have now settled on 6000 without any errors since yesterday.  Do you think this could be related?  Or signs of problems with the memory?

Cheers & Thanks Again!

_Chris

dlCheckGLError: Texture::unbind()
dlCheckGLError:   GL_OUT_OF_MEMORY
dlCheckGLError: Texture::unbind()
dlCheckGLError:   GL_INVALID_VALUE
dlCheckGLError: Texture::pre-bind()
dlCheckGLError:   GL_INVALID_FRAMEBUFFER_OPERATION
dlCheckGLError: Texture::unbind()
dlCheckGLError:   GL_INVALID_FRAMEBUFFER_OPERATION
dlCheckGLError: Texture::pre-bind()
dlCheckGLError:   GL_INVALID_FRAMEBUFFER_OPERATION
Out of memory allocating 22478848 bytes.
Could not read: Frame (fId:0x2d384de5400000a6): Out of memory
Out of memory allocating 17104896 bytes.
Could not read: Frame (fId:0x2d384dd240000039): Out of memory

[Anonymous]

Smileys?  Silly forum!

Those lines should read...

dlCheckGLError: Texture:: pre-bind()

[Steve_Curley]

Tip. Use the "Insert Code" button on the toolbar (if using Rich Text, as you were), or if using HTML wrap the code in
< PRE >...< /PRE > tags (without the spaces).

 Result = no smileys ;) 

[Robert.Adam]

Hi Chris.

Yeah, those messages look like out of memory issues. It sometimes takes some playing around to find the right value. Looks like you found a good one at 6000.

Hope you don't get any more such messages from now on, but when you get a chance you might want to think about adding more RAM to the system.

Regards,

Robert.

[Anonymous]

Thanks Steve! That's good to know!

[Anonymous]

Thanks Robert, I ended up getting the out of memory errors all the way down to 4000 eventually.  But there is more RAM on the way!

I don't think that is my problem with these BFX not loading though.

I have one that I have split out as many layers into precomp BFXs and ended up with a fairly simple BFX, but it is still locking up on me.

These lines were the end of the log when it hung and just after I closed the shell to force the smoke to quit.

[notice] 1349595856 messageAccumulator.C:291 03/05/14:17:08:12.880 Batch FX : Creating node connections.
[notice] 1397156160 CoThreadPthread.C:200 03/05/14:17:08:14.586 Registered thread 'MepDispatcher' [ 1397156160 ]
[warn] 1349595856 mp.C:2613 03/05/14:17:14:40.226 Flushing log in signal handler

Any insights?

Cheers & Thanks Again!

_Chris

[Robert.Adam]

Hi Chris.

Those log entries are normal operation messages. Looks like the app hung without reporting any error before.

Can you please send me the entire log, or even better can you please zip up the entire /usr/discreet/log directory and send it over?

Do the issues only occur with BFX in this specific project? Are other projects ok?

Any chance you could send me a small project archive (without any media) to take a look at?

Thanks,

Robert.

[Anonymous]

Hi Robert, thankyou so much for looking into this for me.

The forums won't allow an attachment over 5MB so here is a wetransfer.com link.
SHOT010_MEDIA_UNLINKED.tar.gz
This is a version of my timeline BFX that mostly doesn't work, but for some reason opened first thing this morning (but only once).  I have been trying to compress an archive of another version of the shot that more consistently crashes, but nothing will do the trick.  I have archived that one, but could not unlink the media without a crash so it's about 10GB.  I can restore from it just fine, but I can't edit it or unlink the media.  Can I send it to you via FTP?

I have attached the logs to this post as well.

I haven't run into this issue before in other projects, but I have had to create new projects for this shot because of the Smoke trying to open the BFX as the project is starting up.  Effectively locking me out of the whole project.  (I am currently on my 3rd project for this shot)

Thankyou again!
Cheers!
_Chris

 

[Anonymous]

Hi Robert, since I keep missing you online I thought I would go ahead and PM you the FTP details of the larger archive.

No worries if it's too big, I was just hoping you might take a look at it.  The other one I sent by wetransfer was working fine again for me later in the day today, which seems weird, but I was being very careful with the system so I guess that helped a little maybe.  I didn't have any other versions of the shot loaded in other rec tabs, and I shut the system down and booted it up cold when I got it to work.

More importantly, I restored this larger archive (the one on the FTP) on our backup Smoke (a Z800 with only 6GB of ram) and it hung the same way when entering the timeline BFX, so I can fairly confidently say that whatever the problem is, it exists in this archive.  But for some reason it kept giving me errors when I tried to compress it.

Anyways, the FTP info should be in your inbox.

Thankyou so much for your help Robert.

I hope the bandwidth isn't a problem.

Cheers,

_Chris

[Anonymous]

Hi Robert!

Have you had any luck?  I've finally managed to get myself completely stuck.

Attached to this message is a batch setup that I can't open, even on another system without any media.

Do you think it's possible to salvage it?

Cheers & thanks,

_Chris

[Robert.Adam]

Hi Chris,

I had sent you a private message yesterday. I tested your archive with media on a Z800 with 12GB of RAM and though some operations in BFX took a few long seconds to execute, the software never froze or crashed. I also tested on a 9400 with 8GB of RAM where it consistently crashed, especially when selecting the color correct nodes in your setup. A z820 with 32GB RAM had absolutely no issue or lag with any operation I threw at it.

Today I also tested the archive with no media you sent me the other day. It freezes the 9400 pretty much instantly and throws out of memory errors, and it's also noticeably slower on the z800, but it never froze or crashed the software. Available RAM on the z800 was around 1.5 GB (out of 12) while rendering or fiddling with that BFX. The logs didn't contain any clues.

I'll try opening the archive you have just attached. I'll try to fix it to the point where you can open it again.

Regards,

Robert.

[Anonymous]

Hi Robert, I got your PM after posting my message.  Sorry about all the bouncing back and forth.

So, I thought I was a very lucky man today, because the memory fairy arrived and doubled my RAM from 12GB to 24GB.

But it hasn't helped, I still can't open the last version of my comp.  (the batch setup posted above)

I've attached a screen grab of the output from the "top" and "free" commands in the terminal.  These varied very little over the course of the 10 minutes I left it to see if it would come back to life.

What else can I try?

Cheers & Thanks,

_Chris

[Robert.Adam]

Hi Chris.

So it does take up to 18GB of RAM... have you tried just leaving it alone for several minutes?

I have tried your latest setup on the Z800 with 12GB and it does not freeze but it sometimes takes long minutes to enter BFX or to select nodes. I also tried rendering it and it took about 26 minutes which I found quite long. I did notice that if I try to turn the Action node or anything downstream of that into a BFX clip, the app crashes with an error about the Paint node. I've tried removing the Paint node and still getting the crash. I'll do some more testing tomorrow trying to split your BFX into several chunks until i get to the bit that causes the slowness and crashes.

Stay tuned.

Regards,

Robert.

[Anonymous]

Will do.  Thanks Robert!

[Anonymous · Accepted solution]

I think I got it.  Thankyou for mentioning about the action node, it wasn't doing a whole lot so I hadn't suspected it could be the issue.

Yesterday I was experimenting with a duplicate batch setup and I was removing node files from the directory to see if it would load it up without all of the node files.  It didn't work out for me at the time.  But today, when I removed only the action node, it worked!  I immediately suspected the 3rd output from the node.

Output 1 = comp

Output 2 = matte

Output 3 = media matte

So I loaded just the action node setup into an otherwise empty BFX, deleted Output 3, and then saved it overtop of itself.  And voila!  Suddenly I could load my BFX setup again!  And I didn't really need that media matte to come through action anyways!  Hah!

Hopefully this is the end of my troubles with this project!

Thankyou Robert! Cheers!

_Chris

[Robert.Adam · Accepted solution]

Hi Chris.

I'm glad to hear you have found the culprit and made it work again.

This is usually the best approach to take when large BFX start acting up: start splitting it out or taking nodes out until you find the corrupt one.

Cheers,

Robert.