Smoke archive filling up hard drive and then crashing

[renee.busse]

Hey there! I'm trying to archive a rather small project. It might be 3GB worth of media.

It's not going very well- the archive process takes about an hour before it fills up my 1TB hard drive with 1GB segments and then crashes Smoke. I've flushed renders, source media cache, and consolidated handles on all my media. What am I doing wrong?

Thank you!

[Grant.Kay]

Hi Renee,

I believe Smoke is crashing because the destination drive for your archive is full.  I'll log that as a bug so we can get it sorted.

As for your archives being very large, this has to do with the way Smoke archives material.     Even though you are archiving compressed media, Smoke will archive the media as uncompressed RGB frames.   

Unfortunately this will increase the size of archives substantially so you need to have enough drive space.

However, this preserves the image quality as well as maintains image compability between Mac and Linux offerings.

I will also log the option to make archives with compressed media.

Many thanks for your feedback!

Regards

Grant

[Robert.Adam]

Hi Renee.

You can create a slim archive by selecting the "links only" option in the Archive Options tab.

This way only source media that has already been cached or rendered will be included in the archive. If you flush renders and cache before archiving, only audio media will be included, so the archive should be very small.

Otherwise, Smoke will forcefully cache ALL source media in your project/clips, which will result in a very large archive. (since frames are stored in uncompressed RGB)

Regards,

Robert.

[renee.busse]

Thanks but I need to send the media also. It would be better if I could archive it all together. :]

[renee.busse]

Sorry, where's the option to make the archive with compressed media?

[Robert.Adam]

There's no option to use compressed media in Smoke archives. Even if you're using a compression codec for your project's frames, they will be uncompressed into the archive. The reason we're doing that is that we want archives to be as future-proof as possible. There's no guarantee that a compression codec we might use today will still be around x years from now, and we want to avoid the situation where you're stuck with an archive you can't restore.

Do you need to archive the entire project or can you just archive the sequence? Have you tried that? The archive filesize might be more reasonable.

Another trick is to archive links only, as I mentioned before, and then manually copy the compressed source media to the same drive as the archive. That way you have everything in one place without filling the drive. You might need to relink the source media when restoring though, as the path to the source clips will be different once you move them to the drive.

Regards,

Robert.

[A_Over_B]

There is NO option to archive with compressed media.

To simplify:

In the olden days when inferno/flame/flint/fire/smoke used uncompressed RGB frames you could archive in one of two ways:

1) Archive soft-import links - which translated to "archive an alias to the original source media but don't copy or move the media in any way".

Anything that is generated in the software, like mattes or color-corrections, is not a link & is therefore archived as full uncompressed version.

2) Archive hard copies - this involved hard importing all the soft imported material and converting it to uncompressed RGB material, then writing it immediately back out to the archive.

As with option 1, anything that is generated in the software, like mattes or color-corrections, is not a link & is therefore archived as full uncompressed version.

Option 1 creates tiny archives, option 2 creates enormous archives, especially if your source material is heavily compressed material like ProRes, that gets expanded to uncompressed sizes.

These options were a flexible way to keep a handle on lots of media - for those within structured companies, with rigid/semi-rigid file systems, option 1 was a godsend.

You load the source material into a predefined directory, load the smoke archive and the aliases or softimports find all the source material.

For those with a more haphazard approach to computing, option 2 was a good catch-all situation at the expense of time & disk space.

You could effectively lose all the source material as it would have been converted to an autodesk friendly format and bundled up into an autodesk friendly archive.

Smoke 2013 still archives similarly to the legacy software but there is a subtle difference in that you can now work natively with compressed material.

Personally I'm a dinosaur & still use uncompressed material - to eliminate the tiny fraction of a second that it takes to compress & decompress my frames when I'm viewing/using them.

When I archive, I mostly use the links only method - production usually have between two or three instances of the media before I get it - making another iteration seem wasteful & pointless.

So, my archives end up being links to my source material, which weighs nothing, then hard copies of all the useful stuff that I make and want to save for later. If I were working with ProRes or DNxHD then that media would be "compressed" but since I do not, it is not.

If you are looking for a button that converts all of your media to compressed media at the time of archiving then, for now at least, you are out of luck.