Setting up a Render Farm

[Anonymous]

Hello,

My company is researching into setting up a render farm to speed up render time for our marketing material and CG video content. I have done some research into the subject and I understand that each computer in the render farm must meet the 3DS Max minimum requirements, but what are the recommended settings? We'd like to configure the render farm to exceed the minimum requirements, to create the most efficient render farm we can. We are planning to begin with 5 render machines, two workstations, and the File Server.
We have a few follow up questions we'd like clarification on as well:

Currenty, we have two copies of 3DS Max in the office- one 3DS Max 2014 and one 3DS Max 2013.  Do both workstations need to be running the same version of 3DS Max in order to speak to the render machines?

Do the render machines need to be configured identically? Do they need to match the configuration of the workstations?

We primarily render with mental Ray, though we do have V-Ray for the 3DS Max 2013 machine, and occasionally we use that. Not sure if it matters, but we seldom if ever render anything but .pngs. Even when rendering video we use PNG sequences until the video is composited in After Effects and Premiere.

[Anonymous · Accepted solution]

From what I have read and my recent experience...

All machines must have the same version of Max to run a render farm. In my experience, the machines should be close in their configuration to avoid render issues, especially with lighting. My render machines are Dell desktops that vary from T3600, T3610 and a couple of T7600's. Render results have been good. I tried adding a Thinkpad laptop to the farm which resulted in a serious flickering issue with my lights. I now use the Thinkpad as a workstation, so my workstation is completely different from the render stations. I do notice a very big difference in the image colors/gamma. Good thing the Dells produce the preferred image.

All materials and image maps should be in a location that all render machines can access/rights and must have UNC paths (no drive letters). This is also where the renders should be saved. Luckily there is an option within Max to always convert to UNC paths in the Customize > Preferences.

Hope I've helped.

[Alfred.DeFlaminis · Accepted solution]

Hello @Anonymous and welcome to the community,

@Anonymous has some good points here.  I'm not really in a position to recommend hardware, but lots of RAM is always a plus.  You must have identical service packs on every machine, and you can have both 2013 and 2014 installed on all farm machines.  Job sent from 2013 will render with 2013 and so on and so forth.  Mismatched versions of plugins, service packs, scripts, etc. will cause all sorts of problems, so you must ensure an identical versioning between systems.  Please let me know if you have questions.

Best Regards,

[Alfred.DeFlaminis]

Hello @Anonymous,

I wanted to follow up here, did you have any other questions or concerns?  

Best Regards,

[Anonymous]

Hi,

I do actually have a few follow up questions. How many licenses are required for network rendering? I thought only the workstations need licenses. Do the render machines and host each need a license as well? My graphics lead is also concerned that that our workstations may not actually be usable while rendering from a render farm. My understanding is that they would be freed up this way.

Thank you.

[Anonymous]

The workstations used for rendering (servers) do not require a license. Just the same Max and Backburner version installed.

Rendering is very CPU intensive which makes it very difficult to perform other tasks during a render.

My coworkers have been able to do simple task within email or databases but are not able to effectively use Photoshop, Illustrator or InDesign during a render.

I run my renders overnight so I do not disrupt any other work that needs to be completed.

[kris]

You'd be surprised at how much you can get done with old hardware.  

First, to address your last question.  You only need one max license to render on your farm, as long as you're only using max itself.  Different renders have different licensing requirements; vray, for example, requires additional licensing past whatever number you have in your base license.  I have the Workstation prime, so I have 1 user, 11 render nodes.  Most plugins will render on an unlimited number of nodes for rendering.

As far as how powerful the machine has to be, the ability to run max is a requirement but not necessarily it's minimum requirements.  You can get away with a lot depending on how you set up your scenes.  They should be set up differently for farms, at least I think so.  A render box isn't having to deal with viewport overhead, and so forth, so you don't need a video card of any substance if you're not doing GPU rendering.

My farm is made up of machines that are 5-10 years old, mostly government servers that were surplused. You have to balance the cost of the machine against the cost of power - older machine will provide less power per watt, though CPUs really haven't made a whole lot of advancement in the last 10 years, so that's less of an issue unless you're really tight.  My servers range from X5690s to e5320s, with 32 to 8GB of ram each.  For three years, I was in a location that had unlimited power as part of the rent, and I took advantage of that.  I used (base on my estimate) $3500/mo (CAD) in power.  Now that I'm back home, I am much more frugal as to when I turn on the farm.  I have a total of 38 machines I can turn on as needed.

If you break down your scenes adequately, you can render on just about anything.  I'm in the habit of splitting out my scenes into as many layers as I possibly can, as long as it doesn't take much time for setup; this reduces the ram overhead, and keeps frames down to 5-15 seconds, even on lesser machines.  Generally, this gives me the flexibility to re-render very small slices, and have a fast fix time since clients can't make up their minds about some things.  I'm a one person shop, so that's paramount to me.  If your scenes are very heavy, in geometry, textures, and heavy f/x loads, slower computers will help less.  Generally speaking, a rendered sequence for me might be made of of 20-50 layers, and my fastest render box will render 2.5x faster than the slowest.  So when time is tight, all those ancient machines make a non-trivial difference.

Just remember having a lot of machines means you have to deal with the heat - my old server room never went below 25C, even though it was well below freezing outside, with a window open.

In any case, retiring machines from front line service makes total sense.  I don't sell old mobos/cpus/ram, generally.  Don't know for sure?  Just turn on BB, and run a test, and you'll know.

[kris]

---

@Anonymous wrote:

Rendering is very CPU intensive which makes it very difficult to perform other tasks during a render.

My coworkers have been able to do simple task within email or databases but are not able to effectively use Photoshop, Illustrator or InDesign during a render.

I run my renders overnight so I do not disrupt any other work that needs to be completed.

That's true, but there are workarounds if you only have one machine - specifically, in Task Manager, you can change the affinity of tasks.  This means you can force the rendering instance of max to use one core, or all but one.  As long as you have the ram, you can keep working on the machine.  Even with the farm, I do this all the time.

Kris.

[Anonymous]

Thanks Kris. Good to know for future renders.

[Alfred.DeFlaminis]

I just wanted to add that if you are on Vray 3.4 you can no longer set affinity on the processors without adding an environment variable to Windows.  There is a thread on the Chaosgroup forums somewhere about this.  Just wanted to FYI since there is so much cool info in this thread. 

Best Regards,

[3dfh]

Hi everybody,

I'm also interested by a render farm.

I want to buy a servor rack, because my company is growing slowly and i want to upgrade the servor everytime i have a new emlploye.

I

f i understand well, the graphic card is not so important. THe most important is the CPU and the memory?

Do you have recomanded servor?

I have an estimate with 2 servor bixeon but it's a bit expensive to start...

Do you recomand many servor with less CPU/memory? or less servor with more memory/CPU?

Thank for your help!!

Regards,

Fabien

[kris]

@3dfh  What your server has in it is proportional to what kind of renderer you use, etc.  If you use a GPU renderer, having a video card would be a good idea.

What you do is affected by a great number of factors, but it does generally come down to the type of renderer you're using.  For example, VRay's licenses are per machine, and they're not exactly cheap.  In that case, it behooves you to have  the most powerful server you can buy for the dollar.  If you're using a renderer with unlimited nodes, then you can get away with older, slower computers.

Either way, you have to optimize your pipeline to take advantage of those respective servers.  I have a lot of older (5+ year old) machines in my farm, and I set up my scenes so that I can get renders that take no more than 5 seconds/frame.  This generally means I can get away with having only 8GB of ram in most of them.  These are all old government surplus machines, which are about 50% as fast as their modern equivalent; dual processor, quad core.  But they're only $250/each.

Cost of power is also a factor.  Newer machines are generally better performance per kW.  Older machines, less so.  The cost of power can become a serious consideration: my farm can cost as much as $4000/month in electricity when it's on and running full tilt, and that doesn't include cooling costs.  That said, you only need to turn it on when you need the performance, and the cost is much less than using a third party render farm. 

You have to balance the starting cost versus the operating cost, basically.  Also, if you can pass those costs on to your customers.  Note that MR will no longer ship with max going forward, so if that is your primary renderer, you might want to look into the cost of that (and/or Arnold).  This is something you need to understand about your workflow, because it's hard to generalize it.

Kris.

[dgorsman]

Adding to that, once you've got a purchase and operating cost (including maintenance time) for an in-house system, be sure to compare it to commercial render farms.  They can be very cost-competitive, especially for intermittent work or as a "bridge" to handle higher capacities while you grow.

[3dfh]

When you talk about commercial render farms, you mean online renderfarm?

[Anonymous]

I would like to add, if you have an IT department and IT Security, you may want to check with them regarding the use of 3rd Party online render farms.
Some companies, such as mine, will not approve the use of these farms because of client confidentiality.

Even with a non-disclosure agreement, some companies may determine the service is not secure enough. Getting approval internally and from the client organization can be a lengthy process.

[dgorsman]

---

@Anonymous wrote:

I would like to add, if you have an IT department and IT Security, you may want to check with them regarding the use of 3rd Party online render farms.
Some companies, such as mine, will not approve the use of these farms because of client confidentiality.

Even with a non-disclosure agreement, some companies may determine the service is not secure enough. Getting approval internally and from the client organization can be a lengthy process.

---

And yet they likely are mum about DropBox or other online storage/transfer systems without prior approval.  Policies can be so... absurd... at times.

Sometimes procedures need updating to meet modern realities, or at least revised to be consistent.  But otherwise yes - ensure security policies are adhered to.

[Anonymous]

Hi Kris,

We are planning also to set up a small render farm here and would like to know what are the requirements, set-up and licensing requirements. We use 3dmax 2017,vray 3.0 ans vizpark plug-ins for now. my workstation right is  intel xeon e3-1270 at 3.60 GHz, 32 gb RAM,nvidia quadro k2200.  We are planning to setup initial a 4 or maybe 5 render nodes + 1 for the file server.

--- do we need to install a copy of 3dmax to all nodes? so my 3dmax in my workstation plus 1 copy 3dmax for all the nodes?

--- how about the other plug-ins?

--- also how do you configure your server so the nodes can see it? do i need to put all the maps and other textures on the server?

--- I never did this before, for example i have a animation scene in my workstation that is ready to render, should i render it from my workstation directly         to      the farm?  Can you share some sort of illustration?

Hope you can share some though on this subject very confusing for the beginner like me.

Thanks a lot.

Arnel 

[darawork]

Here is one that I am planning to build at a later date:

https://ie.pcpartpicker.com/list/NRjZVY

32 Threads

Start your own build on PCpartPicker first, to check compatibility issues.

[Alfred.DeFlaminis]

Hello @Anonymous,

You can install 3ds Max on as many nodes as you'd like, you just can't activate it for workstation use.  But a non-activated version of Max can still be used for rendering on a farm.  So you do need to install it, but that's it.  If your renderer requires render node licenses, you'll need those as well.  Each renderer has different rules for it.  You can get 1 license of Arnold with Max, but beyond that it does cost.  Vray, Mental Ray, Final Render, and Arnold all require render node licenses for farm use.  

You should use network pathing and make sure your drive letter path or UNC path is available to all servers and workstations.  Most plugins will have render node installations such as Forest Pack.  

To send your local job to the network, you need to use some kind of render manager.  Backburner comes with Max and can be used and there are others as well.  Backburner is 32-bit and does have file size restrictions, but the most popular render manager is called Deadline.  

If you have trouble please make a thread specifically about it and the community or myself would be happy to help.  Thanks!

Best Regards,

[brewdamaster]

cloudrender.farm is a useful farm that allows you to not worry about the licensing component and offload the compute entirely leaving your system free to make edits/new scenes