Error in SimCFDSolver.exe

[john_minthorne_corbinengineering_com]

I am getting a repeated error in Sim CFD 360 during a local solve, with the following relevant log information:

[11:57:27]  output:local to global mapping
[11:57:27]  output:Distributing field arrays
[11:57:27]  output:Field Vector updates
[11:57:28]  output:Distributing nDistinctDRRegs data
[11:58:46]  output:Distributing DR Face data
[11:58:46]  output:Distributing wall elements
[11:58:46]  output:Distributing wall node data
[11:58:47]  output:Distributing DR node data
[11:58:47]  output:Distributing transient field arrays
[11:58:47]  output:Distributing element BC data
[12:11:26]  output:job aborted:
[12:11:26]  output:[ranks] message
[12:11:26]  output:[0] fatal error
[12:11:26]  output:Fatal error in PMPI_Barrier: Other MPI error, error stack:
[12:11:26]  output:MPI_Barrier(MPI_COMM_WORLD) failed
[12:11:26]  output:Out of memory
[12:11:26]  output:[1-16] terminated
[12:11:26]  output:---- error analysis -----
[12:11:26]  output:[0] on Bim11-PC
[12:11:26]  output:mpi has detected a fatal error and aborted C:\Program Files\Autodesk\Simulation Flex 2015\Simulation CFD 360 2015\SimCFDSolver.exe
[12:11:26]  output:---- error analysis -----
[12:11:26]  output:C:\Program Files\Autodesk\Simulation Flex 2015\Simulation CFD 360 2015>set E=-4
[12:11:26]  output:C:\Program Files\Autodesk\Simulation Flex 2015\Simulation CFD 360 2015>exit -4
[12:11:26] Job finished with exit code = -4

Any suggestions on how to troubleshoot?

[scott.werner]

John,

It looks like SimCFDSolver is running out of memory.

[12:11:26]  output:Fatal error in PMPI_Barrier: Other MPI error, error stack:
[12:11:26]  output:MPI_Barrier(MPI_COMM_WORLD) failed
[12:11:26]  output: out of memory
[12:11:26]  output:[1-16] terminated
[12:11:26]  output:---- error analysis -----

1. How many total elements is your job and how much GiB of RAM do you have? We recommend 2 GB of memory per 1 million elements.
2. To conserve memory, close the UI during solve time, and reopen it when it is done solving.

You can find total # of elements in the Output Bar:

** FINITE ELEMENT SUMMARY FOLLOWS...

704237 Total Nodes, 581270 Fluid Nodes , 122967 Solid Nodes

3248311 Total Elements, 2536461 Fluid Elements , 711850 Solid Elements

1 Inlets 1 Outlets 0 Unknowns

*********** ANALYSIS STARTED ***********

[john_minthorne_corbinengineering_com]

The model is between 1 and 2 million elements. I have 64 GB of RAM. I watched the resource monitor during the solve, and the RAM was nowhere near used at the time of the crash. Does the output bar content save to a log somewhere?

[waited]

John, Scott,

I believe the memory fault occured inside the MPI Barrier.  This may occur when a memory address is over-ridden and garbage is access while a MPI request is being executed.  This is the kind of problem where a cfz file would be very helpful so that the model could be placed in a debugger for analysis.

Dave.

[john_minthorne_corbinengineering_com]

Posting details of the geometry on a public forum is a non-starter. I could send it to an individual for analysis if they can sign our standard NDA form first. Can this be changed to an advanced support ticket number so I can work with a single individual?

[john_minthorne_corbinengineering_com · Accepted solution]

We were able to get this to solve by activating the cad_surf_data_option flag, increasing the mesh density, and adjusting the model geometry to eliminate some of the smaller features. I am not 100% certain which of these actions had the most effect on avoiding the error.