topic Re: Reinforcement Learning with FlexSim Model. Fps decreases with training time in FlexSim Forum

Reinforcement Learning with FlexSim Model. Fps decreases with training time

nitin_s2 — Fri, 23 Sep 2022 07:06:16 GMT

[ FlexSim 22.0.0 ]

I am following the tutorial (link) on the FlexSim website to integrate RL with an existing FlexSim model. Using the codes provide there too for training and inference.

As I was training the RL agent, I could see that the fps (frames per second) showing a consistent decrease (see graph) with model training time. High fps means more iterations of FlexSim environment (faster training) and it usually should fluctuate around a constant value.
I was wondering if this is a known case and if someone could point me to a way to fix is.

fps-drop.PNG

Re: Reinforcement Learning with FlexSim Model. Fps decreases with training time

Jacob_White — Fri, 23 Sep 2022 19:12:38 GMT

Hi @Nitin S2,

In reinforcement learning the fps generally will decrease over time because the algorithm takes more and more time to think about what to do. There is a possibility that we have a bug FlexSim that slows FlexSim down, but you'd probably notice that running the model over and over. Example: run an experiment with 50 replications, only using 1 core. Does the model run slower and slower? If not, then the slowdown must be the RL tool just thinking a lot.

Re: Reinforcement Learning with FlexSim Model. Fps decreases with training time

nitin_s2 — Mon, 26 Sep 2022 03:56:01 GMT

Thanks for the response.

I am not sure that fps decreasing with time is an RL exploration issue. Will run the experiment and find more on it.

Re: Reinforcement Learning with FlexSim Model. Fps decreases with training time

Jeanette_Fullmer — Thu, 29 Sep 2022 16:33:29 GMT

Hi @Nitin S2, was Jacob W2's answer helpful? If so, please click the "Accept" button at the bottom of their answer. Or if you still have questions, add a comment and we'll continue the conversation.

If we haven't heard back from you within 3 business days we'll auto-accept an answer, but you can always unaccept and comment back to reopen your question.