topic Re: transfer learning in FlexSim Forum

transfer learning

a9080109 — Wed, 05 Jul 2023 18:32:47 GMT

[ FlexSim 20.0.10 ]

This is a new way of learning, and when I read the literature, I saw that transfer learning can be implemented using reinforcement learning, but my question now is how can I implement this problem in Py provided by Flexsimallcombos-22-0-1.fsm flexsim_training.zip

https://github.com/trunghieu-tran/Transfer-Learning-in-Reinforcement-Learning

Re: transfer learning

allister_wilson — Wed, 05 Jul 2023 20:42:27 GMT

This seems like a Python question, not a FlexSim question.

Is there something specific you're struggling with which directly involves FlexSim, or how the reinforcement learning tool interacts with Python?

Re: transfer learning

julie_weller — Wed, 05 Jul 2023 21:18:59 GMT

Have you looked at the Reinforcement learning tutorial on the docs website?

https://docs.flexsim.com/en/23.1/ModelLogic/ReinforcementLearning/WorkingWithGym/Training/Training.html#app

Re: transfer learning

a9080109 — Mon, 10 Jul 2023 03:14:58 GMT

I have implemented what is mentioned about flexsim

Re: transfer learning

julie_weller — Mon, 10 Jul 2023 13:45:29 GMT

What specific questions do you have now that you've looked at the tutorial?

Re: transfer learning

a9080109 — Wed, 12 Jul 2023 09:38:13 GMT

@Kavika F

I just want to use this as a basis to make some extensions. For example, I want to know the py file about the training. I want to know how my neural network is written. Can you provide relevant information? Or Explain the logic of this py file a little bit

I understand the meaning of the picture but I don't understand what the overall training logic is

Re: transfer learning

kavika_faleumu — Wed, 12 Jul 2023 15:44:32 GMT

Hey @mark zhen, if you want to understand the logic of the training, you'll have to read the documentation posted by stable_baselines3. For example, the PPO documentation gives links to research papers on the algorithm and examples on how to use it.

The "MlpPolicy" is the first parameter of the PPO constructor, stating that it will use a Multi-layered Perceptron. The "env" is a FlexSimEnv, which is an OpenAI Gym Environment (which is the second parameter required by the constructor). The rest of the parameters are left to their default values (which can be found on the documentation) except for "verbose", which is set to 1 so you can see in the output console what's happening.

Re: transfer learning

a9080109 — Wed, 12 Jul 2023 15:52:49 GMT

@Kavika F

So if I want to complete transfer learning or incremental learning, do you have a suggested way?

Re: transfer learning

kavika_faleumu — Wed, 12 Jul 2023 23:18:28 GMT

Those are two very different ways of training a RL model. It ultimately depends on what your end objective is.

If you're trying to train on a small data set and improve the model over time or if your target objective changes frequently, then incremental learning may work for you.

If you're trying to train your model on one environment then use it (and potentially train it) in a separate simulation or environment, then transfer learning would be what you want.

What is your desired objective? What problem are you trying to solve?

Re: transfer learning

a9080109 — Thu, 13 Jul 2023 06:54:46 GMT

My current situation should require the establishment of incremental learning,allcombos-22-0-1.fsm

Re: transfer learning

kavika_faleumu — Thu, 13 Jul 2023 19:27:04 GMT

Well what problem are you trying to solve? Maybe that will help me understand how I could help you implement an incremental learning algorithm. I'm not an expert in RL, but I could try to help you get started with regards to FlexSim and RL.

Re: transfer learning

a9080109 — Fri, 14 Jul 2023 07:01:26 GMT

@Kavika F @Felix Möhlmann

The problem I want to solve (these are all preset by me)

From my understanding incremental learning can be used to update the overall database,

Then I now have a trained agent and I want to perform a dynamic training (because the reinforcement learning we completed on flexsim is to send fixed data into py for training, right?)

But because it is a factory schedule, the orders for each day will be independent, which may require retraining. I want to solve this problem

The second problem to be solved is

There may be new or reduced products in my factory, then I may have to retrain the overall model, then I wonder if incremental learning can be used to solve my problem

Re: transfer learning

a9080109 — Tue, 18 Jul 2023 05:24:00 GMT

@Kavika F Do you have any good ideas?

Re: transfer learning

a9080109 — Thu, 20 Jul 2023 03:19:44 GMT

I've tried to make some changes to PY but it's not going very well, do you know what's wrong?flexsim_training.py

Re: transfer learning

kavika_faleumu — Thu, 20 Jul 2023 19:35:47 GMT

What kind of actions is your incremental model taking? When you build a RL model, it will have to examine the environment/state of the system and make informed decisions (actions) based on the state and rewards involved with those actions.

Are the model's actions assigning operators? Is it sending items to specific places?

Re: transfer learning

joerg_vogel_HsH — Fri, 21 Jul 2023 02:43:02 GMT

@Kavika F, please check all questions and threads of Mark Zhen. Perhaps you are able to find a satisfying answer for him, which other developers or users weren’t able to.

Re: transfer learning

a9080109 — Fri, 21 Jul 2023 08:18:29 GMT

@Kavika F @Joerg Vogel

The problem I want to solve is this, but I don't have a good way to clarify all the details

Then I now have a trained agent and I want to perform a dynamic training (because the reinforcement learning we completed on flexsim is to send fixed data into py for training, right?)

But because it is a factory schedule, the orders for each day will be independent, which may require retraining. I want to solve this problem

I don't quite know what you mean, I can provide you with the field of my incremental learning application0326_autosave.fsm

Re: transfer learning

a9080109 — Wed, 26 Jul 2023 16:31:24 GMT

@Kavika F @Joerg Vogel @Felix Möhlmann Is this problem very difficult to solve?

Re: transfer learning

kavika_faleumu — Wed, 26 Jul 2023 17:29:26 GMT

Hey @mark zhen, the model you've posted doesn't look setup for RL. It won't give you much value.

The Processors are in-line with fixed processing times and no setup times. From what I can tell, each item has to go through each processor once, in order (Polish > Slit > Cut > Pack). There is no action needed by a RL agent because the process will take the same amount of time regardless of what items get chosen in Queue1.

Re: transfer learning

a9080109 — Wed, 26 Jul 2023 18:20:18 GMT

@Kavika F

I need to use rl for production scheduling, and the process of each product is different, so I just want to know how the best scheduling is performed