transfer learning

[a9080109]

[ FlexSim 20.0.10 ]

This is a new way of learning, and when I read the literature, I saw that transfer learning can be implemented using reinforcement learning, but my question now is how can I implement this problem in Py provided by Flexsimallcombos-22-0-1.fsmflexsim_training.zip

https://github.com/trunghieu-tran/Transfer-Learning-in-Reinforcement-Learning

[allister_wilson]

This seems like a Python question, not a FlexSim question.

Is there something specific you're struggling with which directly involves FlexSim, or how the reinforcement learning tool interacts with Python?

[julie_weller]

Have you looked at the Reinforcement learning tutorial on the docs website?

https://docs.flexsim.com/en/23.1/ModelLogic/ReinforcementLearning/WorkingWithGym/Training/Training.h...

[a9080109]

I have implemented what is mentioned about flexsim

[julie_weller]

What specific questions do you have now that you've looked at the tutorial?

[a9080109]

@Kavika F

I just want to use this as a basis to make some extensions. For example, I want to know the py file about the training. I want to know how my neural network is written. Can you provide relevant information? Or Explain the logic of this py file a little bit

I understand the meaning of the picture but I don't understand what the overall training logic is

[kavika_faleumu]

Hey @mark zhen, if you want to understand the logic of the training, you'll have to read the documentation posted by stable_baselines3. For example, the PPO documentation gives links to research papers on the algorithm and examples on how to use it.

The "MlpPolicy" is the first parameter of the PPO constructor, stating that it will use a Multi-layered Perceptron. The "env" is a FlexSimEnv, which is an OpenAI Gym Environment (which is the second parameter required by the constructor). The rest of the parameters are left to their default values (which can be found on the documentation) except for "verbose", which is set to 1 so you can see in the output console what's happening.

[a9080109]

@Kavika F

So if I want to complete transfer learning or incremental learning, do you have a suggested way?

[kavika_faleumu]

Those are two very different ways of training a RL model. It ultimately depends on what your end objective is.

If you're trying to train on a small data set and improve the model over time or if your target objective changes frequently, then incremental learning may work for you.

If you're trying to train your model on one environment then use it (and potentially train it) in a separate simulation or environment, then transfer learning would be what you want.

What is your desired objective? What problem are you trying to solve?

[a9080109]

My current situation should require the establishment of incremental learning,allcombos-22-0-1.fsm

[kavika_faleumu]

Well what problem are you trying to solve? Maybe that will help me understand how I could help you implement an incremental learning algorithm. I'm not an expert in RL, but I could try to help you get started with regards to FlexSim and RL.

[a9080109]

@Kavika F @Felix Möhlmann

The problem I want to solve (these are all preset by me)

From my understanding incremental learning can be used to update the overall database,

Then I now have a trained agent and I want to perform a dynamic training (because the reinforcement learning we completed on flexsim is to send fixed data into py for training, right?)

But because it is a factory schedule, the orders for each day will be independent, which may require retraining. I want to solve this problem

The second problem to be solved is

There may be new or reduced products in my factory, then I may have to retrain the overall model, then I wonder if incremental learning can be used to solve my problem

[a9080109]

@Kavika F Do you have any good ideas?

[a9080109]

I've tried to make some changes to PY but it's not going very well, do you know what's wrong?flexsim_training.py

[kavika_faleumu]

What kind of actions is your incremental model taking? When you build a RL model, it will have to examine the environment/state of the system and make informed decisions (actions) based on the state and rewards involved with those actions.

Are the model's actions assigning operators? Is it sending items to specific places?

[joerg_vogel_HsH]

@Kavika F, please check all questions and threads of Mark Zhen. Perhaps you are able to find a satisfying answer for him, which other developers or users weren’t able to.

[a9080109]

@Kavika F @Joerg Vogel

The problem I want to solve is this, but I don't have a good way to clarify all the details

Then I now have a trained agent and I want to perform a dynamic training (because the reinforcement learning we completed on flexsim is to send fixed data into py for training, right?)

But because it is a factory schedule, the orders for each day will be independent, which may require retraining. I want to solve this problem

I don't quite know what you mean, I can provide you with the field of my incremental learning application0326_autosave.fsm

[a9080109]

@Kavika F @Joerg Vogel @Felix Möhlmann Is this problem very difficult to solve?

[kavika_faleumu]

Hey @mark zhen, the model you've posted doesn't look setup for RL. It won't give you much value.

The Processors are in-line with fixed processing times and no setup times. From what I can tell, each item has to go through each processor once, in order (Polish > Slit > Cut > Pack). There is no action needed by a RL agent because the process will take the same amount of time regardless of what items get chosen in Queue1.

[a9080109]

@Kavika F

I need to use rl for production scheduling, and the process of each product is different, so I just want to know how the best scheduling is performed