Mimic Task Scheduler Migrate with a Custom Task

[petestrycharske]

All,

I am working on a large file migration project with lots of folders that contain 1 or 2 files in each, so it is impractical to select each folder and the migration task keeps failing when I try to migrate the entire overarching folder.  Therefore, I decided to go down the route of a custom task based on a blog post that I read (here is the post) and can get most of the way there with a ton of assistance from one of my programming colleagues.  However, I am a little stuck because I get different results if I migrate with this custom task versus using the task scheduler.

In this case, I migrated a fileset of 678 IPT files with the custom task and all the files passed successfully.  However, when I migrated the same set of files as a test, I get three failures via the Task Scheduler migration task.  These failures are based on a mass update failure.  This got me thinking that I am probably not checking the same criteria that the task migration task is checking.  Does anyone know the  criteria that one should be checking to closely mimic task scheduler when doing custom file migration?

Currently I am opening, rebuilding, saving and closing each file in a CSV file list.  I will look to add criteria to update the mass, but want to make sure that there isn't something else that I am missing.  If you have any questions, please do not hesitate to contact me.  Hope all is well and have a most blessed night!

Peace,

Pete

[VinodBalasubramanian]

Hi Pete,

Thanks for letting us know. Can you able to provide us your current inventor version and the service pack level you are in. Also if possible please attach the log files generated from task scheduler for both full task and the custom one.

http://knowledge.autodesk.com/support/inventor-products/learn-explore/caas/CloudHelp/cloudhelp/2014/...

I would appreciate if you could provide one or two of the part files that have different results. I can private message you for a secure FTP link to upload those files.

Looking forward to your inputs.

[petestrycharske]

Vinod,

Unfortunately, this data is from a client, so I will have to check in with them for permission.  Here is a portion of the file though, if this gives you any clues:

From Failed file log:

XXXXXX.ipt--Mass Properties check error

From the Custom Task Log: (note that this is a log file that is being generated thanks to the work of my colleague and is not the standard Task Scheduler log file)

31;Success;XXXXXX.ipt

What I am hoping for is a list of criteria that Inventor goes through to validate a file: IPT, IAM, IDW; when performing the migration task.  Once I know that criteria, I can write that into the external iLogic rule.  I can try to get you a copy of the logs and part files, but won't be able to do so until tomorrow afternoon, after I consult with the client.  Hopefully this helps to clarify my request and feel free to ask additional questions.  Hope all is well and have a most blessed night!

Peace,

Pete

[petestrycharske]

Oh, I forgot...

We are migrating using Inventor 2015, SP1.

[petestrycharske]

Does anyone know what the Check operation does in the Task Scheduler Migrate Task?  It looks as though it checks the Mass Properties before and after the migration, but does it do anything else?

[petestrycharske]

We are looking at the back ground of the Task scheduler and have discovered that it pulls a lot of information from .mdb tables.  A new thought that we have is that we would like to populate a table with the values that we want.  So instead of picking each folder individually, we can simply enter in a table for the folders that we want to migrate and have the normal task scheduler run.  Unfortunately, we looked high and low and couldn't find the task scheduler table that shows the folder list for each task.  The folders are listed when I look at the Task Scheduler task, so this data must be kept somewhere...  Does anyone know where this information is located?

[petestrycharske]

Just checking back in to again request if anyone knows of a way to access the tables of folder and file lists behind the scenes of task scheduler migrate?  Otherwise, can anyone confirm what the Check operation does in the task scheduler migrate?  I was doing some double checking and I noticed at the top of the migration log files, there is the following verbiage:

-----Task Scheduler Start Time:6/2/2015 10:08:35 PM -----

Start Migrate Files

----------------------Information for every file a user migrates or updates-----------------------
Open - time in seconds (column 5)
Make all precise drawing views - time in seconds (column 5)
Update - difference between the number of comatose (column 5) and sick (column 6) features before and after the update
Rebuild - difference between the number of comatose (column 5) and sick (column 6) features before and after the rebuild all
Check - difference in the volume from when it was opened up to the point after the update and rebuild are done (column 5)
Save - time in seconds (column 5) and the difference in file size (byte) before and after save (column 6)
Close - time in seconds (column 5)
---------------------------------------------------------------------------------------------------

This shows that the Check operation seems to be comparing the Volume of the assembly after opening and then after the update.  Is this true for assemblies?  Are there other things that are checked?  If I can confirm all the checks that task scheduler makes, then I could perhaps greatly expedite this migration process.  Please let me know if you have any questions and have a most blessed day!

Peace,

Pete

[petestrycharske]

Another question I have.  Is it possible to call the Task Scheduler migrate function from the custom task?  If so, could I populate a custom list that way?  If this is possible, then I don't have to try and mimic the task scheduler migrate functionality, I can just focus on creating the batch lists that I want to run.  Please let me know what your thoughts.  Hope all is well and have a most blessed day!

Peace,

Pete

[ChrisMitchell01]

Hi Pete,

All this sounds overly complex for simply migrating a folder full of ipt files. You should be able to migrate 000's of ipt's just using TaskScheduler alone. I've done migrations in the past which run for days & process several hundred thousand files.

If you create a migration task & pick the top level folder & set the filetype extension to be .ipt, what's the error ? Does it fail or just sit there without doing anything? It can appear to take some time for the progress to start since all of the tables are being created in the default mdb file in the background, before any of the migrations actually occur.

I've seen cases where you see a mass props error but when you look at it, the difference is only in the order of 10-4 or so, which is essentially noise & may be due to very slight changes in the ASM kernel, etc.

From Task Sceduler you can pick on a job & then go to File export erros to Excel & that may give you an easier way to look at what's failing.

Also, why are you migrating the files in the first place ? In many cases that's not really necessary anymore.

Thanks,
Chris

[petestrycharske]

Chris,

Thank you so much for looking into this for me!  Unfortunately, I was not present to see the error, as it occurred over the weekend and the programmer noticed the error, but I believe it wasn't just preprocessing, but had actually stopped working.  I understand about the migration approach of doing an entire folder at once and letting it churn through all the subfolders, but we just have the confidence that this will work and don't want to wait a day or two to discover that it won't work.

Some of the IPT files are quite large, so we wanted task scheduler to work through many of the smaller file sizes while opening up the larger files on some dedicated computers.  Some of the larger IPT files are also derived components which seem to have file referencing issues, which have had to be worked through anyway.  So we were able to target many of these part files and then manually key in the folders to capture most of them.  However, there were 20k - 30k IPT that were only 1 file per folder...  That is why we had tried the mass migration over the weekend, but couldn't get it to work.  We have worked around the IPT issue by batching these large quantities into subfolders and then migrating those subfolders as as whole.

Additionally, the files are being added to weekly, so we are having to go back and re-migrate files that are still in the older Inventor version.  Life would be much easier if everyone was already using Inventor 2015, but that wasn't an option in this case. 

Also we face the same challenge with the assemblies, where several thousand of the files are organized one file per folder, so we would like to create batches of these files to process whenever we wish.  I will likely try to migrate all the assemblies over the weekend again, but based on my experiences already, I am not confident it will work.

So for the reasons above, we thought it would be nice to run task scheduler, but just based on preconfigured CSV files.  We could then have the faster computers migrate the more intense files and handle any new additions quickly and easily.  Again, we can do this successfully, but are not confident that we are covering all the aspects that the Task Scheduler - Migrate covers, so want to know what all the checks are.  Otherwise, we would be content to populate a table with the files we want, similar to keying them in through task scheduler.

Hopefully this clarifies what I am trying to accomplish and I wish that the weekend migration had been successfully completed, but I am leery of it based on the large IPT, IAM and IDW sizes.  Feel free to let me know if I have been unclear regarding any of the steps or desired outcomes.  Please let me know if I am way off the mark and if you can think of anything else to try.  Hope all is well and have a most blessed day!

Peace,

Pete

[ChrisMitchell01]

Hi Pete,

That clarifies what you're doing, but not "why".

Many customers now don't worry about migrating all their files & essentially rely on more of a migrate on demand philosophy - have you had the discussion with whoever you are working on all this for, as to why this is all being done ?

-Chris

[petestrycharske]

Chris,

We did have this discussion with them and actually advised that the files should be migrated on an as-needed or as-opened basis.  However, they desire all the files to be migrated as it takes the individual users too long to open the files and then save / migrate them.  The idea is for a small group of us to experience the "pain" of having to migrate all the files, so that the other 80 or so users are not disrupted.  I am not saying I completely agree with the approach, but that is the direction that we have decided to take and are trying to pound through these files as quickly as possible.  Hope this helps illuminate the why aspect and welcome any additional feedback / questions.  Hope all is well and have a most blessed day!

Peace,

Pete

[ChrisMitchell01]

OK, that's fair.

All I would suggest then is to keep a track of anything that you see as a potenital failure from all these various tasks & then open that (hopefully small) subset of files interactively, do the rebuild/save & check that all is as expected.

-Chris

[frederic.joubert]

Hi,

I read your post about "the pain" of migrating files and I understand very well Pete to create a more plaisant tool to migrate. We also have a multiple products folders structure containing one to hundreds of Inventor files as per complexity of the product. First folder is product #0001 and we have pass product #6500.

We used to be told that you should always migrated files when you update Inventor version. Since 2010, when Inventor was deploy here, I'm in charge of that unpleasant task. We have a subscription for our licences so we update every year. The last complete migration was at the 2015 version. In 2016 I skip some old product projet because of the time consumption. Now we're at 2018 and we decided to completely migrate all files. About the non-migrated portion of 2016, I think, when I start the Task-Scheduler, it preprocess, and preprocess... sometime longer then it migrate! Sometime all day for less than 1000 files. it seem to stuck on some files. I try, after no sign of life of the preprocessing, restart computer and manually open the file, corrected it if so, save it and close. I verify that the files are migrated by the iProprety and everythings ok. I restart the same migrated task and it stuck on the same place again! What the...

So I'm a bit confuse when I read that Chris suggest to migrate on demand (Chris are you from Autodesk?)

Does my problem could come from the non migrated part?

Also why the selection fonction of the top folder with the shift key and the last folder to select a batch doesn't work here? We have to clic all the folders to select them one by one. But in the Migrated task window you can select the first and the last with the shift!? **** it!

Ok enough for today, lets get a beer!

Ciao

[ChrisMitchell01]

Yes, I'm from Autodesk.

Sometimes the pre-processing taking a long time is related to the fact that the data is in in vault & it's actually the vault check-out processes which takes a long time. Other times the wait is due to an excessively large Access dB file which is where the Task Scheduler stores all of the info to be batch processed. You can compress the Access dB from from File menu of Task Scheduler. Or you can just delete the dB file itself & then it will be recreated (providing you don't need to retain any of the legacy data in it - it's in C:\Users\Public\Documents\Autodesk\Inventor 2018\Task Scheduler

Regarding the migration of multiple folders, the Ui does only allow the selection of a single folder & then uses the recursive option to iterate down through sub-folders.

Thanks

Chris

[frederic.joubert]

Hi Chris,
Thanks for the recommendation about "cleaning" the Acces dB file.
I will wait for Monday because I have actually 3 computers sleeping on the preprocessing of 3 task migration. But definitely I will try it.
Last time I checked, after 1 hour, 2 were still stuck on each other same files and one was stuck but on another file then the one before.
I monitor the task scheduler with the Windows Task management to see if something run. Is it normal that on a 8 processor system (64 bits 16GB i7-4790CPU @ 3.60Ghz) the CPU usage is only at 6-8% during preprocessing? It's the biggest CPU of the 3 and it's the slowest! Past day I have to run 5 times the same task without success on that one. I think he fall in coma again. The Task scheduler window get empty of all previous task suddenly.
[cid:image004.jpg@01D312B2.21C76270]
On a 4 processor I'm at 40-50% during preprocessing (64 bits 10GB i7-870CPU @ 2.93Ghz)
I'm pretty discourage.
We don't have the vault (luckily)!


«Regarding the migration of multiple folders, the Ui does only allow the selection of a single folder & then uses the recursive option to iterate down through sub-folders.»
Why not?

Thanks and have a good week-end

Frédéric Joubert

[cid:image001.png@01D312AF.1C5A19C0]

Concepteur Mécanique

Solutions Élastomères sur Mesure

Ingénierie d'usine

Engineered Rubber Solutions

Mechanical Designer
Plant Engineering

Tél./Phone : 450-436-2433, poste/ext. : 6268
Téléc./Fax : 450-436-8495
851 Baron, St-Jérôme, Québec, Canada, J7Y 4E1
Frederic.joubert@soucy-group.com
www.soucybaron.com

[frederic.joubert]

Hi Chris,

About the Acces dB files that you suggested to clean, is it this one?

 

Because when I deleted it, the Task Scheduler don't want to start anymore because db files is missing!

On my station I got only one file but on the stations which I migrate there is 2 files; one of 1Ko and one of about 15000Ko.

I try to delete the 15000Ko without success. I had to replaced it.

And what about the questions about stations performances?

Yesterday it take all day to migrate 319 files! Normal?

«Regarding the migration of multiple folders, the Ui does only allow the selection of a single folder & then uses the recursive option to iterate down through sub-folders.»

Why not?

And what about migrating on demand or migrating every Inventor update?

Thanks

Fred

[ChrisMitchell01]

My apologies Fred. I thought we recreated that mdb file if it was missing - seemingly not. If it's that large then it needs to be compacted from the File menu in TaskScheduler.

It shouldn't take that long for 319 files, but I guess it depends on whether they are large assemblies or just parts & the specification of the workstation it's running on.

-Chris

[frederic.joubert]

Hi Chris,

As you suggested, I compress the Taskscheduler.mdb from 114520ko to 1816ko on 1 of the 3 computers I use to migrate and makes no differences. The pre-processing still stuck at divers files. I will compress the other 2 when their tasks end.

I think that it's not normal that it take all day to migrate 66 "normal" files. By "normal", I mean not to big; single part and 10-20 parts ass'y.

I think I could do better opening myself the 66 files, saving it and close. No vault here.

I inform our IT services of my migrated problems. They investigated a bit; size of files and performance of the CPU thru Windows Task Manager. While pre-processing the CPU, is sleeping! They also mentioned, if the migration went well last year, there is no reason to have problem this year, Autodesk must have change something somewhere that impact the migration by the Task Scheduler.

Am I the only one having problem to migrate files with the Inventor Task scheduler, especially 2018?

Is there a marketing thing to have a non performance tool to have more people go to Vault or get to the "cloud" licences option and get more $$$? How come Autodesk haven't upgrade the Task Scheduler since?

We are on a subscription plan, is there someone that could log on our computer and do some check?

«Regarding the migration of multiple folders, the Ui does only allow the selection of a single folder & then uses the recursive option to iterate down through sub-folders.»

Why not?

What do you mean by: «& then uses the recursive option to iterate down through sub-folders.»?

And what about migrating on demand or migrating every Inventor update?

What is suggested?

Thanks

I'm a pro-Autodesk user since 1996 who normally wouldn't ear nothing about other CAD on the market but I will maybe reconsider and at least listen to our sisters plans to study the deploy of a Catia Dassault system.

[ChrisMitchell01]

This Idea is not really the best way to get Product Support - you should be able to get that through https://knowledge.autodesk.com/contact-support/technical-support/troubleshooting?p=Inventor%20Produc...

Have you tried disabling any virus checking ? That's all I can think of right now although I asked the developers for any further suggestions. I haven't heard of any other cases of this taking excessive time in 2018. Others on the public discussion forum may have experienced it though so it might be worth asking there too. - 

This is the recursive option I mentioned:

I assume you're also using the Immediately option ?

Many of our customers with smaller assemblies only migrate on demand; it's only with very large assemblies that you would really notice any performance impact from not migrating. Another method to migrate is to check the option from App Options, Save tab for "Prompt to Save for Migration", open the assembly & then use Migrate from the File, Manage menu.

Hope this helps,
Chris