topic Re: Vault Purge (2023.1) Purge to 3,650 days? in Vault Forum

Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Fri, 21 Oct 2022 11:08:49 GMT

Gang...

Our Vault is approaching it's 20th birthday, and we have decided to turn on our Purging again. Our powers to be decided to turn it off about 10 years ago because of patent litigation. Now, since the patent rules have changed they have decided to allow us to turn it back on and save the last 2 versions that are over 10 years old. So, it appears, from ADMS you can type in a value other than what is in the drop down.

I tried this on our test server and it's been running for 3 days now. Still stuck on same screen. Now keep in mind, our filestore is 3.9 Terabytes.

So, can I put in 3,650 days? How do I know if it's still running?

From Task Manager I can tell that SQL is doing something, but not sure what.

Best,

Steve H.

Re: Vault Purge (2023.1) Purge to 3,650 days?

CGBenner — Fri, 21 Oct 2022 12:12:13 GMT

@steveh5

You've never been one to shy away from trying to break things, have you...? 😂

Re: Vault Purge (2023.1) Purge to 3,650 days?

Gabriel_Watson — Fri, 21 Oct 2022 15:30:55 GMT

I believe your setting is wrong, so I would kill the process ASAP! If your intent was to save the last 2 versions from 10 years ago, this will not do it!

1 - The "versions except latest" setting guarantees to keep/save the top versions of the file. This is in parallel and regardless/disconnected from the next setting "versions older than days".
2 - The "versions older than days" will erase anything that IS older than the number of days you selected. So, you are guaranteeing to lose the versions you wanted to keep from 10+ years ago by keeping this setting as is.
3 - Three TB is nothing compared to my previous Vault at 10 TB. That was a whale, and erasing just a year's worth of unused purgeable data there took about 12-24 hours. I would recommend doing a layered approach of skimming one year at a time from your records, but you must purge from older to newer, given the available tools mentioned above.

4 - I do not believe you can pick your intended copies as you stated via this tool unless you add a "flag" comment on those specific versions (to exclude versions with that comment from your purge), which will be probably a LOT of work if you don't have an amazing programmer that could automate this via API (somehow).
5 - I am expecting a lot more development for this tool in the future, after some feedback I provided to the UX team behind this. Purging is still scary in Vault 2022 (and perhaps 2023). Not sure if a new interface with new options may come in 2024 Vault, but if your files are really precious I would seriously wait for the next release and hope better tools are provided. You should be good until you hit 10 TB of Vault.

Re: Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Fri, 21 Oct 2022 16:18:55 GMT

@Gabriel_Watson ...

I'm running on a test server which is a restored backup of our production server. So, no worries 😁.

Interesting, that's not my comprehension of how it work (at least the way the dialog box is laid out).

Not arguing here... I'm just not sure.

My interpretation is that you have the toggle boxes to combine the filter or not. So, here's my interpretation....

If you just select the "Versions except latest", and if that version is not attached to a lifecycle (item or file lifecycle), all version will be purged except for the last three versions unless when of those version is tied to a lifecycle. No matter what the date.

Ver 10 - latest checkin --> kept

Ver 9 --> kept

Ver 8 --> kept

Ver 7 --> purged

Ver 6 - (Released) --> kept

Ver 5 --> purged

Ver 4 (Released) --> kept

Ver 3 --> purged

Ver 2 --> purged

Ver 1 --> purged

Now, if you check both the toggles (Versions except latest / Version older days)....it should keep all versions of the file unless it's older than 10 years. Of the files that are 10+ years, the last 3 will always be kept and any version linked to a lifecycle will be kept. All other will be purged.

Ver 10 [10/21/2020] - latest checkin --> kept

Ver 9 [10/21/2020] --> kept

Ver 8 [10/21/2020] --> kept

Ver 7 [10/21/2020] -->KEPT

Ver 6 - (Released) --> kept

Ver 5 [10/21/2020] -->KEPT

Ver 4 [10/21/2020] (Released) --> kept

Ver 3 [10/21/2010] --> purged

Ver 2 [10/21/2010]--> purged

Ver 1 [10/21/2010]--> purged

Difference between 1st and 2nd set are the dates. Since dates are not 10 years or more, files would be kept until the 10 years are up. Ver 7 and 5 are kept because they are within 10 years.

Other interesting thing is that Help says .....

4. You are prompted to confirm locking the vault to prevent users from accessing it during the operation. Click Yes.

I don't see this prompt. It just disappears back to ADMS. If I Purge Files again...I get this....

Best,

Steve H.

Re: Vault Purge (2023.1) Purge to 3,650 days?

Gabriel_Watson — Fri, 21 Oct 2022 16:34:17 GMT

Ok, phew! Good thing you are just testing this...

Anyhow, I am very certain that the purge command will work as I described. I think you also misinterpreted the "not controlled by lifecycle" portion of this. Let me see if I can unpack this explanation here:
1- "Purge versions not controlled by Lifecycles": means that any "base" category files that were not ever placed under a lifecycle will also be purged. This is a check required because you can imagine some files may have been forgotten without a lifecycle but are also important, and if your "versions except latest" is set to ZERO, then ALL versions of those no-lifecycle files will be gone (since there can be NO "released" state versions on those files).
2- versions except latest I think you got it right, except for when you believe that it also combines with the remaining options. This setting is just so we can keep any "Work in Progress" versions on top that may be important and currently being worked on before a new release/revision.
3- "versions older than days" is set for us to skim the bottom of the pan and leave any newer versions on top. The rationale is that maybe you had 20 versions created in a single day by a user that checks in a lot, but if you only save the top 3, it will be an uneven cut file to file (compared to other designers checking in less, for example).

Finally, the reason why I do not believe on the two settings you selected combining together to form one combined condition is because nothing alludes to that. You can run a simple test if you have a blank sandbox Vault for this.

Re: Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Tue, 01 Nov 2022 12:50:19 GMT

Thanks @Gabriel_Watson ....

That makes sense...appears I was reading more into what capabilities I wish it had versus what it had 😖

I did find an idea that is wishing for what I am wanting...

https://forums.autodesk.com/t5/vault-ideas/ability-to-purge-lifecycle-controlled-files-with-nunber-of/idi-p/8156036

Till then....going to have to leave purge off and let our filestore grow.☹️

Thanks for the guidance....

Steve H.

Re: Vault Purge (2023.1) Purge to 3,650 days?

ihayesjr — Tue, 01 Nov 2022 15:09:47 GMT

@steveh5

While Purge is a complex command, I think there were some keep points missed in explaining how it works.

This is not to take away how @Gabriel_Watson explained it.

Steve, your files are not controlled by "File" lifecycles because you use Item lifecycles. Therefore, all files are not controlled by Lifecycles. All of your files are "uncontrolled."
The three options are "And" together to determine how many files might possibly be purged. In your example, the files cannot be the last 2 versions "And" they have to be older than 3650 days.

This is the first set of criteria that is looked at but it isn't the final criteria. You now have to through Items into the mix. Since you do use Item lifecycles, versions of the file that are used in the "Released" state of the Item cannot be purged. So your example below will save all of the "Released" versions of the file because they are associated to the "Released" version of the Item.

Your two examples above are correct. All "Released" versions of the files are kept regardless of what you enter in the latest and older than days options. They are kept because they are associated with a Released version of the Item. If we purged them, we would break the Release Item version.

The other versions may or may not be kept depending on their association with the Item. You will need to purge your Items first. Then Purge the files so that those WIP versions of the files can be purged.

Also, the purge process will delete data in the database first. The server will delete the files later during the vacuum cleaner task.

Hope this helps.

Re: Vault Purge (2023.1) Purge to 3,650 days?

ihayesjr — Tue, 01 Nov 2022 15:03:25 GMT

@steveh5

Here is an example of what I was talking about.

File Version and Date	Item Version and lifecycle State	Purgable	Comments
Ver 10 [10/21/2020]	Item (C) - WIP	Kept	Kept because it is the last version and associated to the tip version of an item
Ver 9 [10/21/2020]	Item (C) - WIP	Kept	Kept because it is within the last 3 latest versions
Ver 8 [10/21/2020]	Item (C) - WIP	Kept	Kept because it is within the last 3 latest versions
Ver 7 [10/21/2020]	Item (C) - WIP	Kept	Kept because it is not older than 3650 days
Ver 6 - (Released)	Item (B) - Released	Kept	Kepts because it is associated to a Released version of an Item
Ver 5 [10/21/2020]	Item (B) - WIP	Kept	Kept because it is not older than 3650 days
Ver 4 [10/21/2020] (Released)	Item (A) - Released	Kept	Kepts because it is associated to a Released version of an Item
Ver 3 [10/21/2010]	Item (A) - WIP	Purgable	A candidate for purge because it is older than 3650 days and not the last 3 versions of the file. If this is still associated to an Item version that has not been purged, it will be kept.
Ver 2 [10/21/2010]	Item (A) - WIP	Purgable	A candidate for purge because it is older than 3650 days and not the last 3 versions of the file. If this is still associated to an Item version that has not been purged, it will be kept.
Ver 1 [10/21/2010]	Item (A) - WIP	Purgable	A candidate for purge because it is older than 3650 days and not the last 3 versions of the file. If this is still associated to an Item version that has not been purged, it will be kept.

Re: Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Tue, 01 Nov 2022 15:15:33 GMT

@ihayesjr

THANKS Irvin...

Just to make sure we are all on the same page...

So, my last example matches your post before this.

So, a file purge is doing what I thought it was doing. Correct?

It takes both toggles into account?

Best,

Re: Vault Purge (2023.1) Purge to 3,650 days?

ihayesjr — Tue, 01 Nov 2022 15:20:57 GMT

@steveh5

That is correct. It takes both into account and makes them "candidates" for purge. We use the word "candidate" because of the behind-the-scenes associates that need to be checked also.

My recommendation for you is to go ahead with trying it in your test environment. However, purge your Items first.

Re: Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Tue, 01 Nov 2022 15:47:28 GMT

@ihayesjr .....

Thank you for the clarification....

Now on the ITEM purge, not seeing any toggles/controls on that.

Appears this is coming from how the lifecycle was originally set up.

Guessing that we took the Out-of-the-Box settings way back when.

We have three unreleased states (WIP, In Review - Released, In Review - Obsolete), which all have these settings...

For the WIP state, my guess is that if you do 10 updates to an item before you make a state change, you will get the first and last update versions. All others in between will be purged.

Although, if an item is in either of the two In Review states, you can't technically perform an update because the item is locked. Therefore, you really can't have versions in between because it's locked, I'm thinking there will be nothing to purge.

Am I correct?????

The other two (Released and Obsolete) have this setting....

For the Released/Obsolete states, again.....you can't really perform an update to an item because it's locked down.

So, nothing will be purged on these items which are in that state.

So, in a nut shell....every transition from one state to another will be kept. WIP is the only state which will allow something to be purged.

Can you clarify if my assumption is not accurate?

Best,

Steve H.

Re: Vault Purge (2023.1) Purge to 3,650 days?

ihayesjr — Tue, 01 Nov 2022 16:22:21 GMT

@steveh5

We have three unreleased states (WIP, In Review - Released, In Review - Obsolete), which all have these settings...

For the WIP state, my guess is that if you do 10 updates to an item before you make a state change, you will get the first and last update versions. All others in between will be purged.

Although, if an item is in either of the two In Review states, you can't technically perform an update because the item is locked. Therefore, you really can't have versions in between because it's locked, I'm thinking there will be nothing to purge.

Am I correct?????

Yes, you are correct.

The other two (Released and Obsolete) have this setting....

For the Released/Obsolete states, again.....you can't really perform an update to an item because it's locked down.

So, nothing will be purged on these items which are in that state.

So, in a nut shell....every transition from one state to another will be kept. WIP is the only state which will allow something to be purged.

Can you clarify if my assumption is not accurate?

Yes, you are correct.

Re: Vault Purge (2023.1) Purge to 3,650 days?

ihayesjr — Tue, 01 Nov 2022 16:23:54 GMT

@steveh5

I missed one key thing.

For versions 1 and 10 of the Item that is in WIP, the versions of the files linked to Item versions 1 and 10 cannot be purged.

Re: Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Tue, 01 Nov 2022 16:31:47 GMT

Yes...files linked to version 1 and 10 will be kept. That is the reason for purging items prior to purging files.

Awesome... @ihayesjr THANKS for the insight and clarification.

I will definitely test, just wanted to make sure of what I was expecting to see.

Just for the records...I ran the file purge as described (3 versions/10+ years) and it ran for 6 days and cleared out ~0.3 Tb of data (Started with 3.9 Tb, now at 3.6Tb)

I will next purge the Items, then follow up with a file purge again.

Stay tuned 😁

Thanks to all....

Re: Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Mon, 07 Nov 2022 12:40:06 GMT

So, after running the item purge, I then proceeded to run the file purge, again.

Ran the file purge keeping last 3 versions and 3,560 days (same as before).

This time it purged another 0.14 Tb.

So, in total after I did the initial file purge, then an item purge, and then a file purge again...

We gained ~440 Gb ~350,000 files.

It took around five days to run the file purge, ADMS logs doesn't tell you when it's done. So, not sure how long it took.

So, @ihayesjr ....while purge is running, users can still use Vault correct?

Best,

Steve H.

Re: Vault Purge (2023.1) Purge to 3,650 days?

ihayesjr — Mon, 07 Nov 2022 13:00:23 GMT

@steveh5

When you looked at your files, is it keeping the files you expected it to keep?

Purge should be a low-priority process which is another reason for it to take a long time.

Therefore, users should still be able to use Vault during the day without much of a performance hit.

Re: Vault Purge (2023.1) Purge to 3,650 days?

steveh5 — Mon, 07 Nov 2022 19:32:08 GMT

@ihayesjr ....

So, interrogating data and what was purged.

All in the files world seems fine so far.

But....

When looking at one particular item and its lifecycle change history.....

All seems well until I get to version 114 of the item...Here all of the WIP and In Review states disappear. All that remains is the Released states. I'm thinking you should see the first and last WIP and In Review state.

Version 114 was in July of 2015 would have been Vault 2014.1 at that time.

Comparing to the pre-purge of the items....this is what I see...

Thoughts?

Re: Vault Purge (2023.1) Purge to 3,650 days?

ihayesjr — Mon, 07 Nov 2022 20:10:26 GMT

Here are my thoughts on that.

Back then, there wasn't a control bit set for the WIP states that told them to keep the first and last. Sometime after that, someone set the First and Last control bit, and then it would have only applied to WIP states created after it was set.

If you are willing to test this theory try the following.

Restore your environment again.
Inside of Vault, edit the Lifecycle Definition for the Items
Select the Control tab on the Work in Progress state.
Check the box next to "Include existing file versions". I am hoping the text is wrong and works for Items also.
Run purge on Items again.
See if some of the missing Items stay this time.