Vault Project Archiving

[john.stewart]

Project based implementations using Vault have definite beginning and end. AEC and Infrastructure Customers are Project driven.

MFG customers serving the Oil and Gas are also Project driven since machinery is custom specific for the project.Typical projects can create a large number of files throughout the Project lifecycle. E.g. Filestore = 5GB

Customer Scenario: A customer can have 20 active projects and 100 completed projects over a 5 year period.

In this scenario IT run weekly backups. 100GB for active projects and 500GB for completed projects.

Currently Vault will require that all 600GB be backed up.

Vault Project Archiving would allow IT to classify the 500GB filestore as Archive and exclude these Archive Filestore folders from the weekly backup.

Solution :

Allow administrator to archive Project folders from the ADMS Console.

Project Files would be moved to a new ARCHIVE Folder that can be excluded from the ADMS backup or 3rd party Backup.

The Vault Database metadata would point to the new ARCHIVE Folder location.

The Project Files would be moved on all ADMS Filestore Servers and supported on Full replicated databases.

The metadata for the Archived Completed Project would remain in the Vault Database.

Folder ACLs can used to set the Archived Project to Read Only

[Redmond.D]

You might want to take a look at my File Catalog app, which attempts to solve the same problem.  It uses custom objects to keep a record of the file, so that it can be archived and deleted from the Vault.

It's obviously not as good has having a built in Vault feature, but have a look.  Let me know what you think.

[tmoney2007]

Couldn't this be achieve by maintaining a running backup of all release versions separate from the rest of a full backup?  It kind of ends up being an incremental backup, but a full backup would only ever update the released versions section and all newer WIP versions.  This should save space when multiple "full" backups are saved. 

A full backup would consist of:

Released Versions + WIP Versions (as of a particular date) + Record of which released versions are new since the last backup + configuration setting and miscellania.  This should allow a restore to a particular point, without requiring multiple copies of things that haven't changed.

[Richard.Rankin]

As Enterprise customers Vaults get larger and larger, it becomes increasingly costly to support. This is exacerbated, when Connected Workgroups are implemented or even a simple Backup retention policy. Backups, SQL Maintenance, Service Pack installation, data transfer and Restoration all take longer and longer.

Enthusiastic Purging is a possible solution, but involves the destruction of data. Therefore the ability to Archive obsolete files\datasets, but bring them back temporarily from the client UI, albeit at a slower rate, would be desirable.

I have heard this request from multiple customers in the past. If you need this functionality supported in Vault, please vote by giving this request Kudos!

[johannes.bauer]

Challenge:
- Data gets added to Vault and the database size grows

- Maintenance gets more costly the bigger the database grows. Investment in Backup/Storage Strategy is needed. 
- Completed /Obsolete projects need to be archived

Proposal:
- implement an archiving system to „move“ obsolete data to a separate database based on specific criteria's (status/last modified date/…)

Impact:

- Database stays smaller and performance stays at normal level

- archived projects can be stored on cheaper storage in Enterprise Systems

[karthur1]

Can I vote more than once?  Just kidding.  This is really needed in Vault.  We have projects that are in Vault that have not been touched in 8 years.  

[plembourg]

Hello you really need to do something on this, i have customer with more than 3 TBytes in a vault..!

[braeuchler]

We need a solution to archive old projects in Vault.
Target: minimize the Vault Filestore of the produktive System. Keep the archive data in a lower performance system.

[kinplex]

Can you copy the data file from the server and say save it with a new file name.  Delete the files that you want to archive on standard vault file and then access the new file for archived data when needed?

[AndriesMuylaert]

Purge is great to keep current running projects 'lean' but at some point in time the project is finished and the files are no longer interesting. This is where an 'archive project' option on the folder would be interesting.

e.g.: The option removes the folder from Vault and archives the latest version of the files as a zip file on the server.

The result is a 'lean and clean' Vault. 

Even better would be to have an option to 'recall' the archived project folder in case it is needed again...

..of course the disadvantage is you loose some history...and perhaps there are some restrictions needed.

[P.Markovic]

I would like to see the ability to set a Vault on an ADMS server to Archive Mode.  This would serve two purposes and maybe others. 

The first purpose would be for customers that multiple Vaults on the ADMS and they don't want all of them to be included in a backup.  Once a full backup has been done for all of the attached Vaults and archived off for safe keeping, I would like the ability to set a Vault to Archive Only that would then exclude it from full ADMS backups but still allow users to log in to the archived Vault, as Read Only, for reference.

This brings me to the second purpose of when the Vault is set to Archive Only, it would become Read Only without having to go in and change all of the user's access.  This would be very useful during upgrades when we are migrating a Vault environment from one server to another, but want to keep the current Vault online for users to use in a Read Only mode until the new Vault environment is online.  Currently, there isn't an easy way to make a Vault Read Only without going in changing everyone's access.

[florian_baillifard]

The idea was accepted in 2013... Please develop it now.

[walid.driouch]

Absolutely needed in Vault !

[benjamin.huntLSN5B]

I agree that this is needed in Vault as an administrator function. We have 650GB of data to manage currently, that we have on a hyper-V environment (backup taken of Hyper-V) but currently haven't got the recommended / supported process in place.

The file store data backup and supported methodology of Vault is also out dated.

If we had an archive facility that was easy to implement / use then at least we could use the supported methodology of seed, increment and seed and so on, on a much reduced foot print as our business is project based.  

We are currently running 2019 and will be upgrading next year to 2021. Is there any indication of development in this area?

[johnmondro4888]

Just looked into doing this (since most of our engineering department is on furlough.) Guess not. Can't believe I waited this long to try archiving our Vault. Guess it didn't matter LOL! Please add this functionality - thanks!

[Mike.Wohletz]

With a 4TB vault and growing, I would like to see something on this. I don't know if more time is needed since it has only been like 7 years since it was suggested. Some of my other smaller vaults are easier to maintain, but this thing is starting to get to be a pain to backup regardless of the methods used. 

[punisher]

I’d like to toss my 2 cents in. Certain types of AEC companies could benefit greatly by being able to target older project files by type and age. These could be written off to a vault archive. These would still contain their respective meta data. Obviously, file dependencies would be a challenge. However, after 10 years, the project files are no longer touched until there is a specific need. The project metadata would remain in tact, and is all they need in order to find the archived project in a search. 
For Many AEC forms having the archived project header in the search would be enough. Then hopefully the interface could be scheduled to restore the missing project files from the context menu… or something similar. 
Im building a new Vault implementation for a company and rather worried that they will have TB of data rapidly. 
Hell, while I’m asking Santa for the goods, it would be even nicer if the admin can choose to retain only project header data, or the file database data as well as the project; this would allow shrinkage of the database as well as the data store. I’m actually concerned about the database size as much as the file store. 

[tmuel]

This is a huge need.

@john.stewart described our situation exactly. Our Machines are "custom projects" that rarely change after completion but need to be accessible for reference. Our Vault filestore is approx. 4TB and an "ADMS Backup" takes several days.

In order to increase the speed of the backup process and make it easier to manage the filestores, I would like to be able to separate our "active" projects from our "archive" projects during the backup process.  It would be helpful to have the ability to identify a folder or file(s) as "archive" and have the related filestore files be in a different "partition" and/or "backup package" (perhaps a different filestore). The new backup option would capture only the "active files". A restore would require the "active" backup along with the corresponding "archive"

Tim