Hello everyone,
Hoping somebody can help me with this challenge:
We have done a complete migration of 80 000 files on Vault. (drag and drop)
Our Vault serveur is in version 2020. I have the Vault client 2018 + SDK installed on my computer. There is no problem to log on Vault, working with, and so ...
All the files are in Vault now and I need to fill all their properties from our old data base with a csv file. Some properties are UDP (user defined properties) and some properties are mapped properties with the cartridge of dwg drawings.
First I updated all file which wasn't "dwg" extension to avoid mapped properties problem in a loop in C# with:
DocumentService.CheckoutFile
DocumentService.UpdateFileProperties
DocumentService.CheckinUploadedFile
It worked with success. Then now I have filled all UDP of 40 000 files without any problem.
Then we have the other 40 000 files with UDP + mapped properies to fill. I have try with a program in C# inspired from this link: https://justonesandzeros.typepad.com/blog/2015/02/another-way-to-update-properties.html
I follow the trick getting a ticket, DocumentService.CheckoutFile and with FilestoreService.CopyFile to update the properties trought the PropWriteReq[] and then DocumentService.CheckinUploadedFile. I tried with 1 file and it worked to change mapped property.
I added in the code the line to update the UDP DocumentService.UpdateFileProperties befor making the FilestoreService.CopyFile(updating mapped properties).
It worked fine. It changed the UDP and the mapped properties.
After that I launched the process trought a loop and it runs good while the 100 first files, then the loop make an error on the line FilestoreService.CopyFile. I looked the file where the program crashed. It was not a corrupted file. I tested the code only on this file which crashed before and now it worked to update UDP and mapped property... strange.
I restarted the programm and it worked fine with the 100 first files and crashed again on FilestoreService.CopyFile but with another file. After this second crash when everybody on the company tried to connect with the Vault client, they got this error message:
And when I tried with the program in C# I got the same error message on log on.
The IT has to restart Vault server to fix. And then everybody was able to connect again on Vault...
I restarted the program 10 times and in the 10 times he completed more than 100 files, he crashed on the line
FilestoreService.CopyFile. And all the company got again this erro message.
Then I tried to change the method to update these properties to avoid disturbing all the company. I tried with the Vault client.
Our IT has changed the "pagination" propertie on the Vault server from 100 to 1000 to change more than 100 files properties in one shot.
I selected 1000 files in Vault, then the properties I wanted to change ( there was UDP + mapped properties) and edit them. I fill the whole table with all information and I launched the properties updating.
The process is ok but when he reach more than 500 files completed of the 1000 files, it become very very slow for each files. I canceled the process but the windows frozen then I forced Vault client to close. After that I try with 1 file, it take 2 minutes to complete the process but when I got the result of properties change all property was checked on updated done but the last line show "time out". And no properties was done for the file... It was no more possible to change mapped properties of files, it always says time out. I tried only to change UDP and it worked ! But changing mapped property always says "time out".
The IT has not restarted the server and the day after I could again change mapped properties...
I tried again to update trought this method but now after 500 files and more updated it goes very fast and finished. But looking the result it show the 500 files wich has successfully updated the properties but all other got the Server 503 error message.
And now when I try to connect again with Vault client I got the same error message I got when it crash with the API:
I need to finish the complet migration(just need to update the propeties of these 40 000 dwg files) for the next week, how can I update the properties without crashing the whole vault server for all our employees everyday ?
Another way is making a change of 100 files per shot but 400 times doing edit properties, waiting,copy the column of filename to excel and filling with macro trought our old database, copy again in table of vault editing properties and launching the updating, waiting until it is finished and starting again 400 times...
We are in 2021, I think we have something to do that faster, did I miss something?
Thank you very much for your help !
Below the code of the loop:
private void RempliPropDeKLIO()
{
Autodesk.Connectivity.WebServices.Folder DossierPlanKLIO = m_conn.WebServiceManager.DocumentService.GetFolderByPath("$/Espace de travail/KLIO/Plans");
CollecteLesFichiers(DossierPlanKLIO);
VDF.Vault.Currency.Properties.PropertyDefinitionDictionary PropsFichier = m_conn.PropertyManager.GetPropertyDefinitions(VDF.Vault.Currency.Entities.EntityClassIds.Files, null, Vault.Currency.Properties.PropertyDefinitionFilter.IncludeUserDefined);
Cat[] CategorieFile = m_conn.WebServiceManager.CategoryService.GetCategoriesByEntityClassId("FILE", true);
long oCatId = 0;
foreach (Cat oCat in CategorieFile)
{
if (oCat.Name == "SGM")
{
oCatId = oCat.Id;
break;
}
}
//line in csv
long DONR = 0; // 0
long Nplan = 0; // 1
long NPage = 0; // 2
long Indice = 0; // 3
long Description1 = 0; // 4
long Description2 = 0; // 5
long InseredansKLIOpar = 0; // 6
long InseredansKLIOle = 0; // 7
long Remplacele = 0; // 8
// filename // 9
////Method2
//Autodesk.Connectivity.WebServices.PropDef DONR = 0; // 1
//Autodesk.Connectivity.WebServices.PropDef Nplan = 0; // 2
//Autodesk.Connectivity.WebServices.PropDef NPage = 0; // 3
//Autodesk.Connectivity.WebServices.PropDef Indice = 0; // 4
//Autodesk.Connectivity.WebServices.PropDef Description1 = 0; // 5
//Autodesk.Connectivity.WebServices.PropDef Description2 = 0; // 6
//Autodesk.Connectivity.WebServices.PropDef InseredansKLIOpar = 0; // 7
//Autodesk.Connectivity.WebServices.PropDef InseredansKLIOle = 0; // 8
//Autodesk.Connectivity.WebServices.PropDef Remplacele = 0; // 9
CatCfg oConfig = m_conn.WebServiceManager.CategoryService.GetCategoryConfigurationById(oCatId, new string[] { "UserDefinedProperty" }); //{ "UserDefinedProperty" });
foreach (Autodesk.Connectivity.WebServices.Bhv propduFichier in oConfig.BhvCfgArray[0].BhvArray)
{
if (propduFichier.DisplayName == "DONR")
{
DONR = propduFichier.Id; //method1
}
if (propduFichier.DisplayName == "N°plan")
{
Nplan = propduFichier.Id;
}
if (propduFichier.DisplayName == "Indice")
{
Indice = propduFichier.Id;
}
if (propduFichier.DisplayName == "Page")
{
NPage = propduFichier.Id;
}
if (propduFichier.DisplayName == "Description 1")
{
Description1 = propduFichier.Id;
}
if (propduFichier.DisplayName == "Description 2")
{
Description2 = propduFichier.Id;
}
if (propduFichier.DisplayName == "Inséré dans KLIO par")
{
InseredansKLIOpar = propduFichier.Id;
}
if (propduFichier.DisplayName == "Inséré dans KLIO le")
{
InseredansKLIOle = propduFichier.Id;
}
if (propduFichier.DisplayName == "Remplace le")
{
Remplacele = propduFichier.Id;
}
}
using (StreamReader sr = new StreamReader(@"C:\Je\TRAVAUX\MIGRATION\klio\BDDKLIO.csv"))
{
string strResult = sr.ReadToEnd();
string[] result = strResult.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);
for (int i = 2; i < result.Length - 1; i++) //result.Length; i++)
{
string[] oProp = result[i].Split(';');
//long DONR = 0; // 0
//long Nplan = 0; // 1
//long NPage = 0; // 2
//long Indice = 0; // 3
//long Description1 = 0; // 4
//long Description2 = 0; // 5
//long InseredansKLIOpar = 0; // 6
//long InseredansKLIOle = 0; // 7
//long Remplacele = 0; // 8
// // filename // 9
char[] charsToTrim = { ' ' };
List<PropInstParam> propInstParams = new List<PropInstParam>();
PropInstParam propInstPar0 = new PropInstParam();
PropInstParam propInstPar1 = new PropInstParam();
PropInstParam propInstPar2 = new PropInstParam();
PropInstParam propInstPar3 = new PropInstParam();
PropInstParam propInstPar4 = new PropInstParam();
PropInstParam propInstPar5 = new PropInstParam();
PropInstParam propInstPar6 = new PropInstParam();
PropInstParam propInstPar7 = new PropInstParam();
PropInstParam propInstPar8 = new PropInstParam();
propInstPar0.PropDefId = DONR;
propInstPar0.Val = oProp[0];
propInstParams.Add(propInstPar0);
propInstPar1.PropDefId = Nplan;
if (!string.IsNullOrEmpty(oProp[1]))
{
propInstPar1.Val = oProp[1].Trim(charsToTrim);
}
else
{
propInstPar1.Val = oProp[1];
}
propInstParams.Add(propInstPar1);
propInstPar2.PropDefId = NPage;
propInstPar2.Val = oProp[2];
propInstParams.Add(propInstPar2);
propInstPar3.PropDefId = Indice;
propInstPar3.Val = oProp[3];
propInstParams.Add(propInstPar3);
propInstPar4.PropDefId = Description1;
if (!string.IsNullOrEmpty(oProp[4])) { propInstPar4.Val = oProp[4].Trim(charsToTrim); } else { propInstPar4.Val = oProp[4]; }
propInstParams.Add(propInstPar4);
propInstPar5.PropDefId = Description2;
if (!string.IsNullOrEmpty(oProp[5])) { propInstPar5.Val = oProp[5].Trim(charsToTrim); } else { propInstPar5.Val = oProp[5]; }
propInstParams.Add(propInstPar5);
propInstPar6.PropDefId = InseredansKLIOpar;
if (!string.IsNullOrEmpty(oProp[6])) { propInstPar6.Val = oProp[6].Trim(charsToTrim); } else { propInstPar6.Val = oProp[6]; }
propInstParams.Add(propInstPar6);
propInstPar7.PropDefId = InseredansKLIOle;
propInstPar7.Val = oProp[7];
propInstParams.Add(propInstPar7);
//Convert to the required array of arrays
PropInstParamArray propInstParamsArray = new PropInstParamArray();
propInstParamsArray.Items = propInstParams.ToArray();
PropInstParamArray[] propInstParamsArrays = new PropInstParamArray[1];
propInstParamsArrays[0] = propInstParamsArray;
var ListDeTousLesFichiersConcerne = TousLesFichiersDeVault.FindAll(s => s.NomFichierVault.Equals(oProp[9])); // on cherche dans la liste des fichier recupéré du dossier toutes les ligne qui porte le nom du fichier peu importe l'extension
DocumentService documentService = m_conn.WebServiceManager.DocumentService;
string ticket = documentService.SecurityHeader.Ticket;
long userID;
userID = documentService.SecurityHeader.UserId;
foreach (var Ligne in ListDeTousLesFichiersConcerne)
{
string lext = Ligne.ExtensionFichierVault.ToLower();
if (lext == "dwg" || lext == "dxf")
{
bool ErreurCopy = false;
bool ErreurTicketouFilePropMapper = false;
CtntSrcPropDef[] fileProps = new CtntSrcPropDef[] { };
CtntSrcPropDef ItemfileProps = new CtntSrcPropDef();
ByteArray downloadTicket = new ByteArray();
try
{
downloadTicket = m_conn.WebServiceManager.DocumentService.GetDownloadTicketsByFileIds(Ligne.IDFichierVault.ToSingleArray()).First();
fileProps = m_conn.WebServiceManager.FilestoreService.GetContentSourcePropertyDefinitions(
downloadTicket.Bytes, true);
}
catch
{
ErreurTicketouFilePropMapper = true;
}
if (ErreurTicketouFilePropMapper == true)
{
Console.WriteLine("ErreurTicketouFilePropMapper :" + Ligne.NomFichierVault);
}
else
{
//Recupere les propriété non mapper
List<PropInstParam> propInstParamsUnamaped = new List<PropInstParam>();
PropInstParam propInstPar0Unamaped = new PropInstParam();
PropInstParam propInstPar6Unamaped = new PropInstParam();
PropInstParam propInstPar7Unamaped = new PropInstParam();
propInstPar0Unamaped.PropDefId = DONR;
propInstPar0Unamaped.Val = oProp[0];
propInstParamsUnamaped.Add(propInstPar0Unamaped);
propInstPar6Unamaped.PropDefId = InseredansKLIOpar;
if (!string.IsNullOrEmpty(oProp[6])) { propInstPar6Unamaped.Val = oProp[6].Trim(charsToTrim); } else { propInstPar6Unamaped.Val = oProp[6]; }
propInstParamsUnamaped.Add(propInstPar6Unamaped);
propInstPar7Unamaped.PropDefId = InseredansKLIOle;
propInstPar7Unamaped.Val = oProp[7];
propInstParamsUnamaped.Add(propInstPar7Unamaped);
bool PasDePropMapper;
// to make the sample easy, we are only going to allow the edit of string properties
//var propSet = "";
//IEnumerable<CtntSrcPropDef> pPage = null ;
CtntSrcPropDef pPage = new CtntSrcPropDef();
CtntSrcPropDef pNplan = new CtntSrcPropDef();
CtntSrcPropDef pIndice = new CtntSrcPropDef();
CtntSrcPropDef pDescription1 = new CtntSrcPropDef();
CtntSrcPropDef pDescription2 = new CtntSrcPropDef();
if (fileProps == null)
{
PasDePropMapper = true;
//propSet = "";
}
else
{
var propSet = fileProps.Where(n => n.Typ == DataType.String);
pPage = propSet.FirstOrDefault(n => n.DispName == "gentitle.Page");
pNplan = propSet.FirstOrDefault(n => n.DispName == "gentitle.N° plan");
pIndice = propSet.FirstOrDefault(n => n.DispName == "gentitle.Indice");
pDescription1 = propSet.FirstOrDefault(n => n.DispName == "gentitle.Titre 1");
pDescription2 = propSet.FirstOrDefault(n => n.DispName == "gentitle.Titre 2");
}
//code du moniker en string propriété qui map
//long DONR = 0; // 0
string mNplan = ""; // 1
string mNPage = ""; // 2
string mIndice = ""; // 3
string mDescription1 = ""; // 4
string mDescription2 = ""; // 5
List<ListPropReq> ListDesPropReq = new List<ListPropReq>();
if (pPage != null)
{
if (pPage.DispName != null)
{
mNPage = pPage.Moniker;
var addNPage = new ListPropReq(mNPage, oProp[2]);
ListDesPropReq.Add(addNPage);
}
else
{
PropInstParam propInstPar2Unamaped = new PropInstParam();
propInstPar2Unamaped.PropDefId = NPage;
propInstPar2Unamaped.Val = oProp[2];
propInstParamsUnamaped.Add(propInstPar2Unamaped);
}
}
else
{
PropInstParam propInstPar2Unamaped = new PropInstParam();
propInstPar2Unamaped.PropDefId = NPage;
propInstPar2Unamaped.Val = oProp[2];
propInstParamsUnamaped.Add(propInstPar2Unamaped);
}
if (pNplan != null)
{
if (pNplan.DispName != null)
{
mNplan = pNplan.Moniker;
if (!string.IsNullOrEmpty(oProp[1]))
{
var addNplan = new ListPropReq(mNplan, oProp[1].Trim(charsToTrim));
ListDesPropReq.Add(addNplan);
}
else
{
var addNplan = new ListPropReq(mNplan, oProp[1]);
ListDesPropReq.Add(addNplan);
}
}
else
{
PropInstParam propInstPar1Unamaped = new PropInstParam();
propInstPar1Unamaped.PropDefId = Nplan;
if (!string.IsNullOrEmpty(oProp[1])) { propInstPar1Unamaped.Val = oProp[1].Trim(charsToTrim); } else { propInstPar1Unamaped.Val = oProp[1]; }
propInstParamsUnamaped.Add(propInstPar1Unamaped);
}
}
else
{
PropInstParam propInstPar1Unamaped = new PropInstParam();
propInstPar1Unamaped.PropDefId = Nplan;
if (!string.IsNullOrEmpty(oProp[1])) { propInstPar1Unamaped.Val = oProp[1].Trim(charsToTrim); } else { propInstPar1Unamaped.Val = oProp[1]; }
propInstParamsUnamaped.Add(propInstPar1Unamaped);
}
if (pIndice != null)
{
if (pIndice.DispName != null)
{
mIndice = pIndice.Moniker;
var addIndice = new ListPropReq(mIndice, oProp[3]);
ListDesPropReq.Add(addIndice);
}
else
{
PropInstParam propInstPar3Unamaped = new PropInstParam();
propInstPar3Unamaped.PropDefId = Indice;
propInstPar3Unamaped.Val = oProp[3];
propInstParamsUnamaped.Add(propInstPar3Unamaped);
}
}
else
{
PropInstParam propInstPar3Unamaped = new PropInstParam();
propInstPar3Unamaped.PropDefId = Indice;
propInstPar3Unamaped.Val = oProp[3];
propInstParamsUnamaped.Add(propInstPar3Unamaped);
}
if (pDescription1 != null)
{
if (pDescription1.DispName != null)
{
mDescription1 = pDescription1.Moniker;
if (!string.IsNullOrEmpty(oProp[4]))
{
var addDescription1 = new ListPropReq(mDescription1, oProp[4].Trim(charsToTrim));
ListDesPropReq.Add(addDescription1);
}
else
{
var addDescription1 = new ListPropReq(mDescription1, oProp[4]);
ListDesPropReq.Add(addDescription1);
}
}
else
{
PropInstParam propInstPar4Unamaped = new PropInstParam();
propInstPar4Unamaped.PropDefId = Description1;
if (!string.IsNullOrEmpty(oProp[4])) { propInstPar4Unamaped.Val = oProp[4].Trim(charsToTrim); } else { propInstPar4Unamaped.Val = oProp[4]; }
propInstParamsUnamaped.Add(propInstPar4Unamaped);
}
}
else
{
PropInstParam propInstPar4Unamaped = new PropInstParam();
propInstPar4Unamaped.PropDefId = Description1;
if (!string.IsNullOrEmpty(oProp[4])) { propInstPar4Unamaped.Val = oProp[4].Trim(charsToTrim); } else { propInstPar4Unamaped.Val = oProp[4]; }
propInstParamsUnamaped.Add(propInstPar4Unamaped);
}
if (pDescription2 != null)
{
if (pDescription2.DispName != null)
{
mDescription2 = pDescription2.Moniker;
if (!string.IsNullOrEmpty(oProp[5]))
{
var addDescription2 = new ListPropReq(mDescription2, oProp[5].Trim(charsToTrim));
ListDesPropReq.Add(addDescription2);
}
else
{
var addDescription2 = new ListPropReq(mDescription2, oProp[5]);
ListDesPropReq.Add(addDescription2);
}
}
else
{
PropInstParam propInstPar5Unamaped = new PropInstParam();
propInstPar5Unamaped.PropDefId = Description2;
if (!string.IsNullOrEmpty(oProp[5])) { propInstPar5Unamaped.Val = oProp[5].Trim(charsToTrim); } else { propInstPar5Unamaped.Val = oProp[5]; }
propInstParamsUnamaped.Add(propInstPar5Unamaped);
}
}
else
{
PropInstParam propInstPar5Unamaped = new PropInstParam();
propInstPar5Unamaped.PropDefId = Description2;
if (!string.IsNullOrEmpty(oProp[5])) { propInstPar5Unamaped.Val = oProp[5].Trim(charsToTrim); } else { propInstPar5Unamaped.Val = oProp[5]; }
propInstParamsUnamaped.Add(propInstPar5Unamaped);
}
PropInstParamArray propInstParamsArrayUnamaped = new PropInstParamArray();
propInstParamsArrayUnamaped.Items = propInstParamsUnamaped.ToArray();
PropInstParamArray[] propInstParamsArraysUnamaped = new PropInstParamArray[1];
propInstParamsArraysUnamaped[0] = propInstParamsArrayUnamaped;
// check out the file pour propMapper
ByteArray downloadTicket1;
Autodesk.Connectivity.WebServices.File checkedOutFile = m_conn.WebServiceManager.DocumentService.CheckoutFile(
Ligne.IDFichierVault, CheckoutFileOptions.Master, Environment.MachineName, @"c:\temp",
"MàJ Prop de KLIO", out downloadTicket1);
// CtntSrcPropDef selectedProp = selectedItem.Property;
downloadTicket1 = null;
if (ListDesPropReq != null)
{
PropWriteReq[] writeProps = ListDesPropReq.Select(
p => new PropWriteReq()
{
Moniker = p.MonikerString,
CanCreate = false,
Val = p.NouvelleValeur
}
).ToArray();
PropWriteResults results;
// on mets les UDP non mappé en même temps
m_conn.WebServiceManager.DocumentService.UpdateFileProperties(new long[] { Ligne.MasterIDFichierVault }, propInstParamsArraysUnamaped);
byte[] uploadTicket = new byte[] { };
try
{
uploadTicket = m_conn.WebServiceManager.FilestoreService.CopyFile(
downloadTicket.Bytes, true, writeProps, out results);
}
catch
{
ErreurCopy = true;
}
if (ErreurCopy == true)
{
Console.WriteLine("ErreurCopy :" + Ligne.NomFichierVault);
}
else
{
Autodesk.Connectivity.WebServices.File newFile = m_conn.WebServiceManager.DocumentService.CheckinUploadedFile(
checkedOutFile.MasterId, "MàJ Propriétés KLIO", false, System.DateTime.Now,
null, null, true, checkedOutFile.Name, checkedOutFile.FileClass,
checkedOutFile.Hidden, uploadTicket.ToByteArray());
}
}
else
{
m_conn.WebServiceManager.DocumentService.UpdateFileProperties(new long[] { Ligne.MasterIDFichierVault }, propInstParamsArraysUnamaped);//{Ligne.IDFichierVault},propInstParamsArrays);
checkedOutFile = m_conn.WebServiceManager.DocumentService.CheckinUploadedFile(checkedOutFile.MasterId, "MàJ Propriétés KLIO", false, System.DateTime.Now, null, null, false, checkedOutFile.Name, checkedOutFile.FileClass, checkedOutFile.Hidden, null);
}
}
}
if (lext != "dwg")//toutes les autres extensions sauf dwg
{
ByteArray oByteArray;
Autodesk.Connectivity.WebServices.File oVaultFile = m_conn.WebServiceManager.DocumentService.CheckoutFile(Ligne.IDFichierVault, CheckoutFileOptions.Master, Environment.MachineName, @"C:\\ProjetVault\\coffre\\000", @"PropUpdate", out oByteArray);
m_conn.WebServiceManager.DocumentService.UpdateFileProperties(new long[] { Ligne.MasterIDFichierVault }, propInstParamsArrays);//{Ligne.IDFichierVault},propInstParamsArrays);
oVaultFile = m_conn.WebServiceManager.DocumentService.CheckinUploadedFile(oVaultFile.MasterId, "MàJ Propriétés KLIO", false, System.DateTime.Now, null, null, false, oVaultFile.Name, oVaultFile.FileClass, oVaultFile.Hidden, null);
}
}
}
}
MessageBox.Show("MàJ finit !");
}
