Not able to delete Extensible Storage schema

[snajjar]

I'm trying to put together a command to erase Extensible Storage (ES) data and the corresponding Schemas for our addin in case if unwanted by the users or corrupt. I had no luck so far. 

After the ES data has been written to the file it won't go away. I am being able to delete the DataStorage elements with no troubles, but the schemas keep popping up after erasing them.

The procedure I'm trying is:

1. Open the Revit file with the saved ES data.
2. Delete all the DataStorage elements holding the data.
3. Save and close Revit, reopen the file again.
4. Erase all schemas created by our addin.
5. Save and close Revit to make sure I clear the schemas in memory.
6. Start Revit again.
7. Use the Schema.ListSchemas() function with no documents open to make sure our custom schemas are not loaded, this clears out.
8. Open the file and use the Schema.ListSchemas() function, now all the schemas that I have erased previously reappear after opening the file!

I'm using the same procedure found in the "ExtensibleStorageUtility" code example in the Revit SDK in a macro (code below).

One observation I have that if I close Revit without saving after step 4 I get no warning to save the file as if no changes happened to the file, although I'm doing the erase inside a transaction.

This article has some info on ES and Schema behaviors that I tried to utilize but didn't work.

Below is the macro function I'm using:

		public void DeleteSchemas() {
			
			var uiDoc = this.ActiveUIDocument;
			var document = uiDoc.Document;
			
			var message = "";				
			IList<Schema> schemas = Schema.ListSchemas();
			int deleted = 0;
			int count = schemas.Count;
			
			using(Transaction tErase = new Transaction(document, "Erase EStorage")){
				
				tErase.Start();
				
				foreach (Schema schema in schemas.Where(s=> s.VendorId == "MYAD"))
				{
					//Note-this will delete storage of this schema in *all* open documents.
					try{
						document.EraseSchemaAndAllEntities(schema);
						Schema.EraseSchemaAndAllEntities(schema, true);
						deleted++;
					} catch {
						
					}
				}
				tErase.Commit();
			}
			
			message =string.Format( "{0} storage elements out of {1} were deleted.", deleted.ToString(), count.ToString());
			
			TaskDialog.Show("ExtensibleStorageUtility", message);
		}

Any light on this would be much appreciated.

Thanks!

[jeremy_tammik]

Beautiful! Thank you for testing, reproducing, and sharing the proof of the pudding!

[jeremy_tammik]

Dear Marek,

Thank you for your report, clear description, and sample material.

Sorry to hear about this.

I logged the issue REVIT-196204 [IFC link prevents extensible storage delete] with our development team for this on your behalf as it requires further exploration and possibly a modification to our software. Please make a note of this number for future reference.

You are welcome to request an update on the status of this issue or to provide additional information on it at any time quoting this change request number.

This issue is important to me. What can I do to help?

This issue needs to be assessed by our engineering team and prioritised against all other outstanding change requests. Any information that you can provide to influence this assessment will help. Please provide the following where possible:

• Impact on your application and/or your development.
• The number of users affected.
• The potential revenue impact to you.
• The potential revenue impact to Autodesk.
• Realistic timescale over which a fix would help you.
• In the case of a request for a new feature or a feature enhancement, please also provide detailed Use cases for the workflows that this change would address.

This information is extremely important. Our engineering team has limited resources, and so must focus their efforts on the highest impact items. We do understand that this will cause you delays and affect your development planning, and we appreciate your cooperation and patience.

Best regards,

Jeremy

[Songohu_85]

Dear Jeremy,

InternalException when trying to remove schemas is currently not a big issue for me.

I landed on this thread because of difficulties related to upgrade from Revit2020 to Revit2023. In a few models we experience some critical errors when trying to upgrade to version 2023. We came to the point that it could be directly or indirectly related to schemas. After we remove ExStorage schemas the upgrade to 2023 is successful.

I noticed that removing schemas and entities sometimes causes Errors and Warnings, for example:
Error: Can't keep elements joined. (Model Lines, Walls)
Warning: Highlighted lines overlap. Lines may not form closed loops. (Model Lines)
Edit: But most likely it is just because the elements are regenerated after the entitiy is removed.

I wouldn't believe that ExStorages could cause any upgrade issues  if I didn't see it happening.

What is your opinion on the possibility that ExtensibleStorage schemas and entities may cause upgrade issues?

Kind regards

Marek

[RPTHOMAS108]

A link opens a document in the background but you can't edit that linked document can you.

Does the linked document contain the same schema itself?

If so, then I would see it as logical why the linked document causes issues. Should not cause an exception though.

[jeremy_tammik]

Extensible storage schema is an application-wide object. If it exists at all in the application, it will populate and "infect" every single document that you touch. That makes it hard to remove, and complicated to understand.

[jeremy_tammik]

Dear Marek,

Thank you for your patience

The development team analysed the issue REVIT-196204 [IFC link prevents extensible storage delete] and decided not to make any changes, explaining:

The Document class provides the method EraseSchemaAndAllEntities:

  # Erases Schema and all its Entities from the document.
  remarks
     # The Schema remains in memory.
  in ESSchema* schema
     # The Schema to erase.
  since 2021
  validate DocumentValidation::isDocumentModifiable(this)
  throws ArgumentException
     # No write access to this Schema.

The version of EraseSchemaAndAllEntities erasing Schema from all open documents had been deprecated and removed:

• https://www.revitapidocs.com/2023/80983aac-0cca-c211-1c7b-b5350624f046.htm

• 2023 | Resource Not Available for the Active API Year: EraseSchemaAndAllEntities Method -- Erases all Entities corresponding to this Schema from all open documents and erases this Schema from memory.

The document method should be used instead:

• https://www.revitapidocs.com/2023/50debcb0-3c4f-b32b-2edb-8a6ef7b4bf8d.htm

• Erases Schema and all its Entities from the document.

Best regards,

Jeremy

[jgrayYXDAB]

OK i guess i need to jump on board the Schema issue.  I'm also getting an error for an EnigmaSchema that is in multiple files.  No idea how to get this to be removed.  Any help would be appreciated.

[RPTHOMAS108]

You have to write a macro/addin that calls the above function and if that doesn't work it is usually due to entities on elements in nested families etc.  So it is not easy because sometimes it involves iterating families opening them calling the above and reloading them. This issue is also evident on links where the documents for such are loaded in the background and are not editable from the file they are linked into.

Once a schema is loaded then a different one loaded with the same id will cause this. Developers often don't appreciate that even changing the documentation for a schema makes it different. Any change to the structure of the schema or documentation makes it different. In this case prevention is better than cure because the cure is very hard to achieve.

Not sure who wrote the EnigmaSchema but they obviously wanted to leave no details.

What I've also noticed in the past (not sure it is still true) is that once you get this dialogue then you can't remove the schema because it is in conflict. So in the case of links for example you had to unload them all and deal with each file separately i.e. to have Revit always in a state where the conflict had not arisen when you go to remove the schema. It made removing them from loaded families virtually impossible. Hopefully things are not that hard anymore.

See also about purging which may work from an end user perspective

[JeffStuy]

Thanks for looking into this.  However, the Document version of EraseSchemaAndAllEntities also is leaving the schemas behind and not removing them from the model.  That is, I create a DataStorage element, and populate it.  Then, to test, I remove the schema using Document.EraseSchemaAndAllEntities, remove the DataStorage element, save the file, re-open the file - the Schema still exist and the DataStorage element is gone as expected.

[jeremy_tammik]

Wow. Oh dear. That sounds unexpected. What version of Revit are you talking about? Did you also include the step recommended above and execute purge after deleting the extensible storage entity and the DataStorage element? If so, can you please provide a complete minimal reproducible case for detailed analysis and possible rectification by the development team? Thank you!

https://thebuildingcoder.typepad.com/blog/about-the-author.html#1b

[JeffStuy]

I did purge as suggested above with mixed results - see below.

I have tried to create a minimal example but I am in the middle of creating a "generic" storage system and, in order to save time, I created an example by extracting only the parts needed from this.

Scenario 2, below, applies to 2021, 2022, & 2023

I use Revit DB Explorer to check the status of the elements.

I have two scenario:

both start with a new, "empty" model.

scenario 1 - "purge first"

1. open model
2. run ExStoreTest
3. Write
4. Delete
5. Exit
6. check schemas: they are still in memory (expected)
7. purge unused: purge all
8. close & save model
9. open model
10. schemas do not exist

applies to 2021, 2022, & 2023
scenario 2 - "save first"
1. open model
2. run ExStoreTest
3. Write
4. Delete
5. Exit
6. check schemas: they are still in memory (expected)
7. close & save the model
8. open the model
9. check schemas: they ARE still in memory (not expected)
10. purge unused: no ex storage schemas listed
11. purge all anyway
12. close & save the model
13. open the model
14. check schemas: they ARE still in memory (not expected)

But of course, I cannot have the user manually, purge the schemas - especially, in my case, they may change multiple times during a session.

[jeremy_tammik]

Dear Jeff,

Thank you for your report and sample material.

Sorry to hear about this.

I logged the issue REVIT-199932 [Unable to delete extensible storage] with our development team for this on your behalf as it requires further exploration and possibly a modification to our software. Please make a note of this number for future reference.

You are welcome to request an update on the status of this issue or to provide additional information on it at any time quoting this change request number.

This issue is important to me. What can I do to help?

This issue needs to be assessed by our engineering team and prioritised against all other outstanding change requests. Any information that you can provide to influence this assessment will help. Please provide the following where possible:

• Impact on your application and/or your development.
• The number of users affected.
• The potential revenue impact to you.
• The potential revenue impact to Autodesk.
• Realistic timescale over which a fix would help you.
• In the case of a request for a new feature or a feature enhancement, please also provide detailed Use cases for the workflows that this change would address.

This information is crucial. Our engineering team has limited resources and must focus their efforts on the highest impact items. We do understand that this will cause you delays and affect your development planning, and we appreciate your cooperation and patience.

Best regards,

Jeremy

[JeffStuy]

In answer to your questions:

Impact on your application and/or your development: I hesitate to use Extensible Storage as I do not want to pollute user’s models with left-over schema’s they cannot delete.  In addition, I am developing an add-in that will require the usage of Extensible Storage – given this issue, I am considering canceling this project.

The number of users affected: I have one app that uses Extensible Storage.  I do not know the number of users but I’d guess around 50 to 100.

The potential revenue impact to you: Cannot estimate as I cannot determine if my add-in in development will be popular / be purchased.

The potential revenue impact to Autodesk: I doubt that the users will care about this enough to affect AutoDesk’s revenue.

Realistic timescale over which a fix would help you: I am a single person developer.  I will take me a while – 6 to 9 months at least, to develop my add-in.  The fix would be needed by then.  But more important would be an assurance that there will be a fix coming in a reasonable time frame – that is within 6-months – then I would need to cancel my project.

The other questions do not apply.

The bigger question, not asked, is how many other developers are using Extensible Storage and having their voided schema polluting user’s models and what affect this has on those models. 

[jeremy_tammik]

Dear Jeff,

Thank you for your business case. I added it to the ticket REVIT-199932 [Unable to delete extensible storage].

Best regards,

Jeremy

[josephsWYW3U]

Need to lean in on this. 

We are experiencing something similar. Somehow we got a schema in our models (various  models from across the firm) that we absolutely cannot get rid of despite deleting the families, and purging the schema's. This has caused issues when linking in consultant models. The conflicting Schema's "Enigma Schema" and "TraceIt" Schema.

We've never heard of, or used either plugin, however the company that created TraceIt noted this as an issue and they had a plugin that would automatically remove the schema when a model was loaded. When Autodesk updated their API in 2021, it broke their fix. We're currently experiencing loss productivity on one model when linking in an Electrical Revit model for coordination.

I've identified the schema sporadically in other projects, and am trying to find out what it's connected to and how to get it out of the models completely. We were able to track the conflict down to two families, but even when we isolate them and purge the schema's, they come right back.

[jeremy_tammik]

I believe I have heard of people occasionally submitting their corrupt RVT or RFA to Autodesk product support to have it purged or otherwise made reusable and accessible. Can you try to find out if that is possible for you too, via product support? If you cannot find a way to get them to help you, and worst comes to worst, you can make the files available here or elsewhere and I can try to pass them on to the development team for fixing. It's neither their job nor mine, so I would much prefer if the product support team can provide assistance and help you get the problem fixed. Thank you.

[josephsWYW3U]

Yep, we're working with customer support to try to resolve the issue on this project. The support personnel has been great. Right now we have a work-around and have identified the families. It's a large active project that we can't stop production on currently, so we're running tests to confirm how to fix it, and will send it in for fixing as a last resort.

Ultimately however, I believe we're going to need to have the ability to delete schema's in models again. While it's a pressing issue on this specific project, I've found the schema in other models as well. While they're not currently giving conflict errors, there is every potential for them to do so in the future when other subs come onto the project.

I'm working internally to figure out where the schema is coming from, but even if I do, if I can't delete the schema we'll be unable to avoid the issue.

[jeremy_tammik]

Very glad to hear you are working fruitfully with product support. I find it hard to believe that a situation can arise in which it is impossible to delete a schema, if you really process individual file by file. I assume you have read Richard  @RPTHOMAS108  Thomas' post above?  I summarised all I have heard about it here as well, probably mostly or completely based on this thread:

https://thebuildingcoder.typepad.com/blog/2022/11/extensible-storage-schema-deletion.html

Maybe if you can share one of the problematic files plus the code that is supposed to delete the schema, packaged as a complete, minimal, reproducible place, I can either add it to the development ticket  REVIT-199932  mentioned above or even create a new one for your case. 

The API is officially declared to support this straightforward task, so let's ensure it really does.

[jeremy_tammik]

Dear Jeff and other interested parties,

Thank you for your patience with this. The development team completed the analysis of the ticket REVIT-199932 [Unable to delete extensible storage] and concluded that a code fix is required. They opened a new ticket REVIT-203545 [Unable to delete extensible storage] for this work to be performed. Please make a note of this number for future reference.

Best regards,

Jeremy

[jeremy_tammik]

Good news! The development team completed work on REVIT-203545 [Unable to delete extensible storage] and scheduled it for inclusion in an update release of Revit 2024.