Hello everyone,
We are currently developing a pretty elaborate Revit plugin (you can read more about it here). During the course of this work we've ran into many issues (and surely will run into many more) and this forum has helped us resolve many of them.
We'd like to contribute something back to the community by telling you about the improvements we've made to the awesome RevitTestFramework (RTF) by the Dynamo team.
@jeremytammikhas talked about RTF on his blog in the past so if you're not familiar with it, you can read about it here. It's an invaluable tool if you are developing a Revit addin as it allows you to run integration tests and control Revit in an automated fashion.
However, the RTF hasn't had much development done on it in a while (other than making it compatible with Revit 2019) and as such we found a few things lacking; the biggest such shortcoming was a lack of Nuget package making usage of the RTF on a build server/continuous integration server very difficult. Along the way, we made a few other improvements that I think you will find interesting and useful.
Summary of main changes we've made so far:
1. Created a Nuget package,
you can download it here: https://www.nuget.org/packages/revittestframework
2. Added the ability to group tests by the model
Opening a new RVT file for each test significantly slows down the execution of the tests. We wanted our Revit tests to run for each pull request our engineers make, so it has to be fast (and reliable).
The groupByModel option significantly improves performance if you have multiple tests that operate on the same model.
When using --groupByModel the RTF will order the tests such that all tests that operate on the same model file are executed sequentially without closing and reopening the model. Naturally, it requires the --continuous parameter to be effective.
Also, goes without saying that if your tests leave side-effects on a model which may break a subsequent test this option will not work for you. Luckily for us, none of our tests have side-effects (which, in general, is a good practice).
3. Added ability to use wildcards for model filenames
We have multiple models which we want to run a particular test on. Today, every time we make a new model we have to copy/paste a unittest with a new model file.
To make this simpler, you can now specify a wildcard for your model file on a test and RTF will enumerate all files in the directory and run the test on each one. For example:
[TestFixture] public class TestAllModels { [Test, TestModel(@"C:\Models\test_models_2019\*.rvt")] public void SomeTest() {
... } }
will run SomeTest on every RVT file it finds under C:\Models\test_models_2019.
One of the ways we use this is for performance testing. All that this test does is measure the speed with which we finish processing a given model. This way, we can run the test suite on each release and assess whether we have significantly regressed our performance from our previous release.
4. Clear messaging & indication of failures
Engineers don't love writing tests and they hate debugging tests. Especially, when diagnosing what went wrong is very difficult.
We wanted to make sure that an engineer could:
a) look at the log output from a build server and quickly be able to tell what went wrong and where to start debugging
b) run the test easily with nice UI on their local computer to quickly iterate on a solution to fix the problem
This included:
We've also made a bunch of small bug fixes around the stability of the RTF. There is still work to be done on the framework, but we hope you find these changes useful.
We are working with the Dynamo team to bring all these changes back to the main branch of RTF, but for now, you are welcome to contribute with us in our forked repo.
Mark - Engineering at UpCodes
p.s. we are hiring - if you love building Revit addins please reach out!
Dear Mark,
Thank you very much for your appreciation and sharing this.
It looks like a great piece of work to me, very useful indeed.
I'll repost your contribution on The Building Coder, if I may.
Thank you!
Cheers,
Jeremy
Thank you, @jeremytammik! That would be great, if you don't mind, please attribute these improvements to UpCodes instead of just me, since this was an effort from our whole team.
Mark
UpCodes Engineering
Dear Mark,
Thank you very much. Done:
http://thebuildingcoder.typepad.com/blog/2018/08/revit-unit-test-framework-improvements.html
Cheers,
Jeremy
Thank you, Jeremy!
Btw, in case you are interested how we run RTF in our automated CI, you can read the instructions here:
https://github.com/upcodes/RevitTestFramework/blob/mark/Revit2019/docs/using_with_ci.md
Mark
Dear Mark,
Thank you for pointing them out!
Duly noted.
Cheers,
Jeremy
I'm very interested in your tools. But it seems that there is too little introduction about how to use the tool. I mean, maybe there are a lot of persons trying the tool, but don't know how to open the door...
Maybe we need something like, say, Getting Started with RTF, step1, 2, 3,... .
More details, more better