Revit Test Framework improvements

Anonymous — Wed, 08 Aug 2018 22:33:25 GMT

Hello everyone,

We are currently developing a pretty elaborate Revit plugin (you can read more about it here). During the course of this work we've ran into many issues (and surely will run into many more) and this forum has helped us resolve many of them.

We'd like to contribute something back to the community by telling you about the improvements we've made to the awesome RevitTestFramework (RTF) by the Dynamo team.

@jeremytammikhas talked about RTF on his blog in the past so if you're not familiar with it, you can read about it here. It's an invaluable tool if you are developing a Revit addin as it allows you to run integration tests and control Revit in an automated fashion.

However, the RTF hasn't had much development done on it in a while (other than making it compatible with Revit 2019) and as such we found a few things lacking; the biggest such shortcoming was a lack of Nuget package making usage of the RTF on a build server/continuous integration server very difficult. Along the way, we made a few other improvements that I think you will find interesting and useful.

Summary of main changes we've made so far:
1. Created a Nuget package,

you can download it here: https://www.nuget.org/packages/revittestframework

2. Added the ability to group tests by the model
Opening a new RVT file for each test significantly slows down the execution of the tests. We wanted our Revit tests to run for each pull request our engineers make, so it has to be fast (and reliable).
The groupByModel option significantly improves performance if you have multiple tests that operate on the same model.
When using --groupByModel the RTF will order the tests such that all tests that operate on the same model file are executed sequentially without closing and reopening the model. Naturally, it requires the --continuous parameter to be effective.

Also, goes without saying that if your tests leave side-effects on a model which may break a subsequent test this option will not work for you. Luckily for us, none of our tests have side-effects (which, in general, is a good practice).

3. Added ability to use wildcards for model filenames
We have multiple models which we want to run a particular test on. Today, every time we make a new model we have to copy/paste a unittest with a new model file.
To make this simpler, you can now specify a wildcard for your model file on a test and RTF will enumerate all files in the directory and run the test on each one. For example:

[TestFixture]
public class TestAllModels
{
    [Test, TestModel(@"C:\Models\test_models_2019\*.rvt")]
    public void SomeTest()
    {
        ...
    }
}

will run SomeTest on every RVT file it finds under C:\Models\test_models_2019.
One of the ways we use this is for performance testing. All that this test does is measure the speed with which we finish processing a given model. This way, we can run the test suite on each release and assess whether we have significantly regressed our performance from our previous release.

4. Clear messaging & indication of failures
Engineers don't love writing tests and they hate debugging tests. Especially, when diagnosing what went wrong is very difficult.
We wanted to make sure that an engineer could:
a) look at the log output from a build server and quickly be able to tell what went wrong and where to start debugging
b) run the test easily with nice UI on their local computer to quickly iterate on a solution to fix the problem

This included:

Clear indication when a model file is not found (prior to that RTF would just silently complete the test suite without errors)
Automatically expand failing tests in the UI so that errors are super obvious
Have a clear summary at the end of a test run for the number of passed/failed/etc tests. This way you can go grab a coffee while the tests run and know if there were any errors with a quick glance at the summary
We utilize categories (e.g. [Category("Doors")]) for grouping. But RTF UI didn't show tests without a category, this is now fixed
All Console.PrintLine() messages from the actual unit tests are now sent back to the RTF server so you can see them in a single contiguous log - yay (this one is my favorite)!
Test completion information is displayed in the console as soon as the test itself is completed, RTF used to wait till all tests have finished before showing you the status of all individual tests, now each pass/fail is printed as soon as the test is completed.

We've also made a bunch of small bug fixes around the stability of the RTF. There is still work to be done on the framework, but we hope you find these changes useful.

We are working with the Dynamo team to bring all these changes back to the main branch of RTF, but for now, you are welcome to contribute with us in our forked repo.

Mark - Engineering at UpCodes

p.s. we are hiring - if you love building Revit addins please reach out!

Re: Revit Test Framework improvements

jeremytammik — Mon, 20 Aug 2018 10:48:47 GMT

Dear Mark,

Thank you very much for your appreciation and sharing this.

It looks like a great piece of work to me, very useful indeed.

I'll repost your contribution on The Building Coder, if I may.

Thank you!

Cheers,

Jeremy

Re: Revit Test Framework improvements

Anonymous — Tue, 21 Aug 2018 00:07:46 GMT

Thank you, @jeremytammik! That would be great, if you don't mind, please attribute these improvements to UpCodes instead of just me, since this was an effort from our whole team.

Mark

UpCodes Engineering

Re: Revit Test Framework improvements

jeremytammik — Tue, 21 Aug 2018 08:36:23 GMT

Dear Mark,

Thank you very much. Done:

http://thebuildingcoder.typepad.com/blog/2018/08/revit-unit-test-framework-improvements.html

Cheers,

Jeremy

Re: Revit Test Framework improvements

Anonymous — Fri, 31 Aug 2018 18:20:15 GMT

Thank you, Jeremy!

Btw, in case you are interested how we run RTF in our automated CI, you can read the instructions here:

https://github.com/upcodes/RevitTestFramework/blob/mark/Revit2019/docs/using_with_ci.md

Mark

Re: Revit Test Framework improvements

jeremytammik — Thu, 06 Sep 2018 16:53:55 GMT

Dear Mark,

Thank you for pointing them out!

Duly noted.

Cheers,

Jeremy

Re: Revit Test Framework improvements

哀家爆 — Thu, 10 Dec 2020 16:07:56 GMT

I'm very interested in your tools. But it seems that there is too little introduction about how to use the tool. I mean, maybe there are a lot of persons trying the tool, but don't know how to open the door...

Maybe we need something like, say, Getting Started with RTF, step1, 2, 3,... .

More details, more better

topic Re: Revit Test Framework improvements in Revit API Forum

Revit Test Framework improvements

Re: Revit Test Framework improvements

Re: Revit Test Framework improvements

Re: Revit Test Framework improvements

Re: Revit Test Framework improvements

Re: Revit Test Framework improvements

Re: Revit Test Framework improvements