Unit tests for Fusion 360 add-ins and running them on the CI/CD server

[rbin_robotik]

Hello,

I have been developing some Fusion 360 add-ins for a while. I am using the Python API mainly but I'll be switching to C++ API very soon. Some add-ins are getting more complex and testing all the features takes a lot of time. I am wondering if it is possible to write tests, for instance; using pytest, and run them on the CI/CD server. I've found this topic from 2017, I am not sure if it is still valid but as far as I can see, it suggests using events and text commands.  I also remember reading about creating mock objects but it didn't sound convenient at that time.

I am aware that there might be some major issues, ex: licensing, ability to run a GUI app on the CI/CD server due to various limitations and others. The licensing might not be a big problem, but technical issues might be; like GUI apps and headless systems won't do together very well (I am hoping I am wrong here). I believe there might be some other alternatives, like robotic process automation, but I haven't tested such alternatives yet.

I'd really appreciate if you have any other suggestions regarding this issue.

Thanks in advance!

[BrianEkins · Accepted solution]

Nothing has been added to the API to explicitly support creating or running unit tests. However, the suggestion from Marshal in the post you linked to is still valid. Here's a short summary, along with an additional idea.

Your add-in creates command definitions and for each command definition should set up a command created event handler. When the command is executed, Fusion will fire the command created event for that command, which you can handle and do whatever that command is supposed to do. A command can be executed in three different ways.

First, and the most common, is that when you created the command definition, you also created a button in the UI that the user can use to run the command.

The second is running the command through code. You can obtain any command, both native Fusion commands and those created by add-ins, by using the CommandDefinitions collection. Once you have the CommandDefinition object associated with the command you want, you can call the execute method to run the command.

The third method is what Marshal described, where you can execute the command using the TEXT COMMAND window by running the "Commands.start" command and providing the name of the command definition.

The suggestion is that your add-in can support some additional commands that would be used only for testing. For these commands, you wouldn't want to use option 1 because you don't want to expose these to the user. Option 2 is probably what makes the most sense from a testing perspective because then your add-in or a separate script can execute the testing commands.

I don't know how Fusion will behave on a server with no graphics display. It's not intended to be run that way and is not a tested or supported configuration.

[rbin_robotik]

Hi @BrianEkins , thank you very much for the detailed answer and the additional idea.

It makes sense to me add unit tests right in the add-in code. It is usually a common practice (and makes the developer's life easier) when you develop a individual library/application, or an add-in/plug-in for a software suite.

It feels like the 2nd option might be applied through a script, i.e. create the script through the "Scripts and Add-Ins" menu option and add the testing code (like you described or in various different ways) to the newly created script. Whenever the developers need testing, they could run the script and see the outcome.

Perhaps the 3rd option might also be applied through a script, and to be honest, I like the idea.

In addition, I have been exploring what hidden capabilities Fusion 360 has and the simplest method would be calling the main executable with "--help" argument to see if it shows anything useful. Actually, I can get the following output on MacOS Ventura with the latest Fusion 360 as of today.

➜ MacOS ./Autodesk\ Fusion\ 360 --help
Following are the existing command line options in the system (case in-sensitive):
--help, -? Display the help information.
--showInternal Display internal command line actions to the user.
--nologo Disable splash screen
--exit Shutdown the application before loading Addins
--exitAfterAddIn Shutdown the application after loading Addins
--safeMode Run in safe mode that disables automatic script loading and certain optimization
--serviceUtil Run the Fusion service utility
--activateTestingContext Activate testing context
--standaloneNTestStartFusion The flag represent fusion is started up by standalone NTest
--AutoCAD Show return to AutoCAD button
--Inventor Show return to Inventor button
--TestingDashboard Set testing flag to disable dialogs to support automation of dashboard JS tests
--ClearUserCache Try to delete Fusion user's cloud cache folders when we are online.
--offlineStart Start from offline
--script, -s <Script> Execute the given script after loading addins
--execute, -e <TextCommand> Execute the given text command after loading addins
--enable <Addin1_AppName Addin2_AppName...> Enable the given addins (in double-quotes)
--disable <Addin1_AppName Addin2_AppName...> Disable the given addins (in double-quotes)
--print Not implemented yet
--log <Log file> Save command-line output into a log file
--Diagnostic, -d Dump out user's diagnostics data to a zip file via command-line
--autologin username,password
--autoregister firstname,lastname,email,password
--fullTest [Result file] Run full regression test after UI loaded
--perfTest <Baseline> [TestCase|TranscriptFileOrDir] [/log <Result file>] Run performance tests after UI loaded, and exit the application after running the tests. Either relative path to "Performance" folder or a full path is acceptable in Baseline and TranscriptFileOrDir

It looks like there are some ways to run scripts and text commands through the command-line interface and a little bit more that we can do too. I haven't tested if they actually work, but I would think they should.

Regarding running Fusion 360 on a headless server, I was actually thinking positively before your answer due to how the coding examples are structured on Fusion 360 documentation. For instance, it is very common to see a structure in a Fusion 360 example like this:

import adsk, adsk.core

app = adsk.core.Application.get()
ui = app.userInterface

# Some script body here

try:
  # Do something
except:
  # Is there a state where ui is not available?
  if ui:
    ui.messageBox("Error occurred")
  # Log to the console or file
  app.log("Error occurred with details", adsk.core.LogLevels.ErrorLogLevel, adsk.core.LogTypes.ConsoleLogType)

I always thought that checking the availability of the the "ui" object was mainly to understand whether Fusion 360 started with a GUI or not. For sure, unavailability of the "ui" object might happen due to various reasons (which are mainly fixable), but it always felt like there might be a non-UI way to run it. I guess I understood that part a bit differently as you said, it wasn't intended to run this way.

[BrianEkins]

Some terms are used internally for things that are different than the equivalent in the API. For example, an internal "add-in" has nothing to do with an API add-in, so the available command line options are different than what you think.

However, you can create your own command line arguments for Fusion. If you add a command line argument, Fusion will log that it encountered an unknown command line argument but will continue to start up. Fusion starts your add-in automatically during start-up, and you can read the command lines argument using the standard Python sys.argv function. Using this, your add-in can check to see if the command line argument exists and if it does, it will run in a testing mode.

The other alternative is to create a script that would execute a command created by your add-in to do the testing. Or you can run the text command to execute the same testing command. With these two options, your add-in does the same thing by creating the special test command, and then it's just a question of how you execute that command. Using the command line option you suggested, you wouldn't need to create a test command, but the add-in will either run normally or in the test mode based on whether the command line switch exists.

[rbin_robotik]

Thank you for your response and all the details, @BrianEkins ! They all make better sense now.

I like the idea of reading from sys.argv to launch tests, but I'll need to use C++ for some add-ins in the near future. Still, it is a very neat idea. I guess I might also combine script/add-in and text commands, e.g. add the tests as commands and use the script to run them, as an alternative to reading from sys.argv. There might be even a way to automate via creating an add-in and run a multithreading.Thread to check for changes on the source code repository and run the tests as the code gets updated. It is a bit tricky to implement, but it can be achievable. However, this might not be the solution to make the Fusion 360 to run in perfectly headless mode, but it would be as close as possible, I guess.

Thanks again for your responses and ideas.

P.S. I am not sure how possible, but for the sake of implementation of unit tests on a headless server, it would be great to have an option to run Fusion 360 without the GUI. Would it be possible to request a feature like this? I am not sure if this is the best place to write about feature requests, I just wanted to try my chances 🙂