Action log for Educational versions

[OG-Work]

Dear Autodesk,

Problem:

I work at a local university and have among other things been responsible for "cheat-detection" in Revit models that students submitted. Cheating in this case, means copying the assignment or parts of it from another student.

I begun by developing a small application that compares Revit-UIDs between each students submissions.

It works well for detecting large similarities, such as students sharing a project (single new project) and changing small parts of it before submitting.

Since Revit-UIDs are not unique between projects, and each new instance gets a new UID, this solution does not detect the copying of instances between projects and similar scenarios.

Solution:

My colleagues and I would love to see a feature like an action-log that could show us dubious behavior such as a large amount of instances being pasted into the project. I'm sure other teachers would love this as well!

I've discussed this idea with a developer at Autodesk, and he said there's already something similar implemented behind the scenes - not exposed by the public API. It would be fantastic if you could expose parts of this functionality or build a small action-log upon it.

Preferably this action-log should be usable without the need for scripting etc, since many of the teachers may struggle with this.

I hope you find this idea appealing and that it may help other teaching facilities as much as it would help us.

Best regards,

Oscar

[boostyourbim]

@OG-WorkPlease take a look at https://boostyourbim.wordpress.com/2020/12/09/how-when-were-your-revit-objects-created/

[boostyourbim]

@OG-WorkPlease take a look at https://boostyourbim.wordpress.com/2020/12/09/how-when-were-your-revit-objects-created/

[OG-Work]

Cheers @boostyourbim, I appreciate your excitement!

It's however impossible to have every student run Revit with the tool installed. Firstly because of security policies at the campus, and secondly if they work from a personal computer there is no way to guarantee that they will actually install the tool and we can't fail them just for that reason.

It looks like an interesting tool however, did you implement some hook/callback functionality upon instance creation or are there already some parameters hiding in the API somwhere?

Kind regards,

Oscar

[spelelo]

Honestly, I would like this for reviewing things my designers/engineers do in drawings. Sometimes "things happen" to drawings and we can't figure out what or why, and its usually the result of someone making a weird mistake, accidently using a hotkey, etc. Being able to look at action logs, filter out certain commands, etc. Especially when its a file several people are working on, being able to see who did what when, so we can figure out what problems are happening and how to fix them.

[OG-Work]

That's a fresh point of view @spelelo, thank you! I did not even consider that, I guess I've been struck by a bit of tunnel vision. Glad you like the idea!

[michael-coffey]

Potentially, you could take the code and embed it as a macro in a template.  Students would be required to use the template and allow the macro to run.

[boostyourbim]

@OG-Work I understand the challenge of using a 3rd party tool, but it is unlikely that Autodesk is going to add this feature to the core product for you. As Michael notes, it might be possible to adapt the code I wrote (which is based on the DocumentChanged event, ExtensibleStorage, and an ExternalEvent) into a document macro, but I'm not sure how much better that is than what I have already done.

[Samuel.Arsenault-Brassard]

I would suggest for the students to use BIM360. Not only for collaboration, but also for the logs.

https://knowledge.autodesk.com/support/bim-360/learn-explore/caas/CloudHelp/cloudhelp/ENU/BIM360D-Ad...

This will give you an idea of the engagement of each student and allow your students to work collaboratively with each other and other disciplines at the school (interior design, engineers, etc). You will need to be part of each project, where you can add annotations to their evolving design and use BIM 360 for design reviews.

-

Many architecture students lie about their Revit skills to get a job and will do anything to stay only in Sketchup or Rhino. The solution is to have live tests with no internet for students or new potential employees. I would make a list of questions and also tasks like creating families or specific tasks that an be accomplished in roughly one hour. This will weed out the ones who have made no effort to learn the software.

Generally, Revit needs better logging. At the moment, it is possible for a user to come in and delete many elements, synchronize and no one will ever know who deleted the items or when until it gets noticed by a user. In a world where offices hire lowly paid interns that will 'learn Revit as they go', the reality is that many users deleting things by accident and it's quite hard to figure out who is causing the damage in the team.

[RPTHOMAS108]

In your situation you could probably enforce the presence of a third party add-in quite easily.

If the add-in is responsible for producing an encrypted log (similar to journal but simplified) that the student presents to you at the end of the exercise along with the model then you decrypt it and see what has been done. No log = no validation. They would need the add-in to encrypt the log and the encryption key they don't know.

Obviously there is scope for cheating if you decompile assembly and get key but is the student studying computer science or Revit use? These things are imperfect they are about level of effort involved to thwart vs level of effort to do the task without cheating. Also in remote learning how do you ever know the person doing it is the candidate, do you stare at them through a web cam whilst they do it (they meanwhile put it on a loop like in one of those bank heist movies)? 

[OG-Work]

Thank you all for your replies and suggestions! I'll do my best to reply to all comments here when time allows.

@michael-coffey, yes that sounds like a good idea, I'll try it out!

Thank you @boostyourbim for those hints, I will look into it. Your plugin seems very nice don't get me wrong, but since our campus is large the IT-department are not too happy with the idea of installing 3rd party tools for every course.

Interesting idea @Samuel.Arsenault-Brassard! So far we haven't used BIM360 in our courses. One downside with using BIM360 is that it will steal some focus from learning the Revit, especially in the beginners' courses. It seems like a good solution for group assignments, but I'm not sure that it would solve the problem with students copying individual assignments
I'm sorry to hear that you've had experiences with students lying about their skills. We've recently been incorporating computer-examinations to improve the base skill level, pretty similar to what you're describing.
In the end, it sounds like Revit could benefit a lot from better logging, both in workplaces and education. An action log would probably solve this issue for everyone and it wouldn't be too hard for Autodesk to implement one.

Unfortunately not @RPTHOMAS108. We could definitely enforce students using it on their personal computers, but apart from the current situation due to the pandemic we mostly use the computers on campus. Since the IT department runs hundreds of computers, it's not a simple task to convince them to install 3rd party plugins unless they're verified on a large level
Encryption is a cool idea! If I have the time to develop a template-macro I'll consider it if needed.
Yes you're very right that there will always be a scope for cheating. Our students are engineering students but most of them are quite young and not very technical yet (first year students), so our efforts does not need to be too advanced. Remote exams has been a huge national issue here since covid, cheating in various forms has increased a lot all over the country.