Russian letters doesn't export in lookup tables

[Anonymous]

Hello!

We've faced with the following issue: lookup tables with russian letters exports incorrectly (via GUI or via API, doesn't matter). The problem is that a data with at least one russian symbol doesn't export at all, however Revit undertand and show this letters correctly.

Steps to reproduce:

1. Import .csv file with the following data:

,B##length##millimeters
РУССКИЕ БУКВЫ,150.000000

РУССКИЕ LETTERS,150.000000
ENGLISH LETTERS,150.000000

2. Export this table. Exported table will contain:

,B##length##millimeters
,150.000000

,150.000000
ENGLISH LETTERS,150.000000

Revit's interface is Russian (of course :)). The issue is reproducing in Revit 2018, Revit 2019 and Revit 2020.

[jeremytammik]

Thank you for moving this discussion from our private email discussion out here into the forum.

Can you flesh out the steps to reproduce in a little bit more detail, please?

E.g., something like:

• Start up Revit version XYZ in language UV
• Start a new family/project using template ...
• Import CSV file ... using ...

Please assume that your reader has not the slightest idea how to perform these steps (like me).

Thank you!

Best regards,

Jeremy

[jeremytammik]

Sorry that nothing further happened yet with this thread and your report.

I was awaiting your further feedback.

Now another developer reported a similar issue, so I submitted a ticket on it for the development team to analyse:

• REVIT-152268 [Encoding issue reading data from size lookup table with Cyrillic characters -- 15738224]

Just letting you and the rest of the community know, for the sake of completeness.

Best regards,

Jeremy

[Anonymous]

Hello, Jeremy!

Sorry for not answered on your more steps-to-reproduce request, I was at vacation and then another tasks distracted me. 😞 Sorry once again.

As for the current state of this thread. As far as I understand, you don't need any additional information from me any more? Now all is clear to you? Please, let me know if I can be helpful.

Thanks

Vladimir

[jeremytammik]

Thank you for your update and no problem whatsoever.

Yes, all is clear now, the development team are looking into it.

[jeremytammik]

Can you provide more detailed steps to reproduce, please?

The development team is looking into this now, and the developer says:

Sorry, I need an explicit cookbook set to replicate this in Revit. In other words, I launch Revit, then what? I have the description and the link you pointed to and those are not clear enough for me to get to the code I updated to see if it is being called or not. Sorry, but in my day to day development work on Revit, this is an area I do not play in much.

Also please provide the exact Revit version and build number that this is occurring in.

Thank you!

[Anonymous]

Hello, Jeremy!

Here is my detail steps to reproduce:

1. Create a .csv file with data in CP1251. Here is my example (is attached to this answer as test_russian.csv):

,B##length##millimeters
Тест1,1.000000
Тест2,2.000000
Test3,3.000000

Take note, that the last row contains only english letters, it's important.

3. Create a new family (my test family is added to this answer as test_russian.rfa).

4. Import the lookup table from step 1.

5. Add a new parameter with lookup table (in my test family it's called TestFieldTable).

6. Add a new lookup parameter (in my test family it's called TestFieldValue):

size_lookup(TestFieldTable, "", "", 2 мм)

Result will be: "Тест2", which is correct. This means that Revit has read lookup data correct.

7. Export this lookup table. Here is what you'll get (my exported file is attachet to this answer as test_russian (exported).csv):

,B##length##millimeters
,1.000000
,2.000000
Test3,3.000000

As you can see, russian letters is now missed in first two data rows, while the last row with english letters is ok.

This issue is reproduced on the following versions of Revit:

1. 20.0.0.377 / 20190327_2315(x64)

2. 19.2.10.7 / 20190225_1515(x64) / 2019.2.1

3. 18.3.2.7 / 20181011_1500(x64) / 2018.3.2

Do not hesitate to contact me if you need some additional information!

Thanks

Vladimir

[jeremytammik]

Dear Vladimir,

Thank you very much for raising this issue and all your support and patience in nailing it down.

The development team has reproduced and understood the issue now.

Code is in place to handle this properly in a future release of Revit. However, some command sequences were still making use of older implementation paths. Thanks to your input, they are now being corrected to use the updated approach as well.

Best regards,

Jeremy

[adam.krug]

Hello Vladimir @Anonymous ,

I'm actually this above mentioned "another developer" :).

I'm very curious how did you manage to import the "test_russian.csv" lookup table first place? When I'm trying to re-load it from your attached zip I only get some strange characters instead of "Тест2"...

I tried switching my Revit to Russian but still the same result.

@jeremytammik - it seems like there is already some trick to import Cyrillic text in a csv table...

[Anonymous]

Hello, Adam!

Glad to see you. 🙂

Maybe, this is because of I'm using russian version of windows? I'm using Win10, which version are you using? Do you have set russian language for non-unicode programms (https://answers.microsoft.com/en-us/surface/forum/surfbook-surfaccess/how-to-change-system-locale-in...)?

Vladimir

[Anonymous]

Adam, hello again! @adam.krug 

I've made some tests (switched language for the non-unicode apps to English (USA)) and Revit still displays Russian symbols correctly (surpise! :))). So, this is not the cause of your issue, forget about my previous answer. 🙂

If I view my CSV file (in external viewer program) and switch code page from WIN1251 to WIN1252, then I see the same symbols as on your screenshot (not cyrillic, of course). Attached it. Which code page Revit suppose the CSV file is during the import? System (1252 in your case and 1251 in my accordingly)? If so, then this is the cause of your issue with import.

Anyway, if you see any characters in the formula cell, then Revit has read it from the CSV file. Then, it should export it, even in WIN1252 code page, isn't it?

How else can I help you?

Vladimir

[adam.krug]

Hi Vladimir,

You might be correct with the Russian Windows set-up. I also considered this to be the reason but haven’t checked that yet. Maybe next week. I have my Windows 10 in US English with a Polish programmers keyboard.

The problem is not that I’m unable to see the proper Cyrillic chars in the Revit family, I can see them fine from your example. I only cannot successfully load the Cyrillic csv because then the chars get messed up.

After exporting the csv it has ANSI encoding which must be some kind of System’s default.

thanks for your feedback!

[Anonymous]

Hello Adam,

I understood and I'm waiting your research results.

You can count on my help if you need additional tests.

Thanks

Vladimir

[Anonymous]

Hello, Adam!

Any updates?

Thanks

Vladimir

[adam.krug]

Hi Vladimir,

I myself haven't done that yet but my colleague did - it works. Now when he switches to the Russian keyboard he can insert a csv to a family with Cyrillic text with no loss.

Thanks a lot for the advice!

Adam

[jeremytammik]

Dear All,

The development team have continued work on this issue REVIT-153006 [Encoding issue reading data from size lookup table with Cyrillic characters -- 15738224] and report the good news:

We have now fixed and closed this issue in the development release.

It has been addressed in work I did for REVIT-161509.

That fix is present in Revit 2021.1 and any version of Revit after that.

For previous versions, one option to try is to see if generating the csv files as UTF-16 (i.e., 2 byte Unicode) will cause them to be read properly, even in older versions of Revit that do not contain the fix.

Please note that our support for loading UTF-8 and UTF-16 requires that the files contain the proper UTF signature BOM at the beginning of the file. The UTF-8 files provided as samples DO NOT, but you can re-save them as UTF-8 in almost any text editor and force generation of this signature. Then Revit knows how to properly load and encode them with the fix. Revit is not a full blown text processor, so it relies on the presence of that BOM to properly handle files. Without it, Revit assume they are 'normal' ANSI encoded files and thus will not process them correctly.

I hope this helps.

Best regards,

Jeremy

[jeremytammik]

Result saved for posterity:

https://thebuildingcoder.typepad.com/blog/2020/07/cyrillic-encoding-and-categories-for-legends.html#...