Help to improve performance when reading large files

[blakestone]

Back again... I am reading an XML file which is 15,000+ lines (~280MB) where I need to check lines for keywords and collect its data. Currently it's taking ~8 minutes to read this file which I am hoping to reduce this to anything less. Any ideas would be greatly appreciated!

Here is an example of how my script looks

(
 
 local my_file = "c:\\my_file.xml"
 local my_fileOpen = undefined
 local my_fileRead = undefined
 local my_array = #()
 
-- open file
 my_fileOpen = openFile my_file

-- read file (line-by-line)
 while not eof my_fileOpen do
 (
 my_fileRead = readLine my_fileOpen

 if (matchPattern my_fileRead pattern:"*<node>*") then append my_array my_fileRead
 )

-- close file
 close my_fileOpen
 
 print my_array

)

[barigazy]

Look this threat to see how to read from xml

[blakestone]

This is great for an XML but what about non-XML files... as per the title "when reading large files" ?

[Steve_Curley]

Have a look at the MemStream interface. It loads the entire file into memory in one go which is obviously a lot faster than repeated calls to the OS to read a line at a time.

[blakestone]

memStream == awesome........... I got this process from ~8 minutes down to 2.5 seconds!

(

 local my_file = "c:\\my_file.xml"
 local my_memOpen = undefined
 local my_memRead = undefined
 local my_array = #()

-- open memStream
 my_memOpen = memStreamMgr.openFile my_file

-- read file (line-by-line)
 while not my_memOpen.eos() do
 (
 my_memOpen.skipSpace()
 my_memRead = my_memOpen.readLine()

 if (substring my_memRead 1 7) == "<layer>" then append my_array my_memRead
 )

-- close memStream
 memStreamMgr.close my_memOpen

 print my_array

)

[Steve_Curley]

I take it you're happy then? 😉

[blakestone]

I am very happy, thanks again for your assistance!

You wouldn't know how to speed up

importFile

would you, I am importing 100+ objects one after another and yeah it's a slow process. About 2 minutes for 100 objects.

[Steve_Curley]

You're quite welcome 🙂

ImportFile() I doubt can can be sped up at all. Each filetype is probably already optimised - when you consider everything that the import has to do, from reading the file through to reconstructing the mesh and the UVs and the materials and so on, 2 mins for 100+ objects is probably not bad.

Try manually importing one and time the actual import - even if it only takes a second that's 100 secs or 1:40 for them all. Larger objects will take longer, obviously. I suspect that reading the file is not the bottleneck, it's everything else which has to happen.

[blakestone]

a random thought... can i run importFile multiple times instead of 1 at a time?

[Steve_Curley]

Under script control? Don't see why not.

Create a simple text file with the necessary details - filename, import type, object name (if applicable) then just read that a line at a time (using memStream of course 😉 ) and set up the import for each one. Would that help?

[blakestone]

I am not sure how that would be much faster than what I am currently doing... I can visually see the delay with the importing process via its progress bar. I think the key here is to kick off the import process without pausing the script so that I can execute one after another without waiting for the previous one to finish.

Can you force the script to continue after executing importFile or maybe I could generate some maxscript files and then kick them off using the HiddenDOSCommand doNotWait:true

[Steve_Curley]

I'm not sure about not waiting for a command to complete - I don't think so, and as most of Max is not multi-threaded you could give it, and yourself, a real headache.
As for using DOS's doNotWait - that would mean running multiple copies of Max (I think), which besides being slow would eat up memory at a rate of knots.