ReadBlock limitation

[lightcube]

There is a very bothersome limitation in <Interface:MemStream>.readBlock

And that is that it fails if the close token is more than 5120 characters from the open token, the function returns undefined.

I've had problems for a couple years with this limitation. My workaround has been to use this basic solution from http://forums.cgsociety.org/archive/index.php/t-1082753.html

fn readBlock ms =
(
local p1 = "("
local p2 = ")"
local s1 = "\""
local s2 = "\\"
local s3 = "-"
local s4 = "/"
local s5 = "*"
local s6 = "\n"

local token = ms.readChar()
while not ms.eos() and not token[1] == p1 do
token = ms.readToken()

local ss = stringStream ""
format token to:ss
local n = 1

local inString = false
local inComment1 = false
local inComment2 = false
local last1 = ""
local last2 = ""

while not ms.eos() and n > 0 do (
local c = ms.readChar()
format "%" c to:ss

if not inString and not inComment1 and not inComment2 then (
case c of (
s1: if last1 != s2 or last2 == s2 then inString = true
s3: if last1 == s3 then inComment1 = true
s5: if last1 == s4 then inComment2 = true
p1: n += 1
p2: n -= 1
)
) else (
case c of (
s1: if inString and last1 != s2 or last2 == s2 then inString = false
s6: inComment1 = false
s4: if inComment2 and last1 == s5 then inComment2 = false
)
)

last2 = last1
last1 = c
)

if n > 0 then undefined else ss as string
)

But this code has a serious flaw. It very often generates an Unknown System Exception. My guess is that it goes past the end of the string.

Anyone have any working solution for this that is fool-proof? I really can't understand this hard-coded limitation of 5120 characters.

[akira.kudo]

Hi,

As you can see source code of it in the sdk, maxsdk\samples\maxscript\gScript\iMemStream.h.

There is a definition at the line 22 as below.

// russom - 08/24/01
// Put 5k limit on line lengths
#define MEMSTREAM_MAXLINE    5120

So you are hitting this limition unfortuantely, I think.

By the way, you can also use python script. Can you use it instead?

You can see some samples in the C:\Program Files\Autodesk\3ds Max 2015\scripts\Python.

I hope this helps.

[lightcube]

Thanks for responding... for some reason the notification went to my spam folder.

I think I have a working MAXScript solution that I will post when I've used it more. However, I still wish this function would be updated and the MemStream class would not be so naive and return Unkown System Exceptions so much. The functions should really just be more user-friendly and robust.

[akira.kudo]

Certainly, I will be reporting your request to engineering team.

Again, we are sorry for inconvenience and thank you very much for your patience.

[lightcube]

For those who are having this problem, I've posted a working Solution for working around the ReadBlock Limitation in MAXSctipt.

[lightcube]

The functions I've used to work around the readBlock() limitation have worked fairly well. By now I've got a few working solutions. However, I now think that I've discovered the root of the Unknown System Exception surrounding the various solutions I've come up with (and the MemStream methods all around).

As far as I can now tell, the Unkown System Exception occurs when the input data has one or more of these qualities:

• Has mixed EOLS (End of Line : LF with CRLF)
• File does not have a final EOL

I've found that if you are iterating over a memstream that was derived from memStream.openFile() and the file does not have CRLF at the end of the file, the odds increase that the Unkown System Exception will occur significantly. Adding CRLF to the end reduces (or maybe eliminates) the error.

Knowing this is helpful in dealing with it. I'd still like some attention from AD on adding methods to MemStream to allow handling these cases because one of the main reasons for using MemStream functions is to read/import data--and you can never be sure where the data came from. Failure from CRLF vs LF or no line ending on a file feels like the stone age!

Increasing the ReadBlock character limit and fixing the problem with EOL would be super fantastic!

For the record, here is a MXS function to help detect final line ending:

function wallworm_checkForEndlines vmfMemStream  = (
	local pos = vmfMemStream.pos()
	vmfMemStream.seek -2 #seek_end
	ln = (vmfMemStream.readChar() + vmfMemStream.readChar())
	local isOK 
	if ln == "\r\n" then (
		isOK = true
	) else (
		isOK = false
	)
	vmfMemStream.seek pos  #seek_set 
	isOK
)

There is also the function replace_LF_with_CRLF() that can help sanitize a document. But it doesn't fix the final line ending.