READ string too long on input

[CodeDing]

I'm trying to convert a JSON response to a list which I can then manipulate with AutoLISP.

Normally I can accomplish that with this simple function:

(defun JSON->LIST (json / tmp dbl nwl)
;json - string, as json data
;returns - list or nil, list of data converted from json
(if (eq 'STR (type json)) (read (vl-string-translate "[]{}:," "()()  " json)) nil)
);defun

However, the READ function must have a limit which I cannot directly find. Because when I send the Attached JSON information as a single string to the function, I get this error:

; error: string too long on input

I have tried removing excess spaces and newline characters with:

- (while (vl-string-search "  " str) (setq str (vl-string-subst " " "  ")))

- (while (vl-string-search "\n" str) (setq str (vl-string-subst " " "\n")))

Still no luck. (my shortened string length comes to 37,367 characters).

So, I was hoping that I could create any function (perhaps recursive?) that will allow me to merely construct lists on the fly since, in THIS case, I know that I have many 'Lists' inside of my JSON data. Then I could break my 'Reads' into smaller chunks... I just am not able to understand how my workflow should go and I have been racking my brain on this for hours..

This isn't a function but hopefully it can help you get an idea of how i'm trying to break this down:

Instead of this:
(read "(\"abc\" 123 (\"def\" 456) (\"ghi\" 789) \"jkl\" 000)")
...Do this...
(list "abc" 123 (list "def" 456) (list "ghi" 789) "jkl" 000)

Best,

~DD

[cadffm]

You have Lisp at your disposal, so program a function to find out the limit!

Symbolname length limit = 2305 (works) 2306 (too long)

[CodeDing]

@cadffm ,

Thank you for helping me find the limit. However, I cannot just break my string down into multiple strings so easily.. because if I were to run the read function on a string that is an incomplete list, then it will error out.

Example:

(read "(\"here is some data\" \"but the list is incompl\"")
; error: malformed list on input

Therefor, i cannot just send ANY partial string to my READ function.. It must be a complete list.. which I must also remember where the position of this list is in my overall string (the complete list of JSON data).

[cadffm]

I am not familar with that stuff (it's ok for me), 

but you can not change basic API limits.

The maximum length of a symnolname is 2305 (said my system),

nothing to do with the read function.

(But read convert a string to a symbol, and symbolname is limited to 2305)

Sorry

Are you sure that you need that read in this way? 

(Sorry again because i am not familar with that)

[Sea-Haven]

You can (read-char which does just that a single character at a time so if you say have 50  character strings you do just that in a repeat.

[cadffm]

>"This isn't a function but hopefully it can help you get an idea of how i'm trying to break this down:"

This is easy, easy to understand and easy to do,

but i can not see what this "sample" to do with your special *.txt content.

A shorted sample from your *.txt

Line 87 for example (shorted!)

"points" : "iomiGhsocNIo@c@oDGg@AASaB[_CGi@E_@EWAGIo@?EA.........................ICSAI?KAI?O?M?Q?G?G@I@Q?I@G?EAE?G?EACAKa@{Ca@oCc@eDEW[wBYwB]eCCSYsBQsAMeA_@qCa@wCa@qC[_CAIWgBe@gDU_BMcAOiAUyAWiB]aCOeAg@kDUuACQ?ICUGs@KaBAUCYEq@KaB"

"points":  is ok, you can check the string from left to the right to find this part from the Line87.

But i can not understand what the rest is or why do you need this as a symbol - It can not be the right or the only way.

Hopeful you find another helper with knowledge about Json and what you try,

i am out at this point

I would love to look at it, but currently I don't have the time for such complex things, I have to start at 0 for it.

[Sea-Haven]

Not sure if this helps

(setq fo (open "d:\\acadtemp\\testchar.txt" "r"))
(while (setq ans (read-char fo))
(princ "\n")
(princ (chr ans))
)
(close fo)

[CodeDing]

@cadffm ,

Thank you for your help.

I am not looking to return a Symbol from the read function. I am trying to return a List.

Since the read function returns '...the first list or atom obtained from a string...'

...and the string returned from (vl-string-translate "[]{}:," "()() " json) is one big List...

...I am trying to save the value from Read as a List inside a variable.

...Which means that ALL of the json data would be just one large list I can easily manipulate.

However, since my string being passed into Read is too long, I need to SOMEHOW loop through my string and break it down into smaller Lists, either with or without the Read function.

@Sea-Haven ,

Thank you for the input.

But, the Read-Char function does not return a list nor provide me with more guidance as to how to come toward a solution of my question. I understand the concept of looping through characters, but as I stated previously, I cannot think of a function that will help me return actual Lists that I can save as a variable when I pass in my json String.

Best,

~DD

[Sea-Haven]

 Is this what your talking about ?

To parse -- to separate into parts. 

When JSON data is received, it comes as a massive concatenation of characters (one huge string).  Inside of this string is encoded tags as well as "name : value" pairs.  The job of the parser is to take this huge string and break it up into the data structures (Objects?) that are indicated by the string. 

The answers are here in google "json lisp parser" not sure about a Autocad lisp variant. They talk about Gb files. Did not look to far.

[Sea-Haven]

Look into VBA found pretty quick some good examples and code to download.

https://www.youtube.com/watch?v=CFFLRmHsEAs&feature=youtu.be

[CodeDing]

@Sea-Haven ,

Yes, at this point I think it's pretty safe to say I'm looking for a JSON to Lisp parser haha. I guess I wasn't ready to accept that in my mind yet.

Looks like I have quite a bit of work in front of me. I can't find much that seems ready to use by just Googling for a Parser.

Thank you for helping look into it though!

[Sea-Haven]

If you find a VBA answer you can call a VBA from lisp 

(vl-vbaload "P:/AutoDESK/VBA/acces-rev2.dvb") 
(vl-vbarun "draw_vehicle")

[martti.halminen]

A few years ago I needed a JSON -> Lisp parser, didn't find one for AutoLISP. Several for Common Lisp, though.

If you don't like working with VBA (or possibly C#), you could do what we are doing: run the parser in a separate Common Lisp program, write the parsed result as a convenient list into a file and read that with AutoLISP.

-- 

[CodeDing]

@martti.halminen ,

I'm interested in your response. I've never ran a Common Lisp routine though. Are they the same file extension? Do I need an IDE? Can you maybe post a sample routine for me (doesn't have to be a parser routine) and example how to call it with AutoLISP?

I'd definitely appreciate it.

Best,

~DD

[hak_vz · Accepted solution]

What will you do with that JSON data. Do you have to recreate whole database structure, or are you interested just about some data that are siblings to some parent  object (let say routes->legs->steps in your sample). I would attack this task using sequential read line by line and look for particular data only.

[CodeDing · Accepted solution]

@hak_vz ,

Not sure why it didn't cross my mind to use those newline characters to my advantage!

I use the newline character approach "\n" because I do not write the JSON data to a text file, I get it as response text straight from the browser.. I guess thankfully, Google returns it formatted (for some reason) so I can use those newline characters to my advantage.

So, once I get rid of those painful "points" elements, then it's smooth sailing for the Read function to continue the heavy lifting!

Thank you for the inspiration!

Here's my finished function:

(defun DirectionsJSON->LIST (json / tmp str)
;json - string, of json data (as returned by Google Directions API)
;returns - list, of converted JSON data
;helper functions(s)
  (defun LM:str->lst (str del / pos)
    ;By Lee Mac: http://www.lee-mac.com/stringtolist.html
    (if (setq pos (vl-string-search del str))
      (cons (substr str 1 pos) (LM:str->lst (substr str (+ pos 1 (strlen del))) del))
      (list str)
    );if
  );defun
(while (vl-string-search ":" json) (setq json (vl-string-subst "" ":" json)))
(setq json (vl-string-translate "[]{}," "()() " json)
      json (LM:str->lst json "\n")
      tmp '())
(foreach str json
  (if (not (or (> (strlen str) 2000)
  	       (wcmatch str "*\"points\"*")))
    (setq tmp (cons (vl-string-trim " \t\n" str) tmp))
  );if
);foreach
(setq json (apply 'strcat (reverse tmp)))
(read json)
);defun

Best,

~DD

[hak_vz]

Actually, your post has reminded me about one of my old private project I have been working on at a time when Autodesk had active project "Butterfly" as one of test environments for Autodesk 360. I have been working on a in-browser CAD like environment, and writing JSON parser (for CAD) was one of the task on my long to-do list that I never started working on. Unfortunately W3C specifications for technology behind it changed so often that I stopped development, but always hopping that once they'll finally figure it out. After 10 years almost nothing changed. Maybe its a time to restart my working on it. Here are some of the screenshots from my beta.

[hak_vz]

JSON data are mostly written in human readable form. It is a really good database form applicable to almost anything and with many sub specifications. I personally often use GeoJSON for transferring some geolocation data to  state cadastre.

[CodeDing]

@hak_vz ,

Not to detract too much, but I use API calls to ArcGIS GeoServices all the time, which I can have return GeoJSON. I use the Read function to translate these all the time to lists, and create the linework directly into AutoCAD using a recursive function. It's a beauty and a life saver.

Best,

~DD

[Sea-Haven]

If there are new line in the file then using what I posted about (read-char just look for 10 which is a linefeed.