Import xyz DEM1 only certain area

[t_ott]

Hello everyone,

I'm desperately trying to read point data from a .xyz file using LISP over a polyline area.
After all my attempts failed, I decided to give AI a try.

With SPIPv5.lsp the LISP points were read, but very strange, see SPIPv5.dwg.

In the SelectPointsInPolyline.dwg all points are read in and the polyline that I want to use and only want to read in the points within this polyline.

I'll add the lsp dwg and xyz files.

Would be great if someone could help me.

LG Tom

Addendum:
Only 3 attachments allowed, so here is partial content / format xyz data:
420726,000 5554796,000 377,803
420727,000 5554796,000 377,814
420728,000 5554796,000 377,816

[Sea-Haven]

Please post in English also.

Ok I think you want points inside a pline with a tolerance to outside pline. Then maybe use this method offset pline inwards the tolerance, then get the new pline pts note must add last point to list of points to make closed boundary. Then use (setq  pts (ssget "WP" pts '((0 . "POINT")))) you will have a selection set of points. Can then write to a file. 

No points file so no code.

[t_ott]

Good morning,

here as an addendum is the point file.

Original file name is: dgm1_32_420_5554.xyz

Greetings Tom

[t_ott]

Hi Sea-Haven,

I tried it a bit and it seems like this code works now.

See SPIPv5f.lsp

Thank you in advance for your food for thought and perhaps you have suggestions for improvement.

Kind regards, Tom

[Moshe-A]

@t_ott  hi,

check this PT-IN command.

some points on the edge are not colored as red even they are fully inside - why?

if you want them out, then i need to give a fix.

many thanks to mr Lee Mac, without his functions this could never been done.

enjoy

Moshe

(vl-load-com) ; load activex support


;; Intersections  -  Lee Mac
;; Returns a list of all points of intersection between two objects
;; for the given intersection mode.
;; ob1,ob2 - [vla] VLA-Objects
;;     mod - [int] acextendoption enum of intersectwith method

(defun LM:intersections ( ob1 ob2 mod / lst rtn )
    (if (and (vlax-method-applicable-p ob1 'intersectwith)
             (vlax-method-applicable-p ob2 'intersectwith)
             (setq lst (vlax-invoke ob1 'intersectwith ob2 mod))
        )
        (repeat (/ (length lst) 3)
            (setq rtn (cons (list (car lst) (cadr lst) (caddr lst)) rtn)
                  lst (cdddr lst)
            )
        )
    )
    (reverse rtn)
); LM:intersections


;; String to List  -  Lee Mac
;; Separates a string using a given delimiter
;; str - [str] String to process
;; del - [str] Delimiter by which to separate the string
;; Returns: [lst] List of strings

(defun LM:str->lst ( str del / pos )
  (if (setq pos (vl-string-search del str))
      (cons (substr str 1 pos) (LM:str->lst (substr str (+ pos 1 (strlen del))) del))
      (list str)
  )
); LM:str->lst



(defun c:pt-in (/ draw_xline IntersectWidth draw_point ; local function
		  adoc savPDMode savPDSize ss fname f AcDbPLine rec m0 cross^ p1 p2)

 (defun draw_xline (pt vec)
  (entmakex
    (list
      '(0 . "XLINE")
      '(100 . "AcDbEntity")
      '(100 . "AcDbXline")
      (cons '10 (list (car pt) (cadr pt) 0.0))
      (cons '11 vec)
    )
  )
 ); draw_xline

  
 (defun IntersectWidth (b0)
  (vl-some
    (function
      (lambda (vec / ename AcDbXLine cr^)
       (cond
        ((and
	   (setq ename (draw_xline b0 vec))
           (setq AcDbXLine (vlax-ename->vla-object ename))
           (setq cr^ (LM:intersections AcDbPline AcDbXLine acExtendNone))
	   (= (vl-list-length cr^) 2)
         )
	 (vla-delete AcDbXLine)
	 (vlax-release-object AcDbXLine)
	 cr^ ; return
        ); case
	((eq (type AcDbXLine) 'VLA-OBJECT)
         (vla-delete AcDbXLine)
	 (vlax-release-object AcDbXLine)
	 nil ; return nil
	); case
	((eq (type ename) 'ENAME)
         (entdel ename)
	 nil ; return nil
	); case
       ); cond
      ); lambda
    ); function
   (list '(0.0 1.0 0.0) '(1.0 0.0 0.0)) ; bisect vectors
  ); vl-some
 ); IntersectWidth


 (defun draw_point (pt)
  (entmake
    (list
      '(0 . "POINT")
      '(100 . "AcDbEntity")
      '(100 . "AcDbPoint")
      (cons 10 pt)
    )
  )
 ); draw_point

  
 ; here start c:pt-in
 (setq adoc (vla-get-activedocument (vlax-get-acad-object)))
 (vla-startUndoMark adoc) 
 
 (setq savPDMode (getvar "pdmode"))
 (setq savPDSize (getvar "pdsize"))
  
 (setvar "pdmode" 35)
 (setvar "pdsize" 0.25)
  
 (if (and
       (setq ss (ssget ":s:e+." '((0 . "*polyline"))))
       (setq fname (getfiled "Select point data file" "" "txt" 8))
       (setq fname (findfile fname))
       (setq f (open fname "r"))
     )
  (progn
   (setq AcDbPLine (vlax-ename->vla-object (ssname ss 0)))
   
   (while (setq rec (read-line f))

    (if (and
          (setq m0 (mapcar (function (lambda (s) (atof s))) (LM:str->lst rec " ")))
	  (setq m0 (list (car m0) (cadr m0) 0.0)) ; put point on zero elevation
	  (setq cross^ (IntersectWidth m0))
          (setq p1 (car cross^) p2 (cadr cross^))
          (equal (distance p1 p2) (+ (distance p1 m0) (distance p2 m0)) 1e-3) ; is it inside pline?
        )
      (draw_point m0)
    ); if

   ); while

   (vlax-release-object AcDbPLine)
  ); if
 ); if

 (setvar "pdsize" savPDSize)
 (setvar "pdmode" savPDMode)

 (vla-endUndoMark adoc)
 (vlax-release-object adoc)
 
 (princ)
); c:pt-in

[t_ott]

Hi,

I have already noted programmers in my documentation, including Lee Mac.
The mention of this person is still missing in my code because it is not finished yet.
Sorry.

Thank you very much Tom

[Moshe-A]

@t_ott 

---

@t_ott wrote:

Hi,

I have already noted programmers in my documentation, including Lee Mac.
The mention of this person is still missing in my code because it is not finished yet.
Sorry.

Thank you very much Tom

---

what?

[t_ott]

sorry, I'm still at work and just skimmed over your answer and misunderstood it...

[t_ott]

Hello
Unfortunately, I only got around to continuing working on the LISP yesterday.
I noticed something very decisive.
I worked all the time with a file with about 740 points, now I have taken the actual file for which I would like to use such a LISP.

But it has 1000200 points, now it is so that first the points in the graphic are created and which are within the polygon are selected.
Of course, this takes a very long time.

Now my question, is there also a way to make this selection before the actual points are created?

Greetings Tom

[Moshe-A]

@t_ott ,

i do not understand what do you mean by "and which are within the polygon are selected" - you mean selected manually?

i am curious to know how AutoLISP handle such a big file? specially i address to open the file and reading 1000200 records this is huge 😱

any how the pt-in does not first read the whole file into AutoLISP memory but process each record and draw the point (only the ones inside the polygon) only AutoCAD regenerates the graphic once lisp is end (to reduce regeneration time).

"Now my question, is there also a way to make this selection before the actual points are created?"

there is no way to select objects that are not exist.

maybe for this kind of big data files it better to process the file by chanks about 50000 records and maybe send each chank to a separate layer so you can freeze them?

Moshe

[Sea-Haven]

I did a google and  found 2 possible ideas, need to  test on multiple shapes and look at speed. Will use entmake or VL add object to get more speed.

(setq ss (ssget "WP" pts '((0 . "DUMMY")))) ; read point into DUMMY layer 1st
(if (= ss1 nil)
(princ)
(vlax-put (vlax-ename->vla-object (ssname ss 0)) 'layer "Insidepts")
)

Another is to draw a line from the point in a single direction and check does it intersect the polygon.

(setq plent (vlax-ename->vla-object (car (entsel "\nPick pline "))))
(setq pt (getpoint "\nPick point "))
(command "line" pt (polar pt 0.0 10000) "")
(setq obj (vlax-ename->vla-object (entlast)))
(setq intpt (vlax-invoke plent 'intersectwith obj Acextendnone))
(command "erase" (entlast) "")
(if (= intpt nil)
(vlax-put (vlax-ename->vla-object (entlast)) 'layer "Insidepts")
)

A 3rd

Solution 2 (2D)

Another solution forwarded by Philippe Reverdy is to compute the sum of the angles made between the test point and each pair of points making up the polygon. If this sum is 2pi then the point is an interior point, if 0 then the point is an exterior point. This also works for polygons with holes given the polygon is defined with a path made up of coincident edges into and out of the hole as is common practice in many CAD packages.

[Sea-Haven]

Had a quick go and got something working. The critical thing is make a "Point" but it must have its pdmode set to 0, if using other values that touch or go out side the polygon causes it to miss points. This did 740 points instantly. Can you post your huge file please. Need a dwg with correct polygon shape matching points. 

There may be a mathematical formula to work out point is inside a polygon so avoid using CAD to check.

; thanks to Lee-mac for this defun
(defun csv->lst32 ( str / pos )
(if (setq pos (vl-string-position 32 str))
    (cons (substr str 1 pos) (csv->lst32 (substr str (+ pos 2))))
    (list str)
    )
)

(defun Point (pt)
 (entmakex (list (cons 0 "POINT")
                 (cons 10 pt))))

(defun test1 ( / )
(setq ss (ssget "WP" co-ord (list (cons 8 "DUMMY")))) ; read point into DUMMY layer 1st
(if (= ss nil)
(princ)
(vlax-put (vlax-ename->vla-object (ssname ss 0)) 'layer "Insidepts")
)
(princ)
)

(defun c:test3 ( / oldsnap plent co-ord fname )
(setq oldsnap (getvar 'osmode))
(setvar 'osmode 0)
(setvar 'pdmode 0)
(command "-layer" "m" "Dummy" "C" 2 "" "M" "Insidepts" "C" 1 "" "")
(setq plent (entsel "\nPick pline "))
(setq co-ord (mapcar 'cdr (vl-remove-if-not '(lambda (x) (= (car x) 10)) (entget (car plent)))))
(setq co-ord (cons (last co-ord) co-ord))
(setq fname (open (getfiled "Pts FILES" "d:\\acadtemp" "XYZ" 16) "R"))
(setvar 'clayer "Dummy")
;(setq x 0)
(while (setq newline (read-line fname))
(setq ptslst (csv->lst32 newline))
(setq pt (list (atof (car ptslst))(atof (cadr ptslst))(atof (caddr ptslst))))
(point pt)
(test1)
;(princ (setq x (1+ x))) ; for testing
)
(setvar 'pdmode 34)
(close fname)
(princ)
)
(c:test3)

[t_ott]

Hi, I can send you the data on Monday. the file is 30mb in size. Greetings Tom

[Sea-Haven]

As its a text file zip it.

[Moshe-A]

@t_ott  hi,

---

@t_ott wrote:

Hi, I can send you the data on Monday. the file is 30mb in size. Greetings Tom

---

don't bother.

Took your data file from message #3 and duplicate it content until i got 200000 records and run my lisp on it and it took about 3:35 minutes to complete (this on my 2 year old laptop i7 with 16gb ram and 500gb disk on AutoCAD R2022 nothing fancy 🤣) this means for 1200000 records it will need at least 22 minutes to run.

if you are expecting to load such a huge data and it will complete in for seconds or few minutes, think again. even if @Sea-Haven will find more efficient method (which i doubt) it will still be around 22 minutes.

we are talking here about AutoCAD System which is full Graphics System not a Financial banking system.

add to that, AutoLISP is High interpreter language which converts the code at run time, all that is significant on overall run time.

Moshe

[Sea-Haven]

Like @Moshe-A I tried with 103,460 points just importing the points. Around 4 seconds. So for the sort of numbers you have just import all the points then run the select boundary and use (ssget "WP" co-ord) this will find all those inside hopefully and you can get X&Y of every point. Write out to a file. Erase all the imported points. Import the new file of points

I drew a random pline over the 100k+ and it returned 20000 points. 2 seconds.

So @t_ott your home work is take what I have posted and make it into 2 steps, all the code is there. A rearrange of the code . A good time to have a go at learning lisp.

[t_ott]

Hello Moshe-A,
Hello Sea-Haven,

I happen to work in a job where I'm busy with CAD all day.
I now had a project where I had to load 11 DGM1 datasets to create a 6.7 kilometer terrain section.

To cope with this amount of raw data, I used Cloudcompare, where you can actually read in 10 million points within minutes.
Then a traverse on top for the area you actually need and the points that are within it output again as DXF.

Now my idea / question was whether you can't do all this via Autocad.

Thank you for your support.

Greetings Tom

[Moshe-A]

@t_ott ,

i do not recommend to put such amount of points (data) in one dwg file but divide it to few files at reasonable volumes that gives you reasonable edit times. take in consideration, these files will grow along the way.

Moshe

[Sea-Haven]

Autocad gets a bit fussy when palying with say a million points. " I now had a project where I had to load 11 DGM1 datasets to create a 6.7 kilometer terrain section. So even loading 1 section at a time will be like a million pts. I wrote somewhere I thought to import all the points then write out those co-ords for the ones inside the boundary to a file, sya then do a couple of undo's and import next section.

One of the things with lisp is to open a file using the "A" option this adds to a file. So you can repeat the import selection in one dwg. because your dealing with big numbers it may be better to use a script and close each dwg as you make the file, doing a garbage clean up as you go just dont close last file rather undo and import internal points.

Your welcome to start with what I previously posted. It may be better to use a true cloud software to get the internal points as it will be optomised for big numbers of points. eg Recap ?