Remove Hidden Data and Reduce File Size

[JC_BL]

I am trying to reduce the drawing file size.  This is for uploading the drawings to AutoCAD WS.  AutoCAD WS doesn't display or edit a drawing that is over 15MB.  Unfortunately some of our drawings are over 15MB even after I have removed the background and others and have purged.  I believe the reason why a drawing can be so big has to do with each object (grouped object) has a ton of information on it.  For example, the properties of a typical object is like this:

(
   (-1  . <Entity name: 7ffffbece60>)
   (0   . "PRCD_FITTING")
   (5   . "145B6")
   (102 . "{ACAD_XDICTIONARY")
   (360 . <Entity name: 7ffffbece70>)
   (102 . "}")
   (330 . <Entity name: 7ffffbf59f0>)
   (100 . "AcDbEntity")
   (67  . 0)
   (410 . "Model")
   (8   . "BON-SupplyMP-3.0-4.0 RECT")
   (62  . 1)
   (100 . "CPrcdEntity")
   (280 . 10)
   (90  . 601861454)
   (70  . 24058)
   (70  . 4567)
   (280 . -65)
   .
   . Skip 3000 lines here for clarity
   .
   (90  . 0)
   (90  . 0)
   (90  . 0)
)

You can see from above that the list of properties are huge.  All I really need are the visible parts of the object.  This means most of the info above are junk.

I try to reduce the size of object by exploding all the objects on the drawing.  This sort of work in the sense that the file size of a test drawing is reduced from something like 8MB down to 6MB.  But I am expecting far more reduction than this.  The reason why I expect far more reduction is that the total number of lines of the exploded items of the same object mentioned above is reduced from 3000+ lines to only 300 lines as shown below:

(
   (-1  . <Entity name: 7fffe8654f0>)
   (0   . "LINE")
   (330 . <Entity name: 7ffffa1b9f0>)
   (5   . "3B11F")
   (100 . "AcDbEntity")
   (67  . 0)
   (410 . "Model") 
   (8   . "BON-SupplyMP-3.0-4.0 RECT")
   (62  . 1)
   (100 . "AcDbLine") (10 4188.87 -423.095 156.0) (11 4188.87 -387.095 156.0) (210 0.0 0.0 1.0)
)
(
   (-1  . <Entity name: 7fffe8655f0>)
   (0   . "LINE")
   (330 . <Entity name: 7ffffa1b9f0>)
   (5   . "3B12F")
   (100 . "AcDbEntity")
   (67  . 0)
   (410 . "Model")
   (8   . "BON-SupplyMP-3.0-4.0 RECT")
   (62  . 1)
   (100 . "AcDbLine") (10 4248.87 -453.095 156.0) (11 4212.87 -453.095 156.0) (210 0.0 0.0 1.0)
)
.
. Skip 250 lines here for clarity
.
(
   (-1  . <Entity name: 7fffe865260>)
   (0   . "MTEXT")
   (330 . <Entity name: 7ffffa1b9f0>)
   (5   . "3B0F6")
   (100 . "AcDbEntity")
   (67  . 0)
   (410 . "Model")
   (8   . "BON-SupplyMP-3.0-4.0 RECT-ANNO")
   (62  . 1)
   (100 . "AcDbMText") (10 4193.39 -418.589 150.0)
   (40  . 3.0)
   (41  . 19.8)
   (46  . 0.0)
   (71  . 1)
   (72  . 1)
   (1   . "\\Fdim.shx;\\W0.7000000000;\\T1.0000000000;\\o\\l24X30THRT")
   (7   . "BonDuctTXT") (210 0.0 0.0 1.0) (11 1.0 1.46161e-017 0.0)
   (42  . 17.5)
   (43  . 3.0)
   (50  . 1.46161e-017)
   (73  . 1)
   (44  . 1.0)
)

Therefore, I am expecting a 10-to-1 reduction, not just a 1/4 reduction.

I am wondering there may be hidden data on the drawing that are left behind after I have exploded the objects.  Unfortunately, purging the drawing doesn't help because there is nothing to be purged.

Would you please tell me if there is a way to get rid of the hidden data?  If there is a better alternative, please also let me know.

Jay Chan

[scot-65]

Apologies for being a little late on this topic.

Have you tried to WBLOCK all the exploded items, etc.?

How about inserting the existing drawing into a blank Drawing1.dwg

then do a Save As...?

I posted a routine called PDI a while back - Purge Dictionary Items.

Have you tried looking into the "Reference Manager"?

???

[JamesMaeding]

This is an easy subject, as it is well known how to clean native entities.

You wblock out like you said, then strip the xdata and any xrecord data.

The trick is how to explode the custom aec objects and not end up with a million pieces.

The OP needs to try some things and tell us a little more.

Its something he would have to play with as he has the object enablers or prog on his system.

[marko_ribar]

Try following 2 routines for removing XDATA and LDATA... Unfortunately invisible entities that may contain also XDATA or LDATA like ModelSpace entity are not considered for removal by these 2 codes, so you have to do the rest manually... Simply there is no possible filter for those kind of entities to take them into sel. sets...

Link :

http://www.theswamp.org/index.php?topic=44080.msg493392#msg493392

M.R.

[dbroad]

Given that dataset (testSomePieces.dwg) 713Kb, the following process reduces it and keeps some 3d features:

explode all

overkill all

wblock objects 0,0 ....

This leaves you with a 115kb file

To get to 2D, open that file and

flatten all no

overkill

That gets is to 97 kb

wblock objects 

That gets it to 75kb (as a 2d file).

Some of these operations take some time, particularly flatten and overkill.  What you will end up with will display in WS but will lose almost all its functionality.

[JamesMaeding]

I believe the xdata and xrecords will still be present, as overkill and wblock does not affect them.

Is it an issue, NO.

xdata and xrecords are likely the least of the worries on this, I bet they don't affect file size 1%.

The killer is exploding the aec objects, which can make a million pieces.

Hope the OP is reading all this, some good people commenting here.

[dbroad]

I agree that killing the AEC objects is bad and can result in lots of extra objects but the end result would target the reduction requested.  Look at the file size results reported (10 percent of original size) which was requested.  Wblock objects generally takes care of any xdata problems assuming the xdata was attached to the parent and not to every child.  Flatten puts into 2d.  overkill takes out the bottom or top entities.  You could also get rid of the zero length lines that probably result to further reduce the size.

Large amounts of Xdata could conceivably affect the file size significantly but the explode and wblock should take care of most of that.  The intermediate file size could baloon until the sequences are complete.

The main issue is the loss of functionality but if the WS version is only a copy its a moot point.

[JC_BL]

Thanks all the tips that have been pouring in.  I have been checking this forum for the past few days and today.  But I was sick for one day, and I needed to check each suggestion carefully.  I can only report back until now.

I have checked the possibility of removing xdata and ldata.  I have tried the XDLIST command (from Express\xdata.lsp) and some other LISP programs from the net, and I cannot find any in the objects on the drawings.  This means I should focus on removing the excessive amount of graphical elements from the drawing instead of trying to remove hidden data.

I find that the explode/overkill/wblock commands actually work in reducing graphical elements from the drawing and reduce the drawing size.  I tried and found them didn't seem to work before.  Actually I might have forgot to do one thing: I need to ask AutoCAD _not_ to show proxy graphics after I have unloaded the third party app.  I need to unload the third party app in order to explode the objects.  And seem like I might have applied those commands on the proxy graphics by mistaken instead of the objects themselves.  That may have explained the reason why previously I didn't see any size reduction after trying those commands.

This time I have tried the commands sequence that dbroad has provided with the changes that I mentioned above, I can see a significantly size reduction.  The actual sequence is the following:

1.  Open the drawing.

2.  Unload the third party app, and ask AutoCAD not to show proxy graphics.

3.  Explode All

4.  Overkill All

5.  WBlock Objects 0,0...

The results are:

1.  A drawing that has one piece is reduced from 158 kb to 83 kb.

2.  A drawing that has some pieces is reduced from 713 kb to 257 kb.

3.  A drawing that has many pieces is reduced from 8156 kb to 3515 kb.

The result is not as low as dbroad has reported.  But this is still a very good size reduction.

By the way I cannot try the "flatten" command that dbroad has suggested.  Somehow that doesn't work in my drawing.  I am still looking into this issue.

Although the approach above works, the overkill command is very slow if the drawing has many pieces - takes 3 hours to process the drawing that has many pieces.  The workaround is to divide the drawing into smaller sections, and overkill each small section at a time.  The total time is reduced from 3 hours to 30 minutes.  This is a significant speed improvement.  But this is still very slow.  At the end I may not use this approach after all.

Currently I am playing around with the idea of using ExportLayout.  This is very fast and file size is very small (8156 kb reduced to 1552 kb).  Yes the drawing will lose some functionality.  But this is OK with the intended use in AutoCAD WS.

I have still not committed to use ExportLayout yet.  If the Flatten command works and the Overkill command works faster with flattened objects than non-flattened objects, I may go back to use flatten/overkill/wblock approach.  I will see how this goes.

Thanks for all the suggestions that I have received.  I appreciate your help.

Jay Chan

[JamesMaeding]

telling acad to no show the proxy grapgics is no good. You said those objects are part of your data, so they must be shown and exploded.

Use the map cleanup tools in Map 3D or C3D to get a 50x speed increase in the overkill part of things, its that much faster.

ts lame the arx prevents aec objects from being exploded, we would never tolerate that of Civil 3D.

[JC_BL]

Thanks for the suggestion.  But Map 3D is only available in Civil 3D version of AutoCAD.  We don't have it.  Oh well...

As of now, I am trying to figure out why I keep getting error message "Point or option keyword required" when I try to use Flatten command on a drawing.  That may have to do with some unit of measurement not compatible with that command in the drawing template.  I am still working on this.

Jay Chan

[JamesMaeding]

map is in both c3d and Map 3d, but if you don't have it....

I am working on ultra fast tools for those kinds of things, not done yet though.

[JC_BL]

How do you flatten the 3D wire frame objects on the drawing?  I am trying to do this for hours.  But I keep getting a "Point or option keyword required" error.  I tried unload the third party app and tried to ask AutoCAD to show or not to show proxy graphics - either way doesn't make any difference.

I can see that by flattening all the objects into 2D, the drawing size should be significantly reduced.  Therefore, I really want to be able to get this to work.

Would you please tell me how you did this in details?  Thanks in advance.

Jay Chan

[JamesMaeding]

Please post a dwg with the wireframe object, and I can take a shot at it.

If you did already, pls point me to the post, I did not look at your drawings yet.

You are on the right track though, dealing with those items correctly is the key.

Note that my prediction on the xdata was right, its generarally not the bulking problem, as not too many progs use tons of xdata, they keep it in other forms that get nuked with the explode process and wblock out.

[JC_BL]

Thanks for your help in advance in looking into the drawing.

I posted a drawing called Test_SomePieces.dwg in message 11 of this message thread.  That is the drawing that I am trying to flatten.

Jay Chan

[JamesMaeding]

I tried the same process in Civil 3D 2013, results were:

Exploded, overkilled, and wblock to new dwg (3d):

445kb

Exploded, flattened, overkilled, wblock to new dwg (2d):

379 kb.

So that is not bad, about 50% reduction.

I tried the map cleanup tools, not better end result, but likely faster process.

My tolerance for speed is waiting about 2 minutes.

Anything beyond that makes me think a lockup has happened.

[dbroad]

@Jay Chan,

I really don't know how to explain it better than I did >>>here<<<.

Open the drawing. Saveas to another drawing to preserve the original.  Explode all.  You may need to repeat this, depending on the nature of the objects to get to just lines.  Be sure you are in a top view.  Flatten all.  Choose no to the hide objects prompt.  There is no point argument required but flatten may have a bug I haven't discovered.  Use overkill to reduce the duplicate lines and to merge colinear lines.  Wblock the remaining objects to a new file.  Open that file and then purge (although I don't think that is necessary).  Qselect lines of 0 length (if they exist) and delete them.  Save it if the purge command or delete command worked otherwise close without saving.  

[JamesMaeding]

don't forget to run overkill.

The purge is not needed in the dwg you wblock to, the whole point of the wblock is it leaves behind anything invisible or not used that purge would remove.

Note that the "Civil Batch Converter" tool in Civil 3D does this batch style, without opening the drawings. Takes seconds per file.

Its the ultimate cleanup tool, and even exploded the aec objects.

It will not flatten or overkill though, but those steps are particular to the compromises you want for low file size.

[JC_BL]

Thanks for trying the drawing that I posted previously.

I found that the problem with Flatten has to do with something in my copy of AutoCAD.  I have no problem flattening the drawing using several other PCs.  This explains the reason why no one here has any problem flattening.

In any case, I have determined flattening indeed can reduce the file size - but just a little bit (this reduces the size of a file that has already shrinkled previously by an additional 3%).  This reduces the functionality of the drawing too much, and the pay back seems to be too little.  Therefore, I will not purse this idea further.  I will focus on the following approaches:

1.  ExportLayout to DWG file.  The process is fast and the resulted DWG file is small.

2.  Explode/Overkill/Wblock.  Unfortunately Overkill is too slow; therefore, I am not likely to use this approach.

3.  Upload DWF file to AutoCAD WS instead of uploading DWG file.  DWF file is quite small and the process of making DWF file is well known.  I will be trying out DWF Tool Kit 7.7 to see if I can write a C++ program to get access info in DWF file.

Thanks everyone who has helped me exploring different ideas.  I greatly appreciate that.

Jay Chan

[JamesMaeding]

I think there is a com (vba) and .net api to dwf also, that will be far easier than c++.

You can even use lisp with the com api.

[JC_BL]

Thanks for the suggestion of using VBA and .NET to get access to DWF files.

I will stick with C++ for now because it seems to be what the DWF Tool Kit was originally designed for (I want to have the least "surprise" as possible).  I have spent years in C++.  Therefore, using C++ is not an obstacle to me.

Jay Chan