Problem with CAM Processor BOM regarding attributes including the " character

[tedj1]

Hi, wanted to note a complicating factor with the CAM Processor.

The presence of a " character in an attribute of a device can complicate the later import of a BOM into a worksheet program, for example the one that is part of OpenOffice. If the 'List attributes' box is unchecked, then one can export as text and import as fixed width columns, and begin importing at the second line or reassemble the first line after import. One can export as CVS and import using " as the Text delimiter and loose all the lines past the offending attribute (not good), or clear the text delimiter field and maintain all the double quotes in all fields, which might complicate things for the manufacturer, or manually search and strike all double quotes and lose the ones indicating "inch" within the offending attribute, or laboriously strike being careful to leave the ones that aren't delimiters.

One can check the 'List attributes' box and list many parts duplicated with quantity 1 for each row, which some manufacturers may not like or manually condense the rows while counting quantities.

One could identify those offending attributes and modify the devices in libraries in which they exist, which is laborious. Work-arounds might be to automatically replace " characters with 'inch' or 'in', complain to part manufacturers to please not use " characters in their part attributes and instead use 'inch' or 'in' when necessary, or (my favorite) not include " characters redundantly delimiting fields in the BOM in addition to the ';' delimiters, which are sufficient (assuming part manufacturers don't sometimes include ';' characters in attribute fields.  Just trying to help and smooth the workflow.  I've attached a typical CVS export BOM which demonstrates the problem.

[jorge_garcia]

Hi @tedj1,

I hope you're doing well. Typically when I do this I will export as CSV and list the attributes. The only delimiter you should ever use when exporting CSV is the semi-colon. If you only use the ; you should be good.

I may be misunderstanding what you are trying to do. Are you actually adding the " to the text in your design? I don't think that's the case but I want to make sure I'm not missing something.

Let me know if there's anything else I can do for you.

Best Regards,

[one-of-the-robs]

---

@jorge_garcia wrote:
I may be misunderstanding what you are trying to do. Are you actually adding the " to the text in your design? I don't think that's the case but I want to make sure I'm not missing something.

---

I'd read it as being exactly that. A possible case where you might do so is if you add a descriptive attribute to a 0.1" pin header, for example. I know I've hit a problem with such things in the past (although it wasn't in an attribute but rather in the parts database for the old BoM export ULP).

[tedj1]

I hope you're doing well. Typically when I do this I will export as CSV and list the attributes. The only delimiter you should ever use when exporting CSV is the semi-colon. If you only use the ; you should be good.

Hi jorge.  I do the same.  The issue comes into play occasionally when I've downloaded parts to the SamacSys library with the Library Loader.  I don't add them myself.  I'm curious why Eagle includes not only the ';' character as a delimiter, but also redundantly includes " characters surrounding each field as well.  The combination causes the trouble.  I wasn't aware I had a choice in what delimiter Eagle uses when it exports CSV files during CAM Processing.

[jorge_garcia]

Hi @tedj1,

I hope you're doing well. By default EAGLE uses the ; as the delimiter. It does enclose the value of every field in " " . When you open it in LibreOffice, Calc should only be concerned with what the delimiter for the fields is which is ;. I was able to import the csv without issue into Calc just by going File > Open select the csv file. In the import settings the delimiter should only be semi-colon nothing else. If you see a setting for TEXT delimiter you can set it to " or if you want everything to come in as is pick something else as the text delimiter.

Let me know if there's anything else I can do for you.

Best Regards,

[tedj1]

Hi Jorge.  How are you?  Try the attached file.  It produces the problem on my end.  Does it work for you?   I tried downloading the one I originally posted, but it's not producing the problem.  I may have inadvertently downloaded one that I already fixed.  Sorry.  Under List Types, I chose 'Values' and unchecked 'List Attributes' and Output Type 'CSV'.

[jorge_garcia]

Hello @tedj1,

I'm using the latest stable version of LibreOffice(6.3.2). I imported this one and it came in fine also.

Let me know if there's anything else I can do for you.

Best Regards,

[tedj1]

On import, if I 'Separate by' a semicolon, then I see no lines below the fourth (the fourth just continues and continues to the right) if leaving double quote as 'Text Delimeter'.  Or if I clear the 'Text Delimiter', then all lines are included and all fields are surrounded by double quotes.  That's acceptable, if the manufacturer has no problem with all fields being surrounded by double quotes.  Thanks.

[one-of-the-robs]

This is the offending 4th line:

"3";"xxx-xxxxxxxx";"xxx-xxxxxxxx";"HDRV3W95P0X254_1X3_762X254X869";"J5, J8, J15";"HEADER 3 POSITION 2.54MM 0.100" VERTICLE UNSHROUDED BREAKAWAY THROUGH-HOLE";"";

Note that the penultimate field, the "Description", includes

2.54MM 0.100" VERTICLE UNSHROUDED

See, there's a terminating quote in the middle of the string, which will put LibreOffice out of sync and cause all subsequent line breaks to be quoted as part of a cell.

Exactly as I predicted earlier.

[tedj1]

Exactly my point as I carefully explained earlier.  Why not just dispense with redundant double quotes surrounding all fields?  The ; character is sufficient.

[jorge_garcia]

Hi @one-of-the-robs , @tedj1 ,

This is really weird, on Windows LibreOffice 6.3.2 Calc processes the file correctly as shown in the attached screenshot. I find it hard to believe that there would be a difference because of Windows vs Linux. I have to believe that there is some difference in the versions being used.

I can request a bugfix if I can't duplicate the bug. The situation seems to point more to a deficiency in older versions of LibreOffice then an actual issue with the bom ulp. This ULP has been around almost since the beginning of EAGLE so making this type of change could result in some unintended consequences.

Let me know if there's anything else I can do for you.

Best Regards,

[one-of-the-robs]

It may well be that some versions of LibreOffice are tolerant of the error (6.2.5 on Linux is partially) but CSV files using quote characters to "hide" potential delimiters within cells should never include an unescaped quote character within a string. The correct behaviour, which Eagle should follow, is to double-up on them, i.e.

;"HEADER 3 POSITION 2.54MM 0.100"" VERTICLE UNSHROUDED BREAKAWAY THROUGH-HOLE";

This is what LibreOffice calc does if you ask it to re-save the file.

As to unintended consequences... I've been using a Python script to repair this error (indirectly - I export the BoM in fixed-width and the Python script converts it to the correctly formatted CSV) for years. Any breakage is likely to just require removal of such work-arounds.