Displaying DB results too slow

Displaying DB results too slow

Anonymous
Not applicable
435 Views
10 Replies
Message 1 of 11

Displaying DB results too slow

Anonymous
Not applicable
Hello

I've written a macro that works well, it's only problem is speed.
It utilities an .mdb database with with a single table of approx. 35,000 rows.

The SQL queries are quite quick; a string with LIKE and wildcars searching in multiple columns, that I know won't return any results, finishes in about 2s.

The problem is the UI. I have a form with a ListBox, that I populate with the the results from the query with a DO-LOOP. If I wish to view all 35k entrys, it takes more than 10s to display the form.

As the form is displayed for every block entered, this is unacceptable.

So far, the only thing I found that may help is the DoEvents statement. The Idea is to let the user browse the ListBox even before it's fully populated, but everything I've tried with it doesn't work and the documentation on the function is pretty sparse.

Any ideas/comments/suggestions?
0 Likes
436 Views
10 Replies
Replies (10)
Message 2 of 11

Anonymous
Not applicable
My only suggestion is do not put 35,000 items in a combobox. Find a way filter it based on some criteria to reduce that number. It's holding that much info in memory and probably has to write to the hard drive to hold that much info on memory since it probably exceeds your RAM. I'm surprised it didn't crash AutoCAD.
0 Likes
Message 3 of 11

Anonymous
Not applicable
Take a look at RowSourceType and
RowSource in the Access Help

~'J'~
0 Likes
Message 4 of 11

dgorsman
Consultant
Consultant
Are you continuously rebuilding the list each time a block is inserted?
Does the database change in that interval?
Do you require every single piece of data in the database every single time?

10 seconds (I hope I read that right) is a reasonable amount of time to be waiting for large data sets.
----------------------------------
If you are going to fly by the seat of your pants, expect friction burns.
"I don't know" is the beginning of knowledge, not the end.


0 Likes
Message 5 of 11

Anonymous
Not applicable
Move *everything* in your code that you can, out of the do-loop. There may well be code that can be performed once before/after the loop is done. Secondly, lock/disable the combobox while you re seeding it, then unlock/enable it when you are done - that way, if you have them displayed in some kind of sorted order, or generating a refresh, the sort (or refresh ) isnt re-running every time you add an item to the combobox.
0 Likes
Message 6 of 11

Anonymous
Not applicable
It's a ListBox, not a combobox.

The criteria is the block name, witch is the basis of an SQL search. But sometimes, it doesn't find anything, so I have to display everything. Hope that makes sense.

As for the stability:
Memory allocated to Autocad goes from 120MB to 134MB immediately upon load, and further to 148MB after the search ends and the ListBox is populated.
Aside from the wait, there seem to be no degradation of stability.
0 Likes
Message 7 of 11

Anonymous
Not applicable
This was my first attempt. But the form also has a culpe of TextBoxes that display data from other columns, depending on witch row is selected in the ListBox.
0 Likes
Message 8 of 11

Anonymous
Not applicable
Well, here's the code, if anyone can find lines that can be trimmed:

Private Sub UserForm_Initialize()

If effName = "" Then
MsgBox "Invalid block name" & vbNewLine & "all database entries will be shown."
effName = "*"
End If


Set baza = DBEngine.Workspaces(0).OpenDatabase("C:\AtttribDB\sifrant.mdb")
Set recset = baza.OpenRecordset("Artikli", dbOpenDynaset)

Dim sqlString As String
Dim i As Long
Dim effNameSplit As Variant
effNameSplit = Split(effName)
sqlString = "Naziv LIKE '*" & effNameSplit(i) & "*' OR NazivDob LIKE '*" & effNameSplit(i) & "*' OR Opis LIKE '*" & effNameSplit(i) & "*'"
i = 1
If i < UBound(effNameSplit) Then
Do
sqlString = sqlString & "OR Naziv LIKE '*" & effNameSplit(i) & "*' OR NazivDob LIKE '*" & effNameSplit(i) & "*' OR Opis LIKE '*" & effNameSplit(i) & "*'"
i = i + 1
Loop Until UBound(effNameSplit)
End If
recset.FindFirst sqlString
i = 0

If recset.NoMatch = True Then
MsgBox "Inserted element does not match any database entries;" & vbNewLine & "all database entries will be shown."
recset.MoveFirst
Do
recset.MoveNext
If recset.EOF = True Then
Exit Do
Else
i = i + 1
End If
Loop
' THIS DO-LOOP IS QUITE FAST (SUB 1 SECOND)
ReDim listPos(i) As Long
i = 0
recset.MoveFirst
Dim dbcount As Long
dbcount = 0
Do Until recset.EOF = True
lbxNaziv.AddItem (recset!Naziv)
listPos(i) = recset.AbsolutePosition
recset.MoveNext
dbcount = dbcount + 1
i = i + 1
Loop
'THIS ONE IS THE CULPRIT
recset.MoveFirst
txtGovkoda = recset!Koda
txtOpis = recset!opis
txtProizvajalec = recset!Dobavitelj

Else
Do
recset.FindNext sqlString
If recset.NoMatch = True Then
Exit Do
Else
i = i + 1
End If
Loop
ReDim listPos(i) As Long
i = 0
recset.FindFirst sqlString
lbxNaziv.Clear
txtGovkoda = recset!Koda
txtOpis = recset!opis
txtProizvajalec = recset!Dobavitelj
Do
lbxNaziv.AddItem (recset!Naziv)
listPos(i) = recset.AbsolutePosition
i = i + 1
recset.FindNext sqlString
If recset.NoMatch = True Then
Exit Do
End If
Loop
effName = ""
End If

txtFind.SetFocus

End Sub



I've translated a few of the strings, but some of the variables will be confusing to English speakers. Just think, database, Manufacturer, ProductCode,....
0 Likes
Message 9 of 11

Anonymous
Not applicable
Maybe this is more of a connectivity issue after all!

I haven't analyzed it in detail, but two points:

1. It looks as if you are using DAO. That has been obsolescent (not
obsolete) for several years. ActiveX Data Objects (ADO) is its replacement,
and is much more full-featured and likely to be faster. It's also much
easier and simpler to use.

2. It looks as if you are grabbing the whole table into a recordset then
using your SQL string to repeatedly query that recordset. If I'm right then
that's very inefficient. The database driver is heavily optimized for
retrieval, so it should be doing that job rather than having VBA do it, and
you only need to apply the SQL once, and you don't need to retrieve all the
columns. Here's an example of doing it with ADO:

Dim oConn As ADODB.Connection
Dim oRS As ADODB.Recordset
Set oConn = New ADODB.Connection
Set oRS = New ADODB.Recordset
oConn.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;persist
security info=false"
oConn.Open "C:\AtttribDB\sifrant.mdb"

Dim sqlString As String
Dim i As Long
Dim effNameSplit As Variant
effNameSplit = Split(effName)

' Note the change in the SQL statement
sqlString = "Select Koda, opis, Dobavitelj from Artikli where Naziv
LIKE '*" & effNameSplit(i) & "*' OR NazivDob LIKE '*" & effNameSplit(i) &
"*' OR Opis LIKE '*" & effNameSplit(i) & "*'"
i = 1
If i < UBound(effNameSplit) Then
Do
sqlString = sqlString & "OR Naziv LIKE '*" & effNameSplit(i) &
"*' OR NazivDob LIKE '*" & effNameSplit(i) & "*' OR Opis LIKE '*" &
effNameSplit(i) & "*'"
i = i + 1
Loop Until UBound(effNameSplit)
End If

' Back into the new stuff
oRS.Open sqlString, oConn
' Now you have a recordset of zero or more length, containing
' only records that match your SQL statement.
' Do a MoveLast then MoveFirst to populate the size before querying it
' If it's zero length then you can grab the whole table with
' oRS.Close then
' oRS.Open "Select "Select Koda, opis, Dobavitelj from Artikli", oConn
.
.
.
oRS = Nothing
oConn.Close
oConn = Nothing


--
jrf
Autodesk Discussion Group Facilitator
Please do not email questions unless you wish to hire my services
On Thu, 10 Jan 2008 19:02:49 +0000, gregri wrote:

> Well, here's the code, if anyone can find lines that can be trimmed:
>
> Private Sub UserForm_Initialize()
>
> If effName = "" Then
> MsgBox "Invalid block name" & vbNewLine & "all database entries
will be shown."
> effName = "*"
> End If
>
>
> Set baza =
DBEngine.Workspaces(0).OpenDatabase("C:\AtttribDB\sifrant.mdb")
> Set recset = baza.OpenRecordset("Artikli", dbOpenDynaset)
>
> Dim sqlString As String
> Dim i As Long
> Dim effNameSplit As Variant
> effNameSplit = Split(effName)
> sqlString = "Naziv LIKE '*" & effNameSplit(i) & "*' OR NazivDob LIKE
'*" & effNameSplit(i) & "*' OR Opis LIKE '*" & effNameSplit(i) & "*'"
> i = 1
> If i < UBound(effNameSplit) Then
> Do
> sqlString = sqlString & "OR Naziv LIKE '*" & effNameSplit(i)
& "*' OR NazivDob LIKE '*" & effNameSplit(i) & "*' OR Opis LIKE '*" &
effNameSplit(i) & "*'"
> i = i + 1
> Loop Until UBound(effNameSplit)
> End If
> recset.FindFirst sqlString
> i = 0
>
> If recset.NoMatch = True Then
> MsgBox "Inserted element does not match any database entries;" &
vbNewLine & "all database entries will be shown."
> recset.MoveFirst
> Do
> recset.MoveNext
> If recset.EOF = True Then
> Exit Do
> Else
> i = i + 1
> End If
> Loop
> ' THIS DO-LOOP IS QUITE FAST (SUB 1 SECOND)
> ReDim listPos(i) As Long
> i = 0
> recset.MoveFirst
> Dim dbcount As Long
> dbcount = 0
> Do Until recset.EOF = True
> lbxNaziv.AddItem (recset!Naziv)
> listPos(i) = recset.AbsolutePosition
> recset.MoveNext
> dbcount = dbcount + 1
> i = i + 1
> Loop
> 'THIS ONE IS THE CULPRIT
> recset.MoveFirst
> txtGovkoda = recset!Koda
> txtOpis = recset!opis
> txtProizvajalec = recset!Dobavitelj
>
> Else
> Do
> recset.FindNext sqlString
> If recset.NoMatch = True Then
> Exit Do
> Else
> i = i + 1
> End If
> Loop
> ReDim listPos(i) As Long
> i = 0
> recset.FindFirst sqlString
> lbxNaziv.Clear
> txtGovkoda = recset!Koda
> txtOpis = recset!opis
> txtProizvajalec = recset!Dobavitelj
> Do
> lbxNaziv.AddItem (recset!Naziv)
> listPos(i) = recset.AbsolutePosition
> i = i + 1
> recset.FindNext sqlString
> If recset.NoMatch = True Then
> Exit Do
> End If
> Loop
> effName = ""
> End If
>
> txtFind.SetFocus
>
> End Sub
>
>
>
> I've translated a few of the strings, but some of the variables will be
confusing to English speakers. Just think, database, Manufacturer,
ProductCode,....
0 Likes
Message 10 of 11

Anonymous
Not applicable
Hi,

Another point to consider is the practically of the design intent.

No matter how long it takes to load it, no user is going to want to
manipulate their way through that many lines in a list box.

Think of an alternative action for when you don't find the required data.

--


Regards

Laurie Comerford

"Jon Fleming" wrote in message
news:5817999@discussion.autodesk.com...
Maybe this is more of a connectivity issue after all!

I haven't analyzed it in detail, but two points:

1. It looks as if you are using DAO. That has been obsolescent (not
obsolete) for several years. ActiveX Data Objects (ADO) is its replacement,
and is much more full-featured and likely to be faster. It's also much
easier and simpler to use.

2. It looks as if you are grabbing the whole table into a recordset then
using your SQL string to repeatedly query that recordset. If I'm right then
that's very inefficient. The database driver is heavily optimized for
retrieval, so it should be doing that job rather than having VBA do it, and
you only need to apply the SQL once, and you don't need to retrieve all the
columns. Here's an example of doing it with ADO:

Dim oConn As ADODB.Connection
Dim oRS As ADODB.Recordset
Set oConn = New ADODB.Connection
Set oRS = New ADODB.Recordset
oConn.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;persist
security info=false"
oConn.Open "C:\AtttribDB\sifrant.mdb"

Dim sqlString As String
Dim i As Long
Dim effNameSplit As Variant
effNameSplit = Split(effName)

' Note the change in the SQL statement
sqlString = "Select Koda, opis, Dobavitelj from Artikli where Naziv
LIKE '*" & effNameSplit(i) & "*' OR NazivDob LIKE '*" & effNameSplit(i) &
"*' OR Opis LIKE '*" & effNameSplit(i) & "*'"
i = 1
If i < UBound(effNameSplit) Then
Do
sqlString = sqlString & "OR Naziv LIKE '*" & effNameSplit(i) &
"*' OR NazivDob LIKE '*" & effNameSplit(i) & "*' OR Opis LIKE '*" &
effNameSplit(i) & "*'"
i = i + 1
Loop Until UBound(effNameSplit)
End If

' Back into the new stuff
oRS.Open sqlString, oConn
' Now you have a recordset of zero or more length, containing
' only records that match your SQL statement.
' Do a MoveLast then MoveFirst to populate the size before querying it
' If it's zero length then you can grab the whole table with
' oRS.Close then
' oRS.Open "Select "Select Koda, opis, Dobavitelj from Artikli", oConn
.
.
.
oRS = Nothing
oConn.Close
oConn = Nothing


--
jrf
Autodesk Discussion Group Facilitator
Please do not email questions unless you wish to hire my services
On Thu, 10 Jan 2008 19:02:49 +0000, gregri wrote:

> Well, here's the code, if anyone can find lines that can be trimmed:
>
> Private Sub UserForm_Initialize()
>
> If effName = "" Then
> MsgBox "Invalid block name" & vbNewLine & "all database entries
will be shown."
> effName = "*"
> End If
>
>
> Set baza =
DBEngine.Workspaces(0).OpenDatabase("C:\AtttribDB\sifrant.mdb")
> Set recset = baza.OpenRecordset("Artikli", dbOpenDynaset)
>
> Dim sqlString As String
> Dim i As Long
> Dim effNameSplit As Variant
> effNameSplit = Split(effName)
> sqlString = "Naziv LIKE '*" & effNameSplit(i) & "*' OR NazivDob LIKE
'*" & effNameSplit(i) & "*' OR Opis LIKE '*" & effNameSplit(i) & "*'"
> i = 1
> If i < UBound(effNameSplit) Then
> Do
> sqlString = sqlString & "OR Naziv LIKE '*" & effNameSplit(i)
& "*' OR NazivDob LIKE '*" & effNameSplit(i) & "*' OR Opis LIKE '*" &
effNameSplit(i) & "*'"
> i = i + 1
> Loop Until UBound(effNameSplit)
> End If
> recset.FindFirst sqlString
> i = 0
>
> If recset.NoMatch = True Then
> MsgBox "Inserted element does not match any database entries;" &
vbNewLine & "all database entries will be shown."
> recset.MoveFirst
> Do
> recset.MoveNext
> If recset.EOF = True Then
> Exit Do
> Else
> i = i + 1
> End If
> Loop
> ' THIS DO-LOOP IS QUITE FAST (SUB 1 SECOND)
> ReDim listPos(i) As Long
> i = 0
> recset.MoveFirst
> Dim dbcount As Long
> dbcount = 0
> Do Until recset.EOF = True
> lbxNaziv.AddItem (recset!Naziv)
> listPos(i) = recset.AbsolutePosition
> recset.MoveNext
> dbcount = dbcount + 1
> i = i + 1
> Loop
> 'THIS ONE IS THE CULPRIT
> recset.MoveFirst
> txtGovkoda = recset!Koda
> txtOpis = recset!opis
> txtProizvajalec = recset!Dobavitelj
>
> Else
> Do
> recset.FindNext sqlString
> If recset.NoMatch = True Then
> Exit Do
> Else
> i = i + 1
> End If
> Loop
> ReDim listPos(i) As Long
> i = 0
> recset.FindFirst sqlString
> lbxNaziv.Clear
> txtGovkoda = recset!Koda
> txtOpis = recset!opis
> txtProizvajalec = recset!Dobavitelj
> Do
> lbxNaziv.AddItem (recset!Naziv)
> listPos(i) = recset.AbsolutePosition
> i = i + 1
> recset.FindNext sqlString
> If recset.NoMatch = True Then
> Exit Do
> End If
> Loop
> effName = ""
> End If
>
> txtFind.SetFocus
>
> End Sub
>
>
>
> I've translated a few of the strings, but some of the variables will be
confusing to English speakers. Just think, database, Manufacturer,
ProductCode,....
0 Likes
Message 11 of 11

Anonymous
Not applicable
Good point.

--
jrf
Autodesk Discussion Group Facilitator
Please do not email questions unless you wish to hire my services

On Sun, 13 Jan 2008 09:00:48 +0000, Laurie Comerford wrote:

> Hi,
>
> Another point to consider is the practically of the design intent.
>
> No matter how long it takes to load it, no user is going to want to
> manipulate their way through that many lines in a list box.
>
> Think of an alternative action for when you don't find the required data.
0 Likes