Good to hear from you Bob!
Merging my sql statements together is exactly what I have been doing the last 24 hours, unfortunately without luck.
For inserts, I have used this syntax:
INSERT INTO table (column_1,column2) VALUES
('value 1 for column 1','value 1 for column 2')
,(value 2 for column 1','value 2 for column 2')
,(value 3 for column 1','value 3 for column 2')
..etc
The problem is that the majority of my queries are update queries, so I changed my DSN to be able to handle multiple queries within the same <cfquery> by adding "allowMultipleQuery=true" in the text box "connection string" under advanced settings for my DSN.
Then I merged chunks of 250 update statements together throughout my loop (of about 6000 rows), and then outputted them in a <cfquery>, then starting on a new chunk and so on. This decreased the number of queries vastly off course, but my request ran even slower.
I produce the sql statements as strings (off course), and therefore I can't use any cfqueryparams, but that's ok. I read that they could cause memory leaks in some situations, so in this case, I was only glad to be able to remove them (I ALWAYS use them otherwise, to avoid sql injection off course).
I should also mention that I have a cftransaction around every 250 rows, that commits the insert and update queries, throughout the loop (every 250th row). From the beginning I had the transaction around the entire loop, which worked pretty well up to about 5000 rows.
Now I'm looking into memory leak issues. One guy wrote that he had a similar project like ours, but had problems with the memory usage, so he said he wrote a function that imported a part of the content, then "did a gateway call to the CFC" at the end, to the same function, with an argument that told the function to import the next part and so on. By doing this he said that the memory was flushed during the import, between the different parts of the import, resolving his issue. You can read about it here (one of the replies)
http://stackoverflow.com/questions/2364682/memory-implications-of-retu rning-a-query-from-a-cfc
I'm not sure though what he means with "gateway call". If it has anything to do with event gateways? If so, I'm really lost. Never tried that.
Anyway, now I will try to run the cf debugger, and see how my queries are doing, hopefully it will bring some clarity. I'm in Sweden, so it's getting late. 10:14 PM.
/Jörgen