Out Of Memory Exception For Large Result Set

Jul 13, 2010 at 4:20 PM

Sir,

When comparing result set for around ~ 200000 records i am getting Out Of memory exception. The exception is happening when the tool is evaluating the differences.

Coordinator
Jul 13, 2010 at 5:05 PM
As all processing is done in memory, you may run into limits depending on the shape of the data (i.e. both row count and column count are factors). In my own experience I have successfully run over 500,000 records with 10 or so columns. There are a few things you should check: -- Your machine: How much memory do you have on your machine? If at least 3GB did you choose the install option for large address space model? See section 2.1.2 "Component Options" in the user manual. -- Your difference engine: SqlDiffFramework includes 3 separate difference engines that vary dramatically in their memory usage (the Hertel engine consumes the most memory). These are selectable on the dropdown attached to the Diff button. See section 4.1.1.9 "Toggle Auto-Differencing Mode" in the user manual. If those suggestions do not help, you will need to divide up your data into multiple data sets. Either split up your columns (reduce the number of fields in your select) or split up your rows (add a WHERE clause to select just half the number of rows). Let me know how it goes! ~~Michael
Jul 15, 2010 at 3:38 PM
Edited Jul 15, 2010 at 3:44 PM
msorens wrote:
As all processing is done in memory, you may run into limits depending on the shape of the data (i.e. both row count and column count are factors). In my own experience I have successfully run over 500,000 records with 10 or so columns. There are a few things you should check: -- Your machine: How much memory do you have on your machine? If at least 3GB did you choose the install option for large address space model? See section 2.1.2 "Component Options" in the user manual. -- Your difference engine: SqlDiffFramework includes 3 separate difference engines that vary dramatically in their memory usage (the Hertel engine consumes the most memory). These are selectable on the dropdown attached to the Diff button. See section 4.1.1.9 "Toggle Auto-Differencing Mode" in the user manual. If those suggestions do not help, you will need to divide up your data into multiple data sets. Either split up your columns (reduce the number of fields in your select) or split up your rows (add a WHERE clause to select just half the number of rows). Let me know how it goes! ~~Michael

   Sir,

   I have 2Gb RAM. Changing the difference engine did not help much. I have split the number of rows i get back from DB and i am fine now.

 Thanks,

 Pradeep.Y