Error 7

[Phillip Mathis]

I'm working with large .csv files (above 500,000 rows) and when I try to plot more than three columns from either Dplot or Excel I'm prompted with a run time error 7 "Not enough Memory". Is this a limitation of Dplot, lack of RAM(2G installed), or an Excel limitation?

Is there a work-around, besides splitting the files?

[DPlotAdmin]

Definitely not a DPlot limitation, though the Add-In may be fouling something up. What version of Excel? 32-bit or 64-bit?

[DPlotAdmin]

FYI I just now tried a sheet with 5 columns of 500,000 points and the XYYY command. It took at least a minute, but it worked: in DPlot I get 4 curves with 500,000 points each. This is with Excel 2010 64-bit. When saved as a CSV file DPlot read the file, no problem.

If there's any more information you can give me about what you're trying to do, what Add-In command you're using, etc., that might be helpful.

When reading a CSV file directly into DPlot, if you know how many data rows are in the file then you should first double-click the "Array size" area of the status bar and enter values large enough to accommodate your data. The reason for that is that if, for example, the max. number of data points is N and DPlot reads N rows but doesn't reach the end of the file, it estimates how many rows are in the file based on the file size and how many bytes were processed for those N rows. It might get this badly wrong if the first N rows use many fewer characters per row than the remainder of the file, and so overestimate the required number of points.

2Gb should be plenty for what you're talking about, assuming you aren't running everything under the sun. With the example above, Task Manager shows DPlot eating up ~46Mb.

[DPlotAdmin]

PS: With that same sheet open in Excel, Excel eats up 242Mb. You should still be able to do this on a 2Gb system. If not you might want to try rebooting. If that doesn't help then you'll obviously be better off reading the CSV file directly into DPlot with Excel closed.

[Phillip Mathis]

David, my computer crashed. I reloaded Dplot and got updates on my new computer(Windows 7 64 bit). I bought office 2010 and loaded it on my computer. I added Dplot as an add-in and this when i started having this problem. I was not having this problem with 2007.

When I adjusted the array size, Dplot will now plot these large files from the import function, but when plotting an XYYY from the Excel add-in it still gives the error 7 code when plotting more than 3 columns. The Debug function on the error form highlights at the ReDim X(DataRowCount * NumCurves -1) in DplotData.XLA...

I have feeling that I haven't set something up correctly in 2010. Just noticed that Dplot is in xla, and if I remember correctly 2007 is xls. Am I having a backwards compatibilty problem? Anyway, I hope this is the info you need.

Thanks

[DPlotAdmin]

I have feeling that I haven't set something up correctly in 2010. Just noticed that Dplot is in xla, and if I remember correctly 2007 is xls. Am I having a backwards compatibilty problem? Anyway, I hope this is the info you need.

The Add-In has always been an XLA file, not XLS. It is possible that I sent you an XLS file previously to help debug something. But there's nothing magic about an XLA file other than once set up it is automatically loaded when you start Excel.

"Error 7" is an Excel thing, nothing to do with DPlot. It indicates that you have insufficient memory for the operation intended. (And really, 2Gb is pretty lean for Windows 7.) On my same setup, for example (same OS and same Excel version) I have 6Gb. I know that doesn't make your day, sorry.

You can kinda sorta predict this with Task Manager. It will show how much memory Excel is gobbling up and you can also tell pretty quickly how much memory you need to send the data to DPlot. dplot.exe with a small default array size takes ~9Mb. The data you want to plot will require NC*NP*16 bytes, where NC is the number of curves and NP is the number of points/curve.

The bad thing (for low memory situations) is that the requirements are at least doubled when sending data to DPlot from Excel: Excel owns one copy, DPlot the other. Actually it's quite a bit worse than that as Excel stores more information than just the raw data values (formulas, formatting, etc.). As mentioned previously, a data set that takes ~46Mb in DPlot takes ~240Mb in Excel. This relationship will vary depending on the Add-In command used, but the important part is that the total memory requirement will be, at the very least, twice what is eventually required in DPlot. So, short version, if you're running out of memory, save your sheet as a CSV file, exit Excel, and import that file directly into DPlot (for XYXY, XYYY, XYZSurface. For other data types... no surefire solution other than installing more RAM.)

I'll at least think about intercepting that particular error and offering suggestions within the Add-in. For the XYYY, XYXY, and XYZSurface commands the solution is fairly simple: save as CSV, exit Excel, import into DPlot. Some of the other Add-In functions aren't that straightforward, but those 3 will cover the majority of cases (he says, crossing his fingers).

[Phillip Mathis]

Got it, Thanks David.