DPlot

Hi. I am implementing a file conversion software from my raw data file to DPlot unformatted binary format (Option E).

When the created file contains some large number of sample points, DPlot can't load that file. For the small sized file, DPlot has no problem to open the file created by my program.

I tried to find this limitation at DPlot help file, but it wasn't successful.

If I get the maximum limitation in terms of the number of sample points, I will modify my program to create several DPlot files.

Thanks,

Do Park

Do,
The only limit is the maximum magnitude of a signed 32-bit integer: 2^31 = 2147483648 (but you'd no doubt run out of memory if trying to read that many points). How many points are you trying to read? And what exactly is the error message you get?

The error message looks like

"Insufficient Memory for 138412032 data points".

The reason why I am asking you the maximum limitation is... my raw data file can contains the very large number of data samples such as 150M Samples ( = 2.5 M Samples X 60 Seconds) or 1200M Samples ( = 20 M Samplex X 60 Seconds) or more.
I think a few plot software can handle with this kind of big data file in earth.

So, the proper way for this big file will be to create several files that contains a small chunk of the whole data.

Thanks for your quick response.

You're welcome. But just so we're clear: the limitation here isn't so much DPlot as it is your available memory. If you had the memory for 138 million points (16 bytes/point) DPlot would work with it just fine. You can try forcing Windows to use a larger chunk of your hard disk for virtual memory and it will probably work, though it will be so slow that you'll most likely find it to be useless.

Hi. I increased the virtual memory space to 4096MB in my PC which has Windows XP Professional.

But I still have the same error message.
How big virtual memory are you recommending ?

First... I misspoke earlier and I need to rewrite some of my documentation. DPlot will fail to allocate memory for an array that requires more than 2Gb. So in effect the limit is 1/8 what I said before: 2147483648/(8 bytes per X or Y) = 268,435,456 points total for all curves.

Just so we're clear, I'm not recommending any virtual memory setting. DPlot will be painfully slow if you're working with data that requires VM, as would any other program. But if you really want to do this, your best bet is to start DPlot w/o a plot, then run Task Manager and click the "Performance" tab to see how much memory you're currently using and how much is available ("Mem Usage" at the bottom of the window). Subtract used from available. If the result is not greater than 16*N (where N is the number of points you want to plot), then you'll need to increase your VM maximum size. This is assuming you're using a fixed page file size so that the "available" setting shown in Task Manager is truly the available amount. To use a fixed size page file set the "Initial Size" and "Maximum Size" to the same value.

But again... unless you're willing to be very, very patient I wouldn't bother with this.

One of the engineers in our mechanical analysis group was looking for some inexpensive software to recommend to his "customers" in our company. I had suggested DPlot, but then he tried it on a 656Mb 4-column x 13,057,002 row ascii file. It was one of his "smaller" files! I didn't try tinkering with virtual memory (I have 1gb actual RAM on a 1.73 gHz year-old laptop running XPProSP2), but trying to load this file into DPlot (2.1.0.6) was an immediate crash. I split the file and loaded in about 4 million rows x 4y without Dplot complaining. But speed was close to my limit of patience - I might be looking for another tool, or splitting up the file into smaller chunks.

Nowadays, memory management and software tuning seems to have gone by the wayside because of the ready availability of RAM and cpu speed, but is there any possibility of tweaking DPlot to handle these larger datasets? (I still go back to a version 5 of Paint Shop Pro from the early days because it eats big files so much better than the newer versions or competing software, because it seems to have been designed in a day when memory and cpu speed was much more limited.)

Another possibility might be to come up with an internal "DSplit" (!) function in the file open dialog to allow the user to specify some range of data to work with, ie, start at 30% and read to 50%, and let DPlot extract this "window" from the main file. Is that something you have or would consider for the future?

...but trying to load this file into DPlot (2.1.0.6) was an immediate crash.

More than likely there was something about the file that DPlot didn't like, rather than the number of points being a problem (but please don't send me the file to test it ). If it was an "immediate" crash then DPlot didn't have time to read enough points to know it should die . And you can fairly easily test the extremes using something like Y=f(X) with a very small interval. After 20 million or so on my system DPlot gets very, very sluggish (as VM is used), but I can't increase the number of points enough to make it do anything ugly.

Having said all that, though... I need to test this tonight with a humongous file. It's possible that a crash might be occuring due to an integer overflow when DPlot is estimating the number of points based on the file size.

Nowadays, memory management and software tuning seems to have gone by the wayside...

Yes it does. Before Windows, a DOS-based version of DPlot used a lot of overlays and all sorts of weird stuff to plot very large data sets. It was slow as molasses, but it worked.

Another possibility might be to come up with an internal "DSplit" (!) function in the file open dialog to allow the user to specify some range of data to work with, ie, start at 30% and read to 50%, and let DPlot extract this "window" from the main file. Is that something you have or would consider for the future?

Sure, that's a good idea.

David, The original file was a tab delimited text file with no headers, ie, all digits in line 1. I loaded the file into UltraEdit, and added a Y0{tab}Y1{tab}Y2{tab}Y3{CR} to the first line and resaved the file. It took about a minute to even write the file to the hard drive. I then started DPlot, did the File>Open>D-Multiple columns> Pick columns to plot, clicked 0 in the Use Column n for X axis, click the first row in the Use these columns for Y then click ok and DPlot immediately "disappeared". Afterward, I did try adjusting the array size before loading, and DPlot did complain that the 4x13,000,000 was too large of an array. I do forget what it did allow, but after splitting the file into 100mb chunks, it read the first 2 million rows. Then I combined the first two files into a 200mb file, and Dplot handled 4 million rows, but it was getting sluggish. Task Manager indicated about a 250 mb load for DPlot with this number of points.

Thanks for considering the possibility of the "DSplit" concept. I think that might be attractive to a lot of your users. Might that be worth a poll or a topic to get other peoples suggestions? (Unless you are overloaded with "Suggestions" already!)

Two things, neither one of them especially good. Just wanted to check in and let you know I'm working on it.

1) The crash when reading the file. Stupid mistake that is fixed and will be available in the next release. But.... aaarrggghh...

2) I apparently don't know as much about memory allocation in Windows as I thought I did. Old versions of DPlot used a compiler-specific memory allocation function that would definitely use virtual memory if necessary. The only problem with that function is that the memory would remain committed to DPlot until DPlot shut down. So if you plotted a zillion points, then closed that document but not DPlot, you'd still be short on (as in no) RAM for use in other programs. So I started using Windows API stuff for memory allocation, and I thought those would use the page file, in fact was reasonably sure that I had tested that it would. And the docs say they should. But no matter what I set the page file size to, memory allocation fails for me at ~1.2Gb, works fine for half of that. I haven't investigated enough to find what the actual limit on this system is, but am not sure that's really important. Short version: I'm working on it, and this is my priority until I figure it out.

Didn't mean to open a can of worms!

Thanks for taking a look.

Well... the can of worms was already open, you just called attention to it
I appreciate your input.