Avoiding File Fragmentation when logging

[John Owen (EnTegra Ltd)]

It is undesirable when logging to disk to create a file that is fragmented. Firstly that can impact disk access speed when writing since the heads have to seek further, and secondly it can impact disk access speed wehn reading for post processing.

I have found through experimenting (on an NTFS disk system) that by setting a filesize when creating the file, that fragmentation is avoided, unless there aren't enough contiguous clusters to store the file, in which case you need to clean up the disk and defragment it.

To set the filesize, use:
// Set expected filesize
_LARGE_INTEGER FilePos;
FilePos.QuadPart = TargetSize;
SetFilePointerEx( FileHandle, FilePos, NULL, FILE_BEGIN );
SetEndOfFile( FileHandle );

where TargetSize is an __int64.

[John Owen (EnTegra Ltd)]

Once the filesize is set to the expected filesize, then the file pointer must be repositioned to the beginning of the file,
// Move FP back to beginning
FilePos.QuadPart = 0;
SetFilePointerEx( FileHandle, FilePos, NULL, FILE_BEGIN );

During writing to the file, the class must keep track of how many bytes have been written, because before the file is closed the actual size should be set again incase the file did not receive the TargetSize bytes. ie if the TargetSize was 1GB but only 900MB's were written, then the size must be set back to 900MB otherwise the file will be listed as 1GB and the last 100MB will be garbage.
// Set actual size of number of bytes that was written to it
SetFilePointerEx( FileHandle, FilePos, NULL, FILE_BEGIN );
SetEndOfFile( FileHandle );

// Close File
CloseHandle( FileHandle );

If your application requires that you log a very large amount of data, but the actual amount is unknown, then I recommend creating say 250GB files and creating a new file everytime 250GB is reached. I also recommend Raxco's PerfectDisk for keeping disks defragmented.

[bkilty21]

Why do you recommend 250GB files?

[John Owen (EnTegra Ltd)]

That was a misprint, it should of been 250MB as a suggestion. The idea is to keep the file large so that the task of creating a new file is infrequent, but keep the number of bytes to less than 2^31, so that any code that uses signed ints to represent the file position is not compromised by a number greater than 2^31 wrapping round. So a more accurate limit would be 2GB to be safe.

[bkilty21]

Well I am glad that you misprinted the recommended value. I was working on a high speed radar data recorder, and as soon as files grew over a couple of GB's the overhead of file allocation was apparently too much; the process CPU usage grew consistently and it was not able to maintain the required data rate.

For each file, I started pre-allocating the file size to nearly the size available on the disk, and that made the CPU usage independent of the file size. Then after done logging, I would go back and pair down the file size. Although pre-allocation wasn't quite enough by itself to sustain the required data rate, using the SetFilePointerEx(...), and SetEndOfFile(...) functions required me to use the Win32 API file functions, which gave me much more flexibility than the Innovative::DataLogger class of functions.

I ended up using Overlapped I/O, with no buffering, and that seems to have done the trick. I tried to pass it on in a post in the Host Development Corner forum under the title of "Obtaining High Sustained Data Rates in dataLogging app."

Thanks again for your post,

Brennan Kilty

[jhenderson]

Issue closed