Slow save while inserting bytes near start of giant file.

Wishlists for new functionality and features.
Post Reply
poleguy
Posts: 2
Joined: 09 Nov 2017 22:34

Slow save while inserting bytes near start of giant file.

Post by poleguy »

I recently filled a hard-drive with a giant wav file only to realize the header was wrong (RIFF) instead of (RF64). To fix it I copied a header from an RF64 file, adjusted the length fields, and pasted it in place.

I then hit save and it said it couldn't make a backup because it wouldn't fit on the disk. It was a 800GB file, so no surprise.

So I let it continue without a backup. (One enhancement request: provide more information in the dialog about the drawbacks of letting it continue. What risk am I taking on exactly if I don't back up? Does it mean that cancelling the subsequent operation causes corruption?)

It then started saving... and it said it would take 4 hours, and presented a Cancel button.

I'm now three hours in to the 4 hours, and it is reporting an additional 3 hours.

So here's the questions:

If I hit cancel:
What will be the state of the file on disk? (If it is going to be corrupted by cancelling, will the GUI warn me and give me a second chance to not cancel?)
Will it take 3 hours to back out the changes? (What if I cancel that process?)
If I do hit cancel and then close HxD will it be corrupted on disk?
If I hit cancel and then hit save again, will it be corrupted on disk? (I did this already, so I'm quite interested.)

Second question:
Is there a way to improve the write speed of a small block inserted at the beginning or middle of a large file? Can this be implemented in hxd? (That's my primary enhancement request)

I'm probably not going to cancel this, but I imagine other people get themselves in this situation occasionally, and it would be nice if HxD handled this as well as it handles modifying a byte in a large file (which I think is a lightning fast save, right?).

Nick
Maël
Site Admin
Posts: 1455
Joined: 12 Mar 2005 14:15

Re: Slow save while inserting bytes near start of giant file.

Post by Maël »

poleguy wrote: 09 Nov 2017 22:48 If I hit cancel:
What will be the state of the file on disk?
First it will move the data towards the file end to make space, which will take almost all of the time. Then it will write the new data, which should be fast (since a header is maybe a few KB at most).

Interruption of the saving progress can only be done when an entire block was moved properly, so in your scenario, the worst case should be that you are left with a file that has a gap somewhere (to make space for the next blocks to move) or duplicate data, but nothing is missing.
No data would be lost, but it would be a bit cumbersome to find that place where the data movement was interrupted, and you would have to delete that part of the file to remove this gap, then save (which will take roughly as long as the time that remained before you interrupted).

I wouldn't recommend interrupting the saving progress, as you wont save any time and you'll have to fix the file afterwards as mentioned above.

(If it is going to be corrupted by cancelling, will the GUI warn me and give me a second chance to not cancel?)
No, that's the idea of the backup file. But I understand how that could be improved.
Will it take 3 hours to back out the changes? (What if I cancel that process?)
If I do hit cancel and then close HxD will it be corrupted on disk?
See above.
If I hit cancel and then hit save again, will it be corrupted on disk? (I did this already, so I'm quite interested.)
It will be in an inconsistent state as mentioned above. Saving again (if it is possible) would not be a good idea. In this case it is better to remove the gap/duplicated data that exists as mentioned above.
Second question:
Is there a way to improve the write speed of a small block inserted at the beginning or middle of a large file? Can this be implemented in hxd? (That's my primary enhancement request)
I replied to this in detail in an email you sent me earlier. The short answer was: I am not aware that any file system supports this, just efficient appending, or efficient writing if no data needs to be moved due to insertions or deletions. Without file system support, no program can implement this feature.
I'm probably not going to cancel this, but I imagine other people get themselves in this situation occasionally, and it would be nice if HxD handled this as well as it handles modifying a byte in a large file (which I think is a lightning fast save, right?).
Regarding the speed nothing can be done (unless you know of some special file system API).

But warnings about possibly inconsistent states when canceling are reasonable. Or maybe another way to handle this.
This issue hadn't come up yet, but it's indeed a valid concern.
I'll think about it.
poleguy
Posts: 2
Joined: 09 Nov 2017 22:34

Re: Slow save while inserting bytes near start of giant file.

Post by poleguy »

Primum non nocere... you've done a great job with HxD! Keep up the good work...

I appreciate the thought and attention to detail you put in.

I can't disagree with anything you're saying, and having done some searching, haven't found a method of doing this that would improve the write speed.

Cheers!
Post Reply