Exploring mostly blank drives

Wishlists for new functionality and features.
Post Reply
pTd
Posts: 18
Joined: 22 Sep 2009 18:16

Exploring mostly blank drives

Post by pTd »

Could you add a 'search for non-zero byte' option?

And/or possibly add a 'skip blank sectors' option to the edit window?

Ideally if you do eventually extend the drive imaging options, would it be viable to create a non-compressed image of non-blank sectors only, in a way which remains viewable as an image file?
pTd
Posts: 18
Joined: 22 Sep 2009 18:16

Re: Exploring mostly blank drives

Post by pTd »

To expand further, what I have in mind is a simple compression scheme where each sequence of blank sectors gets represented in the image file by just 2 sector images. The first is a representative blank sector which acts as a marker for the sequence. The second contains the total number of sectors in the sequence. All other sectors are saved unaltered. OK, so it's a form of RLE, but this could save hundreds of GBs as just 2 sector images and leaves the non-zero sectors still human-readable as an ordinary file, even if it proves unfeasible to have HxD view it as an image (but skipping blank sectors of course, which is the point of the scheme!). I only envisage this scheme being useful for drives which are nearly blank, even if there are lots of single length blank sectors which necessarily translate to double sector images. The scheme could probably adapt for sectors byte-filled with any specific value.

If anyone knows of a utility that already does this I'd be grateful for a link?

No rush for this enhancement by the way, next week will be fine. :wink:
Maël
Site Admin
Posts: 1455
Joined: 12 Mar 2005 14:15

Re: Exploring mostly blank drives

Post by Maël »

What I could think of as an immediate solution would be to enable NTFS file compression. You can set it on a per file basis (i.e. also for a disk image) and you can still use such files in any program as if it were uncompressed.
pTd wrote:Could you add a 'search for non-zero byte' option?
You could try grep or such a tool now. HxD's search will have support for regex in future, so that would be covered.
pTd wrote:And/or possibly add a 'skip blank sectors' option to the edit window?
You mean collapsible sections as in the RAM-editor?
pTd
Posts: 18
Joined: 22 Sep 2009 18:16

Re: Exploring mostly blank drives

Post by pTd »

I hadn't used the ram editor before, but yes having just tried it, that's just what I had in mind. I probably wouldn't ever need to expand the blank sections however. Perhaps all I really need is some form of sector bookmarking.

Search for non-zero byte was really a poor alternative for blank skipping. grep wouldn't be suitable for searching a raw drive, would it? I've used other tools to do it, as well as save compressed images, but I'm not happy with those tools for other reasons. HxD is such a pleasure to use btw, if I haven't already said so.

I had been considering Virtual Hard Drives for compression options but they don't seem suitable for raw drives, ie non-PC drives, which is what I'm primarily dealing with. I'd forgotten about NTFS compression though, good suggestion, I'll try that later this week when I get chance. I can't really see how it would help with skipping blank sectors however, but I'll see what happens; it might at least be a way of saving a compressed image. But ideally I would like to have a manageable image file which is not solely dependent on Windows to be readable.

Thanks for your comments, looking forward to the next update :D .
Maël
Site Admin
Posts: 1455
Joined: 12 Mar 2005 14:15

Re: Exploring mostly blank drives

Post by Maël »

pTd wrote: Search for non-zero byte was really a poor alternative for blank skipping. grep wouldn't be suitable for searching a raw drive, would it? I've used other tools to do it, as well as save compressed images, but I'm not happy with those tools for other reasons. HxD is such a pleasure to use btw, if I haven't already said so.
You could try and use dd (exists for Windows, too) together with grep. I am not sure if it would work, though.
pTd wrote:I had been considering Virtual Hard Drives for compression options but they don't seem suitable for raw drives, ie non-PC drives, which is what I'm primarily dealing with.
Could you elaborate on that? What are the requirements and the tools you need to be interoperable with?

Regarding NTFS compression I just came across comments saying that it has issues with large files, leading to corruption. Of course it wouldn't help in skipping blank sectors, that's what grep was meant for (I am aware it's nothing but a temporary solution).
pTd
Posts: 18
Joined: 22 Sep 2009 18:16

Re: Exploring mostly blank drives

Post by pTd »

Purely for convenience I'm trying to avoid using command line tools, or tools requiring a separate boot disc, and the Windows versions of dd I've used so far don't scan the whole drive. They seem to be based on CHS partitioning and don't map one to one with the LBA address, leaving typically the top 5103 sectors of a 'fractional' cylinder inaccessible. Access to these is crucial.

It's a purely personal interest, the drives are for Sagem DVRs, which are terrestrial dvb recorders for UK Freeview tv. The problem is that there is no user-method of partitioning and formatting new drives or even reformatting a non-booting drive; there is a hidden menu format but only available from a working machine, and probably amounts to no more than resetting the filing system anyway. The machine will format a blank drive during an Over Air Download update though, but these are infrequent.

The format is proprietary, now strongly believed to be based on ERTFS on which there is no publicly available information. The first goal was to be able to create a backup drive image of a manageable size. Using Dubaron's Diskimage and a drive which never had any recordings on it (hence the large blank areas), a smallish 160GB drive produces a compressed image (*.cdi) of 700MB, which further zips to about 6MB.

Which is fine, but restoring that image takes a long time. OK, it would rarely be needed, but avoiding writing 95%-ish blank space would help, so restoring just the non-blank areas would be preferable.

But the real goal is to work out how to format any size drive and not be limited to the original image size, and produce a tool to do it. Or failing that maybe reverse engineer the code to find the format routine itself. This is beyond my abilities, but another contributor to this forum, minipc, has elsewhere expressed an interest in solving this problem. Naturally I don't expect HxD to play any part in this aspect, but it shows promise as a very good general analysis tool for this and many other situations.
pTd
Posts: 18
Joined: 22 Sep 2009 18:16

Re: Exploring mostly blank drives

Post by pTd »

Maël wrote:
...

Regarding NTFS compression I just came across comments saying that it has issues with large files, leading to corruption. Of course it wouldn't help in skipping blank sectors, that's what grep was meant for (I am aware it's nothing but a temporary solution).
Having looked further into VHD I'm not keen to try it, it looks more trouble than it's worth. Partly because I'm not sure how to set it up, but mostly because there seems to be a limit of 127GB drive size.

In any case I have had some good success using NTFS folder compression. I managed to save a 320GB drive image to a compressed folder, occupying <12MB on disk, better even than Selfimage managed. The resulting image file is transparently readable in HxD as the full image file, with the added advantage that text searches are now about 8 times faster than searching the drive directly. I'm still in favour of blank skipping though (or sector bookmarking, I may later expand on that in a separate thread).

One drawback is that HxD won't save the drive image to the compressed folder, failing with a not enough space message. So I tried Selfimage, uncompressed option, which gave a similar message but with an option to proceed anyway, which was successful. I suspected there might be difficulties because the compressed size can't be predicted, so it must be left either as a user choice or allowed by default if a compressed folder can be detected.

One further request in the light of this result. Would it be possible to allow file compare between a drive image file and the drive itself? I was hoping to be able to verify the above results in this manner but realised there isn't an option to compare file with drive.
Post Reply