On 22 March 2012 23:34, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote: <...>

You could use something other than md5sum, such as sha256sum in the same manner.

Using strong crypto would probably be overkill for this particular use-case. -- Joel Shea <jwshea@gmail.com>

James Harper <james.harper@bendigoit.com.au> wrote:

Those are just hashes, and I already know that whatever internal crc the drive uses has missed a single bit error, so I'm settling for nothing less than a byte for byte comparison :)

Md5 and SHA256 won't miss single bit errors. As I remember, they're designed so that single bit or other small differences in the input yield very large differences in the hash value.

I don't think that there is any evidence that a hash collision occurred. An error on the wire or a memory error are both more likely.

The drive itself is failing. It isn't reporting any errors under Linux (although Windows was reporting timeouts) but certain files were taking an incredibly long time to read as the drive silently tries to obtain a good read. It would have to be an astonishing coincidence that there was a wire error or memory error at the same time as I was trying to extract data from a failing disk. I think it is much more reasonable to assume that there was a checksum collision on that sector such that the single bit error wasn't detected. Either the head or the media is failing so it will be returning incorrect data most of the time which is rejected by the controller due to incorrect checksum, but given enough read attempts it is possible that a checksum-valid combination of random bytes could sneak past the checksum verification. I don't know what checksum algorithm the drive uses internally but I bet it isn't nearly as strong as MD5. One thing I guess I haven't considered is that there is a more widespread fault on the drive that could cause data errors past the point of the checksum (eg nearer the IDE interface side)... a failing capacitor causing unclean DC could do this I suppose. The test rig (USB to IDE disk adapter plugged into a Linux machine) has been used many times before without issue, just never on a disk that is failing this badly, so I'm reluctant to suspect that as the culprit. James

In the end I did this: #!/bin/sh #!/bin/sh find sp6a | while read filename do basename=`basename "$filename"` othername=`find C/WINNT -iname $basename` if [ "$othername" != "" ] then diff "$filename" "$othername" if [ "$?" != "0" ] then ls -al "$filename" "$othername" fi fi done which showed me a few files that have a few possible 0xF0 -> 0x90 difference (comparing the hex dumps)... but that's a double bit transition vs the files from the service pack and I don't know if that's the way the files are supposed to be... James

Andrew McGlashan wrote:

Hi James,

Okay, here is one method for bulk comparing two directory trees that will give you a start as you can then eliminate some files from further analysis.

If you do an md5sum of all files of each tree sending the output to a file for each tre, then you sort those files. Once you have those files, you can start eliminating files that share the same basename and md5 checksum.

You could use something other than md5sum, such as sha256sum in the same manner.

finddup (from the perforate package) might likewise help. $ ls good-copy/ bad-copy/ $ finddup -l # links duplicate files $ find -links 1 # find files that have no duplicate AFAIK it checks length and sum. You can use finddup -n -v instead to have it merely report duplicates, but IIUC you want to find the NON- duplicates. finddup theoretically can compare dirs in separate areas, but IME it only works properly on $PWD.