Even if you run your JPEGs through jpegtran -optimise then sometimes they can differ in the initial header bytes. This small Python script resets the version, units and density fields of a (directory of) JPEGs. Once its run then remove-duplicates.py will have a better chance of eliminating the duplicate images.

Example

To recursively process all JPEGs within a directory:

$ resetjpegdpi.py ./photos/