Tuesday, September 16, 2014

Pycasa - Group duplicate files

A problem that I have been battling is how unorganized my photos and videos are. I have multiple backups taken from the various devices, which has resulted in multiple copies of images and videos. Recently, I started thinking about how to organize those with the following goals:


  • Remove duplicates
  • Organize by year and by event
  • Remove images that are out of focus or not relevant


After manually going through a few folders, I realized that it was going to be a very onerous task. So I started thinking about writing a program to identify duplicates. I recently watched a great video by David Beazley on Python generators and it was exactly what I needed for my purposes.

So I created this program that groups the duplicates together and prints a list for review.

import os
import time
import fnmatch
from collections import defaultdict

#http://szudzik.com/ElegantPairing.pdf
#This is used to combine size_of_file_in_bytes and last_modified into one number associated uniquely to both
#using a pairing function.
def elegant_pairing_fn(x,y):
    if x == max(x,y):
        return (x*x)+x+y
    else:
        return (y*y)+x

#generates a sequence of file names for the given search pattern starting from the top directory.
def gen_find(filepat, top):
    for path, dirlist, filelist in os.walk(top):
        for name in fnmatch.filter(filelist, filepat):
            yield os.path.join(path,name)

#generates a sequence of dictionary objects with detailed file information.
def gen_stat(filelist):
    for name in filelist:
        dstat = (dict(zip(osstatcolnames,os.stat(name))))
        dstat['filename'] = name
        dstat['unique_id'] = elegant_pairing_fn(dstat['size_of_file_in_bytes'], dstat['last_modified'])
        yield dstat

osstatcolnames=('protection_bits', 'inode_number', 'device', 'number_of_hard_links'
          , 'user_id_of_owner', 'group_id_of_owner', 'size_of_file_in_bytes'
          , 'last_accessed', 'last_modified', 'created')

t0 = time.time()
jpgfiles = gen_find("*.jpg", "C:\\Users\\Shantanu\\Pictures")
jpgfilestats=gen_stat(jpgfiles)

#Now group the file names with the same unique_id value
#The algorithm assumes that the combination of file size and last modified date is unique per image file.
# Meaning it is almost impossible to have images that are different and yet have the same size and last modified date.
# This is a safe assumption as long as the files have not been modified using some photo editing software.
filecount = 0
jpgfilegroups = defaultdict(list)
for l in jpgfilestats:
    filecount += 1
    jpgfilegroups[l['unique_id']].append(l['filename'])

#Print groups with more than one file names in it.
#  We do not care to review files that appear only once in our stash.
count = 0
groupcount = 0
for k,v in jpgfilegroups.items():
    groupcount += 1
    if len(v) > 1:
        count += 1
        print(count, k, v)
print ("Total files processed: ", filecount)
print ("Total file groups processed: ", groupcount)
print ("Total time taken: ", time.time() - t0)


On my laptop, the program processed ~11K files in 2.8 seconds and came up with 2689 groups of possible duplicates. Manually reviewing those 2689 groups is a much simpler task.

The algorithm assumes that the combination of file size and last modified date is unique per image file. Meaning it is almost impossible to have images that are different and yet have the same size and last modified date. This is a safe assumption as long as the files have never been modified using some photo editing software (which is conveniently true for my case).

I learnt some new tools as a result of this exercise:

  • A practical use of Python generators for scratching my own itch.
  • The Python os library and its functions
  • The mathematical concept of 'pairing functions' to uniquely represent two numbers as one.
  • EDIT 20141020: I just realized that jpgfilegroups is actually a hash table using chaining for collision resolution.
What problems have you solved using these tools? Feel free to use this code to organize your photos and videos.

Monday, September 15, 2014

My Latest Laptop

I wrote the first half of this post in 2011 with an update in February 2014.

After 7 years of superb performance from my IBM Thinkpad T40, it finally died of a power connection break inside the Motherboard. After doing much research, I decided to stick to Lenovo instead of changing brands.

I noticed in my behavior and of some of my friends too, that one doesn't switch laptop brands that easily.

So here is what I ended up with:
Lenovo Z570
Second Generation i7 (2.0 GHz)
8 GB RAM
240 GB OCZ Vertex 3 SATA 3 6GBPS SSD
700 GB 5400 RPM WD HDD
External Optical Drive

Here is how much it cost me (including taxes & shipping):
1. $10.59 - For 3 new Philip Head Screwdrivers
2. $32.85 - SATA - ESata connector cable for External Optical Drive
3. $51.65 - Optical Drive HDD Caddy
4. $742.69 - Lenovo Z570 Laptop
5. $49.99 - Acronis True Home Image
6. $459.99 - OCZ Vertex 3 SSD 250 GB
For a total of - $1347.76

That's the price I paid for a laptop with:
1. Almost 1 TB of total disk space
2. Super Fast SSD Performance (~20 Second Boot times)
3. Security of backups given the instability of SSDs!
4. External Optical Drive that is not taking up space in the laptop.

Update 20140205: Return of the IBM Thinkpad T40.
First, an update on the Z570. It has been working great. Like Jeff Atwood of Coding Horror fame says "A solid state hard drive is easily the best and most obvious performance upgrade you can make on any computer for a given amount of money. Unless your computer is absolute crap to start with.". It is well worth the price. Also, I have been fortunate that I have not had any catastrophic SSD failures yet.

Going back to the T40. I am glad that I did not throw it away. I was able to move most of the working parts from this T40 to another one that the IT department at work was trashing. I moved the HDD, Internal Wireless Card, Internal Bluetooth Card, RAM to the new chassis.

In turn, I learnt a lot about the inside of a laptop. Most of it is like a jigsaw puzzle. The parts are built in such a way that they fit in perfect slots, mostly. The biggect challenge I had was keeping a track of all the screws. Everytime I opened the laptop, I ended up with a few screws that I could not figure out where I took them out from!

One change that I did make was to move to Lubuntu 12.04. With Windows XP end of life in April 2014, and knowing the fact that the T40 hardware is too weak for Windows 7 and above, I decided to switch. And that was a smart move.

By switching to Lubuntu, I have extended the life of my T40 by at least another 2 years.

Now all I need is a new battery pack, 1 GB of RAM and a 60 GB SSD (SATA 2 will be fine). I will then put the current HDD into the optical drive bay and make the SSD the master. I should end up with the meanest T40 out there!

Do you have any insights to share on ways to alter laptops to make them more useful from a practical point of view?