Hard drive chaos due to huge ammount of files (alot of files from M8, post processing

E

eleskin

Well-known
Local time
10:10 PM
Joined
Feb 5, 2008
Messages
1,080
I am a 24" Imac user and I am not the greatest in organizing my files (M8 files as well, raw and edited. In film, I always kept negatives in sleeves and marked boxes, and mounted the best photos in protective slide mounts. When I went digital with the M8, I generated many files both raw and edited (tiff). I was organized when I started, but being the attention deficit disorder artist type person that I am, I started to create more files of different things, and over time things became scattered all over my hard drive (I have 3 externals, 1TB each that mirror each other). The only way now that I can find anything is if I type "tiff" in the search function or "Leica". That way, I can see all at once what I have like a giant light table. Raw, I have over 40,000 M8 files (I never delete. I save EVERYTHING). So does any of this sound familiar to all of you? Some here may have better organizational skills, while others like me do not due to ADD or something similar. Would there be a better method to my digital madness? I thought about buying another 2 hard drives only dedicated to photo files. One for raw, the other for edited Tiffs.
 
Use a program like Aperture or Lightroom. Everything will be sorted by date and it makes it much easier to find things. But you might want to consider not saving everything. If you have several shots of the same thing, keep 1 or 2. I believe digital clutter today is similar to past generations saving string, foil etc.
 
I agree; use a single program to import and organize all files. If you should have any stragglers for some reason, be sure to put them in the pictures folder at least, hopefully you won't have too many. It will take some effort to put them all in the program at first, but after that everything will be much easier. Also be sure to note if it moves the files to new locations or copies them. If it copies them, then you will have a lot of wasted space on your hard drive.
 
It sounds sorta familiar, but you have gotten things nearly beyond repair. Just devote a week to not shooting any more shots until you fix this! You're going to have to sit down and go through them ruthlessly and delete, delete, delete. Otherwise, the idea of saving EVERYTHING will result in saving NOTHING. If you can't find it, it may as well not exist anyway.
 
At some point we are all forced to organize our files in some manner. Or not. I find it too much work to organize my files by anything other than date. My own concern is primarily with redundancy and preservation than in being able to find a particular photograph, which has never been a problem for me. You seem to have the same set of concerns about data integrity. Hard drives are fairly inexpensive these days. I think your plan of buying additional hard drives is economical and practical. Why not purchase a 1 TB hard drive for each year, move your files and organize that way?
 
Buy 2 1TB drives for each year. One in active (on-line) and the other is stored in another location for backup (off-line).

I would recommend that you edit down the really good stuff to keep in a different folder.

Another option is to look for storage up in the Clouds. Apple or Google would be my choices to look into but Flickr and others might be OK (read secure). You need a BIG pipe to the internet to move that much data.

Steve M's comment about not being able to find it is just as bad as not having it is very true. Being an old Records Manager I'm really big about having one paper copy of everything and well organized. As digital has this little problem with electromagnetic pulses I'm big into a strong backup copy policy.

Keep in mind that you can point iPhoto to a remote storage location (search the Apple support files) which might help. If you really want to get crazy look at investing into a NAS solution that holds 10 TB of space. Keep in mind that the larger the volume the larger the minimum file size. But with tossing around RAW and large JPEGs this should not be a problem.

B2 (;->
 
Do NOT delete anything, you don't know what you'll want in the future. You wouldn't cut individual 'bad' pictures out of a negative strip would you? The ones either side of your 'best' photograph also give context to it and help remind you of the conditions and the work you did to get your 'best' picture.

But do as Bill says, get two external hard drives and use one for main storage and one for backup, eventually adding others as you fill the originals. Its cheap, cheaper than files and negative sheets. And use Lightroom and its Bridge browser for keeping track of your images with tags and a searchable database.

Steve
 
I am the same, but I have a handy little program called ImageIngester, which automatically downloads my files organized by date, whilst adding that date to the image name.
 
It really doesn't matter what software you use, if you don't label and/or tag the image files in a logical way you are always going to have trouble finding what you are looking for. Looking at 40000 thumbnails to find what you are looking for will drive you nuts.
So you better work out a logical referencing system pretty darn fast and then accept you need to spend time organising what you already have.
The best policy is to use a logical directory structure even if it is only by date. That way if you know something was photographed in June 2009 you will immediately be able to find the folder to search rather than doing a massive search on date of all files on your system,
Then name files with a location and subject. There is software to help with this.
Then some of the software already suggested will make life easier.
If you didn't have ADD when you organised film then aren't likely to have it now. You just got trigger happy with digital and lazy about being organised. Computers do that to you.
 
archiving your images on a hard drive is a bad idea, spinning drives sitting on the shelf unused will fail in less time than you think. Unfortunatley, there is no cost effective long term storage. Currently Blu-ray or DVD is probably best, but be sure to use high quality blanks. I use LTO Tapes, but this is an expensive option.
 
This sounds like a severe case of Obsessive Compulsive Disorder (OCD). I would wager that this is mirrored in most (if not all) aspects of your daily life.

No amount of software would help in this instance as OCD is, by its very nature, accumalative.

I would advise that you begin this lengthy process by examining which pictures are important to you (ALL of them is not an option).

Place the unimportant ones (or ones you are not sure about) in a seperate folder called `Not Sure`. Work quickly. Carry out the first decision that enters your mind. Do not labour over individual images. They either hit a `note` or not.

Lightroom would greatly assist with this as it does not physically move your files (only adds them to a catalogue).

This will take you several evenings but persevere.

Next, place the pictures you have kept, in folders by Camera Used. (Lightroom).

Then by Year Taken and maybe Month as well (Lightroom).

Many other posters mention Lightroom. It was developed specifically for photographers to managed vast quantities of pictures.

Obviously, more hard drives will be needed as your collection grows but without a structured system of archiving, this will make your picture `hell`more unbearable until it becomes a serious issue (for your pictures and your emotional wellbeing).

Good Luck (and happy Cleansing).
 
Sounds like you are using iPhoto?
I had a similar problem on my last laptop (mac too) where iPhoto saved multiple images that were somehow linked even after manipulating and saving them. Made the error of deleting some which deleted the originals. In cleaning up I lost HEAPS of stuff. Be careful if you use iPhoto.
Don't use iPhoto at all now.
I have LR now which works flawlessly and I can recommend it!!
 
I've started treating each card like a roll of film. I note and number it in my book same as films, with a prefix of D. The card gets copied to the drives with this number as the folder number. I've been making up proofsheets in Bridge (Lightroom does this as well) of the images I've picked from the first edit of each card, folder number at top, with file numbers listed below each image. MPix prints them out pretty cheaply. Much easier for me to page through a binder of proofsheets looking for something, as I can manage to keep relationships in my head so can find images by association within the binder. This has worked well with my film images over the last 25+ years.

As I work on files the new saves are kept in the same folder, with the same file number followed by some descriptor (v1, edit4client, redadjust, final etc.). This keeps all the versions together so I can find one of them more easily.
 
just a word of caution about relying solely on a catalogue created in database by a software product.
If that database is lost or corrupted you lose all references to where and what stuff is. That is why its important to physically organise and name stuff logically before you make it known to one of these software tools. Then if you lose the database catalogue you can still find stuff and re-cataloging is much easier.
 
I use Adobe Bridge to import, and let it organize files based on date + some keyword such as location. Year separates bigger chunks of files. few times a year backup new files to external backup drive. System and SW does not matter, there are many good options. what matters is to decide one that works for you, and then stick with it. would also re-consider of storing everything, and already when importing new photos from camera, filter "saves" from the rest. as someone already commented, amount of 40k files is already so big that it probably takes some time to organize it. but its like a snowball growing, if you dont do anything.
 
The simplest program I recommend, is Google's Picasa, it searches your drives and finds every image, what you do after that is up to you, but it is a quick way to just find the locations and the extant images.

It found all my images from my installed instructor manual DVD's as well.

The price is right as well.

Regards, John
 
At the moment I rely on a good file and folder structure. All image downloads (from my cameras) go into a separate folder within an image download folder on a drive I reserve for the purpose and are named / sorted by by date and name. When I process them I select those I think work especially well and put these into a favorites folder also on the same drive/ This is not yet categorised but I will need to do so one day. For me this works well enough. I have never really liked using image file databases as most I have tried are either clunky or memory hogs. By using plain old Windows Explorer - with the file view set option to thumbnail its very easy and quick to scan any individual folder to find a specific file by sight. Of course I need to recall more or less when it was shot but I guess you do with file databases too- unless you want to spend hours after each and every photoshoot categorising files further and / or adding search metadata. I do not have the patience for this.

I do sometimes supplement this with the freeware FastStone Image Viewer simply because it has some nice features - its slideshow viewer is better than the Microsoft one for example but its still reliant on a basic Windows style folder structure. I suppose one day I will need to do something more clever by using metadata but you would be surprised how well it works.
 
Last edited:
percepts said:
just a word of caution about relying solely on a catalogue created in database by a software product.
If that database is lost or corrupted you lose all references to where and what stuff is.
Click to expand...

also, besides corrupted DB, its usually also proprietary solution, made by vendor that can go bankrupt or get bought off the market by Microsoft. or, simply gets outdated quickly due to fast pace of upgraded versions.
 
I use the following file structure:

Pictures -> Digital SLR -> 30D
| -> 50D
|
|-> Film SLR -> Elan 7
| -> Pentax K1000
| -> Pentax K2
|
|-> Digital P&S-> Canon SD400
|
|-> Rangefinder -> Leica M2
|-> Bessa R2A
|-> Bessa R3A
|-> Yashica GTN

Within each sub folder are a folders in the form "YYYY_MM_DD - <description>" and within each of those folders I create an "edits" folder for any shots I work on in photoshop and a folder called "JPEG" if necessary if I do a batch process of all the raw files without any edits. Adobe Bridge will do most of the work from me, when I import the card its set up to dump all my files into folders in the format YYYYMMDD, if a folder is still in this format it means I haven't looked at the shots yet... once I've looked at them I change the folder format to the standard YYYY_MM_DD - <description>, that lets me know later on I've at least taken a browse through the shots already.

My film scans get dumped into folders just like my digital shots... it helps to keep on top of things and organize as you go. I've found this to be the easiest way for me to be able to locate specific shots later on down the road... I'm clueless about dates, but I can almost always remember what camera I shot the picture I'm looking for with :)

*EDIT* ok.. so it won't show all the spaces I used to show the file structure, but I hope you get the point :)
 
You must log in or register to reply here.
Back
Top Bottom