47,900 MP3's - How To Organize Hard Discs?

[matthewhargrove]

I have a little over 310 GB of music in MP3/WMA and am wondering the
best way to organize my hard discs to maximize computer speed. I have
two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
processors.

The two discs are currently divided into about ten 50GB partitions and
the MP3's are distributed somewhat evenly between the partitions.

Is this most efficient way to store these files or should I have less
partitions with bigger capacity?

At times my computer runs slow when accessing the whole list and I
just want to make sure its not because of my file organization.

Thanks!
Matthew

 

[Arno Wagner]

I have a little over 310 GB of music in MP3/WMA and am wondering the
best way to organize my hard discs to maximize computer speed.

What do you mean by "speed"?

I have
two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
processors.

The two discs are currently divided into about ten 50GB partitions and
the MP3's are distributed somewhat evenly between the partitions.

Is this most efficient way to store these files or should I have less
partitions with bigger capacity?

It depends on what you want to optimize.

Search speed would need something like separate subdirectories for
each starting letter or use of a database system in the first place.

Space would mean to store them in archives, e.g. .zip.

At times my computer runs slow when accessing the whole list and I
just want to make sure its not because of my file organization.

That may be due to directory fragmentation. A simpe disk defrag
may cure it.

Arno

 

[Andre Majorel]

The two discs are currently divided into about ten 50GB partitions and
the MP3's are distributed somewhat evenly between the partitions.

Is this most efficient way to store these files or should I have less
partitions with bigger capacity?

At times my computer runs slow when accessing the whole list and I
just want to make sure its not because of my file organization.

On what OS ? For Unix filesystems, the bigger the filesystem the
farther away a directory entry will be from the inode on
average. That's no good.

Directory fragmentation will slow things down, as Arno pointed
out. On the other hand, if you create subdirectories to fight
that, fewer directories will be fragmented but there will be
more of them so the total seek time may actually increase.

There are good reasons for avoiding directories with large
number of entries, though (CPU).

 

[chrisv]

I have a little over 310 GB of music in MP3/WMA

Steal much?

 

[matthewhargrove]

Steal much?

No Chris, I don't steal at all, but thanks for the implication. I
have been collecting music for more than 30 years and have digitized
most of my albums and tapes, the more than 3,000 CD's I have purchased
since my first one in 1985, and have been a subscriber to eMusic since
its launch. Digitizing all the music of has taken me years of
meticulous cataloguing, expense, and tons and tons of time. Since I
am not a computer guy I came here for assistance on how to best
organize it. Your post was not very helpful.

 

[matthewhargrove]

Thanks to Arno and Andre for the posts.

I have defragged and have an organized directory tree set-up (each
letter of alphabet --> band name --> album --> song files), and am
running Windows XP.

I am trying to decide if I should have just a few large partitions or
a bunch or smaller ones to organize the hard disks for speed of
accessing the data. Thanks again.

 

[Franc Zabkar]

What do you mean by "speed"?




It depends on what you want to optimize.

Search speed would need something like separate subdirectories for
each starting letter or use of a database system in the first place.

Space would mean to store them in archives, e.g. .zip.

MP3s are already compressed. Winzip doesn't appear to shrink them by
very much.

- Franc Zabkar

 

[Arno Wagner]

On 10 Sep 2007 05:34:59 GMT, Arno Wagner <[email protected]> put finger
to keyboard and composed:

MP3s are already compressed. Winzip doesn't appear to shrink them by
very much.

Indeed. But if you have small mp3s and a large cluster size, you get
less filesystem loss. Does not matter that much, admittedly.

Arno

 

[Ian R]

I have a little over 310 GB of music in MP3/WMA and am wondering the
best way to organize my hard discs to maximize computer speed. I have
two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
processors.

The two discs are currently divided into about ten 50GB partitions and
the MP3's are distributed somewhat evenly between the partitions.

Is this most efficient way to store these files or should I have less
partitions with bigger capacity?

At times my computer runs slow when accessing the whole list and I
just want to make sure its not because of my file organization.

Thanks!
Matthew

Hi Matthew

I see youve already had good advice re managing the file distribution.

FWIW I did wonder if you have any backup regime for all your music data
files?

Given the amount of time and effort it has taken it would be more than a bit
upsetting if one of you drives expired (sorry to say that IMHO Maxtors dont
have the best reputation for reliabilty).

Given the amont of data I'd suggest using disk imaging software and another
drive to save to - perhaps in a removeable caddy.

Food for thought....

Hope this is helpful.

Ian

 

[matthewhargrove]

FWIW I did wonder if you have any backup regime for all your music data

files?

My backup regime is not very good... I have everything burned to DVD's
using Nero's Back It Up. My initial backup took over 50 disks, and
since then I just periodically burn a new disc with what I have
transferred recently. Would love advice on how to back it up if an
easier way exists. I have never tested the backup, so I have
nightmares that I lose the data and it doesn't work, then I am back to
transferring everything again.

However, I am still unclear if a few larger partitions are better than
a bunch of smaller ones.

 

[Franc Zabkar]

Indeed. But if you have small mp3s and a large cluster size, you get
less filesystem loss. Does not matter that much, admittedly.

Arno

IME MP3 files seem to be about 3MB in size. Assuming a max cluster
size of 32KB, the average slack space per file would be about
32K/3MB/2 = 0.5%. As you said, that isn't much of a saving.

- Franc Zabkar

 

[timeOday]

There are good reasons for avoiding directories with large
number of entries, though (CPU).

Depends on the filesystem. ReiserFS handles them very well, but I guess
its future is in doubt.

 

[timeOday]

I have a little over 310 GB of music in MP3/WMA and am wondering the
best way to organize my hard discs to maximize computer speed. I have
two new Maxtor 250 GB hard discs (7200 rpm), 4GB RAM, and 3.2GHz dual
processors.

The two discs are currently divided into about ten 50GB partitions and
the MP3's are distributed somewhat evenly between the partitions.

Is this most efficient way to store these files or should I have less
partitions with bigger capacity?

At times my computer runs slow when accessing the whole list and I
just want to make sure its not because of my file organization.

Thanks!
Matthew

Over-partitioning. Why all the partitions? They don't do anything but
add complication.

What you should do is get two discs, each capable of holding the entire
collection plus some room to grow. Use one for every day use, and
another for backup. On each, put a single partition with the entire
collection on it.

A 500GB drive is only a few bucks more than a 250. For the time and
expense in your collection, it would be a real shame not to have a
backup. (And RAID is not a backup - you need protection against
accidental deletion or corruption, too).

If searching through the music is taking too long, just dump out a text
file with the filename of every track, and search that instead. It's as
easy as

dir /s > all_music.txt

then look through it with a text editor

 

[Franc Zabkar]

If searching through the music is taking too long, just dump out a text
file with the filename of every track, and search that instead. It's as
easy as

dir /s > all_music.txt

then look through it with a text editor

Here's another way:

dir drive:\ /b /s /on | find /i "name_of_artist_or_album_or track"

- Franc Zabkar

 

[timeOday]

Here's another way:

dir drive:\ /b /s /on | find /i "name_of_artist_or_album_or track"

- Franc Zabkar

Since it will take a while to traverse the whole directory tree, I would
still recommend dumping the listing to a file only once (each time you
add a new album), then searching that file with "find," or a text
editor, etc.

 

[Andre Majorel]

I have defragged and have an organized directory tree set-up (each
letter of alphabet --> band name --> album --> song files),

So you have about 4000 directories with a dozen of entries each.
Directories with so few entries are unlikely to become
fragmented. On the other hand, that makes a lot of them.

Scanning that tree means opening 4000 directories. That's
expensive. Unless NTFS wildly differs from the file systems I
know, opening a directory has a significant fixed cost. Assuming
no fragmentation, listing 10 directories with 10 entries each
takes more time than listing 1 directory with 100 entries.

You could make fewer directories with more entries. For example,
one directory per artist. But that won't solve everything
because scanning the tree still means stating nearly 50,000
files.

Alternatively, you could have a cron job (or whatever periodic
tasks are called on Windows) slowly scan the tree every few
minutes. A simple "DIR D:\ E:\ F:\ ... /-O /-P /S >NUL" would be
a good start. That would help the directories and file metadata
remain in the cache. Not exactly elegant, but IMO more
convenient and effective than one directory per artist.

I am trying to decide if I should have just a few large partitions or
a bunch or smaller ones to organize the hard disks for speed of
accessing the data. Thanks again.

Your many-small-partitions scheme makes day-to-day management
more difficult but I don't think it makes things slower. In
fact, it probably makes them a bit faster by improving locality.

 

[Andre Majorel]

My backup regime is not very good... I have everything burned to DVD's
using Nero's Back It Up. My initial backup took over 50 disks, and
since then I just periodically burn a new disc with what I have
transferred recently. Would love advice on how to back it up if an
easier way exists. I have never tested the backup, so I have
nightmares that I lose the data and it doesn't work, then I am back to
transferring everything again.

Buy a 500 GB disk (A). Copy everything you have on it. Give it
to a friend or relative.

On the next visit, buy a second 500 GB disk (B). Copy everything
you have on it, give it to him/her and take disk (A) back with
you. Repeat, alternating between (A) and (B).

At any time, you have a backup at hand and another one that's
not as fresh but will survive a flood/fire/burglary of your
residence.

 

[Rod Speed]

So you have about 4000 directories with a dozen of entries each.
Directories with so few entries are unlikely to become fragmented.

It aint the directory that gets fragmented, its the files.

On the other hand, that makes a lot of them.

Scanning that tree means opening 4000 directories. That's
expensive. Unless NTFS wildly differs from the file systems I
know, opening a directory has a significant fixed cost. Assuming
no fragmentation, listing 10 directories with 10 entries each
takes more time than listing 1 directory with 100 entries.

Fragmentation has no effect on listing directorys.

You could make fewer directories with more entries. For example,
one directory per artist. But that won't solve everything because
scanning the tree still means stating nearly 50,000 files.

But you only need to scan directory of the artist you are looking at.

Alternatively, you could have a cron job (or whatever periodic
tasks are called on Windows) slowly scan the tree every few
minutes. A simple "DIR D:\ E:\ F:\ ... /-O /-P /S >NUL" would be
a good start. That would help the directories and file metadata
remain in the cache.

No it wouldnt with that many files.

Not exactly elegant, but IMO more convenient
and effective than one directory per artist.

Wrong again.

Your many-small-partitions scheme makes day-to-day management
more difficult but I don't think it makes things slower.

It can do.

In fact, it probably makes them a bit faster by improving locality.

Pity about farting around changing the current partition.

 

[Folkert Rienstra]

Rod Speed wrote in news:[email protected]
Provided they all fit in the initially allocated directory space.
If there is even such a thing in NTFS.

It aint the directory that gets fragmented, its the files.

Nope. He only adds to his collection.
So it's the directories that can get fragmented, not the files.

Fragmentation has no effect on listing directorys.

Of course it does, if they're scattered all over the drive.

[snip]

 

[Andre Majorel]

It aint the directory that gets fragmented, its the files.

If I'm not mistaken the OP has not been complaining about file I/O.

Fragmentation has no effect on listing directorys.

Unless your disk drive has zero seek time, it has.

But you only need to scan directory of the artist you are looking at.

I assume the OP is scanning all (or a significant portion of)
the tree. Otherwise it wouldn't be slow enough to notice.

No it wouldnt with that many files.

$ time find / | head -n 50000 | wc -l
50000

real 0m7.794s
user 0m0.169s
sys 0m0.291s
$ time find / | head -n 50000 | wc -l
50000

real 0m0.301s
user 0m0.134s
sys 0m0.166s

Works for me, on Ext3 at least.

It can do.

In the OP's situation ? How ?

Pity about farting around changing the current partition.

That's Windows for you. :->