iTunes and Network Attached Storage

Deets on recent iTunes weirdness and attempted solutions, mostly for people asking about it on Twitter.

I’ve been storing my music collection on an Infrant ReadyNAS RAID system for more than a year. Aside from slow write speeds over a lame 10 megabit connection, it’s worked really well, and it’s comforting to know that, even though I’m not backing up the collection, at least I’m reasonably well-protected from disk failure.

But over the past month or so, I’ve been noticing more and more of those little exclamation marks in iTunes indicating that a track could not be found. Ah… turned out I had accidentally run iTunes for a while with the NAS unmounted, and iTunes had re-set the base dir to my home (thank you, how nice!), so now the collection was partially split across volumes.

I could re-navigate to find missing tracks individually, but there were too many to catch them all, and because the files weren’t on a local volume, the process per-file was agonizingly slow. Tried the Advanced | Consolidate menu option to try and force iTunes to put everything back on the NAS, but no dice – still a sea of exclamation points.

Decided it was time for some XML surgery. It’s no fun editing a 50MB file in TextMate, but the edits looked good. After iTunes re-indexed… No change – all missing tracks were still missing. Rassafrassa…

Fine. Time to rsync the collection over to a USB drive, just in case something bad was brewing.

rsync -a -v /Volumes/eggplant/MP3/ /Volumes/Jarlsberg

After a couple of hours rsync completed, but with an error message: “Couldn’t synchronize all files.” No further info. Crapola. Digging deeper, decided to get a directory count from both the USB drive and the NAS to see what had been skipped in the rysnc process:

ls -1 /Volumes/Jarlsberg > ~/Desktop/usb-dir.txt
ls -1 /Volumes/eggplant/MP3 > ~/Desktop/nas-dir.txt

Then ran a diff from within TextMate and… the light went on. All of the skipped directories were case contrast issues:

Half-Man Half-Biscuit
half-man half-biscuit

Wherever two dir names were identical but the case was different, iTunes was having trouble locating them, and rsync wasn’t able to transfer both to an HFS-formatted drive. So now I was getting somewhere. But how did it happen, and what to do about it?

The NAS runs Linux, and its volumes are ext2fs-formatted. The network layer is AFP. The USB drive is HFS+ formatted. ext2fs is completely case sensitive; HFS+ is case-insensitive but case-respecting. So this works on Linux, but not on the Mac:

mkdir foo
mkdir Foo

You’d think that iTunes would never create two dirs with the same name but different cases, even when writing to a filesystem that supports that… but the evidence proved otherwise – iTunes had created a directory structure it couldn’t read.

So I’m content using USB external storage for my music, as long as I can come up with a way to back it up (maybe I’ll just push monthly tarballs to the NAS, or use S3, or something). The remaining problem was getting all of my playlists, play counts, etc. back. Again, more XML surgery. More wrestling with a 50MB XML file, re-importing, things getting weird. Running out of patience, decided to throw in the towel and start fresh. Dragged in 30,000 tracks from the USB drive and damn the database. Not ideal by a long shot, but the process did manage to resurrect a bunch of long-orphaned tracks, ghost podcasts, etc.

So my question is, did I screw something up, or hit a genuine bug? Is this an edge case? And why did it take so long to manifest? For now, I’m not feeling particularly comfy about using non-Apple network storage for iTunes, though obviously thousands of people do it every day. Hrm.

Music: Ken Nordine :: Brown

11 Replies to “iTunes and Network Attached Storage”

  1. Have you thought about creating a .dmg file on the NAS, putting your music on that, then mounting it automatically at boot?

    If you make it a sparse image, it should be able to grow with your collection. You’ll also be able to set the volume format to:

    Mac OSX Entended (Journaled)

    I don’t know if this will solve your problem, but it might be worth a shot.

  2. Hmmm…maybe I’m glad I manage my music collection & the loading & unloading of tracks onto various iPods with Amarok under Linux ;) Linux Mint to be exact – an Ubuntu derivative with a lovely & well thought out interface…

  3. Hrm, cool idea PJ. I know there was some talk of using sparseimage as a way to use Time Machine against a non-Apple NAS – can’t think of a reason why it wouldn’t work for this purpose too.

  4. IMO, the correct solution is to mount the remote share, then:

    ln -s /path/to/remote/music ~/Music/iTunes/iTunes\ Music

    This symlinks the directory iTunes auto-creates in “panic mode” to your actual remote music library location.

    HTH.

  5. It’s also possible to mount your ext fs with flags that make it not differentiate between case. I don’t recall what those flags are, but I had to do it a couple years back to fix a similar problem.

    What’s happening is that iTunes is just blindly creating dirs based on track info. In a non-case-sensitive filesystem this isn’t a problem, so if you make your filesystem not sensitive (after fixing all theirs and files, of course, the problem shouldn’t occur again.

    Not exactly ideal. Another option would be to store them on a vfat partition. In fact, rsyncing them over to a vfat partition might just clean things up for you, come to think of it…

  6. I’d be interested in how an OSX install on case-sensitive HFS+ deals with this issue. Does the kernel then automagically start handling case-sensitivity across mounts?

    The fact remote shares are mounted case-insensitive by OSX is a bit daft, considering the variegated filesystem types you might expect.

  7. @mneptok: Just did a little test with an Interesting result. With an ext2 volume mounted via AFP I cannot create both folders “Foo” and “foo” via the Finder, but I can create both from the shell. Both folders then appear and behave normally from the Finder. So there’s definitely some weird abstraction going on — the Finder prevents it even on a supporting filesystem, but then also respects the situation if it already exists.

    I’ll try a similar experiment later today with a networked HFS+ volume.

  8. Result of same experiment at work: Can’t create dirs foo and Foo to an HFS+ formatted volume over AFP. Also can’t create them via CLI.

Leave a Reply

Your email address will not be published. Required fields are marked *