Deets on recent iTunes weirdness and attempted solutions, mostly for people asking about it on Twitter.
I’ve been storing my music collection on an Infrant ReadyNAS RAID system for more than a year. Aside from slow write speeds over a lame 10 megabit connection, it’s worked really well, and it’s comforting to know that, even though I’m not backing up the collection, at least I’m reasonably well-protected from disk failure.
But over the past month or so, I’ve been noticing more and more of those little exclamation marks in iTunes indicating that a track could not be found. Ah… turned out I had accidentally run iTunes for a while with the NAS unmounted, and iTunes had re-set the base dir to my home (thank you, how nice!), so now the collection was partially split across volumes.
I could re-navigate to find missing tracks individually, but there were too many to catch them all, and because the files weren’t on a local volume, the process per-file was agonizingly slow. Tried the Advanced | Consolidate menu option to try and force iTunes to put everything back on the NAS, but no dice – still a sea of exclamation points.
Decided it was time for some XML surgery. It’s no fun editing a 50MB file in TextMate, but the edits looked good. After iTunes re-indexed… No change – all missing tracks were still missing. Rassafrassa…
Fine. Time to rsync the collection over to a USB drive, just in case something bad was brewing.
rsync -a -v /Volumes/eggplant/MP3/ /Volumes/Jarlsberg
After a couple of hours rsync completed, but with an error message: “Couldn’t synchronize all files.” No further info. Crapola. Digging deeper, decided to get a directory count from both the USB drive and the NAS to see what had been skipped in the rysnc process:
ls -1 /Volumes/Jarlsberg > ~/Desktop/usb-dir.txt
ls -1 /Volumes/eggplant/MP3 > ~/Desktop/nas-dir.txt
Then ran a diff from within TextMate and… the light went on. All of the skipped directories were case contrast issues:
Wherever two dir names were identical but the case was different, iTunes was having trouble locating them, and rsync wasn’t able to transfer both to an HFS-formatted drive. So now I was getting somewhere. But how did it happen, and what to do about it?
The NAS runs Linux, and its volumes are ext2fs-formatted. The network layer is AFP. The USB drive is HFS+ formatted. ext2fs is completely case sensitive; HFS+ is case-insensitive but case-respecting. So this works on Linux, but not on the Mac:
You’d think that iTunes would never create two dirs with the same name but different cases, even when writing to a filesystem that supports that… but the evidence proved otherwise – iTunes had created a directory structure it couldn’t read.
So I’m content using USB external storage for my music, as long as I can come up with a way to back it up (maybe I’ll just push monthly tarballs to the NAS, or use S3, or something). The remaining problem was getting all of my playlists, play counts, etc. back. Again, more XML surgery. More wrestling with a 50MB XML file, re-importing, things getting weird. Running out of patience, decided to throw in the towel and start fresh. Dragged in 30,000 tracks from the USB drive and damn the database. Not ideal by a long shot, but the process did manage to resurrect a bunch of long-orphaned tracks, ghost podcasts, etc.
So my question is, did I screw something up, or hit a genuine bug? Is this an edge case? And why did it take so long to manifest? For now, I’m not feeling particularly comfy about using non-Apple network storage for iTunes, though obviously thousands of people do it every day. Hrm.