As Miguel reports, people have already figured out the hashing algorithm.

Still hoping for a response from Apple on this.

It’s sad, but not in any way surprising. When it comes to the iPod database checksum story, not only are some people quick to jump to conclusions based on pure speculation; they also continue to propagate the never-been-true assertion that iPods are a means of locking people in to the iTunes, or its store.

It’s true that Apple markets iTunes, the iPod, and the store as a powerful bundle. The synchronization is designed to work well, and makes for a platform (or, in Microsoft’s parlance, an “ecosystem”) of its own. In fact, it worked well enough to motivate Microsoft to create their own variant of this, the Zune, with its Marketplace application. PlaysForSure largely (though not exclusively) failed because the integration between Microsoft’s software and third parties’ hardware (such as from SanDisk, Creative and others) was less than ideal. By controlling the hardware as well, Zune was to fix this part of the problem. Sales-wise, that didn’t quite work out anywhere near as well as they had been hoping, going from “3 million in the first quarter” to currently 1.2 million – after almost a year.

But there’s a line between integration and lock-in. Is AAC an Apple-specific format? No. Do iPods only play music purchased from the store? No. Did both iPods and iTunes predate said store by years? Yup. As well, iTunes has always had built-in support for “ripping”. There is nothing iTunes does to the files that marks them as unplayable by other software after the fact. Even the DRM’d files from the store play elsewhere (including, say, Winamp), provided you use QuickTime, which will handle the authorization for you.

There is no economic motivation whatsoever for Apple to force you to use your iPod with iTunes. There is, however, motivation for them to encourage you in iTunes itself as well as in the store to buy an iPod: namely, their iPod revenues.

I do not know the reason Apple added this additional check. And I do feel that it was a bad decision on Apple’s part, particularly without giving a good explanation. dfiler thinks it may simply be a means of ensuring an atomic sync operation, as to prevent potential crashes of the iPod firmware.

The fact is, we don’t know.

iPods have an iTunesDB binary database file that provides an index for fast file lookup as well as additional metadata. It is essential for any kind of meaningful operation, as iPods fail to find the files themselves (likely for performance reasons, not to mention needlessly draining battery life and shortening lifetime by wasting too many accesses, particularly on hard drive-based ones). iPods are also unable to generate this database on bootup or on-the-fly; it has to be done at sync time, thus making iTunes the primary application to create it. That, and iTunes’s own database having a similar format, make the likely explanation for the naming.

The format has been reverse-engineered years ago, not too long after the initial release of the iPod in November 2001, then with the primary objective of using the then-Mac-only iPods on Windows. (Half a year later, Apple shipped an “iPod for Windows” with MusicMatch Jukebox; soon after, iPods were “universal” in that they could be formatted for either platform; in late 2003, iPods finally shipped with the then-new iTunes for Windows.)

I believe Apple had a stance (implicitly?) of “use whatever you like, but we won’t support it”, which is only fair. They already have the revenues from your iPod purchase, and contrary to popular belief, they make next to nothing off the iTunes Store; indeed, it is and always has been primarily a product to drive revenues for other products (namely, iPods, iPhones, Apple TVs and arguably even Macs, in the sense that the iTunes Store experience is supposed to be reminiscent of that of the Mac OS UI in general). In other words, just like they let you toy around with the Apple TV, and to a point even with the iPhone, they do the same with regard to software that fills up the iPod. And, for that matter, software that runs on the iPod through means of alternative firmwares, be it Linux or Rockbox. Again, there is no conceivable reason why Apple wouldn’t have this stance: they have nothing to lose, and potentially a number of additional customers (who wouldn’t have accepted using iTunes, but still want an iPod over an alternative) to gain, negligible though their number may be.

But now, heise reports that the new generation of iPods introduced on the 5th expects a checksum for the database, and will refuse to read it (thus, again, providing no music, photos, videos, etc. whatsoever) if incorrect or – as is the case with current non-iTunes iPod clients – missing.

Why?

Well, heise’s report implies that it’s clear Apple was deliberately trying to lock out third parties. As far as I’m concerned, that part is still speculation, and will continue to be until we have official confirmation. Effectively, they’re locked out, but this doesn’t mean that was Apple’s intent.

However, I cannot think of any other reason myself either, and will therefore have to conclude for the moment that this marks yet another bafflingly silly move on Apple’s part. Much like, say, allowing you to edit contacts on your iPod touch, but not calendar events – even though you can on the iPhone, which ships a slightly older version of otherwise the exact same operating system.

People will eventually figure out how to copy the iPhone’s calendar application over to the iPod touch to work around that. And they’ll probably also reverse-engineer the checksum algorithm and reproduce it in their own apps, such as EphPod, ml_iPod in Winamp, gtkpod, Hipo, and so on. Moreover, both of those are unlikely to take more than a few weeks; months at most.

That’s not the point, though. Apple made a design decision here that they failed to explain, leading people to conclude something even less explicable, and coming across as draconian, arrogant and ignorant to a simple principle: that any single piece of software can never be for everyone. Nor should it try to be: as 37signals puts it, make your software opinionated. Normally, Apple understands that concept very well. Most of their apps include deliberate feature omissions for the sake of focusing on what they’re particularly good at. Then, why can’t they leave room for an iPod client that is good at different things than iTunes is, whether it’s being a lot more lightweight, or supporting abilities such as on-the-fly transcoding to lower bitrates on devices other than the shuffle, and with settings other than AAC at 128 kbit/s?

They can. They do it all the time on the Mac platform. And they ought to do it here as well.

Gruber links to a piece lovingly titled “It’s Official: Apple’s Stupidest Interface Innovation Ever”.

Now, I can’t deny I expected this UI change to be rather controversial when it was first leaked. And I haven’t seen it in person, so I admittedly cannot judge it well. But I think Kirk may be exaggerating just one little bit.

Like others, I think a division of two-thirds/one-third might work a lot better than the current 50-50. But in general, I understand why they went with the split screen (to display just the traditional menu on such a high-resolution screen is so wasteful!), and do think it’s a neat idea. Again, though: I can’t really judge how distracting the (Ken Burns-like?) movement animation feels in real life. Perhaps I’d hate it too.

I just don’t think I would. On the contrary, I feel that cover art, when done well, can be quite pleasant to look at while navigating the menus.

Today’s Apple event was solid. Not overwhelming, but also very little to complain about.

• Shuffle: I’m disappointed, but not astonished, that its price hasn’t been dropped. I don’t care for the new colors; if I wanted one, I’d prefer the old selection. Apple must have an extraordinary profit margin on these by now, as they’ve introduced virtually the same models about a year ago, when Flash memory prices were significantly higher. I was hoping for $49, but even a minimal drop from $79 would have made me happy. Then again, there’s refurbished ones for $49, which isn’t too bad.
• Nano: Strong. 4024 hours of battery life is nice, though most people I know prefer to sync it once a day anyway, in which case it only helps for video playback. (Keep in mind the inevitable capacity decrease after a year or two, however.) Update: Actually, the 40 hours are for the 160 GB iPod classic, not the nanos. However, the same point applies – just not to the nanos, which have the sweet spot of 24 hours.
• Classic: In sharp disagreement with someone whose name I can’t reveal, I’m happy about this new moniker for, well, the “classic” iPods. I used to refer to them as “regular”s, which always struck me as a subpar name. What Apple chose won’t win prizes for creativity, but at least everyone will associate it correctly.
• Touch: What’s up with no Mail, and not even the oh-so-important Stock Ticker? The Springboard looks so empty it’s stupid, and that Mobile iTunes is one of the worst designs Apple has done in years. (Yes, I get it. An arrow. For downloads. What was wrong with the cute lil’ green shopping bag?)
• Touch vs. Classic: That they kept Classic alive is an unusual move for Jobs 2.0-era simple-lineup Apple, but makes sense for a simple reason: Touch doesn’t offer anywhere near the capacity some people require (whereas Classic delivers), and Classic’s hard drive wouldn’t work well at all for the iPhone-esque Touch UI. As Gruber stressed:
  

  A cold boot takes about 20 seconds. (Sleep/wake is effectively instantaneous — far faster than any Mac.)

  [..]

  Everything is very fast, very responsive. When you drag something — whether it’s the slider button to unlock the phone, a zoomed-in photograph, or a web page — the drag keeps up with your finger. I haven’t found a single element of the iPhone UI that doesn’t feel super-snappy. The whole thing feels very realistic.

  Update: Real-time dragging is such a priority that if the iPhone can’t keep up and render what you’re dragging in real-time, it won’t even try, and you get a checkerboard pattern reminiscent of a transparent Photoshop layer until it catches up (typically, an instant later). I.e. iPhone prioritizes drag animation over the rendering of the contents; feel over appearance.

  Flash memory plays a large role in that “responsiveness”, namely through extremely low seek times. Having the hard drive spin up, seek, park, unpark, seek would completely ruin the experience. So, as long as Flash memory in capacities of, say, 64 or 128 GB isn’t available in an affordable fashion, merging the two lines won’t be an option. Going by Samsung’s rough estimates from earlier this year, even the former would cost Apple about $480 for the memory alone; hardly doable when an 80 GB hard drive equivalent can be had for $299 for the entire piece. Perhaps in a year, or two, or three.

  (Sure, one could provide 1 GB of Flash combined with a hard drive, but then you’d still have a rather bulky device compared to the Touch.)

• iTunes Wi-Fi Music Store: Awesome! I know a lot of people have been asking for this. Nice. One can only hope Apple waited this long so they could make for a great implementation, particularly with regard to the now-required two-way syncing… The Starbucks announcement is a clever idea as well, but… did anyone say Tim Hortons? Also, I’m sure the “LOL V CAST” suits at Verizon are none too happy about this: with V CAST, songs are $1.99 unless purchased on the computer; with iTWFMS (heh), songs are still the same 99 cents. ‘Tis a big difference: Verizon chooses to discourage people to fully use their phones’ capabilities.
• Ringtones: 99 cents. Family-friendly words cannot describe this. Being able to select your own portion of the song is great, though the idea doesn’t take a genius to come up with. Having to pay this much is ludricous, and ought to be boycotted. The sad part? People have been spending much more than this on ringtones for years! If I ever do get an iPhone, there’s always manually copying the file over. Fission. ’nuff said.
• iPhone price break: they removed the 4 GB model, and stressed customers preferred the 8 GB. So far, so good. But they cut the price by $200, which is unusual of Apple and downright suspicious. Were they planning this all along in anticipation of the holiday season, deliberately screwing over early adopters (it’s only been a little over two months!)? Did the models not sell as well as they had hoped? Or did they sell far better than they had anticipated, thus causing component pricing to decrease more rapidly? Finally, perhaps it’s simply that they sensed too much criticism from the press for relatively high pricing.

Overall, I’m very pleased with the new iPod line-up. The ringtone pricing struck a nerve particularly because I don’t expect it to ever go down and because you are effectively advertising a song, so you should be getting money, not having to pay almost 2 dollars worth of it, but it’s not the worst thing in the world. Perhaps most surprising about this event was the amount of information we previously had thanks to rumors, particularly with regard to the new nano’s format, dimensions and especially user interface.

Whichever the reason or combination thereof for the iPhone’s drastic price change, the iPods and iPhone combined should make for a terrific holiday quarter.