Unpaid PingPongBall

(This is all just an idea about the 1999/12/31 8001050F crash that "fat" PS3 owners seemed to universally experience today... nothing I say here is fact or certain.)

I'm going to guess that the PS3 thinks that you bypassed the hypervisor to set back the hardware RTC and as a result, the hypervisor flips a kill switch that permanently disables certain features of the PS3 (like trophies and PSN). Who knows... but, keep reading to hear out my explanation of this wild theory:

I'm going to randomly guess that the PS3 has two time sources: it has a monotonic hardware RTC that can never be directly updated, and it has a fake system-level RTC which is correlated to the hardware RTC.

The system-level RTC keeps time by catching interrupts while the system is on.  When the system gets turned off, it saves the current system RTC time and the hardware RTC time into non-volatile memory (e.g. CMOS or Flash).  When the system gets turned back on, it looks at the saved hardware RTC time and the current hardware RTC time, computes the difference, then adds that to the saved system RTC time ... and voila, now it has a new updated system-level RTC time.  Optionally, it can also go grab the time from a network time server too.  But none of this will ever cause the real hardware RTC time to be updated by external forces (the hardware RTC just chugs along and is only updated by its own internal crystal).  Only the system-level RTC time can be updated.

Thus, under-the-hood, there is actually another time (the hardware RTC time) that users normally never get to see.  This time is assumed to be monotonic (i.e. always increasing).  If the hardware RTC time ever goes backwards, then that probably triggers some fail safes in the PS3 software or hypervisor that are intended to protect the system from cheating or hacking.  Or perhaps maybe trophy data is timestamped with this RTC time. So, the reason everything failed is because the hardware RTC time is assumed to be monotonic, but in reality, there is a bug that caused it to jump back several years ... who knows how many years, but the PS3 clamped the number to 1999-Dec-31. And for whatever reason, this totally breaks the underlying assumption that the hardware RTC is monotonic and thus breaks system software.  The reason is possibly that the PS3 thinks you bypassed the hypervisor (i.e. it thinks you hacked into it and bypassed its security layer). But in reality, this was just an obscure bug of the hardware RTC (which is a lesson in why you should keep time in seconds-since-epoch, not counting leap seconds, instead of trying to manage a calendar which has very complex rules). Nevertheless, the system thinks you hacked it.  And the penalty for hacking is that you lose access to the PSN and you lose your trophies and you get locked out of games that require trophies....

Assuming that the affected PS3s are still able to download system updates, it is conceivable that Sony could un-flip this hypothetical kill switch.  So maybe there is hope...

Or for all I know, the whole thing will start working again tomorrow.

Other thoughts, conspiracy theories, etc. are welcome if you want to reply or link to a page in my comments....

UPDATE: http://boards.ign.com/ps3_general_board/b8267/189824800/r189827162/ says (I don't this is a primary source though.... the actual primary source is somewhere on IRC... but until I can hunt that down, this is interesting):

Alright guys. We figured out how to fix this issue. You have to pull out the Real Time Clock battery for 10 minutes. The main problem is that the Real Time Clock handled by syscon is out of sync with the system clock and that newer models are not affected because they have an up to date syscon firmware that appears to have this bug fixed. All Sony needs to do to fix this problem is to provide a syscon update. You can do this to fix it if things have screwed up for you or you can wait a couple days until your system reaches March 1st. This has been tested.

Well the gist is that the ARM SYSCON CPU that is used to power up the front panel of the ps3, that is responsible for doing things like sleep mode, eject, RTC etc. Is an old batch that sony picked up from the shelf like other manufacturers that has that calendar year bug regarding feburary 29th on certain periods. Causing the ps3 and the real time system clock to desync, messing up timing like Digital Rights management software and sometimes games that relies on clocks for whatever reason.

PS3s which have not connected with the internet for years are reverting their clocks to 12/31/1999 for some reason. This will cause many games like Heavy Rain and Demon’s Souls to not run regardless of whether or not they normally require a PSN connection to function

UPDATE: I just remembered I have an old PS3 Linux installed, but that doesn't seem to matter... I was a twit and didn't check what date the RTC was reporting at startup (because I accidentally grabbed the date from NTP because I forgot to unplug the ethernet cable).  I tried changing the date with hwclock, but it failed (I'm not sure if it ever worked):

root@ps3linux:~# date -s "Mar 1 01:00:00 MST 2010"
Mon Mar  1 01:00:00 MST 2010
root@ps3linux:~# date
Mon Mar  1 01:00:00 MST 2010
root@ps3linux:~# hwclock --systohc
Timed out waiting for time change.
root@ps3linux:~# boot-game-os

Then I tried setting the date forward and backward from the PS3 OS.  The system does accept the change, but the system still won't boot trophy-based games and still won't connect to PSN.

I just read http://moss.csc.ncsu.edu/~mueller/cluster/ps3/doc/LinuxKernelOverview.html which says:

Built-in real time clock keeps the wall clock time for PS3 system. Like PC, it is backed up by battery, so it ticks time even if external power is out. RTC value can be retrieved by a hypervisor call, but be never set. PS3 RTC monotonically increases and never rolls back.

UPDATE: See my newer post for my personal, contrived theory of why the PS3 died today.  It's an interesting theory, even if it's not true :-)

• M-Audio MIDISPORT 2x4
  • The good: Worked great in Linux under Ubuntu 7.10 (or maybe it was earlier, like 7.04). I was using it with an Alesis QS6.1, a Yamaha MU80 module, and a Korg TR-RACK module.
  • The bad: As soon as I upgraded to the next version of Ubuntu, the latency on the MIDISPORT 2x4 became terrible! I still haven't figured out why :-( (Latency of other MIDI interfaces remained very good).  If I switch back to the old version of Ubuntu, the latency of the 2x4 becomes good again.
• M-Audio MIDISPORT Uno
  • The bad: If you have a Yamaha keyboard, especially a Yamaha PSR keyboard, the MIDISPORT Uno just will not work. It drops MIDI events like crazy. It doesn't matter what OS you are using (supposedly).
  • The good: None.  Sure it is cute, compact, cheap, and has integrated cables.  But it doesn't work with the Yamaha PSR keyboards and potentially other keyboards.
• M-Audio MIDISPORT 2x2 Anniversary Edition
  
  • Tested with: Alesis QS6.1 keyboard, Yamaha P-70 keyboard
  • The good: Works great in Linux (Ubuntu 8.10).
  • The bad: Drops MIDI events in OS X (hmmm, this is similar to the Uno...?) (might be because the P-70 keyboard is sending MIDI timing events, but this is not really an excuse)
• Here is a device from a different vendor, just to test that my setup is okay: AKAI MPD24 (it has a MIDI interface builtin)
  • Works beautifully in Mac OS X and in Linux. It's just what I would expect from any vendor.
  • You don't even have to install drivers! It just automatically gets detected and starts working.