Sunday, March 09, 2008

Later...or Not

Last night, I set my watch forward before going to sleep. I always feel virtuous when I do that, the night of Time Change. Since there aren't all that many things that can make me feel virtuous, this is a cheap thrill. It's particularly pleasant when I think that I didn't work last night, because thats the night -- one of two, each year -- when people go kerwhacko in the computer industry. The primary problem: trying to remember, when the clocks change, what it is that you're supposed to do, and when you're supposed to do it. The secondary problem: not all computer systems can handle time change gracefully. Some can't handle it at all. To this day -- to this day! -- major computer companies have to shut down their massive database systems and leave them down in the Spring for one hour because the system can't handle records created at 1:59:59...click (time change!) and then one microsecond later at 1:00:00 . They just can't do it. They either try to sort those records back in with the records reflecting what happened at the first 1:00:00 that day, or they just barf and say RECORD OUT OF SEQUENCE; DELETING RECORD or something equally helpful. Which means someone has to go in there and fix the record flow... and as you know, computer systems are great for letting you turn a small problem into a great big one. Its usually simpler to leave the damn system down for an hour, so that the next record the system sees looks like it happened one second after the last one.... even though it really happened one hour later.

But the primary problem is the killer. Every year, when I was at IBM, in our little data center (it was just for our small account, but I understand that this same discussion used to happen at the big supergihugic ones, too), you'd hear people say, over and over and over and over "Okay, now at two o'clock WALL CLOCK TIME NOT SYSTEM TIME" ... and every so often, we'd still get it wrong. We'd swear mighty oaths that we would NOT adjust the system time... or maybe it was that we would adjust the system time... and we'd get it wrong, every so often. And once in a blue moon, that'd cause a problem, and less than that, a problem that actually was visible to the customer. This did not make us look like we knew what we were doing, and so we would try very very hard to shield them from it.... and we'd usually get it right. When I worked at EDS, we'd actually have a chart that was hand-drawn (later, someone wrote a Rexx exec to do this) to show, for the life of the maintenance window on time change weekend, what the real time was (ie, the National Bureau of Standards time for our time zone), what the system time was (which could be changed at will), and what the wall clock time was (usually the same as the real time, but not always). To add a little spice to the mixture, sometimes we'd have a system in, say, the Central Time Zone that was doing work for a customer in the Eastern Time Zone -- so we'd have to adjust the system time for that customer at one time, without affecting any other customer, and then an hour later, do it for all the other customers on that same system, without affecting that first customer. Remember what I said about the ability of computers to change little problems into big ones?

First time I saw Windows just do it, I thought: Why can't we do that? I'm still wondering about that one. I'm sure it'll get fixed. Someday.

Does anybody really know what time it is?

2 comments:

Unknown said...

Hmm... I'm going to have to write about that. I had two systems, Windows NT and Solaris, on a bunch of servers - and not a single one agreed with any other about the time. I did inititate a project to get them to all agree, but as no one seemed to think it important - the project failed. And the guy who was supposed to write the Unix piece produced this incredibly complex bit of code that almost worked. Ah well. :-)

OS X "just does it", too!
Carolyn Ann

Cerulean Bill said...

When I was a missile launch officer in training, the first time we did a 'launch', the MCCC just glanced at his watch and said 'Looks like time to me', whereupon we did the keyturn. I was disappointed to learn that in reality we had this Procedure to synch up all of the capsules' clocks every damn night...even if it did give me the chance to get a Reputation as being a little strange because I'd occasionally precede the synch announcement with an ad for 'Strategic Air Command, First Name in Nuclear Deterrence...If your needs include Armageddon, think SAC! Countdown from ten.....'