Blog

"NOTHING works." That's what the editor of the mid-size daily paper told Point Click & Drag after a series of storms went through. It was late evening, she had a 12:30 a.m. press deadline for her morning daily, and she had a full newsroom of staff with storm stories.

The paper had a Support Agreement with PCD, and there was no question whether the Boys in Mac were going to respond - but how? They were onsite at another publication, 4 hours away, and in the midst of scheduled upgrades.

Phil began asking questions. "Nothing works? Are any computers on?"
"Well, most of them came back on after the power came back, yes."
"Good. Can they see the servers?"
"Let me check...OK, some can see one of the servers, some can see another."
"Can anyone see the printers?"
"Well, ok, yeah, a couple can see one of the printers, and one sees the imagesetter."
"How about the internet?"
"It's weird, some of us have it, and some don't."

"OK, your network switch has taken a hit, and some of the ports must be out. Sounds like the jolt came through and took out some of your computers too, at least their network ports."

Phil worked with the distraught editor on the phone, talking her through inspecting the switch to figure out which ports were working, and which devices were connected to those ports - a task made much more difficult because the network wiring had never been labeled at either end. Of 80 ports on the switch, and 50-some devices on the network, about 20 could be made to work. With some workflow improvisation, it was enough for the staff to meet deadline and get the paper out.

PCD buttoned up the job where they were, so that staff could work seamlessly next day, explained the emergency status of the storm-struck client, and promised to return and finish the upgrade later. Was the abandoned client upset?

Harman explains, "When we have two clients with contracts and their needs for our services come into conflict, we have to respond to the most urgent "can't-publish" situation. If we have to miss or leave early from a job, we tell the customer the situation, and assure them that we'd do the same for them if they had an emergency. Everyone seems to understand."

The next morning, PCD methodically worked through the stricken network, definitively determined the damage, ordered the necessary replacement gear, and helped with documentation for the insurance claim. The client ended up with a new switch (with a backup for redundancy),15 new computers - and surge protection on every network drop.

Later, PCD rewired the network and labeled everything.

PCD President Tim Harman says he'll always remember where he was and what he was doing when he heard about three catastrophes: Kennedy's assassination, 9/11, and the day a customer lost 1.2 terabytes of data from a server PCD had recently installed.

It was a Friday morning when Phil received the cal. The customer's entire 200-page 4-color slick retail catalog was at risk, the deadline was next Wednesday, and the company stood to lose thousands of dollars a day if the data was lost. The most recent backup was over a week old, and did not incorporate hundreds of crucial changes.

Phil rushed onsite, confident he could bypass a momentary glitch and bring the volumes back. But nothing worked. The 3rd-party software used to format the drives didn't recognize that the volumes had ever existed. None of the tools in the usual bag of tricks were of any avail.

These are the times that try men's souls (and pucker their sphincters). After hours of terror and sweat, Phil called Tim to explain that the customer's data appeared to be gone. Tim steeled himself to call Bob (the young president of the company, but not his real name) with the grim news.

"What can I do?" he asked.
"Bob," answered Tim, "from all we can tell, it's gone. We've done all we know to do. We can send the drives to DriveSavers in California to see what they can recover, but that will take days and cost thousands, and no guarantees."
"How many thousands?" Tim had already called DriveSavers and knew what it would cost to get emergency recovery service on four hard drives. "Minimum 6,000.00, as much as 25,000.00."
"When can they start? I'll fly out with the drives tonight." Tim had to explain the raw physics: "Bob, the drives are SO big that if they started just scanning the platters right NOW, it would be Tuesday before they even knew if anything could be recovered."
"Well, this is just unacceptable. It's unacceptable."
Sometimes people walk into a brick-hard wall of reality but refuse to acknowledge it. "I know it's unacceptable. If I could wave a wand and fix the the problem, I'd do it. Let me talk to Phil again, see if he's had any luck, and get back to you."
"Tonight?"
"Yes, of course, immediately."

Harman called Boger - who had, in a last-ditch effort to avoid the wall, searched the internet and found an obscure utility program that promised to recover any data. He'd already downloaded it and found that it recognized the missing drives - and paid the license fee to get the data recovery started.

Till 11 o'clock Friday night, and beginning at 8:00 Saturday morning, Phil traveled back and forth between his house and the company's office, checking on progress as the software recovered every bit of the data from all four drives. By late Saturday night, the process was done.

Monday morning, the design staff at the company sat down to their workstations and found the server volumes on their desktops. Every single file was just where they'd left it Thursday night before the crash. They'd lost just one day of work, and had plenty of time to get the catalog to press. Disaster was averted.

Bob - and Phil and Tim - felt their chests (and their sphincters) relax. The 3rd-party drive array was replaced by a more robust storage solution, backups were reconfigured, and the server has been stable ever since.

Total cost? Two days of Boger's labor, which were discounted because PCD had installed the server just a few weeks before.

Moral of the story? When everything goes right, it's easy. When it goes wrong, you want someone in your corner who doesn't know when they're beat.

Dear Technicia,

Yes, it's true, I have to admit it. I've always wanted to go to work in my underwear. It's just one of those things, you know? You wear a shirt and tie to work for so many years, and the pressure just builds and builds.

Well, I have to confess to someone - I finally did it. It was about six weeks ago, and early morning when I got the call from a newspaper customer several states away. Their server, installed nearly a year ago, was "losing files" and "not saving changes to files."

You know how that is. If a computer tech has heard that one once, we've heard it a thousand times! But all the users' stories were consistent, and the production manager insisted it was happening. So what could I do? I had to take it seriously.

But I didn't want to drive 8 hours to start working, and besides that, they needed to get up and running quicker than that. Then I remembered! We had cleverly installed software to let us access their server over the internet. Could I diagnose, repair, or reinstall a server over the wire?

It was time to find out - and as it turned out, we could. The production manager on the other end acted as my hands when I needed a CD inserted or a drive swapped. To make a long story short, I took over the screen and ran diagnostics. Sure enough, the server was sick. By this time, it was about 9:30 a.m. - and I realized I was sitting on my couch in my skivvies!

No one was home at the cabin in the woods, so I decided to go for the boxer rebuild. The first thing I did was figure out how to configure a temporary server with the day's data so the paper's staff could meet the daily deadline. Then, to make a long story short, I documented the server setup, backed up all the data, reformatted and mirrored the drives, reinstalled and updated the server software, reconfigured the user accounts, restored the data, and configured the sharing. Hot stuff!

There was some time out, but I was online with the server - and on the phone with the production manager - off and on till about 10 PM!

The server was up and running for the staff next morning, and the work was completely done shortly after I would have arrived if I'd had to drive there. Remote access saved the day - and saved the customer a day of production and over 1,000.00.

And I got to work a whole day in my boxers! I feel so guilty, but it's good to get it off my chest.

Sincerely, Pantless in Indiana

Dear Pantless,

You rascal, you! You can work on my server anytime.

Technicia

Roy (names changed to protect the innocent) was feeling overwhelmed. It had been years since his newspaper in a mid-sized midwestern town had upgraded hardware, system and applications.

He was long overdue to implement OS X, and his publisher was pushing to implement more online and video services. His circulation, classified, and editorial systems were all long in the tooth and dead-ended at OS9. More and more websites were out of reach of his obsolete browsers, and users were talking sweet about InDesign and trash about QuarkXPress.

Plus, he needed 25 new workstations and four additional servers. His new software vendor was coming in mid-January to put in editorial, classified, and circulation software. It was early December and he hadn't ordered everything yet - not because he hadn't planned, but because his publisher and changes in technology presented a moving target.

But whatever happened, his 6-day daily could not miss a deadline.

His hardware vendor, Jet Micro, suggested he call Point Click & Drag. By sheer coincidence, PCD personnel were passing through his area from a job further west. They stopped for the meet, took stock of the situation, and put together a plan.

Here's what happened between December 15 and mid-February:

• PCD built all workstations and servers offsite at its office, with all system and application software in place and configured, including fonts and server users
• Machines were shipped to the customer location for installation
• PCD personnel spent 5 days onsite, transferring existing user data to the new computers, installing and configuring all workstations and servers, providing basic OS X training to ease the transition, and assisting in any way needed
• Software vendor personnel were onsite from mid-January through late February, installing and training in their software

It was not easy. It was painful. But the cooperative working relationship of the three companies made it possible, and the job was done.

The paper hit the streets on time, every edition. And Roy slept a little better at night.

In this interview with PCD personnel, this publication asks about the logistics of being everywhere at once.

How DO you guys be everywhere at once?
Boger (chuckling): Well, we're not really everywhere at once, it just seems like it sometimes.

But seriously, I see from your client list that you work all over the country...how do you get there, how do you respond to people in time?
Marhsall: Planes, trains, and automobiles. Mostly automobiles. We've found that if a customer is within an 8-hour drive, it doesn't make sense to fly. Costs too much, and doesn't save much time over driving. So yeah, we drive a lot. I've driven to Texas, Florida, Missouri, Kansas...

But waitaminue, some of those are further than 8 hours from your office in southern Indiana.
Marshall: Well, yeah, sometimes if we have to take a lot of gear, we'll drive anyway. And on the Texas deal, I had to make emergency detours back through Missouri. It's just whatever it takes.

Harman: And I don't like to fly, I'd rather drive.

So you won't fly?
Harman: Oh, I'll fly if it makes sense. We flew to Utah, and to Florida. No big deal. Like Bruce says, whatever it takes.

What's the worst travel experience you've ever had?
Boger (emphatically): The train!

The train?
Harman (chuckling): Oh, that's Phil's problem. We did a gig in southern California last November, and he drove out to take his daughter to Vegas, then rented a car. But I took the train. I like the train. On the way back, we both took the train. Phil...well, Phil didn't like the train.

Boger: I'd crawl naked over broken glass before I'd take the train again.

What's the problem with the train, Phil?
Boger: Everything! It's slow, it's uncomfortable, you're all cooped up, it rocks like a kiddie ride.

But surely it's not worse than a car?
Boger: It is for ME. At least in a car, I can do the driving. I can stop and get out and stretch.

Harman: Well, you can get up and walk around the train.

Boger: I hate the train.

Harman: I like the train.

Boger: You would, you had breakfast with Kirstie Alley on the train.

Wait. You had breakfast with Kirstie Alley?
Harman: I did. But I didn't know it at the time.

You had breakfast with Kirstie Alley and didn't KNOW it?
Boger: It's a long story, don't get him started. We don't have time.

Bruce, how did you get to California?
Marshall: Me? I don't get to go on those exotic trips! Someone has to stay back home and mind all the other clients.

Boger: Yup, that's right. We try to share the travel around, but we have obligations to many clients, and someone is always available to deal with their issues when the rest of us are away.

Speaking of emergencies, surely you must run into weather on these adventures...
Harman: Oh yeah, we do.

What's the worst?
Boger: I guess there are several worsts. We tried to take a scenic route across Illinois to Missouri once, to avoid St. Louis. Only there had been a lot of rain, and the day we traveled ended up being very ... stormy.

Harman: Huh! I guess stormy's the word. We drove between cornfields that looked more like lakes. About dark, we had to take a ferry across the Illinois River. Did you know there are places they still don't have bridges?

Boger: It's true, yeah, and Harman was having some ... diIgestive disorder waiting for the ferry. We finally got across - just our van on a little ferry against the current - and found a little gas station. I never saw a guy so happy to see a Citgo in my life.

Harman: And that's just the start of it. We still needed to get across the Mississippi; guy at the station gave us a shortcut across the last hills. THAT was a wild ride, in the dark, trees down, storming and thundering, road covered with slugs, and oh yeah - watch out for deer.

Boger: And when we got to the other side, the road to the Mississippi bridge was washed out. Detour upriver to another bridge, another hour out of our way, in the storm...

Harman: and when we got to the motel, everyone was hiding under tables in the lobby. Desk clerk told us there was a tornado warning, one had been sighted 10 miles upriver, like where we'd just come from, get the hell under a table!

Boger: But we avoided disaster.

You've always avoided disaster?
Harman: For the most part. Got caught in an April snowstorm in northern Ohio, but we made it. And we had a customer in Kentucky whose wall caved in after spring storms when his area flooded.

Boger: Yeah, I remember that! And his network rack and servers were along the wall...

Harman: Yup. But we'd made sure he was backing up every night, and he had a backup drive of the server and some of the workstations offsite...

Boger: ... so they collected the Macs that still worked, grabbed an extra switch and some cables from Radio Shack, and set up in the local armory. Set up file sharing to the backup drive, and got their paper out...but that's another whole story, I guess.

Well, what does all this travel cost your clients?
Harman: Good question. We always have to work that out, and it varies. But we try to be fair, and clients seem to understand there are costs involved with travel. I guess they think we treat them fairly, so it works out.

What was the worst?
Harman: Uh - that would be the afternoon I was working at a small paper and a massive storm hit the area. It was just after closing time, and I had to rebuild the server before I left. Then came the high winds, driving rain, reports of tornadoes. The power went out. The sirens went off, and I listened to the scanner - reports of trees down, power out, flooding everywhere. The sheriff's department reported that it was running on reserve power. Then the scanners went out and nothing worked. The wind howled and the streets filled with water.

What did you do?
I did what any self-respecting newspaper tech would do. I hid under the press till the howling stopped. When I came out, the phone started ringing - local folks wanting me to dispatch writers and photographers to the scenes of their disasters. The staff of one local factory was trapped on the top floor of their building, and the downstairs was flooding.

Did you send anyone out?
Who would I send? The staff was all home hiding or already out riding around, and no one had working cell service.

Then what?
Things calmed down, power came back. Reports were that the roads all around town were flooded or blocked by fallen trees. I slept on the floor in the paper closet. Next day, I drove home, the long way around.