Apr 27, 2013

Operator Headspace: I suffer from it, too.

I love doing what I do for a living because it affords me the opportunities to do all sorts of different things related to technology. My latest misadventure involves statewide information that my organization hosts, Sharepoint, a Fortigate firewall appliance, and a double round of my operator headspace.

Last week, I had an error on our Sharepoint 2007 server - out of disk space. I had forgotten to delete old backups that accumulate each night. After I had cleared off the old backups, I thought, "Hey, we no longer host certain statewide programs on this server. If I delete those, then the backups won't be as large and I won't have to delete old backups as often."

So... I ran a couple of preliminary tests: visited the sites by url, watched what happened. Well, sorta. More on that in a moment.  I hopped over to the Central Admin (we run MOSS2007), fired up the application list, and promptly deleted several statewide sites. I also deleted the content databases (the real way to free up space in the first place in MOSS2007) and the IIS sites. Thiry minutes later, I receive an email, asking if one of the sites had been taken down and could they have the content, as they still needed to use it. Oops.

Since remote access from my house is basically non-existent due to CentruyStinks DSL, I had to wait until the next day. Er, the next day I was out of town at a meeting. So, the next day I was back in the office, I fired up MOSS Central Services and recreated the sites on the same ports. Nothing worked.

I headed into my Fortigate and checked various settings. Everything seemed okay, but I opted to delete the related entries and create them over again. Nothing. I changed http requests from the default "Port 80" setting to the "Auto" setting, thinking that maybe the problem was my sites were not on 80, but were on ports chosen by Sharepoint (Never mind that these same sites had worked before I deleted them).

After spending a bunch of time on finding a solution, I opted to set the site up on Port 80 of a different IP address. I called our folks at the state level and had them create/move the errant fqdn. After a couple hours, things still weren't working. I decided to sleep on it.

When I woke up, I tested the IP from home and it resolved, but would not bring up the site in question. The state dept called me and we talked for a bit about the problem and decided the issue was definitely internal. Great.

I checked the Fortigate, MOSS2007, and IIS. Everyone seemed to be set up to talk to each other. It just wasn't happening. So, I called Fortinet for help. The tech logged in after several false starts - our system was just acting weird. In addition to those issues, staff members told me that they were having trouble with certain sites. They called the tech support for those sites and they, too, determined the problem was internal to us. Lovely.

When the Fortinet tech was able to finally get into my desktop, he pulled up the policies and said, "The site in question has a policy, but that policy is disabled." I said, "Of course it is! I was testing other statewide sites we host, and I disabled it during testing." We enabled the policy and the errant site we host suddenly jumped up in my browser. I told the tech he was free to mark "Operator Headspace" as the reason for the failure. He and I laughed about it, and he assured me that he would "find a more polite way to say it." We hung up.

In the meantime, my users were having trouble with certain sites, still. I rebooted the firewall. Nothing. I decided to go step-by-step through every setting on the firewall. I finally came to the http settings. Remember, I had changed it from Port 80 ro Auto. I changed it back to Port 80, and TAH-DAH, staff could suddenly access everything they couldn't before. Once again, blame it on Operator Headspace.

One of my "love-to-hate" errors comes by way of the infamous, "Contact your system administrator." My response is usually something akin to, "Yeah. I talked to the guy, and he has no idea. In fact, he's mostly lost in his own operator headspace."

Some people call it job security. I suppose that depends on which side of the server one happens to be at the time.

Apr 6, 2013

Emily and Lathan in the hot tub.

The view from nine floors up!

My brother's grandson Lathan at #jamfesthotsprings

Childhood Memory: Tic-Tac-ing

I grew up in Western Pennsylvania, living in various suburbs around Pittsburgh. From 5th Grade through 9th Grade in school, my family lived in an apartment complex where my mother ultimately became the manager. I include that last bit because some of the craziest things my brother and I did as kids happened in this complex - while my mother was manager. It's a wonder we are still alive today.

Behind the apartment complex, and down a hill or two, there lived a farmer who grew feed corn. Yes, even in the 'burbs of da 'Burgh, there are (or at least were) farmers. The feed corn was multi-colored, very tightly packed in each husk, and the kernels were as hard as, well, Tic Tacs. A group of us would sneak down to the fields and, er, procure several ears of corn. By "several," I mean dozens. We stopped gathering the corn when we thought we had enough of them or when the farmer would come out and start shooting his salt gun at us. If you aren't familiar with a salt gun, it is a shotgun loaded with salt shot instead of buckshot.  Salt burns after it wounds you. I learned this firsthand.

We would shuck the corn, strip the kernels from the ears, and put the kernels in little brown lunch sacks. Then, after dark, each of us would run down a neighborhood street (at least we were smart enough not to do it in the complex). As we passed by each townhouse, we would throw handfuls of corn at doors, windows, siding - anything and everything.  After we threw it, we would hide and watch the reactions.  People would open their doors and look around or start yelling at us.  We had great fun tormenting folks.

One night, we began Tic-Tac-ing (as we called it) a row of townhouses. Very quickly, a huge man yanks open his front door and starts yelling. And then, he comes out into his front yard after us. We scrambled like wild horses, diving behind bushes, hiding behind trees, rolling under parked cars.

My brother chose the latter. He also chose to roll under a car parked acrossed* the street from a street light. He was in plain sight of the very angry man, who trudged straight for him. The whole time, the guy yelled, "I see you! I see you under that car! I am calling your mother!"  He repeated this several times as he got closer and closer. Our solution was to throw corn at his house and other houses to distract him long enough for my brother to scooch** out from under the car on the far side, away from the guy.

By now, lots of people had come out of their houses, and we were running headlong over a hill into the weeds and woods to a clubhouse we had built out of spare plywood and random supplies.

I would like to tell you that we never Tic-Tac-ed again after that. But, I would be lying.

*I wrote "acrossed" because as a kid, we always pronounced it "acrossed" but research has shown me that is not a formal word.  I don't care. I use it much like I say "jumbo" for bologna or "nebby" for a nosy person. It's part of my growing up in Pittsburgh.

** "Scooch" is a term that means "move over a little bit." It's probably Yiddish or something.