I love doing what I do for a living because it affords me the opportunities to do all sorts of different things related to technology. My latest misadventure involves statewide information that my organization hosts, Sharepoint, a Fortigate firewall appliance, and a double round of my operator headspace.
Last week, I had an error on our Sharepoint 2007 server - out of disk space. I had forgotten to delete old backups that accumulate each night. After I had cleared off the old backups, I thought, "Hey, we no longer host certain statewide programs on this server. If I delete those, then the backups won't be as large and I won't have to delete old backups as often."
So... I ran a couple of preliminary tests: visited the sites by url, watched what happened. Well, sorta. More on that in a moment. I hopped over to the Central Admin (we run MOSS2007), fired up the application list, and promptly deleted several statewide sites. I also deleted the content databases (the real way to free up space in the first place in MOSS2007) and the IIS sites. Thiry minutes later, I receive an email, asking if one of the sites had been taken down and could they have the content, as they still needed to use it. Oops.
Since remote access from my house is basically non-existent due to CentruyStinks DSL, I had to wait until the next day. Er, the next day I was out of town at a meeting. So, the next day I was back in the office, I fired up MOSS Central Services and recreated the sites on the same ports. Nothing worked.
I headed into my Fortigate and checked various settings. Everything seemed okay, but I opted to delete the related entries and create them over again. Nothing. I changed http requests from the default "Port 80" setting to the "Auto" setting, thinking that maybe the problem was my sites were not on 80, but were on ports chosen by Sharepoint (Never mind that these same sites had worked before I deleted them).
After spending a bunch of time on finding a solution, I opted to set the site up on Port 80 of a different IP address. I called our folks at the state level and had them create/move the errant fqdn. After a couple hours, things still weren't working. I decided to sleep on it.
When I woke up, I tested the IP from home and it resolved, but would not bring up the site in question. The state dept called me and we talked for a bit about the problem and decided the issue was definitely internal. Great.
I checked the Fortigate, MOSS2007, and IIS. Everyone seemed to be set up to talk to each other. It just wasn't happening. So, I called Fortinet for help. The tech logged in after several false starts - our system was just acting weird. In addition to those issues, staff members told me that they were having trouble with certain sites. They called the tech support for those sites and they, too, determined the problem was internal to us. Lovely.
When the Fortinet tech was able to finally get into my desktop, he pulled up the policies and said, "The site in question has a policy, but that policy is disabled." I said, "Of course it is! I was testing other statewide sites we host, and I disabled it during testing." We enabled the policy and the errant site we host suddenly jumped up in my browser. I told the tech he was free to mark "Operator Headspace" as the reason for the failure. He and I laughed about it, and he assured me that he would "find a more polite way to say it." We hung up.
In the meantime, my users were having trouble with certain sites, still. I rebooted the firewall. Nothing. I decided to go step-by-step through every setting on the firewall. I finally came to the http settings. Remember, I had changed it from Port 80 ro Auto. I changed it back to Port 80, and TAH-DAH, staff could suddenly access everything they couldn't before. Once again, blame it on Operator Headspace.
One of my "love-to-hate" errors comes by way of the infamous, "Contact your system administrator." My response is usually something akin to, "Yeah. I talked to the guy, and he has no idea. In fact, he's mostly lost in his own operator headspace."
Some people call it job security. I suppose that depends on which side of the server one happens to be at the time.
No comments:
Post a Comment