Now I was fairly nervous this morning before work as today marks my 90th day or put another way today was my 3 Month Probation review. I wasn't expecting to be fired or be told off or anything, but still found myself quite nervous regarding my position in the company.
In my typical morning routine, shower/dress, breakfast & morning news I was interrupted by an urgent report from the HR manager... (this I would realize would be just the beginning) ... it seemed that for some reason the MS Exchange server was down which resulted in everyone's emails being unavailable (about a 9/10 on the oh-shit meter)
And so I skipped breakfast and rushed (possibly breaking a few traffic rules to do so) into work to asses the situation and try and resolve it as promptly as possible. On my arrival I was presented with not just one problem but a magnitude of multiple (in hindsight minor) issues, but fixing the Exchange was top priority.
The exchange server/server was configured entirely by someone else as was the scheduled back-ups and the hard-drive configuration, while this is the case I am obviously hamstrung in what I can and can't change
Okay so it seems that the disk partition that the exchange public folders was residing on was at capacity and so exchange wasn't able to steam/load up. Now given that this partition is only used for the public folder database files it was peculiar that it was now full as it was quite happily at 95% for the past few days...
I have to admit right here that my experience and knowledge of MS Exchange administration is quite rudimentary, however I am fairly intelligent (or so I have been told) and research as diligently as the internet will allow
Upon inspection of the hard-drive there seemed to be an alarming number of 5,120 KB text documents (log files) in the format E000000x.log where x was a different number. I say alarming because I hadn't seen these particular log files in the past and they seemed to be the exact reason that the disk partition was now full... and so I deleted most of them (there was a few that were write protected)
Pleased that I had identified the cause of the problem and had also rectified it I re-mounted the Exchange Database files (innuendo much ?) and as expected the companies emails started working again. Yay little pat on the back for me and back onto trying to solve the million + one other issues staff were having...
Day progressed as typical with my probation review being quite anti-climatic as I had half-expected, with the exception that for some reason those log files that I had so brazenly cleared out previously (I would soon realize the error of my ways) were constantly being re-made, only with ever-increasing numbers...
I soon determined that as I was accessing the Public Folders more (I was attempting to reduce the overall size of the public folders by copying emails to my personal store and then using Outlook to Archive them to a .pst file to be stored elsewhere) the more of these log files were being made.
Now I didn't want the disk partition to fill up again and cause the exchange database to fail (in hindsight that would be much preferable to what actually occurred) so as I was moving/archiving old public folder emails I would watch the folder where these log files were being propagated and as I had done previous, remove them without a second thought.
I must admit I was curious as to why these log files were appearing and for what reason and in hindsight should have investigated that before doing anything further, however I felt that I was making good progress on archiving the ridiculously old (from 2007!) emails.
I had finished copying over one section of the public folders into my personal mailbox and had an archive process starting up, so I did what I had been doing for the last hour or so and deleted what I thought redundant log files.... and this is where it went haywire...
About 30 seconds after I had deleted the log files my email client stopped responding (it's about this time I started freaking out) so I attempted to re-create the steps that had seemingly saved the server this morning, un-mount the exchange server & re-mount, only this time when I attempted to re-mount I got a wall of nasty error code.
I had hoped that rebooting the server itself might re-create the necessary log files and then I'd be able to resolve the problem and get on with reducing the public folder stores, unfortunately this was not the case and I soon discovered that I was in much deeper shit than I first expected.
And now I am sitting here almost 9 hours later still trying my hardest to recover the database files on the server, and I expect it will likely take 9 hours more before it is even up and running, and this is IF I'm very lucky and the repair utilities actually work... if not I have a feeling that the information stored in the databases is quite possibly totally lost.
1 comment:
so what happened???
Post a Comment