[isomerica-user] Unexpected downtime

[Dan Noe]

On Mon, Sep 04, 2006 at 06:51:58PM -0400, Dan Noe wrote:
> On Mon, Sep 04, 2006 at 02:05:55AM -0400, Dan Noe wrote:
> > Starting around midnight we encountered a bizarre kernel bug which
> > resulted in colobus becoming wedged with a large number of processes
> > (~95) stuck in the "uninteruptible sleep" state.  This required a reboot
> > to clear although the system was minimally usable.  As a precautionary
> > measure, I am going to upgrade to the latest version of the Linux kernel
> > tomorrow or Tuesday when I have the opportunity to fix things if they
> > break.  I am pretty confident there hasn't been any lasting damage to
> > the filesystem.
> 
> As you may have noticed, the earlier downtime was not the end of the
> story and there was some further damage which manifested itself
> similarly this morning.  I went to Boston and repaired the filesystem,
> then backed it up and recreated the system array from the backup.  This
> should permanently fix any problems.

The issue reared its ugly head once more.  I have downgraded the kernel
to the version I know was stable because it was fine for quite some
time.  Furthermore, colobus will be moved to a different cabinet where
it will have a remote reboot functionality in case of further issues.
This is as a precautionary measure and is now being offered at no
additional charge.  The only drawback is it will take between 30 minutes
and an hour to complete the move.

The move to the other cabinet will be at approximately 17:00 or shortly
afterwards tomorrow, Wednesday September 6th.

Once again, my sincerest apologies for the downtime,
Dan

-- 
                    /--------------- - -  -  -   -   -
                   |  Dan Noe, freelance hacker
                   |  http://isomerica.net/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://isomerica.net/pipermail/isomerica-user/attachments/20060905/3f44b027/attachment.pgp