Unplanned Power Outage

On Sat Aug 23 at  07:55 CST there was insufficient power on the UPS to keep systems alive during a multi-hour power outage.

Systems are coming up now.

Local Network Outage

There is construction happening nearby and they managed to knock out communications for a few hours today.  

Service is restored now, all should be functional again.

Natural Disasters / Upcoming Outages

There was a pump failure at the facility these systems are run upon.  Though the floor was covered in 2 inches of water, the equipment was safely off the gro7und.  I did have to do some equipment re-arranging and cable re-sorting today which may have caused minor network outages as equipment was optimized.  The servers themselves were not rebooted.

More work will be done in this space to repair the floor damage which may require additional outages.  Dates still to come on that.

Unplanned Downtime – DDoS Attacks

For almost a week now my servers have been under attack on and off by multiple sources which seem to change by the hour as some asshole continually floods my connection for purposes I do not understand.

There is also a lot of rain happening lately and for some reason this always seems to cause my line quality to go badly.

Please bear with me, as there is little that can be done about both of these issues.

UPDATE:   I tracked down all of the DoS activity finally and services should be stable again.

Maintenance

Maintenance on the faulty hard disk is completed.  All systems should be operational again.

Unplanned Outage

Overnight one of the disks in the mirror stopped respondign to SCSI commands, causing a bad loop of retries.   A hard reset of the motherboard seems to have cleared it up.

FYI – I do plan to bring it down again this weekend to replace some SATA cables. 

Unplanned Network Outage

The server was being hit by “DRDoS” attack on the network time protocol daemon since this weekend.   I apologize for the delay troubleshooting it, as the effect looked very much like the packet loss previous experienced when line quality issues are in play.

Updates…

Home directory to web server mappings have been fixed.

You can once again access your public_html directories at http://www.probsd.net/~username/ as you could pre-migration.

Unplanned Outage

A NFS filesystem used during migration activities was left mounted when it was turned off at the source, which hung up mail retreival services and needed a reboot to clear up.   Sorry about that!

Migration Update

Migrated Services:
1. Mail (fully migrated)
2. HTTP/Web Services (mostly migrated, some remain)

Return top