Our upstream hosts (the geeks who farm the servers) have not been able to fix the rather wonky DNS issues that have been causing roving blackouts to surfers trying to access your sites, and yesterday they pulled the CPanel backup module from my despairing grasp. Can’t do backups… what?

Since this is the tail end of a number of other issues (PHP errors, slow service response time, and so forth), I’m afraid we’re hitching up the covered wagons and we’re headed to our new home.

If you are a client of mine and you need to do anything, I’ll contact you.

If you don’t hear from me, I’ll be doing the move myself, and you should not experience more than an hour’s outage.

Do contact me if you need anything.

I’m on it.

Update: This is an odd error seemingly with the Telus nameservers at the root. Only Telus clients seem to be affected by it — the server itself is having no downtime issues at all.

I’m trying to get in touch with Telus to have them fix this glitch, but… you know. Telus.

Update: this is definitely a Telus issue.

While I’m spending days arguing with techs who are oblivious, following the instructions here will fix your problems.

http://www.handcoding.com/archives/2005/04/15/alternate-dns-servers/

It’s not ideal, as you should be able to depend on your ISP for domain name services, but at least it will get you access to your sites again.

I’ll continue to poke Telus.

Here’s a screenshot of the solution (using 4.2.2.1 and 4.2.2.2) working on my very own computer. (Win XP)

Works on Vista, Mac, Linux, of course, but you have to find your own TCP/IP settings to input these DNS settings.

Click to enlarge:

-Elaine

- Elaine

I’m all over it. Watch this space for updates.

5:44 pm

Still working on finding out what’s wrong, and getting on the path to righting it.

6:43pm

Haven’t forgotten about y’all.

7:54pm

Still on it, still frustrated.

9:05pm

Sorry folks, I’m getting nothing from the physical people who run the physical server. It’s like they disappeared into thin air. So not cool. I’m trying to track down a responsible human, still.

12:48am Dec 20

We’ve made contact and we’re working on getting the damn server back up. Thanks for your patience so far… Watch this space.

1:49am Dec 20

I’m beginning to be worried that they had some sort of gas leak at the server farm, and all the server techs fainted as they… tried…to…crawl…to..the…door. <gasp>  Ask with me — why hasn’t this been fixed yet? Why won’t they update me, so I can update you?

I’m to bed, and I’ll be back on it first thing in the morning.

Thanks again for your patience.

10:03am Dec 20

I’m highly dissatisfied with the service we’re not getting in this outage. Not just that we’ve been down since yesterday, when I could have mail-ordered a brand new machine and had it delivered in that space of time — but also with the lack of any reasonable communication from the company herding the servers.

Sorry, folks — it seems like the holidays is the worst time to have tech issues. I’m starting to think that everyone’s on vacation and they left someone’s 8-year old niece alone in the facility to type “We are aware of the issue and currently have staff working on the server to get it back online as soon as possible.” to everyone who desperately inquires, and then go back to playing with her dolls.

Now that the office is open, I’m on the phone.

10:25am

Apparently, some folks work only Mon- Fri, 9 – 5. I’ve heard of such a thing, but never experienced it. Electronic support is 24/7.  I mean, that’s what they say right there in the advertising.

I’m whiling away the time shopping for a new home for us.

10:42am

(Rant)

The server should not have been down (worst-case scenario) more than 4 hours. That’s my guess of how long it would take if the entire machine burst into flames. That’s approx. the time it would take to unplug it, pull it out of the rack of millions of other servers, put in a fresh machine, initialize it, and restore backups. And I’ve included in that estimate the time it would take to take a new machine out of the styrofoam and cardboard box.

(/Rant)

If I can’t get service and/or information very soon, I’m considering moving us to a new host. As this always involves considerable disruption and a site-by-site re-up and re-configuration, I would be happy to avoid this, expecially over the holiday season; thus I’m hoping they’ll correct their retrocranial inversion and solve the problem as sooon as possible. If we take this course, I’ll be contacting you both collectively and individually.

11:50am

We are, unaccountably, up. As far as I can tell, it’s a measure of spontaneous server regeneration, because I’ve not yet rec’d any explanation from the company minding the server herd. It’s a miracle! Do your backups, quick.

I’m experiencing annoyance, and passing it right along to the folks who mind the servers like as if they’re little cows in a field.

Watch this space.

Edit, 1:41am: The rebooting is proceeding. Hold your breath!

Edit, 1:44am: Hooray! We’re up and running again.

Edit, 1:49am: Okay, we’ve got HTTP and POP3 and all, but the database server is still hiccuping. Watch this space…

Edit: 2:05am: Everything seems operational!

-Elaine

Edit:  Dec 16, evening

The hard drive on the server failed. A new drive has been installed, and backups reinstated. Everything should be up and running now — but please let me know if anything is still awry for you.

edit: I just found out that SMTP (mail SENDing from yourdomain.com) seems to be affected as well, which is odd as they’re not in the least connected. I’ll look into that as well. POP3 (Mail receiving) is working fine.

edit: 3:46pm PST: MySQL is going up and down like a yo-yo. I’m still on it. Bugger.

edit: 6:34 pm: I’m getting a huge number of very different error messages from MySQL — but a few seem to tell the tale that we may be seeing a DOS attack, either accidental (a script gone wrong, querying the database over and over and over and over and over and over) or on purpose (a script written to go wrong, querying the database over and over and over and over and over and over). Watch this space for updates.

edit: 11:19pm: The database has settled somewhat, but is still blanking at intervals. We’re still working on it!

Resolved: The rogue script has been found and stomped upon. If you have any database troubles now, contact me, and I’ll investigate!