Our server fainted some time this morning. I became aware of it at 10am, and have been in contact with the folks who manage the hardware. They’re tightening bolts, greasing cogs, and heaving wrenches around as I type. No, wait, that greasing cogs thing was a what machine techs did a century ago. They’re, instead, checking hardware and running diagnostics. Watch this space.
I’ve just discovered the outage, and am taking steps to get it kickstarted. Watch this space.
Edit, 1:41am: The rebooting is proceeding. Hold your breath!
Edit, 1:44am: Hooray! We’re up and running again.
Edit, 1:49am: Okay, we’ve got HTTP and POP3 and all, but the database server is still hiccuping. Watch this space…
Edit: 2:05am: Everything seems operational!
Please watch this space for updates, or contact me through this site if you need to!
-Elaine
Edit: Dec 16, evening
The hard drive on the server failed. A new drive has been installed, and backups reinstated. Everything should be up and running now — but please let me know if anything is still awry for you.
The MySQL server’s down for anyone on Abaris server (that’s most folks). I’m on it and will keep you posted here.
edit: I just found out that SMTP (mail SENDing from yourdomain.com) seems to be affected as well, which is odd as they’re not in the least connected. I’ll look into that as well. POP3 (Mail receiving) is working fine.
edit: 3:46pm PST: MySQL is going up and down like a yo-yo. I’m still on it. Bugger.
edit: 6:34 pm: I’m getting a huge number of very different error messages from MySQL — but a few seem to tell the tale that we may be seeing a DOS attack, either accidental (a script gone wrong, querying the database over and over and over and over and over and over) or on purpose (a script written to go wrong, querying the database over and over and over and over and over and over). Watch this space for updates.
edit: 11:19pm: The database has settled somewhat, but is still blanking at intervals. We’re still working on it!
Resolved: The rogue script has been found and stomped upon. If you have any database troubles now, contact me, and I’ll investigate!
Right about now, the “signal” from my main hosting server is interrupted someplace between my ISP and the server farm, and it’s being dropped instead of re-routing. In layman’s terms, the phone line is frayed between here and there — and sadly, it’s not frayed in the place that I control.
I can’t FTP, collect email or access the sites I run without hideous proxy contortions. I have people on it, but if you’ve tried to reach me, that’s what’s up.
Watch this space.
-Elaine
DEC 3, morning — update
Everything seems to be righted. A firewall at the server end had gone a bit rogue. It’s been soundly thumped, and should behave slightly better hereafter.
7:59 PM. PST
We’re experiencing the same troubles as before.
I’m communicating with the server techs again to try to straighten it out. I’m also gnawing on the ears of the folks who manage the server farm about settling the server thing once and for all.
If you send using your domain’s SMTP, swap temporarily to your ISP’s (eg: Telus or Shaw). The SMTP is unreliable in routing.
More info as it happens.
1:26am September 23
Okay, we discovered the cause.
One of the accounts on the server was so bombed by incoming spam that the entire mail queue choked.
Things should be back to normal for you. If not, please get right back to me via the support interface (link to your right —> )
I hate spammers. But you knew that, right?
Notice: this is NOT a high-security solution for database backup! Use at your own risk.
Verbose instructions, including a step-by-step example, are included in the zip file.
Download: techdonkeycom-database-backup.zip
This package will enable you to set yourself up a browser-visitable simple one-button backup of your MySQL database.
It was cobbled together by Elaine Miller from a few different scraps of GPL code, and specifically was constructed to have a bit of security on the index file while side-stepping the problems caused by .htaccess directory-level protection battling with the URL rewrite rules of the omnipresent Wordpress.
Users on a different architecture than my techdonkey.com clients may need to tweak their settings differently, and advanced users may wish to tweak even more. Have at it. If you make changes that are Very Useful To Others, please shoot ‘em back to me and I’ll incorporate ‘em in this package.
time: 5 min
geekiness level: minimal
——————————————————-
You’ll need these skills to start:
be able to edit a text file
be able to transfer files to your account, whether by FTP or SCP or Cpanel’s FileManager
——————————————————-
You’ll need this info before you start:
your database name
your database username
your database password
your account/site username
——————————————————-
Notice: this is NOT a high-security solution for database backup! Use at your own risk.
Verbose instructions, including a step-by-step example, are included in the zip file.
Download: techdonkeycom-database-backup.zip
Today, since about 3pm PST, we’ve all been having some email issues.
Both SMTP and POP, plus mailman mailing lists: all are affected.
I’ve been working with the on-site server tech for the last hour to resolve this. More info as it happens.
If you need to reach me, elaine *at* techdonkey *dot* com is the way to go.
11pm update:
After a great deal of harassing my hapless server cowboys, things are coming back online. Some sent mail and some rec’d mail is intact, but I’m not yet getting an accurate count of what’s working correctly.
noon Sept 15th update:
Everything seems to be working correctly, according to our tests.
CPanel has been upgraded, and this apparently may cause some changes in the optimum email account setup. I’m researching it, and will send out an announcement if it affects you.
If you have lingering email issues, please open a Support Ticket here, and I’ll get under the hood again.
Sorry for the technological inconveniences!
-Elaine
Hullo, O Clients…
Within your CPanel is a nifty thing called Fantastico. Great name and all, but it should really be called something descriptive, like The Thingy That Installs Nifty Web Software For You (As Long As You Have An Available MySQL Database) And By The Way, It Also Monitors Whether The Installed Software Is Up To Date.
On second thought, Fantastico is not such a bad name.
Did you get an email saying you needed to upgrade? Fantastico! Here’s what you do!
If you are running a standard (no adaptations, mods, frills, and bells and whistles) version of the program in question, and know it, you can do the upgrade yourself by pushing the appropriate button. Go into Cpanel, Find the Fantastico button and click it. You’ll see a listing of fantastico-installed software, and a notification that one or more is out of date.
STOP! Is there a lot of custom work on your application? Jeepers, don’t push that button! Call your techdonkey.
If you’re sure (sure, now) that you’re running a standard version, go ahead and… get back out of the Fantastico upgrade screen and into your backups screen. Make a backup and download it to your home ‘puter. Do it now.
Okay, now go back and tell Fantastico to upgrade. You need a bit of room to do so.
Out of room? As a quick fix, throw out the contents of your tmp folder your the root directory IF you don’t care about reading your fascinating webstats.
Go back and check your application, whether Wordpress or phpBB or what have you.
Does it work? Hooray! In your root folder, you’ll find a folder named something like “fantastico backups”. It’s now safe to delete the *.tgz file within. Or download it to your home ‘puter and then delete it from the server, in case you’re paranoid about backups.
Have you done a backup lately? Have you?
Somewhere around 12:19pm PST (give or take a millisecond), the database server (MySQL) for all our sites fainted.
I’m on it. Watch this space.
Update: 12:30pm
It’s up and running. I’d like to state that I did something extraodrinary, but I’d just started kicking the tires when it revived and bounced back to work without a murmur of explanantion. Okay, MySQL server, if that’s the way you’d like to handle it. I see how you are.