RK Server Replacement

By: Eric
Published On: 1/22/2008 2:00:00 AM

WERE DONE.  IF YOU SEE THIS MESSAGE YOU'RE ON THE NEW SERVER!!!! 

We apologize for the minor glitch on the old site as we made this transition.  We were not able to turn off the commenting feature without hiding all the comments. 

ONE FINAL UPDATE!!!!

We will push back the time until 10PM EST tonight in order to allow everyone a little time to discuss the SC results. 

UPDATE  UPDATE UPDATE UPDATE 

Due to ongoing hardware problems we will be replacing the server RK currently uses.  While this change will have no long term impact (except, of course, that our hardware will function better), there will likely be a short term impact due to a DNS change. 

The new plan is as follows:

a) At approximately 8pm (EST) on SATURDAY Jan 26 we will set RK to be "read-only" - meaning you can view the site but will not be able to post comments or diaries.

b) We will bring up the new server (with up-to-date data) and make the required changes to the DNS.

c) The new server will function normally - you will be able to post diaries and comments.  The new server availability will depend on DNS propagation (individual experiences may vary with some seeing the new server and some seeing the old for a short period of time) - it may take up to 48 hours but we expect most users to be seeing the new server by talk show time on Sunday morning.

 

 

Specifically, for a period of up to 48 hours (although usually it takes much less time), while the DNS change is propagated through "the tubes", we may lose some or all of the diaries, comments, registrations, or anything that is entered into our database (which pretty much covers everything you can do here). 

We will do our best to retain as much as possible during this period.  However, we highly recommend that you keep a copy of anything you definitely do not want to lose.  This will likely be any diaries you write during the transition time, but you may have comments or information in comments (like links to sites, video, etc) that you may also want to keep.  

After the change is complete and we have restored the transitional data entries as best we can, we will notify the community.  At this point, if any of your entries have been lost, we ask that you repost them.

Scheduled DNS change: 12 noon, Sunday Jan 27

If this date changes we will notify you in this diary. 

Please post any questions/concerns in the comments section below.


Comments



Sorta reminds me about the White House Email system... (Robespierre'sGhost - 1/22/2008 3:08:48 AM)
Ever heard of a backing up your data?

Google "Website Downloader for Windows" It will pull off everything and archive it for you, no problem- in fact I have already done so with several interesting diaries / comments!

"You can't erase e-mails, not today," said Leahy, D-Vermont. "They've gone through too many servers. They can't say they've been lost. That's like saying, 'The dog ate my homework.' "

Leahy said the e-mails would have remained on party or campaign computer servers, and he compared the situation to the famous 18½-minute gap in one of the Watergate tapes." http://www.cnn.com/2007/POLITI...

Just Kidding Lowell! I am sure you don't do anything for your own personal political purposes!  



It's not a backup problem (Eric - 1/22/2008 10:40:37 AM)
It's a matter of keeping the auto-generated primary keys of two versions of the same database in sync while we wait on the varied lag time of DNS propagation.  


Hmm (tripinva - 1/22/2008 12:10:08 PM)
The ideal thing to do would be to just disable the site on the old server, so people don't post things to it once you've backed up that database.  It'd make everyone's lives much easier.


We're looking into (Eric - 1/22/2008 12:47:55 PM)
that sort of thing as an alternative.  We wanted to post our initial plan ASAP to give everyone a warning - but we're still shooting for a smoother transition so the plan may change slightly.

Please keep an eye on this post (it'll be near the top for the rest of the week) for updates.



One thing you could do... (ericy - 1/22/2008 4:22:14 PM)

Is in advance of this, change the DNS TTL (time-to-live) down to something fairly short.  Perhaps only an hour - maybe 20 minutes or even less.

When you actually get around to doing the migration and changing the IP address, various clients won't hold onto the old IP address for very long before they go and ask for a fresh copy, and thus minimizing the disruption.

Once the migration is complete, you can change the TTL back up to 48 hours or so.



Good Luck Eric :-p (citizenindy - 1/25/2008 5:27:31 PM)
n/t


Also (relawson - 1/26/2008 6:29:58 PM)
If you have two servers (instances of the RK web app), you should be able to point them both to the same database.  As DNS is updated, users are simply pointed to the new server.  You can take the old server offline in a few days.  This should result in no interuptions.

Are there application specific settings that prevent you from running the application on two different domains?  Is the database on the same server as the web app?

If you guys run into problems and need an extra hand feel free to email me - relawson AT sdsflorida dot com.



don't answer those questions here (publicly) (relawson - 1/26/2008 6:31:48 PM)
That information is not something you want the general public to know.


Thanks! (Eric - 1/26/2008 6:39:55 PM)
The db will be moving as well, so we'd need off time to run backups and install as well.  We'll have some sort of down time no matter what.

We're pretty sure that with the latest plan there will be minimal impact.  No data lost - just a small window where people won't be able to post.  By morning all should be well.

But I do remind people that sometimes, on occasion, the DNS propagations take longer than expected.  That's why the 24-48 hour warning... but it is a worst case scenario.  

Thanks for the offer for help.  We'll ping you if we get stuck, but things are looking good at the moment.



DNS up really quick (Greg - 1/26/2008 11:41:40 PM)
Didn't take long for it to propagate. I'm on Verizon DSL.


Yes, that wasn't so bad. (Lowell - 1/26/2008 11:47:33 PM)
Eric's a genius! :)


Well Done (relawson - 1/26/2008 11:56:36 PM)
I didn't notice the interruption.


whats up with DNS? (relawson - 1/27/2008 12:00:10 AM)
Did you update the name server manually?

This is exactly why I like to have control over DNS and use my own name server for all of my sites.  That way, when you change servers there is no interruption.  I've been a big fan of http://www.no-ip.com/ for years.



Per ericy's comment (Eric - 1/27/2008 12:01:34 AM)
above - TTL is set low.


what is default ttl? (relawson - 1/27/2008 12:04:06 AM)
You may want to set it back in a few days.  The reason for a long TTL is that if your DNS server goes down the people who already have it cached won't have a problem.  Buys you time.


Cool (Eric - 1/27/2008 12:10:00 AM)
We'll have a look in a few days.  But so far things seem to be going well - that's what counts for now!


Yes, live for the moment! (Lowell - 1/27/2008 12:10:41 AM)
"The fierce urgency of NOW," as Barack Obama likes to say. :)


read ericys post (relawson - 1/27/2008 12:06:02 AM)
He knows what he is talking about ;-)


i see new servers (soapblox - 1/27/2008 12:00:25 AM)
yyayaya


And dead people (Lowell - 1/27/2008 12:02:29 AM)
heh


There may be a problem (relawson - 1/27/2008 12:14:10 AM)
Just thought of this.  I have been on this site looking around tonight.  So, if you updated your TTL to a short duration my cache would have been updated during that period - hence my ability to post here.

But, the people who aren't frequent visitors (meaning tonight) may not be able to connect for whatever the old TTL was set to.  

A real test would be to go to a computer that you know hasn't accessed RK today (ideally on a different network).  Try hitting the site from there.

Someone remind me how DNS is replicated ;-) but I think this may be an issue.



If I'm understanding (Eric - 1/27/2008 12:44:50 AM)
what you're saying correctly, I don't think there will be a problem because the TTL has been set low for the past few days.  

Either way, even with a long setting, I'm expecting everyone to join us sometime in the next day or so.  But we will keep an eye on things over the next few days.  And please do let us know if you experience any problems (or hear of others who are running into problems) so we can troubleshoot as quick as possible.



No problem then (relawson - 1/27/2008 1:16:10 AM)
Yes, you understood correctly.  If it was lowered a few days ago, all should be well.  I thought it was changed tonight.

Great job.



Fits like a glove 8) (Will Write For Food - 1/27/2008 4:55:54 AM)