Downtime for new servers

WebmasterAlex · May 26, 2015

We are moving to new servers. The GOOD NEWS for those of you that have been around for a while is this will be nothing like the old days where that involved major changes and days of screwed up boards.
Basically there will be 2 short maintenance windows
Wednesday May 27 @ 7:30pm CST/8:30pm EST for approximately 30 minutes
Saturday May 30 @ 9pm CST/10pm EST , this migration should theoretically not involve any downtime at all however reality is we may have a few periods of downtime and some degraded performance for a few hours

Technical for those that want to geek- The first maintenance period is to replace the firewall with a higher capacity unit
The second migration period is to vmotion the virtual machines from some 3 year old Dell iron to HP DL 380's with dual 12 core (E5-2680) procs, 128 gigs of ram and a lot of fiber channel SAN goodness

WDWKOOK · May 27, 2015

WebmasterAlex said:
We are moving to new servers. The GOOD NEWS for those of you that have been around for a while is this will be nothing like the old days where that involved major changes and days of screwed up boards.
Basically there will be 2 short maintenance windows
Wednesday May 27 @ 7:30pm CST/8:30pm EST for approximately 30 minutes
Saturday May 30 @ 9pm CST/10pm EST , this migration should theoretically not involve any downtime at all however reality is we may have a few periods of downtime and some degraded performance for a few hours

Technical for those that want to geek- The first maintenance period is to replace the firewall with a higher capacity unit
The second migration period is to vmotion the virtual machines from some 3 year old Dell iron to HP DL 380's with dual 12 core (E5-2680) procs, 128 gigs of ram and a lot of fiber channel SAN goodness

Thank you for letting everyone know, @WebmasterAlex. Sounds nice and geeky :thumbsup2

with not too much :badpc:

PollyannaMom · May 27, 2015

Thanks for the heads-up!

WebmasterAlex · May 28, 2015

The firewall migration went well! We have upgraded the firewall from a 300 Mbps unit with 50,000 concurrent connections to one that will handle 1.2Gbps with 250,000 concurrent users. Have to love more power!

andrewilley · May 28, 2015

Those numbers sound refreshingly biggerer. 250k concurrent users, wow!

Also, I assume it goes up to '11' ?

Andre

WebmasterAlex · May 28, 2015

andrewilley said:
Those numbers sound refreshingly biggerer. 250k concurrent users, wow!

Also, I assume it goes up to '11' ?

Andre

If you hear reports of massive explosions in Chicago you know we went too far.....

twocat · May 28, 2015

Will this upgrade shorten the 60-90 minute downtime we experience every morning around 4 am EDT?

WDWKOOK · May 28, 2015

twocat said:
Will this upgrade shorten the 60-90 minute downtime we experience every morning around 4 am EDT?

Whoa there, that "downtime" has allowed me an extra 10 minutes of interwebs surfing since I can't log-on till 0530.

Since we're on the topic, @WebmasterAlex, what DISmagic is occurring till 0530 :confused3

WebmasterAlex · May 28, 2015

It may shorten it a bit as these servers have more performance but it's hard to say without just trying it. That time period is the backup system working it's magic, a pain obviously but really important

bankr63 · May 29, 2015

Wow, backup windows still. I guess we know what the next hardware investment should be. We just swapped onto Unitrends for our 7*24 datacentre and really liking them (have a second unit mirrored to an offsite location). Check out the gab on SpiceWorks to see what's hot now; lots of options.

WebmasterAlex · May 29, 2015

bankr63 said:
Wow, backup windows still. I guess we know what the next hardware investment should be. We just swapped onto Unitrends for our 7*24 datacentre and really liking them (have a second unit mirrored to an offsite location). Check out the gab on SpiceWorks to see what's hot now; lots of options.

Well couple of issues there.. I have used Unitrends for years in other places. They have no MySQL agent, only some cobbled together ideas.
That wouldn't even be the issue here though, our servers are hosted in a large datacenter and they supply the backup systems, we don't really have any options
So unfortunately backup window it is

andrewilley · May 29, 2015

If it was my site I'd still be very concerned that they have a daily 90 minute backup window which takes the site completely offline (from 9am until 10:30am every day where I'm based). That's the equivalent two whole days of site downtime per month, which is a pretty poor statistic in anyone's uptime logs. I can't think of any other commercial site that I use (large or small) which accepts a 90 minute daily outage as normal. Surely they could operate some sort of concurrent access lower priority SQL backup task, which might slow the site down a bit during off-peak times, but wouldn't kill it completely? Or better, operate a live mirror.

Andre

WebmasterAlex · May 29, 2015

andrewilley said:
If it was my site I'd still be very concerned that they have a daily 90 minute backup window which takes the site completely offline (from 9am until 10:30am every day where I'm based). That's the equivalent two whole days of site downtime per month, which is a pretty poor statistic in anyone's uptime logs. I can't think of any other commercial site that I use (large or small) which accepts a 90 minute daily outage as normal. Surely they could operate some sort of concurrent access lower priority SQL backup task, which might slow the site down a bit during off-peak times, but wouldn't kill it completely? Or better, operate a live mirror.

Andre

It's not like there are any options. It's a huge operation backing up 1000's of servers. You either use their system or you don't. They supply the agent and guarantee the backup. They actually developed the software http://hollandbackup.org/ . Part of the problem is that all of their clients want to be in that same time slot.
I could do a mirror but the cost and complexity wouldn't even come close to the traffic that is affected at that time period. You have to remember that the vast majority of our traffic is US based.

andrewilley · May 29, 2015

I can see you're pretty much stuck with what they are offering you, either that or not having a backup regime at all which is not a viable option of course. Their docs do say that their system offers a MySQL backup option that is either near-lockless or fully-lockless though - http://docs.hollandbackup.org/overview.html. Not sure if that might help in allowing the database to still service requests while a backup is in progress?

I assume you've probably already asked them the question anyway, as I'm sure you don't enjoy having 90 minutes of hard downtime for the site per day. I do understand that it's your lowest traffic period, but it's still a bit annoying for Brits and other Europeans for whom it's just at the start of our usage day.

Andre

Carriemel · May 30, 2015

WebmasterAlex said:
Technical for those that want to geek- The first maintenance period is to replace the firewall with a higher capacity unit
The second migration period is to vmotion the virtual machines from some 3 year old Dell iron to HP DL 380's with dual 12 core (E5-2680) procs, 128 gigs of ram and a lot of fiber channel SAN goodness

*drool*

WebmasterAlex · May 30, 2015

35 minutes in and 11 servers down 2 to go... and everything running well
After this we still have to migrate the SAN storage but that's not so bad.....

WebmasterAlex · May 30, 2015

And we are done! I can remember when this would have taken weeks with a whole bunch of 16 hour days. We still have to move the storage SAN's but that should go pretty seamlessly. For those who are curious here is my playlist for the migration. I try stay off the heavy metal for this kind of thing, my adrenaline is usually high enough!

lanejudy · May 30, 2015

Thanks @WebmasterAlex ! :music:

WDWKOOK · May 31, 2015

WebmasterAlex said:
And we are done! I can remember when this would have taken weeks with a whole bunch of 16 hour days. We still have to move the storage SAN's but that should go pretty seamlessly. For those who are curious here is my playlist for the migration. I try stay off the heavy metal for this kind of thing, my adrenaline is usually high enough!

Music to migrate by! I think you've come up with a whole new iTunes genre

. Thank you for the seamless migration and upgrades, :surfweb:

@WebmasterAlex.

PrincessShmoo · Jun 1, 2015

WebmasterAlex said:
Saturday May 30 @ 9pm CST/10pm EST , this migration should theoretically not involve any downtime at all however reality is we may have a few periods of downtime and some degraded performance for a few hours

OK, it's now 2 days later and I'm still having issues with DISboard recognizing that I'm logged in. I log in when I first get here and for a while I'm good, then I'm not and the "log in/sign up" button appears again. BUT, when I click on my avatar it's showing the little green triangle in the corner that says I'm logged in. Anyway, I try logging in again, and it just stays on the window telling me to log in.

OR, when I try to post a photo on the DCL photo of the day thread it says "You must be logged in to do this". So I try logging in and it doesn't take, and then totally drops my log in out. I can post photos fine on other threads, so I'm not sure what the issue is here. Although that's mostly because I'm computer challenged.

How long is this going to go on for?

Downtime for new servers

Administrator

NOW WITH LESS TAG!

I was a click-clack champ!!

Administrator

GNU Terry Pratchett

Administrator

80 miles door-to-door to Disney.

NOW WITH LESS TAG!

Administrator

DIS Veteran

Administrator

GNU Terry Pratchett

Administrator

GNU Terry Pratchett

Emma Clare's Mommy

Administrator

Administrator

Attachments

Moderator

NOW WITH LESS TAG!

DIS veteran

Share this page