CAUTION: If you are checking in Tuesday, October 9th

[SheilaHeartsDisney]

What, pray, do you do if you went to a sit down restaurant in the park, and after your meal you found that your room key doesn't work and you took out your only credit card and that does not work either because it was maxed out?

Would they let you write out a note payable (an IOU) using ordinary pen and paper and send it to your resort to resolve later, in the evening, after the systems are back up?

I guess that's why they are telling you in advance with the letter so you can plan for that situation. You would either bring cash or a credit card that is not maxed out.

 

[Cheshire Figment]

I am (thankfully) not involved with this in any way. However, some information I do know (in no particular order).

There are three major systems at WDW which affect Guests. Sometimes they do not talk well to each other.

Oldest - Automated Ticketing System - This is at all Park ticket and Guest Relations locations. It controls all ticketing and turnstiles and does tie in to FastPass.

Medium - a la carte - Everything dining, especially reservations and dining plan.

Newest - LILO - which has everything to do with Rooms and Reservations. It replaces the old DMPS which was based on the Las Vegas model. It connects into both ATS and a la carte by interfaces and does not have all the capabilities of those systems.

And, as far as complexity, I know a person who has worked for Disney IT for years, her previous position was in charge of the team that was responsible for keeping over 2,000 Disney servers in synchronization. That's a large amount of computer power.

And several years ago I was the Accountant in charge of the Central Accounting System of the Federal Courts. When we did a major system upgrade we had to shut our system down early on Friday, do a complete system back-up, and (even though we had done lots of testing) then install and test the new software in the live system to make sure it worked. Normally it took us until late Saturday to be sure everything worked right. So there really is a lot of work that has to be done and the system must be down so they can be sure it was done correctly.

 

[GillianP1301]

The system that's being discussed has (by definition) a single, non-redundant point of failure. You have a database that is transactional and relational. It doesn't matter how many copies of that database you have, or how you're syncing updates. A schema change means no writes. A new interface (that does writes) means no writes.

I agree that they're probably stating a longer window than they expect. Industry standard is the greater of double or implement+QA+rollback. I disagree that the outage is likely to start at midnight. I'd anticipate that there's a brief grace period where staff is told to finish their transactions, and then it goes read-only.

My "when will they be back?" hunch is that they're expecting to be out of QA and ready for customers ~7am on the 6th, if all goes well. Then the rest of the window would be insurance in case of a roll-back.

Yup! Unfortunately, it wouldn't help much. I suspect that they chose their window to impact as few people as possible, but that doesn't make it any easier to be one of them, and there will be a lot more if the window goes badly.

Some good points here, but I still think the length of time is a bit suspect.

Regardless, they are forewarning people, which is about all they can do and have chosen, what is probably, one of their lower transaction days. Doesn't make it any less annoying or inconvenient for people though. Sorry for those impacted and for the CMs who will have to deal with the fallout. The extra time they will have to spend explaining/justifying the situation is going to slow down their productivity and cause longer waits for people. Hopefully Disney will compensate for that and up the staffing levels a bit.

 

[OrangeCountyCommuter]

Disney has the worst IT shop I have ever heard of. (And I work with idiots in my shop! LOL!)

There is NO excuse for a downtime of this long but "we wanted to screw up EVERYONE at once" Working in healthcare I know that there is NOTHING you can't figure out how to do during the wee hours of the morning with minimal impact, Sure we may have to work 2 AM to 5 AM for 10 days in a row but our patients live! If they depended on Disney then.....

And of course since it's Disney the testing will be Wed when we all discover together that it doesn't work!

 

[OrangeCountyCommuter]

What, pray, do you do if you went to a sit down restaurant in the park, and after your meal you found that your room key doesn't work and you took out your only credit card and that does not work either because it was maxed out?

Would they let you write out a note payable (an IOU) using ordinary pen and paper and send it to your resort to resolve later, in the evening, after the systems are back up?

Here is a much more realistic one I can see happening a lot

Family A either (A) didn't get the notice or (B) didn't see it. (I was there last week there was a notice in my room on the road closures etc for the Tower of Terror race. I saw it on Monday as I was leaving... two days after the race! )

So they head off to the park with ONLY the KTTW card. Since the kids have cards to buy snacks etc on the DDP they did NOT put a credit card on thier room so the kids couldn't buy stuff. They eat dinner and BOOM no way to pay! Disney is screwed IMHO! I would just tell Disney that's there problem, write my info down and I am heading to the park LOL!

 

[kamik86]

The system that's being discussed has (by definition) a single, non-redundant point of failure. You have a database that is transactional and relational. It doesn't matter how many copies of that database you have, or how you're syncing updates. A schema change means no writes. A new interface (that does writes) means no writes.

There ARE ways to do schema changes and new interfaces without bringing the system down. It takes more time and better planning. And that means it costs more but it can be done.

Do you think banks never change their databases (can you imagine the outcry if no one could take money out of a bank for a day? No debit cards, ATMs etc? or worse those databases used for security applications!)

I work for a company that is installing systems that need to be up as close to at all times as possible. It is very dangerous and very expensive if this system has to be taken down, even for an hour so they pay very well for us to find ways to upgrade the system without ever taking it down.

 

[HayGan]

Here is a much more realistic one I can see happening a lot

Family A either (A) didn't get the notice or (B) didn't see it. (I was there last week there was a notice in my room on the road closures etc for the Tower of Terror race. I saw it on Monday as I was leaving... two days after the race! )

So they head off to the park with ONLY the KTTW card. Since the kids have cards to buy snacks etc on the DDP they did NOT put a credit card on thier room so the kids couldn't buy stuff. They eat dinner and BOOM no way to pay! Disney is screwed IMHO! I would just tell Disney that's there problem, write my info down and I am heading to the park LOL!

Disney really needs to have informed guests when they check in of the outage. That way families can make a decision on how to handle their own finances to deal with the situation if it should arise.

In the case of the dining plan, they should provide families with vouchers to use in case of not being able to deduct credits for meals. Those vouchers could then be deducted from their allotment once the system is back up and running properly.

It is really poor CS to promote a method of payment and then say sorry but you can't use that method when the guests need it.

This isn't their first rodeo when it comes to system upgrades. They really should have something in place for this stuff!!!

 

[LadyBeBop]

What, pray, do you do if you went to a sit down restaurant in the park, and after your meal you found that your room key doesn't work and you took out your only credit card and that does not work either because it was maxed out?

Would they let you write out a note payable (an IOU) using ordinary pen and paper and send it to your resort to resolve later, in the evening, after the systems are back up?

I'd have fears they'd send me into the kitchen to wash the dinner dishes.

That said, the restuarants would have to know something is going on. In the case like that, perhaps take the customer's name, address and telephone number. In a few hours, they could debit his and his family's dining credits with the meal. And in the odd circumstance that someone not on the DDP is pulling a fast one, contact him in a few weeks to make good on the meals.

 

[donaldlovesdaisy]

Disney really needs to have informed guests when they check in of the outage. That way families can make a decision on how to handle their own finances to deal with the situation if it should arise.

In the case of the dining plan, they should provide families with vouchers to use in case of not being able to deduct credits for meals. Those vouchers could then be deducted from their allotment once the system is back up and running properly.
It is really poor CS to promote a method of payment and then say sorry but you can't use that method when the guests need it.

This isn't their first rodeo when it comes to system upgrades. They really should have something in place for this stuff!!!

That does make more sense than having to reimburse people for their dinning. I wonder why they choose the reimbursement route over this one? Maybe they are hoping people will get caught up in their vacation and forget or lose their receipt. I can't say that I would ever make it back up there to be reimbursed because I would probably forget and misplace my receipt.

 

[mackenziefaith]

Here is a much more realistic one I can see happening a lot

Family A either (A) didn't get the notice or (B) didn't see it. (I was there last week there was a notice in my room on the road closures etc for the Tower of Terror race. I saw it on Monday as I was leaving... two days after the race! )

So they head off to the park with ONLY the KTTW card. Since the kids have cards to buy snacks etc on the DDP they did NOT put a credit card on thier room so the kids couldn't buy stuff. They eat dinner and BOOM no way to pay! Disney is screwed IMHO! I would just tell Disney that's there problem, write my info down and I am heading to the park LOL!

I have to say, THIS is why I'd be upset. My husband and I are those people. We run out in the morning with everything but our KTTW cards and kids. We'd be the ones panicking about suddenly not being able to get into a park, or pay for the meal we'd just eaten simply because our cards weren't working, not because we couldn't afford it.

I will say, we had the opposite happen during our last trip. We had our snacks accidentally reset during our trip. We typically don't eat a ton of snacks, but suddenly on the last night of our trip we had all of them back. I asked a CM about it and she said to just consider it pixie dust and take the kids to Goofy's to stock up on treats for the drive home.

 

[Tiger926]

There ARE ways to do schema changes and new interfaces without bringing the system down. It takes more time and better planning. And that means it costs more but it can be done.

Do you think banks never change their databases (can you imagine the outcry if no one could take money out of a bank for a day? No debit cards, ATMs etc? or worse those databases used for security applications!)

I work for a company that is installing systems that need to be up as close to at all times as possible. It is very dangerous and very expensive if this system has to be taken down, even for an hour so they pay very well for us to find ways to upgrade the system without ever taking it down.

Hubby is a bank manager and system upgrades are ALWAYS after hours and especially on the weekends (Sat afternoon - Monday morn).

The only time the system has a glitch during the day (it's rare) is during an emergency.

I am confuzzled as to the length of time Disney has the outage for. There are going to be scores of guests who will have no idea - some major issues I'm sure over the next few days...

Tiger

 

[Disney_1derland]

Well, wish I had found this thread earlier. We received no notification from our resort. We are at MNSSHP tonight and just tried to feed our kids. Were told no and that there is no recourse and no way they can help us. We didn't bring anything else other than room keys into the parks because in 14 trips we have NEVER had a problem. It's almost midnight, even at our resort nothing would be open by the time we got there and we have hungry kids who haven't eaten since 2pm. Their answer was "really, you can't rely solely on your room key, we are sorry but there is nothing we can do". Well, if it weren't midnight, it wouldn't be a huge deal, I would go back, get the credit cards out of the safe and go on. Unfortunately, that's not an option at this time of night. He finally comped us one meal to split between the three kids but told us he wasn't willing to take responsibility beyond the cost of one meal because it would look bad on him. So, for dinner my three kids shared one hot dog and one order of fries. Guess it's a good thing they got all this candy tonight :-/

I'm rambling, but I'm beyond furious at the way they handled it. Or, rather, didn't handle it. Our room key has enough info on the front of it for them to be able to track us back for anything we incur. He could have taken the info and submitted it when the system comes back up. If we didn't have enough credits, they could have charged it to our credit card on file.

He also told us to expect it to be out the entire day tomorrow. I told him tomorrow wouldn't be an issue, it's tonight I'm dissatisfied with.

 

[smw]

$5 says everyone makes it out of this alive.

 

[SaraJayne]

Well, wish I had found this thread earlier. We received no notification from our resort. We are at MNSSHP tonight and just tried to feed our kids. Were told no and that there is no recourse and no way they can help us. We didn't bring anything else other than room keys into the parks because in 14 trips we have NEVER had a problem. It's almost midnight, even at our resort nothing would be open by the time we got there and we have hungry kids who haven't eaten since 2pm. Their answer was "really, you can't rely solely on your room key, we are sorry but there is nothing we can do". Well, if it weren't midnight, it wouldn't be a huge deal, I would go back, get the credit cards out of the safe and go on. Unfortunately, that's not an option at this time of night. He finally comped us one meal to split between the three kids but told us he wasn't willing to take responsibility beyond the cost of one meal because it would look bad on him. So, for dinner my three kids shared one hot dog and one order of fries. Guess it's a good thing they got all this candy tonight :-/

I'm rambling, but I'm beyond furious at the way they handled it. Or, rather, didn't handle it. Our room key has enough info on the front of it for them to be able to track us back for anything we incur. He could have taken the info and submitted it when the system comes back up. If we didn't have enough credits, they could have charged it to our credit card on file.

He also told us to expect it to be out the entire day tomorrow. I told him tomorrow wouldn't be an issue, it's tonight I'm dissatisfied with.

I'm sorry. That is an unacceptable response.

I'd be pretty darn irritated if I was told I had to pay again for my food (that I had already paid in full for) and oh, by the way, we'll maybe reimburse you within 30 days.

Yup, not gonna happen. You screwed up, you make it right.

No, no one is going to die but it sure is irritating and an inconvenience, especially for those like the above poster who weren't notified properly and were stuck with no options.

 

[mom2rtk]

I'm sorry. That is an unacceptable response.

I'd be pretty darn irritated if I was told I had to pay again for my food (that I had already paid in full for) and oh, by the way, we'll maybe reimburse you within 30 days.

Yup, not gonna happen. You screwed up, you make it right.

No, no one is going to die but it sure is irritating and an inconvenience, especially for those like the above poster who weren't notified properly and were stuck with no options.

I absolutely agree. For a company known for its customer service this falls way short of the bar. To sell a prepaid meal plan, then have it be unavailableto guests for ANY legth of time......... is just nuts.

 

[leelee9878]

I know a lot of people are upset but I am here now and saw the notice and think this is a HUGE improvement got Disney. I was here in December of 2010 I think it was (the ridiculously cold December) and Disney decided to upgrade on December 15th with no notice to guests. We weren't able to use dining plan, or check out and get our bill or anything. It was nuts and very busy and I felt so bad for the CM's. They did this upgrade during the busiest time of year, and didn't inform any guests. At least they are now informing guests ( I saw the notice in my room when I checked in), and using a slower time of year for the upgrade. It will still be an inconvenience but Disney is still a business that needs upgrades and no matter when they do it they will upset some people.

 

[SaraJayne]

I know a lot of people are upset but I am here now and saw the notice and think this is a HUGE improvement got Disney. I was here in December of 2010 I think it was (the ridiculously cold December) and Disney decided to upgrade on December 15th with no notice to guests. We weren't able to use dining plan, or check out and get our bill or anything. It was nuts and very busy and I felt so bad for the CM's. They did this upgrade during the busiest time of year, and didn't inform any guests. At least they are now informing guests ( I saw the notice in my room when I checked in), and using a slower time of year for the upgrade. It will still be an inconvenience but Disney is still a business that needs upgrades and no matter when they do it they will upset some people.

That is why most companies do their upgrades in the wee hours of the morning.

 

[Jujubee727]

Wow, I hope this is for today ONLY! We check in tomorrow!!!

 

[labattblue]

Sorry, but if the IT staff can't figure out a way to get this done in a 4-6 hour overnight window(with the ability to go back if the upgrade fails), then they should be replaced.

An outage this long is completely unacceptable. If they can't figure it out, then block out a two day window where WDW is completely shutdown(parks & resorts) to guests during that period.

 

[kaytieeldr]

I'm sorry. That is an unacceptable response.

I'd be pretty darn irritated if I was told I had to pay again for my food (that I had already paid in full for) and oh, by the way, we'll maybe reimburse you within 30 days.

Yup, not gonna happen. You screwed up, you make it right.

No, no one is going to die but it sure is irritating and an inconvenience, especially for those like the above poster who weren't notified properly and were stuck with no options.

While I agree the response was unacceptable, it would be really great if people didn't relay unsupported assumptions - 30 days? to 'maybe' get reimbursed?

That is why most companies do their upgrades in the wee hours of the morning.

I bet if this system changeover could have been done in the wee hours of the morning, it would have been. Apparently it's going to take somewhere in the vicinity of 20 or so hours to complete; early mornings just don't last that long, and spreading it out over five or six days wouldn't work.

Wow, I hope this is for today ONLY! We check in tomorrow!!!

If you read the original post, other posts by the OP, and responses from two Cast Members - you'll see it's expected to be completed by 3 PM today.