"Frozen Ever After" or "Frozen Until It Breaks Again"?

[walker0829]

The Maelstrom Dökkálfar are angry that Disney told them they were being cut due to Shanghai overruns but it was really so they could be replaced with rocks that call themselves trolls. There's no other explanation for these issues.

 

[Bree]

We went last night around 8pm with a fastpass. MDE & TP showed the ride as closed on our walk over from Beach Club, but it was open for fastpass. The standby line was not open. We waited a total of 22 minutes from the time we entered the line until we were in the boat. What surprised me is that they didn't change the load/unload situation. They kept the same system from Maelstrom! Boat empties then moves forward to load more guests. No wonder the line backs up so much when they can only load one boat at a time! It's slow as molasses!

 

[mom2rtk]

Hate to disagree, but that's just not true in the IT world. I just got finished talking to sales about a bid where a client wants "always up" which they defined as "they want to walk into the core room and unplug the entire system and everything stays working with zero data loss". That's high availability to an extreme level. Can it be done? technically; it can - but at a significant loss to performance (meaning it will be slower). Plus, we are talking HUGE dollar amounts. The kinds that a small country would balk at. I guarantee in a week or two we will be having talks about "reasonable downtime" and defining what reasonable is - 1 second? 1 minute? 30 minutes? what's your pain?

All of this known downtime aside, the simple fact of IT is that you cannot plan for EVERYTHING. We test and test and test. As mentioned - we test according to the level of failure we are willing to accept. No matter how much we test, we CAN'T find everything in complex systems with sometimes hundreds of thousand or millions of lines of code and dozens of systems all coordinating activities. there are just too many interactions, too many unknowns, too many modeling problems compared to real world scenarios.

No one works in infinite budgets. We work in probabilities. We routinely make decisions about what costs we are willing to accept in terms of downtime to the cost of making something always work. A major factor in that is the criticality of the system in question. We accept less failure in systems that protect human lives (or put lives at risk if they fail) than we do on ones that are convenience items. There is far more risk in the breaking of a safety harness then in Elsa's arm waving about for example.

Entertainment systems are by nature not critical infrastructure. In the words of Jurassic Park, "The pirates don't eat the guests when the ride breaks down" (although it would make for a more interesting ride). It wouldn't surprise me in the least if Disney in fact had a formula for how much is costs them for a ride to break down for 1 minute, 5 minutes, 10 minutes, an hour, etc - both in the costs of operating numbers and in public relations. We do - and use it all the time to determine acceptable down times. (Somewhere in the accounting offices that I am not privy to, I am sure there are even numbers for how much a life costs for those critical systems).

Even with all that, we factor in that when we "go live", we will ALWAYS have failures. It WILL happen. The main thing we have to decide is what the level of probability of failure we are willing to accept is and to which systems. It's a difficult choice and I am sure that there were similar conversations had in Disney where they determined and accepted a level of risk given the potential costs involved. Make no mistake - Disney didn't do anything "improperly". They took a risk based on a statistic that they were willing to accept.

I don't question any of that. I'm just saying they should have opened without fastpass for a while, or certainly with fewer fastpasses in the mix.

 

[Stacy's a freak]

@TiggerBouncy, that really does put things into perspective and I appreciate it. Totally agree that this ride working or not works does not rise to the "life or death" risk so it makes sense to have some threshold minimum value (whether dollar or PR) to decide whether to roll out or not.

 

[CS Torkelson]

Hate to disagree, but that's just not true in the IT world. I just got finished talking to sales about a bid where a client wants "always up" which they defined as "they want to walk into the core room and unplug the entire system and everything stays working with zero data loss". That's high availability to an extreme level. Can it be done? technically; it can - but at a significant loss to performance (meaning it will be slower). Plus, we are talking HUGE dollar amounts. The kinds that a small country would balk at. I guarantee in a week or two we will be having talks about "reasonable downtime" and defining what reasonable is - 1 second? 1 minute? 30 minutes? what's your pain?

All of this known downtime aside, the simple fact of IT is that you cannot plan for EVERYTHING. We test and test and test. As mentioned - we test according to the level of failure we are willing to accept. No matter how much we test, we CAN'T find everything in complex systems with sometimes hundreds of thousand or millions of lines of code and dozens of systems all coordinating activities. there are just too many interactions, too many unknowns, too many modeling problems compared to real world scenarios.

No one works in infinite budgets. We work in probabilities. We routinely make decisions about what costs we are willing to accept in terms of downtime to the cost of making something always work. A major factor in that is the criticality of the system in question. We accept less failure in systems that protect human lives (or put lives at risk if they fail) than we do on ones that are convenience items. There is far more risk in the breaking of a safety harness then in Elsa's arm waving about for example.

Entertainment systems are by nature not critical infrastructure. In the words of Jurassic Park, "The pirates don't eat the guests when the ride breaks down" (although it would make for a more interesting ride). It wouldn't surprise me in the least if Disney in fact had a formula for how much is costs them for a ride to break down for 1 minute, 5 minutes, 10 minutes, an hour, etc - both in the costs of operating numbers and in public relations. We do - and use it all the time to determine acceptable down times. (Somewhere in the accounting offices that I am not privy to, I am sure there are even numbers for how much a life costs for those critical systems).

Even with all that, we factor in that when we "go live", we will ALWAYS have failures. It WILL happen. The main thing we have to decide is what the level of probability of failure we are willing to accept is and to which systems. It's a difficult choice and I am sure that there were similar conversations had in Disney where they determined and accepted a level of risk given the potential costs involved. Make no mistake - Disney didn't do anything "improperly". They took a risk based on a statistic that they were willing to accept.

We could be overthinking this whole thing. It's probably just those darn trolls playing pranks. They're famous for that.

 

[JimmyV]

Make no mistake - Disney didn't do anything "improperly". They took a risk based on a statistic that they were willing to accept.

Ahhh. But isn't that what they did improperly? They miscalculated the level of failure that guests would be willing to accept, and the reaction to that failure. Anyone who has been to WDW more than once has gotten stuck on Spaceship Earth. Or walked by Pirates when it was closed. Or heard the deafening silence when no Test Track cars whiz by. Those attractions all fail from time to time, and their failure rate must be within some range of "acceptable" or else there would be torches and pitchforks all over the Boards about these failures. So 99% of what you posted is true. Attractions will fail and Disney has to come to terms with what an acceptable failure rate is or will be. But with FEA, the failure rate (as measured in the pink in the posted chart) goes well beyond "normal" failure and well beyond the public's tolerance level of same. So Disney did one thing "improperly". It either rolled out the attraction with a gross underestimation of its actual failure rate, or, rolled it knowing its actual rate of failure, but grossly underestimating the reaction to that level of failure. So one of two things was wrong. Either their failure rate estimation was wrong, or the prediction regarding the reaction to the failure rate was wrong. They took a risk that the present failure rate would be overlooked, or at least accepted. They were clearly wrong about that.

 

[TiggerBouncy]

I don't question any of that. I'm just saying they should have opened without fastpass for a while, or certainly with fewer fastpasses in the mix.

Well - you know I can never disagree with my favorite cat.

In any case, I don't. Had I been running the thing, I certainly wouldn't have given out fastpasses until I had a reasonable acceptance level.

Sometimes these things are done for accounting reasons. We were forced to open a warehouse before it was ready because "as soon as we shipped one box, it was open" as far as the bean counters told us and you could begin to capitalize it. We literally shipped one order the first day.
The next day we shipped 2.

This is quite literally all we were able to force thorough the system. After the third day where we shipped 4 (we routinely shipped 2000 from our other facility, and this was supposed to handle the workload of 2 distribution centers totaling 3,000 orders / day) the CEO was quite upset and letting us all know it in a meeting. The CIO (same one from above) told him "we have consistently doubled our output for 3 days in a row. Most people would be ecstatic with those figures. What more could you possibly want?"

I really, really miss working for him.

 

[TiggerBouncy]

Ahhh. But isn't that what they did improperly? They miscalculated the level of failure that guests would be willing to accept, and the reaction to that failure. Anyone who has been to WDW more than once has gotten stuck on Spaceship Earth. Or walked by Pirates when it was closed. Or heard the deafening silence when no Test Track cars whiz by. Those attractions all fail from time to time, and their failure rate must be within some range of "acceptable" or else there would be torches and pitchforks all over the Boards about these failures. So 99% of what you posted is true. Attractions will fail and Disney has to come to terms with what an acceptable failure rate is or will be. But with FEA, the failure rate (as measured in the pink in the posted chart) goes well beyond "normal" failure and well beyond the public's tolerance level of same. So Disney did one thing "improperly". It either rolled out the attraction with a gross underestimation of its actual failure rate, or, rolled it knowing its actual rate of failure, but grossly underestimating the reaction to that level of failure. So one of two things was wrong. Either their failure rate estimation was wrong, or the prediction regarding the reaction to the failure rate was wrong. They took a risk that the present failure rate would be overlooked, or at least accepted. They were clearly wrong about that.

Actually, I tend to think it's us who miscalculate the level of concern Disney has for a few outspoken individuals. They have yet to make an "official" announcement - speculation waiting on the ride to become stable. They probably consider this a testing phase for the ride. Right now you are riding a ride that hasn't been "officially" announced as open. Reasonably, they can claim you should expect some rough edges.

The question is HOW MUCH IS THIS COSTING DISNEY?

it's a dollars and sense question. How many people have cancelled their trips because the ride shut down during an unofficial opening? Did anyone purchase less trinkets? Less Anna and Elsa gift shop items? They handed out a few fast passes. Guess what? Those don't cost them a penny.

If you asked the core of Disney Vacationeers (i'm not talking the .0001% of Disney fans who frequent these boards), I suspect few if any are concerned about a breakdown in that ride for the first 4 days of operation. I highly doubt it has affected their plans to attend Disney at all or had any considerable impact.

So I don't think they miscalculated at all. I bet the financial impact is exactly what they planned on it being. Immeasurably small.

Trust me - they are far more concerned about the effect of gators on their customer base than a few irrate posts on DIS about ride shutdowns on FEA.

 

[AngiTN]

Actually, I tend to think it's us who miscalculate the level of concern Disney has for a few outspoken individuals. They have yet to make an "official" announcement - speculation waiting on the ride to become stable. They probably consider this a testing phase for the ride. Right now you are riding a ride that hasn't been "officially" announced as open. Reasonably, they can claim you should expect some rough edges.

It's not? I could have sworn they gave an official opening date of 6/21, when they opened FP booking for it. No?
If it's not officially open now, what constitutes officially open?

 

[Luckybee]

One of the most impressive rides I've experienced anywhere was Escape from Gringott's at U. If you take a look at any U board you'll see that during the first few months of operation they had huge problems with this ride and eventually they got all the bugs out. I am far from being a Disney apologist but this is no different than when we put a new system in at our offices. Takes a while then everything runs smoothly. It isn't a "Disney" problem...its just the nature of IT.

 

[famy27]

Well, they "only" had what..... 18 months to retheme a ride. Maybe we all expect too much.

20+ months. You forgot to carry the 2

I have FPs for next Monday and very little confidence I'll be able to use them. Oh well, perhaps I'll get another ride on new Soarin instead.

 

[TiggerBouncy]

It's not? I could have sworn they gave an official opening date of 6/21, when they opened FP booking for it. No?
If it's not officially open now, what constitutes officially open?

I guess when I say "official" I mean they are not blogging about it, never held a release about it, etc. The release was cancelled because of the shooting.

Until Disney starts trying to hype it up - publishing blogs, etc - they are not really pushing the ride.

two sources:
"Disney and Comcast have stumbled in June, and they're being quiet about their shortcomings. Disney -- which is typically pretty chatty about new attractions on its official parks blog -- has yet to publish an entry about Frozen Ever After's opening. It could be waiting for the ride to operate reliably, but who knows when that will happen."

"Disney has yet to publish a full entry about Frozen Ever After's opening on it'sofficial blog, possibly because they're waiting for the ride to operate reliably. "

So in my view it's "Open", but they are not out there trying to sell the ride yet.

 

[Stacy's a freak]

I guess the dollars and cents aspect is what is peeving me the most. They knew that their attendance numbers would skyrocket once this ride opened. So maybe they opened it too early just to get the money in the door. Maybe they didn't really care of it operated properly if it got people in the park in droves. But there I go again speculating about Disney's reasoning.

 

[AngiTN]

I guess when I say "official" I mean they are not blogging about it, never held a release about it, etc. The release was cancelled because of the shooting.

Until Disney starts trying to hype it up - publishing blogs, etc - they are not really pushing the ride.

two sources:
"Disney and Comcast have stumbled in June, and they're being quiet about their shortcomings. Disney -- which is typically pretty chatty about new attractions on its official parks blog -- has yet to publish an entry about Frozen Ever After's opening. It could be waiting for the ride to operate reliably, but who knows when that will happen."

"Disney has yet to publish a full entry about Frozen Ever After's opening on it'sofficial blog, possibly because they're waiting for the ride to operate reliably. "

So in my view it's "Open", but they are not out there trying to sell the ride yet.

Ah, ok. I'm not a blog follower so I wouldn't notice what they did there.

 

[mom2rtk]

20+ months. You forgot to carry the 2

I have FPs for next Monday and very little confidence I'll be able to use them. Oh well, perhaps I'll get another ride on new Soarin instead.

Dang Disney math!

I'd say you have a far better chance of riding than your counterparts trying to find the standby line open!

 

[TiggerBouncy]

I guess the dollars and cents aspect is what is peeving me the most. They knew that their attendance numbers would skyrocket once this ride opened. So maybe they opened it too early just to get the money in the door. Maybe they didn't really care of it operated properly if it got people in the park in droves. But there I go again speculating about Disney's reasoning.

Yup, and that's kinda what I am saying. From a financial perspective (and lets be honest that's what a company really cares about), I doubt Disney did anything "incorrectly".

This opening is probably considered "a win" to the bean counters and therefore to the brass. It raises guest levels and it can officially be capitalized (which is crucial).

The only place that I think they absolutely messed up as far as they are concerned is in it is the handing out of fastpasses. I agree with mom2rtk there.

 

[MichiganScott]

For a moment there I thought this was the "Name your favorite first world problem" thread.

 

[mom2rtk]

For a moment there I thought this was the "Name your favorite first world problem" thread.

That could probably apply to every single thread on a Disney message board.

 

[Jennasis]

Disney can't even keep barges afloat lately. I'm quite concerned about PAndora and Star Wars meeting any potential deadlines and actually working if they do.

 

[Angel Ariel]

A couple of people I've talked to who worked on or around the construction of FEA have said that it needed another 3 or 4 weeks of work, plus time for a real CM testing/soft open period. But, they didn't get that, so right now is basically it.

So very unfortunate.