Why I Am Skeptical of the Touring Plans Website DLR Crowd Calendar

[HydroGuy]

I'm an engineer and a lover of math, so I appreciate your study of this. You have unquestionably proven that a TP 9 and 10 cannot mean what they have defined it to mean. However, you admit that some of those days that were labeled as 9's or 10's were probably 8's. If I'm planning, I really only care about large differences between say a 3 and an 8. I really don't care about small differences like an 8 vs. a 9. If they are getting within 10% of the right number, I still say there is value in that. Nobody is going to say, "I'm was going to go to the park if it was an 8, but I'm staying home because it's a 9."

You also don't really provide any proof that the data isn't correct with respect to relativity on a short term. For instance, I'm going to be in SoCal for 10 days, but only going to DL for 5. To me, I care more about which of those 5 are the lowest 5 days, and less about whether those days are 5's, 7's or whatever. If they help me correctly choose the best 5 days, the site has done it's job.

Now while you haven't disproved these things, I suppose they also haven not been proven true by any study either. It's kind of a hard thing to quantify either way, given that Disney doesn't release turnstile numbers and ride wait times for certain rides can fluctuate for lots of non-attendance reasons. Also, given that you proved there was one flaw in their method, one could assume the rest of their analysis could be equally flawed. So I guess my takeaway is that your finding is interesting and there is reason to be skeptical of TP's numbers. But given few if any other superior predictive sites out there, I would still use them over going blind, especially for someone like me who does not have the number of trips under his belt like you do.

Thanks for the feedback.

I guess I am saying a couple things with this thread...

If they got this most fundamental part of their prediction wrong then it shows a disturbing lack of rigor that to me calls everything into question they say.

It causes an emotional response in folks that is disproportionate. People are scared by 9's and 10's and maybe say "wow, my trip will be miserabe" or "there is no way I will go to DLR on those days". Even if we assume that TP is relatively correct (e.g., a 9 day is worse than a 7 day even if it is not really a 9 or a 7 because of their lack of rigor) the fact remains that some of the days they are calling 9 or 10 must be a 7 (because 33% of the days are 9 or 10 days when only 20% can be - so some of those must be 7's and 8's even when they say they are 9 or 10). In my book a 7 is not that bad. But a 9 or 10 sounds really packed. That is a huge flaw.

Folks on this forum use their dates to try to avoid 9 or 10 days (by TP account) and feel good about themselves when they can plan their vacation for 6 or 7 days (by TP account). But what is the impact if the days they avoided and thought were 9 or 10 but were in fact a 7 - but maybe with long park hours 8AM-Midnight at DL - and instead went on a TP 7 ranked day with park hours of 9AM-10PM? They traded barely lower crowds for a day with much lower park hours. They got messed up by TP big time.

 

[thornton0511]

Yeah, I really wish there was a west coast equivalent of EasyWDW. And Kennythepirate for character hunters.

This
This has been my #1 frustration with planning our first trip to DLR. We have been so spoiled by the abundance of wonderful resources for WDW that the lack of them for DLR is leaving me feeling....well

 

[pixar mark]

duplicate post

 

[pixar mark]

Good points Hydroguy.

Not to defend TP or do their explaining for them, but I wonder if ties has anything to do with it. Obviously if we had exact turnstile numbers, we would find that likely no two days in a year would tie in attendance, and so it would be easy to split them up into 10% subgroups. But if I'm not mistaken, the way TP measures attendance is, in part, by historical wait times. Therefore, there are lots of days that could vary by a few thousand people that all end up having 60 minute waits on Space Mountain. If that is the longest wait time of the year, you have no choice but to call all of those days 10s even if it accounts for 15% of the days and 5% of those days technically have fewer people in the parks. Obviously they take into account more than SM ride times, but you get my point. I'm not saying that's what's going on here, and it also doesn't necessarily explain your experience with seeing crowds that don't match what you think the number would be. But remember that they really are predicting wait times more than actual crowds, even though they call it a crowd calendar.

 

[YYCmom]

I just discovered Touring Plans this year. I've had a lot of fun creating custom touring plans for my family (parents, spouse and myself, and our kids). It's helped me think about what order I'd like to experience attractions in and when/where we should split up. For example, my 8 year old loves rides that spin and go fast, where my dad and my 4 year old absolutely won't get on. So I can think about California Screamin' paired with the carousel, etc. And try to work my 4 year old up from the tamest rides to the somewhat less-tame.

I'm not trying to go with the most efficient plan, but find one that will work for my family. We'll see whether or not my family will go along once we get there. And also I'll find out if TP's estimates of how long the touring plan will take are anywhere close to accurate.

 

[figment_jii]

I was thinking a little about this and out of curiosity I looked at a couple of days in detail. They've begun posting predicted-versus-observed comparison on the Touring Plan Blog so you can get a sense of how accurate they are. I'm not sure about the marble description anymore because it seems like they base their estimates on predicted daily average wait times for certain attractions (and other things, but they aren't specific about what those are). I looked a couple of the attractions they said they used in their predication and in many cases just a minute or two difference in terms of the daily average wait time changed the crowd calendar predication for that attraction. For example, if Space had a daily average wait of:
39 to 42 minutes, then it's a crowd level of '5',
44 minutes, then it's a crowd level of '6'
52 minutes, then it's a crowd level of '8'
So a daily average wait time increase by 10 minutes jumps the calendar from a '5' to an '8' for Space.

I just thought it was interesting how sensitive the crowd calendar appears to be to slight changes in daily average wait times.

 

[T0rrent]

Hydroguy, do you ever use the Mousewait lines app? Wife and I have used it for the last two years and while the results can be skewed by someone trolling the line estimation, it seems to also have a good current crowd index based off of wait times. We were in the parks last week and when it said the index was in the 60's (out of 100) it really felt it, and over the weekend when it said mid to high 70's, it also did feel like that.

 

[sunflare]

I use Lines for the touring plans more than anything else. I do have a lot of faith in their crowd predictions, though...they were accurate (to ME) about 90% of the time. I think one of the issues they might have with the predictions is the abundance of fickle locals...it's hard to predict crowds when so many of those who visit only come for a few hours at a time.

 

[lindyv321]

I used TP for my last trip in 13 and loved it! The crowd calendar was great and I felt like the lines app was pretty spot on. I picked the dates for my upcoming trip based on the crowd calender and I look at the wait times app a few times a day to check out wait times vs the crowd calender number. I don't expect them to be 100% accurate since they are basing a lot of the info on people actually logging correct wait times etc.

 

[HydroGuy]

I was thinking a little about this and out of curiosity I looked at a couple of days in detail. They've begun posting predicted-versus-observed comparison on the Touring Plan Blog so you can get a sense of how accurate they are. I'm not sure about the marble description anymore because it seems like they base their estimates on predicted daily average wait times for certain attractions (and other things, but they aren't specific about what those are). I looked a couple of the attractions they said they used in their predication and in many cases just a minute or two difference in terms of the daily average wait time changed the crowd calendar predication for that attraction. For example, if Space had a daily average wait of:
39 to 42 minutes, then it's a crowd level of '5',
44 minutes, then it's a crowd level of '6'
52 minutes, then it's a crowd level of '8'
So a daily average wait time increase by 10 minutes jumps the calendar from a '5' to an '8' for Space.

I just thought it was interesting how sensitive the crowd calendar appears to be to slight changes in daily average wait times.

I have been paying more attention to TP lately in connection with a May trip to WDW. In my little bit of reading there are some who are calling into question their predicted vs. observed claims. So not everyone is sold on them at WDW either.

Regarding your comment about slight changes in daily wait times, those slight changes mean thousands of people of extrapolated across the entire park - assuming other rides are also 20% longer wait times too. I personally would expect a "5" day "compared to an "8" day would have 20% more crowds. That seems reasonable to me.

After I spent time in the OP I have not studied it anymore so your further study is interesting.

 

[HydroGuy]

Hydroguy, do you ever use the Mousewait lines app? Wife and I have used it for the last two years and while the results can be skewed by someone trolling the line estimation, it seems to also have a good current crowd index based off of wait times. We were in the parks last week and when it said the index was in the 60's (out of 100) it really felt it, and over the weekend when it said mid to high 70's, it also did feel like that.

My kids use MouseWait and I use Lines from TP just because I like how it works. Usually they are not too different.

 

[CalgaryTroy]

HG - I LOVE your analysis! thanks for doing all the heavy thinking
I'm curious if there's a disproportionate number of days that are 9s and 10s - what about the frequency of 1s/2s & 3s - is there such a thing? how often are they? and WHEN are they?

 

[lindyv321]

We went to the parks at the end of September in 2013 in the middle of the week and those days were listed as 1's and 2's on the crowd calendar. That translated to us walking on the majority of the rides, not standing in crazy lines for food, etc. In fact I think the longest line we stood in was 25-30 minutes. Downside is that there were things closed for refurbs, the park hours were shorter, and not much in the way of nighttime entertainment offered. We will be back in a few weeks and the days are listed as 3's and 4's but per the app which I have been checking the wait times are not all that bad on days they predict as 3's and 4's. I suppose we'll see once we get there.

 

[BobearQSI]

A number of regulars around here have expressed some skepticism of the TP crowd calendar for DLR.
...
So TP is projecting 56 total days this year as days of 10 level crowds. How do they define a 10? Here is what they say:

If you sort the marbles from smallest to largest you can divide them into ten groups of ten, according to size. The largest ten marbles get put into Group Ten. The next ten largest get put into Group Nine, etc.

What that means is that with 365 days in the year there will be 36.5 days ranked as a 10, 36.5 days as a 9, and so on. In other words 10% of all days in a year will be a 10 and 10% a 9.

But TP has 56 days ranked as a 10 - or over 15% of all days at DLR.

I know I'm bringing up an old thread here, but being a math person myself, I have to point out that this is not what the analogy is describing. Although I do agree it has been worded very poorly. It should have been clarified like this - when they say sorted into ten groups, they don't mean total number of marbles divided by ten, they mean the 'diameter' of the largest marble divided by 10, then fit the marbles into their diameter groups. Some groups may have a lot more marbles, but for some reason they decided to normalize to 100 in 10 groups in their bad analogy. They are grouping the crowd levels, not the number of days of the year.

Here's a crude example: Say you had 356 days of the year with the crowds being 90,000-100,000 people (bear with me here). The next 9 days the crowds are 80k-90k, 70k-80k, 60k-70k, etc. Each group gets a number, so 356 days the crowd level is 10, the remaining days are 9, 8, 7, 6, etc.

Another example: Days 1-100 are 90k-100k, day 101 (a single day) is 80k-90k, days 102-150 are 70k-80k, days 151-180 are 60k-70k and so on. The resulting groups are exactly the same. Days 1-100 are a 10, there is only one day a year (day 101) with a level of 9, etc.

To summarize, the 100 marbles don't represent days, but crowd levels. 91k people, 92k people, 93k people, to 100k people are the '10 largest marbles.'

So a daily average wait time increase by 10 minutes [on Space Mountain] jumps the calendar from a '5' to an '8' for Space.

Going strictly by their numbers, this could very well be correct. You also have to consider their wait times are averages only from a period in the middle of the day. And, staffing levels and number of ride vehicles running for different crowds - take a look here: http://touringplans.com/disneyland/attractions/space-mountain - a 4 is actually a smaller average wait than a 3!

----------------

Okay, all of that being said, I myself have become very skeptical of TouringPlans crowd calendar after subscribing for an upcoming trip. They talk about having professional statisticians and all that, which may work well for WDW, but doesn't seem to work as well for DLR. While I believe all of their math is sound, It seems to be seriously lacking a human element and/or adaptability. Case in point is the Diamond Celebration. It seems be attracting a slightly larger than normal crowd. Looking over the past couple months of crowd calendar data, they appear to constantly be underestimating the crowd sizes. Just the past two weeks have an almost daily underestimation of 2 levels - 7's turning out to be 9's, 8's being 10's, etc. Yet, so far, I haven't seen any of the future days for the rest of the summer change.

Another issue that really bugged me is their historical calendar - the month view shows their historical predictions, not the actual crowd levels measured on those days. You have to go into each individual day in order to see the 'what we saw' numbers. The example here is Labor Day of last year. Everywhere else you read online, Labor Day is busy with the half marathon and locals being out of school. But last year, and as seen on their historical calendar month page, Labor Day was predicted to be a 2 out of 10 for Disneyland. Click through to the day itself, and you find it was actually a 6.

Worst of all, with my paid subscription and these statements on their web page: "got a question about a specific day ... We're happy to help! Please Contact Us" and "Want to ask any questions directly? Contact Us!" I decided to send an e-mail. I had very specific questions about why their historical calendar shows predicted instead of actual, pointed out the large error last Labor Day, pointed out the Half Marathon, and was very specific that I meant Disneyland (even mentioned California Adventure) and included direct links to the calendars and pages in question.

The response I got? A single sentence about a link to a blog post of theirs that described how school is starting earlier than normal this year (in respect to Labor Day) so Walt Disney World vacationers would be far less on Labor Day.

----------------

There, I feel better now

I got Touring Plans because I needed to make a booking decision quick, and figured I'd pay a little bit of money (used a discount, it was like $7) for their seemingly rigorous mathematical models and staff statisticians. I'm a low season person, and after using it and watching it for awhile, I'm just really hoping they aren't too wrong on their medium-low crowd predictions for Aug 25-29.

 

[jeffg4]

Good points Hydroguy.

Not to defend TP or do their explaining for them, but I wonder if ties has anything to do with it. Obviously if we had exact turnstile numbers, we would find that likely no two days in a year would tie in attendance, and so it would be easy to split them up into 10% subgroups. But if I'm not mistaken, the way TP measures attendance is, in part, by historical wait times. Therefore, there are lots of days that could vary by a few thousand people that all end up having 60 minute waits on Space Mountain. If that is the longest wait time of the year, you have no choice but to call all of those days 10s even if it accounts for 15% of the days and 5% of those days technically have fewer people in the parks. Obviously they take into account more than SM ride times, but you get my point. I'm not saying that's what's going on here, and it also doesn't necessarily explain your experience with seeing crowds that don't match what you think the number would be. But remember that they really are predicting wait times more than actual crowds, even though they call it a crowd calendar.

That indeed is what's going on. TP's crowd calendars are based on average historical wait times, so the calendars are based solely on wait times, not the number of people in the parks. You can see the historical charts if you go to the historical crowd calendar, click on the day, then click on the park. They will show each ride's wait times on one side of the y-axis and the corresponding "crowd levels" on the other side. So that's why you don't always get 10% of the calendar days to fall within each of the ten different crowd level rankings. They assign a particular "crowd level" to a corresponding attraction's wait time. Then it appears like they average all of the attraction "crowd levels" to get an overall park crowd level for the day. It's hard to tell if the overall crowd level ratings are weighted in any fashion (i.e. the more popular attractions get more weight).

Now here's another problem. While in general, attendance is generally reflected in average wait times, that is not always the case. As mentioned previously, you have to take into account park hours and staffing. With a certain number of people entering the park on a normal winter day with shorter park hours, it will "feel" more crowded than the same number of people entering on a normal summer day with longer hours. Now throw in another wrench - park staffing. We all know that many attractions do not operate at full capacity during lower peak crowd levels. So you may be waiting in line the same amount of time for a ride on a "slow" winter day than on a "busy" summer day even though there are less people in the park.

And yet another problem with TP predictions...The predicted wait times rely on user submittals, so they are only as good as what's being submitted. At 4:45 PM on a day 2 weeks ago at DCA, the Lines app showed a FP return of 7:50 PM for TOT. When I got to the FP machine at 5:00 PM, they were gone. The CM stated there were a reduced number (4, I think) of elevators running throughout the entire day, which raised the waits. So for that day, the predictions were way off, I'm guessing, due to the lack of accurate submitted wait and FP return times.

That all being said, Hydroguy's assessment of the rigor (or lack thereof) with TP is not unfounded. I uncovered many simple inconsistencies when I was researching things a couple of years ago - the main one being that wait times in my TP did not even match their own published wait times for a particular attraction based on the predicted crowd level. In other words, if TP showed a 45-min wait at 3:00 PM on their published Matterhorn "wait times versus crowd level" graph on a CL 8 day, a completed TP would give me a different wait time at 3:00 PM for Matterhorn on a CL 8 day for which I was creating a plan. This year, a generated TP had me pull a FP at 8:50 AM with a return time of 10:00 AM, then had me pulling another FP at 9:40 AM, which is not possible. I emailed and got a response stating "that should not happen" and they were working on a fix, but something as basic as this should not have been an oversight in the software algorithms. By only digging a little into the details, I was able to uncover multiple flaws and inconsistencies.

So, in summary, take the crowd calendars and TPs for what they're worth...a general guideline that gives you a basis from which to plan. Nothing more.

 

[Winnowill]

What's annoying me most is that TP seems to arbitrarily be taking attractions out of circulation. For instance, I went in to adjust my plan for 8/10 today, only to find that I couldn't because, apparently (unbeknownst even to Disney themselves), Autopia is closed until 12/31/16. Really? So any plan that includes it cannot be optimized or evaluated. They did this with the Flag Retreat a few weeks ago, and still haven't put it back on the list.

 

[tarheelalum]

Touring Plans or any other similar website is a waste of money. Anyone who regularly attends the parks can far better tell when the parks are going to be crowded or not so crowded.

 

[texasgingerbread]

Can anyone recommend a site other than TP for DL crowds? Are there other crowd calendars out there? I'm not finding much. So different than WDW I've got to retrain my brain. Looking at first week in March 2016. Specifically flying in Tuesday night and doing Wednesday - Sunday but could change flights and fly in and do Tuesday-Saturday. Thoughts?

I too wish there was a kennythepirate site for DL that was character and touring plan intensive. I ran across RideMax...?

 

[jeffg4]

RMH Travel has a crowd calendar at the following link. Ridemax does not have a crowd calendar for DLR.

http://www.rmhtravel.com/disneyland-365-crowd-calendar.html

 

[BobearQSI]

RMH Travel has a crowd calendar at the following link. Ridemax does not have a crowd calendar for DLR.

http://www.rmhtravel.com/disneyland-365-crowd-calendar.html

Strangely, they have 2 predictions that don't match each other. There's the predictions on that page, and then predictions on the monthly calendar:
http://www.rmhtravel.com/disneyland-aug.html

They aren't the same numbers For example, Friday, August 28th shows a 7 on one and a 3 on the other.