We were there about two weeks ago, and did this several times. However, I wouldn't suggest it with a new ride, or something very popular during peak season.
With that said, we went through once where the cast member aske us jokingly "Well, where have you been, we were waiting for you?".
And a second time, I said "We're a little bit late!" to the CM as I was handing them the fastpass, and he replied "Not to worry. You can always be late. Never early, but late is fine."