Just to clarify, the 2013 F&W festival ends 11/10, so the OP won't be catching much of the festival.
We did the week after Jersey week 3 years in a row. We always arrived on Thursday and stayed through the following week, so about 10 days, and we would usually not do much the first few days until the crowds thinned after Veterans Day weekend. That weekend (the weekend you arrive) is always busy b/c of Veterans Day, the end of F&W, and the end of Jersey week. You'll see a huge change in crowds even around the hotel from the 9th to the 11th. I will caution that F&W will be MOBBED on the last Saturday (9th), stay away. However, the 10th is the true last day of the festival and we have attended on that Sunday evening in the past and found it tolerable.