We just returned last week from a Thursday-Monday trip for the first weekend. Our main goal was to only see a band that was performing at the Flower Power concerts (also free during your trip to Epcot). That didn't quite work out the way we wanted (the band performed the same songs for each first set, the second set, and the third set each night so multiple performances were not in the picture). We are not gardeners since our backyard kills anything but trees.
However - we had a great time! We didn't partake in too much of the food offerings but enjoyed the topiaries and the special displays all over World Showcase.
I would not hesitate to go back again during the Festival, gardener or not.