You certainly can. The Outer Rim offers a pretty good goat cheese spread, as well as the usual lounge fare like chicken wings, etc. It starts serving these at 5 pm, when the Concourse Steakhouse (from whose kitchen the OR gets its food) opens for dinner.
The Cali Grille's lounge offers sushi, but I'm not entirely sure.
I like the Outer Rim, and when we stay at the Poly or Grand Floridian we always make it a point to monorail over to have a quiet afternoon drink. It opens pretty early (1 pm) and is deserted for most of the afternoon.