How badly do you want to stay in your room? Did you HAVE to book preferred for the second two legs of your stay? It you could switch it all to garden view, I think the CMs would try their best not to move you.
If you really want preferred, my guess is that you will have to change rooms once, switching from garden view to preferred. That first part of your stay is Memorial Day weekend, so I would imagine it is pretty booked up. I will be there at the same time, and couldn't get preferred spring discount for my whole stay, but was able to get river view, so I was happy.
But who knows, you might get lucky! Request to move as little as possible.
And no, you are not crazy! I think that is a decent plan, given the situation. I have a family of five too. You gotta do what you gotta do!
ETA... And maybe your garden view room will end up being great, and you will ask to stay there the whole time.