I will qualify my response by saying we are huge Poly fans and have not stayed at CR (it just doesn't ring our bells for some reason - we have visited and eaten at Chef Mickey's, just not stayed).
It really comes down to what is more (or most) important to you. Walking to the park? TPV room? Saving money?
That is a pretty good savings and I would be tempted to switch it...but if you have your heart set on being able to walk to MK, it might not be worth it to you. As far as distance from the park, yes, CR is closer even via monorail, but we have made it from Liberty Square back to our room at the Poly in about 20 minutes (via monorail) - not too shabby if you ask me. (We did have some monorail mojo that evening and didn't have to wait long at the platform for it).
We stay TPV at the Poly, and we love being able to sit on our balcony and watch the activity on the lagoon and the fireworks and EWP at night. The fact that you are switching from TPV to Garden View might be the deal breaker for me. We did NOT switch from Poly to Boardwalk in May, even though it would have saved about $1000 and we would have been closer to DHS for Star Wars Weekends. I just couldn't give up the ambiance of the Poly.
And just to counter-balance the Mousekeeping naysayers, we have stayed at Poly twice in the past year and have not had a bad Mousekeeping experience at all. I'm sure it happens, but it is not something unique to the Polynesian and I wouldn't switch resorts (or not switch) based on the likelihood of that happening. (We also like CBR and hear frequently about poor Mousekeeping there, but never experienced it first hand).
Whatever you decide, I hope you have a great trip - both resorts are magical in their own ways and I'm sure you'll have fun either way!