May is actually more refurb heavy in my opinion since it sits between spring breaks and the summer crowds/grad nights so they try to get stuff done in the lull between those. November doesnt have always ton of refurbs but small world will be down the first week for holiday overlay and space mountain will down for the removal of its halloween overlay. Your 5 year old might be able to ride this but your 2 year old won't be able to but even if you dont want to ride it having it down add to a more crowded feeling since the people would have been lined up to ride Space will now be elsewhere making things more crowded. Also the jungle cruise goes down for its jingle cruise overlay.
Might I actually suggest the second week of November (but going home prior to the avenger half marathon) as opposed to the first-if your trip would start with the 1st or the even october 31 it will be more crowded with halloween crowds. People love Mickey's Halloween party and it does draw crowds (myself included) and you might get to experience the rides with holiday overlays as they reopen.
November has christmas decor and special Christmas entertainment like holiday parades, fireworks, and a special version of World of Color-I personally have always wanted to see these thing but never have since I could only go during the schools winter break which everyone knows is crazy crowded. May doesn't have seasonal decor or special entertainment and may even have a limited entertainment schedule if it follows with what Disney has been doing post 60th (october only had a night parade and longer, more involved fireworks during the special ticketed parties other nights had a very short fireworks display lasting only a few minutes and only a daytime parade) but you may also get to see the main street electrical parade since its coming back for a limited time-I'm not sure if we know when "a limited" time is though?
May could have fabulous weather but it could also be rather hot, whereas November would definitely be chilly (but not cold) with a chance of rain. But I love it when it rains at Disneyland ( more than a sprinkle but a true rain shower) since people often leave the park and I feel like it is the least crowded I'll get to see it. I had a clear rain cover for my stroller when my son was young and he stayed quite warm, dry and happy under it and I packed real rain stuff (no flimsy ponchos) for my own comfort like a raincoat and rain boots and an umbrella that clips to the stroller.