Sand Lake Road is basically the dividing line. I-Drive south of Sand Lake is quite nice, where a lot of upscale hotels have popped up around the Convention Center. Further down past Sea World (where I live), it is less developed and still quite pleasant.
North of Sand Lake is where you find all the tacky gift shops, run-down hotels and rip-off "amusement parks" (I mean that in the loosest sense of the term, they are basically permanent carnivals). Other than Sweet Tomatoes and the area surrounding Festival Bay, there is nothing worth seeing up that way, IMO. While it's not the worst neighborhood in America, it is dicey after dark, especially compared to the level of security on Disney property. If you find yourself up there, lock your doors, be aware of your surroundings, and use common sense.
Neither half offers a whole lot for toddlers, getting back to the OP's question.