I definitely think you should go. There are so many awesome rides and shows and fun things to do that skipping a handful of them shouldn't be a factor. They are at an age where they will enjoy the fantasy and theme of it so much.
Personally, I agree with the other poster who said that she wouldn't take a four year old on most of those attractions anyway, even if he did meet the height requirement.
Some rides have a 38" requirement, some have 40", some even 48", so it's not always the same. Here are the requirements- hope it helps you choose what to do and what not to do, but I definitely think you should go!
Magic Kingdom Park
Goofy's Barnstormer - Must be at least 35" height requirement
Big Thunder Mountain Railroad - Must be at least 40 inches tall to ride. Warning.. For safety and comfort, you should be in good health and free from heart, back or neck problems, motion sickness, or other conditions that could be aggravated by this adventure. Expectant mothers should not ride.
Space Mountain - Must be at least 44 inches tall to ride. Children under 7 years old must be accompanied by an adult & ride in their own seat (no longer seats two people together). Warning.. Guests with health restrictions and mothers to be may wish to bypass the ride.
Splash Mountain - Must be at least 40 inches tall to ride. Warning.. Guests with health restrictions and mothers to be may wish to bypass the ride.
Tomorrowland Speedway - Must be at least 52 inches to ride alone. Warning.. Guests with health restrictions and mothers to be may wish to bypass the ride.
Toontown Playground - Maximum height 40"
Epcot
Soarin: Must be at least 40" minimum to ride. Warning! For safety, you should be in good health and free from high blood pressure, heart, back or neck problems, motion sickness, or other conditions that could be aggravated by this adventure. Expectant mothers should not ride. Young children should be accompanied by an adult. Guests in wheelchairs must transfer from their wheelchairs to experience this attraction.
Mission Space: Must be at least 44" minimum to ride. Warning! For safety, you should be in good health and free from high blood pressure, heart, back or neck problems, motion sickness, or other conditions that could be aggravated by this adventure. Expectant mothers should not ride.
Body Wars (Wonders of Life) - Must be at least 40 inches tall to ride. Warning.. Guests with health restrictions and mothers to be may wish to bypass the ride.
Maelstrom (Norway) - Minimum age 3 to ride. Warning.. Guests with health restrictions and mothers to be may wish to bypass the ride.
Test Track - Must be at least 40 inches tall to ride. Warning.. For safety, you should be in good health and free from high blood pressure, heart, back or neck problems, motion sickness, or other conditions that could be aggravated by this adventure. Expectant mothers should not ride.
Disney-MGM Studios
Backlot Tour (Catastrophe Canyon) - Minimum age 3 to ride. Warning.. Guests with health restrictions and mothers to be may wish to bypass the ride.
Great Movie Ride - Special effects may be too intense for children and some adults
Rock 'n' Roller Coaster Starring Aerosmith - Must be 48 tall. High Speed (3 inverts) Coaster. Warning! For safety, you should be in good health and free from high blood pressure, heart, back or neck problems, motion sickness, or other conditions that could be aggravated by this adventure. Expectant mothers should not ride.
Star Tours© Disney/Lucasfilm, Inc. - Must be at least 40 inches tall to ride. Warning.. Guests with health restrictions and mothers to be may wish to bypass the ride.
The Twilight Zone Tower of TerrorTM - Must be at least 40 inches tall to ride. Warning...Cannot suffer from any neck, back, or heart problems. Cannot suffer from motion sickness or claustrophobia! Wheelchair guests must be able to transfer to ride vehicle. Pregnant women may not ride.
Animal Kingdom Theme Park
DINOSAUR - Height minimum is now 40 inches. Warning! For safety, you should be in good health and free from high blood pressure, heart, back or neck problems, motion sickness, or other conditions that could be aggravated by this adventure. Expectant mothers should not ride. May be too Scary for younger children
Its Tough to be a Bug! - May be Scary for younger children
Kali River Rapids - Must be at least 38 inches tall to ride
Primeval Whirl - Must Be 48 Inches Tall
DisneyQuest Indoor Interactive Theme Park
Buzz Lightyears AstroBlasters - Must be at least 51" tall to ride
CyberSpace Mountain - Must be at least 51" tall to ride
Mighty Ducks Pinball Slam - Must be at least 48" tall to ride
Note: For the safety of all guests, especially our children There are rules for the Water Parks they are as common sense as our back yard pool rules.
Disney's Typhoon Lagoon Water Park
Crush 'n' Gusher: Must be at least 48" tall to ride.
Castaway Creek (continuous river raft ride) - Adult supervision is required
Humunga Kowabunga (speed slide) - Must be at least 48" tall to ride
Ketchakiddee Creek (childrens area) - Must be
under 48" tall to experience
Mayday Falls (raft ride) - Must be at least 48" tall to ride
Shark Reef (saltwater reef environment) - Children under age 10 must be accompanied by an adult. Mothers to be may wish to bypass the ride and those with neck or back problems are strongly discouraged from experiencing any water slide attraction.
Typhoon Lagoon (wave pool) - Adult supervision is required
Disney's Blizzard Beach Water Park
Blizzard Beach Chair Lift - Must be at least 32 inches tall to ride. No infants or lap-riding
Cross Country Creek (continuous river raft ride) - Adult supervision is required
Downhill Double Dipper (mat slide) - Must be at least 48" tall to ride
Slush Gusher (speed slide) - Must be at least 48" tall to ride
Summit Plummet (speed slide) - Must be at least 48" tall to ride
T-Bar (in Ski Patrol Training Camp) - Must be
under 48" tall to ride
Tikes Peak (childrens area) - Must be
under 48" tall to experience