I like to give piggy banks! Buy an inexpensive one, and personalize it by writing the child's name with a permanent marker or paint pen. Stick a $5 bill in it, and consider it your contribution to the development of good money habits!
If you decide to go the "piggy" bank route, try to buy one that's not overly "cutesy" and babyish as it will hopefully be around for many years to come. Of course, it doesn't have to be a pig either - DD12 has one in the shape of Noah's Ark, DD18 still has the clown that my mom bought her when she was born. We had a break in a few years ago - the thieves took the "obvious" coin banks (i.e. the pigs!) but the non-pig designs that just looked like figurines, remained untouched!