You can purchase them at your local bank. I recommed getting the EE bonds (for $50, you can get a $100 bond) or, if you want value for value, you can get an I bond ($50 = $50).
All you need is the application, which they will have at your bank. You can either obtain the child's social security number or use your own, it doesn't matter. You do need to have the right mailing address for the child, however, so make sure you have that on you. You can also name one of the parents as co-owner if you so choose. I usually do this for little ones, just in case - it does not hurt their ability to cash it later on. Then, you pay for the bond, they process it in their computer, and the bond is mailed to the address you provide. Ask for a gift certificate that you can put in the card so you can let the owner know there is a bond coming in the mail. It usually takes 2-4 weeks for the bond to be received by the owner. HTH!