When you're considering an item, ask yourself if it was available 50 years ago. If not, ask yourself whether it's genuinely an improvement for you or the baby (i.e., car seats) or whether it's just "more for the sake of more".
Research carefully and spend heavily on the things you will use long term, possibly for other children in the future:
- Crib
- Car seat
- Stroller
- High chair
And skimp on pretty much everything else. Except photographs and books. Don't skimp on those items.
The one thing I'd suggest buying -- though it isn't a budget item -- is a nursing pillow. You'll spend HOURS holding the baby and feeding him or her, and a nursing pillow saves strain on your back muscles, making the experience of feeding the baby so much easier. With my first child I kept a bed pillow on the sofa all the time, but with my second I found a nursing pillow at a yard sale and bought it. The smaller, shaped pillow was so much more convenient.