When I started my niece on scrapbooking, I bought a 8 x 8 pad of coordinating paper from JoAnn's that was on clearance (you can also look for coupons, etc.) Then, a couple colored markers and some adhesive. I cut some of the paper down to matting size for her with my trimmer, and otherwise she had a pair of more kid friendly sissors. I also bought an 8 x 8 album on clearance.
I think the total cost - pad, album, markers and adhesive was under $15 (this was 5 yrs ago, but I don't think prices have changed that much). It took her about 6 - 8 months to fill her album, and I bought her more things on clearance over time once I realized she would keep working on it. She's now 14, and even so, I generally fund her scrapbooking with my "extras" and a few basic supplies.
If only I could be more budget with my own stuff...