IMO best not to engage someone like that. A previous poster was right, a noncommittal okay would serve well here.
A neighbor across the street from us has white stones along a kneewall. The squirrels bring them into mine and the other neighbors' front and back lawns and flower beds.
Sounds like he takes a lot of pride in keeping the landscaping looking a certain way and it's important to him. Maybe he misses having a garden of his own, I know I would if I moved into a condo. It's stinks that he didn't ask nicely but I think I'd accommodate him just the same seeing as it's not a huge inconvenience to do so. You never know when you may need a friendly neighbor on your side, especially in a condo with an HOA.