The birds WILL NOT abandon it because of human scent. We just had a couple of houses finches build their nest and hatch eggs in a hanging basket of ivy. I had to water it every few days so I'd get the ladder out and water it-avoiding watering the nest/eggs. There were 7 eggs-5 hatched. Then last weekend while we were sitting in our porch (it's open) a hawk flew in and took the babies. We didn't hear a thing-except the parents ranting. So I got up and looked-the basket was swinging-the babies gone

I looked out on the driveway and there was a baby-still alive. I picked it up,put it back in the nest and turned the basket around so the hawk couldn't see the nest and wouldn't be brave enough to fly into the porch to try again. The parents came back, fed it all week and voila it flew away a couple of days ago to a nearby tree. I think the parents are teaching it to fly and eat. I see the parents (at least I think they're the same ones) still gathering food and twice I've seen the baby up in a tree.
So best advice is to leave it alone if you can-from nest building to fly away was almost a month. Then when they leave remove the nest and discourage them from rebuilding somehow. Bluebirds are smaller than blue jays. Buebirds have red breasts and jays have white breasts. Maybe it a Indigo bunting which would be all blue and closer in size to the bluebird.