Believe it or not my parents lived in the Magic Castle before it was a landmark. They rented it for a couple of years in the 1950's.
The Castle is about 10 minutes from my parents house, where I grew up. Their house was up in the hills but the Castle is in Hollywood proper and the neighborhood there does have a certain element that tends to come out at night. I would check to see what the parking situation is so that you are not parking in an unsecure dark lot. (Same precautions you would take in any large city).
Any other questions, feel free to PM me..I lived in Hollywood most of my life .