There are boats leaving from San Pedro, Long Beach, and Newport Beach, I think. It takes 1-2 hours to get there by boat, and is an enjoyable ride. You can visit Avalon and take some various tours there, and yes you can do it in a day.
I am sorry, I do not know what their handicap accessibility is.