I don't think you are crazy at all. Like another poster said, you take care of them all everyday, you would be fine.
We have seven children and take them all to WDW several times a year.
What works for us, is a laid back approach, don't overplan and usually we are pleasantly surprised with what we are able to accomplish. Also, with our youngest being so close together (dd/ds twins almost two and dd just turned 3), we find that an afternoon break and staying on property (and using Disney transportation) work best for us (once we rented a beautiful house off property and it was wonderful but the kids wouldnt get out of the pool). Admittably, the first time we took the twins when they were infants and their sister wasnt quite two, it was difficult.
We don't stand in line for autographs and pix and stuff, thats one thing I cant stand. We prefer to get photos at character meals. Less hassle for me and dh that way.
We have taken grandparents, but I end up pushing myself and the kids harder when they are there because I am afraid they are bored or feeling like they are missing out spending too much time in Fantasyland or at the hotel pool.
Our gang also really loves
Disney Cruise Line. I actually feel like I get to relax when we are on the cruise.
I knew my children had officially become Disney fanatics when they got their haircuts last week and kept wanting to know where the pixie dust and pink hair were (mainstreet barbershop memories).
Good luck and have a grea time.