Sounds good to me, but I'm kinda weird - my only attendant will be my brother (the... Man of Honor?).
One thing to consider is that the Best Man and Maid/Matron of Honor traditionally have certain responsibilities - planning the bachelor party/bridal shower, keeping track of rings, license, etc - so your attendants may be relieved not to be picked as BM/MH.
Then again, I'm not sure who would do all those things for you (if you even want them done). Maybe all the bridesmaids would work together on your shower and all the groomsmen would plan the bachelor party?