I agree, it's okay to keep it relatively short.
I only skimmed people's responses so I don't know if this was mentioned. Since you are so nervous and wouldn't want to be caught like a deer in headlights if you completely forget what you were going to say, I think it would be fine to have an index card with you. Of course, it always comes across better when you can talk naturally and not have to read something, but you would at least have something to save you should you freeze.
Even if you put down a few keys points to mention (i.e. funny prank during high school, felt like brothers growing up, bride and groom perfect together, best wishes for lifetime of happiness, etc.), then you will have something to jog your memory.
My DH had a toast written out for his brother's wedding, it was a page long. He's not the nervous type so he kind of used it as a guide, reading some parts but also ad libbing parts here and there. I think it's fine if you bring something to help you along, just try not to completely rely on reading it word for word. You can prevent the temptation to read it word for word by only writing short phrases as reminders.
Also, what I've noticed is that the guests like to laugh; they love hearing humerous stories of something funny that happened to the groom, or some funny quip about when he was a kid. For example, DH in his speech to his brother mentioned they had a lot of fun memories growing up: Saturday morning cartoons, getting back out of bed after their parents were asleep, and throwing darts at each other in the basement!
If you can think of 1 or 2 funny memories to mention, you'll get the people laughing and then you can start to bring it to a close.
Good luck!