Ahah! Goldfish, my specialty!
Well, in the first place, good for you for putting it into a tank instead of a bowl. However, goldfish are large, messy fish that need big tanks (minimum 10-20 gallons per fish, depending on breed), heavy filtration and oxygenation (they are a pond fish, after all). Goldfish have a horrible reputation and are the last fish that can be kept in bowls or even small tanks. And they can come in many shapes and colors.
First off, here is a great beginner goldfish site that has wonderful, correct information that I link to all the time:
http://kokosgoldfish.com/
Here's the best place to start, it gives you a nice rundown on the basics.
http://www.kokosgoldfish.com/care.html
I'll try to keep this quick. Goldfish are indeed a very hardy fish that can live many years (up to 40+ have been reported, but 10 or so years is the average). However, they are very messy (also called a high bio-load) and need either a large pond with a good surface ratio, or a completely cycled tank of appropriate size with good filtration (a number often referenced as the preferable filtration power is 10*GPH, or gallon per hour. Meaning, if you have a ten gallon tank, for a goldfish you're going to need a filter with at least 100GPH. Though more will not hurt at all).
So while it may be ok in a two gallon for a very short time, it will need an upgrade and soon. How big of a tank it will need depends on the breed. If it is a fancy type, as in having a double tail, ten gallon per fish is the minimum (They'll get to around 4-8 inches generally, though they can get bigger). However, if it is a single tailed type (like most carnival fish are) it's going to need 20 gallons as a minimum (these puppies can push 12 inches and are very active)! So if you do get it into the 5 gallon, please don;t get it a buddy! They are not a schooling fish, and do fine on their own.
Also, due to their high bio-load (aka their giant mess) they absolutely need a cycled tank. Goldfish can push out ammonia so much it rivals other big fish like piraña and oscars. Cycling is a necessary but long process (which I'm assuming you're familiar with since you had neons). Here they go through the process much better than I could and give good tips on cycling with goldfish.
http://www.kokosgoldfish.com/cycle.html
For now, you're going to need to do some heavy duty water changes to keep the water quality within non-lethal ranges (an ammonia level of over 0.5ppm and a nitrate level of over 20ppm can be very detrimental or even lethal for your fish). If you do have a filter, I would recommend a 30-40% waterchange every day until he can get in a properly size tank. If not, you are going to have to do a 80% waterchange.
Other than that, the link I gave you should basically cover it. Here's another good site:
http://thegab.org/Goldfish/goldfish-basics.html
That being said, be careful about information on goldfish you find on the internet. Most of it is ripe with misinformation. Good places to go are aquarium or goldfish boards. Untimatebettas.com, for example, also has exemplary information on goldfish care.
Please, do not hesitate to ask me any questions or concerns you have, I would love to help!
Just another quick ?, 
If you don't really want animals, stop saying yes to people that try to drop them off at your door.
