If it were me, travelling across the border in either direction, I think I would want to have my passport, or birth certificate along with the drivers license picture ID.
You have to think. "What would I do if they wouldn't let me back into the country?". And they are perfectly within their rights not to allow you back into your own country if they don't want to for whatever reason, if you do not have proof of citizenship. A driver's license does not prove whether you are American or Canadian or any other nationality.
Say something terrible were to happen and they tightened security at the borders, if you didn't have proof of citizenship you might find yourself locked out of your own country.
Just something to think about.