2nd grade was toughest for my son in a general way, but 3rd grade the most challenging academically. 4th grade was easily his best all around year, and 5th his easiest academically.
Each grade is a huge step from the grade before. Lots of children have trouble in K who haven't been in school before, because they are trying to learn new things while also figuring out how to fit in at school. Lots of children in 1st have trouble, especially if they weren't reading by the end of K, or didn't understand amounts in math. Lots of children have trouble in 2nd because it all becomes much more academic, faster paced, no centers, little or no recess. Lots of children have trouble in 3rd because they have to be more self-motivated, more homework. Lots of children have trouble in 4th because they have to do long-term projects, which takes perseverance, forethought and organization. Lots of children have trouble in 5th because peer pressure causes social problems either with other children or administration and if they missed any of the critical steps before, they are behind academically, this is also the age that they should start actual studying outside of class without a specific assignment.