99.9% of the time, the answer will be in the form (coefficient * a^2 +/- coefficient * b^2) * (coefficient * a +/- coefficient * b)
A good way to start out is to make the coefficients in the second parentheses the same as the coefficients of the first two terms of the equation. For example, the first two terms in your equation are 10a^3 + 4a^2b, so make the coefficient of a be 10 and the coefficient of b be 4. So far, we have:
(coefficient * a^2 +/- coefficient *b^2) * (10a +/- 4b)
Next, pretend the coefficents of a^2 and b^2 are 1. FOIL like you normally would.
You would come out with 10a^3 +/- 4ab^2 +/- 10ab^2 +/- 4b^3. The multiplication with a^2 came out correct, but the multiplication with b^2 didn't, so the coefficient of b^2 isn't 1. You just have to figure out what it is, which is just simple arithmetic that I'm sure you can figure out. The coefficient of b^2 comes out to be 3/2.
Now, the equation should look like (a^2 +/- 3/2b^2) * (10a +/- 4b)
Only thing left to do is to determine whether the signs are pluses or minuses. The easiest way to do this is to just guess and check. There are only four possible combinations, so it shouldn't take long.
In the end, you should come out with (a^2 - 3/2b^2) * (10a + 4b)