The second and third week are usually the best in May crowd wise. Smart to go during the week, beware that Monday's can still be a bit busy, even busier than Fridays.
Yes generally most of Japan is busier during golden week. People are out and about enjoying their time off.