It's for the most part true. There are exceptions, such as the example you provided where there are food stalls near temples, where there are festivals, celebrations (watching the sakura trees while in bloom is a big one), holidays, school events, etc. Not everyone adheres to the "unspoken rule"...