I think that the answer to a lot of these problems is restrictive zoning rules and (especially) very tight examination of variance requests.
The biggest problem killing affordability in my area is the infill tear-down market. Builders are coming into established neighborhoods of small homes, buying up entire streets, and then replacing those 1000-1600 sqft homes with new builds at least 2X the size on the same lot. The homes that are there may be a little dated, but they are perfectly sound and completely liveable; these areas are not blighted. So the house sells for maybe $200K, and then it is replaced with a much bigger one that is put on the market at $800K.
IMO, that has to stop, and it can be stopped, by making it a rule that if there is an existing home on the property, the one that replaces it cannot be more than a certain percentage larger (maybe 10-15%). Put that rule in place, and the neighborhoods of affordable starter-size homes will be left alone by the spec builders. (I am OK with exceptions for homeowners who wish to substantially enlarge their own homes that they already live in; I just want to prevent the wholesale destruction of affordable neighborhoods by builders and investors who are doing it purely for profit, and not because they had 2 more kids, need to move Mom in, or have to have a home office.)