Well it's a very subjective question. Many folks at the Fort will say it's enough time at the parks but not enough time at the Fort.
For us, there is no way it would be enough time for either. Given that there are 4 parks, that's a park a day, which can be tough (depending on how you do the parks). We also like time in between to relax at the Fort and the pool (or other things). If you are not a "Full Day" park person (or even something close to that) then you could be okay. That said, are you planning on coming back sometime (soon)? Even if you don't get to do everything you want this trip (and with 4 days that could be tough for most folks), you can save some for future trips.
We are going to WDW again in June (or at least it's booked) but will be staying at AoA this time. DW wants to try something new. Also our park trips and ADRs we are also trying to do something new (for us) so that we can get new experiences... We know we will be down again at some point too, so our 1 week stay is good for us (plus we are trying to maximize the AP we got in August 2016)