Very familiar with the Port Jefferson area since we live very close, and the ferry there. Also, very familiar with driving from the Port Jefferson area to upstate NY (Albany, Saratoga area) - done that drive more times that I care to count!!
Oyster Bay is a considerable drive to the west from Port Jefferson (about an hour) - so you are halfway to the NY bridges when you drive over to OB, and it doesn't make sense to backtrack to PJ to catch a ferry back across the sound - you might as well just drive back upstate from Oyster Bay.
The North Fork is considerably less crowded than the south fork this time of year - the north fork is still busy in the summer, but not the mind-numbing bumper to bumper traffic of the Hamptons and Montauk (where Hither Hills is). The north fork will give you a more rural feel, pick-your-own farms, farm festivals, wineries and more wineries, multiple breweries, and a hard apple cider orchard too - and some small town centers. It will be hard to hit both the north and south fork in a 3-day trip (with time also allocated to Stamford, and Oyster Bay) - so you will probably need to pick a 'coast' to concentrate on. The south shore on the east end will give you ocean beaches, 'touristy, tony' towns, and a different feel in general during the summer months. It is not difficult to get from Port Jeff to the south shore once in the Riverhead area. If traveling on the south shore, once you get east of Hampton Bays you are generally down to one main road that gets very, very backed up in the afternoons.
Fall is also beautiful on the east end of Long Island (the last 2 weeks of October will become very busy on the north fork, with pumpkin and apple picking season).
The one thing you don't mention is where you will stay - this will be the harder part of the equation. Once you get east of Riverhead - you only have boutique hotels, small and expensive resorts (and many overcharge substantially during the high-season for the level of 'luxury' you are getting), and B&Bs.
If you were to come into Port Jeff, there is a small hotel right in town (mixed reviews, and I think it depends on whether you have a renovated room or not) - but, it is nice for convenience since it is a nice walking town with numerous restaurants and bars (and shops). There is a brand new Hilton Garden Inn a 5 minute drive away in Stony Brook. There is a Holiday Inn Express in the area also. In Riverhead there is a Hyatt, Holiday Inn, and Hilton Garden Inn.
Also, if your husband enjoys history - the Old Bethpage Restoration Village is a 20 minute ride south from Sagamore Hill and you would need to make that drive south to get to the main highways to head back towards the city anyway.