Sometimes the "other" person is a ghost that somehow gets sucked into the farm with you. I have about 10 strangers on ignore because they ended up on my farm after I've been to the marketplace.
When I harvest with a DIS person usually if there is more than 1 of us we ended up chatting, harvest a few fields, chat some more, harvest a tree, chat again.....
I once had a PITA that hired everyone in the marketplace, sent us all to his farm and there were no crops for us to harvest. Apparently he thought he was funny or was eliminating his competion in the marketplace.