I have had one problem with direct debit, but it had nothing to do with authorized debtors pulling money out of my account.
Years ago, I had direct debit for all sorts of things (insurance, mortgage, utilities). Then my checkbook got stolen and someone wrote checks off the account, so the bank had to close the account.
It was a pain in the back end to change the direct debit arrangments to a new account, and I missed several payments that I had forgotten about.
I ended up not changing most of them, and went to setting up automatic payments in Quicken, where I can switch the accounts much easier.