We have the E-PL1. There is a Dummies book for it. Despite having an E510 and an E30, it took me a while to figure out how to work it. (Still not 100%. Often in a rush I cannot find the settings I want so I tend to keep it on Auto a lot, unlike my dSLR which I am very comfortable finding manual settings for, even in the dark.) I literally sat down with the Dummies book and went step by step to get the basics down. Because there are not as many knobs and buttons, you have to learn to use the controls differently. It's confounding at times.

But it takes a very nice picture, so I'm relatively happy with it (and the family loves it). Like anything else, it takes a lot of practice to get good at it.
Personally, if I were you, I'd stick with your original choice, as, IMO (and if you're like me), you're going to have the same issues working the PEN that you had with the E-M5 - or any other mirrorless, given that they're all smaller and controls are condensed and technology is becoming more complex. Your ultimate results, despite whatever painful process you have to go through with whichever camera you choose, are going to be best with the E-M5 long term thanks to the new sensor. (Although the sensor in the PEN isn't bad, same one that's in my E30, improved over the E510. Not sure if the same one that's in your E3.) I agree I love my Olys and I would have difficulty switching at this point. So I say you try it again and just know it's going to take a while to learn to use it. At least you have your vast experience with Oly menu basics and terms, and a lot of help here now with others who have the camera.
I'd like to buy an E-M5 myself at some point but dread the learning curve. (And am fairly happy with the set up I have now. It would probably be more of a "fun" purchase.) Wish it were simpler sometimes. In my next life I hope to be born technically savvy!