You can do one-way rentals from Shannon; the issue is that they don't often do them for drop-off in the North, which is another country altogether (though you can certainly drive the rental back and forth over the border, I've done it many times.) Agencies in the UK are less draconian about taking cars to NI on the ferry, so people sometimes do it that way.) If you return the car in Dublin and take a train up to Belfast for the ferry you would be fine.
Another option would be to take a short flight to Belfast from Shannon and pick up your car there, then do a round-trip.
As a general rule, flights into/out of Dublin are cheaper than Shannon, so it may make sense to fly to Dublin, train to the west coast, then drive back.
The ferry is an interesting experience, but as you can see, it can often be easier and less expensive to just fly from one island to the other.