it's well worth it. i make the trip from sd several times a year and often times leave at 11am, 12pm, etc. traffic will be nothing during the times you are leaving so don't worry. downtown disney, dca, and
disneyland are all in the same area and thus the entire area is walkable.
you should look into AAA tickets, i think you get free parking, which is a 15 dollar discount essentially. you'll want to park at the toy story lots (fastest from 5 south) and head into the park via the shuttle. if you really don't want to see DCA, then sure you can stick with disneyland. if you haven't eaten lunch yet, what i like to do is first go inside the park to tomorrowland, get a fast pass for space mountain, and go on some of the rides there, and then take the monorail over to downtown disney (a fun ride in itself). right next to the monorail is the best value imo in the entire park, earls of sandwich (a yummy and cheap counter-service sandwich chain). if you want a more sit-downey place, storytellers cafe in the grand california hotel is great. if you want to stay in the parks, good places to eat are the mexican place in frontierland, and the restaurants in new orleans square.
dtd shops close later than the parks on weekdays typically, esp the restaurants. you can wander around there after the park close, although the last shuttle back to the lots is supposed to stop 1 hr after park close, so keep that in mind. there is a gigantic world of
disney store that is located right outside the esplande that connects the two parks, literally a 1 minute walk from disneyland's gate.
honestly given that u can see wdw magic kingdom since your east coast, if you can afford it it would be a lot of fun to get a park hopper so you can at least see DCA. world of color for example is something you can only see in DCA. you can also single-rider RSR which is probably the single most in demand disney ride in the world.