Okay, I know go to the bank and find out what we can qualify for/get pre-approved,
You said that you are good with living below your means. The most complicated part of your house buying process will be the monthly cost of house.
You don't need to go to a bank to know what you are comfortable spending. Each month you will need to account for principle & interest (mortgage), property taxes, homeowner's association (if applicable), homeowner's insurance, and potentially flood insurance. This will cover the monthly expense.
How much you can qualify for will also depend on how much cash on hand you have to bring to closing. You will need to pay the down payment, title company fees, and any bank-related fees. Depending on whether or not you plan to do your own escrow account, and depending on when taxes are due in your area, you will be required to contribute a certain number of months worth of tax payments (keep in mind that the seller will likely contribute to this as well, but depending on how conservative the bank is, they will want a certain amount of money to ALWAYS be in the escrow account).
So your money available to pay at closing and what you can pay each month will determine what you can afford. Considering your earlier statements, it is likely that you will be picking a price point that is within guidelines. These are all things that you can research and figure out without having to talk to a mortgage person. The mortgage person will generally also want to check your credit, because their interest rates will partly be determined by your credit score.
It doesn't sound like you have put too much time into researching the money end of the deal which is why I have tried to detail it here. My mother was a real estate and then mortgage broker for years and years. I used to work with her when she did mortgages. She was a very helpful mortgage broker, she worked so that people would not get hurt as a result of working with her. If you really get into it, and have more questions about the mortgage end of the deal, I'd be happy to answer and help as I can. There is a whole pricing structure to the way that mortgages are offered and how people who broker mortgages make their money that can be helpful to understand when you are trying to qualify for your first home and make a sound financial decision.
As to the real estate end of the deal. There are all types of Realtors out there. Especially with a Toddler in tow, the best Realtor you could find is someone that has either worked a neighborhood for so long that they are familiar with floor plans and features of the homes in the area, or they are willing to preview the houses for you. You would need to be clear on your requirements for a home and take full advantage of the Realtor's service to show you homes that ONLY meet your criteria. Not all Realtor's are willing to preview, it can take a lot of time to do this, so this could be a deciding factor for you.