Closing costs are the elephant in the room when you refinance. You've got to make sure the new, lower interest rate is worth it given the sometimes hefty fees you have to pay to close the loan. If you plan to sell the house before you have recovered the closing costs you must pay to refinance, that is the No. 1 deal breaker. To figure this out, all you do is divide the cost of closing by your monthly savings to see how long it's going to take for the new loan to pay for itself. I suggest the five-year cutoff for recouping closing costs simply because stuff happens, life changes and you want to know you are out from under those closing fees should you need to sell unexpectedly.