sounds familiar @MDT  except substitute jobs/work for trip. 

starts off sounding like the thing that's going to fix everything

then as it gets closer feels completely unbearable

Now dreading seeing people from work, but it is what it is

Anyway. They care enough to ask.

You are out and about. 

No point in trapping yourself with your thinking ie couple days ago thought of going away wa unbearable, now you are a failure somehow?

just have a smile to yourself maybe, and if you have to, just have a quick throw away answer about trip, say change of plans or something and walk away