I get aware if I'm anywhere and there even are balloons nearby.
If anyone carries them or interacts with them in any way, I get nervous, and if anyone touches them directly at all, I start to get afraid.
If that someone is children or anyone I don't trust, I just freak out and want to leave immediately.
While my phobia comes and goes a little, depending a lot on the place and situation itself, I can safely say I'd be really nervous in the described situation. If it was a helium balloon, maybe less so, as the goal then usually is to inhale to make funny voices, but I would never feel comfortable as long as I knew someone are handling an inflated balloon nearby.
So please, consider us phobics a little