Be noncommittal. If they ask that they can come too, let them know you still haven't gotten the details down as to how many people will be allowed to come depending on the space provided.
Take charge of the situation. If they don't get the hint after that, be a little more strong with them, and tell them, "I'm sorry I'm in a rush to get going on a project/going to lunch, and pretend like you didn't hear them as you run off to where ever. Then make a visit to your manager/supervisor and ask for more help.
