Is there actually any evidence any of these things are bad for you?

I know the whole artificial sweetener thing is bollocks but are there actual valid grounds to care about these things in your food or is it basically people being a bit funny about things that aren't "natural"?