Yesterday I said something to my colleague to which he replied 'do you think I came over on a banana boat?'

I was like WOAH thats massively racist, you can't say that.
And he is insisting its definitely not.

So...is it racist?