Did they kill humans? Yes.
Did they kill a large portion of mankind? Absolutely not.
Vortigaunts, even if they wanted to post Black Mesa, were not in any condition to go to war with humanity or constantly seek conflict with them. There absolutely was conflict, especially depending on the area, but its not like there was some concentrated genocide for either side.
Regardless, that was nearly twenty years ago, and Vortigaunts compared to things like Antlions, xenian infestations, Race-X and especially the Combine, are honestly pretty irrelevant in terms of 'things that have screwed us over these last few years.'
Outright racism towards Vortigaunts is a bit...overcooked 7 times out of 10. Distrust or a general social hesitation typically makes more sense. Hating a Vortigaunt 'Because its an alien' is really strange when you are currently being ruled by even more 'alien' aliens. Rather than that, it makes more sense to blame Vortigaunts for their small role in bringing fate to where it is now (IE basically having fucked everything they've tried to do up), and just generally distrusting their future ambitions.
Generally though, for most people, Vortigaunts are not enough of a threat nor enough of a historical foe to hate them just because 'alien' Personal loss or history is a more likely driving force.