No doubt other factors played a role. There was a low turn out aggravated by rain and a lack of enthusiasm for Cantor. His reelection was taken for granted. He isn't especially likeable and he is detached from the district. There were cross over votes by Democrats who wanted to get rid of him. There was a grass roots push for Dave Brat, and a push by extremist right wing radio personalities for him.
However, there is a reason for Democrats to downplay the importance of the voter opposition to immigration reform. They want to minimize the flight of scared Republicans away from any hint of compromise. Fear of the Tea Party will also solidify the already obsessive opposition to anything else Obama supports in addition to immigration reform. Not all Democrats downplay the importance of immigration in the primary. Others do. Perhaps intentionally for strategic purposes, or perhaps they rationalize what they want to believe. Many probably do genuinely believe it wasn't as important as I believe it was.
There are some rational arguments against immigration reform. Why bring more people in when there aren't enough jobs for the people who are already here? Why should people who came illegally jump ahead of people who are following the legal immigration procedures? There are also arguments against these positions. My point isn't whether the positions are valid or not. My point is that someone doesn't have to be racist to oppose immigration reform. I'm establishing that qualifier before I elaborate on the role that racism does play. Racist Americans are obviously going to oppose immigration reform. It is likely to be a very important issue to them. An issue that motivates them to go out and vote when other people stay home.
Tea Party types are the most racist white Republicans. Robert D. Putnam of Harvard and David E. Campbell of Notre Dame documented the evidence for this in their book, American Grace. There was a survey of more than 3,000 people's social and racial views before the Tea Party began. They went back and questioned as many of the same people as they could again, after the Tea Party was established. Not all Republicans had joined the Tea Party. Democrats and Independents hadn't joined the Tea Party as the Tea Party liked to claim. The people who joined the Tea Party were the most conservative and most racist white Republicans before the Tea Party started.
In the study everyone ranked twenty four groups and individuals, Catholics, Jews, atheists, Muslims, etc. Most people liked the Tea Party and the Christian Right (another trait of people who joined the Tea Party) least, but the Tea Party members liked blacks and Mexicans the least. Their most common answer for why they don't like Obama amounted to, "I don't know, I just don't like him." The study also supported the case that government spending was less important to them than protesting that the president is black.