Prejudice: Judgments that are not held on rational grounds and are used to infer or justify conclusions.
Evidently, there are prejudices and humans have and have had very many prejudices - and evidently one problem with the term "prejudice" is that humans are quick to accuse ideological opponents that their views rest on prejudice.
The logical grounds to maintain that a certain belief is a prejudice are:
(1) Important ground: It is used to justify other beliefs the believer in the prejudice has and believes important.
(2) Not provably true: There is no logical proof from true premisses that it is true.
(3) Insufficient rational evidence: There is not sufficient rational evidence for it to support it.
(4) Based on biased and sealed off evidence: The evidence of believers in the prejudice is biased and sealed off from conflicting evidence.
First, let's consider these in turn:
Important ground: Beliefs that are used to infer beliefs that are not held important may have little rational evidence, but are not prejudices, and indeed are much easier given up by those who maintain them, precisely because they are not deemed important to their central beliefs.
Not provably true: Beliefs that are provably true, such as the truths of logic and mathematics, are not prejudices - only beliefs that are not provably true may be prejudices.
Insufficient rational evidence: Evidence is rational if it is not based on prejudice and there is good support for it. Prejudices are based on evidence that is not rational or on evidence that has no good support, and in both cases often involves a considerable amount of wishful thinking.
Based on biased and sealed off evidence: True believers in prejudices base their belief in it on biased, partial, slanted evidence, on wishful thinking or on belief in authorities and leaders, and seal off their evidence normally by refusing to seriously investigate contrary evidence, on the grounds that they already know the truth, namely their prejudice.
In short: The main reasons to characterize a belief as a prejudice involve the manner in which it is held and the grounds on which it are based.
Next, it is well to notice that prejudices are very hard to avoid, and indeed children and young people can hardly do without them, and tend to be offered a lot of them by family and friends, often in good faith i.e. because the others sincerely have these prejudices themselves, and believe them important and true.
Also, there are rational prejudices: Beliefs one has for which one has insufficient evidence oneself, but for which others have the necessary evidence. Thus, most people go to medical doctors based on the belief that they are in a better position to explain what ails the patient medically than people who are not medically qualified. Most laymen who use the services of medical doctors do so because of prejudice, that is fairly called rational when compared to the prejudices of those who refuse to visit medical doctors for their ailments, but go to "alternative healers".