Priming and Prejudice: The Bias Effect of Origin Information on Peer Review, Judgment and Evaluation

Priming y prejuicio: el efecto de sesgo de la información de origen el juicio y la evaluación

Priming e preconceito: o efeito de viés da informação de origem no juízo e na avaliação

Luiz Victorino
Universidade de Brasilia, Brasil
Ronaldo Pilati
Universidade de Brasilia, Brasil
Alexandre Linhares
Universidade de Brasilia, Brasil

Priming and Prejudice: The Bias Effect of Origin Information on Peer Review, Judgment and Evaluation

Avances en Psicología Latinoamericana, vol. 37, no. 1, 2019

Universidad del Rosario

Received: 05 May 2017

Accepted: 10 October 2018

Additional information

To quotes this article: Victorino, L., Pilati R, & Linhares, A. (2019). Priming and Prejudice: The Bias Effect of OriginInformation on Peer Review, Judgment and Evaluation. Avances en Psicología Latinoamericana, 37(1), 169-178. Doi: https://doi.org/10.12804/revistas.urosario.edu.co/apl/a.5635

Abstract: The peer review system is a core tenet of scientific publishing, and has been studied throughout the years. Recent discussions on judgment and evaluation biases have highlighted the importance of priming effects on those processes. We have conducted two experiments with evaluation tasks (a scientific article evaluation, with professors and PhD students as participants, and a chocolate tasting, with undergraduates) in which they were exposed to a footnote acknowledging financial support from a fictional agency. In one condition, the agency’s name was associated with the African con­tinent, while in another condition, it was associated with the European continent. There were statistically significant differences in judgments on both tests, with individuals on the European condition giving better evaluations of the article and the chocolate, even though the stimuli were subtle enough not to be remembered by 92.5 % of all participants. We also found evidence of a moderating effect of academic experience on the priming process.

Keywords Peer review, priming, cognitive bias, ju­dgment.

Resumen: El sistema de revisión por pares es un principio básico de la publicación científica, que ha sido estudiado a lo largo de los años. Las discusiones recientes sobre ses­gos en los juicios y las evaluaciones han destacado la importancia de los efectos de priming en esos procesos. Hemos realizado dos experimentos con tareas de eva­luación (una evaluación de un artículo científico, con profesores y estudiantes de doctorado como participan­tes, y una degustación de chocolates, con estudiantes universitarios) en las que estuvieron expuestos a una nota a pie de página que reconocía el apoyo financiero de una agencia ficticia. En una condición, el nombre de la agencia se asoció con el continente africano, mientras que en otra condición, fue asociado con el continente europeo. Hubo diferencias estadísticamen­te significativas en los juicios en ambas pruebas, tales que los individuos en la condición europea evaluaron mejor el artículo y el chocolate, a pesar de que los es­tímulos fueron lo suficientemente sutiles como para no ser recordados por el 92.5% de todos los participantes. También encontramos evidencia de un efecto moderador de la experiencia académica en el proceso de priming.

Palabras clave: revisión por pares, priming , sesgos cognitivos, juicio.

Resumo: O sistema de revisão por pares é um princípio básico da publicação científica, e tem sido estudado ao longo dos anos. As discussões recentes sobre viés nos juízos e as avaliações têm destacado a importância dos efeitos de priming nos processos. Temos realizado dois expe­rimentos com tarefas de avaliação (uma avaliação de um artigo científico, com professores e estudantes de doutoramento como participantes, e uma degustação de chocolates, com estudantes universitários) na que estiveram expostos a uma nota de rodapé que recon­hecia o apoio financeiro de uma agência fictícia. Em uma condição, o nome da agência associou-se com o continente africano, enquanto que em outra condição, foi associado com o continente europeu. Houve dife­renças estatisticamente significativas em juízos em ambas as provas, com indivíduos na condição europeia dando melhores avaliações ao artigo e ao chocolate, apesar de que os estímulos foram o suficientemente sutis como para não serem lembrados pelo 92.5% de todos os participantes. Também encontramos evidência de um efeito moderador da experiência acadêmica no processo de priming.

Palavras-chave: revisão por pares, priming , vieses cognitivos, juízos.

Introduction

Science stands on the process of peer review. Peer review aims to assure reasonable and fair consideration of scientific papers, for publishing in journals but also for research funding, tenure, prize awarding and so forth. Even though among journals (and institutions) there are small differ­ences in the procedure itself, the process is based on the concept of scientific experts analyzing and scrutinizing studies in their specific field, in order to determine which ones are publishable, how revisions should be conducted, or if publication should be supported by a journal.

Such evaluation is carried out to verify if a study is well conducted, if its methodology is adequate and, obviously, if the results are significant. Since those specialists conducting the review are experi­enced in publishing in the field, it is reasonable to assume that the process enhances the development of specific knowledge. Nevertheless, the equality and transparency of the process is far from achiev­ing a consensus, and questions concerning eval­uation biases in the traditional peer review have been brought to attention (Giner-Sorolla, 2012; Ietto-Gillies, 2012; Kaatz, Gutierrez & Carnes, 2014; Maner, 2014)its role, and the criticisms levelled at it. An analysis of specific problems in economics leads to a full discussion of the Open Peer Review (OPR, and amongst those, bias against author’s origin (Rubinstein & Brenner, 2014; Ver­legh, 1999; Wolff, 1973).

There are numerous situations in which ste­reotyping and judgment are connected, and un­derstanding this relation can improve evaluation processes (Kahneman, 2003; Wall, Liefeld & Hes­lop, 1991). In the present study, we focus on sci­entific papers evaluation, and how the stereotype of a low-prestige institution, or of a low-prestige country, can affect evaluation and judgment of a scientific paper. The literature on stereotyping in judgment and evaluation of scientific papers indi­cates that scientists at prestigious universities tend to have higher rates of publication and higher rates of citation, in part attributable to the prestige of their institutions (Giner-Sorolla, 2012; Ietto-Gil­lies, 2012; Robert C. Calfee, 2010; Wolff, 1973). Even if we consider that prestigious institutions may have better research centers, superior infra­structure, attract better scientists and that, devel­oped countries usually have well-developed sci­entific incentive programs and higher investments on science, it is reasonable to think that this will have an impact on their scientific production. On the other side, it could not affect the evaluation of a scientific work, since those aspects are not solely responsible for the quality of a research —also, this kind of judgment bias could easily be defined as stereotyping.

Stereotyping is a cognitive strategy, with evo­lutionary origins stemming from the beginnings of the human race, to optimize mental processing —after labeling a person, an object, or a situation, it is easier for the brain to identify and react. The interesting fact about stereotyping, and the focus of the present study, is that stereotypes can be activated even when the stimulus presented is unknown, that is, when the brain makes connec­tions with past experiences in order to develop a subsequent coherent response (Shanks et al., 2013). Even when something is shown for the first time, based on our previous experiences, we automatically develop a labeling process which will influence our present and future responses to that stimulus (Loersch & Payne, 2014).

Stereotypes arise from unconscious associa­tions, sometimes through priming effects, which have been attracting more attention to the phenom­ena in the last years, not only for their potential to influence behavior, but also for their widespread presence. The literature shows that the automatic processing can lead to biases and errors in judg­ment (Morewedge & Kahneman, 2010) and even though we are aware that this is a possibility, it is hard, if not impossible, for individuals to control the prejudicial effects (Holroyd, 2015).

The literature on priming is extensive on how it can influence the basic processes of evaluation and judgment (Cokely & Feltz, 2009; Henderson & Wakslak, 2010; Smith & Mackie, 2014), even though the cognitive process behind it is not com­pletely described. The standard perspective is that, once a particular stimulus is shown, it activates neural networks through a process called spread­ing activation —the stimulus activates pieces of information in memory that are related or asso­ciated to the content, influencing the response to the stimulus (Molden, 2014), and since the acti­vation of those networks is not conscious, there is no participation of conscious will in the process.

It is well accepted that not only simple and ba­sic responses can be primed, but even extremely complex cognitive processes can be affected non­consciously through priming, such as goal activa­tion (Marien, Custers, Hassin & Aarts, 2012), ob­served and simulated responses from others (Smith & Mackie, 2014), and, the focus of the present study, judgment and stereotypes (Allen, Sherman & Klauer, 2010; Rubinstein & Brenner, 2014). Since scientific authors are also victims of prejudice and publication biases (Garfunkel, Ulshen, Hamrick & Lawson, 1994; Lee & Schunn, 2011; Papaioannou, Machaira & Theano, 2013) on the present study, we focus on scientific papers’ evaluation, and how the stereotype of a low-prestige institution, or of a low-prestige country, can affect evaluation and judgment of a scientific paper through a priming effect. Priming is a recurrent phenomenon in social cognition, and recently, interest in how it can influ­ence evaluation and judgment has been increasing (Chaxel, Russo & Wiggins, 2016; ­Doyen, Klein, Simons & Cleeremans, 2014; Mohr, Koutrakis & Kuhn, 2015). Beyond the examination of priming effects, recent advances are trending towards the possible mediators and moderators (Pickering, McLean & Krayeva, 2015; Poehlman, Dhar & Bargh, 2016), including attributions related to the origin of primed information (Loersch & Payne, 2014). The influence of priming on complex pro­cesses can be explained through cognitive biases, which can arise from different sources (Hilbert, 2012; Pleggenkuhle-Miles, Khoury, Deeds & Markoczy, 2013), including culture, (Grossmann & Jowhari, 2018)simultaneously evaluating the role of task-compliance, operationalization spec­ificity, and cross-cultural robustness. In the orig­inal study, participants either circled first-person plural (interdependent condition, hence the need to research on and replicate effects of priming on judgment in different cultural groups, such as Brazilian academics.

In the present study, we focus on two different types of evaluation, one presumably more techni­cal (a scientific paper) and the other more subjec­tive and personal (a chocolate tasting), and how the stereotype of origin can affect evaluation and judgment. Our objective is to study if, through a priming effect, subjects will show a bias in eval­uation and judgment on both tasks. In addition, to contribute to the growing body of evidence on the role of moderator variables in priming effects, the moderating effect of academic experience will be researched.

General Method

The present work is based on two studies using the same stimuli, words on a footnote that imply European or African origins. There is consen­sus in the literature that European and African origins are related, respectively, to positive and negative representations, leading to stereotyping effects in many domains (Rubinstein & Brenner, 2014). Based on a pre-test, the words chosen were Welgesteld-Tijdschrift (“wealthy magazine” in Dutch), and Kuranta-Bothata (“problematic mag­azine” in Setswana, a Southern African language).

The experimental design followed the same rationale, which was to subtly introduce the stim­uli to the participants, then present the evaluation object (a scientific paper for experiment 1, and a piece of chocolate for experiment 2) and finally assess the biases in evaluation and judgment for the experimental groups. On both experiments, the manipulation check consisted in asking the par­ticipants if they could remember any information regarding the presented stimuli.

Experiment 1 – Chocolate Testing

Participants

For this experiment, 113 mostly (81.4 %) male graduate students (M = 24.78 years, SD = 7.02) were given an unmarked chocolate, and after tast­ing, asked to evaluate it using a questionnaire.

Instruments

The chocolate evaluation questionnaire (Valde­ci, Bastos, Pereira, Basilio & Leite, 2012) was answered on a Likert scale, from 1 (very poor) to 5 (excellent) and one question asking if any infor­mation on the institution funding the research was remembered, as the manipulation check.

Procedures

Undergraduates were randomly assigned to three groups and conducted to a room, where they were handed out a consent form before being asked to taste and evaluate a piece of chocolate. The variable manipulated was the information present­ed in a footnote placed on the informed consent, regarding the institution which funded that re­search. The institution’s names were Welgesteld-Ti­jdschrift (wt Condition) and Kuranta-Bothata (kb Condition) with the control group (cg) lacking a footnote.

After tasting the chocolate, the evaluation ques­tionnaire was delivered and answered in the same room, and the participants were let off.

Results

The evaluation scores were summed and the average was considered the General Tasting Score (gts). A one-way anova indicated that the dif­ference was significant, F (2, 112) = 5.641, MSE = .28, p = .005, ŋ2 = .06, with participants in kb Condition showing a tendency to evaluate the chocolate more negatively (M = 3.61, SD = 0.61), whereas in wt condition the evaluation was more positive (M = 4.02, SD = 0.54). The mean for the cg condition was very similar to kb condition, (M = 3.78, SD = 0.43) with no significant ­difference.

Paired comparisons and the confidence inter­vals support the initial findings (95 % cis, wt [3.84, 4.19], kb [3.40, 3.82] and cg [3.63, 3.93]), confirming that the difference between kb (M = 3.61, SD = 0.61) and cg (M = 3.78, SD = 0.43) was not significant t (70) = 1.37, p = .173, effect size d = 0.332. Contrarily, the difference between kb (M = 3.61, SD = 0.61) and wt (M = 4.02, SD = 0.54) was significant t (75) = 3.10, p = .003, effect size d = 0.712.

Error bars indicating the differences betweenexperimental groups
Figure 1
Error bars indicating the differences betweenexperimental groups


Discussion

The stimuli changed the evaluation of the choc­olate and even though the differences between the control group and the others were not significant, the trends were in the expected direction. Fur­thermore, most of the participants did not remem­ber any information concerning the institutions (92.7%), which strengthens the assumption that the stimuli were subtle enough.

For the second experiment, there were two conditions, wt and kb. In addition to replicating the priming effect in a different context, the aim of Experiment 2 was to research a moderator-aca­demic experience. Presumably, more experienced academics would be less prone to biases, hence less affected by the stimulus.

Experiment 2 - Article Evaluation Task

Participants

During an academic conference in Brazil, 80 participants, mostly doctorate students (63 %) and PhDs (28.4 %), were randomly selected and averaged 5.44 years of academic experience (SD = 4.51), with no significant difference in gen­der (50.6 % male).

Procedures

Participants were asked to evaluate a sci­entific paper lacking any identification, except for a footnote regarding the funding institution - Welgesteld-Tijdschrift (wt Condition) and Ku­ranta-Bothata (kb Condition). After returning the article, they completed an evaluation questionnaire.

Measures

The questionnaire evaluated different aspects of the paper, such as originality, methodology and conclusions. In the first part of the questionnaire, respondents evaluated based on their opinion, and, on the second part, based on how they thought other scientists would evaluate it. The last part asked if they remembered any information on the funding institution.

The scores for each question were averaged, and named General Acceptance Index (gai) and General Acceptance Index – Others (gai-o). Scores ranged from -2 to +2, with -2 being the worst pos­sible evaluation.

Results

Concerning the gai, participants in the wt con­dition evaluated the article more positively than those in the in kb condition (m’s=-.42 and -.05, respectively). A one-way anova confirmed that this difference was significant, F (1, 79) = 11.55, MSE = .39, p = .001, ŋ2 = .13. For the gai-o, the same pattern was found, with wt condition (M=.29) being significantly different from kb (M=-.14); F (1, 79) = 4.49, MSE = .14, p = .037, ŋ2 = .05, effect size d = 0.459.

Error
Bars indicating the differences between experimental groups
Figure 2
Error Bars indicating the differences between experimental groups


Academic experience was measured in years, self-reported in the questionnaire, and a stepwise regression analysis was used to examine it as a moderator of the relation between priming and the gai. Presumptively, as the academic experience in­creases, a confirmatory bias gets stronger, repeating the well-known discriminatory behavior towards European and African origin (de Bruin, Treccani & Della Sala, 2015; Rubinstein & Brenner, 2014)we estimate the effect of a Sephardic sounding surname on wages. We first compare the wages of Israeli Jewish males born to Sephardic fathers and Ashkenazi mothers (SA).

The slope of the regression lines is consistent with academic experience being a moderator, with the regression line for the wt showing a positive slope, while the line for kb Condition showed a negative slope. A multiple regression was conduct­ed, in which the interaction term was inserted, and the result was significant, F(3.76) = 3.71, p < .05, indicating that the model was a good predictor of the gai, even though the total variance explained was relatively low, R2=0.12, which means that even though the model is adequate, there are other variables that need to be also studied.

 Regression
lines for academic experience and GAI for experimental groups
Figure 3
Regression lines for academic experience and GAI for experimental groups


Discussion

Results indicated a consistent difference in evaluation and judgment between the conditions; the footnote appeared to have operated as a prime, and triggered a biased evaluation in both groups, even though it was subtle enough not to be re­membered.

As for the moderating effect of academic expe­rience, the total variance explained was small, but the slope lines suggest that as the years of academic experience grew, the differences on the evaluation given to the article were more expressive, which may be an indication that the bvias effect was stronger, with participants on wt condition giv­ing better evaluations and those on kb condition giving worse ones. Relatively small effects of priming have often been reported, indicating that the effect itself is subtle enough to make it hard to detect (Cesario, 2014).

General Discussion

On both experiments the priming stimuli were sufficient to affect evaluation and judgment, es­pecially considering that over 90 % of the partic­ipants were unable to remember it. The chocolate evaluation (subjective and non-technical) and the article evaluation (more technical) were similarly affected, which strengthens the case for the reli­ability of this effect.

Individual variables could have affected both experiments, such as personal taste in experiment 1 and experience as a reviewer in experiment 2. Both limitations should be addressed in future work. Nonetheless, the change of a mere two words, if these words resemble the origin of the entity being judged, had an effect.

The study of how moderators are related to the effects found would certainly provide import­ant evidence regarding how automatic biases can be primed, hence forth affecting peer reviewing, judgment and evaluation.

Considering the central role of peer reviewing in science, and adding up to the growing literature on publishing biases, prejudice and questionable practices (Button, Bal, Clark & Shipley, 2016; de Bruin et al., 2015; Koole & Lakens, 2012; Wa­genmakers, Dutilh & Sarafoglou, 2018) further work must continue to address those issues, with different and perhaps larger populations, in an effort to understand the automatic biases and its mechanisms, and how those effects could influence behavior, and the publishing field as well.

Specifically, regarding the peer review system, the results do not prove that there is an established bias, but it indicates that a subtly presented stimu­lus (such as two words on a footnote) can trigger a priming process that alters the perception of the subject, hence affecting its judgment towards an object. If we consider that almost all participants (92,3 %) did not remember any information con­cerning the funding institution, even though the priming stimulus was subtle enough not to be re­membered, it did affect their judgment.

In addition, Kuranta-Bothata and Welgesteld-Ti­jdschrift are not even real as research funding insti­tutions, which indicates that not only the subjects were affected by the priming stimulus, but also that they were able to link those words, possibly through spreading activation, to contents that already exist­ed in their cognition. The sounding-like effect of the words “Kuranta- Bothata” and “Welgesteld-Ti­jdschrift” was sufficient to activate pre-existing concepts, consequently affecting the perception and judgment of the paper.

Even though the subjects were not all reviewers, all of them had experience in evaluating papers, since it is part of their daily activities, as either post-graduate students or faculty members. In addition, recent studies relating self-regulatory dynamics, construal level (Fujita & Trope, 2014) and constraint and affection (Schröder & Thagard, 2014) bring forth new possibilities of more robust explanations for the mechanism of priming itself. Also, the study of how moderators are related to the effects found (Wheeler, Petty & Al, 2014) would certainly bring important evidences on how priming works.

The literature on implicit biases have been deepening the understanding on those kind of ef­fects (Malouff & Thorsteinsson, 2016) but further work must address, in different and perhaps larger populations, the limitations early presented, in an effort to understand the effect and its mechanisms.

References

Allen, T. J., Sherman, J. W., & Klauer, K. C. (2010). Social context and the self-regulation of implicit bias. Group Processes & Inter­group Relations, 13, 137-149. Doi: https://doi.org/10.1177/1368430209353635

de Bruin, A., Treccani, B., & Della Sala, S. (2015). Cognitive advantage in bilingualism: an example of publication bias? Psychologi­cal Science, 26(1), 99-107. Doi: https://doi.org/10.1177/0956797614557866

Button, K. S., Bal, L., Clark, A., & Shipley, T. (2016). Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review. BMC Psychology, 4(1), 1–7. Doi: https://doi.org/10.1186/s40359-016-0167-7

Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psycholo­gical Science, 9(1), 40-48. Doi: https://doi.org/10.1177/1745691613513470

Chaxel, A. S., Russo, J. E., & Wiggins, C. (2016). A goal-priming approach to cognitive consistency: Applications to judgment. Journal of Behavio­ral Decision Making, 29(1), 37-51. Doi: https://doi.org/10.1002/bdm.1874

Cokely, E. T., & Feltz, A. (2009). Individual diffe­rences, judgment biases, and theory-of-mind: Deconstructing the intentional action side effect asymmetry. Journal of Research in Personali­ty, 43(1), 18-24. Doi: https://doi.org/10.1016/j.jrp.2008.10.007

Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. (2014). On the other side of the mirror: Pri­ming and social psychology. Social Cognition, 32, 12-32.

Fujita, K., & Trope, Y. (2014). Structured versus uns­tructured regulation: On procedural mindsets and the mechanisms of priming effects. Social Cognition, 32(Supplement), 68-87. Doi: https://doi.org/10.1521/soco.2014.32.supp.68

Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutio­nal prestige on reviewers’ recommendations and editorial decisions. JAMA: The Journal of the American Medical Association, 272(2), 137-138. Doi: https://doi.org/10.1001/ja­ma.1994.03520020063017

Giner-Sorolla, R. (2012). Science or art? How aesthe­tic standards grease the way through the publica­tion bottleneck but undermine science. Perspec­tives on Psychological Science, 7(6), 562-571. Doi: https://doi.org/10.1177/1745691612457576

Grossmann, I., & Jowhari, N. (2018). Cognition and the self: Attempt of an independent close repli­cation of the effects of self-construal priming on spatial memory recall. Journal of Experimental Social Psychology, 74 (September 2017), 65-73. Doi: https://doi.org/10.1016/j.jesp.2017.08.005

Henderson, M. D., & Wakslak, C. J. (2010). Psycho­logical distance and priming: When do semantic primes impact social evaluations? Personality and Social Psychology Bulletin, 36(7), 975-985. Doi: https://doi.org/10.1177/0146167210367490

Hilbert, M. (2012). Toward a synthesis of cogniti­ve biases: How noisy information processing can bias human decision making. Psychologi­cal Bulletin, 138(2), 211-237. Doi: https://doi.org/10.1037/a0025940

Holroyd, J. (2015). Implicit bias, awareness and im­perfect cognitions. Consciousness and Cogni­tion, 33, 511-523. Doi: https://doi.org/10.1016/j.concog.2014.08.024

Ietto-Gillies, G. (2012). The evaluation of research papers in the xxi century. The open peer discus­sion system of the World Economics Associa­tion. Frontiers in Computational Neuroscience, 6(August), 1-7. Doi: https://doi.org/10.3389/fncom.2012.00054

Kaatz, A., Gutierrez, B., & Carnes, M. (2014). Threats to objectivity in peer review: The case of gen­der. Trends in Pharmacological Sciences, 35(8), 371-373. Doi: https://doi.org/10.1016/j.tips.2014.06.005

Kahneman, D. (2003). A perspective on judgment and choice: Mapping bounded rationality. American Psychologist, 58(9), 697-720. Doi: https://doi.org/10.1037/0003-066X.58.9.697

Koole, S. L., & Lakens, D. (2012). Rewarding re­plications: A sure and simple way to improve psychological science. Perspectives on Psycho­logical Science, 7(6), 608-614. Doi: https://doi.org/10.1177/1745691612462586

Lee, C. J., & Schunn, C. D. (2011). Social biases and solutions for procedural objectivity. Hypatia, 26(2), 352-373. Doi: https://doi.org/10.1111/j.1527-2001.2011.01178.x

Loersch, C., & Payne, B. K. (2014). Situated inferen­ces and the what, who, and where of priming. Social Cognition, 32 (Supplement), 137-151. Doi: https://doi.org/10.1521/soco.2014.32.su­pp.137

Malouff, J. M., & Thorsteinsson, E. B. (2016). Bias in grading: A meta-analysis of experi­mental research findings. Australian Journal of Education, 60(3), 245-256. Doi: https://doi.org/10.1177/0004944116664618

Maner, J. K. (2014). Let’s put our money where our mouth is: If authors are to change their ways, reviewers (and editors) must chan­ge with them. Perspectives on Psychologi­cal Science, 9(3), 343-351. Doi: https://doi.org/10.1177/1745691614528215

Marien, H., Custers, R., Hassin, R., & Aarts, H. (2012). Unconscious goal activation and the hijacking of the executive function., 103(3), 399-415. Doi: https://doi.org/10.1037/a0028955

Mohr, C., Koutrakis, N., & Kuhn, G. (2015). Priming psychic and conjuring abilities of a magic de­monstration influences event interpretation and random number generation biases. Frontiers in Psychology, 6 (Jan), 1-8. Doi: https://doi.org/10.3389/fpsyg.2015.00542

Molden, D. C. (2014). Understanding priming effects in social psychology: An overview and integra­tion. Social Cognition, 32 (Supplement), 243-249. Doi: https://doi.org/10.1521/soco.2014.32.supp.243

Morewedge, C. K., & Kahneman, D. (2010). As­sociative processes in intuitive judgment. Trends in Cognitive Sciences, 14(10), 435-440. Doi: https://doi.org/10.1016/j.tics.2010.07.004

Papaioannou, A. G., Machaira, E., & Theano, V. (2013). Fifteen years of publishing in Engli­sh language journals of sport and exercise psychology: Authors’ proficiency in Engli­sh and editorial boards make a difference. International Journal of Sport and Exercise Psychology, 11(1), 1-10. Doi: https://doi.or­g/10.1080/1612197X.2013.753726

Pickering, M. J., McLean, J. F., & Krayeva, M. (2015). Nonconscious priming of communi­cation. Journal of Experimental Social Psycho­logy, 58, 77-81. Doi: https://doi.org/10.1016/j.jesp.2014.12.007

Pleggenkuhle-Miles, E. A., Khoury, T. A. B., Deeds, D. L. C., & Markoczy, L. D. (2013). Explo­ring cognitive bias: Expert ratings of business schools. Management Decision, 51(9), 1905-1927. Doi: https://doi.org/10.1108/MD-12-2012-0877

Poehlman, T. A., Dhar, R., & Bargh, J. A. (2016). Sophisticated by design: The nonconscious influence of primed concepts and atmospheric variables on consumer preferences. Customer Needs and Solutions, 3(1), 48-61. Doi: https://doi.org/10.1007/s40547-015-0051-2

Robert C. Calfee, R. R. V. (2010). Preparing a ma­nuscript for publication in psychology journals: A guide for new authors. Publication Manual of the American Psychological Association, 1-16. Retrieved from: https://www.apa.org/pubs/authors/new-author-guide.pdf

Rubinstein, Y., & Brenner, D. (2014). Pride and prejudice: Using ethnic-sounding names and inter-ethnic marriages to identify labour market discrimination. Review of Economic Studies, 81(1), 389-425. Doi: https://doi.org/10.1093/restud/rdt031

Schröder, T., & Thagard, P. (2014). Priming: Constra­int satisfaction and interactive competition. So­cial Cognition, 32(Supplement), 152-167. Doi: https://doi.org/10.1521/soco.2014.32.supp.152

Shanks, D. R., Newell, B. R., Lee, E. H., Balakrish­nan, D., Ekelund, L., Cenac, Z., … Moore, C. (2013). Priming intelligent behavior: An elusive phenomenon. PLoS ONE, 8(4). Doi: https://doi.org/10.1371/journal.pone.0056515

Smith, E. R., & Mackie, D. M. (2014). Priming from others’ observed or simulated responses. Social Cognition, 32(Supplement), 184-195. Doi: https://doi.org/10.1521/soco.2014.32.supp.184

Valdeci, A., Bastos, S., Pereira, N. A. B. M., Basilio, J., & Leite, V. (2012). Avaliação Sensorial de Chocolate Amargo Formulado com Blend de Diferentes Variedades Clonais. III Congresso Brasileiro do Cacau.

Verlegh, P. W. J. (1999). A review and meta-analysis of country-of-origin research. Journal of Eco­nomic Psychology, 20, 521-546. Doi: https://doi.org/10.1016/S0167-4870(99)00023-9

Wagenmakers, E. J., Dutilh, G., & Sarafoglou, A. (2018). The creativity-verification cycle in psy­chological science: New methods to combat old idols. PsyArXiv Preprints, 1-13. Doi: https://doi.org/10.17605/OSF.IO/37NTP

Wall, M., Liefeld, J., & Heslop, L. A. (1991). Im­pact of country-of-origin cues on consumer judgments in multi-cue situations: a covariance analysis. Journal of the Academy of Marke­ting Science, 19(2), 105-113. Doi: https://doi.org/10.1007/BF02726002

Wheeler, S. C., Petty, R. E., & Al, W. E. T. (2014). Understanding prime-to-behavior effects: Insi­ghts from the active-self account, 32, 109-123.

Wolff, W. M. (1973). Publication problems in psy­chology and an explicit evaluation schema for manuscripts. American Psychologist, 28(3), 257-261. Doi: https://doi.org/10.1037/h0034678

Author notes

Main contact for publishing purposes: Luiz Victorino, Universidade de Brasilia –UnB–. E-mail: luiz.victorino1@gmail.com