Priming and Prejudice: The Bias Effect of Origin Information on Peer Review, Judgment and Evaluation
Priming y prejuicio: el efecto de sesgo de la información de origen el juicio y la evaluación
Priming e preconceito: o efeito de viés da informação de origem no juízo e na avaliação
Priming and Prejudice: The Bias Effect of Origin Information on Peer Review, Judgment and Evaluation
Avances en Psicología Latinoamericana, vol. 37, no. 1, 2019
Universidad del Rosario
Received: 05 May 2017
Accepted: 10 October 2018
To quotes this article: Victorino, L., Pilati R, & Linhares, A. (2019). Priming and Prejudice: The Bias Effect of OriginInformation on Peer Review, Judgment and Evaluation. Avances en Psicología Latinoamericana, 37(1), 169-178. Doi: https://doi.org/10.12804/revistas.urosario.edu.co/apl/a.5635
Abstract: The peer review system is a core tenet of scientific publishing, and has been studied throughout the years. Recent discussions on judgment and evaluation biases have highlighted the importance of priming effects on those processes. We have conducted two experiments with evaluation tasks (a scientific article evaluation, with professors and PhD students as participants, and a chocolate tasting, with undergraduates) in which they were exposed to a footnote acknowledging financial support from a fictional agency. In one condition, the agency’s name was associated with the African continent, while in another condition, it was associated with the European continent. There were statistically significant differences in judgments on both tests, with individuals on the European condition giving better evaluations of the article and the chocolate, even though the stimuli were subtle enough not to be remembered by 92.5 % of all participants. We also found evidence of a moderating effect of academic experience on the priming process.
Keywords Peer review, priming, cognitive bias, judgment.
Resumen: El sistema de revisión por pares es un principio básico de la publicación científica, que ha sido estudiado a lo largo de los años. Las discusiones recientes sobre sesgos en los juicios y las evaluaciones han destacado la importancia de los efectos de priming en esos procesos. Hemos realizado dos experimentos con tareas de evaluación (una evaluación de un artículo científico, con profesores y estudiantes de doctorado como participantes, y una degustación de chocolates, con estudiantes universitarios) en las que estuvieron expuestos a una nota a pie de página que reconocía el apoyo financiero de una agencia ficticia. En una condición, el nombre de la agencia se asoció con el continente africano, mientras que en otra condición, fue asociado con el continente europeo. Hubo diferencias estadísticamente significativas en los juicios en ambas pruebas, tales que los individuos en la condición europea evaluaron mejor el artículo y el chocolate, a pesar de que los estímulos fueron lo suficientemente sutiles como para no ser recordados por el 92.5% de todos los participantes. También encontramos evidencia de un efecto moderador de la experiencia académica en el proceso de priming.
Palabras clave: revisión por pares, priming , sesgos cognitivos, juicio.
Resumo: O sistema de revisão por pares é um princípio básico da publicação científica, e tem sido estudado ao longo dos anos. As discussões recentes sobre viés nos juízos e as avaliações têm destacado a importância dos efeitos de priming nos processos. Temos realizado dois experimentos com tarefas de avaliação (uma avaliação de um artigo científico, com professores e estudantes de doutoramento como participantes, e uma degustação de chocolates, com estudantes universitários) na que estiveram expostos a uma nota de rodapé que reconhecia o apoio financeiro de uma agência fictícia. Em uma condição, o nome da agência associou-se com o continente africano, enquanto que em outra condição, foi associado com o continente europeu. Houve diferenças estatisticamente significativas em juízos em ambas as provas, com indivíduos na condição europeia dando melhores avaliações ao artigo e ao chocolate, apesar de que os estímulos foram o suficientemente sutis como para não serem lembrados pelo 92.5% de todos os participantes. Também encontramos evidência de um efeito moderador da experiência acadêmica no processo de priming.
Palavras-chave: revisão por pares, priming , vieses cognitivos, juízos.
Science stands on the process of peer review. Peer review aims to assure reasonable and fair consideration of scientific papers, for publishing in journals but also for research funding, tenure, prize awarding and so forth. Even though among journals (and institutions) there are small differences in the procedure itself, the process is based on the concept of scientific experts analyzing and scrutinizing studies in their specific field, in order to determine which ones are publishable, how revisions should be conducted, or if publication should be supported by a journal.
Such evaluation is carried out to verify if a study is well conducted, if its methodology is adequate and, obviously, if the results are significant. Since those specialists conducting the review are experienced in publishing in the field, it is reasonable to assume that the process enhances the development of specific knowledge. Nevertheless, the equality and transparency of the process is far from achieving a consensus, and questions concerning evaluation biases in the traditional peer review have been brought to attention (Giner-Sorolla, 2012; Ietto-Gillies, 2012; Kaatz, Gutierrez & Carnes, 2014; Maner, 2014)its role, and the criticisms levelled at it. An analysis of specific problems in economics leads to a full discussion of the Open Peer Review (OPR, and amongst those, bias against author’s origin (Rubinstein & Brenner, 2014; Verlegh, 1999; Wolff, 1973).
There are numerous situations in which stereotyping and judgment are connected, and understanding this relation can improve evaluation processes (Kahneman, 2003; Wall, Liefeld & Heslop, 1991). In the present study, we focus on scientific papers evaluation, and how the stereotype of a low-prestige institution, or of a low-prestige country, can affect evaluation and judgment of a scientific paper. The literature on stereotyping in judgment and evaluation of scientific papers indicates that scientists at prestigious universities tend to have higher rates of publication and higher rates of citation, in part attributable to the prestige of their institutions (Giner-Sorolla, 2012; Ietto-Gillies, 2012; Robert C. Calfee, 2010; Wolff, 1973). Even if we consider that prestigious institutions may have better research centers, superior infrastructure, attract better scientists and that, developed countries usually have well-developed scientific incentive programs and higher investments on science, it is reasonable to think that this will have an impact on their scientific production. On the other side, it could not affect the evaluation of a scientific work, since those aspects are not solely responsible for the quality of a research —also, this kind of judgment bias could easily be defined as stereotyping.
Stereotyping is a cognitive strategy, with evolutionary origins stemming from the beginnings of the human race, to optimize mental processing —after labeling a person, an object, or a situation, it is easier for the brain to identify and react. The interesting fact about stereotyping, and the focus of the present study, is that stereotypes can be activated even when the stimulus presented is unknown, that is, when the brain makes connections with past experiences in order to develop a subsequent coherent response (Shanks et al., 2013). Even when something is shown for the first time, based on our previous experiences, we automatically develop a labeling process which will influence our present and future responses to that stimulus (Loersch & Payne, 2014).
Stereotypes arise from unconscious associations, sometimes through priming effects, which have been attracting more attention to the phenomena in the last years, not only for their potential to influence behavior, but also for their widespread presence. The literature shows that the automatic processing can lead to biases and errors in judgment (Morewedge & Kahneman, 2010) and even though we are aware that this is a possibility, it is hard, if not impossible, for individuals to control the prejudicial effects (Holroyd, 2015).
The literature on priming is extensive on how it can influence the basic processes of evaluation and judgment (Cokely & Feltz, 2009; Henderson & Wakslak, 2010; Smith & Mackie, 2014), even though the cognitive process behind it is not completely described. The standard perspective is that, once a particular stimulus is shown, it activates neural networks through a process called spreading activation —the stimulus activates pieces of information in memory that are related or associated to the content, influencing the response to the stimulus (Molden, 2014), and since the activation of those networks is not conscious, there is no participation of conscious will in the process.
It is well accepted that not only simple and basic responses can be primed, but even extremely complex cognitive processes can be affected nonconsciously through priming, such as goal activation (Marien, Custers, Hassin & Aarts, 2012), observed and simulated responses from others (Smith & Mackie, 2014), and, the focus of the present study, judgment and stereotypes (Allen, Sherman & Klauer, 2010; Rubinstein & Brenner, 2014). Since scientific authors are also victims of prejudice and publication biases (Garfunkel, Ulshen, Hamrick & Lawson, 1994; Lee & Schunn, 2011; Papaioannou, Machaira & Theano, 2013) on the present study, we focus on scientific papers’ evaluation, and how the stereotype of a low-prestige institution, or of a low-prestige country, can affect evaluation and judgment of a scientific paper through a priming effect. Priming is a recurrent phenomenon in social cognition, and recently, interest in how it can influence evaluation and judgment has been increasing (Chaxel, Russo & Wiggins, 2016; Doyen, Klein, Simons & Cleeremans, 2014; Mohr, Koutrakis & Kuhn, 2015). Beyond the examination of priming effects, recent advances are trending towards the possible mediators and moderators (Pickering, McLean & Krayeva, 2015; Poehlman, Dhar & Bargh, 2016), including attributions related to the origin of primed information (Loersch & Payne, 2014). The influence of priming on complex processes can be explained through cognitive biases, which can arise from different sources (Hilbert, 2012; Pleggenkuhle-Miles, Khoury, Deeds & Markoczy, 2013), including culture, (Grossmann & Jowhari, 2018)simultaneously evaluating the role of task-compliance, operationalization specificity, and cross-cultural robustness. In the original study, participants either circled first-person plural (interdependent condition, hence the need to research on and replicate effects of priming on judgment in different cultural groups, such as Brazilian academics.
In the present study, we focus on two different types of evaluation, one presumably more technical (a scientific paper) and the other more subjective and personal (a chocolate tasting), and how the stereotype of origin can affect evaluation and judgment. Our objective is to study if, through a priming effect, subjects will show a bias in evaluation and judgment on both tasks. In addition, to contribute to the growing body of evidence on the role of moderator variables in priming effects, the moderating effect of academic experience will be researched.
The present work is based on two studies using the same stimuli, words on a footnote that imply European or African origins. There is consensus in the literature that European and African origins are related, respectively, to positive and negative representations, leading to stereotyping effects in many domains (Rubinstein & Brenner, 2014). Based on a pre-test, the words chosen were Welgesteld-Tijdschrift (“wealthy magazine” in Dutch), and Kuranta-Bothata (“problematic magazine” in Setswana, a Southern African language).
The experimental design followed the same rationale, which was to subtly introduce the stimuli to the participants, then present the evaluation object (a scientific paper for experiment 1, and a piece of chocolate for experiment 2) and finally assess the biases in evaluation and judgment for the experimental groups. On both experiments, the manipulation check consisted in asking the participants if they could remember any information regarding the presented stimuli.
Experiment 1 – Chocolate Testing
For this experiment, 113 mostly (81.4 %) male graduate students (M = 24.78 years, SD = 7.02) were given an unmarked chocolate, and after tasting, asked to evaluate it using a questionnaire.
The chocolate evaluation questionnaire (Valdeci, Bastos, Pereira, Basilio & Leite, 2012) was answered on a Likert scale, from 1 (very poor) to 5 (excellent) and one question asking if any information on the institution funding the research was remembered, as the manipulation check.
Undergraduates were randomly assigned to three groups and conducted to a room, where they were handed out a consent form before being asked to taste and evaluate a piece of chocolate. The variable manipulated was the information presented in a footnote placed on the informed consent, regarding the institution which funded that research. The institution’s names were Welgesteld-Tijdschrift (wt Condition) and Kuranta-Bothata (kb Condition) with the control group (cg) lacking a footnote.
After tasting the chocolate, the evaluation questionnaire was delivered and answered in the same room, and the participants were let off.
The evaluation scores were summed and the average was considered the General Tasting Score (gts). A one-way anova indicated that the difference was significant, F (2, 112) = 5.641, MSE = .28, p = .005, ŋ2 = .06, with participants in kb Condition showing a tendency to evaluate the chocolate more negatively (M = 3.61, SD = 0.61), whereas in wt condition the evaluation was more positive (M = 4.02, SD = 0.54). The mean for the cg condition was very similar to kb condition, (M = 3.78, SD = 0.43) with no significant difference.
Paired comparisons and the confidence intervals support the initial findings (95 % cis, wt [3.84, 4.19], kb [3.40, 3.82] and cg [3.63, 3.93]), confirming that the difference between kb (M = 3.61, SD = 0.61) and cg (M = 3.78, SD = 0.43) was not significant t (70) = 1.37, p = .173, effect size d = 0.332. Contrarily, the difference between kb (M = 3.61, SD = 0.61) and wt (M = 4.02, SD = 0.54) was significant t (75) = 3.10, p = .003, effect size d = 0.712.
The stimuli changed the evaluation of the chocolate and even though the differences between the control group and the others were not significant, the trends were in the expected direction. Furthermore, most of the participants did not remember any information concerning the institutions (92.7%), which strengthens the assumption that the stimuli were subtle enough.
For the second experiment, there were two conditions, wt and kb. In addition to replicating the priming effect in a different context, the aim of Experiment 2 was to research a moderator-academic experience. Presumably, more experienced academics would be less prone to biases, hence less affected by the stimulus.
Experiment 2 - Article Evaluation Task
During an academic conference in Brazil, 80 participants, mostly doctorate students (63 %) and PhDs (28.4 %), were randomly selected and averaged 5.44 years of academic experience (SD = 4.51), with no significant difference in gender (50.6 % male).
Participants were asked to evaluate a scientific paper lacking any identification, except for a footnote regarding the funding institution - Welgesteld-Tijdschrift (wt Condition) and Kuranta-Bothata (kb Condition). After returning the article, they completed an evaluation questionnaire.
The questionnaire evaluated different aspects of the paper, such as originality, methodology and conclusions. In the first part of the questionnaire, respondents evaluated based on their opinion, and, on the second part, based on how they thought other scientists would evaluate it. The last part asked if they remembered any information on the funding institution.
The scores for each question were averaged, and named General Acceptance Index (gai) and General Acceptance Index – Others (gai-o). Scores ranged from -2 to +2, with -2 being the worst possible evaluation.
Concerning the gai, participants in the wt condition evaluated the article more positively than those in the in kb condition (m’s=-.42 and -.05, respectively). A one-way anova confirmed that this difference was significant, F (1, 79) = 11.55, MSE = .39, p = .001, ŋ2 = .13. For the gai-o, the same pattern was found, with wt condition (M=.29) being significantly different from kb (M=-.14); F (1, 79) = 4.49, MSE = .14, p = .037, ŋ2 = .05, effect size d = 0.459.
Academic experience was measured in years, self-reported in the questionnaire, and a stepwise regression analysis was used to examine it as a moderator of the relation between priming and the gai. Presumptively, as the academic experience increases, a confirmatory bias gets stronger, repeating the well-known discriminatory behavior towards European and African origin (de Bruin, Treccani & Della Sala, 2015; Rubinstein & Brenner, 2014)we estimate the effect of a Sephardic sounding surname on wages. We first compare the wages of Israeli Jewish males born to Sephardic fathers and Ashkenazi mothers (SA).
The slope of the regression lines is consistent with academic experience being a moderator, with the regression line for the wt showing a positive slope, while the line for kb Condition showed a negative slope. A multiple regression was conducted, in which the interaction term was inserted, and the result was significant, F(3.76) = 3.71, p < .05, indicating that the model was a good predictor of the gai, even though the total variance explained was relatively low, R2=0.12, which means that even though the model is adequate, there are other variables that need to be also studied.
Results indicated a consistent difference in evaluation and judgment between the conditions; the footnote appeared to have operated as a prime, and triggered a biased evaluation in both groups, even though it was subtle enough not to be remembered.
As for the moderating effect of academic experience, the total variance explained was small, but the slope lines suggest that as the years of academic experience grew, the differences on the evaluation given to the article were more expressive, which may be an indication that the bvias effect was stronger, with participants on wt condition giving better evaluations and those on kb condition giving worse ones. Relatively small effects of priming have often been reported, indicating that the effect itself is subtle enough to make it hard to detect (Cesario, 2014).
On both experiments the priming stimuli were sufficient to affect evaluation and judgment, especially considering that over 90 % of the participants were unable to remember it. The chocolate evaluation (subjective and non-technical) and the article evaluation (more technical) were similarly affected, which strengthens the case for the reliability of this effect.
Individual variables could have affected both experiments, such as personal taste in experiment 1 and experience as a reviewer in experiment 2. Both limitations should be addressed in future work. Nonetheless, the change of a mere two words, if these words resemble the origin of the entity being judged, had an effect.
The study of how moderators are related to the effects found would certainly provide important evidence regarding how automatic biases can be primed, hence forth affecting peer reviewing, judgment and evaluation.
Considering the central role of peer reviewing in science, and adding up to the growing literature on publishing biases, prejudice and questionable practices (Button, Bal, Clark & Shipley, 2016; de Bruin et al., 2015; Koole & Lakens, 2012; Wagenmakers, Dutilh & Sarafoglou, 2018) further work must continue to address those issues, with different and perhaps larger populations, in an effort to understand the automatic biases and its mechanisms, and how those effects could influence behavior, and the publishing field as well.
Specifically, regarding the peer review system, the results do not prove that there is an established bias, but it indicates that a subtly presented stimulus (such as two words on a footnote) can trigger a priming process that alters the perception of the subject, hence affecting its judgment towards an object. If we consider that almost all participants (92,3 %) did not remember any information concerning the funding institution, even though the priming stimulus was subtle enough not to be remembered, it did affect their judgment.
In addition, Kuranta-Bothata and Welgesteld-Tijdschrift are not even real as research funding institutions, which indicates that not only the subjects were affected by the priming stimulus, but also that they were able to link those words, possibly through spreading activation, to contents that already existed in their cognition. The sounding-like effect of the words “Kuranta- Bothata” and “Welgesteld-Tijdschrift” was sufficient to activate pre-existing concepts, consequently affecting the perception and judgment of the paper.
Even though the subjects were not all reviewers, all of them had experience in evaluating papers, since it is part of their daily activities, as either post-graduate students or faculty members. In addition, recent studies relating self-regulatory dynamics, construal level (Fujita & Trope, 2014) and constraint and affection (Schröder & Thagard, 2014) bring forth new possibilities of more robust explanations for the mechanism of priming itself. Also, the study of how moderators are related to the effects found (Wheeler, Petty & Al, 2014) would certainly bring important evidences on how priming works.
The literature on implicit biases have been deepening the understanding on those kind of effects (Malouff & Thorsteinsson, 2016) but further work must address, in different and perhaps larger populations, the limitations early presented, in an effort to understand the effect and its mechanisms.
Allen, T. J., Sherman, J. W., & Klauer, K. C. (2010). Social context and the self-regulation of implicit bias. Group Processes & Intergroup Relations, 13, 137-149. Doi: https://doi.org/10.1177/1368430209353635
de Bruin, A., Treccani, B., & Della Sala, S. (2015). Cognitive advantage in bilingualism: an example of publication bias? Psychological Science, 26(1), 99-107. Doi: https://doi.org/10.1177/0956797614557866
Button, K. S., Bal, L., Clark, A., & Shipley, T. (2016). Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review. BMC Psychology, 4(1), 1–7. Doi: https://doi.org/10.1186/s40359-016-0167-7
Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9(1), 40-48. Doi: https://doi.org/10.1177/1745691613513470
Chaxel, A. S., Russo, J. E., & Wiggins, C. (2016). A goal-priming approach to cognitive consistency: Applications to judgment. Journal of Behavioral Decision Making, 29(1), 37-51. Doi: https://doi.org/10.1002/bdm.1874
Cokely, E. T., & Feltz, A. (2009). Individual differences, judgment biases, and theory-of-mind: Deconstructing the intentional action side effect asymmetry. Journal of Research in Personality, 43(1), 18-24. Doi: https://doi.org/10.1016/j.jrp.2008.10.007
Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. (2014). On the other side of the mirror: Priming and social psychology. Social Cognition, 32, 12-32.
Fujita, K., & Trope, Y. (2014). Structured versus unstructured regulation: On procedural mindsets and the mechanisms of priming effects. Social Cognition, 32(Supplement), 68-87. Doi: https://doi.org/10.1521/soco.2014.32.supp.68
Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutional prestige on reviewers’ recommendations and editorial decisions. JAMA: The Journal of the American Medical Association, 272(2), 137-138. Doi: https://doi.org/10.1001/jama.1994.03520020063017
Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7(6), 562-571. Doi: https://doi.org/10.1177/1745691612457576
Grossmann, I., & Jowhari, N. (2018). Cognition and the self: Attempt of an independent close replication of the effects of self-construal priming on spatial memory recall. Journal of Experimental Social Psychology, 74 (September 2017), 65-73. Doi: https://doi.org/10.1016/j.jesp.2017.08.005
Henderson, M. D., & Wakslak, C. J. (2010). Psychological distance and priming: When do semantic primes impact social evaluations? Personality and Social Psychology Bulletin, 36(7), 975-985. Doi: https://doi.org/10.1177/0146167210367490
Hilbert, M. (2012). Toward a synthesis of cognitive biases: How noisy information processing can bias human decision making. Psychological Bulletin, 138(2), 211-237. Doi: https://doi.org/10.1037/a0025940
Holroyd, J. (2015). Implicit bias, awareness and imperfect cognitions. Consciousness and Cognition, 33, 511-523. Doi: https://doi.org/10.1016/j.concog.2014.08.024
Ietto-Gillies, G. (2012). The evaluation of research papers in the xxi century. The open peer discussion system of the World Economics Association. Frontiers in Computational Neuroscience, 6(August), 1-7. Doi: https://doi.org/10.3389/fncom.2012.00054
Kaatz, A., Gutierrez, B., & Carnes, M. (2014). Threats to objectivity in peer review: The case of gender. Trends in Pharmacological Sciences, 35(8), 371-373. Doi: https://doi.org/10.1016/j.tips.2014.06.005
Kahneman, D. (2003). A perspective on judgment and choice: Mapping bounded rationality. American Psychologist, 58(9), 697-720. Doi: https://doi.org/10.1037/0003-066X.58.9.697
Koole, S. L., & Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7(6), 608-614. Doi: https://doi.org/10.1177/1745691612462586
Lee, C. J., & Schunn, C. D. (2011). Social biases and solutions for procedural objectivity. Hypatia, 26(2), 352-373. Doi: https://doi.org/10.1111/j.1527-2001.2011.01178.x
Loersch, C., & Payne, B. K. (2014). Situated inferences and the what, who, and where of priming. Social Cognition, 32 (Supplement), 137-151. Doi: https://doi.org/10.1521/soco.2014.32.supp.137
Malouff, J. M., & Thorsteinsson, E. B. (2016). Bias in grading: A meta-analysis of experimental research findings. Australian Journal of Education, 60(3), 245-256. Doi: https://doi.org/10.1177/0004944116664618
Maner, J. K. (2014). Let’s put our money where our mouth is: If authors are to change their ways, reviewers (and editors) must change with them. Perspectives on Psychological Science, 9(3), 343-351. Doi: https://doi.org/10.1177/1745691614528215
Marien, H., Custers, R., Hassin, R., & Aarts, H. (2012). Unconscious goal activation and the hijacking of the executive function., 103(3), 399-415. Doi: https://doi.org/10.1037/a0028955
Mohr, C., Koutrakis, N., & Kuhn, G. (2015). Priming psychic and conjuring abilities of a magic demonstration influences event interpretation and random number generation biases. Frontiers in Psychology, 6 (Jan), 1-8. Doi: https://doi.org/10.3389/fpsyg.2015.00542
Molden, D. C. (2014). Understanding priming effects in social psychology: An overview and integration. Social Cognition, 32 (Supplement), 243-249. Doi: https://doi.org/10.1521/soco.2014.32.supp.243
Morewedge, C. K., & Kahneman, D. (2010). Associative processes in intuitive judgment. Trends in Cognitive Sciences, 14(10), 435-440. Doi: https://doi.org/10.1016/j.tics.2010.07.004
Papaioannou, A. G., Machaira, E., & Theano, V. (2013). Fifteen years of publishing in English language journals of sport and exercise psychology: Authors’ proficiency in English and editorial boards make a difference. International Journal of Sport and Exercise Psychology, 11(1), 1-10. Doi: https://doi.org/10.1080/1612197X.2013.753726
Pickering, M. J., McLean, J. F., & Krayeva, M. (2015). Nonconscious priming of communication. Journal of Experimental Social Psychology, 58, 77-81. Doi: https://doi.org/10.1016/j.jesp.2014.12.007
Pleggenkuhle-Miles, E. A., Khoury, T. A. B., Deeds, D. L. C., & Markoczy, L. D. (2013). Exploring cognitive bias: Expert ratings of business schools. Management Decision, 51(9), 1905-1927. Doi: https://doi.org/10.1108/MD-12-2012-0877
Poehlman, T. A., Dhar, R., & Bargh, J. A. (2016). Sophisticated by design: The nonconscious influence of primed concepts and atmospheric variables on consumer preferences. Customer Needs and Solutions, 3(1), 48-61. Doi: https://doi.org/10.1007/s40547-015-0051-2
Robert C. Calfee, R. R. V. (2010). Preparing a manuscript for publication in psychology journals: A guide for new authors. Publication Manual of the American Psychological Association, 1-16. Retrieved from: https://www.apa.org/pubs/authors/new-author-guide.pdf
Rubinstein, Y., & Brenner, D. (2014). Pride and prejudice: Using ethnic-sounding names and inter-ethnic marriages to identify labour market discrimination. Review of Economic Studies, 81(1), 389-425. Doi: https://doi.org/10.1093/restud/rdt031
Schröder, T., & Thagard, P. (2014). Priming: Constraint satisfaction and interactive competition. Social Cognition, 32(Supplement), 152-167. Doi: https://doi.org/10.1521/soco.2014.32.supp.152
Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Ekelund, L., Cenac, Z., … Moore, C. (2013). Priming intelligent behavior: An elusive phenomenon. PLoS ONE, 8(4). Doi: https://doi.org/10.1371/journal.pone.0056515
Smith, E. R., & Mackie, D. M. (2014). Priming from others’ observed or simulated responses. Social Cognition, 32(Supplement), 184-195. Doi: https://doi.org/10.1521/soco.2014.32.supp.184
Valdeci, A., Bastos, S., Pereira, N. A. B. M., Basilio, J., & Leite, V. (2012). Avaliação Sensorial de Chocolate Amargo Formulado com Blend de Diferentes Variedades Clonais. III Congresso Brasileiro do Cacau.
Verlegh, P. W. J. (1999). A review and meta-analysis of country-of-origin research. Journal of Economic Psychology, 20, 521-546. Doi: https://doi.org/10.1016/S0167-4870(99)00023-9
Wagenmakers, E. J., Dutilh, G., & Sarafoglou, A. (2018). The creativity-verification cycle in psychological science: New methods to combat old idols. PsyArXiv Preprints, 1-13. Doi: https://doi.org/10.17605/OSF.IO/37NTP
Wall, M., Liefeld, J., & Heslop, L. A. (1991). Impact of country-of-origin cues on consumer judgments in multi-cue situations: a covariance analysis. Journal of the Academy of Marketing Science, 19(2), 105-113. Doi: https://doi.org/10.1007/BF02726002
Wheeler, S. C., Petty, R. E., & Al, W. E. T. (2014). Understanding prime-to-behavior effects: Insights from the active-self account, 32, 109-123.
Wolff, W. M. (1973). Publication problems in psychology and an explicit evaluation schema for manuscripts. American Psychologist, 28(3), 257-261. Doi: https://doi.org/10.1037/h0034678
Main contact for publishing purposes: Luiz Victorino, Universidade de Brasilia –UnB–. E-mail: firstname.lastname@example.org