https://orcid.org/0000-0003-4625-7680
Figures
Abstract
In this paper, we enhance the Best-Worst Scaling (BWS) method by incorporating participants’ preference uncertainty into the conventional BWS, known as case 1. In this context, respondents are tasked with making trade-offs among a set of items of interest. Applying this novel extended BWS method to a sample of Argentinian wine consumers (n = 342), we aim to a) provide a more informative elicitation of consumers’ relative preferences for 16 wine attributes; b) identify the level of uncertainty with each of the attributes, exploring differences between the most and least important wine attributes influencing purchasing wine; and c) compare the results of the extended BWS with the standard BWS. Our findings indicate variability in uncertainty levels on the importance of wine attributes when purchasing wine within and across attributes. Moreover, accounting for participants’ preference uncertainty can alter the ranking of preferences obtained through the standard approach. This alteration is due to both accounting for preference uncertainty itself as well as the uncertainty indicator used. Although this approach is a way to mitigate biases associated with respondents’ preference certainty, it is recommended that preference uncertainty heterogeneity is investigated using different indicators.
Citation: Areal FJ, Perez R (2025) Incorporating preference uncertainty in best worst scaling. PLoS ONE 20(1): e0315705. https://doi.org/10.1371/journal.pone.0315705
Editor: Gabriel A. Picone, University of South Florida, UNITED STATES OF AMERICA
Received: February 26, 2024; Accepted: November 29, 2024; Published: January 30, 2025
Copyright: © 2025 Areal, Perez. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data is published in a public repository (zenodo.org) with the title Best Worst Scale under uncertainty The DOI is 10.5281/zenodo.13889329
Funding: The author(s) received no specific funding for this work.
Competing interests: the authors have declared that no competing interests exist.
Introduction
Best-Worst Scaling (BWS) is a survey-based method used to study and measure individual choice (i.e. measure individual preferences of multiple choices) [1]. In BWS, survey respondents are presented with sets of items and asked to identify the “best” (most preferred or most important) and the “worst” (least preferred or least important) within each set. This process is repeated over several choice cards, providing information from which researchers can derive the individual preference for each of the items presented. The relevance of BWS lies in the fact that it forces respondents to make trade-offs between the items being evaluated, and reduces biases associated with the use of traditional rating scales such as Likert scales, which measures people’s attitudes, opinions, or perceptions by asking respondents to rate their level of agreement or disagreement on a multi-point scale (e.g. 5 point scale, 7 point scale), and ultimately provides a measure of the relative strength of item preferences [1, 2].
The use of the BWS method has gained popularity for evaluating an individual’s relative preferences regarding attributes or items, involving the selection of the most and the least attractive characteristics from a set of choices [1, 3]. The BWS method is important and holds appeal due to its capacity to capture trade-offs and preferences within complex decision-making scenarios, outperforming Likert scale ratings [4]. This approach mitigates issues such as response bias and inadequate differentiation among items [5]. Notably, the BWS method’s simplicity in allowing respondents to choose the best and worst options among alternatives has contributed to its widespread application in diverse research domains, overcoming response biases that may be present in approaches such as rating scales [6]. Examples include health care [2, 7], transportation [8], consumer’s ethical beliefs [9], environmental studies [10–12], forest management [13], energy [14], urban planning [15], education [5, 16], public policy [17], and in the analysis of perceptions of risks [18].
Despite the merits of employing the BWS method in capturing an individual’s relative preferences it is important to note that it does not inherently account for preference uncertainty among BWS respondents. This crucial aspect remains unaddressed within the standard framework of the BWS method. Respondents may have incomplete knowledge about their true preferences for product attributes (e.g. attributes for which they may be less familiar such as environmental and social aspects associated with products) and provide uncertain answers to eliciting methods (e.g. best-worst scaling). Consumer uncertainty may stem from a lack of information (i.e. respondents may feel they need more information about the attributes [19]); choice overload (i.e. respondents are presented with too many choices may hamper their intrinsic motivation and confuse them [20, 21]); psychological factors such as fatigue and cognitive biases (e.g. confirmation bias) [22]. Importantly, the use of heuristics, which are used by consumers to overcome lack of information and expertise on a product, can lead to biases when uncertainty is present under decision making [23]. All these aspects can contribute to respondents facing challenges in decision-making, potentially compromising the reliability of elicited preferences. One effective approach to account for preference uncertainty involves assessing the level of knowledge among respondents. In preference analysis, respondents’ knowledge can be divided into subjective and objective knowledge categories. Subjective knowledge pertains to respondents self-assessment of their understanding about the topic of study, typically measured in Likert scales gauging respondents’ belief levels. Objective knowledge involves an external evaluation through the testing of respondents’ knowledge via a set of questions related to the study’s subject matter. Conversely, although in applied preference analysis, respondents’ subjective and objective knowledge is typically not taken into account [24], with some exceptions [25, 26], preference uncertainty has been addressed by incorporating follow up questions that inquire about respondents’ certainty regarding their stated preferences (e.g. using a scale, such as a 10-point scale) [24, 27–29]. Neglecting preference uncertainty can result in an overestimation of respondents’ willingness to pay, particularly in non-market valuation methods such as contingent valuation and choice experiments [30, 31]. Therefore, understanding and addressing consumer uncertainty in the methods used to assess an individual’s preferences is crucial for obtaining robust preference elicitation.
This paper introduces and implements a method designed to address preference uncertainty within the context of BWS responses. We employ this extended BWS approach in a study involving Argentinian wine consumers, gathering data on their attribute preferences and associated preference uncertainty using a questionnaire. This paper contributes by 1) devising an extended BWS method to enhance understanding of trade-offs and respondent preferences in complex decision-making scenarios; and 2) evaluating the influence of preference uncertainty in attribute/option selection on respondents’ preferences, comparing results obtained from standard BWS with the extended BWS in a sample of Argentinian wine consumers.
The remainder of the paper is structured as follows. We begin by introducing the extended BWS method, detailing the data collection process, and outlining the experimental design. The following section presents and discusses the results, with the last section concluding.
Data and methods
The extended BWS
A variety of approaches, methods and techniques have been used to study consumer wine preferences encompassing sensory analysis; statistical examination of survey data, often through questionnaires and choice experiments; as well as qualitative methods like focus groups, and in-depth interviews. The literature extensively deploys hedonic price models to elucidate the impact of various wine attributes, such as wine variety and quality, grape type, and region of origin. Consequently, the associations between wine consumers’ characteristics and their preferences have been explored through different lenses, including the emotion-based perspective [32, 33]; consumers’ personality traits [34]; wine attitudinal and behavioural differences of consumers differences [35]; frequency of use [36]; and demographic differences [37]. In addition, research has also been conducted on consumers’ perceptions of wine attributes such as production origin, and the sustainability of wine production [38, 39].
To assess the Argentinian consumer preferences for a set of Argentinian wine characteristics, we employed and compared two approaches. We used a) a simple standard BWS approach, known as case 1, where respondents are required to make trade-offs between a set of items of interest [1, 3]; and b) a novel extended BWS, which was developed to account for respondent’s preference uncertainty. A consumer’s relative preferences are revealed through their response choices. The theoretical framework which underlies the analysis of BWS data is Random Utility Theory (RUM), which is widely used in the economic and econometric literatures. RUM assumes that consumers select between alternatives that maximise their utility [1, 40].
Louviere and Flynn [1] show that summarising standard best and worst data can be done by obtaining a simple scale that subtracts the worst count for each attribute (i.e. the sum of every time each attribute has been selected as worst) from the best count for that attribute (i.e. the sum of every time each attribute has been selected as best).
(1)
where a refers to an attribute, with a = 1, …, 16; Bia takes a value of 1 if attribute a is selected as “the most important” by respondent i and 0 otherwise; Wia takes a value of 1 if attribute a is selected as “the least important” by respondent i and 0 otherwise.
We account for respondents’ uncertainty in their choices by asking them “On a scale 0–100 how sure are you that you have selected the best attribute?” and “On a scale 0–100 how sure are you that you have selected the worst attribute?” after each of the choice cards are presented to them and they have made their selections.
(2)
where α represents the level of certainty that respondent i has in their selection of “the most important” attribute from each choice card; and ω represents the level of certainty that respondent i has in their selection of “the least important” attribute from each choice card. The values of α and ω range between 0 and 1 and are obtained by dividing the value given by respondents on the 0–100 scale by 100.
In order to account for uncertainty, we use a respondent’s self-reported uncertainty about the choices made in each of the choice cards presented to them.
Control vs. treatment group
We split the sample into one control group (n = 185) and one treatment group (n = 158), enabling us to examine whether making respondents aware of their uncertainty in answering questions had any effect on their BWS attribute scores and ranking. For this, we compared the standard BWS attribute scores between the treatment and control groups. The treatment group was presented with 16 choice cards, with 6 attributes in each. After choices were made for each choice card, participants were asked to self-report how sure they were of the choices made (i.e. most and least important attribute). The follow-up questions that respondents in the treatment group were asked after each choice card was shown and selections were made were “On a scale 0–100 how sure are you that you have selected the best attribute?” and “On a scale 0–100 how sure are you that you have selected the worst attribute?”. The control group was presented with 16 choice cards with 6 attributes in each, according to Tables 3, 4, with no follow-up questions made on how certain they were on the choices made in the choice cards presented.
We calculated and compared the BWS attribute scores (i.e. the preference (un)certainty scores) and rankings for the standard BWS and extended BWS that accounts for (un)certainty for each of the 16 attributes considered. We focus on the preference (un)certainty median scores for the most and least important attributes in consumers’ decision-making, since preference (un)certainty distributions are not normally distributed, with the median being less influenced by extreme values, thus providing a more robust measure of central tendency. To compare standard and extended BWS scores, we use the sign test for the median This test is a conservative non-parametric test that imposes minimal assumptions on the data distribution. Although we focus on the preference (un)certainty median scores, we also present the results for the preference (un)certainty mean scores for the most and least important attributes in consumers’ decision-making for completeness, and test whether the preference (un)certainty distributions differ using the non-parametric Wilcoxon rank sum test (also known as the Mann-Whitney U test).
The treatment group is divided into two subgroups: a) participants who are relatively more certain about their wine attribute preferences; and b) participants who are relatively less certain, to examine any differences which may exist in the BWS scores and rankings between these subgroups and the control group. It is expected that respondents belonging to the less certain group may state different preferences than the relatively more certain group since their relative uncertainty about choices may cause them to be indecisive. Rankings may be altered compared to the more certain participants, given this indecisiveness. Finding such differences would suggest that heterogeneity in respondents’ level of certainty about their choices matters in attribute preference elicitation. There are two ways in which we measure the relative certainty level of group participants: a) calculating the certainty median of the individual’s average certainty across wine attributes, dividing the sample into those above and below the median of their average certainty across the attributes; and b) calculating the individual’s coefficient of variation:
Questionnaire and data collection
The questionnaire included an introduction to the questionnaire where all participants were presented with the BWS task as follows–“Imagine you are in your favourite wine shop to buy a bottle of wine to drink at home with family and/or friends. You will then be shown a series of tables with 6 attributes or characteristics of the wine in each table. Please select the characteristic that would influence you the most and the characteristic that would influence you the least when considering the purchase of a bottle of wine in the shop. Some attributes or characteristics shown may not be common but please consider all the characteristics as if they were possible”. The questionnaire also included the BWS experiments (control and treatment groups with follow-up questions on preference uncertainty) and socio-demographic characteristics (gender, income, age, and education level).
Snowball sampling, a popular non-probability convenience sampling technique [41], was conducted via an electronic questionnaire using Qualtrics during January and February, 2023. We previously conducted a small survey to investigate wine consumer preferences in March 2022 and piloted the questionnaire in December 2022. Participants were invited to participate through social media platforms, where initial participants recruit others, creating a “snowball” effect. Leveraging social networks allows for amplifying the reach of our survey to respondents, providing access to individuals who would not easily be identifiable through random sampling [42], as well as accessing geographically dispersed populations [43]. We advertised on WhatsApp and Twitter to ensure the inclusion of diverse perspectives [44]. A brief explanation of the research was provided to participants with a link to the questionnaire. A total of 343 valid responses were used in the analysis. We recorded 740 responses but 397 were incomplete (i.e. respondents started but did not finish the questionnaire).
Generalisability of our results is limited given the selection bias inherent in the initial snowball sample that relied upon our personal network [45]. Nonetheless, it is noteworthy that the resulting sample includes all age groups, both genders (Table 1), and various geographical locations (Table 2). We acknowledge that the sample may underrepresent the lower income group and overrepresent the highest income group.
The sample includes a comprehensive representation from diverse regions, as illustrated in Table 2 which compares our sample with the most relevant available 2022 census data. It is noteworthy that we have gathered information from 14 provinces. Within the Gran Buenos Aires area, the sample encompasses 19 municipalities.
Experimental design
The experimental design consisted of 16 choice cards presenting 16 different options for each respondent (i.e. respondents were shown 16 sets of 6 attributes). Respondents are asked to select their most and least preferred wine attribute within each set that they consider. The selection of the 16 options was based on literature review and a small consumer survey conducted to explore individual preferences (Table 3). Note that attributes 7 and 12 are not true attributes, but respondents’ attributes to the particular wine. Regarding the literature on consumers’ preferences for wine attributes Stanco, Lerro [46] and Casini, Corsi [47] studied the wine attribute preferences of Italian consumers. Casini and Corsi [47] found that direct, personal, and sensorial experiences are the most important attributes when choosing wine. Stanco and Lerro [46] found that traditional attributes such as “geographical indications”, “grape variety”, “sustainable certification”, “vintage”, and “price”, are more important for consumers than novel attributes such as “canned wine”, “alcohol-free wine”, and “vegan wine”. Nunes and Madureira [48] found that “tested the wine previously”, “recommended by friends/relatives” and “region of origin” were the preferred attributes for the sample of Portuguese wine buyers studied.
We used a Balanced Incomplete Block Design (BIBD) to construct a number of choice cards presented to Argentinian wine consumer respondents to assess their preferences for the set of 16 Argentinian wine characteristics or attributes.
A BIBD is a type of experimental design used when it is not practical to test all possible combinations of conditions. It assigns each attribute to various choice cards. This type of experimental design ensures that all attributes are compared in a balanced way (i.e. each attribute appears the same number of times across all choice cards), with not all attributes being present in each of them. Hence, BIBD is a) balanced since each attribute is paired with every other attribute an equal number of times across all choice cards; b) incomplete because not all attributes appear in every choice card; and c) block design since each choice card is a “block” that includes a subset of attributes.
To conduct BIBD the attributes need to be firstly identified and assigned into choice cards (blocks) so that each attribute appears the same number of times across all choice cards and every pair of attributes appears together in the same choice card an equal number of times.
Let J be the list of 16 wine attributes, and denote the individual attributes on the list by j = 1, 2, …, J. Let denote k and b as the fixed size of a choice card (i.e. number of attributes in a choice card) and the number of choice cards (blocks), respectively. Each attribute is shown r times and co-occurs with each other attribute λ times. To assign attributes to choice cards attributes can be numbered from 1 to J. BIBDs generate b choice cards from which respondents make their evaluations (i.e. select their most preferred and least preferred attribute). BIBDs are available in a variety of sources [1, 55], but also can be generated using multiple statistical packages (e.g. bwsTools, and AlgDesign for R) [56–58]. Here we used the AlgDesign package for R [56] to generate a BIBD (J = 16, b = 16, r = 6, k = 6, λ = 2)-design. This means that our BIBD consisted of 16 choice cards with 6 attributes in each choice card, where each attribute appears equally often (6 times); and they concur twice with other attributes (Table 4). We selected a BIBD with 16 choice cards and 6 options because, with 16 attributes, this design was the one with the least number of choice cards possible (easing the respondents’ burden of going through too many choice cards) whilst dealing with a reasonable number of choices within each choice card (6).
The case 1 of the best-worst scaling method presents respondents with a set of 16 choice cards, each showing 6 wine attributes, and asked them to select the best and worst attributes based on their preferences. Fig 1 shows one of the choice cards used in this study.
Inclusivity in global research
The Newcastle University Ethics Committee granted the ethical approval for this research. Participants were informed that the aim of the research was to learn about Argentinean consumer’s preferences in Argentinean wine consumption. They were encouraged to express their views freely, were assured that their views would be valued, and that there were no right or wrong answers to the questions asked. They were informed that we did not collect any names or personal details. Participants were asked to voluntarily participate in the survey, and were made aware that they could withdraw from the survey at any point. They were informed that by completing the survey, they acknowledge they understand the terms and conditions of participating in the study, and that they accepted these terms. Additional information regarding the ethical, cultural, and scientific considerations specific to inclusivity in global research is included in the Supporting Information (S1 Checklist).
Results and discussion
This section presents three type of results including those associated with the level of certainty respondents have on their selection of the most and least important wine attributes when purchasing a bottle of wine; results showing the most and least important wine attributes for the sample of Argentinian wine consumers and the score and ranking comparisons between the control and treatment groups, their relative level of (un)certainty, as well as between the use of the standard and extended BWS approach; and the results sensitivity to uncertainty indicators. Finally the section points out the limitations of the study.
Our results indicate that the most important wine attributes are grape variety, previous recommendations, prior testing, price, and the Argentinian region of origin. We also found that individuals are generally certain about what attributes are least and most important when purchasing wine. However, they tend to be more certain about what attributes are least important than those that are most important. We also found that when integrating uncertainty into the BWS analysis, both BWS scores and rankings vary in comparison to the standard BWS analysis. When incorporating uncertainty, scores and rankings also vary depending on the method used to measure uncertainty.
Level of certainty respondents have on their selection the of most and least important wine attributes
Overall, respondents were relatively certain about the choices made. The certainty median scores for the least and most preferred attributes were 99 and 92, respectively (i.e. 50% of respondents have 99% and 92% levels of certainty on what their least and more preferred attributes are, respectively). However, many respondents were uncertain or very uncertain about their choices. Thus, 9.4% and 5.1% of respondents were 50% or less certain about the least and most important attribute chosen, respectively. A two-sided sign test indicates a statistical difference between the median of the certainty score of attributes selected as least important and the median of the certainty score of attributes selected as most important (p<0.01). We found that out of the 2,528 pairs of observations (158 participants in the treatment group were shown 16 cards), there were 777 pairs (30.7%) where the level of certainty about the least important attribute was greater than the level of certainty about the most important attribute, 661 (26.2%) where the level of certainty about the least important attribute is less than the level of certainty about the most important attribute, and 1,090 (43.0%) where the level of certainty about the least important attribute is the same than the level of certainty about the most important attribute. In Fig 2, the density plots illustrate the level of certainty associated with all attributes identified as both most important and least important in the decision-making process regarding the purchase of a bottle of wine.
Fig 3 shows the density plots of the level of certainty associated with each of the attributes identified as both most important and least important in the decision-making process regarding the purchase of a bottle of wine. Additionally, Fig 3 shows the two-sided sign test. The findings reveal variation in the median level of certainty among respondents regarding the attribute selected as both most and least important across the sample. While a majority of participants exhibit a relatively high certainty level in their choices, a significant portion still expresses notable uncertainty, as evidenced by the extended left tail of the distributions.
Participants exhibit a tendency to assign a higher level of certainty when identifying the least important attributes compared to the most important ones. However, it is noteworthy that the scores for most important attributes surpass those assigned to least important attributes. Thus, the average certainty score for most and least important attributes are 87.2 and 85.8, respectively. More specifically, the top five attributes that respondents were least certain about when identifying them as the most important attribute are: grape variety, wine price, whether they had tasted the wine before, the Argentinian region of origin, and the type of closure. The top 5 attributes that respondents were least certain about when selected as the least important attribute are grape variety, wine has been recommended, alcohol content, Argentinian region of origin and ageing of wine.
Table 5 shows that BWS scores and ranking per attribute obtained using the standard approach for the control and treatment groups as well as the extended approach for the treatment group. Our findings indicate two effects: a) when integrating uncertainty (treatment group) into the analysis, specifically by including self-reported uncertainty information through additional follow-up questions for each BWS card in the extended BWS approach, both scores and rankings may vary in comparison to the control group, where uncertainty information is not acquired; and b) scores and rankings may also exhibit variations due to the BWS method employed, the extended BWS approach or the standard BWS. The first effect may be due to treatment group respondents becoming more aware of their level of certainty and thinking more carefully about their choices. However, for this particular sample, such variations are relatively small. Alterations in ranking are dependent on the proximity of the standard BWS scores and the level of certainty respondents have on the choices they make.
Most and least important wine attributes when purchasing wine
We ranked wine attributes from most to least important using the best worst scaling scores through both the standard BWS and the extended BWS (taking into account the level of (un)certainty).
In the context of Argentinian consumer preferences for wine purchasing decisions, the top five most crucial attributes, ranked in descending order, include grape variety, previous recommendations, prior testing, price, and the Argentinian region of origin. Notably, four out of these five attributes are associated with a relatively lower respondent certainty. Previous results on grape variety are mixed with some research indicating it as an important attribute (53) whereas other research found it relatively less important (42, 49). Having tasted the wine previously is usually a top attribute for wine consumers across different countries (43, 51). The region of origin has also been found to be important for French Spanish, Italian and Australian wine consumers (42, 43, 49, 51, 53). Recommendations by friends and relatives were also found as important attributes in consumers wine selection for Spanish and Australian consumers (49, 53), whereas price was found to be important for Spanish consumers (49). The 5 least important attributes in wine purchasing decision-making are the alcohol content, type of closure, attractiveness of the label, the wine having a certificate of rural sustainability, and the wine having a certificate of environmental sustainability. Alcohol content was found to be the least important attribute in wine choice across different countries (50, 51). The extrinsic wine attributes such as the type of closure, labelling, knowing about how the grapes have been harvested, rural and environmental sustainability certification, being on promotion, information on the back label, quality awards and the brand name are all found to be relatively less important attributes when selecting wine for respondents. Being on promotion, back label, quality awards and brand name were also found to be relatively less important for Italian, Australian and Israeli consumers (42, 53).
Sensitivity to uncertainty indicators
Collecting information on the respondents’ level of certainty allowed us to group respondents according to their relative level of certainty (e.g. relatively more certain vs. relatively less certain) on their most preferred attribute.
Tables 6 and 7 show differences in score and ranking between relatively more and less certain respondents on their most preferred attribute using a) the certainty median of the individual’s average certainty across attributes and b) the coefficient of variation as indicators of respondents’ level of certainty, respectively. When using the median of individual’s average certainty across attributes to group respondents, we found that the grape variety, the wine being recommended and having tested the wine before come as the most important attributes for both groups, whereas the alcohol content comes as the least important attribute. When using the median of the individual certainty’s coefficient of variation to group respondents into relatively more and less certain the grape variety, the wine being recommended and having tested the wine before also come as the top 3 attributes, although there are slight differences between less and more certain groups when using the standard BWS compared to the extended BWS. Alcohol content also came as the least important attribute when using the individual certainty’s coefficient of variation to group respondents into relatively more and less certain. Differences in scores and rankings were found between groups and indicators used of respondents’ level of certainty. For instance, wine promotion in the shop/supermarket is an attribute that is relatively important for those who are more certain about their responses (rank 5th) as opposed to those who are less certain on their responses (rank 10th) when using the certainty median as an indicator. Also, the wine been awarded a quality medal/award was relatively more important for more certain respondents (rank 6th) than for less certain respondents (10th). However, when comparing results shown in Tables 6 and 7 differences in ranking and score are found to be sensitive to the method used to distinguish between relatively more and less certain on their attributes preferences. For example, the Argentine region of origin is a relatively more important wine attribute for respondents who are more certain about their preferences than for those who are less certain about their preferences, when using the individual’s average certainty across attributes to group respondents into relatively more and less certain about their relative preferences. However, when using the median of the certainty’s coefficient of variation to group respondents into more and less certain, Argentine region of origin is relatively more important for respondents who are less certain than for those who are more certain about their preferences. This means that the use of different uncertainty indicators may lead to different preference rankings, which can have important implications for decision making (e.g. marketing strategies). Different uncertainty indicators capture/represent different ways to interpret preference uncertainty. Hence, it is important to use a range of indicators to present these different interpretations of preference uncertainty. Decision makers then have a broader understanding of individuals’ preferences and their uncertainty about them, which can help to take informed decisions. It is worth noting that it is possible that the differences in most preferred attributes between less and more certain respondents may be due to factors that determine their expressed level of confidence. For instance, maybe people who are confident in their answers are more frequent consumers and maybe for such consumers, the relevant attributes are different than for those than are less frequent wine consumers.
Limitations
Despite the valuable insights provided in this study, several limitations must be acknowledged. First, the non-random sample used may not be fully representative of the population of Argentinian wine consumers, which could limit the generalisation of the findings. Hence, cautions should be exercised when extrapolating our findings in other contexts. While the sample may not be fully representative of the broader population, the findings remain relevant and provide valuable insights specific to the group studied. These results offer useful information that can inform future research. Secondly, the attributes were presented to the respondents in the order that are listed in Table 4, without randomisation of the order, which can create some biases if the order in which they are presented impacts the respondents’ answers. This study shows one way to account for an individual’s preference uncertainty in their reported preferences. In expression (2), we assumed that individuals can measure their level of uncertainty in their choices precisely. However, individuals could also be uncertain about their stated level of uncertainty, which could impact the results obtained.
Conclusions
A novel approach to account for preference uncertainty using BWS is presented and applied to a sample of Argentinian wine consumers. The approach extends the standard BWS method by including a respondent’s preference (un)certainty on attributes. This approach is shown to yield different scores and rankings of attributes compared to the standard approach when accounting and not accounting for preference (un)certainty through the use of self-reported uncertainty follow up questions.
Although most respondents may be certain about their responses, there may always be respondents who are uncertain about their preferences and, in particular, regarding some individual attributes. Not accounting for an individual’s preference uncertainty may lead to biases and inaccurate representations of true preferences and poor forecasting of decision-making. The approach presented here offers a way to account for preference uncertainty and mitigate the issues it causes. We advocate for the use of this approach to evaluate the impact of preference uncertainty in decision-making since it may make respondents more focused on the task.
We found that attribute preference uncertainty may vary across attributes. The extension proposed here allows for the accounting of (un)certainty by adding follow-up questions for each of the choice cards presented to respondents. Accounting for attribute preference uncertainty may alter the BWS ranking of preferences and, therefore, the recommendations made. This new method offers researchers a way to mitigate biases associated with respondents’ preference (un)certainty. In addition, it is recommended that preference uncertainty heterogeneity is investigated using different indicators.
We use self-reported (un)certainty, asking participates to indicate their level of uncertainty on the choices made on a 0–100 scale. Other alternative (un)certainty indicators could be used. For instance, response times, where longer response times may suggest respondents needed more time to think, indicating uncertainty, other scales (e.g. Likert scales) that then will need to be transformed into weighting parameters α, ω bounded between 0 and 1 for the most (least) preferred respectively. The transformation could be conducted using normalisation methods (e.g. max-min scaling). Although we applied this method to the simplest BWS case (case 1), similar approaches can be developed for case 2 and case 3 as well as other choice experiments to mitigate bias associated with preference uncertainty.
The findings in this study also have policy implications. The use of the extended BWS proposed here can have relevant policy implications in areas such as public health, environmental policy and transportation, for example, by providing more accurate representations of an individual’s true preferences and more nuanced, adaptive policy interventions. The approach and findings presented here indicate that rigid, one-size-fits-all policies may not be appropriate in many situations. With the use of the method presented here, policy makers can use the knowledge of individuals’ preference uncertainty to design more flexible, adaptive, and inclusive policies that better reflect the complexity around individual preferences. The use of information provided by this approach by policy makers could enhance policy effectiveness and public acceptance.
Acknowledgments
We would like to express our sincere gratitude to Dr. Matthew Shannon for his invaluable assistance in editing this manuscript. His thoughtful feedback and meticulous attention to detail greatly improved the clarity and quality of the final version.
References
- 1. Louviere JJ, Flynn TN, Marley AAJ. Best-Worst Scaling: Theory, Methods and Applications: Cambridge University Press; 2015.
- 2. Flynn TN, Louviere JJ, Peters TJ, Coast J. Best–worst scaling: What it can do for health care research and how to do it. Journal of Health Economics. 2007;26(1):171–89. pmid:16707175
- 3. Finn A, Louviere JJ. Determining the Appropriate Response to Evidence of Public Concern: The Case of Food Safety. Journal of Public Policy & Marketing. 1992;11(2):12–25.
- 4. Burton N, Burton M, Rigby D, Sutherland CAM, Rhodes G. Best-worst scaling improves measurement of first impressions. Cognitive Research: Principles and Implications. 2019;4(1):36. pmid:31549257
- 5. Huybers T. Student evaluation of teaching: the use of best–worst scaling. Assessment & Evaluation in Higher Education. 2014;39(4):496–513.
- 6.
Soutar GN, Sweeney JC, McColl-Kennedy JR. Best-worst scaling: an alternative to ratings data. In: Marley AAJ, Louviere JJ, Flynn TN, editors. Best-Worst Scaling: Theory, Methods and Applications. Cambridge: Cambridge University Press; 2015. p. 177–88.
- 7. Tatar O, Haward B, Zhu P, Griffin-Mathieu G, Perez S, Zimet G, et al. Using Best-Worst Scaling to investigate younger adult Canadians’ preferences for COVID-19 vaccination and public health measures: An observational study. Preventive Medicine Reports. 2022;26:101755. pmid:35284212
- 8. Tran V, Nhu Phan QT, Hai Nguyen HT, Trinh AT. Airport low-cost terminals in Vietnam: Using best-worst scaling to identify the salient factors affecting the investment decision. Transportation Research Procedia. 2021;56:127–35.
- 9. Auger P, Devinney TM, Louviere JJ. Using Best–Worst Scaling Methodology to Investigate Consumer Ethical Beliefs Across Countries. Journal of Business Ethics. 2007;70(3):299–326.
- 10. Dumbrell NP, Kragt ME, Gibson FL. What carbon farming activities are farmers likely to adopt? A best–worst scaling survey. Land Use Policy. 2016;54:29–37.
- 11. Rolfe J, Rajapaksa D, De Valck J, Star M. Will greenhouse concerns impact meat consumption? Best-worst scaling analysis of Australian consumers. Food Quality and Preference. 2023;104:104755.
- 12. Jones AK, Jones DL, Edwards-Jones G, Cross P. Informing decision making in agricultural greenhouse gas mitigation policy: A Best–Worst Scaling survey of expert and farmer opinion in the sheep industry. Environmental Science & Policy. 2013;29:46–56.
- 13. Loureiro ML, Dominguez Arcos F. Applying Best–Worst Scaling in a stated preference analysis of forest management programs. Journal of Forest Economics. 2012;18(4):381–94.
- 14. Aruga K, Bolt T, Pest P. Energy policy trade-offs in Poland: A best-worst scaling discrete choice experiment. Energy Policy. 2021;156:112465.
- 15. Larranaga AM, Arellana J, Rizzi LI, Strambi O, Cybis HBB. Using best–worst scaling to identify barriers to walkability: a study of Porto Alegre, Brazil. Transportation. 2019;46(6):2347–79.
- 16. Burke PF, Schuck S, Aubusson P, Buchanan J, Louviere JJ, Prescott A. Why do early career teachers choose to remain in the profession? The use of best–worst scaling to quantify key factors. International Journal of Educational Research. 2013;62:259–68.
- 17. Caputo V, Lusk JL. What agricultural and food policies do U.S. consumers prefer? A best–worst scaling approach. Agricultural Economics. 2020;51(1):75–93.
- 18. Erdem S, Rigby D. Investigating Heterogeneity in the Characterization of Risks Using Best Worst Scaling. Risk Analysis. 2013;33(9):1728–48. pmid:23339686
- 19. He S, Rucker DD. How uncertainty affects information search among consumers: a curvilinear perspective. Marketing Letters. 2023;34(3):415–28.
- 20. Iyengar SS, Lepper MR. When choice is demotivating: Can one desire too much of a good thing? Journal of personality and social psychology. 2000;79(6):995. pmid:11138768
- 21. Lehmann DR. Customer reactions to variety: Too much of a good thing? Journal of the Academy of Marketing Science. 1998;26(1):62–5.
- 22. Jerath K, Ren Q. Consumer Rational (In)Attention to Favorable and Unfavorable Product Information, and Firm Information Design. Journal of Marketing Research. 2020;58(2):343–62.
- 23. Tversky A, Kahneman D. Judgment under Uncertainty: Heuristics and Biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science. 1974;185(4157):1124–31.
- 24. Ready RC, Champ PA, Lawton JL. Using respondent uncertainty to mitigate hypothetical bias in a stated choice experiment. Land Economics. 2010;86(2):363–81.
- 25. Martinez-Ribaya B, Areal FJ. Is there an opportunity for product differentiation between GM and non-GM soya-based products in Argentina? Food Control. 2020;109:106895.
- 26. Rodriguez L, Kulpavaropas S. Factors Influencing U.S. Consumers’ Preference for Positively Versus Negatively Framed GM Food Symbols. Journal of Agricultural & Food Information. 2018;19(1):75–96.
- 27. Champ PA, Bishop RC, Brown TC, McCollum DW. Using donation mechanisms to value nonuse benefits from public goods. Journal of environmental economics and management. 1997;33(2):151–62.
- 28. Champ PA, Bishop RC. Donation payment mechanisms and contingent valuation: an empirical study of hypothetical bias. Environmental and resource economics. 2001;19(4):383–402.
- 29. Li C-Z, Mattsson L. Discrete Choice under Preference Uncertainty: An Improved Structural Model for Contingent Valuation. Journal of Environmental Economics and Management. 1995;28(2):256–69.
- 30. Akter S, Bennett J. Preference uncertainty in stated preference studies: facts and artefacts. Applied Economics. 2013;45(15):2107–15.
- 31. Zheng D, Ao C, Mao B, Xu L. Can respondent uncertainty be ignored when exploring heterogeneity in public preferences for air pollution treatment policies? Comparative results from choice experiment analysis. Science of The Total Environment. 2022;808:152169. pmid:34875324
- 32. Calvo-Porral C, Lévy-Mangin J-P, Ruiz-Vega A. An emotion-based typology of wine consumers. Food Quality and Preference. 2020;79:103777.
- 33. Mora M, Dupas de Matos A, Vázquez-Araújo L, Puente V, Hernando J, Chaya C. Exploring young consumers’ attitudes and emotions to sensory and physicochemical properties of different red wines. Food Research International. 2021;143:110303. pmid:33992323
- 34. Burro R, Branchini E, Capitani E, Barnaba V, Fermani A, Paradis C, et al. Is there an association between consumers’ personality traits and the sensory characteristics they look for in wine? Food Quality and Preference. 2023;105:104767.
- 35. Thach L, Olsen J. Profiling the high frequency wine consumer by price segmentation in the US market. Wine Economics and Policy. 2015;4(1):53–9.
- 36. Goldsmith RE, d’Hauteville F. Heavy wine consumption: empirical and theoretical perspectives. British Food Journal. 1998;100(4):184–90.
- 37. Wolf MM, Higgins LM, Wolf MJ, Qenani E. Do generations matter for wine segmentation? Journal of Wine Research. 2018;29(3):177–89.
- 38. Tait P, Saunders C, Dalziel P, Rutherford P, Driver T, Guenther M. Estimating wine consumer preferences for sustainability attributes: A discrete choice experiment of Californian Sauvignon blanc purchasers. Journal of Cleaner Production. 2019;233:412–20.
- 39. Rodríguez-Donate MC, Romero-Rodríguez ME, Cano-Fernández VJ, Guirao-Pérez G. Analysis of heterogeneity in the preferences of wine consumption. Wine Economics and Policy. 2019;8(1):69–80.
- 40. Janssen M, Hamm U. Product labelling in the market for organic food: Consumer preferences and willingness-to-pay for different organic certification logos. Food Quality and Preference. 2012;25(1):9–22.
- 41.
Johnson TP. Snowball Sampling: Introduction. Wiley StatsRef: Statistics Reference Online2014.
- 42.
Parker C, Orcid, S., & Geddes, A. Snowball Sampling. SAGE Research Methods Foundations 2020.
- 43. von der Fehr A, Sølberg J, Bruun J. Validation of networks derived from snowball sampling of municipal science education actors. International Journal of Research & Method in Education. 2018;41(1):38–52.
- 44. Kirchherr JA-O, Charles K. Enhancing the sample diversity of snowball samples: Recommendations from a research project on anti-dam movements in Southeast Asia. (1932–6203 (Electronic)). pmid:30133457
- 45. Agarwal SK, Srivastava AK. On probability limits in snowball sampling. Biometrical Journal. 1980;22(1):87–8.
- 46. Stanco M, Lerro M, Marotta G. Consumers’ Preferences for Wine Attributes: A Best-Worst Scaling Analysis. Sustainability [Internet]. 2020; 12(7).
- 47. Casini L, Corsi AM, Goodman S. Consumer preferences of wine in Italy applying best‐worst scaling. International Journal of Wine Business Research. 2009;21(1):64–78.
- 48. Nunes F, Madureira T, Oliveira JV, Madureira H. The consumer trail: Applying best-worst scaling to classical wine attributes. Wine Economics and Policy. 2016;5(2):78–86.
- 49. Bernabéu R, Díaz M, Olivas R, Olmeda M. Consumer preferences for wine applying best‐worst scaling: a Spanish case study. British Food Journal. 2012;114(9):1228–50.
- 50. Hristov H, Kuhar A. Young Urban Adults’ Preferences for Wine Attributes: An Exploratory Study of the Republic of Macedonia Wine Market Applying the Best-Worst Scaling. Slovenian Association of Agricultural Economists (DAES); 2013.
- 51.
Cohen E, Lockshin L. How consumers choose wine: using best-worst scaling across countries. In: Flynn TN, Louviere JJ, Marley AAJ, editors. Best-Worst Scaling: Theory, Methods and Applications. Cambridge: Cambridge University Press; 2015. p. 159–76.
- 52. Chrysochou P, Krystallis A, Mocanu A, Leigh Lewis R. Generation Y preferences for wine. British Food Journal. 2012;114(4):516–28.
- 53. Cohen E. Applying best‐worst scaling to wine marketing. International Journal of Wine Business Research. 2009;21(1):8–23.
- 54. García Arancibia R, Rossini GE, Depetris de Guiguet EYN. Wine label descriptors and shelf price paid by argentine consumers. Agricultural Economics Review. 2015;16(2):56–72.
- 55. Street AP, Street DJ, editors. Combinatorics of experimental design1987.
- 56.
Wheeler RE. optBlock. AlgDesign. The R project for statistical computing http://www.r-project.org/; 2004.
- 57. White MH. bwsTools: An R package for case 1 best-worst scaling. Journal of Choice Modelling. 2021;39:100289.
- 58. Aizaki H, Fogarty J. R packages and tutorial for case 1 best–worst scaling. Journal of Choice Modelling. 2023;46:100394.