In each case the question is accompanied by a \u2018guideline\u2019 for interpretation of the reply to the question. That additional guidance follows each question in italics <\/em>we then follow each guideline with Sample Answers response.<\/p>\n <\/p>\n <\/p>\n Q1. What experience does your company have with providing online samples for market research?<\/strong> (This answer might help you to form an opinion about the relevant experience of the sample provider. How long has the sample provider been providing this service and do they have for example a market research, direct marketing or more technological background? Are the samples solely provided for third party research, or does the company also conduct proprietary work using their panels?).<\/em><\/p>\n Sample Answers: <\/strong>We have been providing on-line sample and other web services for more than 10 years. As the principal operating company of CMRGroup.com Ltd we were partly instrumental in running the series of 6 Seminars entitled \u201cThe Internet, Marketing and Research<\/em><\/strong>\u201d which began in July 1996 and finished in November 1998, at which point ESOMAR began their own series of special conferences \u201cNet Effects\u201d <\/em><\/strong>the first of which was in February 1999.<\/p>\n <\/p>\n <\/p>\n Q2. Please describe and explain the types of source(s) for the online sample that you provide (are these databases, actively managed panels, direct marketing lists, web intercept sampling, river sampling or other)?<\/strong> (The description of the type of source a provider uses for delivering an online sample might provide insight into the quality of the sample. An actively managed panel is one which contains only active panel members – see question 11. Note that not all online samples are based on online access panels).<\/em><\/p>\n Sample Answers: <\/strong>We use every type of source to assist our clients with their on-line work, including each of the sources mentioned in the question. Of course, different sources are appropriate for different needs and we will always discuss that aspect with our clients.<\/p>\n <\/p>\n Q3. What do you consider to be the primary advantage of your sample over other sample sources in the marketplace?<\/strong> (<\/strong>The answer to this question may simplify the comparison of online sample providers in the market<\/em>).<\/p>\n Sample Answers: <\/strong>We consider the fact that we can offer a comprehensive range of services to be our USP – rather than seeking to claim to be better in any specific area. Since we also trade with other sample providers we believe that we are able to select the best source for our clients, although not necessarily at the best price.<\/p>\n <\/p>\n Q4. If the sample source is a panel or database, is the panel or database used solely for market research? If not, please explain<\/strong>. (Combining panelists for different types of usage (like direct marketing) might cause survey effects).<\/em><\/p>\n Sample Answers: <\/strong>As far as we are aware most Market Research panels (including our own small panel in the UK) come from a variety of sources, which may include: a) <\/em>direct \u2018on-line\u2019 recruitment to the panel web-site; b) <\/em>e-mail invitations from a direct marketing source; c) \u2018click through\u2019 invitations from other web-sites (normally, specific interest websites) and d) special recruitment from telephone or mail invitations.<\/p>\n Coming from a number of sources rather than just the one source, is more likely to increase the representation of the on-line population. Ultimately, a panel (and indeed any specific survey) is concerned with engagement of the respondent and the results are always subject to potential response bias. In by far the majority of instances, such bias will not affect the use that is made of the data \u2013 but, there are exceptions and, to understand such exceptions, we have developed a special software package which is available for clients to quickly evaluate any potential problems arising from samples from different sources (see www.risk-e.net<\/a> for further information).<\/p>\n <\/p>\n Q5. How do you source groups that may be hard-to-reach on the internet<\/strong>? <\/strong>(The inclusion of hard-to-reach groups on the internet – like ethnic minority groups, young people, seniors etc. – might improve the quality of the sample provided).<\/em><\/p>\n Sample Answers: <\/strong>Evidently, the inclusion of such minorities will enhance the quality of the sample in respect of anyone seeking a \u2018universe\u2019 of the population for their study.<\/p>\n However, when responses are \u2018short\u2019 for any important minority we assist our clients by searching across our suppliers for an additional source, as may be necessary. Often that will involve a \u2018click through\u2019 from specialist minority interest sites. It is normally possible to identify the source of response to the survey and then evaluate the effect of different sources if required.<\/p>\n <\/p>\n Q6. What are people told when they are recruited?<\/strong> (The type of rewards and proposition could influence the type of people who agree to answer a questionnaire or join a specific panel and can therefore influence sample quality).<\/em><\/p>\n Sample Answers: <\/strong>We consider that panelists should be briefly advised about Market Research and the fact that they may be invited to participate in many different types of surveys. They are also informed in respect of data privacy policy and the confidentiality of their responses and assured that it is NOT obligatory to participate in every study. In respect of individual projects we recommend to our clients that they will always inform the potential respondent as to the purpose of the survey and we are always concerned that the invitation and the \u2018look and feel\u2019 of the questionnaire are respectful of them and\u00a0the fact that we wish to use their time. Of course, rewards may bias response \u2013 but they are, on occasion, essential to get some genuine surveys done but may be less essential for many opinion polls. It has for long been the standard practice to reward Doctors and IT specialists and there is, in our opinion, no ethical objection to offering rewards \u2013 we always discuss this issue with our client and agree whatever seems most appropriate for the particular study.<\/p>\n <\/p>\n <\/p>\n Q7. If the sample comes from a panel, what is your annual panel turnover \/ attrition \/retention rate and how is it calculated?<\/strong> (The panel attrition rate may be an indicator of panelists\u2019 satisfaction and, therefore, panel management, but a high turnover could also be a result of placing surveys which are too long with poor question design. The method of calculation is important because it can have a significant impact on the rate quoted).<\/em><\/p>\n Sample Answers: <\/strong>The fact is that attrition rates (however calculated) vary across different classification groups \u2013 most importantly they tend to be higher for younger people than for older. In truth, we do not see this as a significant guide to the ease with which a particular survey may be conducted at a particular cost.<\/p>\n <\/p>\n Q8. Please describe the opt-in process<\/strong> (The opt-in process might indicate the respondents\u2019 relationship with the panel provider. The market generally makes a distinction between single and double opt-in. Double opt-in describes the process by which a check is made to confirm that the person joining the panel wishes to be a member and understands what to expect).<\/em><\/p>\n Sample Answers: <\/strong>Double \u2018opt-in\u2019 is our standard.<\/p>\n <\/p>\n Q9. Do you have a confirmation of identity procedure? Do you have procedures to detect fraudulent respondents at the time of registration with the panel? If so, please describe<\/strong>. (Confirmation of identity might increase quality by decreasing multiple entries, fraudulent panelists, etc).<\/em><\/p>\n Sample Answers: <\/strong>No. We have not found a foolproof process that truly assures this. Our approach is to monitor responses and remove the data (and the respondent) if there is a concern. Our experience is that this is a minor issue compared with other aspects (e.g. children completing the questionnaires for the parent). The latter may be identified by investigating responses.<\/p>\n <\/p>\n Q10. What profile data is kept on panel members? For how many members is this data collected and how often is this data updated?<\/strong> (<\/em><\/strong>Extended and up-to-date profile data increases the effectiveness of low incidence sampling and reduces pre-screening of panelists).<\/em><\/p>\n Sample Answers: <\/strong>We maintain only the standard demographics required to \u2018control\u2019 samples by that means \u2013 more detailed information such as car ownership is only maintained for those concerned about their cars. Even so, it is difficult to guarantee that the information will always be \u2018up to date\u2019. We prefer that the email invitation will always state who is requested for this survey and invite the click through accordingly.<\/p>\n <\/p>\n Q11. What is the size and\/or the capacity of the panel, based on active panel members on a given date? Can you provide an overview of active panelists by type of source?<\/strong> (The size of the panel might give an indication of the capacity of a panel. In general terms, a panel\u2019s capacity is a function of the availability of specific target groups and the actual completion rate. There is no agreed definition of an active panel member, so it is important to establish how this is defined. It is likely that the new ISO for access-panels which is being discussed will propose that an active panel member is defined as a member that has participated in at least one survey, or updated his\/her profile data, or registered to join the panel, within the last 12 months. The type and number of sources might be an indicator of source effects and source effects might influence the data quality. For example, if the sample is sourced from a loyalty program<\/em><\/p>\n – travel, shopping, etc. – respondents may be unrepresentatively high users of certain services or products).<\/em><\/p>\n Sample Answers: <\/strong>There are no standard definitions and, indeed no way in which the \u2018active panel\u2019 size will predict response to a particular survey. Each survey needs to be judged independently. Indeed the \u2018active\u2019 panel in October was rarely that active in August!<\/p>\n <\/p>\n <\/p>\n Q12. Please describe your sampling process including your exclusion procedures if applicable. Can samples be deployed as batches\/replicates, by time zones, geography, etc? If so, how is this controlled?<\/strong> (The sampling processes for the sample sources used are a main factor in sample provision. A systematic approach based on market research fundamentals may increase sample quality).<\/em><\/p>\n Sample Answers: <\/strong>The sampling \u2018procedures\u2019 employed are those most appropriate for each project and depend entirely on the \u2018target\u2019 audience required for the project.<\/p>\n Normally, we will only exclude those who are not eligible but other exclusions (e.g. those who have recently responded to a previous survey) will depend upon the clients wishes. For \u2018on-line\u2019 work we see little value in using time zone differential (unlike telephone, of course). We do like to vary the day of week, particularly for reminders.<\/p>\n <\/p>\n Q13. Explain how people are invited to take part in a survey. What does a typical invitation look like?<\/strong> (Survey results can sometimes be influenced by the wording used in subject lines or in the body of an invitation).<\/em><\/p>\n Sample Answers: <\/strong>The invitation is, of course, always agreed with the client. Aside from seeking to have a simple, engaging, email we will usually include the important details of a) the subject of the survey and b) the \u2018type\u2019 of respondent who is invited to complete it, as well as any incentive that may be offered.<\/p>\n <\/p>\n Q14. Please describe the nature of your incentive system(s). How does this vary by length of interview, respondent characteristics, or other factors you may consider?<\/strong><\/p>\n (The reward or incentive system might impact on the reasons why people participate in a specific panel and these effects can cause bias to the sample).<\/em><\/p>\n Sample Answers: <\/strong>This is very much \u2018horses for courses\u2019. We have not, as yet, brought in a \u2018loyalty\u2019 scheme what we do is vary the incentive according to the project. That incentive can be nothing, a feedback report, a prize draw or be a tangible direct money or voucher reward (or a combination of these).<\/p>\n <\/p>\n Q15. How often are individual members contacted for online surveys within a given time period? Do you keep data on panellist participation history and are limits placed on the frequency that members are contacted and asked to participate in a survey?<\/strong> (Frequency of survey participation might increase conditioning effects whereas a controlled survey load environment can lead to higher data quality).<\/em><\/p>\n Sample Answers: <\/strong>Yes, we maintain a history of all \u2018click throughs\u2019 \u2013 but it is not always possible to obtain completion data from every client who hosts their own surveys.<\/p>\n Our principal \u2018philosophy\u2019 is that we are facilitating the opportunity for panelists and, occasionally, others to participate in a survey if they wish and, in the main, we are doubtful about \u2018conditioning\u2019 effects. Clearly if a respondent has completed a survey in an exceptionally \u2018quick\u2019 time we will remove them from the sample and, if repeated, will remove them from the panel. If the client wishes to ensure that they only contact those who have not completed a survey recently we can accommodate that need. In truth, we are aware that many of those who do have rules of this kind often find it necessary to \u2018break\u2019 them to fulfill a particular survey.<\/p>\n <\/p>\n <\/p>\n Q16. Is there a privacy policy in place? If so, what does it state? Is the panel compliant with all regional, national and local laws with respect to privacy, data protection and children e.g. EU Safe Harbour, and COPPA in the US? What other research industry standards do you comply with e.g. ICC\/ESOMAR International Code on Market and Social Research, CASRO guidelines etc.?<\/strong> (Not complying with local and international privacy laws might mean the sample provider is operating illegally).<\/em><\/p>\n Sample Answers: <\/strong>We provide samples (of all types) across the world and, as members of ESOMAR, we are familiar with, and abide by, all the rules of conduct and data protection as appropriate to each country accordingly.<\/p>\n <\/p>\n Q17. What data protection\/security measures do you have in place<\/strong>? <\/strong>(The sample provider usually stores sensitive and confidential information on panelists and clients in databases. These need to be properly secured and backed-up, as does any confidential information provided by the client).<\/em><\/p>\n Sample Answers: <\/strong>All our data bases are secured and appropriately \u2018firewalled\u2019. Sample Answers maintains many confidential data bases \u2018in house\u2019 and have done for many years, these are maintained in accordance with the guidelines as provided by the Data Protection authorities in the UK and other relevant bodies.<\/p>\n <\/p>\n Q18. Do you apply a quality management system? Please describe it<\/strong>. <\/strong>(A quality management system is a system by which processes in a company are described and employees are accountable. The system should be based on continuous Certification of these processes can be independently done by auditing organizations, based for instance, on ISO norms).<\/em><\/p>\n Sample Answers: <\/strong>Yes, but our systems are not accredited.<\/p>\n <\/p>\n Q19. Do you conduct online surveys with children and young people? If so, please describe the process for obtaining permission<\/strong>. (The ICC\/ESOMAR International Code requires special permissions for interviewing children)<\/em><\/p>\n Sample Answers: <\/strong>Yes, we have assisted with projects involving children \u2013 although that is not a \u2018target\u2019 market for us. We always seek parental permission.<\/p>\n <\/p>\n <\/p>\n Q20. Do you supplement your samples with samples from other providers? How do you select these partners? Is it your policy to notify a client in advance when using a third party provider? Do you de-duplicate the sample when using multiple sample providers?<\/strong> (Many providers work with third parties. This means that the quality of the sample is also dependent on the quality of sample providers that the buyer did not select. Transparency is a key issue in this situation. Overlap between different panel providers can be significant in some cases and de-duplication removes this source of error, and frustration for respondents).<\/em><\/p>\n Sample Answers: <\/strong>Yes, we do supplement samples from other suppliers and yes, normally our clients are informed. De-duping is a serious issue and, unfortunately not everyone is able (or willing?) to supply the information necessary to de-dupe a sample and thus avoid the possibility of a second invitation. In fairness, it is often the case that time is of the essence for the client and there simply is not the time (or money) for such \u2018proper\u2019 procedures. Also, respondents on more than one panel for any length of time are \u2018wise\u2019 to this issue and it does not worry them. Of more concern is the potential for a multi-response and processes are in place to reduce the potential for this to a minimum.<\/p>\n <\/p>\n Q21. Do you have a policy regarding multi-panel membership? What efforts do you undertake to ensure that survey results are unbiased given that some individuals belong to multiple panels?<\/strong> (It is not that uncommon for a panellist to be a member of more than one panel nowadays. The effects of multi-panel membership by country, survey topic, etc., are not yet fully known. Proactive and clear policies on how any potential negative effects are minimized by recruitment, sampling, and weighting practices is important).<\/em><\/p>\n Sample Answers: <\/strong>No, we do not have a policy for this. It is impossible to truly police without complete co-operation across the \u2018industry\u2019.<\/p>\n <\/p>\n <\/p>\n Q22. What are likely survey start rates, drop-out and participation rates in connection with a provided sample? How are these computed? <\/strong>(<\/em>Panel response might be a function of factors like invitation frequency, panel management (cleaning) policies, incentive systems and so on. Although not a quality measure by itself these rates can provide an indication of the way a panel is managed. A high start rate might indicate a strong relationship between the panel member and the panel. A high drop-out rate might be a result of poor questionnaire design, questionnaire length, survey topic or incentive scheme as well as an effect of panel management. The new ISO for access panels will likely propose that participation rate is defined as the number of panel members who have provided a usable response divided by the total number of initial personal invitations requesting members to participate).<\/em><\/p>\n Sample Answers: <\/strong>We are always prepared to share this information with our clients. The statistic proposed by the new ISO seems quite appropriate but (in our view) it is also necessary to consider the click through rate as a % of invitations sent. Poor response from click through to completion is often indicative of questionnaire difficulties rather than attempted participation rates.<\/p>\n <\/p>\n Q23. Do you maintain individual level data such as recent participation history, date of entry, source, etc., on your panelists? Are you able to supply your client with a per job analysis of such individual level data?<\/strong> (This type of data per respondent increases the possibility of analysis for data quality, as described in<\/em><\/p>\n ESOMAR\u2019s Guideline on Access Panels).<\/em><\/p>\n Sample Answers: <\/strong>We are not always able to provide the client with such information nor, in general, do they request it (see comment below). We are able to provide clients with Risk-e to assist them in evaluating potential bias by this (or other) information, when available.<\/p>\n <\/p>\n Q24. Do you use data quality analysis and validation techniques to identify inattentive and fraudulent respondents? If yes, what techniques are used and at what point in the process are they applied?<\/strong> (<\/em><\/strong>When the sample provider is also hosting the online survey, preliminary data quality analysis and validation is usually preferable).<\/em><\/p>\n Sample Answers: <\/strong>Since our own hosting of studies is relatively rare this can only be done occasionally and, normally, upon completion.<\/p>\n <\/p>\n Q25. Do you measure respondent satisfaction?<\/strong> (Respondent satisfaction may be an indicator of willingness to take future surveys. Respondent reactions to your survey from self-reported feedback or from an analysis of suspend points might be very valuable to help understand survey results).<\/em><\/p>\n Sample Answers: <\/strong>No, we don\u2019t measure this on a job by job basis. It is an additional \u2018task\u2019 for which, often, there is insufficient time to include. The response itself is a good indicator of the \u2018satisfaction\u2019 and, of course, the truly dis-satisfied can always \u2018opt-out\u2019.<\/p>\n <\/p>\n Q26. What information do you provide to debrief your client after the project has finished?<\/strong> (One might expect a full sample provider debrief report, including gross sample, start rate, participation rate, drop-out rate, the invitation text, a description of the field work process, and so on).<\/em><\/p>\n Sample Answers: <\/strong>It does depend upon the nature of the job and the client\u2019s requirements.<\/p>\n But as a rule we expect to supply the following information whenever appropriate:<\/p>\n Note – questionnaire design and hosting is, now, normally conducted under our \u2018On-line Answers\u2019 <\/em><\/strong>trading name.<\/p>\n <\/p>\n Quantitative work over the Internet has grown very quickly and provided clients with a cost effective means of gathering information that, previously, would have taken longer and been more expensive. Unfortunately, it is NOT, a method that enables the scientist (statistician) to prove that the results can be truly representative of a population. Indeed, a lot of \u2018quick\u2019 telephone work can be said to be equally non verifiable. Unfortunately, with the web, the practice of paying per interview rather than per project has driven down standards in favour of simply speed and cost. This uncomfortable fact is, in many cases, not that significant insofar as the results are true unto themselves and, in the main, provide marketers with effective guidance the implementation of which is a \u2018low risk\u2019 tactic. Whenever the decision to be taken is a high risk one \u2013 then the results need to be more carefully evaluated and there are any number of means by which that can be achieved \u2013 including use of the program Risk-e, which program provides a \u2018quick guide\u2019 to the significant influences on any chosen key variable.<\/p>\n Note <\/strong>\u2013 this document has been prepared by Tony Dent, Chairman of Sample Answers please advise him of any problems of interpretation you may have \u2013 email: tony@sampleanswers.com<\/a><\/p>\n Alternatively please feel free to contact Mark Dent: Mark.Dent@sampleanswers.com<\/a><\/p>\n United Kingdom<\/strong><\/p>\n Tel. +44 (0) 20 8274 5000<\/p>\n Fax. +44 (0) 20 8274 5020<\/strong><\/p>\n www.sampleanswers.com<\/a><\/em><\/strong><\/p>\n <\/p>\n ACKNOWLEDGEMENT:<\/em><\/strong><\/p>\n The 26 questions were prepared on behalf of ESOMAR by the following project team:<\/p>\n <\/p>\n Vondelstraat 172 1054GV Amsterdam The Netherlands<\/p>\n Tel +31 20 664 2141<\/p>\n Fax +31 20 664 2922<\/p>\n E-mail professional.standards@esomar.org<\/a> www.esomar.org<\/a><\/p>\n <\/p>\n All ESOMAR world research codes and guidelines, including latest updates, are available<\/p>\nCOMPANY PROFILE<\/h3>\n
SAMPLE SOURCE<\/h3>\n
PANEL RECRUITMENT<\/h3>\n
PANEL AND SAMPLE MANAGEMENT<\/h3>\n
POLICIES AND COMPLIANCE<\/h3>\n
PARTNERSHIPS AND MULTIPLE PANEL MEMBERSHIP<\/h3>\n
DATA QUALITY AND VALIDATION<\/h3>\n
\n
SAMPLE ANSWERS FINAL COMMENT:<\/h3>\n
\n62 High Street
\nRedhill, Surrey
\nRH1 1SG<\/div>\n\n
ESOMAR<\/h2>\n
![\"\"](\"https:\/\/www.sampleanswers.com\/wp-content\/uploads\/2020\/11\/Untitled-design-31-150x150.png\")
Contact us<\/strong><\/p>\n![\"magnifying-glass-on-a-person-icon\"](\"https:\/\/www.sampleanswers.com\/wp-content\/uploads\/2020\/11\/Untitled-design-37-150x150.png\")
Order sample<\/strong><\/p>\n![\"shopping-basket-icon\"](\"https:\/\/www.sampleanswers.com\/wp-content\/uploads\/2020\/11\/Untitled-design-32-150x150.png\")
Reach targets<\/strong><\/p>\n