When is a 'survey' not really a survey? Perhaps, when it is a marketing tool.
Keith S. Taber
A research survey seeks information about a population by collecting data from a sample.
Acaudio's 'survey' seems to seek information about whether particular respondents might be persuaded to buy their services.
Today I received an invitation to contribute to something entitled "the first annual International Survey of Research Centres and Institutes". Despite this impressive title, I decided not to do so.
This was not because I had some doubts that whether it really was 'the first…' (has there never previously been an annual International Survey of Research Centres and Institutes?) Nor was it because I had been invited to represent 'The Science and Technology Education Research Group' which I used to lead – but not since retiring from my Faculty duties.
My main reason for not participating was because I suspected this was a scam. I imagined this might be marketing apparently masquerading as academic research. I include the provisos 'suspected' and 'apparently' as I was not quite sure whether this was actually a poor attempt to mislead participants or just a misjudged attempt at witty marketing. That is, I was not entirely sure if recipients of the invitation were supposed to think this was a serious academic survey.
There is a carpet company that claims that no one knows more about floors than … insert here any of a number of their individual employees. Their claims – taken together – are almost logically impossible, and certainly incredible. I am sure most people let this wash over them – but I actually find it disconcerting that I am not sure if the company is (i) having a logical joke I am supposed to enjoy ('obviously you are not meant to believe claims in adverts, so how about this…'), or (ii) simply lying to me, assuming that I will be too stupid to spot the logical incoherence.
Read 'Floored or flawed knowledge?: A domain with a low ceiling'
Why is this not serious academic research?
My first clue that this 'survey' was not a serious attempt at research was that the invitation was from an email address of 'playlist.manager@acaudio.com', rather than from an academic institute or a learned society. Of course, commercial organisations can do serious academic research, if usually when they are hired to do so on behalf of a named academically-focussed organisation. The invitation made no mention of any reputable academic sponsor.
I clicked on the link to the survey to check for the indicators one finds in quality research. Academic research is subject to ethical norms, such as seeking voluntary informed consent, and any invitation to engage in bone fide academic research will provide information to participants up front (either on the front page of the survey or via a link that can be accessed before starting to respond to any questions). One would expect to be informed, at a minimum:
- who is carrying out the research (and who for, if it is commissioned) and for what purpose;
- how data will be used – for example, usually it is expected that any information provided with be treated as confidential, securely stored, and only used in ways that protect the anonymity of participants.
This was missing. Commercial organisations sometimes see information you provide differently, as being a resource that they can potentially sell on. (Thus the recent legislation regulating what can or cannot be done with personal information that is collected by organisations.)
Hopefully, potential participants will be informed about the population being sampled and something of the methodology being applied. In an ideal world an International Survey of Research Centres and Institutes would identify and seek data from all Research Centres and Institutes, internationally. That would be an immense undertaking – and is clearly not viable. Consider:
- How many 'research centres' are initiated, and how many close down or fade away, internationally, each year?
- Do they all even have websites? (If not, how are they to be identified?)
- If so, spread over how many languages?
Even attempting a meaningful annual survey of all such organisations would require a substantive, well-resourced, research team working full-time on the task. Rather, a viable survey would collect data from a sample of all research centres and research institutes, internationally. So, some indication of how a sample has been formed, or how potential participants identified, might be expected.
Read about sampling a population of interest
One of the major limitations of many surveys of large populations is that even if a decent sample size is achieved, such surveys are unlikely to reach a representative sample, or even provide any useful indicators of whether the sample might be representative. For example, information provided by 'a sample of 80 science teachers' tells us next to nothing about 'science teachers' in general if we have no idea how representative that sample is.
It can be a different matter when surveys are undertaken of small, well-defined, populations. A researcher looking to survey the students in one school, for example (perhaps for a consultation about a mooted change in school dress policy), is likely to be in a position to make sure all in the population have the opportunity to respond, and perhaps encourage a decent response rate. They may even be able to see if, for example, respondents reflect the wider population in some important ways (for example, if one got responses from 400/1000 students, one would usually be reasonably pleased, but less so if hardly any of the responses were in, say, the two youngest year groups).
In such a situation there is likely to be a definitive list of members of the population, and a viable mechanism to reach them all. In more general surveys, this is seldom the case. One might see a particular type of exception as elections (which can be considered as akin to surveys). The electoral register potentially lists all enfranchised to vote, and includes a postal address where each voter can be informed of a forthcoming poll. In this situation, there is a considerable administrative cost of maintaining the register – considered worth paying to support the democratic process – and a legal requirement to register: yet, even here, no one imagines the roll is ever complete and entirely up-to-date.)
- How many of the extant Research Centres and Research Institutes, internationally, had been invited to participated in this survey?
- And did these invitations reflect the diversity of Research Centres and Institutes, internationally?
- By geographical location?
- By discipline?
No such information was provided.
The time-scale for an International Survey of Research Centres and Institutes
To be fair the invitation email did suggest the 'researchers' would share outcomes with the participants:
"We will release the results over the next month".
But that time-scale actually seemed to undermine the possibility that this initiative was meant as a serious survey. Anyone who has ever undertaken any serious research knows: it takes time.
When planning the stages of a research project, you should keep in mind that everything will likely take longer than you expect…
even when you allow for that.
Not entirely frivolous advice given to research students
Often with surveys, the initial response is weak (filling in other people's questionnaires is seldom anyone's top priority), and it becomes necessary to undertake additional rounds of eliciting participation. It is good practice to promise to provide feedback; but to offer to do this within a month seems, well, foolhardy.
Except, of course, Acaudio are not a research organisation, and the purpose of the 'survey' was, I suggest, not academic research. As becomes clear from the questions asked, this is marketing 'research': a questionnaire to support Acaudio's own marketing.
What does this company do?
Acaudio offer a platform for allowing researchers to upload short audio summaries of their research. Researchers can do this for free. The platform is open-access, allowing anyone to listen. The library is collated with play-lists and search functions. The company provides researchers data on access to their recordings.
This sounds useful, and indeed 'too good to be true' as there are no charges for the service. Clearly, of itself, that would be a lousy business model.
The website explains:
"We also collaborate with publishers and companies. While our services are licensed to these organizations, generating revenue, this approach is slightly different from our collaboration with you as researchers. However, it enables us to maintain the platform as fully open access for our valued users."
https://acaudio.com/faq
So, having established the website, and built up a library of recordings hosted for free (the 'loss leader' as they say 1), the company is now generating income by entering into commercial arrangements with organisations. Another page on their website claims the company has 'signed' 1000 journals and 2000 research centers [sic]. So, alongside the free service, the company is preparing content on behalf of clients to publicise, in effect advertise, their research for them. Nothing terrible there, although one would hope that the research that has the most impact gets that impact on merit, not because some journals and research centres can pay to bring more attention to their work. This business seems similar to those magazines that offer to feature your research in a special glossy article – for a price.
Read 'Research features…but only if you can afford it'
One would like to think that publicly funded researchers, at least, spend the public's money on the actual research, not on playing the impact indicators game by commissioning glossy articles in magazines which would not be any serious scholar's preferred source of information on research. Sadly, since the advent of the Research Assessment Exercise (and its evolution into the 'Research Excellence Framework') vast amounts of useful resource have been spent on both rating research and in playing the games needed to get the best ratings (and so the consequent research income). As is usually the case with anything of this kind (one could even include formal school examinations!), even if the original notion is well-intentioned,
- the measurement process comes to distort what it is measuring;
- those seen as competing spend increasing resources in trying to out do each other in terms of the specifics of the assessment indicators/criteria
So, as research impact is now considered measurable, and as it is (supposedly) measured, and contributes to university income, there is a temptation to spend money on things that might increase impact. It becomes less important whether a study has the potential to increase human health and happiness; and more important to get it the kind of public/'end user' attention that might ultimately lead to evidence of 'impact' – as this will increase income, and allow the research to continue (and, who knows, perhaps eventually even increase human health and happiness).
What do Acaudio want to know?
Given that background, the content of the survey questionnaire makes perfect sense. After collecting some information on your research centre, there are various questions such as
- How satisfied are you with the level of awareness people have of your centre / institute?
- How important is it that the general public are aware of the work your centre / institute does?
I suspect most heads of research centres think it is important people know of their work, and are not entirely satisfied that enough people do. (I suspect academic researchers generally tend to think that their own research is actually (i) more important than most other people realise and (ii) deserves more attention than it gets. That's human nature, surely? Any self-effacing and modest scholars are going to have to learn to sell themselves better, or, if not, they are perhaps unlikely to be made centre/institute heads.
There are questions about how much time is spent promoting the research centre, and whether this is enough (clearly, one would always want to do more, surely?), and the challenges of doing this, and who is responsible (I suspect most heads of centres feel some such responsibility, without considering it is how they most want to spend their limited time for research and scholarship).
Perhaps the core questions are:
- Do you agree it is important to have a dedicated person to take care of promotional activities?
- How much would you consider to be a reasonable amount to spend on promotional activities?
These questions will presumably help Acaudio decide whether you can easily be persuaded to sign up for their help, and what kind of budget you might have for this. (The responses for the latter include an option for spending more than $5000 each year on promotional activities!)
I am guessing that at even $5000+ p.a., they would not actually provide a person dedicated to 'take care of promotional activities' for you, rather than a person dedicated to adding your promotional activities to their existing portfolio of assigned clients!
So, this is a marketing questionnaire.
Is this dishonest?
It seems misleading to call a marketing questionnaire 'the first annual International Survey of Research Centres and Institutes' unless Acaudio are making a serious attempt to undertake a representative survey of Research Centres and Institutes, internationally, and they do intend to publish a full analysis of the findings. "We will release the results over the next month" sounds like a promise to publish, so I will look out with interest for an announcement that the results have indeed been made available.
Lies, delusional lies, and ill-judged attempts at humour
Of course, lying is not simply telling untruths. A person who claims to be Napoleon or Joan of Arc is not lying if that person actually believes that is who they are. Someone who claims they are the best person to run your country is not necessarily lying simply because the claim is false. If the Acaudio people genuinely think they are really doing an International Survey of Research Centres and Institutes then their invitation is not dishonest even if it might betray any claim to know much about academic research.
"I'm [an actor playing] Spartacus";"I'm [an actor playing another character who is not Spartacus, but is pretending to be] Spartacus"; "I'm [another actor playing another character who is also not Spartacus, but is also pretending to be] Spartacus"… [Still from Universal Pictures Home Entertainment movie 'Spartacus']
Nor is it lying, when there is no intent to deceive. Something said sarcastically or as a joke, or in the context of a theatrical performance, is not a lie as long as it is expected that the audience share the conceit and do not confuse it for an authentic knowledge claim. Kirk Douglas, Tony Curtis, and their fellow actors playing rebellious Roman slaves, all knew they were not Spartacus, and that anyone in a cinema watching their claims to be the said Spartacus would recognise these were actors playing parts in a film – and that indeed in the particular context of a whole group of people all claiming to be Spartacus, the aim even in the fiction was actually NOT to identify Spartacus, but to confuse the whole issue (even if being crucified as someone who was only possibly Spartacus might be seen as a Pyrrhic victory 2).
So, given that the claim to be undertaking the first annual International Survey of Research Centres and Institutes was surely, and fairly obviously, an attempt to identify research centres that (a) might be persuaded to purchase Acaudio's services and (b) had budget to pay for those services, I am not really sure this was an attempt to deceive. Perhaps it was a kind of joke, intended to pull in participants, rather than a serious attempt to fool them.
That said, any organisation hoping for credibility among the academic community surely needs to be careful about its reputation. Sending out scam emails that claim to be seeking participants for a research survey that is really a marketing questionnaire seems pretty dubious practice, even if there was no serious attempt to follow through by disguising the questionnaire as a serious piece of research. You might initially approach the questionnaire thinking it was genuine research, but as you worked through it SHOULD have dawned that this information was being collected because (i) it is of commercial value to Acaudio, and not (ii) to answer any theoretically motivated research questions.
- So, is this dishonest? Well, it is not what it claims to be.
- Does this intend to deceive? If it did, then it was not well designed to hide its true purpose.
- Is it malpractice? Well, there are rules in the U.K. about marketing emails:
"You're only allowed to send marketing emails to individual customers if they've given you permission.
Emails or text messages must clearly indicate:
- who you are
- that you're selling something…
Every marketing email you send must give the person the ability to opt out of (or 'unsubscribe from') further emails."
https://www.gov.uk/marketing-advertising-law/direct-marketing
The email from Hussain Ayed, Founder, Acaudio, told me who he, and his organisation, are, but
- did not clearly suggest he was selling something: he was inviting me to contribute to a research survey (illegal?)
- Nor was there any option to opt out of further messages (illegal?)
- And I am not aware of having invited approaches from this company – which might be why it was masquerading as a request to contribute to research (illegal?)
I checked my email system to see if I'd had any previous communication with this company, and found in my junk folder a previous approach,"invit[ing Keith, again] to talk about some of the research being done at The Science and Technology Education Research Group on Acaudio…". It seems my email software can recognise cold calling – as long as it does not claim to be an invitation to respond to a research study.
The earlier email claimed it was advertising the free service…but then invited me to arrange a time to talk to them for 'roughly' 20 minutes. That seems odd, both because the website seems to provide all the information needed; and then why would they commit 20 minutes of their representative's time to talk about a free service? Presumably, they wanted to sell me their premium service. The email footer also gave a business address in E9, London – so the company should know about the UK laws about direct marketing that Acaudio seems to be flouting.
Perhaps not enough people responded to give them 20 minutes of their time, so the new approach skips all that and asks instead for people to "give us 2-3 minutes of your time to fill in the survey [sic 3]".
Original image by Mohamed Hassan from Pixabay
Would you buy a second hand account of research from this man?
In summary, if someone is looking to buy in this kind of support in publicising their work, and has the budget(!), and feels it is acceptable to spend research funds on such services, then perhaps they might fill in the questionnaire and await the response. But I am not sure I would want to get involved with companies which use marketing scams in this way. After all, if they cannot even start a conversation by staying within the law, and being honest about their intentions, then that does not bode well for being able to trust them going forward into a commercial arrangement.
Update (15th October, 2023): Were the outcomes of the first annual International Survey of Research Centres and Institutes published? See 'The sugger strikes back! An update on the 'first annual International Survey of Research Centres and Institutes'
Notes
1 When a shop offers a product at a much discounted price, below the price needed to 'break even', so as to entice people into the shop where they will hopefully buy other goods (at a decent mark-up for the seller), the goods sold at a loss are the 'loss leaders'.
Goods may also be sold at a loss when they are selling very slowly, to make space on the shop floor and in the storeroom for new produce that it is hoped will generate profit. Date-sensitive goods may be sold at a loss because they will soon not be saleable at all (such as perishables) or only at even greater discounts (such as models about to be replaced by updated versions by manufacturers – e.g., iPhones). But loss leader goods are priced low to get people to view other produce (so they might be displayed dominantly in the window, but only found deep in the shop).
2 In their wars against the armies of King Pyrrhus of Epirus, the Romans lost battles, but in doing so inflicted such heavy and unsustainable losses on the nominally victorious invading army that Pyrrhus was forced to abandon his campaign.
At the end of the slave revolt (a historical event on which the film 'Spartacus' is based) the Romans are supposed to have decided to execute the rebel leader, the escaped gladiator Spartacus, and return the other rebels to slavery. Supposedly, when the Roman official tried to identify Spartacus, each of the recaptured slaves in turn claimed he was Spartacus, thus thwarting identification. So, the ever pragmatic Romans crucified them all.
3 The set of questions is actually a questionnaire which is used to collect data for the survey. Survey (a type of methodology) does not necessarily imply using a questionnaire (a data collection technique) as a survey could be carried out using an observation schedule (i.e., a different data collection technique), for example.
