The DIY guide to spotting bad science in adverts

Some people have asked me about whether products they've seen in adverts actually work or not. Mostly ones I've never heard of because I usually get my toiletries from Savers. What I have told them is this: you don't need an expert to tell you whether a product works or not. Just as you should be wary of a celeb endorsing a product you should probably be equally wary of people like me telling you it doesn't work! Science is not so complicated that you couldn't figure it out, if you know what to look for.

This is my guide to being sceptical of the use of science in adverts.

1. Check whether those "studies" are proper clinical trials or consumer surveys. When as advert says "studies have shown", it doesn't mean much. If you can't see any evidence of a clinical trial be suspicious. A survey can be someone whizzing around an office asking people for their thoughts; not very scientific. A clinical trial should use a rigorous method to test whether something works (e.g. product X gets rid of spots) by eliminating other possible causes of an effect. Design of clinical trials is a long topic, but the basics, if you want to know, are that:
• there should be some control group (e.g. some people don't get given product X but a blank product)
• the people involved in the study shouldn't know whether they have been given product X or the blank (this gets rid of bias - a person knowingly using product X may assume it will work and report positive results because of that)
• the way in which people are allocated into groups to get product X or the blank is randomised (this also gets rid of bias, e.g. product X isn't given to the people who had less spots to start with anyway, meaning you'd get a more positive result). 
2. If it's a survey, look at the numbers. Surveys gain credibility if they are representative of a group of people. If a survey finds that 86% of women were happy with a face cream, you may think great - it works! But if that 86% was 43 women out of 50, the numbers are probably too low to mean much. There are tens of millions of women in the UK, does a sample group of 50 really represent us? There are lots of other problems with surveys, but the numbers do tend to be put in the small print of adverts so that's an easy one to look out for.
3. Listen to the wording. Does skin serum make women's skin more supple or do women report that it makes their skin feel more supple? I call these woolly words; they leave so much room for ambiguity. They don't really mean anything, it's just opinions masquerading as science.
4. Ignore the buzzy scientific images. They often mean nothing and don't prove that something works. Just another selling tool.
5. Ignore jazzy scientific jargon. Yes some of the words are real scientific terms, but others are not. Pantene shampoos contain "Pro-V" - what is that exactly? Well in this case it stands for pro-vitamin B5, a common ingredient in many shampoos. Nowt special about Pantene.The ultimate tool for busting through the jargon is Google, so use it!
6. No cosmetic can stop wrinkles permanently. This seems easy to forget with the constant bombardment of convincing advertising, but ultimately only surgical methods can get rid of wrinkles. Some creams may reduce the wrinkly look for a short while, but that's all. So if you want that cream because the packaging is attractive or you like the smell/feel of it, knock yourself out. Just don't fool yourself into believing it will reverse time on your face and be £100 out of pocket to boot.

Well I'm sure I've left things off this list, but it is 2am after all! I also realise I've focused mainly on the beauty product spectrum of adverts, but as we all know, these are the worst for using science to market spurious claims. Do you have your own tips? Please share in the comments section.

Long hiatus

I've been incredibly lazy at keeping up my little blog and to anyone who keeps an eye on it, apologies! It's partly because I haven't had much rant in me the last few months. Thankfully there aren't too many irritating adverts. However there is a niggling little ad campaign called "the Science behind the beauty" that's got me going and I will be posting a grisly dissection soon.

If you haven't seen it, have a look at this one for example:

Oh Anna Richardson, I quite liked you on the Sex Education Show.

Shoes for perky bums

Some of you will know that I have a bee in my bonnet about fitness shoes. You know the ones I mean. It started with MBTs  that were promoted as an "anti-shoe". Then we had Reebok Re-Tone trainers. You can't have missed the TV advert for Re-Tone. It featured about a hundred hypnotically bouncing perky bums and long legs. You probably didn't even notice the shoes to start with. OK I couldn't resist watching it again on YouTube so here it is. Try to concentrate on the shoes, I dare you.


Next in  line were Fit Flops, a craze that seems to have coincided beautifully with the summer - kudos to the Fit Flops marketing manager. Ladies, it's your dream come true! Now not only can you look cool this summer, your shoes can make you fit! How nice of you to combine two of our most important and all-consuming obsessions.


Finally, Skechers have got onto this bandwagon, with Shape-Ups. 


All these shoes (MBT people, they're still shoes in my book) share broadly the same marketing claim: to improve posture and fitness through working your body a bit harder than normal shoes. What I want to know is whether these claims are backed up by evidence. Unfortunately, my inquiries have been rather fruitless, as you will see. Which is a shame as it's the first time I've ever contacted companies to ask for the evidence behind marketing claims.


First I emailed Reebok to ask about their shoes. My email said:




Dear Sir/Madam
 
I am gathering evidence on toning trainers (including Reebok Retone/Easytone and Fit Flops) for my blog. I am particularly interested in the clinical evidence behind the marketing claims that Easytone/Reetone trainers improve muscle tone/strength and improve fitness. For example the claims in this advert: http://blog.fitnessfootwear.com/wp-content/uploads/2010/03/reetone21.jpg.
 
I was unable to find any references to clinical trials/scientific experiments that support the use of Reetone/Easytone trainers for muscle toning on your website and I would be very grateful if you could send me the original studies. 
 
Thank you in advance for your assistance.
 
Kind regards,
Sabre


I got an auto-reply saying:

Dear Sir or Madam:

We have received materials from you that appear to include an unsolicited idea for our consideration.

While we genuinely appreciate your interest in our company, please be advised that neither Reebok International Ltd (“Reebok”), nor any of its subsidiary or affiliated companies, accepts or reviews unsolicited idea submissions including designs, marketing concepts, or inventions.  In accordance with this policy, none of the materials received from you have been reviewed or evaluated by Reebok.

Please do not send any further unsolicited ideas to Reebok as such submissions will not be reviewed or evaluated.

Sincerely,

Reebok International Ltd

I emailed again but with no further success. Bollocks to you too Reebok!

The Fit Flops people were marginally more helpful. Having emailed them with:

Dear Sir/Madam

I am gathering evidence on toning shoes for my blog which looks at the science behind consumer products. I have looked at the FitFlop website in detail and while I am pleased that there is some explanation of the science behind the products (this is sadly the exception rather than the norm), I would like more information please. I was unable to find references to published clinical research that supports the claims that FitFlops can have health benefits and would appreciate if you could send me the original studies (or full references for them).

In addition it was unclear whether the claims were based on clinical evidence or on consumer testimonials. I would be very grateful if (as well as sending me links to original studies) you could clarify the origins  (clinical evidence or testimonials) for the following claims:
  1.Reduce back stress
2. Reduce joint stress (knees, hip joints and ankles)
3. Increase muscle activation (up to 16% for womens hamstrings, up to 11% for womens lower legs, up to 30% for womens bottoms, up to 16% for mens quadricep muscles, up to 11% for mens calf muscles)

I found the above claims on  http://www.fitflop.com/benefits/. 

Thank you in advance for your assistance.

Kind regards,
Sabre

I got a reply, from a real person this time, saying:

Dear Sabre,

Thanks so much for getting in touch and for your interest in our shoes. Have you seen this webpage which details the results and studies that our shoes have undergone? http://www.fitflop.com/benefits/research/  

Kind regards,

Natasha

Sigh. I emailed back reiterating that yes I had seen their website and wanted the original studies. I also emailed the two people, Dr Philip Graham Smith and Richard Jones at Salford University, who are mentioned on their website as having independently tested the technology. No reply at all, and all these emails were sent over a month ago.

All I've managed to establish so far is that the marketing is great but the customer service is woeful. Unable to actually look at the evidence, particularly for Fit Flops (which I want to buy!), I'm going to cobble together some kind of opinion anyway, based on what information I do have.

As I mentioned in my email to the Fit Flops company, my main interest is in the origin of the health claims made. Are they the result of proper controlled trials of some sort or are they based on surveys and customer feedback? This is what prompted my BS-o-meter, (taken from Fit Flops website):

"Please note that on page 10 of the Spring Summer brochure, we would like to clarify that the claim “you can improve your posture” is derived from hundreds of FitFlop wearer testimonials we have received."

One might ask why it matters that claims are based on testimonials. After all, if people say it works, where's the problem? The problem with surveys is that:

1. the group of people surveyed have a vested interest in the shoes working for them. After all, they've spent money on these shoes and so they are more likely to give a favourable response

2. we don't know how testimonials were sought or if people were randomly selected for views. Therefore we don't know what kinds of bias are present

3. The improvements are self-reported. No health professional has provided an independent assessment of whether the customer's posture or fitness has actually improved

4. There may be hundreds of people giving positive testimonials, but what is the overall number of customers? If a hundred customers out of a thousand are satisfied, that's only 10%.

5. We don't know if customers see improvements because of a placebo effect. Does posture improve because of the shoes or because after buying them, a customer starts walking a little bit taller and straighter? Expectation of an improvement can actually be the sole cause of improvement.

Without the information to hand, it's difficult to say whether there is evidence for health benefits. I have however found an excellent article that evaluates the pros and cons of Fit Flops and MBTs. I recommend you read it. I would add to the article by pointing out the risk of wearing fitness shoes as a substitute for proper exercise. This is a very real danger for lazy people like me. If they don't work and you give up proper exercise you'll end up less fit, not more.

The bottom line (geddit?) is that if they're comfortable and you like the way they look, buy them. But don't pin your hopes on them making you fit because it's difficult to find the evidence to prove it. I know it's jumping to a conclusion, but I do feel that if the evidence was solid, the Fit Flops people would have been happy to send me full references to published research.

I've just realised I didn't put a picture in this post. So here's something to counter all those female bums (cos bad science should be an equal opportunity activity).

Are feminist skeptics "woo filled idiots"?

My command of the social media site Twitter has grown and over the last month particularly I have come across or engaged in discussions about women and their relationship with science. My ramblings below are prompted by a post on the Blag Hag blog called The more feminists mistrust science, the more women look like fools. This was a response to another article published at I Blame the Patriarchy on a scientific study looking at the effects of porn. (BTW I'm not getting into a porn debate!)

I have also recently become aware of groups of people calling themselves "skeptics" who meet in pubs across the UK and take a critical view on pseudoscientific mumbo-jumbo like homeopathy and astrology. I've never been to one of their meetings but they're held all around the country. Recently a group called Ladies Who Do Skepticism formed along a similar vein. Recent discussions on Twitter have focused on why women don't get more involved in skepticism and come to these meetings.

So here are my thoughts.

Science has historically been the preserve of men (as have most professional and academic fields). The situation has improved somewhat but I am aware that the numbers of women in science is still pathetically low, particularly at the top of professions. If there isn't a glass ceiling then women must be very good mimes. The factors involved are numerous, and I won't go into them. However I think that our perceptions of who does science still hasn't changed much. For example, Google search for "scientist" and you usually get something like this or this. Google also suggests the related search "female scientist" which feeds into the assumption that male is the default setting for a scientist (search for nurse and Google suggests "male nurse").  I think that may women who are not scientists don't see science as something that affects them, or at least not as something to get involved with. This is true of non-scientist males too, but the male scientist stereotype is an extra barrier for women.

I've also noticed a certain aggression in some of the skeptical discussions that take place online. While aggression is not an exclusively male trait, I have picked up on an element of machismo and derision towards those who do not agree and I can see how that might be off-putting to women more than men. I haven't been to a skeptics meeting but I'd be interested in seeing for myself whether this is true in real life.

Anyway back to the Blag Hag article, in which the blogger (Jen) basically attacks Jill (from I Blame the Patriarchy) for questioning the validity of a a study on the effects of porn on men. It's interesting because Jen appears to see herself as the voice of scientific reason, defending science from "those feminists who distrust science". And yet a look at Jill's article shows that Jill was simply approaching the subject at hand with a healthy dose of skepticism. The really fascinating part is when  Jen says:

"Every time a feminist treats science like some great big boogeyman, she makes all feminists and women look foolish and ignorant."


I have a big problem with this statement. Do we really have to simply accept any scientific study for fear of looking like "woo filled idiots screaming conspiracy theories"? Why does questioning make us look stupid rather than skeptical? Are male skeptics accused of being foolish and ignorant too?


I actually think the right place is a middle ground and I can find points in both articles to agree and disagree with. Some, but of course not all, scientists and science experiments are misogynistic. But Jen's overreaction to healthy skepticism from a fellow feminist is wrong and serves to discourage women from skepticism. Women are constantly being bombarded by crap science, whether it is surveys on sexual happiness, creams for prolonging youth or all-natural yoghurts to feed the young 'uns. Even if we get it wrong, we shouldn't hold back from questioning the science if we think something's amiss. We should rush to hold those who conduct and benefit from science to account. 


I want to comment also on intuition. Whether it is skepticism, experience or simply (my favourite) an internal bullshit-o-meter, we all need something to prompt us to start questioning where a problem may not be obvious. All of these things make up what I like to think of as intuition, and therefore I don't know why intuition is seen as such a dirty "emotional" thing. What's important is that intuition leads to rational scrutiny. I want to add that I don't think emotion and science are mutually exclusive. Scientists are passionate, they are defensive, they get excited, they are competitive and their work can be biased by emotions and beliefs. So let's not act like science is always cold and rational, because often it isn't. 

To sum up (because I could ramble on forever), I'd say the following are reasons why women aren't more involved in skepticism (questioning bad science).

1. Science isn't by female non-scientists seen as "for us" because of pervasive stereotypes and male domination
2. There is a lot of bad science targetted specifically at women out there. This may have the effect of women building up some kind of immunity and an ability (or need) to simply ignore it all
3. Questioning what appears to be science may draw negative reactions from others and accusations of being anti-science
4. The idea that science is cold and logical combined with the message that logic is for men and emotion is for women
5. Most importantly,  there aren't enough women doing science. This is by far the biggest problem. You don't need to be a scientist to be a science sceptic but it helps, not least with having the confidence to question.

Hey it's not all doom and gloom. Have a look at some drawings by schoolchildren of scientists before and after visiting labs and talking to scientists. If this doesn't cheer you up and inspire hope, nothing will.

L'Oreal Youth Code

Alright, I said I wasn't going to go for the low hanging fruit with this blog and I am trying to steer away from talking about anti-wrinkle products, I really am. But that doesn't stop me becoming infuriated, as I was this week when I saw an advert for L'Oreal Youth Code. Not only was the pseudoscientific tripe in front of me, it was in an ad break for Genius of Britain, a historical look at some of Britain's past scientific genius. The juxtaposition made it seem extra insulting. There really is no safe place!

The headline is "inspired by the science of genes". L'Oreal's website has an explanation of the science here. Gene expression is described as occurring when skin genes are activated following stress. In young skin this happens more quickly than in older skin, indicating that it takes longer to respond the stressor. Having "discovered" this wonderful science, L'Oreal developed the Youth Code Range, which is:

"Enriched with patented Pro-Gen (TM) Technology.

Applied daily, the moisturising formula helps improve the skin's ability to behave more youthfully" (ref)

Ah Pro-Gen Technology, where have you been all my life? And more importantly, what the heck are you? Further digging reveals that Pro-Gen Technology is supposed to speed up the rate of gene expression and contains biolysate. The "code" part of Youth Code refers to the claim that this cream will help your skin reset its genetic code.

Here comes the science bit.

Genes contain information. Gene expression is what happens when that information is used to make a product, usually a protein. Proteins then go away and do get things done in body cells. This whole process is fundamental to all life, it makes the body work. Gene expression happens almost everywhere in the body and so I'm OK with the idea that external stress can lead to gene expression occurring, because so many things can require proteins to be made. The genetic code refers to the sequence of animo acids that make the gene; the blueprint.

The link between this science and what L'Oreal have put into Youth Code is not explained anywhere, and yet that's the crucial part of how this cream should work. That in itself suggests something fishy. I couldn't figure out what Pro-Gen Technology is, but I did find this handy list of ingredients and what they're used for. It's interesting to see that ascorbyl glucoside is whitening, but that's another can of worms!

Taking a historical view of skincare products and science, it seems that products are always peddled to consumers based on the trendy science of the day, be it radioactivity, chemicals, hormones or genetics. Today's trend is of course genetics and it appears that L'Oreal have caught the public fascination and are cashing in. Simple as that. There is no evidence that a cream can tinker with gene expression to produce more youthful skin, and if a cream could affect my genetic code I'd be highly concerned because that would be tinkering with my very DNA and causing genetic mutations. No thanks.

But wait, there are proven results! Oops, it's a consumer test not real scientific evidence.
 
"PROVEN RESULTS



Proven and confirmed by women:
- Skin looks rejuvenated: 78% agree*
- Wrinkles appear reduced: 69% agree*
- Features appear rested: 71% agree*

*Consumer test, 229 women"

At least L'Oreal are careful never to claim that their products reduce wrinkles, just that they appear to. Skin isn't actually rejuvenated, it just looks like it. The "features appear rested" made me laugh because I have no idea what that even means.

Oh L'Oreal. Ever since I went to the Science Museum lates event you sponsored you've really been getting under my skin.

I guess some things never change

I came across this advert today for facial massage from 1926.

The small text reads:

"Too bad she doesn't try to remain the girl her husband fell in love with. The change is so gradual that you never see it. But your friends do.
Then - "How young Lois looks. Wonder how she does it." You'd hardly believe that even Harper Method facial massages could work such wonders.
But they do. Of course a "facial" must be done with science and skill. Delicate muscles and fine aristocratic skin should never be entrusted to any except trained experts.
That's why the women whose glowing youth you envy take care to have their facial massage regularly at a Harper Method shop. There Harper Method graduates, with secrets learned during 38 years of success, help to keep you looking young, attractive - charming alike to family and friends.


Harper Method preparations can be purchased in convenient sizes for home use at all Harper Method shops"


And right at the bottom it says:

"Free - Book on "Scientific care of hair and scalp". Fully illustrated. Secrets every woman should know."


It reminds me of the Harry Enfield comedy sketches "Women, know your limits". If you don't know what I'm talking about, watch this.

It's a bit frightful to realise that although marketing has become much more subtle, the underlying message hasn't really changed since the 1920s.

Women: avoid looking your age and trust in science!

The Science of Beauty

Anyone who follows my tweets will know that I was quite angry last week following my visit to the Science Museum's "Lates" event. Science Museum Lates are free, adult-only evening events held once a month. As well as giving visitors an opportunity to experience the Science Museum without children and with booze (a joy I must admit), the events are themed. Last week's theme was "The Science of Beauty", and it was sponsored by L'Oreal UK and Ireland. Yeah I know, warning bells are ringing already.

The flyer for the event can be found here. The programme included:

• What Makes a Beauty Product: an exhibition on beauty research and development from L'Oreal and the chance to learn what your skin says about you
• Table-top experiments: discover how to make your own beauty potion
• Why Does Beauty Matter: debate with a panel

There were lots of other talks going on that evening but the above are the only ones I had time for, so my comments are restricted to the above.

I headed straight up to the "What makes a beauty product" exhibition as soon as I got in. Finally, I thought, a chance to talk to someone about the science behind beauty products or maybe learn something new and perhaps reassuring. But I was disappointed. The focal point of the exhibition was a computer-based exhibit that told people about their skin type, no doubt coupled with advice on what L'Oreal product would work best to complement it. It wasn't worth my time queueing for and I didn't care to find out more. The part that was actually about beauty R&D was... wait for it.... a pop up stand! Here's what it looked like:

There were two more stands showing the history of L'Oreal. Pretty pathetic eh? Struggling to prevent my face from scowling, I headed towards the table top experiments. Those at least had some element of science, but nothing particularly new to anyone who's done GCSE chemistry and I've got a bloody degree in it. Ooh dry ice. Ooh mixing acid and alkali to make something pop. No real relevance to the science of beauty, just an opportunity to make slightly tipsy people gasp.

With a sinking feeling that I was wasting my time I headed finally to the panel debate for "Why does beauty matter?" The panel members were Béatrice Dautresme, Executive Vice-President Corporate Communications and External Affairs for L’Oréal; psychotherapist Lucy Beresford; Carla Bevan, Editor of Marie Claire online; Sarahjane Robertson, Executive Director of the charity Look Good... Feel Better, UK; science broadcaster and author Vivienne Parry; and Professor David Perrett of the University of St Andrews. The panel members spoke in turns and then there was a Q&A. The whole thing was chaired by journalist Alice Hart-Davis.

Béatrice Dautresme from L'Oreal talked about a project they were doing cataloguing the history of beautification. We were treated to a promotional video. She pointed out that people have been doing this beauty thing forever, before companies came along. I got the feeling she was trying to subtly say "don't blame the beauty industry for making a big deal out of this, people have always wanted to look better". While it's true that people have always engaged in trying to make themselves look better, beauty companies have, I believe, changed it into more of an obsession than it should be. Lucy Beresford spoke about how beauty is tied up with self esteem. She quoted a survey that found 60% of women in London believed that wearing make-up gave them an edge at work. There was no discussion about why women believed this. I can't even remember what Carla Bevan from Marie Claire said. Sarahjane Robertson runs a charity called "Look Good... Feel Better" which gives support to women with breast cancer, namely with combating the visible effects of cancer treatment. A very worthy cause. Vivienne Parry basically took the opportunity to plug her new book on hormones and told us all about why we smell attractive to the opposite sex because of hormones. She's a pretty distinguished science communicator so I was surprised at how bubblegum her talk was. David Perrett showed some slides of faces and asked us to rate them on attractiveness. The point was that symmetrical faces are more appealing. Yawn.

Onto the Q&A. Someone asked the panel what they thought about the use of skin-lightening creams in India. Beatrice batted that one away telling us that light skin has always been seen as desirable because it meant you weren't working in the fields. I'm not kidding. No mention of L'Oreal's own racist policy of excluding non-white women from promoting its products. A man in the audience asked SarahJane Robertson if her charity provided services to men. It does. Other questions were not particularly memorable.

I left the debate feeling hugely disappointed and angry. Nobody talked about the science apart from giving the word itself a bit of lipservice. Nobody talked about whether our society is too obsessed with beauty or why beauty matters to much for women particularly.

In fact the whole evening was laughable. The most popular event of the evening seemed to be one where you could get a Cleopatra-style make-over. It all just felt like a promotional exercise for L'Oreal. Everywhere I went there were freebies. Being a cheapskate I took some. I now have samples of "UltraLift Pro-X re-plumping anti-wrinkle day cream", "Pureology" shampoos and conditioners and "Derma-Genesis Cellular Nurturing" day and night creams. I only took one sample of the anti-wrinkle cream (for research purposes!) and the woman doling them out told me to take as many as I wanted. When I said "it's OK, I'm only 27" she looked confused.

Sigh. I really did expect more from the Science Museum. I didn't expect PhD level science, but I hoped for something a bit more substantial. What I got wouldn't have intellectually stimulated a 5-year old. Even more disturbing was that the Science Museum completely sold out to L'Oreal, forsaking any exploration of the science (or lack of it) for a beautiful yet ultimately shallow promotional event. I know the Science Museum receives funding from companies but I don't expect it to compromise its integrity to do so.

Did I learn anything new? No. But I suppose I should stop thinking and just be grateful that I can re-plump my face with Pro-Xylane. Thanks Science Museum and L'Oreal.