The Yahoo! Data Breach Post

Yahoo! had a data breach in 2014, it declared the breach to consumers on September 22. This isn’t the first large data breach breach that Yahoo! has had over the past few years just the largest.

In 2012, there was a breach of 450,000+ identities back in 2012. Millions of identity records were apparently being sold by hackers in August 2016 that the media initially linked to the 2012 breach. It would be speculative to assume that the records for sale in August was part of the 2014 raid.

The facts so far:

  • 500 million records were stolen by the hackers. Based on the latest active email account numbers disclosed for Yahoo! many of these accounts are inactive or forgotten
  • Some of the data was stored unencrypted
  • Yahoo! believes that it was a state sponsored actor, but it has offered no evidence to support this hypothesis. It would be a bigger reputational issue if it was ‘normal’ hackers or an organised crime group
  • There are wider security implications because the data included personal security questions

The questions

Vermont senator asked the following questions in a letter to Yahoo!:

  • When and how did Yahoo first learn that its users’ information may have been compromised?
  • Please provide a timeline detailing the nature of the breach, when and how it was discovered, when Yahoo notified law enforcement or other government authorities about the breach, and when Yahoo notified its customers. Press reports indicate the breach first occurred in 2014, but was not discovered until August of this year. If this is accurate, how could such a large intrusion of Yahoo’s systems have gone undetected?
  • What Yahoo accounts, services, or sister sites have been affected?
  • How many total users are affected? How were these users notified? What protection is Yahoo providing the 500 million Yahoo customers whose identities and personal information are now compromised?
  • What steps can consumers take to best protect the information that may have been compromised in the Yahoo breach?
  • What is Yahoo doing to prevent another breach in the future?
  • Has Yahoo changed its security protocols, and in what manner?
  • Did anyone in the U.S. government warn Yahoo of a possible hacking attempt by state-sponsored hackers or other bad actors? When was this warning issued?

Added to this, shareholders and Verizon are likely to want to know:

  • Chain of events / timing on the discovery on the hack?
  • Has Yahoo! declared what it knew at the appropriate time?
  • Could Yahoo! be found negligent in their security precautions?
  • How will this impact the ongoing attrition in Yahoo! user numbers?

Additional questions:

  • How does Yahoo! know that it was a state sponsored actor?
  • Was there really Yahoo! web being sold on the dark web in August?
  • Was that data from the 2014 cache?
  • How did they get in?

More information
An Important Message About Yahoo User Security | Yahoo – Yahoo!’s official announcement
UK Man Involved in 2012 Yahoo Hack Sentenced to Prison | Security Week
Congressional Leaders Demand Answers on Yahoo Breach | Threat Post

Oprah time: Heaven’s Bankers – inside the hidden world of Islamic finance by Harris Irfan

I was given Heaven’s Bankers to read as a friend. I can’t say I had thought that much about Islamic finance before. I knew that it had a couple of patches of ‘heat’ behind it in the banking sector. One was in the late 1990s. It then took a back seat post-911 and took off again as Dubai boomed.

It helps that Harris was not only an insider, but passionate about banking in its widest sense. He’s also sickening polymath who is a top flight racing driver.

History never repeats itself, but the Kaleidoscopic combinations of the pictured present often seem to be constructed out of the broken fragments of antique legends. – Mark Twain and Charles Dudley Warner

Irfan delves into the intricacies of how modern Islamic finance grew and contracted. The industry he provides us an inside view of is now worth a trilliion dollars.  The start of history like most things were pretty straight forward. As the industry grew more arcane and complex financial instruments became the norm. This reminded me of a lot of Mark Lewis’ Liar’s Poker. Lewis dealt with bonds and modern derivatives became so complex customers didn’t understand them. The Savings and Loans debacle of 1985-1996 foreshadowed subprime mortgages.

Where Irfan really excels for the non-banker as reader is in his ability to break down the basics. He takes the concepts many of us learned in business or economics classes back into pre-medieval history. He provides a historical perspective on modern capitalism as we know it. So the book becomes invaluable regardless of how you feel about the current economic system. The background gives you a more informed perspective.

PrivaTegrity: the flawed model of distributed keys

Dave Chaum’s idea to to try and balance between state actors demand for internet sovereignty and the defacto end of citizen privacy, with the need to address emotive causes such as terrorism, paedophile rings and organised crime got a lot of attention from wired.
Yesterday evening on a bus stop in Bow
The principle behind PrivaTegrity is that there would be a backdoor, but the back door could only be opened with a nine-part key. The parts would be distributed internationally to try and reduce the ability of a single state actor to force access.

However it has a number of flaws to it:

  • It assumes that bad people will use a  cryptographic system with a known backdoor. They won’t they will look elsewhere for the technology
  • It has a known backdoor, there is no guarantee that it can’t be opened in a way that the developers hadn’t thought of
  • Nine people will decide what’s evil
  • If you’re a state actor or a coalition of state actors, you know that you have nine targets to go after in order to obtain access by hook-or-by-crook. It was only Edward Snowden who showed us how extraordinarily powerful companies where bent to the will of the US government. The UK government is about to grant itself extra-territorial legal powers to compel access. There is no reason why a form of extra-ordinary rendition couldn’t be used to compel access, rather like Sauron in The Lord of the Rings bending the ring bearers to his will. Think of it as Operation Neptune Spear meets a Dungeons & Dragon quest held at a black site

More information
The Father of Online Anonymity Has a Plan to End the Crypto War | WIRED
Privategrity

The changing culture of Silicon Valley

When I was in college I interviewed for a few placements, one was with Hewlett-Packard in Germany. They wanted a marketing student to look after their printing brochures on demand initiative for their UNIX product line. This was going to save them a mint in terms of marketing spend using an Indigo Digital Press rather than brochure runs on litho printing, reducing waste, storage needs and allow for faster document updates. (HP went on to buy Indigo in 2001).

Commercial adoption of the web was around the corner, I was already using it in college, but its ubiquity still seemed quite far away. I decided I didn’t want to go for the job primarily because I wanted to get my degree over and done with and HP weren’t paying that much for the role.

We were interviewed by a succession of people, the only one who was memorable  was a guy called Tim Nolte who wore a Grateful Dead tie and had a Jerry Garcia mouse mat in his cubicle.

At that time HP, had the dressing of the company man but had more than a few hippies on the payroll who permeated its culture. Reading Robert X Cringely’s Accidental Empires made me realise that technology was as much a culture war as technological upheaval.

If one looks at the icons of the technology sector up to and including the early noughties many of the people were influenced by the counterculture movement if not part of it. The  Grateful Dead where one of the first bands to have their own website at dead.net. The Electronic Frontier Foundation was founded by John Perry Barlow, a lyricist with The Grateful Dead. Steve Jobs was influenced by Indian mystics and his experiences using LSD.

Stewart Brand who founded WIRED magazine and The WeLL was the editor of The Whole Earth Catalog, a guide to useful things for people who wanted to get back to the land. He was influential in the early environmentalist movement and had been involved in the counterculture of 1960s San Francisco.
Members of the Golden Circle Senior Citizens Club of Fairmont holding quilt they made. The quilt was raffled off during the Fairmont centennial, May 1973
Ideas from open APIs and creative commons came from their libertarian values. Open Source Software again comes from academic and countercultural attitudes to information and has had to defend itself from accusations of communism, yet it now runs most of the world’s web services and gadgets from smartphones to Google’s search engine.

Reading the Cluetrain Manifesto is like reading a screed that could have come from an alternative Haight Ashbury.

Aeon magazine wrote an article on how yuppies have hacked the hacker ethos, but the truth is they’ve got behind the steering wheel as web2.0 declined. The move from open web API’s and the walled garden approach of Facebook and their ilk marked a changing of the guard of sorts.

Flickr had and ability to move your photos as a matter of pride in their product. Just a few clicks kept them honest and kept them innovating. Joshua Schachter’s similar approach on del.icio.us allowed me to move to pinboard.in when Yahoo! announced that it would be sunset.

Government always is the last to catch up, which is the reason why open data only really gained mainstream political currency in the past five years.

Were now in a Silicon Valley whose values are closer to the Reagan years and I am not too sure what it will do for innovation. I suspect that the change won’t be positive.

More information
Accidental Empires: How the Boys of Silicon Valley Make Their Millions, Battle Foreign Competition, and Still Can’t Get a Date by Robert X Cringely
Don’t listen to Bill Gates. The open-source movement isn’t communism. | Slate
How yuppies hacked the hacker ethos – Aeon

Tim Cook at The White House Cybersecurity Summit

Whilst on the surface this is a puff piece for Apple, but Cook uses the Obama administration’s call to cooperate making life easier for the intelligence industrial complex get access to consumer data and lays out an opposing vision.

He basically kicked Washington DC in the teeth, other significant companies just decided to turn up with a significantly less senior representative to send the same message.

The Facebook paper on mood research paper post

Over the weekend if you went on to quality (not Buzzfeed) news sites you would have probably seen something about a scientific paper that was published by researchers in the pay of Facebook on how emotion spreads through social networks.

There was a lot of copy written already about the experiment, so I recommend that you read The Atlantic‘s piece on it instead. There has been a lot written about whether it is moral, legal or ethical. As far as it being legal, Facebook’s highly paid legal counsel could provide a better steer on it than I could; and I suspect they would tell you it was completely legal.

As for the morals and ethics of it, I rather think that those are a mute point. Consumers emotional states have been tweaked for decades, the question of morality sailed with the rise of the mass market consumer product.
Guilty Viewing Pleasures: They Live
Whilst public relations as it is practiced now is more of a mechanistic craft; its father Edward Bernays viewed propaganda as a ‘modern instrument’ driven by scientific thinking including understanding of audience psychology to move people.

Advertisers utilised motivational research from the early 20th century on to create cognitive dissonance  with a consumer and then provide the product as a solution. The Atlantic carried an article on the psychology of advertising back in 1904. You are a better Mum if you wash your kids clothes with Persil, Cadbury’s Dairy Milk will put a smile on your face.

Political pollsters use voter psychographic profiling to induce a constituency result. We already live in the world of a malleable proletariat envisioned by by George Orwell in his novel 1984.

The people who are outraged by this need to get over it, log-in to Facebook less and realise that they are already sheep with a gallery of multinational shepherds herding them through their consumer lifecycle. What you can do is become more informed and read your environment in a more critical way.

More information
Everything We Know About Facebook’s Secret Mood Manipulation Experiment | The Atlantic
Experimental evidence of massive-scale emotional contagion through social networks by Adam D. I. Kramera, Jamie E. Guillory and Jeffrey T. Hancock
The Oxford Handbook of Propaganda Studies (Oxford Handbooks) the Auerbach and Castronovo edited anthology gives you pretty much everything you need to know from Bernays onwards about psychology and audience manipulation
Be the first to review this item
The Psychology of Advertising by Walter D Scott | The Atlantic (1904) – no that’s not a typo
Frontline: The Persuaders | PBS
Advertising’s Fifteen Basic Appeals | Jib Fowles

Algorithmic Accountability and other Big Data issues

In the past, what is now included in the envelope of big data resided with just a few organisations. The story of big data started with the US government. The government used a young company called IBM and their punch card technology to help tabulate their census data. Punch card technology started in the textile industry, where industrial revolution-era jacquard looms manufactured complex fabric patterns. Punch cards also controlled fairground organs and related instruments. It was with early tabulating machines made by IBM and others that started to change the world as we know it.
Computer History Museum
When the mainframe came along governments used them to manage tax collection and to run the the draft for Vietnam. It came a key part of the US anti-war protesters to destroy machine readable draft cards. (The draft card destruction didn’t affect the draft process. But burning the draft card was still an offence and some people underwent punishment.)

Also around this time, the credit agency was coming into its own in the US. Over a period of 60 years, it had gradually accumulated records on millions of Americans and Canadians. The New York Times in 1970 described the kind of records that were held by Retail Credit (now known as Equifax):

…may include ‘facts, statistics, inaccuracies and rumors’ … about virtually every phase of a person’s life; his marital troubles, jobs, school history, childhood, sex life, and political activities.

These records helped to vet people for job applications, bank loans and department store consumer credit. It was like a private sector version of the J. Edgar Hoover files. Equifax moved to computerise its records. One reason was to improve the professionalisation of its business. This also had an implication on the wider availablity of credit information. Computerisation led to the Fair Credit Report Act in the US. This legislation was designed to give consumers a measure of transparency and control over their data.

Forty years later, mainframe computers are still used to process tens of thousands of credit card transactions every second. New businesses including social networks, search engines and online advertising companies have vast amounts of data; unlike anything a credit agency ever had.

The recent The Social, Cultural & Ethical Dimensions of “Big Data” event held at New York University by the Data & Society Research Institute was important. Events like these help society understand what changes to make in the face of rapid technological change.

The Algorithmic Accountability primer from the event highlights the seemingly innocuous examples of how technology like Google’s search engine can have far reaching consequences. What the Data & Society Research Institute called ‘filter bubbles’. Personalisation of search will change that consumers see from individual to individual. This discrimination could also be applied to items like pricing. Staples has produced an algorithm that based pricing on location of the web user; better off customers were provided with better prices. One of the problems of regulating this area is first of all defining what an algorithm actually is from a policy perspective.

Algorithmic systems are generally not static systems but are continually tweaked and refined, so represent a moving target. During my time at Yahoo! we rolled out a major change to the search algorithm every two weeks on a Wednesday evening US west coast time. I imagine that pace of change at the likes of Google and Facebook has only accelerated.

The problem with many rules based systems now is that we no longer write the rules or teach the systems; instead we give the system access to large data sets and it starts to teach itself – the results generally work but we don’t know why. This is has been a leap forward for what would be broadly based artificial intelligence, but makes these systems intrinsically hard to regulate.
concern with data practices
Given all this it is hardly surprising that research carried out  on behalf of President Obama by The Whitehouse showed a high level of concern amongst US citizens.

More information
Jacquard Loom – National Museums Scotland
Separating Equifax from Fiction | Wired (Issue 3.05)
Data & Society | Algorithmic Accountability primer
This Landmark Study Could Reveal How The Web Discriminates Against You | Forbes
Websites Vary Prices, Deals Based on Users’ Information | WSJ
The 90-day review for Big Data | Whitehouse
Data & Society | Alogrithmic Accountability Workshop Notes
Digital Me: Will the next Cringely be from Gmail? | I, Cringely

Data privacy around the world

The Boston Consulting Group pulled together data about data privacy around the world.  This gives a really good view into consumer’s attitudes towards data privacy.  Location comes across as particularly high compared to other information like credit card details for surfing history.

Jargon Watch: microaggression

Time magazine had an interesting article about older people in the work force. One part of the article struck me; that older people were likely to face microaggression from colleagues. This was categorised as:

“brief and commonplace daily verbal, behavioral, and environmental indignities, whether intentional or unintentional, that communicate hostile, derogatory or negative racial slights and insults to the target person or group,”

Doesn’t sound that serious, until I read that it would

“affect older workers in the same way that they do members of racial minorities, eroding self-esteem,”

You can find out more:
The Costs of Age-related “Microaggressions” | AGEnda Blog
Oldest Boomers Are Increasingly Facing Discrimination in the Workplace | Time

Faster: cannon fodder and real-time media

At the beginning of the week, I read about Sarah Leibowitz  who is an account coordinator at Sparkpr* in New York. Sarah had sent an email to Valleywag on behalf of client Paltalk who had been mentioned in Edward Snowden’s PRISM slide deck.  Sarah mentioned this in her email introduction and the Valleywag journalist dissected her pitch so that the whole of the PR industry and the technology sector could read it. I imagine it’s going to take a good while for Sarah’s personal SEO campaign to bury this article.

As an industry we trust the core part of our value proposition to the most junior people often without the necessary oversight and coaching to prevent these kind of mistakes. It is not intentional, client demands and new business pitches get in the way of the best intentions: most of the time we get away with it. Show me an experienced PR person that hasn’t made a potentially embarrassing mistake at any time in their career and I will show you a liar.

We sell influence: by influencing influential people, be they bloggers, journalists, editors, researchers or DJs. Reputation management is an outcome of this process, strategy is the management of the process: neither are the core value we provide. Ms. Leibowitz is likely to have been a casualty of an industry trying to run ever faster.

Moving forward to this morning I read about David Monagan writing for Forbes who made a grovelling apology for calling Irish president Michael D. Higgins an “acknowledged homosexual”. Mr Monagan in his apology talks about having never made a mistake like that in his 35 years as a journalist; the mistake came about due to a tight deadline. It then went through the Forbes editorial process without question and was published. Mr Monagan is an acknowledged casualty of faster media s we move towards real-time.

Watching the content-free real-time TV coverage of the royal birth and the numerous lame brand #royalbirth hashtag hijack attempts this week showed me as an audience member how an attempt to do real-time media can wear down the the consumer as well as the other parts of the media chain. It is time that we opened a dialogue about taking a step back from this self-defeating treadmill.

More information
Apology as Forbes withdraws ‘homosexual’ claim about President Michael D Higgins | Irish Independent
Reminding People About PRISM Is a Terrible PR Strategy | Valleywag

Archived from blog posts I wrote for PR Week

On the Levenson report

I have been viewing the outcomes of the Levenson Report from afar and decided to revisit my first post on all this:

In the grand scheme of things the impact wasn’t that big. Whilst the News Of The World (NoTW) closed down, the replacement paper by News International has only managed to sell roughly half the NoTW’s circulation. I suspect that this is less about outrage and more about the disappearance of a well-loved brand – I was mildly surprised by the value in the NoTW brand.

News Corporation’s resilience. What is probably most interesting about the whole debacle is the way Rupert Murdoch has used the opportunity to split the firm in two and structure News Corporation for future growth. The company has also changed its approach towards its news media properties. With the split, there is a less sentimental approach and something similar to a fast-failure model has been in play. But this has also spurred innovation:

  • Closing down The Daily
  • The Times adopting a subsidised tablet model in a clear nod to the mobile phone industry

It was interesting that News Corporation used it as such a catalyst for change, either way it’s rivals will be competing against a leaner more dynamic business. They would have been better off with the status quo.

The confluence of interests. Whilst the Levenson Report was quite measured compared to some of the sentiment expressed, there was no way that it was going to get through on all recommendations. This would have upset the eco-system too much and there would have been likely blow-back in the future for the body-politic. Who knows the exact motivations but David Cameron’s administration took things about as far as they could. If one looks at the overall stance on the media industry from the Digital Economy Bill onwards, any greater moves would have been very out of character. The established media industry still has friends in power.

More information
The News Of The World: it’s probably not the revolution that you think it is
An enquiry into the culture, practices and ethics of the press by the Right Honorable Lord Justice Levenson – executive summary (PDF)

Anatomy of a hoax

Around about Monday lunchtime news of a spoof Shell site appeared on my network via Richard Heap’s Twitter account @richheap.


The site was eerily reminiscent of a crowdsourcing advertising stunt that General Motors had done six years ago to get a new advert for one of its large SUVs, The Chevrolet Tahoe. This was then hijacked by environmentalists, which set alarm bells off in the back of my head because this example is well known amongst the kind of people likely to be counseling Shell – a crowdsourcing site of this nature would be well supervised or just not happened as the reputational risk was too great.

Looking at the page source and and web console views (using the Firefox browser) for both ArcticReady.com and Shell.com showed that they were both similar, which made me wonder why ArcticReady.com domain was not just a redirect to a shell micro-site rather than apparently different site but built exactly the same. Some nice touches for authenticity included submitted artwork like this one below, which made it look like there was an agency asleep at the wheel:
trolled
How to spot it as fake:

The definitive evidence for me was looking at the WHOIS report for the domain:

Registrant: c/o ARCTICREADY.COM P.O. Box 821650 Vancouver, WA 98682 US Registrar: Domain.com

Domain Name: ARCTICREADY.COM Created on: 29-APR-12

Expires on: 29-APR-13 Last Updated on: 09-MAY-12

Administrative Contact: UIzZJb@PRIVACYPOST.COM c/o ARCTICREADY.COM P.O. Box 821650 Vancouver, WA 98682 US +1.360-449-5933

Technical Contact: KHJlKb@PRIVACYPOST.COM c/o ARCTICREADY.COM P.O. Box 821650 Vancouver, WA 98682 US +1.360-449-5933

Domain servers in listed order: A.NS.MAYFIRST.ORG B.NS.MAYFIRST.ORG

Here is the key information that the WHOIS record provides you. Firstly, the organisers who registered their domain did a reasonable amount of work to hide who they were.

The privacypost.com domain name used in the email address belongs to a company that provides privacy services to domain name registrars. For someone like Shell I would have expected a contact detail of sorts or maybe even the details of the agency responsible. But I’ve used privacy offerings by registrars before so this isn’t completely out of the question.

The name servers that point to where the website sites was more interesting. Mayfirst.org is a ‘politically progressive’ internet service provider. Whoever set the website up chose to use a hosting provider that wasn’t likely to buckle on first contact with the legal department of a large oil company. They aren’t the kind of provider that Shell or an agency would go to.  This is the big telltale sign.

Greenpeace then showed their hand as reports of closed Shell service stations affected by direct action started to appear on the news.

Ethics

Whilst the environment is important, and the issues surrounding deep-sea drilling in the Arctic deserve greater analysis; I was concerned about the quality of the site. It was not obvious that it was fake or satire fooling many of the the great and the good in the industry.

Instead it masqueraded as a groundswell of community action – what I would have called sock puppetry. As for the site itself, I have seen apparently successful phishing attacks built with less convincing sites.

Which brings me to a larger point. I am concerned the way organisations like Greenpeace are prepared to deceive the general public through online stunts like this. Yet if an opponent used similarly deceptive tactics there would be hell to pay. In the war of public opinion the first casualty is truth, if organisations are prepared to trample on that, how much further are they prepared to go?

There is no moral high ground of conduct, no discourse, there is only the dogmatic belief of the religious extremist; and it is that dogmatism which I find so distasteful. A website like this is the first step on a slippery slope. Not one that was slide down at great speed but in salami slices over time, which is the way moral transgressions usually progress – until you reach a place you never thought you would be and don’t really understand how you got there.

Occupy Hong Kong and the rise of social concern

If you had seen the ragtag nature of the Occupy protests in the UK, the orderly city tents underneath the HSBC building Hong Kong looks genteel by comparison.
Occupy Hong Kong
It seems to be part of a wider social concern springing up there which manifests itself in concern about local independent shops, the power of supermarkets and the excess produce thrown away by supermarkets that could have been beneficial to the less fortunate in Hong Kong society.
Untitled
More information
Hong Kong: are you a friend of the earth? Not if you shop at a supermarket – Jessie Tao’s blog

Is the internet too perfect a market?

The train of thought on this blog post coalesced when I was re-reading Kevin Kelly’s New Rules for the New Economy for the first time in a decade. Kelly’s book built on the work done by fellow Wired contributor John Browning who pulled together The Encyclopedia of the New Economy which was published over a couple of issues of Wired magazine and as a compilation in a now out-of-print pamphlet that used to sold via the Wired web site.

What is the new economy?

Back in the 1990s when the internet started to move out research and academia into the commercial and consumer world lot’s of things were happening.

The cold war had finished, television viewers had seen CNN revolutionise coverage of the Gulf War conflict and the Iraqi army had been routed largely due to technology (and overwhelming firepower). Proto-reality show The Real World was fresh, with David ‘Puck’ Rainey becoming the first reality TV villain to capture the public’s imagination. The M in MTV still stood for music; but also stood for ‘much innovative programming’; Gap had some of the coolest ads on TV and the record industry was making money like music sales were going out of style.

Francis Fukuyama’s political philosophy tract The End of History (and the Last Man) seemed to catch the spirit of the time in terms of a utopian vision of the future, even if most of the people who name-dropped his work had never read it.

People realised that the internet would change things, just in the same way that mobile phones had started to change everyday life (punctuality suddenly became passé, when you could phone ahead give your excuses and have a much more fluid schedule). It was going to change lots of industries perhaps creating a ‘new economy’ of online businesses. From a cultural point-of-view the new economy and the information superhighway was something to hitch one’s utopian hopes to with echoes of Roosevelt’s New Deal some 60 years earlier.

 The assumptions

The new economy was thought to bring about what economists would call a perfect market. Consumers would have information available at their finger tips and be able to compare the price of products throughout the world to get the best deal. There were even those who thought that consumers would have software agents to do this on their behalf and companies would have their power reduced by consumers. All of this change would be brought about by connected information and the rise of hobbyist communities who often knew more about a company’s products than the company themselves. This was seen to be a logical extension based on what people knew of the power of networks.

Consumer opportunities

Many of the early e-commerce businesses were arbitrage plays. Boxman had complex software from IBM that bought CDs from the cheapest distributors across Europe, shipped to its warehouse in Belgium and then shipped to consumers with some of arbitrage gained reflected in their discounted price. CD-WOW.com sold CDs from Hong Kong and other markets to UK consumers at prices that were up to 25 per cent cheaper than other suppliers. In the end, Boxman was brought down by poor software performance due to IBM learning about e-commerce as they went along and eventually CD-WOW had to pay £41 million pounds damages due to a prosecution brought by the BPI under the Copyright, Designs and Patents Act of 1988.

The ruling gave record companies a free hand to continue predation on UK consumers by supporting excessive prices on CDs compared to non-European markets. If it had been a bank instead of a record label, they would have been labeled loan sharks.

I worked on agencyside on the launch of a comparison shopping service called Dealtime UK (it re-branded to Shopping.com and is now part of eBay) which showed the price of CDs, consumer electronics shops and compared them across a swathe of retailers. Eventually search became a big part of the comparison shopping play with Google having its product search function and Yahoo! buying Kelkoo and tapping into that expertise to roll-out Yahoo! Shopping functionality across the international Yahoo! network.

Big Data

The demise of the dot com era saw changes in media consumption that went hand-in-hand with the roughly 30 per cent decline in online advertising spend bottoming out in 2002. Consumers started to find their way around the web in a different manner. Instead of having there homepage of their browser  as a personalised melange of news, weather and horoscopes served up by a portal website like Yahoo!, Excite or MSN; there was instead a search box from Google, Baidu, Naver or Yandex depending where you lived in the world.

As search engines tried to provide better results, they realised that context was important and that a record of what searches people did may make some sense of it. This data is immensely powerful. An example of how powerful it is was show by the AOL Search debacle. In August 2006, an AOL Research project put three month’s worth of search data for 650,000 users online. The data had been anonymised, but that didn’t stop the New York Times tracking down Thelma Arnold based on her search data. At the time I worked at Yahoo! we were gathering as much data each day from consumers as would be held in the US Library of Congress two times over.

It wasn’t only search engines that had this inferred data inside it, other businesses like Amazon had been gathering information about consumer’s preferences towards different products. Netflix like AOL released anonymised consumer data into the public as part of a programme to crowd-source a better recommendation algorithm. Privacy concerns were raised following work done by the University of Texas and Netflix pulled the data set following an agreement with the FTC.

Web 2.0

The web as a platform or web 2.0 came about out the ashes of the dot.com crash. The idea was that the web, had become a web of data that could be used through APIs to build new services and become more useful through mashing the data up. The key concepts that pioneers focused on was making the data usable and ensuring attribution of the data sets – (I’d recommend having a look at Tom Coates’ Native to a Web of Data presentation as a primer.)

One of the key things about this was that a number of the pioneers in this area like Flickr’s founder Stewart Butterfield said that APIs gave consumers power over their data, they could back up their images or take it elsewhere. Their content was exportable and market forces kept all the players honest and competitive.

However it could also be easily matched with existing data sets and much greater inferences derived from it.

Secondly, over time the moral imperative changed in these businesses. Facebook developed its site as being a digital equivalent of the Hotel California where you data can enter, but never leave. So as a marketer you have never had so much consumer information between the big data, inferred data and the ability of blending in further data to refine the knowledge further moves the needle from consumer to marketer in terms of economic power.

How could this be used?

  • Targeted advertising – based on understanding of the consumer behaviour, consumer spending power, life-state information. If you want to know the power of this information, look at how US supermarket Target wants to get hold of consumers as they are ready to start a family.
  • Targeted offers – save your best offers for people who are most likely to act on them
  • Dynamic cross-selling and up-selling opportunities – one of the biggest problems that we as marketers faced when I worked at MBNA a number of years ago was the irate consumers who would reach out when they had been offered a superior deal by us via mail. This need never happen again, instead inventory could be used to target them with additional services from a trusted brand
  • Differentiated pricing – this is where things get interesting. For luxury brands you could deliberately use differentiated pricing as a barrier to the kind of consumers you don’t want. For insurance companies you could use a much wider set of data to make inferences about likely risks for everything from health to their likely driving-style

The trust issue

As the Edelman Trust Barometer has shown for the past decade or so trust is extremely important in consumer – organisation interactions and ongoing relationships. Or as Kelly puts it:

With the decreased importance of productivity, relationships and their allies become the main economic event.

He lists attributes of trust that shows how difficult it is to foster, and how fragile it can be to maintain:

Trust is a peculiar quality. It can’t be bought. It can’t be downloaded. It can’t be instant

So where does trust leave the information imbalance between consumers and the organisation that they interact with? It’s a big challenge, Kelly points out that for trust to work consumers have to know who has the knowledge and a full understanding of that they know.  The problem is that organisations aren’t ready to have that adult conversation and full disclosure, particularly about what they can infer from the data that they have access to. The benefit that the consumer gets in relational activity is much less than what the organisation derives from that data. This would be especially true for someone like Facebook:
netbase on Facebook
Netbase looked at how consumers relate to brands, it indicates that many people feel that they have to be on Facebook rather than they want to be indicating that the consumer benefit is low, so the corresponding trust they are prepared to put into the social network regarding their privacy is low.

Companies like Facebook and Path are treating privacy as an inconvenient hang-up of consumers that they must run an end game around rather than engendering trust. The less engagement these businesses have with their audience the lower the quality of the information and the consequent lower utility that they have for marketers.

In the same way that consumers have a reduced trust in the media following incidents at organisations like News International; there is likely to be an inciting incident at some point between consumers and the online advertising eco-system that is likely to bring trust to a head. The perfect market knowledge advantage then becomes mute.

The internet becoming too perfect a market kills the golden goose being bad for consumers and bad for advertisers. The challenge is that the eco-system is a victim of its own success from 2002 to the end of 2011 the US online advertising market grew over four times to roughly 8 billion dollars a quarter. If someone steps back from the plate to take a more considered approach, someone else will rush in.

A prime example of this is the use of facial recognition software which even Google’s chairman Eric Schmidt agrees is a step too far, Facebook has already implemented it and Google has rolled it out as an opt-in feature on Google+. The problem with the opt-in is that Google doesn’t spell out to the consumer the full ramifications of the technology – yet it was obviously concerned at the highest level in order for Schmidt to go on record about it at a conference.

Looking at all this; it is counter-intuitive, but the market for consumer privacy should actually be with the brands that they are trying to engage them. How powerful would it be if a brand said: we can do all these things skulking around behind your back, pulling the strings but we aren’t going to and we don’t want to. We want to be a brand that you can address on your own terms.

The clock is ticking for the brand that will make that leap and the advertising eco-system that won’t. It was the Cluetrain Manifesto accused PR people of being afraid of their publics; were now in a situation where the online advertising eco-system is afraid of being completely honest with its audiences and afraid of their advertisers.

As Chuck D said:

The easiest and the hardest word to say is NO

More information
Here’s What Really Scares Eric Schmidt – Allthings D
Google+ Introduces Automatic Face Recognition To Photo Tagging (But It’s Completely Opt-In) – TechCrunch

Facial Recognition Technology: Facebook photo matching is just the start – PC World
Netflix Cancels Contest After Concerns Are Raised About Privacy – New York Times
CD Settlement forces prices up – BBC News
Tom Coates famous ‘Native to a web of data’ presentation that he gave at Future of Web Apps back in 2006  and Simon Willison’s write up of the presentation
How Companies Learn Your Secrets – NYTimes.com – How Target zeros in on consumers right around the birth of a child, when parents are exhausted and overwhelmed and their shopping patterns and brand loyalties are up for grabs
Where’s the Market for Online Privacy? | The Precursor Blog by Scott Cleland