Ged Carroll

Search results for: “"knowledge search"”

  • NORA

    Last week I heard the acronym NORA mentioned with regards the kind of problems that Microsoft’s algorithm could solve. NORA stands for no one real answer. Search is already pretty good at answering questions like ‘what time is it in Osaka’ or ‘what is the capital of Kazakhstan’.

    In the mid-2000s NORA would have been called ‘knowledge search‘ by the people at Google, Yahoo! and Bing – who were the main search engine companies. So its not a new idea in search, despite what one might believe based on the hype around chatbot enabled search engines. ChatGPT and other related generative AI tools have been touted as possible routes to get to knowledge search.

    Knowledge search

    Back when I worked at Yahoo! the idea of knowledge search internally was about trying to carve out a space that useful and differentiated from Google’s approach as defined by their mission:

    To organise the world’s information and make it universally accessible and useful

    Our approach to search – Google

    Google was rolling out services that not only searched the web. It also covered maps, the content of books including rare libraries and academic journals. It was organising the key news stories and curating which publications were seen in relation to that story. It could tell you the time elsewhere in the world and convert measures from imperial to metric.

    Google’s Gmail set the standard in organising our personal information, making the email box more accessible and searchable than it had been previously. We take having a journaled hard drive for granted now, but at one time Google Desktop put a search of the files on your computer together with online services in one small search box.

    DSC05757

    Being as good as Google was just table stakes. So when I was at Yahoo! we had our own version of Google Desktop. We bought Konfabulator, that put real time data widgets on your desktop and were thinking about how to do them on the smartphone OS of the time Nokia’s Symbian S60. Konfabulator’s developer Arlo Rose went on to work on Yahoo!’s mobile experiences and Yahoo! Connected TV – a photo-smart TV system that was before the modern Apple TV apps. Tim Mayer led a project to build out an index of the web for Yahoo! as large, if not bigger than Google’s at the time. And all of these developments were just hygiene factors.

    My colleagues at Yahoo! were interested in opinions or NORA; which is where the idea of knowledge search came in. Knowledge search had a number of different angles to it:

    • Tagged content such as my Flickr photo library or social bookmarking provided content from consumers about a given site that could then be triangulated into trusted context, or used to train a machine learning model of what a cat looked like
    • Question and answer services like Quora, Yahoo! Answers and Naver’s Jisik In Service improved search. Naver managed to parlay this into becoming the number one search engine for Korea and Koreans. Google tried to replicate this success with Knol and failed
    • Reviews. Google managed to parlay reviews into improving its mobile search offering. Google acquired Zagat in 2011. This enabled Google to build a reputation for good quality local restaurant reviews. It eventually sold the business on again to another restaurant review site The Infatuation

    The ChatGPT type services in search are considered to provide an alternative to human-powered services. They create NORA through machine generated content based on large data sets trawled from the web.

    Energy consumption

    A conventional Google internet search was claimed to consume 0.3 watt/hours of power according to Google sources who responded to the New York Times back in 2011. This was back when Google claimed that it was processing about one billion (1,000,000,000) searches per day. It accounted for just over 12 million of the 260,000,000 watt hours Google’s global data centres use per day. The rest of it comes from app downloads, maps, YouTube videos.

    But we also know that the number of Google searches ramped up considerably from those 2011 publicly disclosed numbers

    https://flic.kr/p/GN9TwD

    The driver for this increase was mobile search including more energy intensive Google Lens and voice activated searches thanks to Android.

    Large language models (LLMs) are computationally intensive and this will result in a corresponding rise in energy consumption. That also has implications in terms of business profit margins as well as ESG related considerations.

    Legal liabilities

    With NORA content being created by machine learning services, it might be different to the previous generation of knowledge search services. These services were platforms, but machine learning services become publishers.

    This becomes important for a few reasons

    • Increased costs (while they aren’t using an army of writers, they are using a lot of computing power to generate the responses)
    • Legal protections (in the US)
    • Intellectual property and plagiarism issues, currently they can handle it just by taking down the content. Once they become a publisher rather than a platform things become more complicated

    “no provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider”

    Communications Decency Act of 1996.Section 230

    Section 230 has been repeatedly used to regulate Facebook, Google et al in a lax manner as they haven’t been ‘publishers’, with ChatGPT this may change. The question of whether an algorithm is a creator has some precedence. Financial reporting has used machine learning to create news reports on company financial results over a number of years. Combine that with the general political antipathy towards Meta and Alphabet from both of the main US political parties and things could get interesting very fast.

    It is interesting that OpenAI is putting a lot of thought around ethics in LLM, which will impact future services and they probably hope stave off regulation.

    Regulated industries and liability

    Given an LLM’s ability to make things up it can:

    • Gives advice without pointing out health risks by creating a workout plan or a weight loss diets
    • Gives bad legal advice
    • Infringe regulations surrounding different industries like financial services

    This is just the tip of the iceberg that NORA content powered by LLMs face.

    Business model disruption

    Search advertising as we know it has been the same for the past two decades. The disruption to the look and feel of search results through Bing’s chat response has a negative impact on Google’s advertising model with the search ads along the top and down the right hand side of the search engine results page. Instead you’ll end up with the ‘correct’ answer and no reason to click on the search adverts.

    Currently if a non-relevant site shows up in Google. The lack of relevance is blamed on the site rather than the search engine. However an error in a machine learning created NORA response will see the search engine blamed.

    Which is pretty much what happened when Google demonstrated their efforts in the area. Inaccuracies in a demonstration held in Paris cause the share price of Alphabet to decline by 7 percent in one day. Technology news site TechCrunch even went as far as to say that Google is losing control.

    Microsoft probably doesn’t have a lot to lose in Bing. So integrating ChatGPT’s LLM might give them a few percentage points of search market share. Microsoft thinks that each percent gain would be worth 2 billion dollars in extra revenue.

    The 2 billion number is an estimate and we don’t know how the use of NORA results generated by LLM will affect bidding on search keywords. That 2 billion might be a lot less.

    Is NORA the user problem that Google and Bing’s use of LLMs are fixing?

    Around about the time that Google enjoyed a massive uptake in search it also changed search to meet a mobile paradigm. Research type searches done by everyone from brand planners to recruiters and students have declined in quality to an extent that some have openly questioned is Google dead?

    https://flic.kr/p/gWmAWy

    Boolean search no longer works, Danny Sullivan at Google admitted as much here. While Google hasn’t trumpeted the decline of Boolean search, ‘power’ users have noticed and they aren’t happy. That narrative together with the botched demo the other week reinforced each other.

    Unfortunately, due to the large number of searches that don’t require Boolean strings, Google wasn’t going to go back. Instead, chat-based interfaces done right might offer an alternative for more tailored searches that would be accessible to power users and n00bs alike?

    Technology paradigm shift?

    At first the biggest shock that myself and others had seeing the initial reports was how Google and Microsoft could have been left in the dust of OpenAI. Building models requires a large amount of computing power to help train and run.

    Microsoft had already been doing interesting things in machine learning with Cortana on Azure cloud services and Google had been doing things with TensorFlow. Amazon Web Services provides a set of machine learning tools and the infrastructure to run it on.

    Alphabet subsidiary DeepMind had already explored LLM and highlighted 21 risks associated with the technology, which is probably why Google hadn’t been looking for a ChatGPT type front end to search. The risks highlighted included areas such as:

    • Discrimination, Hate speech and Exclusion although there is research to indicate that there might be solutions to this problem
    • Information Hazards – there has already been a case study on how an LLM can be influenced to display a socially conservative perspective.
    • Misinformation Harms – researchers claimed that LLMs were “prone to hallucinating” (liable to just make stuff up)
    • Malicious Uses
    • Human-Computer Interaction Harms
    • Environmental and Socioeconomic harms

    Stories that have appeared about ChatGPT and Bing’s implementation of it seem to validate the DeepMind discussion paper on LLMs.

    The Microsoft question of why they partnered with ChatGPT rather than rolling out their own product is more interesting. Stephen Wolframs in-depth explanation of how ChatGPT works is worth a read (and a couple of re-reads to actually understand it). Microsoft’s efforts in probabilistic machine learning looks very similar in nature to ChatGPT. As far back as 1996, then CEO Bill Gates was publicly talking about how Microsoft’s expertise in Bayesian networks as a competitive advantage against rivals. Microsoft relied on research and the Bayesian network model put forward by Judea Pearl which he describes in his book Heuristics.

    Given the resources and head start that Microsoft had, why were they not further along and instead faced being disrupted by OpenAI? Having worked in the past with Microsoft as a client, I know they won’t buy into anything that they can build cheaper. That raises bigger questions about Microsoft’s operation over the past quarter of a century and its wider innovation story to date.

    Flash in the pan

    At times the technology sector looks more like a fashion industry driven by fads more than anything else. A case in point being last years focus on the metaverse. The resulting hike in interest rates has seen investment drop in the field. Businesses like Microsoft and Meta have shut down a lot of their efforts, or have scaled back. It is analogous to the numerous ‘AI winters‘ that have happened over the past 50 years as well.

    Bing’s implementation of LLM is already garnering criticism from the likes of the New York Times. This new form of search may end up being a flash-in-the-pan like Clubhouse. The latent demand for NORA in search will still be there, but LLM might not be the panacea to solve it. Consumers may continue to rely on Reddit and question-and-answer platforms like Quora as an imperfect solution in the meantime.

    In summary….

    • NORA content generated by LLMs represent a new way to solve a long known about challenge in online search
    • NORA as a concept was previously called knowledge search
    • NORA content competes with: social media including Reddit, specialist review sites including Yelp or OpenRice and question and answer services including Quora
    • ChatGPT and similar services affect human perceptions of search and the experience makes them more critical of the search engine response is not of an acceptable standard
    • LLMs represent a number of challenges that large technology companies have discussed publicly, but were still attractive for some reason
    • ChatGPT shows up the the decades of research that Google, Microsoft and Amazon have put into machine learning, this will negatively affect investors attitudes to these companies and merits a more critical nuanced examination of ‘innovation’. These large companies seem to be struggling to put applied innovation into practice. Microsoft buying into ChatGPT is essentially an admission of failure in its own efforts over at least 3 decades. Even ChatGPT’s deeply flawed product is considered to be better than nothing at all by these large technology companies
    • Use of ChatGPT like services expose Google and Bing to business risks that are legal and regulatory in nature. It could even result in loss of life
    • ChatGPT’s rise has surfaced deep seated concerns amongst technologists, early adopters, power users and investors about Google’s ability to execute on innovation successfully now (and in the future). Google’s search product has been weakened over time by its focus on mobile search dominance. Alphabet as a whole is no longer seen as a ‘leader’
    • LLMs, if successful would disrupt the online advertising business model around search engine marketing
    • ChatGPT and its underlying technology do not represent a paradigm shift
    • There is evidence to suggest that ChatGPT and other LLM powered chat search interfaces could turn out to be a fad rather than a future trend. The service as implemented has underwhelmed

  • Subprime attention crisis

    Subprime attention crisis is a short book, or a long essay depending on the way you want to look at it. It was written by Tim Hwang.

    IMG_1929

    About Tim Hwang

    Hwang is a lawyer working for email newsletter platform Substack. Prior to this he worked in a US think tank attached to Georgetown University: Center for Security and Emerging Technology and in public policy at Google focused on machine learning. So he brings a deep set of knowledge to writing Subprime attention crisis. One also has to bear in mind that his current employee Substack is based on the online media model moving from online advertising driven to subscription driven.

    Timing is everything

    I read this book over a couple of days at the beginning of this month. By this time, Meta and Alphabet has published quarterly results that were below what investors expected with falling sales. Add into the mix that the problems that Twitter and Snap have had (which are are bigger issues than just down to the dynamics of the online advertising market), all of which makes this book feel timely.

    On the other hand, one could also argue that much of the crisis had already landed. Ad tech businesses like Rubicon Project have either gone under or merged with their peers creating a massive amount of consolidation. The latest wave of consolidation happened in 2020 – 2021.

    Meta-specific issues

    Even with Meta and Alphabet there are business specific issues. Meta has struggled to compete effectively with TikTok. The poisonous nature of debates on Facebook, together with an aging audience on the platform hasn’t helped. In fact it’s a wonder that the context collapse that the platform has suffered from for at least the past six years hadn’t dragged it down yet. WhatsApp has helped enrich Facebook data and provided a channel for business services. At the time Facebook bought the business partly because Zuckerberg needed a brain trust for the future. The brain trust is gone and Zuckerberg’s dive into the Metaverse looks very similar to Apple’s peak John Sculley moment with the Knowledge Navigator concept. You can see glimpses of the Knowledge Navigator in the smartphone, the iPad, the now abandoned WikiReader product or the use of contextual information and national language processing like Siri. Apple didn’t waste the kind of money that Meta has spent chasing an illusory vision of the future.

    Alphabet-specific issues

    I was surprised that Alphabet growth had lasted this long based on the following considerations:

    With mobile, Google also pivoted a different type of search from product search to where is my nearest coffee shop with free wifi and has managed to sell search ads against them. This meant that Amazon and eBay managed to capture a lot of product searches, with consumers only hitting up Google afterwards and Amazon’s advertising has been eating Google’s lunch. Secondly a lot of the high street and neighbourhood shops have been eaten alive by food delivery services and this was then exasperated by the COVID which has changed at least some people’s consumer behaviour

    Historically, Google has been too focused on looking for multi-billion dollar opportunities which haven’t panned out and closed down smaller services that were making money and bringing in attention. In essence, over the years they have thought Google Reader, the Google Search Appliance, Google Health, Boston Dynamics and several other projects were the big payday. They weren’t, but they were respectable business opportunities, just too small for Google to want to pursue. In its wake Google had destroyed entire sectors, or turned them into cottage industries such as enterprise search and knowledge management, RSS newsreaders autonomous robots

    Web search in general has become less effective at doing deep research for consumer and B2B needs – no more support for boolean operators is a case in point. This has had some tech forward netizens wondering if the likes of Reddit fulfils the vision of knowledge search in place of Google and Alphabet being concerned about young people using TikTok as their local search box instead

    “something like almost 40% of young people when they’re looking for a place for lunch, they don’t go to Google Maps or Search, they go to TikTok or Instagram.”

    Google internal report quoted by Business Insider

    YouTube seems to struggle getting brand building advertising dollars in the face of TikTok, Instagram and this explains why you saw a decline in sales over 2 percent. Instead you see a lot of D2C product ads a la day trading and drop shipping courses advertised. Part of this might be down to the product. YouTube has been screwing over creators and creators have made it clear that they’re not happy. You don’t need to go to YouTube if you get the directors cut of your favourite creators content on Patreon or Curiosity Stream. Censorship of political analysis content around China or Ukraine seems to be particularly bad. 

    Back to Subprime attention crisis

    Hwang in Subprime attention crisis points out many of the things that agency employees and owners have known for years:

    • Online advertising effectiveness has declined compared to its performance 25 years ago
    • Audiences don’t see a lot of the ads that are displayed. Different reports will give you different numbers on this
    • Online advertising is destroying the very media industry that its content is shown on
    • Online advertising fraud is a big problem
    • Online advertising business practices are an even bigger problem with up to 70 percent of of online programmatic advertising spend going to advertising technology intermediaries such as The Rubicon Project (now Magnite) and Xaxis
    • This has allowed businesses like Procter & Gamble and adidas to reduce advertising spend at no loss in effectiveness. In the case of P&G Subprime attention crisis highlights how they cut $200 million in online advertising spend, moved that spend on to offline media like radio and print AND managed to increase their reach by 10 percent.

    More on adidas via its inhouse head of media Simon Peel

    One of the most notable things for me was being introduced to the work of Australian based academic Nico Neumann who has done some great research on online advertising effectiveness related areas including Frontiers: How Effective Is Third-Party Consumer Profiling? Evidence from Field Studies.

    So nothing surprising for insiders, but….

    Hwang marshals his facts well. Which is what you would expect from a lawyer. He uses analogous examples from the US financial services sector including the 2008 financial crisis. The book itself is 141 pages in length and there is a substantial section detailing his sources. Subprime attention crisis is based exclusively on desk research.

    More on the book here.

  • Animoca Brands + more news

    Animoca Brands

    Animoca Brands: How a big bet on blockchain and NFTs minted Hong Kong’s latest unicorn | South China Morning Post and more here The Sandbox developer Animoca Brands sees private valuation surge to US$5 billion amid metaverse, NFT frenzy | South China Morning Post – Animoca Brands has come up fast with The Sandbox. NFTs have become a bubble in Hong Kong. I was chatting to a good friend of mine who is now based in Shanghai. We talked about people we used to know who worked at various technology vendors. All are now involved in NFT businesses. The South China Morning Post has partnered with The Sandbox themselves. And then there is the speculation in metaverse property by Hong Kong’s oligopoly: Hong Kong property tycoons, brokers snap up virtual land in metaverse as valuations soar | South China Morning Post and deals like: CSOP AM launches metaverse ETF in Hong Kong | Financial Times 

    Animoca Brands is the face of a NFT bubble that feels curiously like the dot com era. Will there be a place for NFTs? Possibly. Will Animoca Brands be its Amazon or its Pets.com? I don’t know. But I feel queasy about the Hong Kong NFT wave that Animoca Brands are the poster child for. Particularly when on most other economic and societal metrics Hong Kong is running the other way; with a brain drain and economic decline.

    Beauty

    Omicron hits Wuhan, centered on cosmetics staff training – Global Times – centred around a training event by western beauty brand and multilevel marketing firm Nu Skin Enterprises – disclosure I used to work on the NuSkin brand in China and Hong Kong

    China

    Indian Tax Authorities Raid China’s Huawei, Triggering Protest From Beijing – WSJ

    Games changer: How China is rewriting global rules and Russia is playing along – European Council on Foreign Relations – Beijing and Moscow are unlikely to rush to each other’s aid during a military escalation, be it in Ukraine or over Taiwan. But the enabling environment of their mutual diplomatic support matters greatly

    Design

    Shedding some light on “dark patterns” and advertising regulation – ASA | CAP – “dark patterns” encompass a range of misleading advertising practices that have long been regulated under the CAP Code, and some of which reflect practices that are banned in all circumstances under consumer protection law.  The CAP Code has long applied to online advertising (including companies’ own websites), and many of the common “dark patterns” align with issues that the ASA is well-versed in regulating

    Economics

    UK risks spending more on defence equipment than it can afford, warns watchdog | Financial Times – damning NAO report

    Ukraine conflict will have a significant impact on Asia – Nikkei Asia – Look for the crisis to consolidate alignment among Asia’s democracies

    China Loosens New Mobile Payment Rules to Put Small Businesses at Ease – Caixin Global – the benefits of mobile payment oversight is going to mean less small and medium sized businesses fiddling their tax returns than currently happens

    Energy

    Tesla’s reverse on battery cells signals shift for electric vehicles | Financial Times 

    Wind Industry Warns EU to Take Urgent Action as China Rises – Bloomberg – wind industry will get screwed over just like solar and telecoms have been

    Ethics

    Ronn Torossian Admits To “Ethical Lapses” Amid News Site Controversy – PRSA-NY’s board of directors unanimously voted to condemn Torossian and 5WPR in response to the story. “In addition to being a cowardly and blatant violation of PRSA’s Code of Ethics, Ronn’s actions are a stain on our profession and undermine our role as guardians of facts and integrity for those we serve. We strongly condemn his and his firm’s direct role in perpetrating disinformation while pretending to be a legitimate industry news site,” said PRSA-NY’s board said in a statement. Torossian is no stranger to controversy, having been criticized over the years for his aggressive PR tactics, and is taking steps to remain in the public eye amid this one. Doing that has included issuing two press releases since the story broke — one offering Torossian’s list of “PR Rules” and another with marketing podcast recommendations. – so the lesson is basically break the rules while you’re small, apologise with no repercussions when you get larger

    EU to punish rights abuses in supply chains, with forced labour ban to follow | South China Morning Post – Bloc will require large companies to ensure their supply chains are free of human rights and environmental abuses, with fines for failing to comply. But the issue of forced labour, particularly complex for firms active in China, is not covered by the EU, which will address it with a separate ban

    Depicting older people in ads – ASA | CAP – Communicating about ageing and older people in a positive way can help to tackle negative perceptions of ageing, and older people, but negative and offensive stereotypes about ageing and older people are still common. Using stereotypes about age in advertising may breach the CAP Code, and our guidance is designed to help advertisers ensure that they do not include offensive depictions of, or references to age in their advertising. – but is this really going to change in the ad industry when ageism is endemic from the top down in the industry – from hiring policies and representation to board level views held by the likes of Mark Read

    FMCG

    Advertising zero alcohol products – ASA | CAP – Marketers should, however, take care not to mislead consumers by implying that a product contains no alcohol at all if it contains any. For some consumers, whether for health, religious, or other reasons, the presence of a small amount of alcohol may be material information and therefore required to be present with reasonable prominence. Although the ASA has not formally ruled on such a circumstance, marketers are best advised to take a cautious approach when marketing a drink that is usually alcoholic (such as a non-alcoholic beer) but has been adjusted to bring it below the 0.5% ABV threshold. For instance, we would strongly recommend that ads contain a reference to the ABV alongside any ‘alcohol free’ or similar claims. – interesting that the ASA felt the need to put this notice out

    Hong Kong

    Hong Kong to allow in doctors from mainland China as Covid cases overwhelm hospitals | Hong Kong | The Guardian – its also got an ideal opportunity to build up a DNA data bank of every Hong Kong resident which will help matching against debris from the 2014 and 2019 protests

    Ideas

    Lecturers admit self-censoring classes with Chinese students | The Times – Academics are self-censoring to avoid causing offence to students from authoritarian states such as China, a new report has said. Two thirds said they believed that academic freedom was under threat in higher education and more than two fifths felt the same about their freedom to select teaching content. The survey of 1,500 social science faculty members across a range of British institutions was conducted by academics from Oxford, Exeter and Portsmouth universities.

    Innovation

    Tencent-backed academic network to launch ‘open access’ journals | Financial Times 

    Startup Turns “Unrecyclable” Plastic Into Giant, Indestructible Construction Bricks it reminds me of Timbuk2’s lamitron experiment and might run into the same legal issues: Target Shuts Down Timbuk2’s Recycled-Plastic-Bag Messenger Bag Project « Mission Mission

    Japan

    Sony Ventures Corporation hits first close of its $215M fourth fund  | TechCrunch

    China temporarily detains Japanese diplomat | The Japan Times – interesting that China is manufacturing a dispute with Japan

    Korea

    Toronto bakery is burnt by a cultural appropriation feud | Financial Times – it is interesting to read about how Chinese cultural appropriation of Korean intangible national treasures has spilled over on to the western social networks. Naturally Vancouver seems to be at the centre of this maelstrom

    Luxury

    Sports car maker Lotus explores IPO options to fund global expansion | Financial Times

    Materials

    Chinese Researchers Uncover Massive Lithium Mine in the Himalayas | Sixth Tone – the Russians did extensive geological surveys and the US did a similar survey using a lot of satellite technology in Afghanistan and the nearby areas

    Easy aluminum nanoparticles for rapid, efficient hydrogen generation from water — Nano Magazine – if this can be commercialised; this has a huge impact for the nascent hydrogen economy

    Media

    Inside Facebook’s $10 Billion Breakup With Advertisers – WSJ – Apple’s privacy settings have affected e-commerce advertising on Facebook and advertising sales have dropped. It would be interesting to see if there was a geographic breakdown on this. If this is people like the Chinese online direct to consumer commerce and drop shippers, thats a big issue for Facebook. It already has issues with big brands in terms of ad quality, brand safety and skepticism over the reliability of Facebook’s ad metrics that is based on past behaviour

    Online

    Google Search Is Dying | DKB – interesting discussion on the Google search experience for early adopters. It is the kind of things that I have complained to friends about. It also shows the relative power of Reddit – which brings us back to the Yahoo! ideas around knowledge search circa 2005/6

    Retailing

    John Menzies accepts sweetened takeover offer | Financial Times 

    Security

    Attack on Ukrainian Government Websites Linked to GRU Hackers – bellingcat 

    Technology

    Intel in Israel: A Semiconductor Success – by Jon Y 

    Web of no web

    TikTok Wants to Avoid Facebook’s Mess. Its Corporate Culture Could Complicate That — The Information – Two years ago, a team of TikTok employees in China—where the hit video-sharing app’s parent company, ByteDance, is based—were excited to show their colleagues in the U.S. a preview of some new features they’d been working on. But the Americans were troubled when they saw one of them, which would let TikTok users darken or lighten their skin tone—a feature the U.S. employees feared would spur the creation of culturally insensitive videos featuring blackface, according to three people with direct knowledge of the matter. In another meeting, the China TikTok team showed their American counterparts a different feature that used an algorithm to scan users’ faces and tell them whether they were “beautiful” or not, according to one person who saw the presentation. After some employees raised concerns about the features, TikTok decided against launching them in the U.S.

    Web3: A Map in Search of Territory

    China introduces state-backed NFT platform unlinked to cryptocurrencies | South China Morning Post

    Tencent-led project becomes first UN-approved standards initiative on NFTs, known as ‘digital collectibles’ in China | South China Morning Post 

    Tencent quietly updates QQ with Unreal game engine in possible metaverse move | South China Morning Post 

    Chinese firms scramble to register metaverse trademarks despite Beijing’s warnings of risks | South China Morning Post 

    China’s growing market for NFTs, metaverse could foster new money-laundering schemes, central bank official warns | South China Morning Post 

    China plans to accelerate blockchain development and adoption in push to become a world leader in the technology by 2025 | South China Morning Post 

    Meta shows ad agencies metaverse—it looks a lot like Snapchat | Ad Age 

    Wireless

    Telenor investors scrutinise Myanmar sale | Reuters 

  • Yahoo Answers

    Back in 2005, I worked in the search group at Yahoo!. One of the projects that I worked on was Yahoo Answers. 16 years later, Yahoo Answers is being closed down. I thought I would capture some of my memories and inside knowledge on Yahoo Answers.

    But first we need some context so that what I write later about Yahoo Answers will make sense.

    The beginning

    Let’s go back to the beginning. Back to the early-1990s. Jerry Yang and David Filo founded Yahoo!. It fits the classic Silicon Valley archetype story and you can find plenty of accounts of it elsewhere. The key is what Yahoo! originally was. Its a list of links for websites. Once the list grew above 200 links or so; Jerry and David came up with a way of displaying this list by grouping it into subject areas.

    grass

    What would later be called a web directory. There were other directories around about this time like:

    • Best of the Web – which surprisingly still exists
    • Netscape Communications had their own directory when they acquired Gnuhoo, this eventually became DMOZ and then Curlie. Gnuhoo did rely on a search engine to help you find things in their directory. This is available as open source code at GitHub

    All of them had a certain amount of editorial input over what was good. Yet Yahoo! became the top one through buzz marketing – cheap ways to do brand building.

    When I was there, I worked with an agency to organise event hijacking at the Harry Potter and the Half Blood Prince book launch at Waterstones flagship store on Oxford Street. Yahoo! would vinyl wrap any employee’s car for free. There were also strategically placed billboards, such as this one in San Francisco.

    https://flic.kr/p/6vXVff

    People who managed this directory were known as web surfers. But there was also search engines out there, like the Knowbot search engine for Telnet developed in the late 1980s. There was Archie which was the closest to what we think of as a search engine now. Archie searched FTP archives around the world.

    As computer science post-grad students, Filo and Yang would have been familiar with the idea of the search engine. At the time David Filo felt that no machine would provide better filtering than a human. Media accounts of the time showed that Silicon Valley venture capitalists were all in favour of search engines over directories.

    Peer companies like:

    • Webcrawler
    • Metacrawler
    • Lycos
    • Ask Jeeves
    • Infoseek
    • Excite
    • AltaVista

    All offered what we’d recognise as reasonable search experiences. But Filo’s comments on human filtering is something that we will revisit later.

    Web portal & web advertising

    Search engines were the future but as the dot com era took off it wasn’t apparent how to monetise them.

    https://flic.kr/p/oFkWeG
    Yahoo! home page early on the morning of March 3, 1999

    At the height of the dot com era; Yahoo! had about 40 million users a month. You have to remember there weren’t that many people online in comparison to now. Internet usage had grown from 45 million users in 1995 to over 410 million by 2000. At the time it didn’t seem to matter that Yahoo! took longer to load as a website compared to its peers. Longer page times, meant that you could get away with less equipment in your data centre hosting the website and supporting infrastructure.

    The internet didn’t give birth to culture in the same way that memes, influencers and platforms do now. Instead it was the meme. It was all over the mainstream media, often tied up with ideas of cyberpunk culture, bulletin boards and the ‘information superhighway’. Examples of this included:

    • The Site by MSNBC
    • The i in iMac was for internet. The idea was that you could take the computer out of its box, plug it in to your wall socket and telephone socket. When you turned it on, it would configure you an internet service. The cool product design was a byproduct of this internet appliance plus personal computer thinking
    • Movies: The Lawnmower Man, Hackers, The Matrix, Ghost In The Shell
    • Books: Snow Crash, William Gibson’s Neuromancer
    • A plethora of internet magazines, including Ziff-Davis’ Yahoo! Internet Life which was a mix of technology and gadget reviews, media and celebrity content and website recommendations. Yahoo! Internet Life was published from 1996 to 2002

    It felt like something big was going to happen, even if we didn’t know what it was. What was obvious was the potential for advertising online. And the clearest analogue was newspaper advertising due to the long page format of web pages.

    Web portals came about for a number of reasons:

    There was now the technology to pull content from different sources together. You would have:

    • Weather forecast
    • Horoscope
    • Up to date news
    • Local information (for major cities like San Francisco)
    • Business
    • Finance
    • Entertainment and celebrity news

    Like the newspaper before it, it offered the first media you needed, but on the web.

    It was mainstream enough for brands to advertise against for brand building.

    By the time I was leaving college, Yahoo! Mail accessed through the Yahoo! home page made perfect sense.

    So before the dot com bubble bursts Yahoo! had a major media business valued at 2.8 billion dollars, or about $70 dollars per user. Which sounds expensive, but when you consider that Google is now worth about $386 per user, it’s not that bad. Secondly, online advertising per impression was much more lucrative back then and ad fraud was much less of an issue.

    What there wasn’t was a way of taking advantage of the highly relevant search results provided by search engines and adequately monetising them. So companies had three ways of monetising search:

    • Companies created portals the so called ‘homepages of the web’ to put display adverts on like My Yahoo! or MSN.com and search was a service alongside news, weather and horoscopes
    • They became infrastructure companies selling search functionality in the background a la Inktomi
    • They sold inclusion in their directory. This was controversial as it went against the editorial integrity of the directory and still a hot button when I arrived at Yahoo! in 2005

    The bubble bursts

    In the US stock market we had was now known as the internet or dot.com bubble. Looking at the NASDAQ composite data, it seemed to start in the last quarter of 1995, six months or so before Yahoo! went public in April 1996. It reached its nadir in the last quarter of 2002.

    In reality, this was more than about websites. Telecoms deregulation, satellite networks and the rise of cellphones had seen a boom in new companies and network equipment providers to support them. The need for servers had created booms in:

    • Computers: SGI, Sun Microsystems and IBM
    • Networking equipment: US Robotics, 3Com, Cisco
    • Software: VA Linux, RedHat, Open Text
    • Software as a service: I2, Salesforce, NetSuite
    • Web hosting and ‘data hotels’: Equinix, Intel, Rackspace, PSINet
    • Telecoms and ISPs: Level3, Global Crossing, Earthlink, Iridium, GlobalStar, AOL, @Home Network
    jjv40_50706349946_o_50743198928_o
    NASDAQ composite index covering the dot com boom and crash

    Add into that artificially high growth in earnings for enterprise IT companies in the run up to the Y2K bug issue and the whole sector was left with a bad hangover.

    Eric Steiner tells his tale as the CEO of Inktomi in 2004

    Steiner’s talk is interesting because it shows how the search business, selling search capability to the likes of Microsoft, Amazon and eBay had slow and steady growth rather than outstanding growth during this time.

    Yahoo! went through a traumatic time. When I worked at Yahoo! Europe, I was told online advertising sales dropped to a third of what they were during the dot com boom. The European business managed to hold on by its finger tips thanks to revenues from online dating services.

    Some of the ‘smart bets’ Yahoo! made during the boom times looked like hubris. The exemplar of this was Yahoo!’s acquisition of Broadcast.com. Broadcast.com provided video streaming (then called web casting) and internet radio services. It was the technology partner for the first online Victoria Secret Fashion Show streamed online. Yahoo! acquired it for 5.6 billion of Yahoo! stock. This was a bad decision, but thankfully, they didn’t pay cash.

    When I joined Yahoo! the Broadcast.com acquisition was still a scar on acquisition decision-making. You can attribute the impact of this to subsequent failed purchases of Google and Facebook.

    GoTo and Google

    In 1998, the company GoTo.com launched paid advertising placement in search engine results. The next year they introduced real time bidding. It was renamed Overture and started providing these services for Yahoo! and others. It started to become successful as a business.

    Meanwhile, Google had moved from a research project to a serious search engine. In 2000, Google began selling advertisements associated with search keywords. This was against Page and Brin’s initial opposition toward an advertising-funded search engine, they saw themselves more as a ‘search appliance’ business rather like Inktomi. Yahoo! adopted Google search around about the same time that Google started its search advertising business.

    This put Google in front of a large number of consumers and helped Google further refine its search engine.

    Google’s own offering was the exact opposite of Yahoo!. It prided itself its clean design with just a search box. Google also had a fanatical obsession with reducing page load times and the time taken to return search results.

    This was what more and more people wanted. Google used the dot com crash to build its business and its infrastructure. It wasn’t until its 2004 IPO that rivals realised how much of a head start Google had.

    Google revolutionised data centre server design, reducing cost and increasing the amount of servers that it could use. By contrast every Yahoo! data centre hardware purchase went via David Filo. If you used Yahoo! small business hosting, you were using tired and almost expired Yahoo! servers. In retrospect, they looked after the datacentre pennies, but let the pounds slip away.

    2003 saw Yahoo! get serious about the search engine business. The company purchased Overture which included GoTo.com and Altavista. But the problem was that even if Yahoo! built a search engine as good as Google, it didn’t matter if people didn’t use it. During my time at Yahoo! there was a push to get the necessary servers in place and a product that was as good as Google. However there was a constant tit-for-tat feature development in the search space. By this time Google had already verbed. The Google habit means that its hard to compete against them.

    I heard that inside Microsoft they tried to take drastic measures to persuade employees to use Bing over Google. When I worked at Yahoo! people used Google a lot too.

    The only way to compete with Google was to have a different idea. Google defined its mission is to organize the world’s information and make it universally accessible and useful.

    Yahoo! needed a new idea that was distinct from Google’s mission. The idea was knowledge search.

    Knowledge search and Yahoo! Answers

    Knowledge search as a concept was well under way by the time that I arrived. It was to capture and make searchable all the ‘knowledge’ (rather than information in the world). Opinions, experience and recommendations are knowledge rather than information. Yahoo!’s web 2.0 acquisitions including Flickr and delicious were made to support this vision.

    Tagging built up words and associations with web links and images, effectively human filtering – some of which would be used to train machine learning algorithms. The next logical step would be to build a repository of knowledge by the people, for the people. That’s where Yahoo! Answers came in.

    The inspiration for Yahoo! Answers came from a product that Yahoo! Taiwan had rolled out. It in turn probably inspired by Korean site Naver Knowledge IN. Bradley Horowitz apparently claimed that Yahoo! Answers was inspired by Naver Knowledge IN directly.

    Knowledge IN was designed to encourage user created content, since there wasn’t much material on the Korean web at the time.

    When I heard Jerry Yang talk about it internally at the time, he talked bruskly about a product built by Yahoo! Taiwan as having inspired it. Jerry didn’t do jet lag well and came across as morose on the couple of times I saw him in Europe, so wasn’t exactly an effusive speaker.

    Yahoo! Answers was championed by Jerry and that blessing allowed it to be pushed through when so many other product died before they got pushed to beta. It makes sense to point out the human crafted nature of Yahoo! Answers. In this respect it can be seen as a direct line back to the original Yahoo! directory product. Both were fuelled by a belief that people had some ability that was better than machines.

    Qi Lu was responsible for new products within the core search business and the troubled Panama search advertising project at the time. Weekly conference calls saw a plethora of existing projects cancelled, or reprioritised by Qi Lu, while new ones would suddenly appear. This constant change in the roadmap mean’t a lot of wasted efforts.

    Yahoo! Answers and much of the knowledge search related acquisitions sat under Bradley Horowitz. Tim Mayer was focused on the commercial side of things, although there was some overlap in the roles. Eckhart Walter sat above Tim. Jeff Weiner was the main shot caller having both Search and Marketplace businesses reporting into him. If you’re thinking, that’s a lot of senior management involved. You’d be right, there were a lot of managers with varying degrees of responsibility involved.

    But they were all good people and I’d be happy to work with them again.

    Prior to Yahoo!,I had been working agency side for Transversal. Transversal powered the support functions for a number of companies including Sony Playstation. I had a good idea how much this service was priced and floated the idea of sponsored channels for instance around Sony Playstation and had a good idea how much Sony must be paying to support user troubleshooting.

    But it didn’t fit that well as an idea with knowledge search.

    Concerns and how is babby formed?

    In the European team we had some concerns about Yahoo Answers like how was it going to get monetised? The quality of the content was also a concern. Knowledge IN and similar services in Asia work partly due to culture. We were worried when it hit a more individual-focused culture like the US or Europe.

    Another problem was calibrating the rewards within the system. Its really hard to get the balance on good quality questions and answers. Generally people who are time rich, aren’t necessarily the best respondents. If you need one proof point to show how much of a failure this was, you only have to look at the how is babby formed? meme.

    Rewards aren’t the only problem however. The second issue was the way the community was built. Generally, a great community is built carefully from like-minded people. With flickr it was around the passion of photography. Facebook is actually closer to Reddit, built on groups of groups. The death of a group dynamic won’t necessarily kill the platform.

    I was involved in early seeding of the initial content on Yahoo! Answers. I answered 42 questions, the first one question I answered was ‘What to take from airport to downtown Munich?‘ My response: The taxi is reasonable, it cost me 30 Euros – which shows the contextual nature of knowledge search. 30 Euros was reasonable for me at the time, since I could expense it back, but it wouldn’t be reasonable for a backpacking traveller.

    I also wrote six questions, the first one was ‘Has anybody got a Pentax K100D, if so what do you think of it? What are its pitfalls and what aspects of it do you particularly like? I wanted to get a a bit more colour beyond the reviews I’ve read online. – I was getting ready to leave Yahoo! and was going to buy a DSLR camera to take better pictures on my Flickr account. I deliberately structured the question to get opinions from early users. The Pentax K100 had recently been launched.

    Careful community management is at odds with a platform trying to capture the world’s knowledge. So the Yahoo Answers community was built for rapid global user growth. For the English language versions at least, there was a global content index, sitting on top of a distributed Oracle database.

    This meant a clash of cultures and variable quality content. I quickly found the site unusable for productive questions. Yahoo! spent the next few years trying to perfect it. People that formerly worked on Yahoo! Directory and front page brought their content and editorial skills to bear on Yahoo! Answers.

    I suspect that trying to monetise the service would have been a constant challenge. Yahoo! Answers provided variable quality answers for children’s homework and was the butt of memes. Neither of which are an ideal recipe for the kind of content large brands like Procter & Gamble would want to put their name against.

    Quora’s lean pickings

    Google tried to do it better with Google Knol and also failed.

    Quora was formed in 2009 and managed to build a better community, but I’ve still seen a steady decline in the quality of their answers. In 2019, they had a user base of 300 million people and total revenue (from advertising) of 20 million dollars. Thats an ARPU of 6.6 cents. That’s not a good internet media business. From that 20 million, they need to pay their infrastructure costs, maintain and improve the product, pay the salaries of their 300 employees. And I haven’t even talked about how their investors must feel.

    Knowledge search is still a technology challenge waiting to be conquered.

    More information

    Yahoo: a history of the internet in 5 acts – Financial Times (July 25, 2016)

    Max Roser, Hannah Ritchie and Esteban Ortiz-Ospina (2015) – “Internet”. Published online at OurWorldInData.org.

    How Yahoo! Won The Search Wars Once upon a time, Yahoo! was an Internet search site with mediocre technology. Now it has a market cap of $2.8 billion. Some people say it’s the next America Online. by Randall E. Stross – Fortune (March 2, 1998)

    Britannica Online – authoritative top level history of Yahoo!

    Room for the Internet; Combining a Data Center With a ‘Telco Hotel’” – New York Times (May 14, 2000)

    Take Naver Global Today! – Korea IT Times

  • Magic donkey

    I was reading Rob Manuel’s Facebook post about the origin of B3ta’s ‘magic donkey’ and its wider connection to the modern dystopian web experience.

    B3ta and magic donkey

    I guess it makes sense to first explain what B3ta is. B3ta is a community of bored driven people with a creative bent and a finely tuned sense of the absurd. The Guardian described it as a ‘purile digital arts community’. To be fair the works are more multi-displinary, than just digital in nature. Brands and memes get hacked.

    Ball's ice cream by S4RK on B3ta

    Its humour and its contributors are mostly British.

    B3ta was founded as a website and forum back in 2001 and I found it as a passive consumer a few years later. The front page and weekly email sent to members curated a selection of the content in the forums. Whilst contributors weren’t paid, there was a lot of kudos to getting your content on to the front page of the website, or into the weekly email that went out to the community of creators and consumers.

    This meant that Manuel was under pressure by contributors to put their work on the front page or in the weekly update email. The ego of the creator is familiar to anyone who has watched TV shows or films such as:

    Manuel invented an imaginary editor to deflect pressure away from himself. Of course, imaginary editor had to be slightly absurd. Hence a magic donkey.

    Flickr and the magic donkey

    While Rob Manuel was responsible for trying to fend off B3ta contributors aspirations to get on the front page, Cal Henderson was responsible for the technology. He had been a co-founder of B3ta alongside Manuel.

    Henderson is better known as a long time collaborator with Stewart Butterfield and CTO of Slack. But before that he was responsible for the technical aspects of B3ta and then moved on to Flickr.

    Flickr had a strong tight community, with agreed well-adhered to rules. A large part of this was down to the Flickr team including co-founder Caterina Fake and Heather Powazek Champ. The community met up in real life; rather like users of Chinese network Douban had been known to do. Flickr users were also good at organising their pictures providing labels or tags for images. But as the community scaled, surfacing the right content at the right time would have been more difficult.

    Henderson word on a algorithm-driven function called ‘interestingness’ that surfaced ‘the best’ content on a particular subject. Here’s what Steve’s Digicams said about is likely to go into the ‘Interestingness’ algorithm.

    There are a number of factors that go into what makes a photo interesting including, the number of tags it has, the number of groups it belongs to, how many people have viewed the image, and how many people have made it their favorite.

    Steve’s Digicms – What is Flickr Interestingness?

    Cal Henderson called the algorithm ‘magic donkey’. This would be a substitute for the curation done by an editor or a community manager and be applied across all subject areas. If the descriptions of Henderson’s interestingness algorithm reminds you a bit of Larry Page and Sergei Brin’s original working paper The PageRank Citation Ranking: Bringing Order to the Web, you’re probably right. At a base level both seem to rely on different feedback mechanisms to provide a reductive way of resolving what to show. Feedback as a concept is a hugely important role in computing and technology. Bell Labs were using feedback in its solutions to reduce noise on telephone lines at the start of the modern electronic age. Now feedback and analysis is done thousands of time a second to try and provide robots with some form of situational awareness.

    Magic donkey, search and social search

    Just over 12 months after it was formed, Flickr was purchased by Yahoo!. Yahoo! was interested in flickr for a number of reasons:

    • Yahoo! (and Microsoft) were fighting a losing battle against Google’s search engine and needed an edge
    • Web 2.0 had started to take off and Flickr was a cool property in this space.

    At that time search lacked meaning and context. To help you understand what search was like back then. I used to use the analogy of a shop assistant

    Imagine going to the supermarket and asking the assistant for an item, they run down the corridor and run back with their arms full of different stuff. They empty the stuff into your trolley and say to you ‘Your item is in there’. If you are lucky, the item is at the top of the pile, it you aren’t you may sort through it all and find you don’t have it anyway. You complain to the manager and he dismisses you with ‘Its your own fault, you asked in the wrong way’.

    Folksonomy.

    In order to deal with the meaning and context problem, all the main search engines brought out vertical search services

    • Video Search
    • Maps
    • Blog search
    • Google Scholar
    • Shopping search

    And ‘easter eggs’ such as providing information on local time in different cities or countries and measurement conversions. Whilst these weren’t great at driving advertising revenue they encouraged usage. Search became a giant Swiss army knife for knowledge workers.

    All the search providers had a keen interest to the GWAP (games with a purpose) work that was being done at Carnegie Mellon University by Luis von Ahn. von Ahn is a specialist in the field of ‘human computation’.

    Human computation was providing machine learning something to learn from. You want to teach a machine learning algorithm how to identify cats? von Ahn’s ESP game was the ideal teacher. In the words of von Ahn’s own page at Carnegie Mellon University:

    The first GWAP developed by von Ahn, the ESP Game displays images to two players who each try to guess words that the other player would use to describe the image. The game improves web image searches by generating descriptions of uncaptioned images. Google Inc. has licensed the game, which the company calls Google Image Labeler.

    Games With A Purpose – Carnegie Mellon University

    von Ahn went on to design other games that would have a similar utility

    • Matchin, a game in which players judge which of two images is the more appealing. (This might introduce cultural bias and would probably be much more problematic now.) Back in the late noughties this was seen as progress towards better search. Automating systemic racial bias just wasn’t on the radar and ‘bro culture’ wasn’t as prevalent in its engineers
    • Tag a tune – which looked to get genres and descriptors like happy or sad music
    • Verbosity – tests ‘common sense’ knowledge to build facts for machine learning platforms like ‘you shouldn’t walk under ladders’

    You can still try Google’s use of GWAP here. Though most people are more used to engaging with GWAP functions as part of CAPTCHA and reCAPTCHA verification services. Google used reCAPTCHA and CAPTCHA technology to digitise the archives of The New York Times and libraries into Google Books.

    Yahoo!’s answer to this has been variously termed knowledge search and social search. The idea was to improve the quality of results through people and provide context through human effort. A few of the things were in Yahoo!’s favour for this approach.

    Heuristics that support social search

    Search like many categories of commerce, tends to follow the principle of the long tail. The bulk of interest or transactions are the head. This existed pre-Internet; if you’re of a certain age you’ll remember that most people seemed to have a Sade or Dire Straits album in their CD collection. Ed Sheeran or Beyonce on their Spotify playlists would be a similar phenomenon now.

    Search is quite similar. The biggest searches on search engines are likely to be something along the lines of:

    • Porn
    • Google (on Yahoo! or Bing)
    • Amazon
    • eBay
    • Facebook

    (The lists that you see published of the top searches put out by Google, Yahoo! etc are usually cleaned first by the PR teams; so have limited value as trustworthy information sources.)

    A lot of searches are looking for things that you’ve found before online. Whether it was a particular article or a website that you use on a regular basis.

    Social search was manifested in a number of different ways. Questions and answer sites had originally got popular in east Asia, notably Korea, Taiwan and Japan.

    Jerry Yang himself got behind the launch of what would become Yahoo! Answers following the popularity of a Q+A service launched by Yahoo! in Taiwan.

    Yahoo! had an interest in tagging and folksonomies as a way of providing context around content. In a similar way to the way lexemes work.

    So if you were listening to a report on the stock market. The report wouldn’t necessarily have to use the phrase stock market to indicate that was what the report was about. There would be lexemes – words associated with the concept of a stock market that would be indicators for instance:

    • Wall Street
    • Bull market
    • Bear market
    • Standard & Poors 500 (S+P500)
    • NASDAQ
    • Share price
    • FTSE
    • The Hang Seng
    • The Nikkei

    There would be similar language for other subjects as well. This allows for one item to be in multiple categories. Yahoo! acquired Flickr which helped because it had a community that tagged their images.

    Yahoo! also launched a series of social bookmarking services. Remember what I said earlier about people often searching for things that they’d found before? Well a social bookmarking tool offers a few benefits

    • Your bookmarks exist online so you don’t have to worry about getting access to your browser bookmark folder at home, at work or on the move
    • You organise things using the language that makes the most sense for you
    • You can search amongst links that you’ve found before
    • Searching amongst content that you and others like you chose to bookmark should raise the overall quality of the links that you are provided with

    There was Yahoo! MyWeb, MyWeb 2 (beta) and it then acquired Delicious. Stewart Butterfield used to joke that Yahoo! bought flickr because they thought flickr were the ‘tagging people’; when they’d really just been copying the feature from Joshua Schacter at Delicious. Yahoo! then went on to buy social bookmarking site Delicious as well.

    The problem is that to tag your bookmarks and content carefully requires a discipline that many people struggled to maintain. I have found it to be personally beneficial over time, but I had a strong incentive to stick with it; even then I have been far more lax on my photo tagging since I no longer use Flickr’s desktop app to upload my photos.

    Changing behaviour is hard; when I worked there, reputedly a higher proportion of Yahoos used Google search than the general population. I heard that there was a similar behaviour pattern at Microsoft.

    There is also a certain irony in Henderson et al falling back closer towards a Google PageRank citation / feedback-type model of algorithm given the nature of a more human-powered and humane ideal of social search.

    Magic donkey and content firehose

    Cal Henderson literally wrote the book on scaling websites to cope with the kind of growth you would see driven by social web applications such as photo sharing, bookmarking and social networks.

    But social networks grew at a phenomenally fast rate. You could never log on to Friendster. That meant that the bar was set very low for MySpace to compete against Friendster.

    MySpace and Facebook were initially very sluggish sites. Twitter and the ‘fail whale’ of the site being down were a cultural touchstone of the late noughties.

    Fail Whale

    This increase in audience, meant a consequent increase in content. YouTube for example was running at over 45minutes of video being uploaded every minute. I am sure that rate is even greater now. How to sort through this firehose of content?

    To engineers the solution would look a lot like Henderson’s magic donkey. Algorithms slowed down the newsfeed to something more manageable, otherwise it would overwhelm the users.

    On the commercial side, the social platforms need to show sticky content that will keep users on their site longer and that they can vend advertising against.

    No great plot to up-end civilisation or spread hatred and bile. But algorithms can have unintended consequences. Content that polarises, engages. The algorithm doesn’t know that’s a bad thing. Soon those that want to engage with audiences in an emotive political way understand how the system can work with them.

    A mix of trial and error with a bit of understanding of behavioural science and continual learning allows political actors to learn how to use the system. Incremental tweaks in approach that their rivals or peers make drives that knowledge at a faster rate than the algorithms seem to evolve. The algorithm is blind to it all. It sees the things it cares about ‘improving’. Time spent on service, engagement with content, commenting and sharing. A human-machine feedback mechanism is created.

    In essence, it’s Google’s ‘stupid shop assistant’ all over again and this time human input in the feedback mechanisms is hurting rather than helping the magic donkey of social platforms.

    More on flickr here and more on internet culture here.

    UPDATE (September 24, 2020): Earlier this week the FT ran the story ‘YouTube reverts to human moderators in the fight against misinformation‘ – in an apparent indication that the ‘magic donkey’ model has reached its limits (at the moment). YouTube’s machine learning algorithms when given control had a false positive bias banning a ‘significant number’ of videos that broke no rules. In a three month period 11 million videos were taken down. (The usual number is about half that).

More posts