17 minutes estimated reading time
Back in 2005, I worked in the search group at Yahoo!. One of the projects that I worked on was Yahoo Answers. 16 years later, Yahoo Answers is being closed down. I thought I would capture some of my memories and inside knowledge on Yahoo Answers.
But first we need some context so that what I write later about Yahoo Answers will make sense.
The beginning
Let’s go back to the beginning. Back to the early-1990s. Jerry Yang and David Filo founded Yahoo!. It fits the classic Silicon Valley archetype story and you can find plenty of accounts of it elsewhere. The key is what Yahoo! originally was. Its a list of links for websites. Once the list grew above 200 links or so; Jerry and David came up with a way of displaying this list by grouping it into subject areas.
What would later be called a web directory. There were other directories around about this time like:
- Best of the Web – which surprisingly still exists
- Netscape Communications had their own directory when they acquired Gnuhoo, this eventually became DMOZ and then Curlie. Gnuhoo did rely on a search engine to help you find things in their directory. This is available as open source code at GitHub
All of them had a certain amount of editorial input over what was good. Yet Yahoo! became the top one through buzz marketing – cheap ways to do brand building.
When I was there, I worked with an agency to organise event hijacking at the Harry Potter and the Half Blood Prince book launch at Waterstones flagship store on Oxford Street. Yahoo! would vinyl wrap any employee’s car for free. There were also strategically placed billboards, such as this one in San Francisco.
People who managed this directory were known as web surfers. But there was also search engines out there, like the Knowbot search engine for Telnet developed in the late 1980s. There was Archie which was the closest to what we think of as a search engine now. Archie searched FTP archives around the world.
As computer science post-grad students, Filo and Yang would have been familiar with the idea of the search engine. At the time David Filo felt that no machine would provide better filtering than a human. Media accounts of the time showed that Silicon Valley venture capitalists were all in favour of search engines over directories.
Peer companies like:
- Webcrawler
- Metacrawler
- Lycos
- Ask Jeeves
- Infoseek
- Excite
- AltaVista
All offered what we’d recognise as reasonable search experiences. But Filo’s comments on human filtering is something that we will revisit later.
Web portal & web advertising
Search engines were the future but as the dot com era took off it wasn’t apparent how to monetise them.
At the height of the dot com era; Yahoo! had about 40 million users a month. You have to remember there weren’t that many people online in comparison to now. Internet usage had grown from 45 million users in 1995 to over 410 million by 2000. At the time it didn’t seem to matter that Yahoo! took longer to load as a website compared to its peers. Longer page times, meant that you could get away with less equipment in your data centre hosting the website and supporting infrastructure.
The internet didn’t give birth to culture in the same way that memes, influencers and platforms do now. Instead it was the meme. It was all over the mainstream media, often tied up with ideas of cyberpunk culture, bulletin boards and the ‘information superhighway’. Examples of this included:
- The Site by MSNBC
- The i in iMac was for internet. The idea was that you could take the computer out of its box, plug it in to your wall socket and telephone socket. When you turned it on, it would configure you an internet service. The cool product design was a byproduct of this internet appliance plus personal computer thinking
- Movies: The Lawnmower Man, Hackers, The Matrix, Ghost In The Shell
- Books: Snow Crash, William Gibson’s Neuromancer
- A plethora of internet magazines, including Ziff-Davis’ Yahoo! Internet Life which was a mix of technology and gadget reviews, media and celebrity content and website recommendations. Yahoo! Internet Life was published from 1996 to 2002
It felt like something big was going to happen, even if we didn’t know what it was. What was obvious was the potential for advertising online. And the clearest analogue was newspaper advertising due to the long page format of web pages.
Web portals came about for a number of reasons:
There was now the technology to pull content from different sources together. You would have:
- Weather forecast
- Horoscope
- Up to date news
- Local information (for major cities like San Francisco)
- Business
- Finance
- Entertainment and celebrity news
Like the newspaper before it, it offered the first media you needed, but on the web.
It was mainstream enough for brands to advertise against for brand building.
By the time I was leaving college, Yahoo! Mail accessed through the Yahoo! home page made perfect sense.
So before the dot com bubble bursts Yahoo! had a major media business valued at 2.8 billion dollars, or about $70 dollars per user. Which sounds expensive, but when you consider that Google is now worth about $386 per user, it’s not that bad. Secondly, online advertising per impression was much more lucrative back then and ad fraud was much less of an issue.
What there wasn’t was a way of taking advantage of the highly relevant search results provided by search engines and adequately monetising them. So companies had three ways of monetising search:
- Companies created portals the so called ‘homepages of the web’ to put display adverts on like My Yahoo! or MSN.com and search was a service alongside news, weather and horoscopes
- They became infrastructure companies selling search functionality in the background a la Inktomi
- They sold inclusion in their directory. This was controversial as it went against the editorial integrity of the directory and still a hot button when I arrived at Yahoo! in 2005
The bubble bursts
In the US stock market we had was now known as the internet or dot.com bubble. Looking at the NASDAQ composite data, it seemed to start in the last quarter of 1995, six months or so before Yahoo! went public in April 1996. It reached its nadir in the last quarter of 2002.
In reality, this was more than about websites. Telecoms deregulation, satellite networks and the rise of cellphones had seen a boom in new companies and network equipment providers to support them. The need for servers had created booms in:
- Computers: SGI, Sun Microsystems and IBM
- Networking equipment: US Robotics, 3Com, Cisco
- Software: VA Linux, RedHat, Open Text
- Software as a service: I2, Salesforce, NetSuite
- Web hosting and ‘data hotels’: Equinix, Intel, Rackspace, PSINet
- Telecoms and ISPs: Level3, Global Crossing, Earthlink, Iridium, GlobalStar, AOL, @Home Network
Add into that artificially high growth in earnings for enterprise IT companies in the run up to the Y2K bug issue and the whole sector was left with a bad hangover.
Steiner’s talk is interesting because it shows how the search business, selling search capability to the likes of Microsoft, Amazon and eBay had slow and steady growth rather than outstanding growth during this time.
Yahoo! went through a traumatic time. When I worked at Yahoo! Europe, I was told online advertising sales dropped to a third of what they were during the dot com boom. The European business managed to hold on by its finger tips thanks to revenues from online dating services.
Some of the ‘smart bets’ Yahoo! made during the boom times looked like hubris. The exemplar of this was Yahoo!’s acquisition of Broadcast.com. Broadcast.com provided video streaming (then called web casting) and internet radio services. It was the technology partner for the first online Victoria Secret Fashion Show streamed online. Yahoo! acquired it for 5.6 billion of Yahoo! stock. This was a bad decision, but thankfully, they didn’t pay cash.
When I joined Yahoo! the Broadcast.com acquisition was still a scar on acquisition decision-making. You can attribute the impact of this to subsequent failed purchases of Google and Facebook.
GoTo and Google
In 1998, the company GoTo.com launched paid advertising placement in search engine results. The next year they introduced real time bidding. It was renamed Overture and started providing these services for Yahoo! and others. It started to become successful as a business.
Meanwhile, Google had moved from a research project to a serious search engine. In 2000, Google began selling advertisements associated with search keywords. This was against Page and Brin’s initial opposition toward an advertising-funded search engine, they saw themselves more as a ‘search appliance’ business rather like Inktomi. Yahoo! adopted Google search around about the same time that Google started its search advertising business.
This put Google in front of a large number of consumers and helped Google further refine its search engine.
Google’s own offering was the exact opposite of Yahoo!. It prided itself its clean design with just a search box. Google also had a fanatical obsession with reducing page load times and the time taken to return search results.
This was what more and more people wanted. Google used the dot com crash to build its business and its infrastructure. It wasn’t until its 2004 IPO that rivals realised how much of a head start Google had.
Google revolutionised data centre server design, reducing cost and increasing the amount of servers that it could use. By contrast every Yahoo! data centre hardware purchase went via David Filo. If you used Yahoo! small business hosting, you were using tired and almost expired Yahoo! servers. In retrospect, they looked after the datacentre pennies, but let the pounds slip away.
2003 saw Yahoo! get serious about the search engine business. The company purchased Overture which included GoTo.com and Altavista. But the problem was that even if Yahoo! built a search engine as good as Google, it didn’t matter if people didn’t use it. During my time at Yahoo! there was a push to get the necessary servers in place and a product that was as good as Google. However there was a constant tit-for-tat feature development in the search space. By this time Google had already verbed. The Google habit means that its hard to compete against them.
I heard that inside Microsoft they tried to take drastic measures to persuade employees to use Bing over Google. When I worked at Yahoo! people used Google a lot too.
The only way to compete with Google was to have a different idea. Google defined its mission is to organize the world’s information and make it universally accessible and useful.
Yahoo! needed a new idea that was distinct from Google’s mission. The idea was knowledge search.
Knowledge search and Yahoo! Answers
Knowledge search as a concept was well under way by the time that I arrived. It was to capture and make searchable all the ‘knowledge’ (rather than information in the world). Opinions, experience and recommendations are knowledge rather than information. Yahoo!’s web 2.0 acquisitions including Flickr and delicious were made to support this vision.
Tagging built up words and associations with web links and images, effectively human filtering – some of which would be used to train machine learning algorithms. The next logical step would be to build a repository of knowledge by the people, for the people. That’s where Yahoo! Answers came in.
The inspiration for Yahoo! Answers came from a product that Yahoo! Taiwan had rolled out. It in turn probably inspired by Korean site Naver Knowledge IN. Bradley Horowitz apparently claimed that Yahoo! Answers was inspired by Naver Knowledge IN directly.
Knowledge IN was designed to encourage user created content, since there wasn’t much material on the Korean web at the time.
When I heard Jerry Yang talk about it internally at the time, he talked bruskly about a product built by Yahoo! Taiwan as having inspired it. Jerry didn’t do jet lag well and came across as morose on the couple of times I saw him in Europe, so wasn’t exactly an effusive speaker.
Yahoo! Answers was championed by Jerry and that blessing allowed it to be pushed through when so many other product died before they got pushed to beta. It makes sense to point out the human crafted nature of Yahoo! Answers. In this respect it can be seen as a direct line back to the original Yahoo! directory product. Both were fuelled by a belief that people had some ability that was better than machines.
Qi Lu was responsible for new products within the core search business and the troubled Panama search advertising project at the time. Weekly conference calls saw a plethora of existing projects cancelled, or reprioritised by Qi Lu, while new ones would suddenly appear. This constant change in the roadmap mean’t a lot of wasted efforts.
Yahoo! Answers and much of the knowledge search related acquisitions sat under Bradley Horowitz. Tim Mayer was focused on the commercial side of things, although there was some overlap in the roles. Eckhart Walter sat above Tim. Jeff Weiner was the main shot caller having both Search and Marketplace businesses reporting into him. If you’re thinking, that’s a lot of senior management involved. You’d be right, there were a lot of managers with varying degrees of responsibility involved.
But they were all good people and I’d be happy to work with them again.
Prior to Yahoo!,I had been working agency side for Transversal. Transversal powered the support functions for a number of companies including Sony Playstation. I had a good idea how much this service was priced and floated the idea of sponsored channels for instance around Sony Playstation and had a good idea how much Sony must be paying to support user troubleshooting.
But it didn’t fit that well as an idea with knowledge search.
Concerns and how is babby formed?
In the European team we had some concerns about Yahoo Answers like how was it going to get monetised? The quality of the content was also a concern. Knowledge IN and similar services in Asia work partly due to culture. We were worried when it hit a more individual-focused culture like the US or Europe.
Another problem was calibrating the rewards within the system. Its really hard to get the balance on good quality questions and answers. Generally people who are time rich, aren’t necessarily the best respondents. If you need one proof point to show how much of a failure this was, you only have to look at the how is babby formed? meme.
Rewards aren’t the only problem however. The second issue was the way the community was built. Generally, a great community is built carefully from like-minded people. With flickr it was around the passion of photography. Facebook is actually closer to Reddit, built on groups of groups. The death of a group dynamic won’t necessarily kill the platform.
I was involved in early seeding of the initial content on Yahoo! Answers. I answered 42 questions, the first one question I answered was ‘What to take from airport to downtown Munich?‘ My response: The taxi is reasonable, it cost me 30 Euros – which shows the contextual nature of knowledge search. 30 Euros was reasonable for me at the time, since I could expense it back, but it wouldn’t be reasonable for a backpacking traveller.
I also wrote six questions, the first one was ‘Has anybody got a Pentax K100D, if so what do you think of it? What are its pitfalls and what aspects of it do you particularly like? I wanted to get a a bit more colour beyond the reviews I’ve read online. – I was getting ready to leave Yahoo! and was going to buy a DSLR camera to take better pictures on my Flickr account. I deliberately structured the question to get opinions from early users. The Pentax K100 had recently been launched.
Careful community management is at odds with a platform trying to capture the world’s knowledge. So the Yahoo Answers community was built for rapid global user growth. For the English language versions at least, there was a global content index, sitting on top of a distributed Oracle database.
This meant a clash of cultures and variable quality content. I quickly found the site unusable for productive questions. Yahoo! spent the next few years trying to perfect it. People that formerly worked on Yahoo! Directory and front page brought their content and editorial skills to bear on Yahoo! Answers.
I suspect that trying to monetise the service would have been a constant challenge. Yahoo! Answers provided variable quality answers for children’s homework and was the butt of memes. Neither of which are an ideal recipe for the kind of content large brands like Procter & Gamble would want to put their name against.
Quora’s lean pickings
Google tried to do it better with Google Knol and also failed.
Quora was formed in 2009 and managed to build a better community, but I’ve still seen a steady decline in the quality of their answers. In 2019, they had a user base of 300 million people and total revenue (from advertising) of 20 million dollars. Thats an ARPU of 6.6 cents. That’s not a good internet media business. From that 20 million, they need to pay their infrastructure costs, maintain and improve the product, pay the salaries of their 300 employees. And I haven’t even talked about how their investors must feel.
Knowledge search is still a technology challenge waiting to be conquered.
More information
Yahoo: a history of the internet in 5 acts – Financial Times (July 25, 2016)
Max Roser, Hannah Ritchie and Esteban Ortiz-Ospina (2015) – “Internet”. Published online at OurWorldInData.org.
Britannica Online – authoritative top level history of Yahoo!
Room for the Internet; Combining a Data Center With a ‘Telco Hotel’” – New York Times (May 14, 2000)