Chat-driven artificial intelligence may be one of the oldest forms of hype around—it’s literally been a point of discussion since the 60s (hi ELIZA)—and now it’s coming for your search engine.
That’s been the big hype train of the last few weeks, and it’s one that seems to be flattening every other tech discussion in its wake. It even got Elon Musk out of the news cycle for five minutes.
Videos by VICE
But, with all the benefits (and the hilarious and dark oscillations) of Bing’s chat-based search add-on, code-named Sydney, and Google’s long-in-gestation Bard product, search is looking like it’s about to go through a dramatic change.
Much has been said about the brokenness of these still-in-beta tools. And many are skeptical about the motives of this technology. Jon Henshaw, the director of search engine optimization (SEO) for Vimeo and the former CEO of a search-optimization software company, is one such skeptic. He points out that it may simply not be practical for most people.
“Chatting with a bot might be fun and magical, but it’s inefficient compared to having a search engine instantly provide a running list of results that one can quickly scan and click on,” Henshaw says.
But it’s worth taking a step back to consider why search is suddenly seen as a thing in play, even if this isn’t ultimately the catalyst. Conversational AI is not the conduit behind a change in search, but the result of an opportunity emerging. It took a quarter-century, but our search engines are finally starting to feel dated.
We simply needed someone to point that out. And on February 14, 2022, a blogger named Dmitri Brereton wrote what a lot of people were apparently thinking.
The Blog Post That Changed Everything
Since the days of Lycos, Excite, and AltaVista, we’ve essentially lived with the primary search model of putting queries into a box and hoping for the best. Google, upon its launch, further emphasized this minimalist approach—sure, the landing page was Google’s trademark logo and a search box, but the search results weren’t loaded with extra junk, either. But over time, it and its competitors became increasingly complex, in part because they had business models to support, and in part because the internet they were indexing had itself gotten a lot more complex.
It’s not a shocker to say that the Google search page has gotten bloated, but Brereton, a software developer and independent researcher, put it into words, and got an extremely viral blog post as a result. The post received more than 3,600 upvotes on Hacker News and, as of this writing, is the 11th-most-upvoted post in that website’s long history. Even if you disagree with him, he ultimately planted a huge seed within the startup world’s most prominent community.
(He also wrote another viral blog post just last week, almost a year to the day of the first one. He was the guy who pointed out that Bing’s chatbot produced a ton of errors during Microsoft’s demo.)
In his post last year, Brereton’s point was simple: All the external pressures on Google’s results, including the dominance of search engine optimization (SEO) and the company’s self-promotive tendencies, had made the search engine less effective at its ultimate goal of helping you find things. That was one factor. But he also took aim at the service’s weak artificial intelligence and failed attempts to guide the user.
“Google increasingly does not give you the results for what you typed in,” he wrote. “It tries to be ‘smart’ and figure out what you ‘really meant,’ in addition to personalizing things for you. If you really meant exactly what you typed, then all bets are off.”
I, admittedly, was skeptical of Brereton’s thesis at first, in part because I tend to practice what I call “Extreme Googling.” Weird site searches, incognito mode, VPNs, image searches? I’ve done it all. (These weird tricks were actually part of Brereton’s argument—particularly that site-specific searches are increasingly necessary to find good content.) I struggled to see that changing, because to me, Google has a depth few other search engines touch—even now.
But with a year of hindsight, I’ve come to believe that Dmitri was on the right track. And it came from someone, who, from his own admission, got into researching search engines as a pandemic hobby. At one point last year, he even started a search engine of his own, Blogsurf.
“Finding information is the most important thing that we do as humans,” he says. “The number one most important product in people’s lives is search. So I think it’s a pretty important thing to think about.”
Brereton tends to avoid general-interest search engines when Googling. Instead, he favors social networks like Reddit, TikTok, or even the Google-owned YouTube.
“I only use Google when I want a fact—I don’t use it to find websites as much,” he says.
In many ways, Brereton’s thinking on this topic is reminiscent of a famous argument about Craigslist, first made by venture capitalist Andrew Parker, that went viral during the Web 2.0 era—Craigslist never went away, but nearly every one of its offerings had been replaced by a better startup. From Airbnb to ZipRecruiter, Craigslist got superseded by them all.
And that might be the true risk facing Google, rather than some algorithm that talks to you. Centralized search is closer to the Yellow Pages than we might want to admit—and the Yellow Pages were full of ads that made that experience imperfect.
But individual apps have a responsibility to deepen their engagement that a general search engine might not. Reddit and YouTube are ultimately incentivized to show you relevant content from Reddit or YouTube. Google is supposed to be motivated to show you the open web, but it ultimately needs to support its business. Naturally, that difference changes to motivation.
And partly for this reason, search in a more narrow service like Reddit or YouTube is less likely to be gamed by SEO experts or dominated by ads.
Considering SEO’s Long Shadow
SEO is one of those things that people either find infuriating, confusing, or obtuse. Perhaps that’s why we’re starting to look for something else—but ultimately it’s important to understand why it exists in the first place.
So let’s try to explain it with a metaphor. It helps to think of Google or Bing search result pages as pressure cookers, with tension happening from all sides. At the top, there is pressure from advertising and knowledge panels—those boxes on the side of the search results that try to explain the thing you’re searching for before clicking a link. (I guess you could call it the steam in this metaphor.) Meanwhile, simmering from the bottom are pages that are vying to be at the top, where it’s most likely you’ll interact with them once you finally let the steam out. The more popular the page, the higher the pressure is. In that environment, it’s no wonder that stuff that doesn’t deserve to be at the top often finds its way there.
This is why every food website has 2,000 words of fluff that go with every recipe, why every startup under the sun has a blog, and why publishers get so many emails asking for a backlink or guest post.
Some of this comes down to how Google’s secret sauce, PageRank, was built. It’s intended as a system to reward relevance, and relevance is ultimately determined by how far an article or website spreads. But while that isn’t as easy to game as a primitive search engine like Lycos, which didn’t have any algorithms of this nature, it’s still possible to game. And because the results are just too large to check, the only way to get ahead on Google is to invest in SEO.
Add to that the tension created by Google actively competing with its own search results on the top half of the page, and you have a recipe for a complex symbiotic relationship, where tactics get passed along among SEO experts.
But it doesn’t have to be that complex. While admitting that search tactics are somewhat of a moving target because of ever-present rule changes, Henshaw says that one big problem he’s seen is that publishers and site owners fail to utilize tactics that are built for long-term growth, instead favoring short-term trends that are often outdated by the time they come into use.
“I’ve learned that the best way to approach SEO is to be curious and observant, do the research and always be testing, and try to collaborate and share knowledge with like-minded people,” he says. “Unfortunately, I don’t think most people in the SEO industry do that. Instead, many learn, trust, and execute ineffectual tactics, which results in clients and executives losing faith in SEO and concluding that it’s a waste of time and money.”
So what should we make of Microsoft’s attempt to so nakedly disrupt the space? Henshaw, who previously worked as the director of SEO for the streaming service Paramount+, says that Google invested in large teams to manage its knowledge panels and search results, with an emphasis on quality—and he saw that work up close in his efforts to get shows like Beavis and Butt-Head into those knowledge panels.
“To my knowledge, Bing hasn’t made the same investments as Google,” Henshaw says. “This is why I think Microsoft is going all-in with ChatGPT for Bing. It’s a Hail Mary pass to try and jump ahead, but I think it will fail because it’s a gimmick, and they haven’t invested in search the way Google has.”
Defending Against Disruption
Of course, if conversational AI does take off, it has a real chance of disrupting a model that, for better or for worse, has worked well for lots of people, from small publishers to massive multinational corporations.
That said, this disruptive risk has come up plenty of times before. And Google has put up strong defenses.
For example, one could point at Google’s extreme diversification as a result of some of these efforts. The neglected-but-still-important Google Books, for example, came as a result of attempting to improve search results. (It, unfortunately, faced lengthy lawsuits over its mere existence.) So was Google Maps. The social network Google+ was also deeply integrated into the search engine, which created security issues that infamously led to its closure. When search needed to become pocketable, it launched Android into the world.
These tools, when done well, created a reason for users to stick around when searching for something—and create reasons to not go to an alternate source offsite. Based on ChatGPT’s wild growth, conversational AI could be sticky content on steroids.
The ChatGPT That Wasn’t
But if you go further back, the discussion around a new technology disrupting search might start to feel a little familiar. This has literally happened before.
In 2009, the equation-solving answer engine WolframAlpha generated significant buzz for adding contextualized data into its search results, but ultimately had no effect on how the average user interacted with search engines, other than maybe encouraging the market leaders to up their game. Conversational AI could go pretty much the same way.
Alex Barredo, a podcaster and developer, recently put this point perfectly in a comment on Mastodon: “You could take all the articles written in 2009 about WolframAlpha, replace the name with ChatGPT and publish them again.”
WolframAlpha is nothing like ChatGPT in practice—for one thing, it’s not guessing in conversation but interpreting and visualizing data on the fly—but in many ways, the tension is the same. A lot of people were betting against Google that year. But nothing happened. WolframAlpha is still great, but it didn’t beat Google.
Maybe if someone combines WolframAlpha and ChatGPT, perhaps that could be the secret weapon. (That, after all, is what WolframAlpha designer Stephen Wolfram wants.) Or maybe not.
What Comes Next
So, setting aside the errors and the hype, there is a case that the community around search is getting restless for the next thing.
Even outside of conversational AI, there seems to be real activity in the search space outside of Google for the first time in a long time. Sites like You.com, Neeva, and Kagi have emerged as attempts to take the search engine in a new direction—whether with more AI, more privacy, or better usability. Even DuckDuckGo, a Bing-derived meta-search engine that has existed in underdog mode for the past 15 years, is prominent enough these days that it can advertise itself on television.
But just because we’re restless for the next big thing doesn’t mean it’s simply going to appear. After all, there’s a reason why the hype bubble of “Web3” eventually popped.
It’s understandable, given all that, to have a healthy dose of skepticism around any forthcoming technology trend. Henshaw, who often writes about open-web topics on his site Coywolf News, struggles to see a future where traditional search gets disrupted—and further, ultimately sees conversational AI as a dangerous trend that further centralizes control.
“With conversational AI, I think society has the most to lose. Having it take over search means people will be spoon-fed information that is limited, homogenized, and sometimes incorrect,” he says. “It will affect our capacity to learn and will suffocate the open web as we know it. I know it sounds like hyperbole, but I don’t see how search taken over by conversational AI doesn’t eventually lead to some form of dystopia.”
Henshaw’s comment points to a tension that keeps coming up in this discussion. Is search for the user, for the websites that show up in the results, or for the company that runs the search engine? One could make the case that conversational AI, if it takes off, threatens to leave content-rich websites out of the equation.
On the other hand, if our search engines don’t get a fresh dose of ChatGPT, perhaps it’s because we need to set our sights bigger. At one point during our conversation, Brereton suggested that the real opportunity for conversational AI might be as a teacher of sorts, a technology that can teach you about the latest trends in quantum physics, to solving equations, to putting together the best possible vacation.
“I guess this might be beyond the traditional definitions of search, but truly the search engine is just a really smart person who knows everything,” he says.
He essentially wants a technology that has all the answers, no matter the source—just way better at the job than Google. After mentioning that this sounded a lot more exciting than trying to optimize a business’ presence in a search engine, his response was basically perfect. “I don’t think SEO is gonna exist in the future,” he says.
I think that’s a big reason why Brereton’s point of view on this discussion feels so refreshing in 2023—and perhaps why it broke out in such a big way a year ago. There are real reasons to want to keep search grounded in reality—for one thing, it’s tied to a whole lot of business use cases that make lots of money for everyone involved.
But the truth is, if search is forever held to the standards of 1998, it will eventually reach a ceiling and lose some of its magic. So the right thing to do, long term, is disrupt it, so it better meets the needs of the present day.
When that disruption happens, businesses will adapt, just like they did when search engines first came onto the scene. Businesses, after all, loved the Yellow Pages … until they didn’t.
Now, is conversational AI, specifically, going to get us to that future? All bets are off. Perhaps it needed another year in the hopper.
