How Yelp Checked competing LLMs for correctness, relevance and sound in order to develop the user-friendly AI assistant

The review app Yelp Has provided helpful information for guests and other consumers for decades. It had experimented with machine learning since its early years. During the recent explosion in AI technology, stumbling blocks still occurred, since it was committed to modern large language models to operate some characteristics.

Yelp realized that customers, especially those who only occasionally use the app, had problems Connect to his AI functionsLike his AI-operated assistant.

“One of the obvious lessons we have seen is that it is very easy to build something that looks cool, but very difficult to build something that looks cool and is very useful,” said Craig Saldanha, Chief Product Officer at Yelp, in an interview with Venturebeat.

It was certainly not all easy. After the Yelp assistant, the AS assistant of the AI-operated service, launched a broader customer amount in April 2024, Yelp saw usage figures for his AI tools that actually started to decrease.

“The one who surprised us was when we started as a beta for consumers – a few users and people who are very familiar with the app – (and they) loved it. We got such a strong signal that this would be successful, and then we condemned it to everyone, (and) the performance just fell away, ”said Saldanha. “It took us a long time to find out why.”

It turned out that Yelps, who occasionally visited the website or app to find a new tailor or a new plumber, did not expect to speak immediately to a AI representative.

From simple to vegetable AI functions

Most people know Yelp as a website and app to search for restaurant reviews and menu photos. I use Yelp to find pictures of Essen in new restaurants and see if others share my feelings about a particularly boring dish. It is also a place that tells me whether a café that I want to use as a work area for the day, WLAN, plugs and seats, a rarity in Manhattan.

Saldanha remembered that Yelp had “invested in AI for most of a decade.

“Already when I would say in the timeline 2013-2014, we were in a completely different generation of AI, so our focus was on creating our own models to do things like understanding queries. Part of the task of establishing a meaningful connection is to help people refine their own search intent, ”he said.

But when the AI developed, the needs of Yelp also developed. It invested in AI to recognize food in pictures submitted by users to identify popular dishes Help the users’ search processes on the platform.

Yelp Assistant helps Yelp users to find the right “Pro” with which you can work. People can tap the chatbox and either use the input requests or enter the task they need. The assistant then poses follow-up questions to narrow down potential service providers before starting a message to professionals who may want to offer for the job.

Saldanha said that professionals are encouraged to react to users themselves, although he recognizes that larger brands often have call centers that process the messages created by Yelps KI assistants.

In addition to Yelp Assistant, Yelp Review started insights and highlights. LLMS analyze the seismion of our and a reviewer that Yelp collects in mood reviews. Yelp uses a detailed GPT-4O entry prompt How to generate a data record for a list of topics. Then it is coordinated with a GPT 4O mini model.

The review emphasizes that information from the ratings is presented and also uses an LLM request to generate a data record. However, it is based on GPT-4 with the fine-tuning of GPT-3.5 turbo. Yelp said it would update the function with GPT-4O and O1.

Yelp joined many other companies Use LLMS to improve usefulness From reviews by adding better search functions based on customer comments. For example, Amazon started RufusA AI-affiliated assistant who helps people find recommended objects.

Large models and performance needs

For many of his new AI functions, including the AI assistant, Yelp turned to Openas GPT-4O and other models, but Saldanha found that Yelps data is the secret sauce for its assistants regardless of the model. Yelp did not want to join a model and was open which LLMS would offer its customers the best service.

“We use Models from Openai, Anthropic and other models on the AWS basic rock,” said Saldanha.

Saldanha said that Yelp had created a section to test the performance of models in correctness, relevance, awareness, customer security and conformity. He said that “it is really the top -end models”, which are best cut. The company runs a small pilot with every model before taking iteration costs and response latency into account.

Teach users

Yelp has also made concerted efforts to educate both casual and power users in order to familiarize themselves with the new AI functions. Saldanha said one of the first things they recognized, especially with the AI assistant, was that the tone had to feel human. It could not react too quickly or too slowly; It could not be excessively encouraging or brucking.

“We put a few trouble to help people feel comfortable, especially with this first answer. It took us almost four months to do this second piece. And as soon as we did it, it was very obvious and you could see that hockey sticks into commitment, ”said Saldanha.

Part of this process included the training of the Yelp assistant to use certain words and sound positively. After all this fine-tuning, Saldanha said that they finally see higher number of uses for Yelps AI functions.