Of Chatbots and Search Engines

they're not all that different after all - author

let's take a 10,000 feet view of both the eponymous search engine as well as the chatbot.

What do we need a chatbot to do? It has to hold a conversation with us and provide us with helpful results, as well as prompts if it is unsure what you're asking. It is a problem of finding out what the user is asking

What do we need a search engine to do? It has to provide us with information, preferably within 1-2 tries, or at least provide good close answers. It is a problem of finding out what the user is asking, it is about narrowing down what the question is, and giving the best answer to it.

prng search engine architecture

by order of priority

1. fuzzy text search on title, subtitle, tags, and excerpt text
   • < 30ms, immediate feedback
2. full text search
   • < 1s, almost instantaneous feeedback, debounced
   • fuzzy search on full text
3. text corpora similarity search
   • &gt; 1s, shou
   • should be able to pick up the most nuanced queries
   • QUERY: MultinomialNB on query (ref: chatbot workshop) (basic version)
     • classify what the intent is
   • RESULT: some sort of clustering model that captures topics
     • e.g. summarizer models
     • captures the answer's intent
     • creates intent classes (that are probably document embeddings) for the query model to classify against
   • requires a new model to be built everytime a new piece of data (blogpost) is added to the database
   • build should happen automatically