We take the data let’s say Bhagavad Gita in this case and embed it using Embedding-ada-002. This will convert the plaintext into vectors.
We store these vectors in Pinecone which is a vector DB. We can’t store it in a normal DB. This is sorta like Algolia or elastic search.
For any new user query, we convert that query into embedding using the same model Embedding-ada-002 and then do a KNN (K nearest neighbors) search. So, in vector space, we are finding the shlokas that are the closest to the user’s search query and returning those top K shlokas.
This is a good video that explains this
https://www.youtube.com/watch?v=ocxq84ocYi0&themeRefresh=1
Right now we’re using just the full-text search in Postgress, and we can improve it using embeddings and this would help us build the AI councilor also 🎉