Development Contribution for the Search in Utopian

So I along with @sirrius was working on improving the search of the current Utopian.io main site. @sirrius has already shared most of the details on his post Added text search ordered by relevance score for Utopian.

So here I will at first show what exactly I have done to create a local environment of Utopian.io. To run Utopian.io in your local you actually have to create 3 servers as shown below

1. Utopian Front End Server
2. Utopian Back End Server for Fetching the Data
3. Mongo Db Server

Setting Utopian Front End Server

This one I have already shared in my previous post, just copying it from there.

The very first thing I have to do it to fork the original project and run it on my local server. Next, I pulled the master branch from my forked repository using command git pull repositoryname master where repositoryname is my forked repository full path.

After pulling, I need to install all the node_modules which are used in the Utopian project for that I have ran npm install in the same repository where my code resides.

It will take some time if you are doing it for the first time. Once done, its time to run the code. For that you need to run npm run dev-server. If it runs successfully you will get a message as shown below.

When you navigate to the http://localhost:3000/ you can see the utopian running on your local server.

But after doing this you will see that you are actually not getting any data. So for that at first we need to set up the Back End Server for Fetching the Data and MongoDB Server.

Setting Mongo Db Server

We need to set up MongoDb Server before setting up the Back End server. For this, you need to install MongoDB in your local. Once done you need to run the command mongod --dbpath c:\mongodb\db, where my db resides in C:\mongodb\db.

Once done you can see that MongoDB has been started in port 27017.

Setting Utopian Back End Server

Similar to the Front End server we need to fork the original project (https://github.com/utopian-io/api.utopian.io). After pulling, you need to install all the node_modules using npm install in the same repository where my code resides.

Before running it on the local server you need to comment some code.

Comment out below code from config.js

UTOPIAN_GITHUB_SECRET: Joi.string().required(),
UTOPIAN_GITHUB_CLIENT_ID: Joi.string().required(),
UTOPIAN_GITHUB_REDIRECT_URL: Joi.string().required(),
UTOPIAN_STEEMCONNECT_SECRET: Joi.string().required(),

Also, change some code in the utopian-bot.js.

 const botAccount = process.env.BOT;
  const refreshToken = process.env.REFRESH_TOKEN;
  const secret = process.env.CLIENT_SECRET;
  const forced = process.env.FORCED === 'true' || false;

to

const botAccount = "utopian-io";
const refreshToken = "";
const secret = "";
const forced = true;

Once done you can run npm run dev-server. If it runs successfully you will get a message as shown below.

Thus now you can see the Utopian running on your server and also with all the data.

Now coming back to the search, as of current environment you will see that we are doing an exact match of the words and not full text search. Full-text search mens that we are searching the whole database against the search criteria specified by the user.

For example, if you say "forked project in github repository", we will search whole database of Utopian to get the results where these matching terms are there. Lets say if we get whole matching terms than that result will be shown as a top one whereas we will also get the result where some or the other keywords from the search is not present but their relevancy will be less so that will be shown below.

It is almost similar to what we search in Google where Google displays relevant result first and then all the result based on the rankings.

To do this type of search at first we need to create Indexes on our existing database as shown below. Where we are creating the text search on the title and body of the post.

db.posts.createIndex({"title":"text","body":"text"})

Now next part is to search the whole database with the exact phrase. So for that we need to find that phrase in our database and then sort it on the score (i.e. relevancy score).

db.posts.find({$text: {$search: "forked project in github repository"}}, {score: {$meta: "text Score"}}).sort({score:{$meta:"text Score"}})

Now this was the basics of the text search and how we have moved forward. If you want to know what exactly we have done the change then you can see the pull request from @sirrius here https://github.com/utopian-io/api.utopian.io/pull/18. We thought of creating only one pull request and we both were working on finding the best possible way to show result.