Elasticsearch : I did it, I used an existing framework, not recreate the wheel :)

Burf.co and Elasticsearch

So, as you can guess by the title, I decided to check out Elasticsearch (and do a couple of courses on it) to see if I could use it with Burf.co Search Engine.  Elasticsearch is built to be a search engine, whereas previously I was using MongoDB’s full test indexing and well, it just wasn’t up to the job of quickly returning results in a few seconds.

I still use the CommonCrawl data, which is filtered and put into MongoDB (the actual HTML document is stored), I then parse this and chuck it into Elasticsearch.  The program is all written in Java and seems to work pretty well.

The only issues I had were around updating my Java Spring Boot API’s to talk to Elasticsearch.  The Spring Boot Elasticsearch component uses an older version of Elasticsearch (current is v6.4.2, Spring Boot I believe is v5.x).  The other issue I had which seems silly is that most examples I found use Maven as the build pipeline, whereas I use Gradle.

The result is that search on Burf.co now works and is a lot quicker than before (disclaimer, I am still adding data to Elasticsearch, last looked it was around 55m pages).  I still need to learn about how to optimise it, how to do better more accurate searches, however, I am happy that I moved to use the correct tool for the job 🙂

Hack24

Last Friday, a few of my old Compsoft work colleagues met up for a GameJam which was fun.  I decided to focus on finishing Hack24 (finally), or to at least evaluate how much work was required.  Sadly the GameJam only lasted 7 hours (2 am) however I did get quite a lot of bugs and features implemented into Hack.  An Android MVP version is coming really soon!

I read a book!!!

So, it appears that you don’t just need to watch tv and play games while you’re on the exercise bike, you can READ, and I did so.  I read a book on flipping businesses (random I know) and quite enjoyed it.  So I have the $100 startup to read next 🙂

 

 

 

 

Leave a Reply