UCI Web Crawler & Search Engine

I worked with a group to scrape the UCI Information and Computer Science Repository. The main code is in scraper.py. We started with a seed URL and extracted its HTML for important text in specfic headings. This was also the way we extracted the next url. Additionally, we did some text processing such as stemming to make our index stronger. This webcrawler is found here https://github.com/richlo01/webCrawler.

Then, we created a search engine, ranks the results, and returns them in under 200ms. We improved query results by implementing cosine similarity between tokens and documents using term frequency - inverse document frequency. The scraper created an index that was sorted alphebetically. Here we would do word preprocessing and document duplicate checks using SimHash. This way, the results are accurate and efficient since they were done before hand.

This was done purely on Python.