Web Crawler

Web Crawler is an operational project focusing on crawling the web based on the given url to extract more urls within the given url. It has visual elements to display the extracted results. A tree element that is drawn with Graphviz and a list of links with MUI list component, which shows the depths of crawling to demonstrate links that have been found and their connection.

JavaScript/React/MUI and Python are used for the front-end and the back-end code development. For handling communication between front-end and back-end I used Flask and Axios. You can view the source codes in repository here!

To crawl within a given link, I have used Beautiful Soup Python Package to loop thought the HTML tags within any websites and extract <a> tags to get the link to other websites. 

Main page, where you enter a URL and add depth of search and how many times each link be searched.

You can view the result of the search as a linked list.

You can view the result as a linked graph that is by Graphiz.