Following, It is best to make a decision how you want the crawler to manage various World wide web technologies. There is an ongoing debate with regards to the intelligence of online search engine crawlers. It isn't really completely obvious When they are full-blown headless browsers or just glorified https://www.youtube.com/watch?v=EVeTP0aV8gY