Social networking Influence: Well-known trends on social media can spill over into search engines, affecting the look for quantity of associated keywords and phrases. txt file is then parsed and may instruct the robot as to which web pages usually are not to be crawled. Being a internet search engine https://www.youtube.com/watch?v=CWAXJPBjGjE