.Google.com's Gary Illyes discloses Googlebot picks up web links instead of "following" them, negating official paperwork.Googlebot picks up web links instead of observing all of them in real-time.Gary Illyes discussed this details in Google's 'Search Off The Report' podcast.Google's records opposes the real crawler habits.