What is a crawler

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

Was just looking at some of the spider and crawler activity on our site, and found quite an interesting description of what they are, how they work, etc, if anyone is interested.

What is a crawler ? LINK

 

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

Interesting Gav, thanks.

 

And here I thought a "crawler" was a wee one who hadn't learned to walk yet. Synonyms include, "curtain climber" and "ankle grabber".