Hello all -
Running a Web report recently, I realized that the third-highest URL in my site was the Faculty & Staff Directory search results page. This struck me as rather bizarre, so I dug a little deeper and learned that, indeed, hits had tracked up rather significantly in the last month.
I dug deeper, and it seems that we are getting hit with quite a few IPs whose reverse DNS points to pipl.com, which appears to be something of a search aggregator.
I did some targeted searches to see if I could pull up something clearly from our directory, but was unsuccessful.
Of course, there are some common-sense precautions in place to prevent wholesale harvesting of the directory, but this seems like a questionable practice, and I've attempted to block access from the offending IPs.
Has anyone else encountered this? Thoughts on blocking as a reasonable response?
Oddly, about the same time (although a week after this started) I converted the directory search results to a compliant Microformat. This raises an interesting question as to whether such specific coding can actually hurt more than help...