Today, a member of the Doll Forum ignored our request to not use automatic site copying tools and effectively brought our server down to its knees.
In under one hour they had made over 12,000 requests to the server for image files. It took me another hour or so to extract their entries from our logs to gain a full appreciation of what they had done which took our process load from its normal 2.5 up to 27.9.9 - No use of automated download programs (bots, web harvesters, etc.). Such programs effect the performance of our server by making too many concurrent requests.
While they were "mining" images files at such a rapid rate, no one else was able to access our server with any degree of responsiveness. The server was very slow and sluggish for everyone else.
Anyone interested in seeing what I am talking about can look at the extraction file which I produced:
warning: this file is 3.5 megs in size:
http://www.dollforum.com/68.211.49.239.txt
I know who this was that did this and I have permanently banned them.
I am posting this because I have been finding that more and more people are trying to use image harvesters (bot, spiders, etc.) in an attempt to grab all image files.
At times, this has gotten so bad that we have now started discussing a very expensive solution to solving the problem. Funds which could have gone into improving our service will eventually be going into installing equipment in which to protect it from those greedy few within our membership.
I am making this announcement because starting with this member today, I will permanently ban anyone found to be using automated data mining tools to rapidly download images and other files.
I am sorry to be hard nosed about this, but I have tried every software solution to prevent this kind of selfish behavior and these people work to find a way around them. And it has brought us to a point where we will very likely be spending 3 to 4 thousand dollars on hardward just so some of our members do not abuse this free service.
Ish