robots.txt (spiders and crawlers)

[JamesB]

anyone here using a robots.txt file to prevent crawlers and spiders and suce from snooping around your sites?

if so do use this command

# go away
User-agent: *
Disallow: /

I currently just have the

# No robot will spider the domain
User-agent: *
Disallow: /

I want to keep as many crawlers away from my site as possible.

James

[Clint_Smith]

hmmm...been getting a lot of "investment" mail lately...maybe should ad this to the contact page meta data...

[Nocturnus]

Only problem with that is, they already have your email, so it might stop new crawlers, but your still gonna get spam. I got ~200-300 on my domain a day now.

[JamesB]

I have never listed my email address anywhere on the web, well I use to, but I removed them years ago due to spam and deleted those email address. I never display my email address anywhere. up until now, I just put them on my new site, but I want to keep all the crawlers away so my site does not generate traffic with out me giving out the link.

James

[haraki74]

Me too on the "investments" emails. For those of us who are web challenged, can you give us a short how to lesson?

[Gordon]

I just use disposable addresses for any web accounts. Makes life easier.

[Detonate]

I use javascript to display my email link.

Still works for customers, fools the harvestors....

<script language="JavaScript" type="text/javascript">
<!-- Hide me from older non-JavaScript-enabled browsers 

var name = "jim"; 
var atsign = "@"; 
var virtual_domain = "jimmarch"; 
var dotcom = ".com"; 

document.write("<a href=mailto:", name + atsign + 
virtual_domain + dotcom, ">email me</a>"); 

// --> 
        </script>

[brad]

it isn't going to stop a spam crawler... robots.txt is for well behaved, friendly crawlers. Spam bots are not.

[markfh]

It won't stop dictionary attacks either. That's the job of your email provider. One other thing to do is make sure you don't have a catch-all account like "nobody" on your domain.

Unfortunately once you use your email address anywhere it can become available to spammers. Mostly by friends or business associates having your address in their address book on a computer that has been, is, or will be infected by a virus or trojan.

Spam campaigns have almost trippled within the last month and has caused quite a number of problems for email administrators.

The problem at Yahoo caused many of their email servers to be unresponsive to the point that some accounts, like mine, were not valid within their own networks for a period of time.

I had to reactivate my Yahoo account for groups that I subscribe to yesterday.

Another problem with email servers are the number of SMTP vulnerabilities that have been publicized lately that in some instances cause the service to be shut down on email servers.

Another area for you web designers is to make sure that any contact forms you write for your web site or clients sites are secured from having spammers take them over and spamming the world through your contact forms.

This appears to be one of the more recent vulnerabilities that many web hosting companies and developers haven't had to deal with in the past but is causing problems now.

We've had to strip out the contact forms on over 50 different web sites on our servers to stop this from happening.

[simon]

robots.txt is a request, not enforceable. If you include a blanket disallow, all the good guys won't crawl you and all the bad guys still will, so it's useless to avoid spam harvesting etc.

By the way, the two entries you show are the same. Did you mean that?

[AlexMorse]

The javascript solution is excellent. Or don't have your email address on your site, have a contact form.

I encourage robots and spiders for cataloging my site in some of the lesser known search engines.

[JamesB]

thanks all, there is some good help here.
I do NOT want my site indexed by search engines. I know some people want that, but not me. I will be adding some scripts and such to help reduce the amount of crawlers to my site.

yesterday I had over 800 hits on my new gallery and 57 users on line at one time and only me and my wife have the link for that. Im waiting on the web stats to update so I can see whats hitting it

I did use a disposable email forwarder address on my site, so if I start getting spam I can change it with no worries and delete the forwarder.

James

[simon]

Originally Posted by JamesB

thanks all, there is some good help here.
I do NOT want my site indexed by search engines.

You can't really stop it entirely, but you can minimize the impact by the way you structure the site. Are you trying to avoid email harvesting, or are you trying to avoid load on the server/bandwidth ?

[JamesB]

Originally Posted by simon

You can't really stop it entirely, but you can minimize the impact by the way you structure the site. Are you trying to avoid email harvesting, or are you trying to avoid load on the server/bandwidth ?

I know I can not stop all the crawlers, but I sure want to reduce the amount that scan my site.

I pretty much want to avoid the server/bandwidth issue. I am not all that concerned with email harvesting as Im using a disposable email address in which I can delete or change at will
James

[markfh]

why are you putting an email address on it at all?