Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
SEO Engine Spiders?
#1
I'm having another bout of early morning insomnia, and was just looking around the forum...Noticed that under the "active users" section for right now, Bing, Baidu, and Google are listed.  That may be by design, but if the site is being spidered by search engines daily and you want to lock that down so that archived posts aren't available for spidering in the future, it's a pretty simple robots.txt file change on the server - I'm sure Admin of Steel is familiar with it...Just wanted to raise the question in case the admins don't want search engines returning results from IOPList.com.  If allowing 'bots to spider the site has already been considered and the admins are OK with it, then please disregard.
Reply
#2
There's another one from time to time as well... "Internet archiver"... That one wants to "archive" the entire internet... All of it! I found that out using g00gle...

Admin is aware of this.... So maybe he's the one to answer this question...

Ice
Semper Fidelis

[Image: SyAa0qj.png]

USMC
Nemo me impune lacessit
Reply
#3
(03-26-2016, 07:47 AM)Back2Good Wrote: I'm having another bout of early morning insomnia, and was just looking around the forum...Noticed that under the "active users" section for right now, Bing, Baidu, and Google are listed.  That may be by design, but if the site is being spidered by search engines daily and you want to lock that down so that archived posts aren't available for spidering in the future, it's a pretty simple robots.txt file change on the server - I'm sure Admin of Steel is familiar with it...Just wanted to raise the question in case the admins don't want search engines returning results from IOPList.com.  If allowing 'bots to spider the site has already been considered and the admins are OK with it, then please disregard.

I am well aware of that my friend. The only reason why I left the site to be optimized by search engines is that we have to speed up to enter the Space and then turn off the thrusters so we may move freely in space. We do not mention Sch 1 or Sch A drugs here though and try to run this very safely and strictly. We would be making very massive changes to the forum but it would require our member quantity to be at least 5000 so that we may sit around relaxed without worrying for advertisement aka recognization. I do thank you for your concern regarding our safety but this has already been discussed before and we would surely give it a go along with https protocols and totally Going private.
The only way to know how strong you are is to keep testing your limits.
Reply
#4
(03-26-2016, 09:03 AM)IceWizard Wrote: There's another one from time to time as well... "Internet archiver"... That one wants to "archive" the entire internet... All of it! I found that out using g00gle...

Admin is aware of this.... So maybe he's the one to answer this question...

Ice

Ice, You have more reputation points than me. Kinda makes me jealous. Kidding.   Tongue
The only way to know how strong you are is to keep testing your limits.
Reply
#5
Big Grin 
(03-28-2016, 05:15 AM)ADMIN OF STEEL Wrote:
(03-26-2016, 07:47 AM)Back2Good Wrote: I'm having another bout of early morning insomnia, and was just looking around the forum...Noticed that under the "active users" section for right now, Bing, Baidu, and Google are listed.  That may be by design, but if the site is being spidered by search engines daily and you want to lock that down so that archived posts aren't available for spidering in the future, it's a pretty simple robots.txt file change on the server - I'm sure Admin of Steel is familiar with it...Just wanted to raise the question in case the admins don't want search engines returning results from IOPList.com.  If allowing 'bots to spider the site has already been considered and the admins are OK with it, then please disregard.

I am well aware of that my friend. The only reason why I left the site to be optimized by search engines is that we have to speed up to enter the Space and then turn off the thrusters so we may move freely in space. We do not mention Sch 1 or Sch A drugs here though and try to run this very safely and strictly. We would be making very massive changes to the forum but it would require our member quantity to be at least 5000 so that we may sit around relaxed without worrying for advertisement aka recognization. I do thank you for your concern regarding our safety but this has already been discussed before and we would surely give it a go along with https protocols and totally Going private.

I suspected you were aware of it and had reasons for leaving SEO engines permission to spider the site, but insomnia leaves one looking for almost anything to do to occupy time until one can go back to sleep, so I thought I'd ask the question.  No disrespect whatsoever intended - And I have noticed the lack of discussion of the items you mention, which is one of the reasons I feel comfortable posting here, where I wouldn't elsewhere.  Thank you for answering my query rather than disregarding it - I know you must have hundreds of other more important things to do!

Oh, and as soon as I reach the point where I'm able I'll drop a reputation point on you for the reply :-) 

Best,

- b2g
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)