Board crashing

Big A · Mar 27, 2009

Some of you have noticed the board crashing almost on a daily basis.

It's due to the Yahoo web crawlers. For some reason, Yahoo releases thousands of spiders simultaneously, causing the server to crash.
I am talking with them now, as they are supposed to release the crawlers staggeredly, not simultaneously.

Just a heads up in case conspiracy theorists want to start talking crap.

RONJON · Mar 27, 2009

thanks for the heads up Big A ...was wondering what was up

VanR__ · Mar 27, 2009

You're admins should have some DOS tolerance contingencies in place.
If an IP range opens more then 100 TCP SYNs, block that range for an hour.

tkav1980 · Mar 27, 2009

VanR__ said:
You're admins should have some DOS tolerance contingencies in place.
If an IP range opens more then 100 TCP SYNs, block that range for an hour.

ive never read anything in my life that made less sence to me. holy crap!

hawkmoon · Mar 27, 2009

tkav1980 said:
ive never read anything in my life that made less sence to me. holy crap!

He is saying that A's webhost should have guards in place which limit a flood of connection reqests from a single source.

JUSTKIDDING · Mar 27, 2009

Great work big A,

I also noticed it and touhgt there where problems with my provider, but today i could log-in again and now i read what the problem is.
Hopefully they will fix it soon.

Pekkerwood · Mar 27, 2009

tkav1980 said:
ive never read anything in my life that made less sence to me. holy crap!

Thats hilarious!

Big A · Mar 27, 2009

VanR__ said:
You're admins should have some DOS tolerance contingencies in place.
If an IP range opens more then 100 TCP SYNs, block that range for an hour.

That's a good idea to stop ddos attacks.

I set up the robots.txt file to slow down the yahoo crawlers. Seems to be working.

Showy0023 · Mar 27, 2009

WTF is a "web crawler"??

Big A · Apr 1, 2009

This latest downtime was the longest (7 hours) because I was sleeping and it happened just after I switched the computer off.

The board will go down for about 30 minutes in the next number of hours as I am SERIOUSLY upgrading the server.

We are going to end up with 600GB of memory and 15GB of RAM in a quad core, so we should have no more issues ever.

Big A · Apr 1, 2009

Showy0023 said:
WTF is a "web crawler"??

A web crawler or a spider is a program of a search engine (google, yahoo, etc) that 'crawls' the web's sites looking for relavant information to put in the search engine directories.

There seemed to be a glitch in the yahoo system as we have been 'infested' with their crawlers, all hitting the site at the same time instead of being staggered by time delays.
Basically, 100 of them at one time can take a normal site down. I have a big server, so it took quite a few thousand of them to do it.

With the new server upgrades, the server can handle millions of open connections simultaneously, so there should be no more issues. Sucked in for any hacker that wants to do a ddos attack

rAJJIN · Apr 1, 2009

Big A said:
This latest downtime was the longest (7 hours) because I was sleeping and it happened just after I switched the computer off.

The board will go down for about 30 minutes in the next number of hours as I am SERIOUSLY upgrading the server.

We are going to end up with 600GB of memory and 15GB of RAM in a quad core, so we should have no more issues ever.

Remember....
All this from a guy that has never Once asked his members for Money, Donations, or ANYTHING more then treating the forum and members with some respect.

gridlock · Apr 1, 2009

how about we put in a donation section

it doesnt mean we have to but it would be nice to give back.

i know big a has saved me and some mates over here some serious money through sponsors and there isnt many places you can get info from professionals in the game that are happy to speak to you when you are just an average joe that they've never even met before

RONJON · Apr 1, 2009

Yea it has been ANNOYING AS HELL!!! I thought it was my shitty computer

..Thanks for taking care of this Big A

RJ

FordTurboDiesel · Apr 1, 2009

rAJJIN said:
Remember....
All this from a guy that has never Once asked his members for Money, Donations, or ANYTHING more then treating the forum and members with some respect.

I was gonna say....DAYUM...

Thats a hell of a thing.

Big thanks A

7_Deadly_Sins · Apr 1, 2009

Big A said:
This latest downtime was the longest (7 hours) because I was sleeping and it happened just after I switched the computer off.

The board will go down for about 30 minutes in the next number of hours as I am SERIOUSLY upgrading the server.

We are going to end up with 600GB of memory and 15GB of RAM in a quad core, so we should have no more issues ever.

J/wing, what kind of connection do you have? T1? And what kind of HD set up?(Raid I assume)

Physick · Apr 1, 2009

rAJJIN said:
Remember....
All this from a guy that has never Once asked his members for Money, Donations, or ANYTHING more then treating the forum and members with some respect.

Was just thinking that. Very impressed that Big A keeps investing in this board at the level he does.
Thanks Big A!

kcbuilder · Apr 1, 2009

Thanks, for investing your money for us big a. I'm new here but in the short time ive been a member Id be more than happy to donate some funds to help maintain a good thing.

Glad the board is back up also, don't want my boss thinking that Im always as productive as I was today.

Tomorows newspaper headline- "PM goes down and the production of the american workforce goes up 1000%."

Thanks again Big A

intensed · Apr 1, 2009

Good Price

Big A,

Not sure who you coloate with but I use http://fdcservers.net/ with great success. Check out their prices. I have a hell of a server 100mbit Unmetered. I'm not sure what you are paying now but I but you can find a hell of a server with 100mbit unmetered for under $400!. 100Mbit unmetered would serve this board just fine with extra bandwidth.

Here is a example with you need 1 GIG unmetered!!

1Gbit Burstable Unmetered Server #5

from 649 /month
only CPU - Dual Xeon 5410
Memory - 16GB RAM
Hard Disk Space - 2x1.5TB HDD

norm1220 · Apr 1, 2009

If you continue to have problems from excessive spidering, consider placing a robots.txt file on the root directory of your server. Be sure that it contains the following directive -

User-agent: *
Crawl-delay: 20

This will allow the spiders to crawl the site, allowing for your pages to get ranked and listed, but will prevent too many concurrent connections from bots. The crawl-delay can be set to what ever you want.

Putting more money into hardware isn't always the best answer. Often configuration changes to Apache and MySQL can squeeze a more performance out of your existing system.