Host constantly attacked, options?

K

Kainzo

Gawd
Joined
Nov 12, 2009
Messages
972
So here's the issue. I run a very popular gaming server for Minecraft.
We seem to get hit by DDOS's every 2-3 weeks. These attacks have lasted a week straight before.

Some of these attacks have reached 10Gbps. This is not a "DOS", I have been dealing with this since August and I am frankly tired of it.

The options that I've seen are:
1) Javapipe.com (Reverse Proxy to Thwart Anti-DDOS through packet filtering)
2) Datacenter that offers some form of DDOS shield
3) Eating the DDOS and waiting (Tried this for a week+ and it never stopped)


Some other things to note:
We are currently hosted with OPlink with their collocated services, we own the hardware we're on (i7 3960x 32gb 4TBB HDD)
A software firewall (like CSF) will not cut it. 90% of these packets never resolve and are spamming/overloading the network card.
The host is trying to nullroute the ip's but have been through 200+ already. It is not possible for them to cut all UDP / ICMP packets form the machine via the switch.


What would you do in my place?
 
Last edited:
They can't filter out UDP and ICMP for you? They must have some terribly old hardware, don't know how to do it, or are unwilling to do it...
 
Blue Fox said:
They can't filter out UDP and ICMP for you? They must have some terribly old hardware, don't know how to do it, or are unwilling to do it...
Click to expand...

oplink tech and I worked until 1AM last night with me reporting the IP's by hand and then him null routing them at the IP.

I requested several times to block ALL UDP frags / ICMP and he said its not possible to do.

Which means he may not know or just may not have the ability to do so.

It's getting very frustrating.

The server has gone up and down probably 50+ times in the last 6 hrs.
 
Here's a quote from the host about blocking it.
We could block ICMP, or UDP, but again, your port will be hit and the bandwidth consequences would be the same.

I believe the attack is ICMP, not UDP. The UDP traffic appears small relative to ICMP. Granted, there are lots of IP's sending UDP packets, but they are not the problem.

Anyway, our uplink router is experiencing a small increase in load right now - indicating that it's blocking the extra ICMP bandwidth.
Click to expand...
 
Kainzo said:
oplink tech and I worked until 1AM last night with me reporting the IP's by hand and then him null routing them at the IP.

I requested several times to block ALL UDP frags / ICMP and he said its not possible to do.

Which means he may not know or just may not have the ability to do so.

It's getting very frustrating.

The server has gone up and down probably 50+ times in the last 6 hrs.
Click to expand...
Even on a 10 year old Cisco router you can do the following. It's really this simple:

Code: 
ip access-list extended [I]name[/I]
   10 deny icmp any xxx.xxx.xxx.xxx 0.0.0.0
   20 deny udp any xxx.xxx.xxx.xxx 0.0.0.0
   30 permit ip any xxx.xxx.xxx.xxx 0.0.0.0

If you have multiple IPs it isn't much more complicated. Depending on how they have their network set up, they could do it for your subinterface instead of as a global thing. Either way, a pretty trivial thing to do.

Kainzo said:
Right, got any ideas for a good collocation host that has protection from massive ddos?
Click to expand...
I think a lot of this depends on where you are and whether or not your server is a tower or something that can be racked.
 
Kainzo said:
Here's a quote from the host about blocking it.
Click to expand...

It is getting into their network and hitting your interface right? They have the network overhead to handle your flood and their other clients? Then they have the ability to stop the ICMP flood at their firewall, router, switch level. Find a new host.
 
They probably don't want to block it as it will affect all customers, but I wonder if they can just make the rule block it only if the destination is your range.

I think you will need to look at a host that specializes in DDoS mitigation. Your host is trying, but they're probably just not equipped as well as a host that can do it. It's sad that DDoS are not illegal and piracy is, because DDoS causes way more damage by being able to bring down entire networks. While your host is being attack everyone else's server is probably very slow as well because the DDoS is using up the host's pipe.

Are you on Carat Networks by chance, they've been getting hit with DDoSes lately as well.
 
Red Squirrel said:
They probably don't want to block it as it will affect all customers, but I wonder if they can just make the rule block it only if the destination is your range.
Click to expand...
I don't understand how it would effect everyone else. I'm not saying block IMCP for the entire data center, just stop before it floods his interface on the switch. It is likely being passed through numerous network devices before hitting his server that could block the attack. It obviously won't stop it, but it will allow the client to regain services. And if they put a stop to it at the edge of the network it won't hit as many devices using network resources. If a client is being bombarded with bad traffic the first thing you do is block the traffic before it hits all your devices then work on mitigation. The traffic doesn't need to hit his server to troubleshoot...
 
sheesh721 said:
I don't understand how it would effect everyone else. I'm not saying block IMCP for the entire data center, just stop before it floods his interface on the switch. It is likely being passed through numerous network devices before hitting his server that could block the attack. It obviously won't stop it, but it will allow the client to regain services. And if they put a stop to it at the edge of the network it won't hit as many devices using network resources. If a client is being bombarded with bad traffic the first thing you do is block the traffic before it hits all your devices then work on mitigation. The traffic doesn't need to hit his server to troubleshoot...
Click to expand...

Once the traffic gets past the core the damage is done. But I'm sure they can just block it for his IP as blue fox posted, if they are using Cisco, if not I'm sure whatever they're using must have some ability to set this rule. There's also the issue of documenting the changes. Normally you can't just go making a change into a core router without documenting or even having to go through CM. It sounds to me like the host is just not equipped, from an equipment, and process point of view, for these type of situations.
 
Just learned that the IP's are actually spoofed. So what I've been working on the last 12 hrs is all for not.

"We have a problem. Despite the null routing the ICMP IP's: 186.182.185.164 and
186.158.246.46 are listed in your log as attacking.

This means that the IP's are spoofed, and nothing I have done has really had
an effect.
Click to expand...
 
Last edited:
I assume your host is nearly crippled by the attack. Unless they block this at the core, odds are the TOR switch uplink port is actually getting cooked by the attack, too. You're going to need to work with someone like BlackLotus or get transit in a DC that exceeds the DoS size + buy your own edge router and eat the traffic.

Your cheapest option is to talk to BlackLotus ( http://www.blacklotus.net/protect/protection-for-minecraft ). The other option will exceed $10k a month, and doesn't include the gear you'll need to buy.

There is really nothing you can do. I work closely with a certain large bank in Pittsburgh and the websites were taken offline this week and everything imaginable was tried. buying200gbpsoftransitnextweeklol
 
Last edited:
I'm more than happy with the current host, the issue is that we're a very large target for attacks =/

I'm trying to find a VPS that has a good amount of protection, if I can find one based in Texas, I'll be able to tunnel the connections through to our collocated host. The issue is, it will put the bulk of attacks onto the VPS and if they aren't ready, they will go down hard.

Ultimately, Javapipe may be the only answer for these sort of attacks since they are "set up" for Anti-ddos reverse proxy, but I'm still all ears.


Brak710 said:
I assume your host is nearly crippled by the attack. Unless they block this at the core, odds are the TOR switch uplink port is actually getting cooked by the attack, too. You're going to need to work with someone like BlackLotus or get transit in a DC that exceeds the DoS size + buy your own edge router and eat the traffic.

Your cheapest option is to talk to BlackLotus ( http://www.blacklotus.net/protect/protection-for-minecraft ). The other option will exceed $10k a month, and doesn't include the gear you'll need to buy.

There is really nothing you can do. I work closely with a certain large bank in Pittsburgh and the websites were taken offline this week and everything imaginable was tried. buying200gbpsoftransitnextweeklol
Click to expand...
Yah, I have had friends who run servers and they went with Blacklotus, they said the latency was pretty high. I guess that's the price you pay for ultimate protection.
 
goodcooper said:
was wondering the same thing... but surely all DDoS are the same, regardless of your host
Click to expand...

adam30k said:
Is something like Cloudflare only applicable to web hosting?
Click to expand...

Correct, Cloudflare is only for websites - it does not (and will not) protect gaming servers due to their "high target" status.

So... the host has been working on blocking all incoming ICMP (and UDP fragments) just to my block of addresses but still for some reason I'm reporting ICMP and UDP on my IPtables. I've rebooted the machine and it still persists..

Here's the info he posted to me:

Extended IP access list HC
10 deny icmp any 216.230.231.160 0.0.0.7 (8892 matches)
20 deny icmp any 66.187.65.240 0.0.0.7
30 deny udp any 216.230.231.160 0.0.0.7 (550 matches)
40 deny udp any 66.187.65.240 0.0.0.7 (18 matches)
50 permit ip any any (71282 matches)
Click to expand...

Shouldn't this completely stop ICMP from hitting our block of addresses/machine? The ICMP flood is bringing the box down and we're able to pick up probably 5-10% of the flood in IPtable logs.
 
I did a reverse IP check on those IPs, and you are getting hit from inside the US. You could try reporting the traffic to their ISPs to get it killed at the source.

We were getting attacked on our firewall at work, but luckily all the addresses reversed to IPs in china. Since we have no business/customers in china, I just started blocked very large subnets that were constantly showing up in the logs. Any time a new subnet showed up again, i would either block the subnet, or increase the range of the existing blocked subnets. Doing this in your situation most likely wont work as you may be blocking ligitimate traffic.
 
If you're serious about running a public gaming server, you're going to either spend the money to do it right, or you're going to disappear.

You need a network admin that doesn't ask questions on [H].

This is not the first time I've seen you post about this. It's been months.

Get a real router, learn how to use it, and get your own uplinks.
 
Biznatch said:
I did a reverse IP check on those IPs, and you are getting hit from inside the US. You could try reporting the traffic to their ISPs to get it killed at the source.
Click to expand...
If you're talking about the ones in the firewall rules he posted, those belong to him and not the attackers.
 
mwarps said:
If you're serious about running a public gaming server, you're going to either spend the money to do it right, or you're going to disappear.

You need a network admin that doesn't ask questions on [H].

This is not the first time I've seen you post about this. It's been months.

Get a real router, learn how to use it, and get your own uplinks.
Click to expand...

There is nothing wrong with asking like he is. It's a bad situation. He's asking here because a lot of us have experience with this. Problem is, our budgets at work are in the millions, and his is whatever him and a gaming community can afford.

Going with BlackLotus or JavaPipe (or someone similar) is the only way to go. They have the multi-million dollar budgets for you.

I have no idea who you managed to piss off this bad that they waste a 10gbps botnet against a Minecraft server. Are these constant DNS Amplification attacks, by chance?
 
Is there a way you can tell your players of a new server, just keep it low profile, and move your whole operation elsewhere? (different host and different domain, IP etc) Of course, this is easier said than done when you have an established setup going on.
 
Brak710 said:
I have no idea who you managed to piss off this bad that they waste a 10gbps botnet against a Minecraft server.
Click to expand...

I wonder this myself. Fair bit of work just to take down a minecraft server.
 
Brak710 said:
There is nothing wrong with asking like he is. It's a bad situation. He's asking here because a lot of us have experience with this. Problem is, our budgets at work are in the millions, and his is whatever him and a gaming community can afford.

Going with BlackLotus or JavaPipe (or someone similar) is the only way to go. They have the multi-million dollar budgets for you.

I have no idea who you managed to piss off this bad that they waste a 10gbps botnet against a Minecraft server. Are these constant DNS Amplification attacks, by chance?
Click to expand...
I'm really not sure what a constant DNS Amplifcation attack is, but that is very possible. Our ntop did go crazy a bit ago and my linux guy did say he had to uninstall it to stop some of the oddities.

Biznatch said:
I did a reverse IP check on those IPs, and you are getting hit from inside the US. You could try reporting the traffic to their ISPs to get it killed at the source.

We were getting attacked on our firewall at work, but luckily all the addresses reversed to IPs in china. Since we have no business/customers in china, I just started blocked very large subnets that were constantly showing up in the logs. Any time a new subnet showed up again, i would either block the subnet, or increase the range of the existing blocked subnets. Doing this in your situation most likely wont work as you may be blocking ligitimate traffic.
Click to expand...

Some of the IPs that we saw were resolving from inside of the USA - we have contacted over 10 ISP's so far in these series of attacks.

mwarps said:
If you're serious about running a public gaming server, you're going to either spend the money to do it right, or you're going to disappear.

You need a network admin that doesn't ask questions on [H].

This is not the first time I've seen you post about this. It's been months.

Get a real router, learn how to use it, and get your own uplinks.
Click to expand...

I completely disagree with you. I built this server and community of 20k+ players from the ground up, sure we may not be professionals at this point in time with million+ income, but we're making it on donations alone and doing pretty well.


Red Squirrel said:
Is there a way you can tell your players of a new server, just keep it low profile, and move your whole operation elsewhere? (different host and different domain, IP etc) Of course, this is easier said than done when you have an established setup going on.
Click to expand...
That isnt possible, we're a high profile server and the reason why is because we broadcast ourselves openly - if we go reclusively we'll eventually die off.

======================
As to who we pissed off? I'm not 100%
Well, the entire store can be found here....http://hardforum.com/showthread.php?t=1710405

That pretty much covers who we "pissed off" - I can't say for 100% that its the same people but I do know that its the same type of attack and with the same intensity - it lasts for days and there's no sign of stopping. The persons involved said they will never stop until we are wiped off the net.
 
hire a brazilian thug to take the guy out...

in the favelas $100 could get you at least the guy beaten up and his computers smashed...
 
You might be able to use adaptive DOS/DDOS protection, so that the ips will automatically be blocked if they make repetitive connections.
 
jimh425 said:
You might be able to use adaptive DOS/DDOS protection, so that the ips will automatically be blocked if they make repetitive connections.
Click to expand...

We've tried most everything, it doesn't appear to be a DNS amplification attack - the IPs hitting us don't resolve to DNS servers.

That is a thought... why doesn't cloudflare offer a premium service that protects game servers? :~(
 
CloudFlare's offering would be very near to what BlackLotus does. I know you're worried about latency, but I think they could fix your issue, and minecraft is more tolerant to latency than an FPS is.
 
Actually to fix the latency issue, how about keep your current host, but also get one at CloudFlare or equivalent, host the DNS there too. Run your stuff off the normal host, but in the event of a DDoS simply have it switch over to the other host. You can probably script it so it copies the data over every 5 minutes or so and when it switches over it attempts to copy over (but a DDoS may stop that from happening). So at worse you'd loose 5 minutes of progress or so.

Are you running it in Windows or Linux? If Linux then your options are better as you can use ssh/rsync scripts.

You would be paying for two hosts though, that's the only thing. So maybe it's best to just switch over and deal with the latency, maybe it wont be so bad.
 
Have you looked into what AWS would cost? They've got more bandwidth than you can imagine and have DDOS mitigation.
 
Brak710 said:
There is nothing wrong with asking like he is. It's a bad situation. He's asking here because a lot of us have experience with this. Problem is, our budgets at work are in the millions, and his is whatever him and a gaming community can afford.
Click to expand...

I agree, but you don't need millions to do it right.

I have no idea who you managed to piss off this bad that they waste a 10gbps botnet against a Minecraft server. Are these constant DNS Amplification attacks, by chance?
Click to expand...

DNS amplification attacks will not have spoofed IP addresses, they will have hundreds, if not thousands of sources.
 
Kainzo said:
I'm really not sure what a constant DNS Amplifcation attack is, but that is very possible. Our ntop did go crazy a bit ago and my linux guy did say he had to uninstall it to stop some of the oddities.
Click to expand...
DNS amplification shouldn't have even been mentioned with the characteristics of the attacks you mentioned (low source IP and spoofed addresses). Cloudflare's DDoS protection even being mentioned in this thread shows that there is a fundamental lack of knowledge about this attack vector among the posters in this thread.

I completely disagree with you. I built this server and community of 20k+ players from the ground up, sure we may not be professionals at this point in time with million+ income, but we're making it on donations alone and doing pretty well.
Click to expand...

I play on TF2 servers with uptime above 95% for free. "doing pretty well" means having uptime close to that, especially if you see any sort of money. Sorry, but you're not doing "pretty well".

You don't need millions of dollars to do this right, not by any means.

Get your own uplink, get a real router, use the upstream's filtering. Problem solved.
 
Brak710 said:
CloudFlare's offering would be very near to what BlackLotus does. I know you're worried about latency, but I think they could fix your issue, and minecraft is more tolerant to latency than an FPS is.
Click to expand...

Last I checked Cloudflare does not offer gaming server anti-DDOS. It's not a feature.

Red Squirrel said:
Actually to fix the latency issue, how about keep your current host, but also get one at CloudFlare or equivalent, host the DNS there too. Run your stuff off the normal host, but in the event of a DDoS simply have it switch over to the other host. You can probably script it so it copies the data over every 5 minutes or so and when it switches over it attempts to copy over (but a DDoS may stop that from happening). So at worse you'd loose 5 minutes of progress or so.

Are you running it in Windows or Linux? If Linux then your options are better as you can use ssh/rsync scripts.

You would be paying for two hosts though, that's the only thing. So maybe it's best to just switch over and deal with the latency, maybe it wont be so bad.
Click to expand...

We're running debian 7 (Wheezy) kernel 3.2 :)

Heading to bed ill read this tomorrow and edit.
 
You must log in or register to reply here.
Back
Top