Server bandwidth limits

Report broken links, improper HTML, or formatting issues here.

Moderator: Moderators

Post Reply
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Server bandwidth limits

Post by koitsu »

Just thought I'd post this here to keep folks informed.

The short version:

Some Chinese dude downloading files in a peculiar way tickled a bug in some software on our webserver, so to avoid that, we've made some changes.

The long version:

Over the past couple of weeks, I've seen a series of Chinese IP addresses downloading specific files on nesdev.com (mainly large ZIP files). Nothing surprising there, but there was an inadvertent "trickle-down" effect as a result of such.

The client software they're using is some sort of multipart downloader (and it spoofs its User-Agent, so I have no idea what it's actually called). Multipart downloaders download only certain sections of the file, then put it all together once all the sections are done. The software is configured for something like 8 concurrent HTTP connections at once. 31457280 / 8 = 3932160 bytes per HTTP connection.

There's really nothing "wrong" with this, and lots of people use them, mainly to get around bandwidth limit rules imposed by their ISP (rather than just rate-limit the port entirely, the ISP chooses to rate-limit each individual TCP connection; by using multiples at once, you can essentially get around the bandwidth limit. I know Telia (Sweden) uses such things...).

Anyway, the above situation was causing an Apache module we use (called mod_cband) for limiting individual websites' speed (per second) to break horribly, resulting in all the other sites locking up until some of the TCP sockets from the multipart downloader had closed. I know, it sounds like a system or Apache tuning problem, but trust me, it isn't: disable mod_cband and the problem goes away entirely. Hmm... ;)

Then I found a couple Debian PRs documenting similar problems with the speed limiting function of mod_cband. We don't use Linux, but the issue isn't Linux-specific. That, combined with the fact that the only home page for mod_cband has been gone/offline for months, caused the Debian folks to pull the software from Debian's stable and unstable branches entirely:

http://bugs.debian.org/375016
http://bugs.debian.org/418645

That pretty much convinced me right there. I took the liberty of saying "fuck this", removed mod_cband entirely, and resorted to using kernel-level bandwidth limiting features (for those who care, pf ALTQ).

This required me to give nesdev.com it's own IP address, rather than use name-based virtualhosts like I've been doing for many years. The IP of the site has moved from 72.20.106.5 to 72.20.106.40, in case you notice DNS changes. I also updated the DNS records for nesdev.com and www.nesdev.com to point to the new IP.

Since DNS takes time to propagate, I've left support in our webserver for answering nesdev.com on both the old and new IP. In a few days, I'll be adjusting Apache to no longer respond for nesdev.com on the .5 address. (Our DNS TTLs are fairly short, so a few days is quite generous)

That's all. :-)

Oh, and one other thing: yes, the bandwidth limit is the same as it was before (384kbit/sec). :P
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

Interesting.

I've seen similar multipart downloaders hit my personal web server that I use for things like my own homepage and files that I post to forums like this one that don't allow attaching files to posts. Most of the time, when I can't hit my own homepage, I check the log, and sure enough, there's a badly misbehaving downloader that repeatedly tries to get one 2 MB zipfile for an hour at a time. I deny the IP, restart Apache, and even though my log starts to fill up with 403 (forbidden) errors, things become responsive again. I haven't seen this attack from more than one IP at once, so it isn't likely to have too much collateral damage.
User avatar
Zepper
Formerly Fx3
Posts: 3262
Joined: Fri Nov 12, 2004 4:59 pm
Location: Brazil
Contact:

Post by Zepper »

- I suppose they download the zipped nesdev site, correct? It's not the first time you complain about that, so why not taking down such zip file? Well, 99% of the relevant NES info is posted around here... so... I see no reason for seeding a chinese crazy guy. ;)
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Post by koitsu »

Fx3 wrote:I suppose they download the zipped nesdev site, correct? It's not the first time you complain about that, so why not taking down such zip file?
No, it's not. The file the user was continually multiparting was SH657XTool_release.zip.

The weekly nesdev ZIP file is actually made automatically via a cronjob and put up on our FTP server -- it's not available via HTTP.
Post Reply