It is currently Fri Aug 18, 2017 3:35 am

All times are UTC - 7 hours





Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: Server bandwidth limits
PostPosted: Wed Jun 25, 2008 9:49 am 
Offline
User avatar

Joined: Sun Sep 19, 2004 9:28 pm
Posts: 3192
Location: Mountain View, CA, USA
Just thought I'd post this here to keep folks informed.

The short version:

Some Chinese dude downloading files in a peculiar way tickled a bug in some software on our webserver, so to avoid that, we've made some changes.

The long version:

Over the past couple of weeks, I've seen a series of Chinese IP addresses downloading specific files on nesdev.com (mainly large ZIP files). Nothing surprising there, but there was an inadvertent "trickle-down" effect as a result of such.

The client software they're using is some sort of multipart downloader (and it spoofs its User-Agent, so I have no idea what it's actually called). Multipart downloaders download only certain sections of the file, then put it all together once all the sections are done. The software is configured for something like 8 concurrent HTTP connections at once. 31457280 / 8 = 3932160 bytes per HTTP connection.

There's really nothing "wrong" with this, and lots of people use them, mainly to get around bandwidth limit rules imposed by their ISP (rather than just rate-limit the port entirely, the ISP chooses to rate-limit each individual TCP connection; by using multiples at once, you can essentially get around the bandwidth limit. I know Telia (Sweden) uses such things...).

Anyway, the above situation was causing an Apache module we use (called mod_cband) for limiting individual websites' speed (per second) to break horribly, resulting in all the other sites locking up until some of the TCP sockets from the multipart downloader had closed. I know, it sounds like a system or Apache tuning problem, but trust me, it isn't: disable mod_cband and the problem goes away entirely. Hmm... ;)

Then I found a couple Debian PRs documenting similar problems with the speed limiting function of mod_cband. We don't use Linux, but the issue isn't Linux-specific. That, combined with the fact that the only home page for mod_cband has been gone/offline for months, caused the Debian folks to pull the software from Debian's stable and unstable branches entirely:

http://bugs.debian.org/375016
http://bugs.debian.org/418645

That pretty much convinced me right there. I took the liberty of saying "fuck this", removed mod_cband entirely, and resorted to using kernel-level bandwidth limiting features (for those who care, pf ALTQ).

This required me to give nesdev.com it's own IP address, rather than use name-based virtualhosts like I've been doing for many years. The IP of the site has moved from 72.20.106.5 to 72.20.106.40, in case you notice DNS changes. I also updated the DNS records for nesdev.com and www.nesdev.com to point to the new IP.

Since DNS takes time to propagate, I've left support in our webserver for answering nesdev.com on both the old and new IP. In a few days, I'll be adjusting Apache to no longer respond for nesdev.com on the .5 address. (Our DNS TTLs are fairly short, so a few days is quite generous)

That's all. :-)

Oh, and one other thing: yes, the bandwidth limit is the same as it was before (384kbit/sec). :P


Top
 Profile  
 
 Post subject:
PostPosted: Wed Jun 25, 2008 1:55 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 18793
Location: NE Indiana, USA (NTSC)
Interesting.

I've seen similar multipart downloaders hit my personal web server that I use for things like my own homepage and files that I post to forums like this one that don't allow attaching files to posts. Most of the time, when I can't hit my own homepage, I check the log, and sure enough, there's a badly misbehaving downloader that repeatedly tries to get one 2 MB zipfile for an hour at a time. I deny the IP, restart Apache, and even though my log starts to fill up with 403 (forbidden) errors, things become responsive again. I haven't seen this attack from more than one IP at once, so it isn't likely to have too much collateral damage.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Jun 25, 2008 5:36 pm 
Offline
Formerly Fx3
User avatar

Joined: Fri Nov 12, 2004 4:59 pm
Posts: 3040
Location: Brazil
- I suppose they download the zipped nesdev site, correct? It's not the first time you complain about that, so why not taking down such zip file? Well, 99% of the relevant NES info is posted around here... so... I see no reason for seeding a chinese crazy guy. ;)

_________________
Zepper
RockNES developer


Top
 Profile  
 
 Post subject:
PostPosted: Wed Jun 25, 2008 9:14 pm 
Offline
User avatar

Joined: Sun Sep 19, 2004 9:28 pm
Posts: 3192
Location: Mountain View, CA, USA
Fx3 wrote:
I suppose they download the zipped nesdev site, correct? It's not the first time you complain about that, so why not taking down such zip file?


No, it's not. The file the user was continually multiparting was SH657XTool_release.zip.

The weekly nesdev ZIP file is actually made automatically via a cronjob and put up on our FTP server -- it's not available via HTTP.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

All times are UTC - 7 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group