We were having some problems with HTTP POST performance from the West coast to a tomcat server farm on the East coast. None of these problems showed up locally (between two hosts in the same datacenter.) The latency between West and East coast sites is on the order of 70ms. After much investigation, we (two of my coworkers and I) managed to reduce a post of a 32MB file from ~87 seconds to ~4 seconds with a combination of two things: changing socket.rxBufSize and changing /etc/sysctl.conf:
net.ipv4.tcp_mem = 196608 262144 1048576
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 1048576 67108864
net.ipv4.tcp_wmem = 4096 1048576 67108864
net.core.netdev_max_backlog = 250000
In this case I think tcp_rmem and rmem_max are the important settings here. One of the odd things about this was the TCP window wasn't scaling up.... the most I was seeing before this was 38K or so. Not at all what I expected.
By just changing the Tomcat 6 server.xml buffer size we got the time down from 87 to 39 seconds. A further change of the sysctl parameters got us down to just over 4 seconds. The changes we made to the sysctl parameters were limited to the second argument to net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. Those other parameters were set for a previous unsuccessful attempt to improve the situation.
Also, these values were arrived at unscientifically. There are almost certainly some better choices out there. Based on some testing with stock CentOS 5.x scp, I think I should be able to get it to 2 seconds or better.