Subversion on 1 Gb uplink
is it possible at all to utilize 1 Gb uplink? I have 2 server with 1 Gb uplink
176.9.xxx.xxx - server
# uname -a
Linux svn.example.net 2.6.32-5-amd64 #1 SMP Mon Sep 23 22:14:43 UTC 2013 x86_64 GNU/Linux
# cat /etc/debian_version
6.0.8
# svnadmin --version
svnadmin, version 1.6.23 (r1485506)
compiled May 29 2013, 10:00:56
Copyright (C) 2000-2009 CollabNet.
Subversion is open source software, see http://subversion.apache.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).
The following repository back-end (FS) modules are available:
* fs_base : Module for working with a Berkeley DB repository.
* fs_fs : Module for working with a plain file (FSFS) repository.
# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on
Supports Wake-on: pumbag
Wake-on: g
Current message level: 0x00000001 (1)
Link detected: yes
144.76.xxx.xxx - client
# uname -a
Linux test.example.net 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS release 6.5 (Final)
# svnadmin --version
svnadmin, version 1.6.11 (r934486)
compiled Apr 11 2013, 16:13:51
Copyright (C) 2000-2009 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).
The following repository back-end (FS) modules are available:
* fs_base : Module for working with a Berkeley DB repository.
* fs_fs : Module for working with a plain file (FSFS) repository.
# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
drv probe ifdown ifup
Link detected: yes
Some basic tests
# iperf -c 176.9.xxx.xxx -t 60
------------------------------------------------------------
Client connecting to 176.9.xxx.xxx, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[ 3] local 144.76.xxx.xxx port 42619 connected with 176.9.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 4.15 GBytes 594 Mbits/sec
# iperf -c 144.76.xxx.xxx -t 60
------------------------------------------------------------
Client connecting to 144.76.xxx.xxx, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 176.9.xxx.xxx port 54666 connected with 144.76.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 3.17 GBytes 453 Mbits/sec
At the same time when I download some project from svn via http speed is about 100 Mbit/s max. But a plain binary file has been downloaded at max speed
# axel -a -v http://176.9.xxx.xxx/test.img
Initializing download: http://176.9.xxx.xxx/test.img
File size: 1101824020 bytes
Opening output file test.img
Starting download
Connection 3 finished ]
Connection 0 finished ]
Connection 1 finished ]
[100%] [..................................................] [ 94.3MB/s] [00:00]
Downloaded 1050.8 megabytes in 11 seconds. (96588.44 KB/s)
# wget http://176.9.xxx.xxx/test.img
--2014-02-01 14:21:13-- http://176.9.xxx.xxx/test.img
Connecting to 176.9.xxx.xxx:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1101824020 (1.0G) [text/plain]
Saving to: “test.img”
100%[=================================>] 1,101,824,020 51.0M/s in 21s
2014-02-01 14:21:34 (49.9 MB/s) - “test.img” saved [1101824020/1101824020]
# curl -o test.img http://176.9.xxx.xxx/test.img
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1050M 100 1050M 0 0 56.8M 0 0:00:18 0:00:18 --:--:-- 57.2M
Any advice would be highly appreciated.
Update1
As you an see the speed is around 100 Mbit
# time svn co http://svn.example.net/Test/ ./Test/
Authentication realm: <http://svn.example.net:80> Authorization required.
Password for 'user':
A Test/test.img
Checked out revision 1.
real 1m40.768s
user 0m48.885s
sys 0m3.738s
Update2 The same file but via svn protocol, the speed around 250 Mbit/s
# time svn co svn://svn.example.net/ ./Test/
A Test/test.img
Checked out revision 1.
real 0m46.075s
user 0m15.338s
sys 0m4.811s
System load during checkout
# top
top - 18:26:34 up 60 days, 10:14, 1 user, load average: 0.25, 0.06, 0.02
Tasks: 214 total, 1 running, 213 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 99.0%us, 1.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.9%sy, 0.0%ni, 99.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16377584k total, 15345624k used, 1031960k free, 203640k buffers
Swap: 8388600k total, 1956k used, 8386644k free, 13597864k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5449 www-data 20 0 409m 68m 3656 S 100 0.4 2:17.41 apache2
6885 root 20 0 10976 1340 944 R 1 0.0 0:00.28 top
ALex_hha2 Answers
We know too little about your subversion test to answer the question. Try downloading test.img from the repository after you commited it, it should fully satisfy your connection. My guess is that you are measuring while trying to clone a repository which normally is represented with many individual small files thus making many connections necessary and thus slowing down the transfer.
ooxiBecause you can get good speed with network-only benchmarks it looks like the problem is SVN itself.
SVN is slow. It does a lot of work when receiving and sending a file.
The processing involves copying the data multiple times: copying from the network buffer to the program's memory; from the program's memory to an outgoing disk buffer; from a disk buffer to the disk. That's at least 3 copies and if the developer wasn't careful there could be more copies (i.e. extracting the data out of the HTTP stream, decoding it from the SVN file format into plain data, etc). Programs like cp/wget/rsync are highly optimized and attempt to do zero copies... just manipulate the data in place.
I'm speculating but.... SVN wasn't designed to be fast. It was designed to be reliable. Therefore keeping things simple (strictly copying data to each layer) is more important than complex algorithms that reduce data copying.
One way you can tell if a program is doing a lot of pre- and post-processing is if the "user" time is high. In your example the user time was 30% of the time: real 1m40.768s user 0m48.885s sys 0m3.738s 48.885s is about 31% of the total time. That's a lot of processing for each block of data received. There is 48.2 seconds of time unaccounted for. This is the system waiting for disk or networking; which could indicate your storage has slow disks).
Compare to this to copying a big (1382445394 bytes or ~1.3G) file using the highly optimized "cp" command:
$ time cp bigfile.mp4 copyofbigfile.mp4
real 0m3.268s
user 0m0.005s
sys 0m0.994s
The "user" time is 0.005s. That means "cp" is basically setting up a bunch of parameters and letting the kernel do all the work. Almost the 70% of the time is spent waiting for disk (and this is on my laptop with all SSDs).
You get better performance using the svn: protocol instead of http:. That indicates that the HTTP library they are using is reducing performance. It makes sense that the native protocol is faster.
Your original question was "is it possible at all to utilize 1 Gb uplink?" I would speculate that the answer is "yes" but you will need to fix the bottleneck. Switching to the SVN protocol is a good first step. It looks like faster CPU or faster RAM would help too.
That's my recommendation for 1 file. If there are multiple files involved then you have a different problem. Between files SVN does other processing. That might be CPU-bound, but it could be disk bound also (if it is waiting for the file to fsync() before it moves to the next file). That's an entire different set of benchmarks and analysis to perform.
TomOnTimeUser contributions licensed under CC BY-SA 3.0