![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
TeeJayHoward
Limpness Supreme
- Joined
- Feb 8, 2005
- Messages
- 12,293
I've got a 2011 Macbook Pro 13" (Sumomo) connecting to a ZFS share on my file server (banshee) via a Cat-6 cable. I'm getting transfer speeds of a little over 50MB/s, when I'm expecting transfer speeds of almost twice that. Can anyone tell me what's going on?
The ZFS share is a home-built box with an Opteron 165 on an nForce4 chipset. The ethernet card used is nge0, Solaris 11 Express' built-in driver. Drives are connected to the board via a pair of Supermicro AOC-SAT2-MV8 PCI-X cards in PCI slots. As you can see from the DD, write speeds are capped by the PCI bus at about 100MB/s. Read speeds are similar. I have 14 drives in 2 ZFS pools on these cards.
Sumomo: Downloads tim$ dd if=/dev/zero of=./deleteme.1gb bs=1m count=1024
Sumomo: Downloads tim$ scp -rv deleteme.1gb 192.168.0.200:/tank
Sumomo: Downloads tim$ ssh 192.168.0.200
Password:
Last login: Tue May 10 06:31:05 2011 from 192.168.0.3
Oracle Corporation SunOS 5.11 snv_151a November 2010
tim@banshee:~$ cd /tank
tim@banshee:/tank$ rm deleteme.1gb
tim@banshee:/tank$ time dd if=/dev/zero of=./deleteme.1gb bs=1M count=1024
tim@banshee:/tank$ scp -r -v deleteme.1gb 192.168.0.3:/Users/tim
Edit: decided to test a copy from one pool to another. Here's the controller setup:
tim@banshee:/tank$ zpool status
tim@banshee:/tank$ time cp -v ./deleteme.1gb /backup/
Hrm. About 75MB/s.
Edit: I'm wondering... Is the fact that it has to write to drives on two different controllers that share a bus the reason that it's going about half speed? But no, then why would DD work at 100MB/s?
The ZFS share is a home-built box with an Opteron 165 on an nForce4 chipset. The ethernet card used is nge0, Solaris 11 Express' built-in driver. Drives are connected to the board via a pair of Supermicro AOC-SAT2-MV8 PCI-X cards in PCI slots. As you can see from the DD, write speeds are capped by the PCI bus at about 100MB/s. Read speeds are similar. I have 14 drives in 2 ZFS pools on these cards.
Sumomo: Downloads tim$ dd if=/dev/zero of=./deleteme.1gb bs=1m count=1024
Code:
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 9.907154 secs (108380453 bytes/sec)
Code:
Executing: program /usr/bin/ssh host 192.168.0.200, user (unspecified), command scp -v -r -t /tank
OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to 192.168.0.200 [192.168.0.200] port 22.
debug1: Connection established.
debug1: identity file /Users/tim/.ssh/identity type -1
debug1: identity file /Users/tim/.ssh/id_rsa type -1
debug1: identity file /Users/tim/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version Sun_SSH_1.5
debug1: no match: Sun_SSH_1.5
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.2
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '192.168.0.200' is known and matches the RSA host key.
debug1: Found key in /Users/tim/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/tim/.ssh/identity
debug1: Trying private key: /Users/tim/.ssh/id_rsa
debug1: Trying private key: /Users/tim/.ssh/id_dsa
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending command: scp -v -r -t /tank
Sending file modes: C0644 1073741824 deleteme.1gb
deleteme.1gb 100% 1024MB 56.9MB/s 00:18
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 1075840384, received 422112 bytes, in 61.9 seconds
Bytes per second: sent 17383530.0, received 6820.5
debug1: Exit status 0Password:
Last login: Tue May 10 06:31:05 2011 from 192.168.0.3
Oracle Corporation SunOS 5.11 snv_151a November 2010
tim@banshee:~$ cd /tank
tim@banshee:/tank$ rm deleteme.1gb
tim@banshee:/tank$ time dd if=/dev/zero of=./deleteme.1gb bs=1M count=1024
Code:
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.8073 s, 99.4 MB/s
real 0m10.812s
user 0m0.003s
sys 0m0.964s
Code:
Executing: program /usr/bin/ssh host 192.168.0.3, user (unspecified), command scp -v -r -t /Users/tim
Sun_SSH_1.5, SSH protocols 1.5/2.0, OpenSSL 0x009080ff
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: ssh_connect: needpriv 0
debug1: Connecting to 192.168.0.3 [192.168.0.3] port 22.
debug1: Connection established.
debug1: identity file /home/tim/.ssh/identity type -1
debug1: identity file /home/tim/.ssh/id_rsa type -1
debug1: identity file /home/tim/.ssh/id_dsa type -1
debug1: Logging to host: 192.168.0.3
debug1: Local user: tim Remote user: tim
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.2
debug1: match: OpenSSH_5.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.5
debug1: use_engine is 'yes'
debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
debug1: pkcs11 engine initialization complete
debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible
)
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: Peer sent proposed langtags, ctos:
debug1: Peer sent proposed langtags, stoc:
debug1: We proposed langtags, ctos: en-US
debug1: We proposed langtags, stoc: en-US
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: dh_gen_key: priv key bits set: 129/256
debug1: bits set: 980/2048
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '192.168.0.3' is known and matches the RSA host key.
debug1: Found key in /home/tim/.ssh/known_hosts:1
debug1: bits set: 1001/2048
debug1: ssh_rsa_verify: signature correct
debug1: set_newkeys: setting new keys for 'out' mode
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: set_newkeys: setting new keys for 'in' mode
debug1: SSH2_MSG_NEWKEYS received
debug1: done: ssh_kex2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug1: got SSH2_MSG_SERVICE_ACCEPT
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /home/tim/.ssh/identity
debug1: Trying private key: /home/tim/.ssh/id_rsa
debug1: Trying private key: /home/tim/.ssh/id_dsa
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive)
debug1: fd 5 setting O_NONBLOCK
debug1: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug1: send channel open 0
debug1: Entering interactive session.
debug1: ssh_session2_setup: id 0
debug1: channel request 0: env
debug1: Sending command: scp -v -r -t /Users/tim
debug1: channel request 0: exec
debug1: channel 0: open confirm rwindow 0 rmax 32768
Sending file modes: C0644 1073741824 deleteme.1gb
Sink: C0644 1073741824 deleteme.1gb
deleteme.1gb 100% |****************************************| 1024 MB 00:18
debug1: channel 0: read<=0 rfd 5 len 0
debug1: channel 0: read failed
debug1: channel 0: close_read
debug1: channel 0: input open -> drain
debug1: channel 0: ibuf empty
debug1: channel 0: send eof
debug1: channel 0: input drain -> closed
debug1: channel 0: rcvd eof
debug1: channel 0: output open -> drain
debug1: channel 0: obuf empty
debug1: channel 0: close_write
debug1: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: rcvd close
debug1: channel 0: almost dead
debug1: channel 0: gc: notify user
debug1: channel 0: gc: user detached
debug1: channel 0: send close
debug1: channel 0: is dead
debug1: channel 0: garbage collecting
debug1: channel_free: channel 0: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 18.3 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0Edit: decided to test a copy from one pool to another. Here's the controller setup:
tim@banshee:/tank$ zpool status
Code:
pool: backup
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
c9t1d0 ONLINE 0 0 0
c9t5d0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c8t0d0s0 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
c10t0d0 ONLINE 0 0 0
c10t1d0 ONLINE 0 0 0
c10t2d0 ONLINE 0 0 0
c10t4d0 ONLINE 0 0 0
c10t3d0 ONLINE 0 0 0
c10t5d0 ONLINE 0 0 0
c10t6d0 ONLINE 0 0 0
c10t7d0 ONLINE 0 0 0
c9t2d0 ONLINE 0 0 0
c9t3d0 ONLINE 0 0 0
c9t4d0 ONLINE 0 0 0
errors: No known data errors
Code:
`./deleteme.1gb' -> `/backup/deleteme.1gb'
real 0m13.565s
user 0m0.009s
sys 0m0.982sEdit: I'm wondering... Is the fact that it has to write to drives on two different controllers that share a bus the reason that it's going about half speed? But no, then why would DD work at 100MB/s?
Last edited:
