2.5Gb Network only transferring at 1Gb/s [SOLVED]

[Byteus]

Upgrading my home network to 2.5Gb for media file transfer to HDD's and am encountering unexpected results where single transfers seem to be limited to ~1.1Gb/s or 126MB/s while simultaneous transfers can achieve the full ~2.3Gb/s speed.

Using two PC in this scenario, one a W11Pro/MSI Z690 with I225-V, Cat7 cable to a Sodola 8 port 2.5Gb switch with a 100ft Cat8 run to a Sodola 2.5Gb 5 port switch and then to an old W10Pro/Gigabyte P55 PCIe2.0x1 with a I226-V NIC adapter with Cat8. Storage on this old PC is SATA2.0 with Seagate 18TB drives capable of 275MB/s, I also have a 1TB SSD in this P55 which is what I'm coping to for this test with a single 40GB file. Just coping files between the local HDD->SSD can do about 245MB/s so I don't believe either PCIe2.0 or SATA2 are the limiting factors here, I expected to be over 230MB/s at least. Both systems show 2500/2500 link connection in Windows and switch port and I have tried hard setting that rate on both adapters.

I've ordered a Realtek 8125B to try in case it's just something flaky with the I226-V & 1.1.4.42 drivers however exactly what is limiting single transfers to just over my previous 1Gb/s LAN is the question, any ideas?

 

[SamirD]

I think the first thing I'd confirm is if the bandwidth is actually there using iperf. Once you have confirmed you have 2.5Gb in iperf, then we can see what's going on with the file transfer.

 

[Dreamerbydesign]

I think the first thing I'd confirm is if the bandwidth is actually there using iperf. Once you have confirmed you have 2.5Gb in iperf, then we can see what's going on with the file transfer.

Good point. Just because the connection is set to 2.5 doesn’t mean the bandwidth is. Even with new cables, I’ve seen bandwidth issues.

 

[LukeTbk]

If transferring multiple file allow reaching 2.3 Gb/s it must mean the network is 2.5 (or could it be more caching occuring in that case, it is strange that it would be close to the network limit and I assume here multiple distinct file and sustained).

What program are you using for the file transfert (windows explorer, teracopy, fastcopy, other) ? Is it true in both direction ?

 

[SamirD]

If transferring multiple file allow reaching 2.3 Gb/s it must mean the network is 2.5 (or could it be more caching occuring in that case, it is strange that it would be close to the network limit and I assume here multiple distinct file and sustained).

What program are you using for the file transfert (windows explorer, teracopy, fastcopy, other) ? Is it true in both direction ?

Remember that full-duplex gigabit ethernet is 2Gb, so a properly maxed out 2.5Gb link full duplex would be near 5Gb. (If I'm remember 2.5Gb correctly--if not, someone correct me.)

 

[LukeTbk]

Remember that full-duplex gigabit ethernet is 2Gb, so a properly maxed out 2.5Gb link full duplex would be near 5Gb. (If I'm remember 2.5Gb correctly--if not, someone correct me.)

Oh I thought the simulatnous vs single file transfert speed was copying multiple file from computer A to B vs a single one, not the combined speed of computer A sending file to B + computer B sending file to A at the same time, because then could be just what you said I think and would make sense

 

[toast0]

Gigabyte P55

This guy is pretty old. What does CPU look like when you're transferring? Possibly you're eating through 100% of one core doing a single transfer, and it's single threaded so you need multiple transfers to use your other core(s?).

 

[Byteus]

I think the first thing I'd confirm is if the bandwidth is actually there using iperf. Once you have confirmed you have 2.5Gb in iperf, then we can see what's going on with the file transfer.

Here's Iperf, shows what I'm seeing for single file transfer bandwidth, not 2.5Gb/s, a bit over 1Gb/s.

ID] Interval Transfer Bandwidth
4] 0.00-1.00 sec 148 MBytes 1.24 Gbits/sec
4] 1.00-2.00 sec 143 MBytes 1.20 Gbits/sec
4] 2.00-3.00 sec 149 MBytes 1.25 Gbits/sec
4] 3.00-4.00 sec 144 MBytes 1.21 Gbits/sec
4] 4.00-5.00 sec 149 MBytes 1.25 Gbits/sec
4] 5.00-6.00 sec 142 MBytes 1.19 Gbits/sec
4] 6.00-7.00 sec 149 MBytes 1.25 Gbits/sec
4] 7.00-8.00 sec 143 MBytes 1.20 Gbits/sec
4] 8.00-9.00 sec 149 MBytes 1.25 Gbits/sec
4] 9.00-10.00 sec 141 MBytes 1.18 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - -
ID] Interval Transfer Bandwidth
4] 0.00-10.00 sec 1.42 GBytes 1.22 Gbits/sec sender
4] 0.00-10.00 sec 1.42 GBytes 1.22 Gbits/sec receiver

Strangely running iperf3 in parallel doesn't show what I'm seeing with Lan Test 3.5 in Simultaneous mode where it's 3.2Gb/s Read/Write. It just shows the 1.22Gb/s speed divided with the sum being the same 1.22Gb/s. Just looking at iperf it would appear I have a bandwidth issue with new equipment and cabling.

LanTest 3.5 to IP
*******************************************************************************************************
Time Computer Write Speed Read Speed
------------ --------------- ----------------- -----------------
14:09:00 P55 1.0101061 Gbps 1.1808937 Gbps Successive mode
14:11:33 P55 3.1904796 Gbps 3.2223502 Gbps Simultaneous mode

 

[Byteus]

If transferring multiple file allow reaching 2.3 Gb/s it must mean the network is 2.5 (or could it be more caching occuring in that case, it is strange that it would be close to the network limit and I assume here multiple distinct file and sustained).

What program are you using for the file transfert (windows explorer, teracopy, fastcopy, other) ? Is it true in both direction ?

Not transferring multiple files but using Simultaneous mode on LanTest which I assumed would simulated that. Perhaps not though seeing iperf parallel mode showed the same 1.22Gb/s.

 

[Dreamerbydesign]

Swap that cable out.

 

[Byteus]

This guy is pretty old. What does CPU look like when you're transferring? Possibly you're eating through 100% of one core doing a single transfer, and it's single threaded so you need multiple transfers to use your other core(s?).

Yep it is, but it really should be able to do better that 126MB/s with 2.5Gb, should be more like >230MB/s, it would appear something is bottlenecking bandwidth. PCIe2.0x1 has 500MB/s available and even considering the overhead it should be faster than 126MB/s with a 40GB file copy with Sata2.0 to a 1TB SSD which can do a local copy at 240MB/s. Sole purpose of this computer is just to house 2 18TB HDD to backup my main system and stay offline most of the time.

2.8Ghz i7-860 shows 15% utilization with the 126MB/s 40GB file copy.

 

[Dan_D]

Remember that full-duplex gigabit ethernet is 2Gb, so a properly maxed out 2.5Gb link full duplex would be near 5Gb. (If I'm remember 2.5Gb correctly--if not, someone correct me.)

Full duplex means that the connection is bi-directional. Meaning you can send and receive at the same time. It's still only 1Gb each direction. Not a combined 2Gb.

 

[Dreamerbydesign]

A cat 6 would be just fine. Don’t get too caught up on the rating unless it’s less than a cat 6. Half of that is marketing crap. I’ve seen so many junk cat 7/8 cables.

You could trouble shoot by swapping one cable at a time. Preferably with a new cable that did not come in the same batch as what you are using.

Or you can swap out everything in between at once, and then you’ll know if you had a cable problem. Then you can trouble shoot backwards by swapping in your old (current) cables until you locate the offending cable.

 

[Byteus]

Swap that cable out.

Which cable would that be, there are several, all brand new CAT8's now, I swapped out the CAT7, same results. You would hope even junk CAT8's can handle 2.5Gb considering CAT6 can and I've heard even 5e with short runs. These are the cables I went with.

Got the new RealTek coming thursday, will try that first, if not then move these systems side by side, it may just be this 100ft Cat8 run. I cable for the future, playing in the crawlspace is something I only want to do every 20 years or so.

 

[Dreamerbydesign]

Which cable would that be, there are several, all brand new CAT8's now, I swapped out the CAT7, same results. You would hope even junk CAT8's can handle 2.5Gb considering CAT6 can and I've heard even 5e with short runs. These are the cables I went with.

Got the new RealTek coming thursday, will try that first, if not then move these systems side by side, it may just be this 100ft Cat8 run. I cable for the future, playing in the crawlspace is something I only want to do every 20 years or so.

I totally understand. You’ll have your answer as soon as you swap out cables. Did I miss what brand 2.5 gbe switches you are using?

 

[SamirD]

Here's Iperf, shows what I'm seeing for single file transfer bandwidth, not 2.5Gb/s, a bit over 1Gb/s.

ID] Interval Transfer Bandwidth
4] 0.00-1.00 sec 148 MBytes 1.24 Gbits/sec
4] 1.00-2.00 sec 143 MBytes 1.20 Gbits/sec
4] 2.00-3.00 sec 149 MBytes 1.25 Gbits/sec
4] 3.00-4.00 sec 144 MBytes 1.21 Gbits/sec
4] 4.00-5.00 sec 149 MBytes 1.25 Gbits/sec
4] 5.00-6.00 sec 142 MBytes 1.19 Gbits/sec
4] 6.00-7.00 sec 149 MBytes 1.25 Gbits/sec
4] 7.00-8.00 sec 143 MBytes 1.20 Gbits/sec
4] 8.00-9.00 sec 149 MBytes 1.25 Gbits/sec
4] 9.00-10.00 sec 141 MBytes 1.18 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - -
ID] Interval Transfer Bandwidth
4] 0.00-10.00 sec 1.42 GBytes 1.22 Gbits/sec sender
4] 0.00-10.00 sec 1.42 GBytes 1.22 Gbits/sec receiver

Strangely running iperf3 in parallel doesn't show what I'm seeing with Lan Test 3.5 in Simultaneous mode where it's 3.2Gb/s Read/Write. It just shows the 1.22Gb/s speed divided with the sum being the same 1.22Gb/s. Just looking at iperf it would appear I have a bandwidth issue with new equipment and cabling.

LanTest 3.5 to IP
*******************************************************************************************************
Time Computer Write Speed Read Speed
------------ --------------- ----------------- -----------------
14:09:00 P55 1.0101061 Gbps 1.1808937 Gbps Successive mode
14:11:33 P55 3.1904796 Gbps 3.2223502 Gbps Simultaneous mode

Try the -d switch on iperf and see if it matches lantest.

 

[SamirD]

Full duplex means that the connection is bi-directional. Meaning you can send and receive at the same time. It's still only 1Gb each direction. Not a combined 2Gb.

Yep, my understanding of it as well, which would show near 2Gb if data was sending and receiving at 1Gb in each direction.

 

[SamirD]

You know, if those Sodola switches are some chinese knockoff crap, I wouldn't put it past them that they're the problem. See if you can eliminate them and just connect the two computers directly. If they hit 2.5Gb, it's the switches.

 

[pendragon1]

am i missing it, i dont see anything in the OP that is capable of receiving those speeds(drives)...
everything sata tops out at ~600, you sata2 spinners are not going to getting much over 175(133 on win10 and older)....

 

[Dan_D]

Yep, my understanding of it as well, which would show near 2Gb if data was sending and receiving at 1Gb in each direction.

No.

A 1Gb connection will always show as such. While its a bi-directional connection that can theoretically send and receive the full gigabit each direction, it never happens in reality. Even if it did, that's not how these connections are shown in the OS and that's not how they are rated.

 

[Byteus]

Try the -d switch on iperf and see if it matches lantest.

-d switch just emits debug info which shows no packet loss or errors. The parallel switch just divides total bandwidth and remains capped at 1.22Gb/s.

The Sodola's are some cheap switches no doubt but reviews indicate they work. I did try removing the 5 port and just run both computers off the 8 port with the same results. Lot of these switches, Mokerlink, Nicgigga ect seem to be the same switch rebranded so I went with the cheapest. Initially planned to go with Qnap with 10Gb links but I really only ever use one computer at a time and then running media off HDD's and have no real need for 10Gb now, future for sure when it's not 2X the cost.

This I226-V card is Chinese for sure, it's hard to find a T1 I226-V. The 1.1.4.42 driver from intel's 28.3 package wouldn't even work, the same version driver from station-drivers released a month before the intel package driver did work however. It's why I'm suspicious of this I226-V nic at the moment and replacing it with a 8125B as next step.

 

[Byteus]

File copy with the 8125B did show an improvement, we're up to 180MB/s from 126MB/s with the I226-V. Here's the Iperf3:

[ 4] local 192.168.1.19 port 54705 connected to 192.168.1.6 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 157 MBytes 1.32 Gbits/sec
[ 4] 1.00-2.00 sec 157 MBytes 1.32 Gbits/sec
[ 4] 2.00-3.00 sec 162 MBytes 1.36 Gbits/sec
[ 4] 3.00-4.00 sec 160 MBytes 1.34 Gbits/sec
[ 4] 4.00-5.00 sec 162 MBytes 1.36 Gbits/sec
[ 4] 5.00-6.00 sec 160 MBytes 1.34 Gbits/sec
[ 4] 6.00-7.00 sec 162 MBytes 1.36 Gbits/sec
[ 4] 7.00-8.00 sec 158 MBytes 1.33 Gbits/sec
[ 4] 8.00-9.00 sec 161 MBytes 1.35 Gbits/sec
[ 4] 9.00-10.00 sec 158 MBytes 1.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.56 GBytes 1.34 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.56 GBytes 1.34 Gbits/sec receiver

Slightly better but there still seems to be a bandwidth issue, will try moving it closer to rule out the 100ft CAT8 run. Also going to try putting this I226-V in my X58 hyper-v/pfsense system which has a 40ft CAT8 run between the switches to my z690.

I'm so glad this isn't just plug-n-play, look at all the fun I'm having.

 

[Byteus]

No change moving the P55 system close enough for 2 10ft cables to a single switch, same 1.22Gbs performance. Sigh of relief I won't have to run another 100ft cable...

The mystery unfolds....

8125B moved to my i7-950 X58 system, which is a hyper-V/pfsense/jellyfin/sonaar/radarr/nas where my old HDD's go to die. Windows actually already has the latest driver for the 8125B, while the vendor is hosting a 2 year old driver...wth BroTrends. Easy VM switch swap of the LAN adapter config and it was up and running. Iperf3 was a welcomed sight...phew!


Connecting to host 192.168.1.6, port 5201
[ 4] local 192.168.1.19 port 12014 connected to 192.168.1.6 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 282 MBytes 2.37 Gbits/sec
[ 4] 1.00-2.00 sec 281 MBytes 2.36 Gbits/sec
[ 4] 2.00-3.00 sec 282 MBytes 2.37 Gbits/sec
[ 4] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec
[ 4] 4.00-5.00 sec 278 MBytes 2.33 Gbits/sec
[ 4] 5.00-6.00 sec 256 MBytes 2.15 Gbits/sec
[ 4] 6.00-7.00 sec 247 MBytes 2.07 Gbits/sec
[ 4] 7.00-8.00 sec 248 MBytes 2.08 Gbits/sec
[ 4] 8.00-9.00 sec 246 MBytes 2.06 Gbits/sec
[ 4] 9.00-10.00 sec 248 MBytes 2.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 2.59 GBytes 2.22 Gbits/sec sender
[ 4] 0.00-10.00 sec 2.59 GBytes 2.22 Gbits/sec receiver

That a boy!!! Good boy! Good boy!

So, the issue appears to be Nic performance on this P55 system, strange considering the system can do 240MB/s with a HDD>SSD file copy. There's no other cards in this system other than an old Nvidia GT710. I did try the other PCIe x16 slot but the card wouldn't even post there only in the x1 slot did it post. The X58 on the other hand has a GTX970 and Syba SATA3 2xSSD caddy, wouldn't work in the x1 slot but did work in a x16 slot. I guess I'll have to live with junk being junk till ArrowLake releases and I'm ready to look for a new system at which point the z690 will take over for the X58. Either way this mystery appears to be solved.

I have to add file copy performance does not appear to be a great improvement. I use to get a substained 113MB/s at 1Gb/s throughout the copy of large files. With 2.5gb it will start at 140MB/s peak up to 230MB/s for about 6 seconds and then drop like a rock to a sustained 70ishMB/s. It's as if the card overheats and or cache no longer benefits. Just have to chalk that up to slow systems that might actually perform better at 1Gb/s.

Update: File copy performance does appear to be cache related, I do have the large server cache set on this 24GB W10pro system. I've actually got up to 280Mb/s to sustain for 3/4 of a transfer only to drop towards the end to the 70Mb/s. This is from old 2TB Seagate drives off the X58's Sata2 controller.