All,
Here's one for you. I'm trying to transfer a large file between two houses both of which are on GF. The source house has 1G service, and the destination house has 500M service. I can't get a better transfer rate than 1.1MB/s. Reversing the direction, I can achieve a much better 14.2MB/s. The entire path inside both houses is wired, so I've already eliminated WiFi from the equation. I am still plugged into the GF supplied network boxes on both ends. I also compared an internet speed test run at the 500M house during the xfer with one while no xfer was running, and the difference was appropriate. The no-xfer speed test reported just over 500M and the xfer 480M, so the 500M service itself isn't degraded.
Thanks,
Here's one for you. I'm trying to transfer a large file between two houses both of which are on GF. The source house has 1G service, and the destination house has 500M service. I can't get a better transfer rate than 1.1MB/s.
To isolate network throughput from other potential bottlenecks, I would start by testing with iperf3 (https://iperf.fr). You’ll run the server at one site and client at either, and alternate, like you’ve done with other tests.
If the speed is less than you expect at either, keep testing within smaller segments (LAN at site 1, LAN at site 2, on the same switch, etc.) and map out the speed on each link to find the bottleneck(s).
If the speed _is_ what you’d expect, then the transfer could be disk I/O limited. While a single 7200 RPM drive can push 80-160 MB/s a fragmented file system, may be much slower than this.
-Aaron
On 2024-06-08 17:22, Hal Duston wrote:
Would compression help?
We do have an iperf3 service running on kclug.org
I presume the LANs are 1G internally.
When we were doing early testing of the 1G service, we found that details if the receiving computer made a big difference. Plenty of RAM and of course an SSD are important.
Protocols can make a difference too, what are you using for the transfer?
All,
When I xfer the same file via sftp from the source server to the source gateway I see a xfer rate of 16.7MB/s. This eliminates the possibilty of the source storage being the culprit. That gets the file directly adjacent to the source GF network box.
I have tried with the destination storage being /dev/null, so no issue there.
The destination network is a consumer grade Linksys WiFi router directly connected to the GF network box with my laptop directly connected to the same WiFi router.
I have tried with sftp and http and received identical results.
The original file is compressed. I have also noticed that a parallel ssh connection between the houses is very laggy during the xfer.
I just noticed that with the two identical NICs (82541PI Gigabit) in the source firewall,
the NIC connected to my internal switch (a Linksys LGS318) shows e1000: enp1s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
while the one connected to the GF Network box shows e1000: enp1s2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
<Several minutes later>
No, that is NOT the cause. They've been that way since 2019. (I checked my kernel logs)
All servers report CPU Idle between 85% to 95% during the xfer so that's also not the issue.
This change in behaviour is fairly recent. I used to see much higher transfer rates with the same HW and SW six months ago and earlier.
Still digging ... ... ...
Thanks,
Hi All,
I'm a newbie to the list so a little hesitant to jump in :), but my Spectrum fibre down here (Peculiar area) exhibits an upload cap about 30 times less than the download bandwidth. Could that be what you are seeing?
I recently installed and switched over to OpenWRT during which I was looking closer at bandwidth performance. I was surprised at some of the poor performances. I found that on on a couple machines I had to turn off TSO offloading (ethtool -K tso off). I found that counter-intuitive.
On 6/8/24 22:09, Hal Duston wrote:
All,
When I xfer the same file via sftp from the source server to the source gateway I see a xfer rate of 16.7MB/s. This eliminates the possibilty of the source storage being the culprit. That gets the file directly adjacent to the source GF network box.
I have tried with the destination storage being /dev/null, so no issue there.
The destination network is a consumer grade Linksys WiFi router directly connected to the GF network box with my laptop directly connected to the same WiFi router.
I have tried with sftp and http and received identical results.
The original file is compressed. I have also noticed that a parallel ssh connection between the houses is very laggy during the xfer.
I just noticed that with the two identical NICs (82541PI Gigabit) in the source firewall,
the NIC connected to my internal switch (a Linksys LGS318) shows e1000: enp1s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
while the one connected to the GF Network box shows e1000: enp1s2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
<Several minutes later>
No, that is NOT the cause. They've been that way since 2019. (I checked my kernel logs)
All servers report CPU Idle between 85% to 95% during the xfer so that's also not the issue.
This change in behaviour is fairly recent. I used to see much higher transfer rates with the same HW and SW six months ago and earlier.
Still digging ... ... ...
Thanks,
Shawn,
Thanks for the ideas/suggestions.
GF advertises the same bandwidth up and down. The behaviour is relatively recent, and does seem to be getting worse.
I've been running the below on bootup since at least July of 2022.
ethtool -K enp1s0 sg off tso off gro off gso off ethtool -K enp1s1 sg off tso off gro off gso off ethtool -K enp1s2 sg off tso off gro off gso off
One other datapoint, don't laugh, but I'm running Linux on a 32 bit machine as my gateway. People who have been around for a long time will not be surprised at this.
Thanks,
Hal,
I see. Symmetric bandwidth must be nice! Still, I do have enough up bandwidth to be able to VPN in and check the front window camera.
Heh heh. My server sports a Q8200 cpu. Core 2 Duo for the win man! Rofl Not sure exactly when that came out. There are probably folks alive today that have never heard of it.
On 6/9/24 14:01, Hal Duston wrote:
Shawn,
Thanks for the ideas/suggestions.
GF advertises the same bandwidth up and down. The behaviour is relatively recent, and does seem to be getting worse.
I've been running the below on bootup since at least July of 2022.
ethtool -K enp1s0 sg off tso off gro off gso off ethtool -K enp1s1 sg off tso off gro off gso off ethtool -K enp1s2 sg off tso off gro off gso off
One other datapoint, don't laugh, but I'm running Linux on a 32 bit machine as my gateway. People who have been around for a long time will not be surprised at this.
Thanks,
Hi Hal,
Any chance you can bisect the issue? Any way to prove out if each router/firewall can separately pass traffic faster? I could set up iperf if you'd like to try to try against another GF box from each of your two ends. I've experienced something similar when changing my router's bridging configuration caused it to drop NAT offloading.
I don't think this is it, but figured I'd mention about Google fiber - optically it's PON, not Ethernet. I think the most commonly deployed ONUs are only capable of 2.5gbps downstream and 1.25gbps upstream, so if two customers are on the same color and on the same OLT, their combined upstream traffic can't exceed 1.25gbps up. The faster speeds are a different color/wavelength and technology. A PON duty cycle meter would tell you the total downstream traffic utilization at one end, but they're not cheap, and you'd need a splitter to measure while your plugged in. A standard optical power meter won't do it. But again, I doubt it's GF over provisioning.
If you're feeling adventurous, there's a local serial console inside the original white fiber jack, they were running kernel 2.6.32 and a minimal user space. There are some optical metrics available, IIRC under / sys/devices/platform/gpon/info/ but it's undocumented, and as part of becoming an adult, I decided I valued my Internet stability more than my curiosity.
-Richard
-
Aaron Spiegel -
Hal Duston -
Jonathan Hutchins -
Richard Allen -
Shawn C. Powell