First about the setup: a bit older hardware - P2B mobo, P2 333, 192 mb ram, compex pci nics (dmesg attached at the end). 2 boxes - 1 obsd with old 20gb disk, and one ftp client on w2k with even older 3gb hdd.
OBSD box is pretty much vanilla 3.8 with vsftpd accepting passive connections at 40000-40050. The other w2k machine acts as a ftp client.
Used pf setup (*very simplified* to only show the issue) is as follows (75 is windows machine):
#-----------
test="192.168.100.75" int_if="wb0"
set skip on lo0 scrub in on $int_if
block drop all
pass in on $int_if inet from $test to any keep state pass out on $int_if inet from any to any keep state
pass in on $int_if inet proto tcp from $test to any port \ {21,40000:40050} keep state pass in on $int_if inet proto tcp from $test to any port \ {22} keep state
#-----------
In this scenarion (or with pf disabled), everything works like a charm. I can get throughput pretty much limited by what w2k and its antique hdd can handle (around 16 Mb - 20 Mb).
Similar setup with priq queues works also without any problems:
#-----------
test="192.168.100.75" int_if="wb0"
set skip on lo0 scrub in on $int_if
altq on $int_if bandwidth priq queue {std, ftp, pri, ack}
queue std on $int_if priority 1 priq(default) queue ftp on $int_if priority 2 priq queue pri on $int_if priority 3 priq queue ack on $int_if priority 4 priq
block drop all
pass in on $int_if inet from $test to any keep state queue (std, ack) pass out on $int_if inet from any to any keep state queue (std, ack)
pass in on $int_if inet proto tcp from $test to any port \ {21,40000:40050} keep state queue (ftp, ack) pass in on $int_if inet proto tcp from $test to any port \ {22} keep state queue (pri, ack)
#-----------
But....
If I change altq line and set bandwidth to something smaller - like 10Mb - problems show up. Throughput on ftp drops brutally to around 150 - 250 Kb
Also if I use for example cbq in the following way (regardless if bandwidth is or isn't explicitely set, and to what value):
Transfers on ftp are around the same - 150 - 250 Kb, instead of what they should be - around 4 Mb
I did some experiments - for example setting parent queue to 100mbit and allowing childs to borrow, will fix the problem. Or setting parent to 100mbit and childs to 20/40/20/20 with or without borrowing. But for example, parent at 100, and ftp child set to 10 or 15 won't do much good, besides small increase. "Good" starts when I set the bw to around 20 Mb on ftp child (or when it can borrow this amount), although at that value, the transfer was "choking" between 500Kb and max (more often low than high), with average around 4.8Mb on 30 megabyte file.
Tests with other tools - sftp, scp - (where I can get around 10 Mb normally) have similar stalls, if assigned bw is set to anything "smaller".
I'm out of ideas what could be wrong. Above setups are trival and don't really differ from examples that can be found in man or obsd faq, or anywhere else for that matter. There were no extra cpu/memory load or problems after enabling queing, or any unusual things reported by pfctl in verbose mode.
Of course, upload to bsd machine always worked fine (no queueing applied to incoming packets).
Any idea what could be wrong ? Some strange incompatibility with [network] driver (maybe compex NICs are problematic for some reason - I know they aren't precisely "state of the art" of what a nic could be) ? Maybe some simple mistake on my side I'm not aware of ?
Attached dmesg:
OpenBSD 3.8 (GENERIC) #138: Sat Sep 10 15:41:37 MDT 2005 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel Pentium II ("GenuineIntel" 686-class, 512KB L2 cache) 334 MHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR real mem = 200843264 (196136K) avail mem = 176390144 (172256K) using 2477 buffers containing 10145792 bytes (9908K) of memory mainbus0 (root) bios0 at mainbus0: AT/286+(bd) BIOS, date 08/27/98, BIOS32 rev. 0 @ 0xfadf0 apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown apm0: flags 70102 dobusy 1 doidle 1 pcibios0 at bios0: rev 2.1 @ 0xf0000/0xb268 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfd8f0/144 (7 entries) pcibios0: PCI Exclusive IRQs: 10 11 pcibios0: PCI Interrupt Router at 000:07:0 ("Intel 82371SB ISA" rev 0x00) pcibios0: PCI bus #1 is the last bus bios0: ROM list: 0xc0000/0x8000 cpu0 at mainbus0 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x02 ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x02 pci1 at ppb0 bus 1 pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x02 pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: <FUJITSU MPF3204AT> wd0: 16-sector PIO, LBA, 19546MB, 40031712 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 atapiscsi0 at pciide0 channel 1 drive 1 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <TOSHIBA, CD-ROM XM-6602B, 1017> SCSI0 5/cdrom removable cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2 uhci0 at pci0 dev 7 function 2 "Intel 82371AB USB" rev 0x01pci_intr_map: no mapping for pin D : couldn't map interrupt "Intel 82371AB Power" rev 0x02 at pci0 dev 7 function 3 not configured vga1 at pci0 dev 8 function 0 "S3 86C968-0" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) wb0 at pci0 dev 10 function 0 "Compex RL100-ATX 10/100" rev 0x00: irq 10 address 00:80:48:dd:b7:89 ukphy0 at wb0 phy 0: Generic IEEE 802.3u media interface ukphy0: OUI 0x00601d, model 0x0033, rev. 1 ukphy1 at wb0 phy 1: Generic IEEE 802.3u media interface ukphy1: OUI 0x00601d, model 0x0033, rev. 1 isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 sysbeep0 at pcppi0 npx0 at isa0 port 0xf0/16: using exception 16 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec biomask fbfd netmask fffd ttymask ffff pctr: 686-class user-level performance counters enabled mtrr: Pentium Pro MTRR support dkcsum: wd0 matches BIOS drive 0x80 root on wd0a rootdev=0x0 rrootdev=0x300 rawdev=0x302
On Sat, Apr 29, 2006 at 09:49:18AM +0000, Michal Soltys wrote:
> But....
> If I change altq line and set bandwidth to something smaller - like 10Mb > - problems show up. Throughput on ftp drops brutally to around 150 - 250 Kb
> Also if I use for example cbq in the following way (regardless if > bandwidth is or isn't explicitely set, and to what value):
> Transfers on ftp are around the same - 150 - 250 Kb, instead of what > they should be - around 4 Mb
just to be clear, you're definately not confusing b with B, right?
eg, when altq/cbq is 4Mb, 'pfctl -vvsq' is saying Kb/s and not Mb/s ?
not to say it is the cause, but in the case of testing/debugging cbq, i'd suggest tossing the priority lines. let them all rock the default of '1'. after you're done and things work as you want, sure, put them back in on the 'pri' queue, or all of them if you fancy.
if i setup the following queues (i have HFSC in my ruleset normally, naming them like this allowed me to not disturb the rest of my rules) :
--- pass on $e inet proto tcp from any to $TESTHOST port 9999 keep state queue( q-yp q-ack ) ---
and then do a 'nc -l 9999 < /dev/zero' on the test-host, and 'nc $TESTHOST 9999 > /dev/null' from another machine on the LAN, i see (take note of the bytecount to tell you how long it ran) :
if you don't have the luxury of using another unix host for the test ( i find the netcat method to be delightful when testing/debugging altq ), you might be able to get away with opening up chargen in inetd.conf on the obsd host and then just telnetting to it from the windows machine. as long as your telnet client doesn't suck eggs, you should be able to get a not-terribly-bad rendition of the netcat /dev/zero||/dev/null stuff.
use that to make sure you have your queues setup right and believe your config -- then move over to seeing how the "real" client app behaves and twiddle as needed.
On Tue, 02 May 2006 09:15:17 +0200, jared r r spiegel <j...@ice-nine.org> wrote:
> just to be clear, you're definately not confusing b with B, right?
> eg, when altq/cbq is 4Mb, 'pfctl -vvsq' is saying Kb/s and not Mb/s ?
> not to say it is the cause, but in the case of testing/debugging cbq, > i'd suggest tossing the priority lines. let them all rock the default > of '1'.
Yes, it's Kb (kilobits). I was extra careful with that. I also tested everything with priority 1, and w/o setting the priority explicitely at all, both in cbq and altq cases. I haven't tried hfsc yet.
In my case, as soon as allowed bandwidth for ftp "cuts" into what transfers I can normaly achieve without queuing (or pf enabled), it all stalls.
> (cut) > and then do a 'nc -l 9999 < /dev/zero' on the test-host, and 'nc > $TESTHOST 9999 > /dev/null' > from another machine on the LAN, i see (take note of the bytecount to > tell you how long > it ran) : > (cut)
Thanks for suggestions. I'll try testing with netcat, in addition to different nics and maybe testing within single host as well, on two loopback interfaces. In a few days though (long weekend here now, heh).
Also this time I had 4 nics - 2 compex nics (the same model as previously - RL100-ATX, running under wb driver) and my two old trusty cards - 3c905 and 3c905b.
First - both 3com cards solved any anomalies I've had - everything worked beautifully and as intended. Unfortunately, changing nics is not an option atm.
But compex cards had similar problems as previously. I tested with ftp, sftp, http, and nc (reading from /dev/zero as well as from a prepared file).
queue std on $int_if bandwidth 2mb cbq(default) queue ftp on $int_if bandwidth 2mb cbq queue pri on $int_if bandwidth 2mb cbq queue ack on $int_if bandwidth 2mb cbq
block drop all
pass in on $int_if inet from $test to any keep state queue (std, ack) pass out on $int_if inet from any to any keep state queue (std, ack)
#12345 - nc ; 40000:40050 - vsftpd pass in on $int_if inet proto tcp from $test to any port \ {21,80,12345,40000:40050} keep state queue (ftp, ack) pass out on $int_if inet proto tcp from any to $test port 12345 \ keep state queue (ftp, ack) pass in on $int_if inet proto tcp from $test to any port \ {22} keep state queue (pri, ack)
#-----------------------------
While using ftp / sftp I've never managed to achieve more than 15-20 kilobytes/s - exactly as in my previous post. NC reading from regular file, to /dev/null on the other machine - in *most* cases behave in the same improper way, but sometimes after simple ifconfig operation (setting ip, up, down) it worked properly. Directly after tests with other programs (ftp, ..) it never worked properly. NC reading from /dev/null worked *usually* well after mentioned ifconfing, but always poorly after ftp, sftp, etc.
Here are few pfctl -vvsq outputs (pf config as above in each case):
In short - sftp, ftp, and others except certain nc uses - always break the throughput. Ifconfig can help - but only with nc and generally only if it reads /dev/zero. But if regular file is used, it's like in ftp & etc. cases (but excpetions happend few times, and I couldn't find a way to repeat the steps leading to them). .... It doesn't really make any sense.
Everything was ran with debug level set to loud - no issues here either.
As noted previously - 3c905 cards had no problems of whatsoever in any case. But this compex model have some (unless queueing is disabled - then it works perfectly too). And in a way that is kinda hard to explain logically. I didn't even manage to find the steps to make it always fail or always work, despite many tests. Also, considering that two different compex cards were tested, it seems unlikely that they were damaged in some unusual way.
Any other suggestions what could be the problem here - wb driver problem (but I guess it's unlikely), incompatiblity of this compex nic with it, some friction between pf and wb driver maybe ?
|
