www.pegasos.org :: View topic - Peg2 irq sharing cause network unstability?

www.pegasos.org

Hardware - Peg2 irq sharing cause network unstability?

christianwn - Apr 05, 2006 - 22:20
Post subject: Peg2 irq sharing cause network unstability?

I have been using the peg2 for about a year. I must say it is a great system.

But there is one problem which have troubled me grately, it is network speed / stability.
When I download or upload from my home server(freebsd)(or internet) on a 100mbit line full-duplex auto neg. using a switch, I get at first good speed, 4 to 5 MB/sec and after a little while(random) the speed drops to under 100kb/sec and even sometimes stalling.
I have done every trick I have found on internet to solve this problem but nothing has helped. Short list: SMB/CIFS vs NFS, every sysctl option possible on both linux and freebsd, linux kernel 2.6.9 to 2.6.14, network full-duplex half-duplex auto, tried 2 switch'es, using shielded netcable, tried 3 different nic's also changeing the one on the server (Realtek, 3com, Intel, Via), linux kernel: different Timer frequency, Preemption Model etc...

cat /proc/interrupts

CPU0
1: 10105 i8259 Edge i8042
2: 0 i8259 Edge 82c59 secondary cascade
9: 1192717 i8259 Edge uhci_hcd:usb1, uhci_hcd:usb2, EMU10K1, EMU10K1, VIA686A, eth0, radeon@pci:0000:01:08.0
14: 129204 i8259 Edge ide0
15: 62 i8259 Edge ide1
BAD: 0

------------------------------------------------------------
Look at all the devices on the same irq!

On a "normal" computer I always set every card or onboard device to a different irq, can this be the problem?
Is there a setting in the peg2 bios or linux to change irqs?

There is one last odd thing, I can not get the rtc to work.

Everything else works stabile fast and without problems.

Hope somebody can help me, and 10k thanks in advance!
Christian

Trizt - Apr 06, 2006 - 03:00
Post subject: RE: Peg2 irq sharing cause network unstability?

No, it's not true that on a BIOS machine you get different irq for every card, as there are limited amount of them and you will fill them up quite fast with the built in stuff.

I'm more starting to thinnkng the problem really is the IDE, I did switch from the internal gigabit nic to a pci one and I halfed the "stops" to like half on my NFS. Next I'm thinking of trying a pci IDE for the LVM I share over NFS and see if that removes the rest of the troubles. You can ignore the switch part, you have this on a direct connection too.

ironfist - Apr 06, 2006 - 03:03
Post subject: RE: Peg2 irq sharing cause network unstability?

Do you get the same results with the onboard gigabit port?

lisardman - Apr 06, 2006 - 03:08
Post subject: RE: Peg2 irq sharing cause network unstability?

how are your pegasos attatched to the network

christianwn - Apr 06, 2006 - 23:32
Post subject:

>Do you get the same results with the onboard gigabit port?

I have compiled support for the gigabit port, just now, need more time to see longterm operation. Now it gives me 1,2Mb/sec stable using 75-90% of cpu. After a while, the hole X crashed, but I'm using X 7.0, I think some crashes are to be expected.

One thing to note, is that watching movies from the server seems to really make the network slow. This is one sign for me about some irq conflict because the sound-card, video card and net-card operate at the same time using the same irq. Even alsa recommends setting the soundcard on a singel irq.
But watching movies via the hdd work flawlessly!

>how are your pegasos attatched to the network

First I used the 100mbit onboard port, then I tried using 3 different pci nic's 3com, Intel Pro 100 and Realtek. They gave all the exact same result. Using cables and switch.

About irq on bios pc's. With lots of onboard devices it gets harder and harder to give every device a singel irq. But reading the mainboard manual, some mainboards have still got some agp and pci ports that can get a singel unshared irq. Not all devices get an irq, I know.
But I have had lots of experience with irq conflicts.

About hdd's. I have tried Seagate and Western digital, both with udma on. Only using the onboard controller. This has not affected network performance.

Thank you all so far
Christian

christianwn - Apr 06, 2006 - 23:43
Post subject:

After a fresh restart transfering files from the server via smbfs (not cifs) gives better performance right away 4-5Mb/sec. After surfing the internet, listening to some music etc... the network drops to 1,2Mb/sec.
Cifs gives after a fresh restart 1,2Mb/sec.

Just a detail
Regards Christian

gunne - Apr 07, 2006 - 08:50
Post subject:

christianwn,

> I have compiled support for the gigabit port, just now, need more time to see longterm operation.

------------------------

May I ask for what reason You do compile kernels ?

Why not use the kernel thats included with your choice of distribution ?

dholm - Apr 07, 2006 - 09:19
Post subject:

gunne wrote:

May I ask for what reason You do compile kernels ?

Why not use the kernel thats included with your choice of distribution ?

Maybe the distribution he is using is based on a system where you compile your own kernel or maybe it's LFS?
Besides, there are a lot of options in the kernel which will actually impact your system in a noticeable way. No distribution could supply binaries to tend to all variations of configuration. You should know this considering you claim to be selling Linux systems.

gunne - Apr 07, 2006 - 09:31
Post subject:

dholm,

> You should know this considering you claim to be selling Linux systems.

--------------------------------------------------

Sure, I know this.

My question was directed too christianwn.

dholm - Apr 07, 2006 - 10:38
Post subject:

Then why ask?

gunne - Apr 07, 2006 - 11:37
Post subject:

dholm,

> Then why ask?

----------------------------

I do not understand why You wonder at all.

I ask the question to christianwn, because I wonder what his reason is.

There is no point in guessing for wild as You do in Your post above.

dholm - Apr 07, 2006 - 12:23
Post subject:

It's an educated guess and you know it.
What is wrong with the phpBB system of quoting?

ironfist - Apr 07, 2006 - 12:55
Post subject:

I moved the topic to the correct forum.

ChristianWn: Why do you have VIA686A on IRQ9? AFAIK
the Pegasos southbridge is VIA82x?

Trizt - Apr 07, 2006 - 17:10
Post subject:

christianwn wrote:

IMHO this seems to be quite slow speeds, think my A1200 with a PCMCIA 10Mbit card did get to a 2/3 of the speed you get on cifs. On NFS I do run somwhere between 20-30Mb/s with a break in transfer once in a while, before it resumes around 18Mb/s.

kozz - Apr 07, 2006 - 19:46
Post subject:

I'm also using NFS on the Pegasos and always get about 10-11 MB/s... yes I only have 100MBit and not 1GBit as trizt but still 4-5 MB/s seems way to slow.

I don't think it is an IRQ issue, all pegs are setup in the same way and they work Smile

Do you get the same speed with an other computer and the same cables etc?

gunne - Apr 07, 2006 - 21:52
Post subject:

dholm,

Yes, I have very high respect for Your knowledge base around Linux and Linux mattters, and You know that as well Smile

Let me try this way...

The open nature of Linux is its strength and can perhaps also in the same time be its weakness. Anyone can practically do what they want with Linux on their own machines, and adapt it to nearly whatever. Anyone can also contribute to Linux, and in this way share improvements and adaptations they made with other people and also with Linux itself. This is one of its strengths.

It is also possible to make the machine malfunction.

People behind Linux-distributions put a lot of effort into testing, and into making the complete distribution to become a stable and good environment to use and work with. Different distributions have their flavours and characteristics, and if You choose Debian, Gentoo, Suse, Ubuntu or which You prefer the people behind work hard with it as their profession.

There is a lot of people around in the world that uses Pegasos in their profession with Linux.

And remember also, we are a family here, so please be kind Smile

To christianwn again.

My question to you above, is just straight, nothing else Smile

Im just curios about Your reason to compile Your own kernel.

Finally about phpBB quoting:

No, I do not know about any fault with it. Please let me/us know if You have found any fault, so we can have a look on it. Thanks in advance !

christianwn - Apr 07, 2006 - 23:57
Post subject:

>May I ask for what reason You do compile kernels ?

I use gentoo, therefore I compile. I use gentoo because the way you can customize your system + speed... I have prior been using freebsd and are used to the ports collection.

>Besides, there are a lot of options in the kernel which will actually impact your system in a noticeable way. No distribution could supply binaries to tend to all variations of configuration. You should know this considering you claim to be selling Linux systems.

I know of meny things that can affect the system via the kernel, I have research all info I could find on internet, I have compiled I think over 70 kernels to try to solve the problem. I have tried using only safe gcc flags also. But saing this, dosen't mean that I can't have misconfigured something...

I have tried yellodog linux, an got the same low speed there also.

>ChristianWn: Why do you have VIA686A on IRQ9? AFAIK
the Pegasos southbridge is VIA82x?

Help me understand this. How can you select proper southbridge in kernel? I have only selected support for the hardware on the peg2 that has options in make menuconfig. The only place in the kernel I can findVIA82x is the ATA device/driver support and it is checked like this [*]. If I exclude this option, the system does not start (kernel boots, but the rest stops).

>I don't think it is an IRQ issue, all pegs are setup in the same way and they work Do you get the same speed with an other computer and the same cables etc?

I have used a new laptop with winxp installed, files where transfered fast and stabile. I have a "video pc (win98)" and it is streaming the files nice.
(The server is a highly optimized freebsd server with 4 hdd's in raid5 with a raid controller.)

Is the mainboard malfunctioning? A bad board? I can put out kernel files, make files, etc on my webserver, if it helps anyone? NFS is much worse than cifs/smbfs.

Thank you to you all
Christian

Trizt - Apr 08, 2006 - 04:32
Post subject:

Here are the VIA specific options that you may want to have in your kernel, much you can just leave unsupported expect the IDE.

CONFIG_BLK_DEV_VIA82CXXX=y
Disable generic IDE support, as that slows down the speed on the harddrive, in my case around 5Mb/s locally on the pegasos (have around the same result on x86 machines too).
I do use the following IO-scheduel:
CONFIG_IOSCHED_DEADLINE=y
CONFIG_DEFAULT_DEADLINE=y
You want to enable DMA in the kernel
CONFIG_IDEDMA_PCI_AUTO=y

CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
Thats what you need for the 100mbit nic

CONFIG_SND_VIA82XX=m
Only needed if you would want to use the onboard sound, as you have EMU10K1, then you most likely want to disable this completly.

CONFIG_SND_VIA82XX_MODEM=m
This is the modem support, but we don't have access to it by default, so there aren't any need to compile this support at all.

If you use preempt, then enable SMP support, don't think the preempt-UP is completly fixed for PPC (dholm, if you know if it been fixed then please tell us).

gunne - Apr 08, 2006 - 09:08
Post subject:

christianwn,

Ok.

My suggestion to You is to remove all PCI-cards from your computer, and then boot up a live cd, for instance mupper, and try the network-speed You then get using the onboard-plugs.

ironfist - Apr 08, 2006 - 17:24
Post subject:

<b>Christian</b>: That's why I wonder why you have VIA686 on
IRQ 9.. PCI IDE-card?

christianwn - Apr 08, 2006 - 19:46
Post subject:

>Christian: That's why I wonder why you have VIA686 on
IRQ 9.. PCI IDE-card?

After some checking, I found that it is the onboard soundcard(VIA).
I am using 3 soundcards.

Regards
Christian

gunne - Apr 11, 2006 - 22:24
Post subject:

christianwn,

You are welcome to tell if You get low transfer speed if You are trying with a clean install, like a live cd, and with additional card removed.

christianwn - Apr 16, 2006 - 09:48
Post subject:

Sorry I haven't replayed for some days, I have been ill.

What I have tried is disableing the onboard via sound card, I have put a smbfs mount option in fstab, I have compiled the nic's as modules in the kernel.

The result and conclusion is:

I get for the first time a stable 4,4 Mbyte/sec transfer using 30-50% of cpu, with the smbfs mount.

Using cifs I still get 1 Mbyte/sec, using 80-100% of cpu.

When I get a little bit better I'm going to try mupper and see what transfer speed and cpu usage I get.

I can not understand what is happening, but I works.

Thank you so far everybody, you have been a great help. Now I can use my network connection.

A big thank you.

Christian

judas - Dec 28, 2006 - 22:42
Post subject:

As I found out in lots of tests, the problem of instability due to irq shareing is stil a problem.

Here*s my setup:

In a PEG2<>PEG2 configuration, I send by gigabit link lots of data. Receiving and sending is done by netcat. The receiver just calculates a checksum , no wrinting to disk.

Here`s my result with kernel 2.6.18:
The mv634_ethxx driver crashed on 35Mb/s receiving rate after 10s. Using the skge-driver (dlink 530T card) the kernel spits out lots of warning about bad hw cksums.
Note: If rate drops below 20Mb/s, crashes are less often.

Now I changed to the latest stable kernel, 2.6.19:
Receiving with the mv634_eth doesnt work, BUT the kernel prints messages about the atkbd.c receiving an unexpected ACK from keyboard !!!

I think, IRQ sharing all devices on IRQ 9 is a very bad thing on the PEG2, as common 386-systems use all IRQs first, before sharing starts ... most drivers are not aware of this sharing .
One more example might be to plug an usb2.0-card (via) chipset into PCI. This makes mupper, niktarix, gentoo lock on booting. Suse and MorphOS still boot without problems.

Guess what IRQ is used ?

Trizt - Dec 29, 2006 - 07:32
Post subject:

For me the onboard audio has cuased me most problems, "disable" it without including drivers for it solves a lot of problems.

Of the onboard devices, it's the Marvell Gigabit network thats the only one thats not on IRQ9.