Linux SMP FAQ <author>David Mentré, <tt/David.Mentre@irisa.fr/ <date>v0.27, 05 may 1998 <abstract> This FAQ review main issues (and I hope solutions) related to SMP configuration under Linux. </abstract> <toc> <sect>Introduction Linux can work on SMP (Symetric Multi-Processors) machines. SMP support has started with the 2.0 family and has been improved in the 2.1 (future 2.2) saga. FAQ maintained by David Mentré (<htmlurl name="David.Mentre@irisa.fr" url="mailto:David.Mentre@irisa.fr">). The latest edition of this FAQ can be found at <htmlurl url="http://www.irisa.fr/prive/mentre/smp-faq/" name="http://www.irisa.fr/prive/mentre/smp-faq/">. If you want to contribute to this FAQ, I would prefer a diff against the <url name="SGML version" url="http://www.irisa.fr/prive/mentre/smp-faq/smp-faq.sgml"> of this document, but any remarks (in plain text) will be greatly appreciated. If you send me an email about this FAQ, please include a tag like <tt>[Linux SMP FAQ]</>. It helps me automatically sort mails (and you will have a faster reply <tt>;)</>). This FAQ is an improvement of a first draft made by <bf>Chris Pirih</bf>. All information contained in this FAQ is provided "as is." All warranties, expressed, implied or statutory, concerning the accuracy of the information of the suitability for any particular use are hereby specifically disclaimed. While every effort has been taken to ensure the accuracy of the information contained in this FAQ, the authors assume(s) no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. <sect> What's new ? <descrip> <tag/v.27, 05 may 1998/ <itemize> <item> New info for the Adaptec and TekRam drivers <item> Micronics W6-LI motherboard works under SMP </itemize> </descrip> <sect>Questions related to any architectures <sect1>Kernel side <enum> <item> <bf>Does Linux support multi-threading? If I start two or more processes, will they be distributed among the available CPUs?</bf> Yes. <item> <bf>What kind of architectures are supported in SMP?</bf> <descrip> <tag/From <bf>Alan Cox</bf>:/ SMP is supported in 2.0 on the hypersparc (SS20, etc.) systems and Intel 486, Pentium or higher machines which are Intel MP1.1/1.4 compliant. <bf>Richard Jelinek</bf> adds: right now, systems have been tested up to 4 CPUs and the MP standard (and so Linux) theoretically allows up to 16 CPUs. SMP support for UltraSparc, SparcServer, Alpha and PowerPC machines is in progress in 2.1.x. <tag/From <bf>Ralf Bächle</bf>:/ MIPS, m68k and ARM does not support SMP; the latter two probly won't ever. That is, I'm going to hack on MIPS-SMP as soon as I get a SMP box ... </descrip> <item> <bf>How do I make a Linux SMP kernel?</bf> Uncomment the <tt/SMP=1/ line in the main Makefile (<tt>/usr/src/linux/Makefile</>). AND enable "RTC support" (from <bf>Robert G. Brown</bf>). Note that inserting RTC support actually doesn't afaik prevent drift, but according to a discussion [Robert G. Brown] remember from a year ago or so it can prevent lockup when the clock is read at boot time. A note from <bf>Richard Jelinek</bf> says also that activating the Enhanced RTC is necessary to get the second CPU working (identified) on some original Intel Mainboards. AND do NOT enable APM! APM and SMP are not compatible, and your system will almost certainly (or at least probably <tt>;)</>) crash under boot if APM is enabled (<bf>Jakob Oestergaard</bf>). <bf>Alan Cox</bf> confirms this : 2.1.x turns APM off for SMP boxes. Basically APM is undefined in the presence of SMP systems, and anything could occur. You must rebuild all your kernel and kernel modules when changing to and from SMP mode. Remember to <tt>make modules</> and <tt>make modules_install</> (from <bf>Alan Cox</bf>). <item> <bf>How do I make a Linux <bf>non</bf>-SMP kernel?</bf> <bf>Comment</bf> the <tt/SMP=1/ line in the Makefile (and not set SMP to 0). You must rebuild all your kernel and kernel modules when changing to and from SMP mode. Remember to <tt>make modules</> and <tt>make modules_install</>. <item> <bf>How can I tell if it worked?</bf> <verb> cat /proc/cpuinfo </verb> Typical output (dual PentiumII): <code> processor : 0 cpu : 686 model : 3 vendor_id : GenuineIntel stepping : 3 fdiv_bug : no hlt_bug : no fpu : yes fpu_exception : yes cpuid : yes wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx bogomips : 267.06 processor : 1 cpu : 686 model : 3 vendor_id : GenuineIntel stepping : 3 fdiv_bug : no hlt_bug : no fpu : yes fpu_exception : yes cpuid : yes wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx bogomips : 267.06 </code> <item> <bf>What is the status of converting the kernel toward finer grained locking and multithreading?</bf> 2.1.x has signal handling, interrupts and some I/O stuff fine grain locked. The rest is gradually migrating. All the scheduling is SMP safe <item> <bf>Does Linux SMP support processor affinity?</bf> No and Yes. There is no way to force a process onto specific CPU's but the linux scheduler has a processor bias for each process, which tends to keep processes tied to a specific CPU. </enum> <sect1>User side <enum> <item> <bf>Do I really need SMP?</bf> If you have to ask, you probably don't. <tt/:)/ <item> <bf>How does one display mutiple cpu performance?</bf> Thanks to <bf>Samuel S. Chessman</bf>, here is some useful utilities: <descrip> <tag/Character based:/ <htmlurl name="http://www.cs.inf.ethz.ch/~rauch/procps.html" url="http://www.cs.inf.ethz.ch/~rauch/procps.html"> Basically, it's procps v1.12.2 (top, ps, et. al.) and some patches to support SMP. <tag/Graphic:/ xosview-1.5.1 supports SMP. And kernels above 2.1.85 (included) have the /proc/stat/cpuX entry. The official homepage for xosview is: <htmlurl url="http://lore.ece.utexas.edu/~bgrayson/xosview.html" name="http://lore.ece.utexas.edu/~bgrayson/xosview.html"> The various forissier's kernel patches are at: <htmlurl url="http://www-isia.cma.fr/~forissie/smp_kernel_patch/" name="http://www-isia.cma.fr/~forissie/smp_kernel_patch/"> </descrip> <item> <bf>How can I program to use two (or more CPUs) ?</bf> Use a kernel-thread library. A good library, the <url name="pthread library made by Xavier Leroy" url="http://pauillac.inria.fr/~xleroy/linuxthreads/">. LinuxThread is now integrated with glibc2 (aka libc6). From <bf>Jakob Oestergaard</bf>: Also consider using MPI. It's the industry standard message passing interface. It doesn't give you shared memory like threads, but it allows you to use your program in a cluster too. <item> <bf>What has changed in the threads packages, linuxthread, etc.</bf> Glibc is the big change. glibc is threadsafe and includes linuxthreads Posix.4 threads by default. Real time signals are also in glibc so POSIX AIO should also be in glibc2.1 (I hope). <item> <bf>How can I enable more than 1 process for my kernel compile?</bf> use: <code> # make [modules|zImage|bzImages] MAKE="make -jX" where X=max number of processes. WARNING: This won't work for "make dep". </code> With a 2.1.x like kernel, see also the file <tt>/usr/src/linux/Documentation/smp</> for specific instruction. BTW, since running multiple compilers allows a machine with sufficient memory to use use the otherwise wasted CPU time during I/O caused delays <tt>make MAKE="make -j 2" -j 2</> actually even helps on uniprocessor boxes (from <bf>Ralf Bächle</bf>). <item> <bf>Why the time given by the <tt>time</> command is false ?</bf> (from <bf>Joel Marchand</bf>) In the 2.0 series, the result given by the <tt>time</> command is false. The sum user+system is right *but* the spreading between user and system time is false. This bug in corrected in 2.1 series. <item> <bf>How will my application perform under SMP?</bf> Look at <url name="SMP Performance of Linux" url="http://www.interlog.com/~mackin/linux-smp.html"> which gives useful hints how to bench a specific machine (from a post made by <bf>Cameron MacKinnon</bf>). <item> <bf>Where can I found more information about parallel programming?</bf> Look at the <url name="Linux Parallel Processing HOWTO" url="http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html"> Lots of useful information can be found at <url name="Parallel Processing using Linux" url="http://yara.ecn.purdue.edu/~pplinux/"> </enum> <sect>Intel architecture specific questions <sect1>Why it doesn't work on my machine? <enum> <item> <bf>Can I use my Cyrix/AMD/non-Intel CPU in SMP?</bf> <bf>Short answer:</bf> no. <bf>Long answer:</bf> Intel claims ownership to the APIC SMP scheme, and unless a company licenses it from Intel they may not use it. There are currently no companies that have done so. (This of course can change in the future) FYI - Both Cyrix and AMD support the non-proprietary OpenPIC SMP standard but currently there are no motherboards that use it. <item> <bf>Why doesn't my old Compaq work?</bf> Put it into MP1.1/1.4 compliant mode. <item> <bf>Why doesnt my ALR work?</bf> From <bf>Robert Hyatt</bf> : ALR Revolution quad-6 seems quite safe, while some older revolution quad machines without P6 processors seem "iffy"... <item> <bf>Why does SMP go so slowly?</bf> or <bf>Why does one CPU show a very low bogomips value while the first one is normal?</bf> From <bf>Alan Cox</bf>: If one of your CPU's is reporting a very low bogomips value the cache is not enabled on it. Your vendor probably provides a buggy BIOS. Get the patch to work around this or better yet send it back and buy a board from a competent supplier. <item> <bf>I've heard IBM machines have problems</bf> Some IBM machines have the MP1.4 bios block in the EBDA, allowed but not supported by <2.1.80. Please update to the right kernel. There is an old 486SLC based IBM SMP box. Linux/SMP requires hardware FPU support. <item> <bf>Is there any advantage of Intel MP 1.4 over 1.1 specification?</bf> Nope (according to Alan <tt/:)/ ), 1.4 is just a stricker specs of 1.1. <item> <bf>Why does the clock drift so rapidly when I run linux SMP?</bf> This is known problem with IRQ handling and long kernel locks in the 2.0 series kernels. Consider upgrading to a later 2.1 kernel (not garenteed to work). From <bf>Jakob Oestergaard</bf>: Or, consider running xntpd. That should keep your clock right on time. (I think that I've heard that enabling RTC in the kernel also fixes the clock drift. It works for me! but I'm not sure whether that's general or I'm just being lucky) <item> <bf>Why are my CPU's numbered 0 and 2 instead of 0 and 1 (or some other odd numbering)?</bf> The CPU number is assigned by the MB manufacturer and doesn't mean anything. Ignore it. <item> <bf>My SMP system is locking up all the time. Black screen, nothing in the logs. Help!</bf> If you're running a 2.0 kernel, consider upgrading to later 2.0.32+ kernels or apply Leonard Zubkoff's deadlock patch. If you still have deadlocks, apply Ingo Molnar's deadlock detection patch and post the results (against your System.map) to linux-smp or linux-kernel. You might also consider running a 2.1 kernel. </enum> <sect1>Possible causes of crash You'll find in this section some <bf>possible</bf> reasons for a crash of an SMP machine (credits are due to <bf>Jakob Oestergaard</bf> for this part). As far as I (david) know, theses problems are Intel specific. <itemize> <item> <bf>Cooling problems</bf> From <bf>Ralf Bächle</bf>: [Related to case size and fans] It's important that the air is flowing. It of course can't where cables etc. are preventing this like in too small cases. On the other side I've seen oversized cases causing big problems. There are some tower cases on the market that actually are worse for cooling than desktops. In short, the right thing is thinking about aerodynamics in the case. Extra cases for hot peripherals are usefull as well. <item> <bf>Bad memory</bf> Don't buy too cheap RAM and don't use mixed RAM modules on a motherboard that is picky about it. Especially Tyan motherboards are known to be picky about RAM speed. <item> <bf>Bad combination of different stepping CPUs</bf> Check <tt>/proc/cpuinfo</> to see that your CPUs are same stepping. <item> <bf>You are running 2.0.33 aren't you ?</bf> If you run 2.0.31 or 2.1.xx you can't be sure that SMP is stable. 2.0.33 is the right kernel for a production system. 2.1.xx kernels perform better, but they are development releases and should NOT be considered stable! <item> <bf>If your system is unstable, then DON'T overclock it!</bf> ...and even if it is stable, DON'T overclock. From <bf>Ralf Bächle</bf>: Overclocking causes very subtile problems. I have a nice example, one of my overclocked old machines misscomputes a couple of pixels of a 640 x 400 fractal. The problem is only visible when comparing them using tools. So better say never, nuncas, jamais, niemals overclock. <item> <bf>2.0.x kernel and fast ethernet</bf> (from <bf>Robert G. Brown</bf>) 2.0.X kernels on high performance fast ethernet systems have significant (and known) problems with a race/deadlock condition in the networking interrupt handler. The solution is to get the latest 100BT development drivers from CESDIS (ones that define SMPCHECK). <item> <bf>A bug in the 440FX chipset</bf> (from <bf>Emil Briggs</bf>) If you had a system using the 440FX chipset then your problem with the lockups was possibly due to a documented errata in the chipset. Here is a reference References: Intel 440FX PCIset 82441FX (PMC) and 82442FX (DBX) Specification Update. pg. 13 <htmlurl name="http://www.intel.com/design/pcisets/specupdt/297654.htm" url="http://www.intel.com/design/pcisets/specupdt/297654.htm"> The problem can be fixed with a BIOS workaround (Or a kernel patch) and in fact David Wragg wrote a patch that's included with Richard Gooch's MTTR patch. For more information and a fix look here: <htmlurl name="http://nemo.physics.ncsu.edu/~briggs/vfix.html" url="http://nemo.physics.ncsu.edu/~briggs/vfix.html"> <item> <bf>DONT run emm386.exe before booting linux SMP</bf> From <bf>Mark Duguid</bf>, dumb rule #1 with W6LI motherboards. <tt>;)</> </itemize> Some hardware is also known to cause problems. This includes: <itemize> <item> <bf>Adaptec SCSI controllers</bf> Latest news (05 may 1998): The latest version of this driver (5.0.13 at this writting) should be SMP safe (both in 2.0 and 2.1 kernels). Doug had verified it through code review and intensive tests (on a dual PII system). This said, some machines still have problems. Doug has kindly given some explanations: <itemize> <item> Doug: The simple fact of the matter is that this version of the aic7xxx driver places a larger demand on DMA transfers than the old driver did, especially for for smaller DMA transactions. So, the point should be made that this version of the driver *is* more demanding on your system than either the 4.1 or 4.1.1 driver, and that certain marginal systems may get hit by this problem (such as my own P133 system is). <item> In 2.1 kernels, they are still problems (with IRQ) with the kernel (2.1.99) and they are not related to this specific driver. People are working on this. </itemize> <item> <bf>3Com 3c905 cards</bf> Some work, some don't. Try disabling busmastering if your system is unstable. </itemize> <sect1>Motherboard specific information Some more specific information can be found with the <url name="survey of SMP motherboards" url="http://styx.phy.vanderbilt.edu/smp/mainboards.html"> <sect2>Motherboards with known problems <itemize> <item> Gigabyte Solution: BIOS upgrade <item> SuperMicro Solution: BIOS upgrade <item> EPoX KP6-LS (<bf>Christopher Allen Wing</bf>, 16 march 1998) It appears to have the same BIOS related BogoMIPS problem as other motherboards. (one CPU only gives about 3 BogoMIPS, the other gives the full amount) All 2.0.x kernels lock up soon after booting, late 2.1.x kernels run slowly but don't seem to lock up. There is no BIOS upgrade available (yet). I wrote the manufacturer but have not received a reply. <item> Tyan Tyan motherboards are known to be picky about RAM speed (<bf>Jakob Oestergaard</bf>). From <bf>Doug Ledford</bf> about the onboard aic-7895 SCSI controller (for which he wrote the driver): "BTW, make sure you have at least BIOS version 1.16 on that Tyan motherboard. The 1.15 and below BIOS versions have a bug relating to IRQ allocation for the 7895 SCSI controller" (submitted by <bf>Szakacsits Szabolcs</bf>). But, as motherboard problems are more close to grayshade than black and white <tt>;)</>, <bf>Richard Jelinek</bf> (from S.u.S.E.) tells that they sell/sold several Tyan Titan Pro (Dual-PPro) Motherboards. They work perfectly with SMP. <item> GA686DLX (<bf>Andrew Crane</bf>) Same BIOS related BogoMIPS problem as other motherboards. Solution from Alan Cox: Congratulations, send the bill for your hair damage to the supplier. You have yet another SMP box with faulty bios. There is a patch for 2.0.x on www.uk.linux.org and there are people working on generic MTRR handling for 2.1.x From <bf>Claus-Justus Heine</bf>: I'm able to boot above MB with 2.1.90 + Ingo's apic + Richard's mtrr patches. <item> MS-6114 More details for this motherboard at <url name="http://www.msi.com.tw/product/6114/6114.htm" url="http://www.msi.com.tw/product/6114/6114.htm"> Solution: BIOS upgrade Somebody experienced solid hangs (nothing in the log files) under constant load of about 5 running processes within less than 12 hours with AMI BIOS v1.1. v1.4b3 runs without problems. </itemize> <sect2>Motherboards with no known problems <itemize> <item> AIR P6NDP and P6NDI (<bf>Leonard N. Zubkoff</bf>) My primary production machine is based on an AIR P6NDP and one of my test machines uses a P6NDI. Both seem to be fine motherboards in my experience. The P6NDI BIOS is a little conservative in its programming of the Natoma chipset for 50ns EDO, but a minor tweek to one register in rc.local took care of that. <item> AIR 54CDP (<bf>Chris Mauritz</bf>) You can also list the following motherboard as working with no problems: AIR 54CDP motherboard / EISA/PCI / onboard aic7870 / dual P120 / Redhat 5.0 (2.0.32 and 2.0.33 kernels) However, <bf>Jon Lewis</bf> adds the following comments: Has Chris tried putting multiple PCI cards in these motherboards...say a SCSI card and a network card? Due to BIOS brain damage, the 54CDP insists on having all PCI devices share a single IRQ. AIR lead me on for 2 weeks and then said "there have been / will be no updates for your board". This means I can't have Tulip network cards and PCI SCSI without hacking the drives (which seems to work, though I was cautioned not to do it). <item> HP XU 6/200 (<bf>Jean-Francois Micouleau</bf>) Works with 2.0 and 2.1 kernels. Some problems under high network load with 2.0.x kernel. Works under 2.1.78 with Ingo Molnar IO-APIC patch. <item> Elitegroup P6FX2-A (<bf>Benedikt Heinen</bf>) Had this mainboard running with ONE PPro on it for several months, and since about a year, it's running without problems with TWO PPro 200MHz. The only crashes this machine ever experienced were before Leonard Zubkoff's deadlock-patches for Linux 2.0.30... <tt>;)</> Elitegroup P6FX2-A / ISA/PCI / Dual PPro200 / Debian "hamm" <item> QDI P6I440LX/DP "Legend IV" (<bf>Tony Kocurko</bf>) The QDI P6I440LX/DP "Legend IV" dual Pentium II motherboard now boots properly with Linux-SMP. As of 0326 hours on 15 April, there is a necessary BIOS upgrade to version 1.6 at the Canadian site (<htmlurl name="http://www.qdi.ca" url="http://www.qdi.ca">). Previously, the Legend IV would bring up the first processor normally, but the <tt>/proc/cpuinfo</> file would show the second processor barely alive. The BIOS update seems to have corrected the problem. <item> ASUS P2L97-DS ASUS P2L97-DS works great under Linux-SMP (from <bf>Richard Jelinek</bf>). <item> Micronics W6-LI From <bf>Mark Garlanger</bf>: 2.0.33 is very stable on the board, and the 2.1.xx kernels are improving. People have also been able to run other OSes including Windows NT(yuck...), FreeBSD-SMP, Rhapsody. </itemize> <sect>Useful pointers <sect1>Various <itemize> <item> <url name="Parallel Processing using Linux" url="http://yara.ecn.purdue.edu/~pplinux/"> <item> <url name="Linux Parallel Processing HOWTO" url="http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html"> <item> <bf>(outdated)</bf> <url name="Linux SMP home page" url="http://www.uk.linux.org/SMP/title.html"> <item> linux-smp mailing list To <bf>subscribe</bf>, send <tt/subscribe linux-smp/ in the message body at <htmlurl url="mailto:majordomo@vger.rutgers.edu" name="majordomo@vger.rutgers.edu"> To <bf>unsubscribe</bf>, send <tt/unsubscribe linux-smp/ in the message body at <htmlurl url="mailto:majordomo@vger.rutgers.edu" name="majordomo@vger.rutgers.edu"> <url name="Linux SMP archives" url="http://www.linuxhq.com/lnxlists/linux-smp/"> <item> <url name="pthread library made by Xavier Leroy" url="http://pauillac.inria.fr/~xleroy/linuxthreads/"> <item> <url name="Survey of SMP motherboards" url="http://styx.phy.vanderbilt.edu/smp/mainboards.html"> <item> <url name="procps" url="http://www.cs.inf.ethz.ch/~rauch/procps.html"> <item> <url name="xosview" url="http://lore.ece.utexas.edu/~bgrayson/xosview.html"> <item> <url name="Pentium Pro Optimized BLAS and FFTs for Intel Linux" url="http://www.cs.utk.edu/~ghenry/distrib/"> <item> <url name="SMP Performance of Linux" url="http://www.interlog.com/~mackin/linux-smp.html"> <item> <url name="Multithreaded programs on linux" url="http://www.e.kth.se/~e94_bek/mthread.html"> </itemize> <sect1>SMP specific patches <itemize> <item> <url name="Forissier kernel patches" url="http://www-isia.cma.fr/~forissie/smp_kernel_patch/"> <item> <url name="Patch for a bug in the 440FX chipset" url="http://nemo.physics.ncsu.edu/~briggs/vfix.html"> <item> <url name="MTRR patch (latest version: 1.9)" url="http://www.atnf.csiro.au/~rgooch/kernel-patches.html"> </itemize> <sect>Glossary <itemize> <item> <bf>SMP</bf> Symetric Multi-Processors <item> <bf>APIC</bf> Advanced Programmable Interrupt Controler <item> <bf>thread</bf> A thread is a processor activity in a process. The same process can have multiple threads. Those threads share the process address space and can therefore share data. <item> <bf>pthread</bf> Posix thread, threads defined by the Posix standard. </itemize> <sect>List of contributors Many thanks to those who help me to maintain this FAQ. <itemize> <item> Emil Briggs <item> Robert G. Brown <item> Samuel S. Chessman <item> Alan Cox <item> Andrew Crane <item> Mark Duguid <item> Jocelyne Erhel <item> Byron Faber <item> Mark Garlanger <item> Claus-Justus Heine <item> Benedikt Heinen <item> Robert Hyatt <item> Richard Jelinek <item> Tony Kocurko <item> Doug Ledford <item> Cameron MacKinnon <item> Joel Marchand <item> Chris Mauritz <item> Jean-Francois Micouleau <item> Jakob Oestergaard <item> Jean-Michel Rouet <item> Ralf Bächle <item> Sumit Roy <item> Szakacsits Szabolcs <item> El Warren <item> Christopher Allen Wing <item> Leonard N. Zubkoff </itemize> </article>