This is a subject near and dear to my heart as I've had a beowulf at home for several years now. Amazing as it may seem, it is now entirely possible to build a beowulf-class cluster at home, and in fact there are some very useful reasons to think about doing so.
It has become possible for several reasons:
The most difficult single thing about setting up a home beowulf with nodes distributed in several rooms is likely to be the physical network. Houses that are properly wired with category five cabling and RJ45 sockets in all the rooms are relatively rare. However, this is compensated for by three things.
For one, wireless networks are now relatively cheap. A wireless access point is less than $200, and a wireless card for a PC or laptop less than $100. Wireless bandwidth isn't terribly exciting (I tend to get 2 Mbps unless I'm sitting on top of the wireless access point with my laptop) so you won't be able to run particularly fine grained code, but wireless is perfectly adequate for coarse grained code and embarrassingly parallel applications.
For another, having your house wired for a network by professional electricians isn't horribly expensive, either. Depending on how hard they have to work, it should cost on the order of $100 per pull from a reasonable central location (your ``wiring closet'' or wiring shelf, where you will locate your switch, a small patch panel, and a cable and/or DSL phone line for external broadband access) to various rooms. Most of this cost will be labor - it is only a little more expensive to pull 2-4 cat 5 cables than it is to pull one.
Having four rooms wired (with perhap 12 ports, three per room) and a terminating patch panel installed might cost around $500 or even less. It will one day increase the value of your house, and in the meantime it will make your life better in lots of ways. Go for it.
The final way to proceed is, in the best of beowulf traditions, to Do It Yourself. The wiring, that is. This is remarkably simple, and cuts out all the labor costs relative to a professional installation. The hardware costs are a few hundred dollars for cabling and termination and tools, and you can install as many lines as you need whereever you need them.
In my own house (what better testbed?) I have owned and operated a beowulf for years now. I've even written it up in an article for ;login, complete with pictures19.1. I used all three of these methods to network my house. I hired professionals to wire our attic as we had it remodelled into a home office. I stuck a wireless access point up there (and am working on my laptop in my living room via wireless as I write these words). I pulled wires myself into all the downstairs bedrooms, terminating them (and all the professionally installed wires) in an RJ45 patch panel. I pulled a category five phone line from our phone service up to the patch panel for DSL, and a cable up against the possibility that we might one day use cable instead of DSL (or might want to watch TV on a computer, as unlikely as that might seem).
I am therefore in a good position to tell you...