About four years ago now the company I work for were investing in some new servers for a project that we were working on and what turned up were quad LGA1366 socket Xeons with support for up to 192Gb of memory. In most cases two sockets were populated with Intel Xeon X5670 CPUs, hex core devices with 12Mb of cache memory. We ran Redhat Enterprise Linux on them and they were, and still are, extremely fast linux servers that could operate as physical boxes in our production environment or virtuals in development.

I wanted one. I still want one. I looked around and noticed that HP were doing a very similar board with two sockets and, crucially, it was packaged up into what looked like a normal PC tower case. And it was very expensive, much too expensive to justify forking out for one.

Fast forward four years and times have changed. You can now pick up a brand new replacement motherboard for an HP Z800 on ebay for £100. So that’s what I did, and here it is.

click for larger

Excitement quickly turned into a daunting realisation that I may have bitten off more than I could chew. The board is massive. It will not fit into any ‘normal’ PC case, not even an EATX tower case. The mounting screws will not mate with any of the ATX holes in a motherboard tray. The large ATX power connector is non-standard. The CPU fan headers are non-standard. There’s a separate power connector for the memory banks with a proprietary connector. The list goes on…

Clearly this is a server motherboard adapted only slightly to fit into HP’s proprietary case with HP’s proprietary power supply and cooling system. Buyers of the Z800 certainly received their money’s worth compared to an anonymous box filled with generic parts.

I’m not one to give up in the face of a technical challenge and besides I’d just forked out a hundred notes on the board so the rest of this article will go through all the steps in detail that you would have to do in order to get one of these beasts up and running yourself. There’s no cheaper way to get 12 cores of Xeon power under your desk.

The BIOS and CPU compatibility

The Z800 board comes in three different revisions, indicated by the AS# number printed on the white sticker located directly below the big black chipset heatsink.

The revisions are indicated by 001, 002 or 003. As you can see from the image this board is an 002 revision. The executive summary to what I’m about to explain is that if you have revision 001 or 002 then you can officially use only Xeon X55xx CPUs. If you have the later 003 revision then you can use either X55xx or X56xx CPUs.

The issue is the BIOS bootblock. It’s physically write-protected within the BIOS and does not get upgraded when you flash the BIOS. So if you have one of the earlier revisions and you flash your BIOS then the X56xx CPUs will be recognised but the bootblock may fail to get that far and you could be presented with a black screen and a POST failure when you power up.

The key is the bootblock date shown in the BIOS System Information screen. A date of 11/10/09 is required to support X56xx CPUs.

Note how I’ve used the word officially a few times back there because the fact is that it might work. Word on the HP forums is that the behaviour of an earlier revision flashed up to the latest BIOS and used with X56xx CPUs ranges from ‘works for me’ to ‘sometimes won’t boot’ to ‘total failure to POST’. The worst issues seem to be with dual-CPU configurations. Later on this article I’ll go through my own setup and experiences.

The case

We need to start with a case for this thing and like I said, even the largest ‘normal’ tower case will be too small. You need an HPTX format case. There aren’t many of these and the one I chose was the Nanoxia Deep Silence 6.

I got it from Quiet PC in the UK and managed to grab a B-grade bargain at £139, that’s a £50 reduction on the full price and I couldn’t tell what made it B-grade because it looks perfect to me. Maybe I have lower standards than most.

I can confirm what the online reviews say when they describe this case as being massive. It is indeed, truly huge. I expect that if it were hollowed out then I could fit my current Fractal Design tower case inside it. It swallows the Z800 motherboard as if it were designed for it. Result.

Modifying the case to hold the motherboard

I mentioned before that the motherboard mounting holes do not match up with the corresponding holes in the case’s motherboard tray. I had to drill and tap new holes for enough screws to hold the board safely with its heavy load of up to two CPUs with large heatsink/fans attached.

This is a fiddly process. Before starting I fitted a small random PCI card into the motherboard and used it to work out exactly where the board needed to be so that the cards lined up with their fixing holes on the side of the case. I used a 2.8mm drill and then tapped the holes out to the correct imperial 6/32 size for motherboard mounting posts.

The hardest part is offering up the motherboard to the tray and accurately marking where to drill. You have to be very accurate or the posts won’t line up with the holes and you only get one chance. A slow, controllable drill that won’t skid is essential and it needs to get into some confined spaces.

Eventually I got all the holes drilled that I thought I could get away with and the board is held securely clear of the base of the case.

This is by far the hardest part of the job and even with all my careful measurement and drilling my expansion cards are slightly bent in their slots due to a couple of millimetres of misalignment. Oh well, at least they’re held tight! If I were to do this again I would probably drill wider diameter holes in the motherboard tray to allow a small amount of adjustment. I’d then use wide washers on either side of the tray, a standoff above the upper washer, a screw through everything and a locking nut at the back of the tray.

The power supply

The original Z800 comes with a power supply engineered by HP to fit the genuine HP case. It is, of course, totally incompatible with a standard PC case so I needed a standard PC power supply that met the requirements of the board.

The HP power supply distributes the main 12V output across 8 different rails, each with a maximum delivery of 18A but with a combined output ceiling of 70A for the 850W unit and 92.5A for the 1110W option. I have no way of knowing how much will actually be drawn by each rail so it’s safest for me to buy a single rail unit with a nice high overall amperage.

I plugged my prospective peripherals into an online PSU amperage calculator and it came out with a recommended 750W supply for dual 130W TDP processors, four 15K SCSI drives, an SSD and an ATI 7970 graphics card. The power supply is so important to the overall stability of the system that I don’t skimp on it but, as usual, I’m determined to get the best deal I can. After much research I bought a SuperFlower Leadex 1000W (80+ gold) supply from Overclockers UK on one of their ‘this week only’ deals for £110. It’s very highly thought of and should be more than enough for this system.

Custom cabling

I mentioned before that some of the board power connectors are non-standard, in fact only the 8-pin EPS connector has a standard pinout and fitting. The main ATX power connector and the memory power connector are custom HP designs. Thankfully the Z800 service manual gives the pinout of these connectors so it’s not hard to make up some custom cables to do the job.

The main ATX cable

The above image is taken from HP’s service manual for the Z800. It shows the pinout of the power cable, taken as you hold the cable and look at the connector. The first issue is the physical cable itself. Each of the pins in the connector is physically keyed with either a square or a slightly rounded socket and there are only 18 pins.

Luckily the order and shape of the pins is identical to a standard 24-pin ATX power cable leaving 6 pins unused at one of the ends. To solve the physical cable issue I bought an ATX power cable extender on ebay for a few pounds and simply sliced off the unwanted pins with a dremel and sanded it to leave a nice finish. The power supply’s standard connector will plug into the unmodified end of the extender and the modified end will go into the motherboard’s socket.

The second issue is the pinout. It’s not the same as the standard ATX pinout at all. To solve this issue I cut the wires of the extension cable around the center and simply remapped them to match the standard by soldering the ends together.

Most of the names in HP’s pinout have an obvious mapping to the ATX standard but there are some that need an explanation. The +12V and V12 lines are all the same and need connecting to the +12V ATX line. The PSU_ID line is an unknown. My guess is that it was designed to allow HP to detect which of the PSU models were fitted and maybe display it somewhere. I’ve no idea whether this should float, or be pulled low or high so I started with the easiest option which was to let it float by simply tying it off with a piece of masking tape.

click for larger

The memory cable

The Z800 has a whopping 12 banks of DDR3 memory available and the designers have, in common with many server motherboards where stability is paramount, opted to give it its own power supply. Again the HP service manual comes to the rescue with the pinout of the connector as you look at the cable.

The connector type is the same as the main ATX power cable and again we are very lucky that the keying of the connector shape matches up to one end of the standard 24 pin ATX cable. I purchased another ATX extender cable and sliced off the part of the connector that was not required.

click for larger

To hook it up to the main power supply I took a Molex ‘Y’ splitter of the type that you often get for free when you buy case accessories and cut off one of the plug ends. Molex cables have GND, 12V and 5V lines which is all I needed to wire up my hacked cable. It doesn’t matter that many of the 12V wire into one from the Molex connector because these lines don’t carry a high current. The unused wires were tied off with insulating tape.

Since the publication of this article a reader sent me a link to a seller on the Chinese TaoBao shopping site who offers a custom made cable set that does everything all my hacking does. If you have access to TaoBao then I’d recommend that you buy one instead of going through the hassle of making one like I did.

The memory

Because this PC is so closely related to a server board it requires registered PC3-10600P 1333MHz DIMMs. A wide range of configuration options is available up to a massive 192Gb when in dual-CPU configuration, explaining the presence of a separate power socket on the board for the memory banks.

After much scouring of the internet I scored 24Gb of original HP memory for about £60.

click for larger

This first lot of 24Gb will do fine for my initial tests. When I come to install the full complement of two Xeon processors then this board’s design means that I can either keep the 24Gb configuration (12Gb will be installed per processor) or I can double up to 48Gb by populating every slot with a 4Gb DIMM.

The original HP design features a large heatsink/fan unit dedicated to just the memory banks. Obviously HP have to design a system that will cope with the full load of 192Gb of memory potentially sharing a case with dual graphics cards and dual 130W TDP processors and a whole raft of hard disks. My more modest target of 24Gb of RAM and a single graphics card means that I’m not going to need a dedicated memory cooling system.

The CPU

At this point it would be a fairly big risk for me to go out and buy the 3.33GHz X5680 hex-core Xeon that I actually want because there’s still a possibility that this whole system will not work and I’ll be left with an expensive CPU to sell on. So for testing purposes I did a bit of bottom-feeding on ebay and scored a brand new quad-core 2.13GHz E5506 for the princely sum of £1.43, less than the cost of a power cable extender. Crazy.

Unloved, the quad-core E5506

The heatsink & fan

HP’s designers must have fallen asleep here because the original HSF is actually rather close to being a standard fit. The only real problem is that it comes with an odd 5-pin fan connector instead of the usual 4-pin. The extra pin is labelled TACH2 and from doing a bit of research it seems that on many boards HP have just grounded the extra pin. The other issue is the price. Even used units are very expensive for what they are so I’m going to go with a standard unit.

The HSF that I bought is the Zalman CNPS10X Optima, costing just £16.50 on Amazon. What’s really great is that no modifications are required to fit it to the board. The board already comes with a backplate ready to accept the screws that go into the fan assembly. All I have to do is screw it down on top of the CPU.

Or so I thought. The screws supplied with the Zalman are not the correct gauge for the thread on the backplate. It’s very close but if you pull on them then they’ll slip out. Not good and potentially fatal to the CPU if it were to let go in operation. I don’t know who to blame here because the thread on the backplate is a standard gauge – normal computer case screws are a perfect fit. Anyway I found some long-ish replacement screws and just fastened it down reasonably tight and as even as I could make it.

I connected the 4-pin Zalman fan connector to the first 4 pins on the CPU fan header, leaving the mysterious ‘TACH2′ pin floating. Hopefully the BIOS doesn’t care about this pin.

Edit: After upgrading to the X5680 Xeon I started to get a POST warning about having a high powered CPU and a low-powered CPU fan so maybe the fifth line is actively driven to a level by HP’s official high-speed fan to indicate that it’s compatible with the 130W Xeon’s. Anyway, there’s nothing low spec about the Zalman so that warning is duly ignored.

The front panel connectors

Every motherboard has a bank of pins for connecting up the power switch, hard disk LED, power LED etc and the Z800 is no different. On this board it’s the bank of pins labelled ‘P5′. Now here we have a problem because the service manual does not list the pinout for this bank.

I scanned the internet for clues, even searching google images to try to find an internal shot of the workstation where I could see the wiring. No luck. The nearest I came was this post on one of HPs forums that related to a similar but not the same workstation board. Just in case that post disappears off the internet, here’s the pinout:

Pin 1 = Hard drive clear plastic lens bottom LED  (which shows HD activity)… white wire
Pin 2 = Frosted lens top LED  (solid or blinking green LED)  (one wire of front panel dual-color LED)… red wire
Pin 3 = Hard drive clear plastic lens bottom LED (may be ground end)… green wire
Pin 4 = Frosted lens top LED Red (solid orange color LED)  (other wire of front panel dual-color LED)… black wire
Pin 5 = No wire attached in xw6400.  May be ground.
Pin 6 = Front panel switch for power on/off… thick white wire attached..  May be the positive.
Pin 7 = No wire attached in xw6400
Pin 8 = Front panel switch for power on/off… thick white wire attached.  May be the ground.
Pin 9 = No wire attached in my xw6400.  May be +5V
Pin 10 = Key (no pin on motherboard header; plastic filling the #10 hole in the receptacle)
Pin 11 = No wire attached in xw6400.  Has gray wire of ambient air temperature thermisitor attached in xw6600 cable
Pin 12 = No wire attached in xw6400.  Has brown wire ofambient air temperature thermisitor attached  in xw6600 cable
Pin 13 = Orange wire, but not used in my xw6400.  For hood sensor connector.  May be ground.
Pin 14 = Orange wire, but not used in my xw6400.  For hood sensor connector.
Pin 15 = Key (no pin on the motherboard; plastic filling the #15 hole in the receptacle)
Pin 16 = Blue wire, but not used in my xw6400.  For hood sensor connector.
Pin 17 = Internal speaker +… yellow wire
Pin 18 = Internal speaker - … yellow wire

In the absence of anything better to go on I decided to give that pinout a try. At the very least I need the power switch to work, everything else is a ‘nice to have’.

That’s what it looks like with the power switch, power LED and hard disk LED connectors in place. I’ve heard from someone else who’s working on one of these boards that the pinout for the internal speaker connector is also correct.

Testing

Now all my modifications are done, it’s time to put it all together in the case and do a quick test. You only need memory, a CPU and a graphics card to do a POST test on the board so I retrieved an old PCIe graphics card from storage in the garage and hooked it up to an old monitor for testing.

click for larger

The board actually looks quite normal inside that case but I can assure you that both the case and the board are very large indeed.

The moment of truth. Switch it on and see what happens.

click for larger

It works!

Everything seems to be OK. The power button and power LED connections are correct, all the memory is detected, the CPU is detected and the CPU fan is spinning. I went straight into the BIOS settings screen and had a look around. I noticed that the BIOS revision was behind the latest so I upgraded it to the latest version via a USB stick.

click for larger

The BIOS is able to to detect the CPU fan speed so I guess it doesn’t need that mystery TACH2 pin. There doesn’t seem to be anything that shows the model of PSU that it thinks is connected so I’ve no idea what, if anything, that PSU_ID pin was for.

Now we’ve got a good system it’s time to get some storage in there.

Storage

The Z800 board comes with a ton of SATA connectors and a SAS RAID controller manages at least some of them. The thing is, they are all SATA-2 3Gb/s and I’m planning on connecting a Samsung EVO 840 240Gb SSD as my primary OS and programs disk.

The Samsung SSD is a SATA-3 6Gb/s device and although in practice nobody can honestly tell the difference between 3Gb/s and 6Gb/s in real-life usage it would be nice to have the primary SSD on a full-speed bus. The answer is a cheap PCIe card with a couple of SATA-3 connectors on it.

I picked this ‘Syba 2 Port SATA 6Gbps PCI-Express x1 2.0 Card’ up on Amazon for less than 20 quid. I picked it because in the reviews there was a Z800 user who’s using it successfully as a boot device which is exactly what I want to do.

They’re using the popular Asmedia ASM1061 chip in a QFN package as the SATA controller. There’s a bit of wonky soldering on a few of the caps there but generally it looks fine. They’re using the same board layout for their SATA and eSATA products which explains the presence of the 0R bridging resistors being used as option selectors.

Working Specification

Now it’s all working I can move in all the rest of my peripheral ‘furniture’ and upgrade the CPU to the final specs. I picked up a Xeon X5680 on ebay for around £200 and thankfully it worked in my 002 revision board without any problems. I actually bought the X5680 before I found out about the issue with the BIOS bootblock and so I do consider myself one of the lucky ones. I think that if I hadn’t used the E5506 to upgrade the BIOS to the latest revision then I would have had a POST failure with the X5680 due to missing microcode for that newer CPU.

click for larger

I’ve got an 003 revision board on the way to me courtesy of ebay so when I do eventually decide to upgrade to dual Xeon’s then I’ll have a board that’s officially supported.

The rather ugly RAID configuration was ported over from my previous system. If anyone were building a new array today with this motherboard then it makes much more sense to use the onboard SAS RAID controller than the Ultra320 expansion card that I’m using.

Under Windows 8.1 with an ambient 20°C temperature all cores are idling at around 29°C. Stress testing with the prime95 application causes the cores to go up to about 58°C after which I got bored watching something that I’m never going to do in real usage and stopped it.

Notes, issues etc.

I did encounter some issues while assembling the full system with all my expansion cards. Here’s what I discovered and how I worked around each issue.

Attempting to install my XFX 7970 graphics card caused the biggest headache. It started OK then as soon as the Windows 8 start screen appeared the monitor signal was lost and I had to power off the computer. Worse, it would not POST afterwards. Worse than worse, restoring the previous graphics card also would not POST. I honestly thought I’d fried the entire motherboard.

However, some minutes later it all started working again with the previous card. I’m guessing that the XFX card somehow triggered a resettable overcurrent or heat-related fuse and a few minutes later the fuse reset. I’m now using a cheap Nvidia 210 single-slot card because I don’t play games so a power-hungry hot-running gaming card would do nothing except block useful expansion slots.

Installing the PERC 4e/DC RAID card caused the BIOS to complain on startup about being out of memory for option ROMs. This is another well documented complaint. The solution was to disable some of the unwanted onboard peripherals, something that also speeded up the POST process.

Samsung’s so called RAPID mode file system filter driver, which is actually just a write-back cache, causes random blue screens as-of version 4.4. Searching the internet shows that this is a common issue and since the SSD is plenty fast enough without dodgy drivers upsetting the stability I simply disabled RAPID mode.

Windows 8.1′s fast boot mode would sometimes cause the computer to do an immediate power-off as soon as the boot started. Fast-boot works a bit like hibernate mode where the previous state is restored from a disk file on boot. I couldn’t narrow this one down to any particular cause so I just disabled fast boot.

The PERC 4e/DC RAID BIOS screen uses the F10 key in some places to operate its menu system. It’s a well-documented problem that if F10 is used by your computer to enter its BIOS then F10 will not work inside option-ROM BIOS screens. The result was that I could not configure my RAID array. The solution was to use the megarc utility from LSI to configure the RAID options. For example, from an Administrator command prompt the megarc -newcfg -a0 -R5[1:1,1:2,1:3,1:4] -strpsz32 DIO WB RAA command would create a RAID-5 array using the SCSI disks on adaptor 0, channel 1, with SCSI ids of [1,2,3,4] using DirectIO, write-back and read-ahead-adaptive options.

You do get some warnings from the BIOS at the final stage of the peripheral initialisation.

These warnings are just for informational purposes and do not affect the operation of the system. They can be ignored.

That’s all folks

I’m done for now, at least for a few months until I can justify adding another X5680 and swapping out the 002 motherboard for an 003 to support it. I can report that the X5680 is very much faster at compiling my projects than the QX9650 that it replaced. I’m happy with the outcome.

Feel free to leave a comment below, or maybe you’re building a Z800-based system yourself and would like to stop by the forum to share your experiences.