Building a Killer Box for Linux development

It’s about time to test the waters, and see if I can ween myself off the Mac and OS X. It’s been a good ride, but I don’t like the direction that train is going. ¹

To make Linux ² more enticing, I wanted to build a Killer Box for a developer.

The Mac Pro has 1 Xeon processor and 2 GPUs, but what I really want is 2 Xeon processors and 1 GPU. I don’t make movies or sort pictures, I write code and I compile.

There are not many dual-CPU 2011-3 board manufactures on the market for the price range I am interested in. The most readily available are Supermicro and ASUS. Supermicro has apparently a good reputation as a multi-CPU board maker, ASUS has a good reputation as a single-CPU board maker, but multi-CPU board not so much. The new ASUS boards support the M.2 SSDs though. Supermicros do not out of the box, you need to buy an additional PCI-E adaptor. That fills up one of the precious slot.

It took me a couple of weeks to get a stable system. That was partly due to components not being as readily available as advertised, and also due to lots of stupid driver and other software problems. Some more stories for another blog entry.

Mainboard: Supermicro X10DAI

I tried out the ASUS Z10PE-D8 WS, wich looks spiffier, has more features and whose BIOS is more modern. Alas the first board was crap. The second board in hindsight was probably OK, yet it didn’t give me that warm fuzzy feeling, when it also crashed during memtest86+ and then needed multiple CMOS resets to come back to life again.

The Supermicro X10DAi is a bit more oldfashioned but it just works. It can take more RAM, its BIOS comes up faster - just appears more reliable to me. Out of all the boards Supermicro sells, the X10DA line is pretty much the only one that makes sense as a developer Linux workstation. Tweaktown

On the negative side, the boot process of the Linux kernel is fairly slow (17s) with most of the time mysteriously spent during ACPI setup. I blame the BIOS and will try to narrow this down sometime. Windows 10 incidentally boots up very fast with this board…

CPUs: 2x Xeon E5 2660 V3

The Xeon 2660 is a good combination of a ridiculous amount of cores (10 each) with still decent speed and a tolerable wattage suck at a comparably affordable price (for a Xeon).

The E5 2660 V3 can turbo one of its cores to 3.3 GHz, so that would be the single-core max.

Place two on the board and you are running with 20 cores + 20 hyperthreads. How cool is that ? CPUBenchmark

Coolers: 2x Enermax ETS-T40 Cooler

The Enermax ETS T40 are quiet and cool very well. Even in stress testing my CPUs never went over 50 deg. C (out of the case). They are also very easy to install. Only apply about 3/4th of the paste - it is a bit too much otherwise. I didn’t research this part very much though. There may be better stuff out there for less.

The cooler is a bit oversized. With the ASUS board the heatsink fins touch the case lightly. In a dual CPU configuration on the X10DAI the coolers are extremely close, but it’s OK. Legitreviews.

The ArcticCooling Freezer 13 is equally as suitable I think, (I have it in my other boxes) pretty much just as quiet and a bit cheaper. A plus is, that it is not quite as wide as the Enermax. In the end probably a better choice.

RAM: Samsung DDR4 M393A1G40DB0-CPB

I bought 32 GB of Kingston KVR21R15S4K4/32 DDR4 RAM. The first ASUS Z10PE-D8 WS board I had was never completely stable with it. Was it really the RAM or just the board ? It turns out the list of supported memory is extremely (!) small and one probably should take note according to ASUS support. The supported Kingston RAM was actually KVR21R15D4/16, a few characters difference…

Then I used the recommended Samsung RAM. I had the same problems with the ASUS, so I ditched that board. The Samsung RAM works fine now in my Supermicro X10DAI. The Kingston may have worked in the X10DAI fine as well, but that RAM is long gone now. Geizhals

GPU: NVIDIA GTX 660 (will upgrade to GTX 970 soon)

I bought a MSI R9 270x, because I thought it was good enough to do 4K/60Hz (a must) and some limited 3D I just did NOT want to buy NVIDIA again. Register, Torvalds.

Bad mistake, the default “free” driver for Radeon cards, crashes and freezes the whole system badly. With the commercial driver it seemed stable enough, but the UI was very unresponsive. These problems all pretty much disappeared, when I used a comparable NVIDIA card from my Hackintosh. The responsiveness when dragging windows could still be better though, but is usable. Sometimes there is some weird screen tearing when scrolling in Firefox. gtx-660 vs. r9-270x

I will throw more money at the problem and buy a GTX 970, since the R9 270x doesn’t boot with OS X well and can’t replace the GTX 660 in the my Hackintosh, so I need a new card anyways. The ASUS 970 Strix doesn’t need to turn on its fans in desktop mode, sweet. Gonna get this.

I did get it. It doesn’t really help with the crappy Linux graphics performance, but it makes the box VR ready “green” according to the the Steam VR test.

I could also run my Dell UP2414Q Monitor in MST mode (-> 4K/60Hz) with the NVIDIA driver after a LOT of xorg.conf tweaking. Unfortunately that meant, that X11 needed to have Option Xinerama enabled. Option TwinMode always failed for me somehow. Enabling Xinerama disables xrandr. With xrandr disabled, Unity can’t figure out the monitor’s name, thus it can’t set the scale-factor in its dconf dictionary. With a scale-factor of 1 (default) everything is too small and unreadable.

HIDPI and Linux still sucks. Wayland can’t come fast enough.

SSD: Samsung XP941 Series SSD, PCIe M.2 Typ 2280 (NGFF) - 512 GB

It turns out that the ASUS board can’t boot from this kind of SSD card and neither can the Supermicro. There is another M.2 SSD from Plextor, that maybe would have been better. I think the Samsung XP941 beats it in terms of speed though. This is faster than what the Mac Pro offers. Anandtech

For the Supermicro board I had to buy a PCI-E adaptor. I still need to compile benchmarks with this to see if a PCI-E SSDs really makes a difference. It surely I/O benchmarks well.

DELOCK PCIe x4 > 1 x M.2 NGFF

This is cheap and just works as expected. They forgot to package the fixation screw though :( And it doesn’t come with a heatsink … Delock

The missing heatsink might be a problem if you really hit the SSD hard for extended periods of time. (ZFS ? :D)

ZNeK: nah, ZFS is greedy for RAM and CPU, but I/O? COW also means (in theory) that it doesn’t wear off your SSD as much… just think about it.

SSD: Samsung 850 Pro 128GB

Just for booting Linux, because the other SSD can’t function as a boot device. CNET I also have enough space here to add another Operating System, let’s say … Windows 10 preview f.e.

Case: Phanteks Enthoo Pro

The Enthoo Pro looks good, got great reviews, best of all, it can take a SSI-EEB board. There are also some Silverstone cases, that work, but the Raven 5, the smallest one, can only take SSI-ECB. A difference I noticed too late. From Raven 4 upward you should be good to go. Then there is also Chenbro and Intel, but those cases are for people who really want the 24/7 server look and price.

The Phanteks is the nicest PC case I ever had. My Lian Li is also nice, but this is even nicer as it has great cable routing and a hiding place for the boot SSD on the back of the mainboard plate.

One negative point: It is missing a hole that is expected with the SSI-EEB board. So the board is held in place only by 8 screws instead of nine.

Installation of the components was super simple.

Anandtech

Power Supply : Be Quiet Dark Power 650 W

I buy Be Quiet PSUs by habit. Probably a mistake. I think I overpayed here. You can get a better power supply for less. On the plus side all cables were long enough for SSI-EEB, which has a different layout. That was one of my worries. Anandtech

This specific PSU still works fine, but two other Be Quiet PSU crapped out badly on me, so I don’t want to recommend that company any more.

For the money a XFX XTR 650W 80+ modular power supply seems to be a better choice, but I am not 100% sure that the mainboard power cables are long enough.

How much, how fast, how wasteful ?

The expected Geekbench score should be somewhere around 50,000 for Linux 64, alas Geekbench can’t deal with that many cores yet :) Total cost if you shop around like I did, was pretty much exactly €5,000.

A Mac Pro with 12 cores scores around 35,000 and costs €7,699.00. In terms of compile power you have to pay twice as much.

In idle w/o an LCD monitor the whole system is taking up less than 90 W. This is without the case and the Samsung XP941 installed. Noise level is like any other PC.

---

¹ What I don’t like about the Mac anymore

• Tendency towards C++ (too sick) and Swift (too noobish)
• Codesigning, Sandboxing, Programming as a form of bureaucracy
• I need more and more firewall rules to keep a plethora of Apple daemons from calling home.
• Too much Apple nannyish “you shouldn’t be doing this, you shouldn’t be doing that.”
• APIs that are “Here today, deprecated later today”.
• Developer tools, that are not supposed to be extensible. alcatraz.io is aptly named.
• I have to wait for Apple Christmas events to make necessary purchases of devices that may or may not be coming (Mac Mini…) may or may not fit my needs.
• I hate to rewrite all the functionality Apple has been and will be taking/deprecating away.
• The user interface totally lost direction anyway and is now a weird mashup of all styles from 10.0 to 10.10. Yosemite icons look childish and cheap.
• Mac Hardware is unexpandable throwaway stuff nowadays.
• Macs are increasingly just playing catch-up to PCs on the hardware front (again).
• I hate to buy new screwdrivers for every new Mac I buy.
• OS X is like a clam, that is closing up more and more.

² Why Linux Desktop appears viable to me now

• The new Unity, Gnome and KDE Desktops actually don’t look too bad now.
• I play games on Windows anyway.
• I can upgrade when I want and how I want.
• I can easily host stuff I write on this platform on remote servers.
• I can shut off the few daemons that may want to call home.
• With Clang I should be able to get something good going on Linux using C and Objective-C.
• I think I can run pretty much run everything I need on a day to day basis also on Linux.
• Sublime Edit could be a decent tool to develop on Linux.
• Spotify runs decently enough in the browser.
• Thunderbird, Firefox and Google Chrome are all available.
• I can probably make do with OpenOffice.
• Some stuff like scanning or printing is going to work much better in Linux.

---