Re: servers and fans (was Re: RAM sizes)

Craig Sanders via luv-main wrote:
On Sun, May 20, 2018 at 01:17:24AM +1000, Russell Coker wrote:
One of the more dedicated members of this list got a free server system from LUV and uses it as his personal workstation. It has something like 96G of RAM but makes more noise than most people want in the same building they are in. There's no reason why you couldn't design a system with 24 DIMM sockets that doesn't sound like an aircraft taking off, but most people who want so much RAM have a soundproofed server room. Or they could replace the shitty server fans with high-quality low noise fans.
Or for $50 - $100; they could just replace the "shitty server fans" , with a proprietary water-cooler; - removing both the noise and the possibility that the clunky aluminium heat sink, loosens from the CPU eg http://www.msy.com.au/453-water-cooling- regards Rohan McLeod

On Sun, May 20, 2018 at 07:32:03PM +1000, Rohan McLeod wrote:
Or they could replace the shitty server fans with high-quality low noise fans.
Or for $50 - $100; they could just replace the "shitty server fans", with a proprietary water-cooler; - removing both the noise and the possibility that the clunky aluminium heat sink, loosens from the CPU eg http://www.msy.com.au/453-water-cooling-
Trouble is that water coolers may not (and probably won't) fit in most server rack-mount cases. Fans, though, are standard sizes and replacing noisy fans with quieter & better versions of same will fit without a problem. There's a huge difference between a good fan running at ~1500-2000 rpm vs a shitty fan running at 3000-4000 rpm. Also, unfortunately, sometimes water cooling can actually be louder than a good heatsink and fan. It depends on how hard the water pump and radiator fan have to work vs how hard the heat sink fan has to work. and what types of fans are being used, of course. And water cooling isn't always better than a good heatsink & fan, anyway. That's why I decided on the Noctua NH-U14S-TR4-SP3 for my Threadripper CPU, a bloody big metal heat sink - about 1kg of copper heat pipes and shiny alumnium fins - with a very quiet 140mm fan. All the reviews said it worked extremely well, that it greatly out-performed most water coolers for cooling the CPU, and was extremely quiet. Also, to get water cooling that was actually better than it, I'd have to spend at least $200 AUD or so (more than double what the Noctua cost me) - and even then the difference would be about minimal, roughly 5-7 degrees C. The Threadripper is a bit of a special case, though - the CPU is huge, much bigger than the Ryzen 3, 5, or 7 CPUs, and bigger than intel i3, i5, i7 CPUs too. Until very recently, most water cooler blocks didn't completely cover the CPU so were far from optimal in performance. The older style water coolers that weren't designed for the TR4 socket actually perform much worse than the heatsink & fan. If I were into extreme overclocking, I might consider going for high-end water cooling...but I'm not. I might overclock it from 3.4GHhz to 3.8 or maybe 4.0, depending on whether it's stable or not (I expect my machines to run 24/7 for months at a time, minimum. if they can't do that then there's something wrong with them that needs to be fixed immediately). Also depending on whether it's annoyingly noisy or not - more heat causes the fan to spin faster, and more rpm = more noise. The fact that, even without any overclocking, this 16-core/32-thread beast can compile a 4.16 kernel from 'make clean ; make defconfig; make bzImage' in about 53 seconds makes me think that even stock speed is more than adequate, so I'll be happy with that or a modest overclock. PS: IMO given what you can get in consumer-grade computer gear these days, the only reason to bother with a free old server machine is the word "free". "really cheap" might be a good enough reason too. If you're actually paying real money for it, you're better off with good quality consumer stuff. and no need to buy some industrial earmuffs :) craig -- craig sanders <cas@taz.net.au>

Craig Sanders via luv-main wrote:
On Sun, May 20, 2018 at 07:32:03PM +1000, Rohan McLeod wrote:
Or they could replace the shitty server fans with high-quality low noise fans. Or for $50 - $100; they could just replace the "shitty server fans", with a proprietary water-cooler; - removing both the noise and the possibility that the clunky aluminium heat sink, loosens from the CPU eg http://www.msy.com.au/453-water-cooling- Trouble is that water coolers may not (and probably won't) fit in most server rack-mount cases.
Yes rack-mount sounds like no extra space, the radiator unit is almost precisely the size of a standard 120mm case-fan; it mounts directly under it, the pump unit on the other hand is much smaller and lighter than the heat-sink, and fan
Fans, though, are standard sizes and replacing noisy fans with quieter & better versions of same will fit without a problem.
There's a huge difference between a good fan running at ~1500-2000 rpm vs a shitty fan running at 3000-4000 rpm.
Also, unfortunately, sometimes water cooling can actually be louder than a good heatsink and fan.
The noise of the water-cooler is entirely the noise of it's case fan; so quiet if the case fan is quiet
It depends on how hard the water pump and radiator fan have to work vs how hard the heat sink fan has to work. and what types of fans are being used, of course.
And water cooling isn't always better than a good heatsink & fan, anyway.
Can't comment, but on the two ASUS motherboards one with AMD and one with Intel CPU; that I have; the water coolers on both seems to hold the CPU to case temperature differntial to <30deg C; at 100% CPU utilisation
That's why I decided on the Noctua NH-U14S-TR4-SP3 for my Threadripper CPU, a bloody big metal heat sink - about 1kg of copper heat pipes and shiny alumnium fins - with a very quiet 140mm fan. All the reviews said it worked extremely well, that it greatly out-performed most water coolers for cooling the CPU, and was extremely quiet. Also, to get water cooling that was actually better than it, I'd have to spend at least $200 AUD or so (more than double what the Noctua cost me) - and even then the difference would be about minimal, roughly 5-7 degrees C.
That is certainly a beast; with regards to cooling these days I consider cooler means longer life; and greater reliability period ! Water-coolers also pump the heat straight out of the case, so there is no reliance on other case fans and possible high case temperatures I do hope that heat sink is connected through the M/B and not just to it! If the M/B is vertical then it is a cantilever load, with 1000gm acting through the CofG; not to mention the larger volume which the fan and heat-sink are taking up in the case.
The Threadripper is a bit of a special case, though - the CPU is huge, much bigger than the Ryzen 3, 5, or 7 CPUs, and bigger than intel i3, i5, i7 CPUs too. Until very recently, most water cooler blocks didn't completely cover the CPU so were far from optimal in performance. The older style water coolers that weren't designed for the TR4 socket actually perform much worse than the heatsink & fan.
Not unexpected
If I were into extreme overclocking, I might consider going for high-end water cooling...but I'm not. I might overclock it from 3.4GHhz to 3.8 or maybe 4.0, depending on whether it's stable or not (I expect my machines to run 24/7 for months at a time, minimum. if they can't do that then there's something wrong with them that needs to be fixed immediately). Also depending on whether it's annoyingly noisy or not - more heat causes the fan to spin faster, and more rpm = more noise.
Given the expletive deleted fasteners that both AMD and Intel use for attaching their heatsinks and fans; I would use water-coolers even if they were heavier, noisier and the cooling less efficient; and even without over-clocking. The M/B's I tend to use are high end consumer, but second hand.
The fact that, even without any overclocking, this 16-core/32-thread beast can compile a 4.16 kernel from 'make clean ; make defconfig; make bzImage' in about 53 seconds makes me think that even stock speed is more than adequate, so I'll be happy with that or a modest overclock.
Tell me when it's for sale !:-)
PS: IMO given what you can get in consumer-grade computer gear these days, the only reason to bother with a free old server machine is the word "free". "really cheap" might be a good enough reason too. If you're actually paying real money for it, you're better off with good quality consumer stuff. and no need to buy some industrial earmuffs :)
Yes, I look with envy at friends M/B's with BIOS support for PCIe SSD's usually as M2 regards Rohan McLeod

On 20.05.18 23:20, Rohan McLeod via luv-main wrote:
That is certainly a beast; with regards to cooling these days I consider cooler means longer life; and greater reliability period !
As Arrhenius says, each 10°C hotter halves the lifespan. It's not just the CPU, but electrolytic capacitors don't like to cook either. (Even if it says e.g. 105°C on them, they'll last 16 times longer at 65°C, and that's not forever.)
Water-coolers also pump the heat straight out of the case, so there is no reliance on other case fans and possible high case temperatures
Heatpipes are compact, incredibly good thermal conductors, and won't leak water over the mobo. There's an awful lot to like about them. (First encountered them in 1973, but they were expensive then.) ...
Yes, I look with envy at friends M/B's with BIOS support for PCIe SSD's usually as M2
The UdooX86 is around 125 x 85 mm, with an optional M.2 SSD, held on the back with one screw and the socket. It is a very neat little package. I ran debian on it, but have switched to devuan. Its graphics performance is very good, though I haven't tried the three screens. Erik

On Sunday, 20 May 2018 11:20:36 PM AEST Rohan McLeod via luv-main wrote:
Yes, I look with envy at friends M/B's with BIOS support for PCIe SSD's usually as M2
Note that this is only actually useful if it's NVMe. NVMe is very fast. SATA in an M2 form factor gives the same performance as any other SATA SSD but with a little extra expense, some convenience, and some pain if you need to connect it to another system (EG if the motherboard dies and you need to get data off the SSD). -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

On Sun, May 20, 2018 at 11:20:36PM +1000, Rohan McLeod wrote:
Yes rack-mount sounds like no extra space, the radiator unit is almost precisely the size of a standard 120mm case-fan; it mounts directly under it, the pump unit on the other hand is much smaller and lighter than the heat-sink, and fan
yeah, i've got an all-in-one water cooler in one of my machines. I've got another sitting in its box because it's no better or quiter than the CPU cooler I was going to replace it with (bought the wrong one carelessly). BTW, for a water cooler to be significantly better than a heat-sink and fan, you really need at least a 240mm long radiator unit - i.e. 2 x 120mm fans. Some tower cases have 2 spots for fans at the rear where they can fit, and some have mounting points for top-mounted radiators (these often have room for a 360mm radiator and 3 fans)
The noise of the water-cooler is entirely the noise of it's case fan; so quiet if the case fan is quiet
If they have to work hard, the pump noise becomes noticable, just like any other mechanical device. if the water cooler is appropriately sized for the CPU it's cooling, then the pump is nearly silent.
And water cooling isn't always better than a good heatsink & fan, anyway.
Can't comment, but on the two ASUS motherboards one with AMD and one with Intel CPU; that I have; the water coolers on both seems to hold the CPU to case temperature differntial to <30deg C; at 100% CPU utilisation
Here's an interesting comparison by Gamers Nexus of some liquid and air coolers for Threadripper CPUs. https://www.youtube.com/watch?v=FffaOYQpI6k BTW, I mis-remembed the difference shown from this - the ~$95 AUD 140mm noctua heatsink is around 9 degrees C worse than the ~ $250 AUD Enermax 240mm liquid cooler (the cheapest price i've seen for this in AU is about $240 plus postage, on ebay). That's under continuous load, running a benchmark. Idle, the difference is about 2 degrees. My typical usage pattern has the CPU mostly almost-idle with occasional bursts of high load, so I'd expect the average difference to be around 3 or 4 degrees. IMO the liquid cooler isn't worth the extra $150-ish. Both of those are designed for the threadripper CPU. The video also compares a good All-in-One watercooler that ISN'T designed for the threadripper - the kind that AMD included a mounting bracket for with the TR4 CPU - it is about 4 degrees worse than the 140mm Noctua. Entirely due to the fact that the cooler doesn't completely cover the CPU's heat spreader.
That's why I decided on the Noctua NH-U14S-TR4-SP3 for my Threadripper CPU, a bloody big metal heat sink - about 1kg of copper heat pipes and
That is certainly a beast; with regards to cooling these days I consider cooler means longer life; and greater reliability period ! Water-coolers also pump the heat straight out of the case, so there is no reliance on other case fans and possible high case temperatures
If I were running it under continuous high load, I'd be concerned. But I don't expect that to be the case. Even when I get around to virtualising my Win 7 steam gaming box(*), my usage pattern will be mostly the same, with occasional gaming sessions where it might max out 4 or 8 cores for a few hours. (*) one of my reasons for getting a Threadripper rather than a Ryzen 7. Twice as many CPU cores & threads and, most importantly, 64 PCI-e lanes rather than 20, so I can have 16 lanes just for VGA passthrough to the VM. I plan to retire the win7 box with the VM's OS disk on my nvme drives and the steam games library on a HDD zfs dataset (accessed by the win7 VM over the local VM bridge network). The games library will benefit from the ZFS ARC and L2ARC - at the moment, if I want top disk performance for any given game, I have to manually move it from the win7 box's HDD to its fairly small SSD, which requires moving some other game from the SSD back to the HDD. I won't have to do that tedious crap any more. Mostly, though, I'm virtualising it so That there'll be one less machine in the house taking up space and using power.
I do hope that heat sink is connected through the M/B and not just to it! If the M/B is vertical then it is a cantilever load, with 1000gm acting through the CofG; not to mention the larger volume which the fan and heat-sink are taking up in the case.
yep, it is. Ryzen and Threadripper have a good design for mounting the coolers. it's a vast improvement compared to older CPUs - mounting the fan was always a PITA. Now it's just 4 spring-loaded screws to bolt it down to the motherboard, which has a metal bracket around the CPU connected to a metal plate on the back of the board. The fan needs 165mm clearance in the case. It's not uncommon to get cases these days with 170 or 180mm or even more clearance above the CPU. I chose my new case carefully since I knew I'd be putting a giant heatsink in it, decided on a Fractal Design R5.
The fact that, even without any overclocking, this 16-core/32-thread beast can compile a 4.16 kernel from 'make clean ; make defconfig; make bzImage' in about 53 seconds makes me think that even stock speed is more than adequate, so I'll be happy with that or a modest overclock.
Tell me when it's for sale !:-)
Well, it aint exactly cheap, but I've been saving up for this upgrade since around 2013. Not that I knew exactly what I'd be getting, but I've been putting money aside for my next upgrade for years....and haven't like the cost to performance-increase ratio of anything up until the Ryzen and Threadripper CPUs were released. Upgrading from DDR-3 to DDR-4 is also very expensive. Seems to be a lot of price-fixing between the three main RAM manufacturers, RAM prices are double or more what they were a year or two ago. There was nothing new by AMD worth upgrading to, and just to switch to Intel would have cost over $1000 to get roughly equivalent performance to what I already had. $1500 to $2000 for significantly better performance. I paid more than that for this Threadripper upgrade but I'm getting a proportionally much bigger performance increase for my money, and unlike the Intel stuff, I know that I won't have to throw out and replace my motherboard or my RAM or anything else for my next upgrade. AMD have committed to this socket and mobo platform until at least 2020. And by then we'll have DDR5 RAM and PCI-e 4.0 or 5.0 (rumour has it that PCIe 4.0 might be skipped because ) Intel makes no such committment to upgrade compatibility - quite the opposite, with very few exceptions their SOP is to make make each new CPU model incompatible with their previous motherboards - even when they're using the exact same socket. So you can't just swap the CPU, you have to get a new motherboard too. I don't expect to be upgrading any time soon, but I want the option to do so. I also want socket compatibility for other machines I might upgrade (like my server box), so I can choose which machine gets which CPU whenever I upgrade.
Yes, I look with envy at friends M/B's with BIOS support for PCIe SSD's usually as M2
if you want a reasonably priced upgrade with support for NVME SSDs, look at something like the Ryzen 3 2200G ($139) or Ryzen 5 2400G ($230) with a B350 ($110-$150) or X370 ($160 to $360) motherboard. So, a minimum upgrade cost of $250. plus DDR-4 memory, of course, if your current system is still DDR-3. Those two CPUs include quite decent radeon vega GPUs built-in. The 2200G has 4 CPU cores / 4 threads with Vega 8 GPU. The 2400G has 4 CPU cores & 8 threads with Vega 11 GPU. The built-in GPUs share memory with the CPU of course, so it's best to use fast RAM with it - DDR4-3200 or better. Another option is combining either of those motherboards with a Ryzen 7 2700X ($469). No built-in GPU, but 8 cores / 16 threads at 4.3 Ghz. The 2700X has a much faster clock speed than my Threadripper (3.4Ghz), so is much faster for things that don't need more than 16 threads running at once (i.e. almost everything). I would have been more than happy with it if I didn't need more PCI-e lanes, and saved myself nearly $1000. It would have been OK even for virtualising windows, I'd just have to have the host's GPU and the VM's GPU using 8 lanes each rather than 16. That would be just fine, except on some games that really need the memory bandwidth. craig -- craig sanders <cas@taz.net.au>

On Sunday, 20 May 2018 10:00:38 PM AEST Craig Sanders via luv-main wrote:
PS: IMO given what you can get in consumer-grade computer gear these days, the only reason to bother with a free old server machine is the word "free". "really cheap" might be a good enough reason too. If you're actually paying real money for it, you're better off with good quality consumer stuff. and no need to buy some industrial earmuffs
People who attend the LUV Saturday meetings at InfoXchange can pick through their e-waste pile. At the last meeting I got a system with 6*DDR3 slots which now has 24G of RAM. The CPU is an i7-950 which is a little slower than the i7-2600 of my previous machine (which only has 2 DIMM slots). But it's still pretty decent. I don't often need to compile anything as big as the kernel, the biggest thing I regularly compile is the refpolicy package for Debian which only takes a couple of minutes on i5 class systems. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
participants (4)
-
Craig Sanders
-
Erik Christiansen
-
Rohan McLeod
-
Russell Coker