Filesystem  ¶

I disqualified integrated NAS embedding their own OS and proprietary apps to avoid vendor lock-ins. I aimed for a standard and widely adopted software stack.

Silent data corruption being my main concern, I need a robust filesystems. The last time I looked years ago, BTRFS was a big upcoming contender. But it did not seemed to have caught up in popularity since then. So I picked ZFS, for its maturity , flexibility and native data integrity capabilities.

OS  ¶

I need my appliance to run stable for the next few years. I can only achieve that with a healthy distribution. Choosing FreeNAS was a no-brainer : big community, open ecosystem, actively maintained, built upon proven technologies.

Especially since it was announced:

• the unification of FreeNAS and TrueNAS

• rebasing of FreeBSD code to ZFS-on-Linux

This will reduce project fragmentation and a concentration of efforts: a strong future for the platform.

Capacity  ¶

Let’s evaluate my raw needs. I need a place to store:

• Copies of all external HDDs lying around the house

• Some entertainment medias

• Time Machine mount points for macOS backups

Adding all that and rounding up, I end up with at least 8 TB of immediate storage requirement. With a 20% margin, I’ll be happy with 10 TB  .

RAID Array  ¶

To provide resiliency, flexibility and allow for future grow, we will rely on a disk array. How should we structure it?

RAID 5 or RAID-Z1 arrays (simple parity) are no longer safe . With drives getting bigger (> 2 TB), reconstructing an array takes too long. So long the probability of loosing an additional disk is real. Cascading failures is not an option.

RAID 10 is a bullet-proof arrangement. But too expensive for our home usage.

RAID 6 / RAID-Z2 is the best compromise by providing double parity. With 4 drives, we have a similar risk profile as RAID 10: only half the capacity is available because of redundancy, but we can loose up to two disks. The upside is that any additional drive will contribute its full capacity to the array.

Speed, space and safety of different RAID-Z types are explored in the ZFS Raidz Performance, Capacity and Integrity  article.

Should we activate ZFS deduplication feature? No. It consumes too much RAM and CPU for minimal gain. Unless you’re a company looking to archive multiple years of backups for legal requirements, don’t bother with deduplication.

Drives  ¶

The first hardware to decide upon should be the drives. This is the core component of a NAS. Once we have these as a starting point, we can figure out the missing pieces.

Case  ¶

Went fishing for another case. I increased my budget three times and got a SilverStone SST-DS380  :

• Plenty of room to grow with its 8 hot-swappable 3.5” front bays

• Mini-ITX motherboard format

• SFX PSU format (already ordered one)

• Fit in my electrical cabinet (extremely important)

I replaced all its 3 default 120 mm fans by three Noctua NF-S12A PWM . I choose the S-series over the NF-F12 / NF-A12 as the latter targets radiators and I needed to optimize for maximal airflow for case cooling . Also the NF-S12A variant is a little quieter than the NF-S12B one. These new fans added:

• Absolute silence

• Variable and dynamic speed, as we moved from 3-pins to 4-pins PWM

• Reduced RPM: the basic fans were stuck at constant 1100 RPM, the new ones allowed the system to lower them to 400~500 RPM for the same cooling performance

Motherboard  ¶

Requirements:

• Had to be x86-64 for FreeNAS

• Mini-ITX, to reduce as much the form factor

• At least 4 SATA III ports

I stumbled upon the Gigabyte GA-J3455-D3H which fits the bill and comes with a pretty fan-less design, thanks to its built-in Intel Celeron J3455 quad-core (up to 2.3 GHz).

Some reviews here and there called upon its capricious nature. To play it safe I carefully choose RAM that was certified by the manufacturer , and maxed it out at 16 GB.

Realtek NICs are Shit  ¶

I tested the setup for a couple of days and everything worked great.

But I overlooked one issue: Realtek ethernet NICs are shit . It’s written everywhere on discussions boards, even in FreeNAS hardware requirements ( Intel recommended they said…). And FreeNAS maintainers are fed up with support requests  :

we have neither time nor information to debug all zillions of permanently buggy Realtek NICs models

😂

But I ignored the warnings thinking it was meant for enterprise users looking for performances.

Sure enough things went south fast the first time I tried to copy 700 GB over the network. I left the transfer unattended in the night and woke up with a NAS nowhere to be found on the LAN. The Realtek NIC just froze, bringing the whole motherboard with it.

A hard reset later, I found an infinite amount of these in the logs:

(...)
May 11 13:06:36 freenas kernel: re0: link state changed to DOWN
May 11 13:06:36 freenas kernel: re0: link state changed to DOWN
May 11 13:06:40 freenas kernel: re0: link state changed to UP
May 11 13:06:40 freenas kernel: re0: link state changed to UP
May 11 13:18:43 freenas re0: watchdog timeout
May 11 13:18:43 freenas kernel: re0: link state changed to DOWN
May 11 13:18:43 freenas kernel: re0: link state changed to DOWN
May 11 13:18:47 freenas kernel: re0: link state changed to UP
May 11 13:18:47 freenas kernel: re0: link state changed to UP
May 11 13:25:17 freenas re0: watchdog timeout
May 11 13:25:17 freenas kernel: re0: link state changed to DOWN
May 11 13:25:17 freenas kernel: re0: link state changed to DOWN
May 11 13:25:21 freenas kernel: re0: link state changed to UP
May 11 13:25:21 freenas kernel: re0: link state changed to UP
May 11 13:28:46 freenas kernel: re0: link state changed to DOWN
May 11 13:28:46 freenas kernel: re0: link state changed to DOWN
May 11 13:28:49 freenas kernel: re1: link state changed to UP
May 11 13:28:49 freenas kernel: re1: link state changed to UP
(...)

I already wasted too much time with hardware. I decided to just throw money at the problem and get a server-grade motherboard.

Meanwhile, I resorted to one of the worst hack I ever came up with. Not sure it will even have an effect. I added a cronjob-based watchdog checking connectivity and rebooting the machine if the local router can’t be found:

$ ping -c 15 -o -t 600 192.168.0.254 || shutdown -r now

Motherboard, take #2  ¶

So I settled on an ASRock Rack E3C246D2I  :

• Intel NICs, finally!

• 8 total SATA by the way of additional OCuLink port

• ECC memory

• PCIe 3.0 x16

• 3 PWM fan (CPU + front + rear) and thermal sensor header for finer (read quieter) cooling

• Dedicated IPMI/BMC LAN ethernet socket, with non-JAVA UI

• All USB 3.1 ports are Gen2, not Gen1, so no time wasted worrying on which is which

• No unnecessary multimedia ports (no audio, no Display Port, no HDMI)

Cooling  ¶

We’re no longer fan-less, so I searched for the most silent cooling I could ever found. I went straight to Noctua for their reliability, performance and quieteness.

I got the Noctua NH-L9i chromax.black , for LGA 1151 sockets CPUs and 4-pins PWM. Which also perfectly fit the low-profile requirement of my case (< 57 mm).

Surprisingly the black version was cheaper than the regular, brown variant  .

CPU  ¶

In this section I’ve cross-referenced all CPUs supported by the motherboard with the cooler compatibility list  :

Source: cpu-compatibility-asrock-e3c246d2i-noctua-nh-l9i.numbers

All CPUs featured the matrix above shares these specs:

• Intel x86-64, 3rd generation enhanced 14nm++ process

• Socket LGA-1151

• TDP <= 65 W to keep cooler within limits

• SMM, FPU, NX, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AES (encryption boost), AVX, FMA3, and AVX2

• VT-x, VT-d

• Dual-channel memory

But most Coffee Lake-R CPUs are missing from Noctua list (tagged as unreferenced ). I have no intention of overlocking, so we can assume they’re still safe with our cooler, based on TDP and filiation to their Coffee Lake S ancestors.

All in all, low power 35 W models (T suffix) are safe choice.

The no-suffix and F variants are great, and their TDP are perfectly in range within the cooler capabilities. If we find out 65 W to be too much, we can still artificially put hard-limits to the CPU in BIOS.

Note: there’s an elusive candidate in the name of the Core i3-9320, based on the i3-9300 , with slightly better turbo. Unfortunately it is not referenced in the motherboard compatibility list.

My final shortlist was:

1. Core i3-9300 to max out cache and frequency

2. Core i3-9300T same as above with lower frequency but safe thermals

3. Core i3-9100 or i3-9100T as a nice fallback

I ended up with a Core i3-9300T , probably overpriced at the time of purchase, as it was harder and harder to find any of the reference above in retail (seems like Intel prioritizing higher SKUs to increase margins).

I also wasted time trying to get a second-hand Intel Celeron G4900T (Coffee Lake-S, 2 cores, 2.9GHz). It was stolen during transit . By the time I recovered it I decided upon a Core i3-9300T.

Bill of Materials  ¶

Upgrade Path  ¶

This time we’ll try to plan ahead future hardware purchase.

Epilog  ¶

After a number of failures and instability, I finally scrapped all that hardware , and replaced my custom configuration with an official TrueNAS Mini X+ (8x cores 2.2GHz CPU, 32GB ECC DDR4 RAM, 2x 10G Base-T ports) from iXsystems.