Don't forget to TRIM

Last year I built a new home fileserver, mostly because I wanted to switch to an all flash pool of storage. Spinning rust is great for cost effectiveness, but it is (relatively) noisy and noticeably slower. The system is not particularly exciting:

• Fractal Design Node 304 case (left over from a previous project),
• Gigabyte B550I Aorus motherboard,
• AMD Ryzen 7 5700G,
• 64GB RAM,
• 512GB Samsung 970 Evo NVME drive as a boot device,
• 4 x 4TB Crucial BX500 SSD drives for data,
• Intel X550 network card (2 x 10Gb ethernet).

Initially the machine was installed with Debian Stable, the four drives set up as a ZFS RAIDZ pool. It worked well, for about 8 months.

After that, things started to slow down. Any significant write IO to the ZFS pool would cause IO stalls, timeouts and generally slowness. I spent quite a bit of time ignoring the problem, then made sporadic efforts to figure it out. None of this produced any useful results, even when I moved the four drives into another machine (also running Debian).

Finally it occurred to me that I hadn’t turned on autotrim for the pool! Getting ZFS to trim the pool manually (using zpool trim) took several hours, but the performance improved immediately afterwards. After that I turned on autotrim (using zpool set) and the IO slowdowns have stayed away for around six months. So if you are seeing IO timeouts and stalls when ZFS flushes transactions, check whether enabling TRIM will speed things along for you.