Posts

Showing posts with the label storage

When Free Disk Space Lies: How I Solved a Silent NTFS Filesystem Exhaustion Outage and Built an AI-Powered Early Warning System

Image
The Incident What Happened In December 2025, one of our production FTP servers went down hard. The server — running Windows Server 2016 — was responsible for ingesting high volumes of files around the clock. At some point during a peak processing window, all write operations on the D: drive failed simultaneously. The symptom was deceptively simple: the operating system reported 0 bytes available on the D: drive. Applications failed, file transfers aborted, and alerts fired across the board. The first instinct of anyone on the team was obvious: the disk is full. Except it wasn't.  Outage was caused by something most infrastructure engineers have never encountered — and that is exactly what makes it dangerous. NTFS metadata exhaustion is invisible to standard monitoring, looks exactly like a disk space problem, and cannot be solved by adding disk space.   The Investigation When we RDP'd into the server and opened Windows Explorer, the D: drive showed over 500GB of free s...