Thursday, March 30, 2017

The Ridiculous NTFS.3G and big file write

I started noticing this issue recently when I used Ubuntu to copy large files - anything more than 600-700MB. When the copy started, it would start with good speed of about 25-30 MB/S, but then after about 300MB it would slow down and soon go into single digits. Also System Monitor would show kswapd0 popping up and bogging one core so that system performance would take critical hit. This would happen only when the source/destination was ntfs partition.
At first I thought kswapd0 to be the culprit since it was the one hogging the system. But after lots of google searches I found out ntfs.3g driver to be the culprit. What happens is that ntfs.3g driver - whatever version is available in kernel 4.4, allocates huge buffers and does not free them, so if you looked in top, you would see the buff/cache part of memory swelling to ridiculous sizes - in my case when copying 1GB file buff/cache swelled to 3.5GB (WTF!!!) from 500MB.
Anyway after trying a number of solutions below one is the one that worked for me:

while true; do echo 1 > /proc/sys/vm/drop_caches; sleep 5; done


I run this above script in a Sudo shell. It clears the cache every 5secs. It's not the best way, but it gives me responsible system with steady ntfs file copy speed of about 16MBps.

Cheers!!! 

No comments:

Post a Comment