Linux – Adjust linux disk flush intervals to avoid blocking user processes

cachelinuxnetworkingperformance-tuningrtorrent

I'm downloading files on a sd card mounted on a raspberry pi using rtorrent, at around 6MB/s (bandwidth cap). The user process doesn't flush the data itself, and the interval at which the disk cache is flushed depends on sysctl values.

Using iostat and nload I can see the network bandwidth and disk write profiles:

Disk:

20M/s .....x.....x....xx...
      .....x....xx....xx...
      ....xx....xx....xx...
0M/s  ....xx....xx....xx...

Network:

5M/s  ..xxx....x.. x.....x.
      .xxxx...xx..xxx...xxx
0M/s  xxxxx...xxx.xxx...xxx

Apparently the IO writer is always paused during flushing. It's a 4-core CPU with 3 cores permanently idle, and top says 100% iowait during flushing.

If I do while true; do sync; sleep 1; done while downloading, the disk flush is smooth at 6MB every second. And the downloader process isn't choked so bandwidth is constant.

I can get the same effect with:

/proc/sys/vm/dirty_writeback_centisecs:
how often disk cache flushing kicks in – unit is 100's of a sec; default: 500

and /proc/sys/vm/dirty_expire_centisecs:
how old a page must be to be eligible for flushing in 100's of a sec; default: 3000

Setting these to 100 and 200 respectively achieves the same even disk flush profile and constant download bandwidth.

Is there a way to make this tuning automatic, i.e. flush as often as possible to avoid having to block the writer process ? For instance, if download speed is 1MB/s then flushing every 5sec would probably be enough.

Best Answer

  • The linux kernel checks ervery dirty_writeback_centisecs if you have dirty pages older than dirty_expire_centisecs. With your default settings it checks every 5 seconds for dirty pages older than 30 seconds.

    The kernel also has a dirty memory limit. When this limit is hit, it blocks every writters process in order to avoid having too much dirty pages in RAM.

    What you should tune is

    • vm.dirty_background_ratio : the amount of system memory that can be filled with dirty pages before the kernel starts flushing to disk in the background.
    • vm.dirty_ratio : the maximum system memory that can be filled with dirty pages. Then the kernel blocks every I/O requests until everything is flushed to disk.

    In order not to get this behavior you need to tell the kernel it should hold at least 30s*6MB/s=180MB. You also need to take into account the slow write speed of the Raspberry Pi because during the time needed for flushing to disk, more dirty pages will be written to RAM.

    This may represents too much RAM for the Raspberry Pi, so you may need to lower dirty_expire_centisecs. Be aware that using to small values could hurt system responsivness because you'll almost get synchronous I/O.

    Here is a more complete explanation of how this works : https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

  • Related Question