2019-06-06: Future plans w/ mergerfs

I've added a bunch of new features and enhancements lately and I've got a list of things to investigate. Wanted to lay them out here and see if there are anything users might be looking for.

* readdir_plus support: Will be easy to add. Probably come in 2.29. Should improve performance of `ls -lh` and the like. It combines the request for files with the metadata rather than separate steps.

* increasing readdir message size: The FUSE kernel module currently hardcodes the message size for `readdir` requests to 4KiB even though the max message size is 128KiB or 1MiB (depending on kernel version). I'd like to do some tests to see if increasing the value would make any noticeable difference in performance.

* rename2 support: This new'ish function allows the atomic swapping of files and conditional overwriting. Not widely used but a nice to have.

* writeback cache: Might improve write performance in some cases. Probably with apps which write small amounts of data often rather than larger amounts. Needs to be implemented to test unfortunately.

* reworking threading: regular libfuse2 has an unbounded thread pool which will grow as needed but shrink immediately when not needed which I found to cause issues when many requests came in. I addressed this by making a fixed size pool but that can cause problems if threads block (like what happened recently with using file locks). I'm building a flexible thread pool library which will offer the features I want and should allow for better thread management.

* parallel readdir: an idea to improve readdir performance. Currently mergerfs just iterates serially over the branches. While deduplication still needs to occur it should be faster to issue the reads in parallel and perform the dedup at the end. Once the thread pool library is done I'll be doing tests of this idea.

* New runtime control: using xattr for runtime controls is convenient but xattrs in general can add significant overhead due in particular to POSIX ACLs. With newer posix_acl=true, xattr=noattr, and/or security_capability=false arguments it's possible to improve performance without losing xattr support generally but it might be nice to have a non-xattr method so runtime changes can be made while xattr is disabled.

* statx: It's a new'ish system call which may be faster in querying underlying filesystems for certain information. An initial test seems to indicate no improvement on a normal filesystem (xfs, ext4) but on network filesystems it might show a bigger difference. More testing is needed.

* clone FUSE fd: the FUSE device can dup a connection while still being associated with the original instance. Allows for less contention in the kernel which may improve performance. Some new features in the kernel to help pin requests to CPUs might require this. Worth testing out.

* sets of branches: might make tiered caching easier. The idea would be to have a ordered sets of branches rather than a ordered list of branches. Rather than "/mnt/a:/mnt/b:/mnt/c:/mnt/d" it could be "/mnt/a:/mnt/b|/mnt/c:/mnt/d" where for creates a and b would be checked first based on the policy and then c and d. That could allow more easily setting up tiered caching by checking the first set for a valid location before the second set.