This document describes topics related to BTRFS that are not specific to the tools. Currently covers:
This section describes mount options specific to BTRFS. For the generic mount options please refer to mount(8) manpage. The options are sorted alphabetically (discarding the no prefix).
most mount options apply to the whole filesystem and only options in the first mounted subvolume will take effect. This is due to lack of implementation and may change in the future. This means that (for example) you can't set per-subvolume nodatacow, nodatasum, or compress using mount options. This should eventually be fixed, but it has proved to be difficult to implement correctly within the Linux VFS framework.
Mount options are processed in order, only the last occurrence of an option takes effect and may disable other options due to constraints (see eg. nodatacow and compress). The output of mount command shows which options have been applied.
acl, noacl
Enable/disable support for Posix Access Control Lists (ACLs). See the acl(5) manual page for more information about ACLs.
The support for ACL is build-time configurable (BTRFS_FS_POSIX_ACL) and mount fails if acl is requested but the feature is not compiled in.
autodefrag, noautodefrag
Enable automatic file defragmentation. When enabled, small random writes into files (in a range of tens of kilobytes, currently it's 64K) are detected and queued up for the defragmentation process. Not well suited for large database workloads.
The read latency may increase due to reading the adjacent blocks that make up the range for defragmentation, successive write will merge the blocks in the new location.
barrier, nobarrier
Ensure that all IO write operations make it through the device cache and are stored permanently when the filesystem is at its consistency checkpoint. This typically means that a flush command is sent to the device that will synchronize all pending data and ordinary metadata blocks, then writes the superblock and issues another flush.
The write flushes incur a slight hit and also prevent the IO block scheduler to reorder requests in a more effective way. Disabling barriers gets rid of that penalty but will most certainly lead to a corrupted filesystem in case of a crash or power loss. The ordinary metadata blocks could be yet unwritten at the time the new superblock is stored permanently, expecting that the block pointers to metadata were stored permanently before.
On a device with a volatile battery-backed write-back cache, the nobarrier option will not lead to filesystem corruption as the pending blocks are supposed to make it to the permanent storage.
check_int, check_int_data, check_int_print_mask=value
These debugging options control the behavior of the integrity checking module (the BTRFS_FS_CHECK_INTEGRITY config option required). The main goal is to verify that all blocks from a given transaction period are properly linked.
check_int enables the integrity checker module, which examines all block write requests to ensure on-disk consistency, at a large memory and CPU cost.
check_int_data includes extent data in the integrity checks, and implies the check_int option.
check_int_print_mask takes a bitmask of BTRFSIC_PRINT_MASK_* values as defined in fs/btrfs/check-integrity.c, to control the integrity checker module behavior.
See comments at the top of fs/btrfs/check-integrity.c for more information.
clear_cache
commit=seconds
Set the interval of periodic transaction commit when data are synchronized to permanent storage. Higher interval values lead to larger amount of unwritten data, which has obvious consequences when the system crashes. The upper bound is not forced, but a warning is printed if it's more than 300 seconds (5 minutes). Use with care.
compress, compress=type[:level], compress-force, compress-force=type[:level]
Control BTRFS file data compression. Type may be specified as zlib, lzo, zstd or no (for no compression, used for remounting). If no type is specified, zlib is used. If compress-force is specified, then compression will always be attempted, but the data may end up uncompressed if the compression would make them larger.
Both zlib and zstd (since version 5.1) expose the compression level as a tunable knob with higher levels trading speed and memory (zstd) for higher compression ratios. This can be set by appending a colon and the desired level. Zlib accepts the range [1, 9] and zstd accepts [1, 15]. If no level is set, both currently use a default level of 3. The value 0 is an alias for the default level.
Otherwise some simple heuristics are applied to detect an incompressible file. If the first blocks written to a file are not compressible, the whole file is permanently marked to skip compression. As this is too simple, the compress-force is a workaround that will compress most of the files at the cost of some wasted CPU cycles on failed attempts. Since kernel 4.15, a set of heuristic algorithms have been improved by using frequency sampling, repeated pattern detection and Shannon entropy calculation to avoid that.
datacow, nodatacow
Enable data copy-on-write for newly created files. Nodatacow implies nodatasum, and disables compression. All files created under nodatacow are also set the NOCOW file attribute (see chattr(1)).
datasum, nodatasum
Enable data checksumming for newly created files. Datasum implies datacow, ie. the normal mode of operation. All files created under nodatasum inherit the "no checksums" property, however there's no corresponding file attribute (see chattr(1)).
degraded
Allow mounts with less devices than the RAID profile constraints require. A read-write mount (or remount) may fail when there are too many devices missing, for example if a stripe member is completely missing from RAID0.
Since 4.14, the constraint checks have been improved and are verified on the chunk level, not an the device level. This allows degraded mounts of filesystems with mixed RAID profiles for data and metadata, even if the device number constraints would not be satisfied for some of the profiles.
Example: metadata --- raid1, data --- single, devices --- /dev/sda, /dev/sdb
Suppose the data are completely stored on sda, then missing sdb will not prevent the mount, even if 1 missing device would normally prevent (any) single profile to mount. In case some of the data chunks are stored on sdb, then the constraint of single/data is not satisfied and the filesystem cannot be mounted.
device=devicepath
discard, discard=sync, discard=async, nodiscard
Enable discarding of freed file blocks. This is useful for SSD devices, thinly provisioned LUNs, or virtual machine images; however, every storage layer must support discard for it to work.
In the synchronous mode (sync or without option value), lack of asynchronous queued TRIM on the backing device TRIM can severely degrade performance, because a synchronous TRIM operation will be attempted instead. Queued TRIM requires newer than SATA revision 3.1 chipsets and devices.
The asynchronous mode (async) gathers extents in larger chunks before sending them to the devices for TRIM. The overhead and performance impact should be negligible compared to the previous mode and it's supposed to be the preferred mode if needed.
If it is not necessary to immediately discard freed blocks, then the fstrim tool can be used to discard all free blocks in a batch. Scheduling a TRIM during a period of low system activity will prevent latent interference with the performance of other operations. Also, a device may ignore the TRIM command if the range is too small, so running a batch discard has a greater probability of actually discarding the blocks.
enospc_debug, noenospc_debug
Enable verbose output for some ENOSPC conditions. It's safe to use but can be noisy if the system reaches near-full state.
fatal_errors=action
Action to take when encountering a fatal error.
bug
panic
flushoncommit, noflushoncommit
This option forces any data dirtied by a write in a prior transaction to commit as part of the current commit, effectively a full filesystem sync.
This makes the committed state a fully consistent view of the file system from the application's perspective (i.e. it includes all completed file system operations). This was previously the behavior only when a snapshot was created.
When off, the filesystem is consistent but buffered writes may last more than one transaction commit.
fragment=type
A debugging helper to intentionally fragment given type of block groups. The type can be data, metadata or all. This mount option should not be used outside of debugging environments and is not recognized if the kernel config option BTRFS_DEBUG is not enabled.
inode_cache, noinode_cache
Enable free inode number caching. Not recommended to use unless files on your filesystem get assigned inode numbers that are approaching 2^64. Normally, new files in each subvolume get assigned incrementally (plus one from the last time) and are not reused. The mount option turns on caching of the existing inode numbers and reuse of inode numbers of deleted files.
This option may slow down your system at first run, or after mounting without the option.
nologreplay
The tree-log contains pending updates to the filesystem until the full commit. The log is replayed on next mount, this can be disabled by this option. See also treelog. Note that nologreplay is the same as norecovery.
max_inline=bytes
Specify the maximum amount of space, that can be inlined in a metadata B-tree leaf. The value is specified in bytes, optionally with a K suffix (case insensitive). In practice, this value is limited by the filesystem block size (named sectorsize at mkfs time), and memory page size of the system. In case of sectorsize limit, there's some space unavailable due to leaf headers. For example, a 4k sectorsize, maximum size of inline data is about 3900 bytes.
Inlining can be completely turned off by specifying 0. This will increase data block slack if file sizes are much smaller than block size but will reduce metadata consumption in return.
metadata_ratio=value
Specifies that 1 metadata chunk should be allocated after every value data chunks. Default behaviour depends on internal logic, some percent of unused metadata space is attempted to be maintained but is not always possible if there's not enough space left for chunk allocation. The option could be useful to override the internal logic in favor of the metadata allocation if the expected workload is supposed to be metadata intense (snapshots, reflinks, xattrs, inlined files).
norecovery
Do not attempt any data recovery at mount time. This will disable logreplay and avoids other write operations. Note that this option is the same as nologreplay.
rescan_uuid_tree
Force check and rebuild procedure of the UUID tree. This should not normally be needed.
skip_balance
Skip automatic resume of an interrupted balance operation. The operation can later be resumed with btrfs balance resume, or the paused state can be removed with btrfs balance cancel. The default behaviour is to resume an interrupted balance immediately after a volume is mounted.
space_cache, space_cache=version, nospace_cache
Options to control the free space cache. The free space cache greatly improves performance when reading block group free space into memory. However, managing the space cache consumes some resources, including a small amount of disk space.
There are two implementations of the free space cache. The original one, referred to as v1, is the safe default. The v1 space cache can be disabled at mount time with nospace_cache without clearing.
On very large filesystems (many terabytes) and certain workloads, the performance of the v1 space cache may degrade drastically. The v2 implementation, which adds a new B-tree called the free space tree, addresses this issue. Once enabled, the v2 space cache will always be used and cannot be disabled unless it is cleared. Use clear_cache,space_cache=v1 or clear_cache,nospace_cache to do so. If v2 is enabled, kernels without v2 support will only be able to mount the filesystem in read-only mode. The btrfs(8) command currently only has read-only support for v2. A read-write command may be run on a v2 filesystem by clearing the cache, running the command, and then remounting with space_cache=v2.
If a version is not explicitly specified, the default implementation will be chosen, which is v1.
ssd, ssd_spread, nossd, nossd_spread
Options to control SSD allocation schemes. By default, BTRFS will enable or disable SSD optimizations depending on status of a device with respect to rotational or non-rotational type. This is determined by the contents of /sys/block/DEV/queue/rotational). If it is 0, the ssd option is turned on. The option nossd will disable the autodetection.
The optimizations make use of the absence of the seek penalty that's inherent for the rotational devices. The blocks can be typically written faster and are not offloaded to separate threads.
subvol=path
subvolid=subvolid
thread_pool=number
The number of worker threads to start. NRCPUS is number of on-line CPUs detected at the time of mount. Small number leads to less parallelism in processing data and metadata, higher numbers could lead to a performance hit due to increased locking contention, process scheduling, cache-line bouncing or costly data transfers between local CPU memories.
treelog, notreelog
Enable the tree logging used for fsync and O_SYNC writes. The tree log stores changes without the need of a full filesystem sync. The log operations are flushed at sync and transaction commit. If the system crashes between two such syncs, the pending tree log operations are replayed during mount.
usebackuproot
Enable autorecovery attempts if a bad tree root is found at mount time. Currently this scans a backup list of several previous tree roots and tries to use the first readable. This can be used with read-only mounts as well.
user_subvol_rm_allowed
Allow subvolumes to be deleted by their respective owner. Otherwise, only the root user can do that.
List of mount options that have been removed, kept for backward compatibility.
recovery
Some of the general mount options from mount(8) that affect BTRFS and are worth mentioning.
noatime
Note that noatime may break applications that rely on atime uptimes like the venerable Mutt (unless you use maildir mailboxes).
The basic set of filesystem features gets extended over time. The backward compatibility is maintained and the features are optional, need to be explicitly asked for so accidental use will not create incompatibilities.
There are several classes and the respective tools to manage the features:
at mkfs time only
after mkfs, on an unmounted filesystem
after mkfs, on a mounted filesystem
Whether a particular feature can be turned on a mounted filesystem can be found in the directory /sys/fs/btrfs/features/, one file per feature. The value 1 means the feature can be enabled.
List of features (see also mkfs.btrfs(8) section FILESYSTEM FEATURES):
big_metadata
the filesystem uses nodesize for metadata blocks, this can be bigger than the page size
compress_lzo
the lzo compression has been used on the filesystem, either as a mount option or via btrfs filesystem defrag.
compress_zstd
the zstd compression has been used on the filesystem, either as a mount option or via btrfs filesystem defrag.
default_subvol
the default subvolume has been set on the filesystem
extended_iref
increased hardlink limit per file in a directory to 65536, older kernels supported a varying number of hardlinks depending on the sum of all file name sizes that can be stored into one metadata block
free_space_tree
free space representation using a dedicated b-tree, successor of v1 space cache
metadata_uuid
the main filesystem UUID is the metadata_uuid, which stores the new UUID only in the superblock while all metadata blocks still have the UUID set at mkfs time, see btrfstune(8) for more
mixed_backref
the last major disk format change, improved backreferences, now default
mixed_groups
mixed data and metadata block groups, ie. the data and metadata are not separated and occupy the same block groups, this mode is suitable for small volumes as there are no constraints how the remaining space should be used (compared to the split mode, where empty metadata space cannot be used for data and vice versa)
on the other hand, the final layout is quite unpredictable and possibly highly fragmented, which means worse performance
no_holes
improved representation of file extents where holes are not explicitly stored as an extent, saves a few percent of metadata if sparse files are used
raid1c34
extended RAID1 mode with copies on 3 or 4 devices respectively
raid56
the filesystem contains or contained a raid56 profile of block groups
rmdir_subvol
indicate that rmdir(2) syscall can delete an empty subvolume just like an ordinary directory. Note that this feature only depends on the kernel version.
skinny_metadata
reduced-size metadata for extent references, saves a few percent of metadata
supported_checksums
list of checksum algorithms supported by the kernel module, the respective modules or built-in implementing the algorithms need to be present to mount the filesystem
The swapfile is supported since kernel 5.0. Use swapon(8) to activate the swapfile. There are some limitations of the implementation in btrfs and linux swap subsystem:
The limitations come namely from the COW-based design and mapping layer of blocks that allows the advanced features like relocation and multi-device filesystems. However, the swap subsystem expects simpler mapping and no background changes of the file blocks once they've been attached to swap.
With active swapfiles, the following whole-filesystem operations will skip swapfile extents or may fail:
When there are no active swapfiles and a whole-filesystem exclusive operation is running (ie. balance, device delete, shrink), the swapfiles cannot be temporarily activated. The operation must finish first.
# truncate -s 0 swapfile # chattr +C swapfile # fallocate -l 2G swapfile # chmod 0600 swapfile # mkswap swapfile # swapon swapfile
There are several checksum algorithms supported. The default and backward compatible is crc32c. Since kernel 5.5 there are three more with different characteristics and trade-offs regarding speed and strength. The following list may help you to decide which one to select.
CRC32C (32bit digest)
XXHASH (64bit digest)
SHA256 (256bit digest)
BLAKE2b (256bit digest)
The digest size affects overall size of data block checksums stored in the filesystem. The metadata blocks have a fixed area up to 256bits (32 bytes), so there's no increase. Each data block has a separate checksum stored, with additional overhead of the b-tree leaves.
Approximate relative performance of the algorithms, measured against CRC32C using reference software implementations on a 3.5GHz intel CPU:
|
Digest |
Cycles/4KiB |
Ratio
|
|
CRC32C |
1700 |
1.00
|
|
XXHASH |
2500 |
1.44
|
|
SHA256 |
105000 |
61
|
|
BLAKE2b |
22000 |
13
|
There are several operations that affect the whole filesystem and cannot be run in parallel. Attempt to start one while another is running will fail.
Since kernel 5.10 the currently running operation can be obtained from /sys/fs/UUID/exclusive_operation with following values and operations:
Enqueuing is supported for several btrfs subcommands so they can be started at once and then serialized.
maximum file name length
maximum symlink target length
The symlink target may not be a valid path, ie. the path name components can exceed the limits (NAME_MAX), there's no content validation at symlink(3) creation.
maximum number of inodes
inode numbers
maximum file length
maximum number of subvolumes
maximum number of hardlinks of a file in a directory
minimum filesystem size
GRUB2 (m[blue]https://www.gnu.org/software/grubm[]) has the most advanced support of booting from BTRFS with respect to features.
U-boot (m[blue]https://www.denx.de/wiki/U-Boot/m[]) has decent support for booting but not all BTRFS features are implemented, check the documentation.
EXTLINUX (from the m[blue]https://syslinux.orgm[] project) can boot but does not support all features. Please check the upstream documentation before you use it.
The first 1MiB on each device is unused with the exception of primary superblock that is on the offset 64KiB and spans 4KiB.
The btrfs filesystem supports setting file attributes or flags. Note there are old and new interfaces, with confusing names. The following list should clarify that:
a
A
c
When set on a directory, all newly created files will inherit this attribute.
C
When set on a directory, all newly created files will inherit this attribute.
d
D
i
S
X
When set on a directory, all newly created files will inherit this attribute.
No other attributes are supported. For the complete list please refer to the chattr(1) manual page.
There's overlap of letters assigned to the bits with the attributes, this list refers to what xfs_io(8) provides:
i
a
s
A
d
There's a character special device /dev/btrfs-control with major and minor numbers 10 and 234 (the device can be found under the misc category).
$ ls -l /dev/btrfs-control crw------- 1 root root 10, 234 Jan 1 12:00 /dev/btrfs-control
The device accepts some ioctl calls that can perform following actions on the filesystem module:
The device is created when btrfs is initialized, either as a module or a built-in functionality and makes sense only in connection with that. Running eg. mkfs without the module loaded will not register the device and will probably warn about that.
In rare cases when the module is loaded but the device is not present (most likely accidentally deleted), it's possible to recreate it by
# mknod --mode=600 /dev/btrfs-control c 10 234
or (since 5.11) by a convenience command
# btrfs rescue create-control-device
The control device is not strictly required but the device scanning will not work and a workaround would need to be used to mount a multi-device filesystem. The mount option device can trigger the device scanning during mount, see also btrfs device scan.
It is possible that a btrfs filesystem contains multiple block group profiles of the same type. This could happen when a profile conversion using balance filters is interrupted (see btrfs-balance(8)). Some btrfs commands perform a test to detect this kind of condition and print a warning like this:
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'. WARNING: Data: single, raid1 WARNING: Metadata: single, raid1
The corresponding output of btrfs filesystem df might look like:
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'. WARNING: Data: single, raid1 WARNING: Metadata: single, raid1 Data, RAID1: total=832.00MiB, used=0.00B Data, single: total=1.63GiB, used=0.00B System, single: total=4.00MiB, used=16.00KiB Metadata, single: total=8.00MiB, used=112.00KiB Metadata, RAID1: total=64.00MiB, used=32.00KiB GlobalReserve, single: total=16.25MiB, used=0.00B
There's more than one line for type Data and Metadata, while the profiles are single and RAID1.
This state of the filesystem OK but most likely needs the user/administrator to take an action and finish the interrupted tasks. This cannot be easily done automatically, also the user knows the expected final profiles.
In the example above, the filesystem started as a single device and single block group profile. Then another device was added, followed by balance with convert=raid1 but for some reason hasn't finished. Restarting the balance with convert=raid1 will continue and end up with filesystem with all block group profiles RAID1.
If you're familiar with balance filters, you can use convert=raid1,profiles=single,soft, which will take only the unconverted single profiles and convert them to raid1. This may speed up the conversion as it would not try to rewrite the already convert raid1 profiles.
Having just one profile is desired as this also clearly defines the profile of newly allocated block groups, otherwise this depends on internal allocation policy. When there are multiple profiles present, the order of selection is RAID6, RAID5, RAID10, RAID1, RAID0 as long as the device number constraints are satisfied.
Commands that print the warning were chosen so they're brought to user attention when the filesystem state is being changed in that regard. This is: device add, device delete, balance cancel, balance pause. Commands that report space usage: filesystem df, device usage. The command filesystem usage provides a line in the overall summary:
Multiple profiles: yes (data, metadata)
The COW mechanism and multiple devices under one hood enable an interesting concept, called a seeding device: extending a read-only filesystem on a single device filesystem with another device that captures all writes. For example imagine an immutable golden image of an operating system enhanced with another device that allows to use the data from the golden image and normal operation. This idea originated on CD-ROMs with base OS and allowing to use them for live systems, but this became obsolete. There are technologies providing similar functionality, like unionmount, overlayfs or qcow2 image snapshot.
The seeding device starts as a normal filesystem, once the contents is ready, btrfstune -S 1 is used to flag it as a seeding device. Mounting such device will not allow any writes, except adding a new device by btrfs device add. Then the filesystem can be remounted as read-write.
Given that the filesystem on the seeding device is always recognized as read-only, it can be used to seed multiple filesystems, at the same time. The UUID that is normally attached to a device is automatically changed to a random UUID on each mount.
Once the seeding device is mounted, it needs the writable device. After adding it, something like remount -o remount,rw /path makes the filesystem at /path ready for use. The simplest usecase is to throw away all changes by unmounting the filesystem when convenient.
Alternatively, deleting the seeding device from the filesystem can turn it into a normal filesystem, provided that the writable device can also contain all the data from the seeding device.
The seeding device flag can be cleared again by btrfstune -f -s 0, eg. allowing to update with newer data but please note that this will invalidate all existing filesystems that use this particular seeding device. This works for some usecases, not for others, and a forcing flag to the command is mandatory to avoid accidental mistakes.
Example how to create and use one seeding device:
# mkfs.btrfs /dev/sda # mount /dev/sda /mnt/mnt1 # ... fill mnt1 with data # umount /mnt/mnt1 # btrfstune -S 1 /dev/sda # mount /dev/sda /mnt/mnt1 # btrfs device add /dev/sdb /mnt # mount -o remount,rw /mnt/mnt1 # ... /mnt/mnt1 is now writable
Now /mnt/mnt1 can be used normally. The device /dev/sda can be mounted again with a another writable device:
# mount /dev/sda /mnt/mnt2 # btrfs device add /dev/sdc /mnt/mnt2 # mount -o remount,rw /mnt/mnt2 # ... /mnt/mnt2 is now writable
The writable device (/dev/sdb) can be decoupled from the seeding device and used independently:
# btrfs device delete /dev/sda /mnt/mnt1
As the contents originated in the seeding device, it's possible to turn /dev/sdb to a seeding device again and repeat the whole process.
A few things to note:
The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. There are some implementation and design deficiencies that make it unreliable for some corner cases and the feature should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.
Do not use raid5 nor raid6 for metadata. Use raid1 or raid1c3 respectively.
The substitute profiles provide the same guarantees against loss of 1 or 2 devices, and in some respect can be an improvement. Recovering from one missing device will only need to access the remaining 1st or 2nd copy, that in general may be stored on some other devices due to the way RAID1 works on btrfs, unlike on a striped profile (similar to raid0) that would need all devices all the time.
The space allocation pattern and consumption is different (eg. on N devices): for raid5 as an example, a 1GiB chunk is reserved on each device, while with raid1 there's each 1GiB chunk stored on 2 devices. The consumption of each 1GiB of used metadata is then N * 1GiB for vs 2 * 1GiB. Using raid1 is also more convenient for balancing/converting to other profile due to lower requirement on the available chunk space.
When RAID56 is on the same filesystem with different raid profiles, the space reporting is inaccurate, eg. df, btrfs filesystem df or btrfs filesystem usge. When there's only a one profile per block group type (eg. raid5 for data) the reporting is accurate.
When scrub is started on a RAID56 filesystem, it's started on all devices that degrade the performance. The workaround is to start it on each device separately. Due to that the device stats may not match the actual state and some errors might get reported multiple times.
acl(5), btrfs(8), chattr(1), fstrim(8), ioctl(2), mkfs.btrfs(8), mount(8), swapon(8)