rjw | 1f88458 | 2022-01-06 17:20:42 +0800 | [diff] [blame^] | 1 | Written by: Neil Brown |
| 2 | Please see MAINTAINERS file for where to send questions. |
| 3 | |
| 4 | Overlay Filesystem |
| 5 | ================== |
| 6 | |
| 7 | This document describes a prototype for a new approach to providing |
| 8 | overlay-filesystem functionality in Linux (sometimes referred to as |
| 9 | union-filesystems). An overlay-filesystem tries to present a |
| 10 | filesystem which is the result over overlaying one filesystem on top |
| 11 | of the other. |
| 12 | |
| 13 | The result will inevitably fail to look exactly like a normal |
| 14 | filesystem for various technical reasons. The expectation is that |
| 15 | many use cases will be able to ignore these differences. |
| 16 | |
| 17 | This approach is 'hybrid' because the objects that appear in the |
| 18 | filesystem do not all appear to belong to that filesystem. In many |
| 19 | cases an object accessed in the union will be indistinguishable |
| 20 | from accessing the corresponding object from the original filesystem. |
| 21 | This is most obvious from the 'st_dev' field returned by stat(2). |
| 22 | |
| 23 | While directories will report an st_dev from the overlay-filesystem, |
| 24 | non-directory objects may report an st_dev from the lower filesystem or |
| 25 | upper filesystem that is providing the object. Similarly st_ino will |
| 26 | only be unique when combined with st_dev, and both of these can change |
| 27 | over the lifetime of a non-directory object. Many applications and |
| 28 | tools ignore these values and will not be affected. |
| 29 | |
| 30 | In the special case of all overlay layers on the same underlying |
| 31 | filesystem, all objects will report an st_dev from the overlay |
| 32 | filesystem and st_ino from the underlying filesystem. This will |
| 33 | make the overlay mount more compliant with filesystem scanners and |
| 34 | overlay objects will be distinguishable from the corresponding |
| 35 | objects in the original filesystem. |
| 36 | |
| 37 | Upper and Lower |
| 38 | --------------- |
| 39 | |
| 40 | An overlay filesystem combines two filesystems - an 'upper' filesystem |
| 41 | and a 'lower' filesystem. When a name exists in both filesystems, the |
| 42 | object in the 'upper' filesystem is visible while the object in the |
| 43 | 'lower' filesystem is either hidden or, in the case of directories, |
| 44 | merged with the 'upper' object. |
| 45 | |
| 46 | It would be more correct to refer to an upper and lower 'directory |
| 47 | tree' rather than 'filesystem' as it is quite possible for both |
| 48 | directory trees to be in the same filesystem and there is no |
| 49 | requirement that the root of a filesystem be given for either upper or |
| 50 | lower. |
| 51 | |
| 52 | The lower filesystem can be any filesystem supported by Linux and does |
| 53 | not need to be writable. The lower filesystem can even be another |
| 54 | overlayfs. The upper filesystem will normally be writable and if it |
| 55 | is it must support the creation of trusted.* extended attributes, and |
| 56 | must provide valid d_type in readdir responses, so NFS is not suitable. |
| 57 | |
| 58 | A read-only overlay of two read-only filesystems may use any |
| 59 | filesystem type. |
| 60 | |
| 61 | Directories |
| 62 | ----------- |
| 63 | |
| 64 | Overlaying mainly involves directories. If a given name appears in both |
| 65 | upper and lower filesystems and refers to a non-directory in either, |
| 66 | then the lower object is hidden - the name refers only to the upper |
| 67 | object. |
| 68 | |
| 69 | Where both upper and lower objects are directories, a merged directory |
| 70 | is formed. |
| 71 | |
| 72 | At mount time, the two directories given as mount options "lowerdir" and |
| 73 | "upperdir" are combined into a merged directory: |
| 74 | |
| 75 | mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ |
| 76 | workdir=/work /merged |
| 77 | |
| 78 | The "workdir" needs to be an empty directory on the same filesystem |
| 79 | as upperdir. |
| 80 | |
| 81 | Then whenever a lookup is requested in such a merged directory, the |
| 82 | lookup is performed in each actual directory and the combined result |
| 83 | is cached in the dentry belonging to the overlay filesystem. If both |
| 84 | actual lookups find directories, both are stored and a merged |
| 85 | directory is created, otherwise only one is stored: the upper if it |
| 86 | exists, else the lower. |
| 87 | |
| 88 | Only the lists of names from directories are merged. Other content |
| 89 | such as metadata and extended attributes are reported for the upper |
| 90 | directory only. These attributes of the lower directory are hidden. |
| 91 | |
| 92 | whiteouts and opaque directories |
| 93 | -------------------------------- |
| 94 | |
| 95 | In order to support rm and rmdir without changing the lower |
| 96 | filesystem, an overlay filesystem needs to record in the upper filesystem |
| 97 | that files have been removed. This is done using whiteouts and opaque |
| 98 | directories (non-directories are always opaque). |
| 99 | |
| 100 | A whiteout is created as a character device with 0/0 device number. |
| 101 | When a whiteout is found in the upper level of a merged directory, any |
| 102 | matching name in the lower level is ignored, and the whiteout itself |
| 103 | is also hidden. |
| 104 | |
| 105 | A directory is made opaque by setting the xattr "trusted.overlay.opaque" |
| 106 | to "y". Where the upper filesystem contains an opaque directory, any |
| 107 | directory in the lower filesystem with the same name is ignored. |
| 108 | |
| 109 | readdir |
| 110 | ------- |
| 111 | |
| 112 | When a 'readdir' request is made on a merged directory, the upper and |
| 113 | lower directories are each read and the name lists merged in the |
| 114 | obvious way (upper is read first, then lower - entries that already |
| 115 | exist are not re-added). This merged name list is cached in the |
| 116 | 'struct file' and so remains as long as the file is kept open. If the |
| 117 | directory is opened and read by two processes at the same time, they |
| 118 | will each have separate caches. A seekdir to the start of the |
| 119 | directory (offset 0) followed by a readdir will cause the cache to be |
| 120 | discarded and rebuilt. |
| 121 | |
| 122 | This means that changes to the merged directory do not appear while a |
| 123 | directory is being read. This is unlikely to be noticed by many |
| 124 | programs. |
| 125 | |
| 126 | seek offsets are assigned sequentially when the directories are read. |
| 127 | Thus if |
| 128 | |
| 129 | - read part of a directory |
| 130 | - remember an offset, and close the directory |
| 131 | - re-open the directory some time later |
| 132 | - seek to the remembered offset |
| 133 | |
| 134 | there may be little correlation between the old and new locations in |
| 135 | the list of filenames, particularly if anything has changed in the |
| 136 | directory. |
| 137 | |
| 138 | Readdir on directories that are not merged is simply handled by the |
| 139 | underlying directory (upper or lower). |
| 140 | |
| 141 | renaming directories |
| 142 | -------------------- |
| 143 | |
| 144 | When renaming a directory that is on the lower layer or merged (i.e. the |
| 145 | directory was not created on the upper layer to start with) overlayfs can |
| 146 | handle it in two different ways: |
| 147 | |
| 148 | 1. return EXDEV error: this error is returned by rename(2) when trying to |
| 149 | move a file or directory across filesystem boundaries. Hence |
| 150 | applications are usually prepared to hande this error (mv(1) for example |
| 151 | recursively copies the directory tree). This is the default behavior. |
| 152 | |
| 153 | 2. If the "redirect_dir" feature is enabled, then the directory will be |
| 154 | copied up (but not the contents). Then the "trusted.overlay.redirect" |
| 155 | extended attribute is set to the path of the original location from the |
| 156 | root of the overlay. Finally the directory is moved to the new |
| 157 | location. |
| 158 | |
| 159 | Non-directories |
| 160 | --------------- |
| 161 | |
| 162 | Objects that are not directories (files, symlinks, device-special |
| 163 | files etc.) are presented either from the upper or lower filesystem as |
| 164 | appropriate. When a file in the lower filesystem is accessed in a way |
| 165 | the requires write-access, such as opening for write access, changing |
| 166 | some metadata etc., the file is first copied from the lower filesystem |
| 167 | to the upper filesystem (copy_up). Note that creating a hard-link |
| 168 | also requires copy_up, though of course creation of a symlink does |
| 169 | not. |
| 170 | |
| 171 | The copy_up may turn out to be unnecessary, for example if the file is |
| 172 | opened for read-write but the data is not modified. |
| 173 | |
| 174 | The copy_up process first makes sure that the containing directory |
| 175 | exists in the upper filesystem - creating it and any parents as |
| 176 | necessary. It then creates the object with the same metadata (owner, |
| 177 | mode, mtime, symlink-target etc.) and then if the object is a file, the |
| 178 | data is copied from the lower to the upper filesystem. Finally any |
| 179 | extended attributes are copied up. |
| 180 | |
| 181 | Once the copy_up is complete, the overlay filesystem simply |
| 182 | provides direct access to the newly created file in the upper |
| 183 | filesystem - future operations on the file are barely noticed by the |
| 184 | overlay filesystem (though an operation on the name of the file such as |
| 185 | rename or unlink will of course be noticed and handled). |
| 186 | |
| 187 | |
| 188 | Multiple lower layers |
| 189 | --------------------- |
| 190 | |
| 191 | Multiple lower layers can now be given using the the colon (":") as a |
| 192 | separator character between the directory names. For example: |
| 193 | |
| 194 | mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged |
| 195 | |
| 196 | As the example shows, "upperdir=" and "workdir=" may be omitted. In |
| 197 | that case the overlay will be read-only. |
| 198 | |
| 199 | The specified lower directories will be stacked beginning from the |
| 200 | rightmost one and going left. In the above example lower1 will be the |
| 201 | top, lower2 the middle and lower3 the bottom layer. |
| 202 | |
| 203 | |
| 204 | Sharing and copying layers |
| 205 | -------------------------- |
| 206 | |
| 207 | Lower layers may be shared among several overlay mounts and that is indeed |
| 208 | a very common practice. An overlay mount may use the same lower layer |
| 209 | path as another overlay mount and it may use a lower layer path that is |
| 210 | beneath or above the path of another overlay lower layer path. |
| 211 | |
| 212 | Using an upper layer path and/or a workdir path that are already used by |
| 213 | another overlay mount is not allowed and may fail with EBUSY. Using |
| 214 | partially overlapping paths is not allowed but will not fail with EBUSY. |
| 215 | If files are accessed from two overlayfs mounts which share or overlap the |
| 216 | upper layer and/or workdir path the behavior of the overlay is undefined, |
| 217 | though it will not result in a crash or deadlock. |
| 218 | |
| 219 | Mounting an overlay using an upper layer path, where the upper layer path |
| 220 | was previously used by another mounted overlay in combination with a |
| 221 | different lower layer path, is allowed, unless the "inodes index" feature |
| 222 | is enabled. |
| 223 | |
| 224 | With the "inodes index" feature, on the first time mount, an NFS file |
| 225 | handle of the lower layer root directory, along with the UUID of the lower |
| 226 | filesystem, are encoded and stored in the "trusted.overlay.origin" extended |
| 227 | attribute on the upper layer root directory. On subsequent mount attempts, |
| 228 | the lower root directory file handle and lower filesystem UUID are compared |
| 229 | to the stored origin in upper root directory. On failure to verify the |
| 230 | lower root origin, mount will fail with ESTALE. An overlayfs mount with |
| 231 | "inodes index" enabled will fail with EOPNOTSUPP if the lower filesystem |
| 232 | does not support NFS export, lower filesystem does not have a valid UUID or |
| 233 | if the upper filesystem does not support extended attributes. |
| 234 | |
| 235 | It is quite a common practice to copy overlay layers to a different |
| 236 | directory tree on the same or different underlying filesystem, and even |
| 237 | to a different machine. With the "inodes index" feature, trying to mount |
| 238 | the copied layers will fail the verification of the lower root file handle. |
| 239 | |
| 240 | |
| 241 | Non-standard behavior |
| 242 | --------------------- |
| 243 | |
| 244 | The copy_up operation essentially creates a new, identical file and |
| 245 | moves it over to the old name. The new file may be on a different |
| 246 | filesystem, so both st_dev and st_ino of the file may change. |
| 247 | |
| 248 | Any open files referring to this inode will access the old data. |
| 249 | |
| 250 | If a file with multiple hard links is copied up, then this will |
| 251 | "break" the link. Changes will not be propagated to other names |
| 252 | referring to the same inode. |
| 253 | |
| 254 | Unless "redirect_dir" feature is enabled, rename(2) on a lower or merged |
| 255 | directory will fail with EXDEV. |
| 256 | |
| 257 | Changes to underlying filesystems |
| 258 | --------------------------------- |
| 259 | |
| 260 | Offline changes, when the overlay is not mounted, are allowed to either |
| 261 | the upper or the lower trees. |
| 262 | |
| 263 | Changes to the underlying filesystems while part of a mounted overlay |
| 264 | filesystem are not allowed. If the underlying filesystem is changed, |
| 265 | the behavior of the overlay is undefined, though it will not result in |
| 266 | a crash or deadlock. |
| 267 | |
| 268 | Testsuite |
| 269 | --------- |
| 270 | |
| 271 | There's testsuite developed by David Howells at: |
| 272 | |
| 273 | git://git.infradead.org/users/dhowells/unionmount-testsuite.git |
| 274 | |
| 275 | Run as root: |
| 276 | |
| 277 | # cd unionmount-testsuite |
| 278 | # ./run --ov |