Microsoft's new ReFS file system initially appeared on servers running Windows control 2012. And only later it was included in Windows 10, where it can only be used as part of the Storage Spaces feature (disk space virtualization technology) of a disk pool. IN Windows Server 2016 Microsoft promises to significantly improve work with the ReFS file system, and, according to rumors in print, ReFS may replace the outdated NTFS file system in new version Windows 10, which proudly bears the name Windows 10 Pro (for advanced PCs).
But what exactly is ReFs, how does it differ from the currently used NTFS file system, and what advantages does it have?
What is ReFS
In short, it was designed as a fault-tolerant file system. ReFS is a new code-based file system that is essentially a redesign and improvement of the NTFS file system. These include improved reliability of information storage, stable operation in stress modes, file sizes, volumes, directories, the number of files in volumes and directories is limited only by the size of the 64-bit number. Let us recall that with this value, the maximum file size will be 16 exbibytes, and the volume size will be 1 yobibyte.
At the moment, ReFS is not a replacement for NTFS. It has its advantages and disadvantages. But you won't be able to, say, format a disk and install a new one on it. copy of Windows so how would you do it on NTFS.
ReFS protects your data
ReFS uses checksums for metadata and can also use checksums for data files. Every time you read or write files, ReFS checks the checksum to ensure it is correct. This means that the file system itself has a tool capable of detecting corrupted data on the fly.
ReFS is integrated with the Storage Spaces feature. If you have configured mirroring with ReFS support, Windows will easily detect file system corruption and automatically repair it by copying the mirrored data to the damaged disk. This function Available for both Windows 10 and Windows 8.1.
If ReFS detects corrupted data and there is no required copy of the data to restore, the file system is able to immediately remove the corrupted data from the disk. This does not require a system reboot, unlike NTFS.
ReFS not only checks the integrity of files during writing and reading. It automatically scans data integrity by regularly checking all files on the disk, identifying and correcting corrupted data. In this case, there is no need to periodically run the chkdsk command to check the disk.
The new file system is also resistant to data corruption in other ways. For example, you update the metadata of a file (let's say the file name). The NTFS file system directly changes the file's metadata. If at this time the system crashes (power outage), there is a high probability that the file will be damaged. When you change metadata, the ReFS file system creates a new copy of the metadata. The file system does not overwrite old metadata, but writes it to new block. This eliminates the possibility of file damage. This strategy is called “copy-on-write” (copy-on-write, highlight-on-write). This strategy is available in other modern file systems such as ZFS and BtrFS on Linux, as well as Apple's new APFS file system.
Limitations of the NTFS file system
ReFS is more modern than NTFS and supports much larger data volumes and longer file names. In the long run this is very important.
In the NTFS file system, the file path is limited to 255 characters. In ReFS maximum amount characters is already an impressive 32,768 characters. There is currently an option in Windows 10 to disable the character element for NTFS. On ReFS disk volumes this limit is disabled by default.
ReFS does not support DOS 8.3 filenames. On NTFS volumes, you have access to the “CProgram Files”, “CProgra`1” folders. They are needed for compatibility with the old one software. In ReFS you will not find the folders we are used to. They have been deleted.
The theoretical maximum data volume supported by NTFS is 16 exabytes, ReFS supports up to 262,144 exabytes. Now this figure seems simply huge.
ReFS performance
The developers did not set a goal to create a more productive file system. They made a more optimized system.
For example, when used with an array, ReFS supports real-time level optimization. You have a storage pool consisting of two disks. The first disk was selected for high speed and fast access to data. The second disk is selected with reliability criteria, under long-term storage data. IN background ReFS will automatically move large chunks of data to a slower disk, thereby ensuring that the data is stored securely.
In Windows Server 2016, developers have added a tool that provides increased productivity using certain functions virtual machines. For example, ReFS supports block copying, which speeds up the process of virtual machine copying and checkpoint merging operations. To create a copy of a virtual machine, ReFS creates a new copy of the metadata on disk and provides a link to the copied data on disk. This is so that with ReFS, multiple files can reference the same underlying data on disk. After you change the data after working with the virtual machine, it is written to disk in another location, but the original data of the virtual machine remains on the disk. This significantly speeds up the process of creating copies and reduces the load on the disk.
ReFS supports “Sparse VDL” (discharged files). A sparse file is a file in which a sequence of zero bytes has been replaced with information about that sequence (a list of holes). Holes are a specific sequence of zero bytes inside a file that were not written to disk. The information about holes itself is stored in the file system metadata.
Discharged file support technology allows you to quickly write zeros to a large file. This significantly speeds up the process of creating a new one, empty file virtual hard drive fixed size (VHD). Creating such a file in ReFS takes a few seconds, while in NTFS such an operation takes up to 10 minutes.
Still, ReFS is not able to completely replace NTFS
Everything we described above sounds good, but you won't be able to switch to ReFS from NTFS. Windows cannot boot from the ReFS file system, requiring NTFS.
ReFS lacks many technologies available in NTFS. For example, file system compression and encryption, hard links, extended attributes, data deduplication, and disk quotas. Moreover, unlike NTFS, ReFS supports full data encryption technology - BitLocker.
In Windows 10, you will not be able to format a disk partition with ReFS. The new file system is only available for storage systems where its main function is to protect data from damage. In Windows Server 2016, you will be able to format a disk partition with ReFS. You will be able to use it to run virtual machines. But you won't be able to select it as a boot disk. Windows boots only from the NTFS file system.
It's unclear what Microsoft's future holds for the new file system. Perhaps one day it will completely replace NTFS in all Windows versions. But on this moment ReFS can only be used for certain tasks.
Application of ReFS
Much has been said above in support of the new operating system. The disadvantages and advantages are described. I suggest you stop and summarize. For what purposes can, and maybe should, be used ReFS.
In Windows 10, ReFS is only applicable in conjunction with the Storage Spaces component. Be sure to format your disk dedicated to data storage in ReFS, not NTFS. In this case, you will be able to fully appreciate the reliability of data storage.
In Windows Server, you can format a partition for ReFS using the standard Windows tool in the Disk Management console. It is recommended to format it for ReFS if you use virtual servers. But remember that boot disk must be formatted as NTFS. Booting from the ReFS file system is not supported by Windows operating systems.
New ReFS file system and Windows 10| 2017-06-28 06:34:15 | Super User | System software | https://site/media/system/images/new.png | The new file system from Microsoft ReFS has replaced the outdated NTFS. What are the advantages of ReFS and how does it differ from NTFS | refs, refs or ntfs, refs windows 10, refs file system, new file systems, ntfs system, ntfs file system
A public beta version was released not long ago Microsoft Windows 8 Server with support for the announced ReFS (Resilient File System) file system, previously known as “Protogon”. This file system is offered as an alternative to the NTFS file system, which has proven itself over the years, in the segment of data storage systems based on Microsoft products, with its further migration to the area of client systems.
The purpose of this article is a superficial description of the structure of the file system, its advantages and disadvantages, as well as an analysis of its architecture from the point of view of maintaining data integrity and the prospects for data recovery in case of damage or deletion by the user. The article also reveals a study of the architectural features of the file system and its potential performance.
Windows Server 8 Beta
The file system option available in this version of the operating system only supports 64KB data clusters and 16KB metadata clusters. It is not yet clear whether there will be support for ReFS file systems with other cluster sizes: currently the Cluster Size parameter when creating a ReFS volume is ignored and is always set to default. When formatting the FS, the only available option to select the cluster size is 64KB. He is also the only one mentioned in the developer blogs.
This cluster size is more than sufficient to organize file systems of any practical size, but at the same time it leads to significant redundancy in data storage.
File system architecture
Despite frequent mentions of the similarities between ReFS and NTFS on high level, we are just talking about the compatibility of some metadata structures, such as: “standard information”, “file name”, compatibility with the values of some attribute flags, etc. The disk implementation of ReFS structures is fundamentally different from other Microsoft file systems.
The main structural elements of the new file system are B+ trees. All elements of the file system structure are represented by single-level (lists) or multi-level B+ trees, which allows you to significantly scale almost any of the file system elements. Along with real 64-bit numbering of all system elements, this eliminates the appearance of bottlenecks during further scaling.
Except for the root record of the B+ tree, all other records have the size of an entire metadata block (in in this case- 16KB); intermediate (address) nodes have a small total size (about 60 bytes). Therefore, usually a small number of tree levels are required to describe even very large structures, which has a rather favorable effect on the overall performance of the system.
The main structural element of the file system is the “Directory”, presented in the form of a B+-tree, the key of which is the number of the folder object. Unlike other similar file systems, a file in ReFS is not a separate key element of the “Directory”, but only exists as an entry in the folder containing it. It may be because of this architectural feature that hard links to ReFS are not supported.
“Leaves of the Directory” are typed records. There are three main types of entries for a folder object: the directory handle, the index entry, and the nested object handle. All such records are packaged as a separate B+ tree with a folder ID; the root of this tree is a leaf of the B+-tree of the “Directory”, which allows you to pack almost any number of records into a folder. At the bottom level in the leaves of a folder's B+ tree is primarily a directory descriptor entry containing basic information about the folder (such as name, "standard information", file name attribute, etc.). The data structures have much in common with those adopted in NTFS, although they have a number of differences, the main one of which is the absence of a typed list of named attributes.
Next in the directory are the so-called index entries: short structures containing data about the elements contained in the folder. Compared to NTFS, these records are much shorter, which reduces the volume's burden with metadata. Last are the directory item entries. For folders, these elements contain the name of the pack, the folder identifier in the “Directory” and the structure of the “standard information”. For files there is no identifier, but instead the structure contains all the basic data about the file, including the root of the B+ tree of file fragments. Accordingly, the file can consist of almost any number of fragments.
On disk, files are located in 64KB blocks, although they are addressed in the same way as metadata blocks (in 16KB clusters). File data “residency” is not supported on ReFS, so a 1-byte file on disk will occupy an entire 64KB block, which leads to significant storage redundancy on small files; on the other hand, it simplifies management free space and allocating free space for new file carried out much faster.
The size of an empty filesystem's metadata is about 0.1% of the size of the filesystem itself (i.e., about 2GB on a 2TB volume). Some core metadata is duplicated for better fault tolerance.
Failure proof
There was no goal to test the stability of the existing ReFS implementation. From the point of view of file system architecture, it has all the necessary tools for secure file recovery even after a serious hardware failure. Parts of metadata structures contain their own identifiers, which allows you to check the ownership of the structures; metadata links contain 64-bit checksums of the blocks being referenced, which makes it possible to evaluate the integrity of the block read from the link.
It is worth noting that checksums of user data (file contents) are not calculated. On the one hand, this disables the integrity check mechanism in the data area, on the other hand, it speeds up the system operation due to the minimum number of changes in the metadata area.
Any change in the metadata structure is carried out in two stages: first, a new (changed) copy of the metadata is created in free disk space, then, if successful, an atomic update operation transfers the link from the old (unchanged) to the new (changed) metadata area. This strategy (Copy-on-Write (CoW)) allows you to do without logging, automatically maintaining data integrity.
Confirmation of such changes on the disk may not take long enough, allowing several file system state changes to be combined into one.
This scheme does not apply to user data, so any changes to the file's contents are written directly to the file. Deleting a file is done by rebuilding the metadata structure (using CoW), which saves previous version metadata block on disk. This is what recovery does. deleted files possible before they are overwritten by new user data.
Data storage redundancy
In this case, we are talking about the consumption of disk space due to the data storage scheme. For testing purposes, the installed Windows Server was copied to a 580GB ReFS partition. The size of metadata on an empty file system was about 0.73GB.
When copying installed Windows Server per partition with ReFS, file data storage redundancy increased from 0.1% on NTFS to almost 30% on ReFS. At the same time, about 10% of redundancy was added due to metadata. As a result, “user data” of 11GB in size (more than 70 thousand files) on NTFS, taking into account metadata, took 11.3GB, while on ReFS the same data took 16.2GB; this means that data storage redundancy on ReFS is almost 50% for this type of data. With a small number of large files, this effect is naturally not observed.
Operation speed
Due to the fact that we are talking about Beta, no measurements of FS performance were carried out. From the point of view of the FS architecture, some conclusions can be drawn. When copying more than 70 thousand files to ReFS, this created a B+ tree of the “Directory” of 4 levels in size: “root”, intermediate level 1, intermediate level 2, “leaves”.
Thus, searching for folder attributes (assuming the tree root is cached) requires 3 reads of 16KB blocks. For comparison, on NTFS this operation will take one read of 1-4KB in size (assuming the $MFT location map is cached).
Finding file attributes by folder and file name in a folder (a small folder with several entries) on ReFS will require the same 3 reads. On NTFS, 2 reads of 1 KB each will be required, or 3-4 reads (if the file entry is in the non-resident “index” attribute). In larger packs, the number of NTFS reads grows much faster than the number of reads required by ReFS.
The situation is exactly the same for the contents of files: where an increase in the number of file fragments on NTFS leads to an enumeration of long lists spread across different $MFT fragments, on ReFS this is carried out by an effective search through the B+-tree.
conclusions
It is too early to draw final conclusions, but from the current implementation of the file system one can see confirmation of the initial focus of the file system on the server segment, and, above all, on virtualization systems, DBMS and archival data storage servers, where speed and reliability of operation are of paramount importance. The main disadvantage of the file system, such as inefficient packaging of data on disk, is negated on systems that operate with large files.
SysDev Laboratories will monitor the development of this file system and plans to include support for data recovery from this file system. Experimental ReFS support for the beta version of Microsoft Windows 8 Server has already been successfully implemented in UFS Explorer products and is available for closed beta testing among partners. The official release of tools for recovering deleted files from ReFS, as well as data recovery after file system damage as a result of hardware failures, is planned a little earlier or simultaneously with the release of Microsoft Windows 8 Server with ReFS support.
Version dated 03/16/2012.
Based on materials from SisDev Laboratories
Reproduction or quotation is permitted provided that reference to the original is maintained.
First in Windows Server, and now in Windows 10, the modern REFS (Resilient File System) file system appeared, in which you can format your computer’s hard drives or created system means storage spaces.
This article is about what the REFS file system is, its differences from NTFS and possible applications for the average home user.
In addition to features related to maintaining data integrity on disks, REFS has the following main differences from the NTFS file system:
- Typically better performance, especially when using Storage Spaces.
- The theoretical volume size is 262144 exabytes (versus 16 for NTFS).
- No file path limit of 255 characters (in REFS - 32768 characters).
- REFS does not support DOS file names (i.e. access to the folder C:\Program Files\ on the way C:\progra~1\ it won't work). In NTFS, this feature was retained for compatibility with older software.
- REFS does not support compression, additional attributes, or encryption using the file system (NTFS has this, but it works for REFS).
Cannot be formatted at this time system disk in REFS, the function is only available for non-system drives (not supported for removable drives), as well as for Storage Spaces, and perhaps only the latter option can be truly useful for regular user, who is concerned about the safety of data.
Please note that after formatting a disk in the REFS file system, part of the space on it will immediately be occupied by control data: for example, for an empty 10 GB disk this is about 700 MB.
It's possible that REFS could become the primary file system in Windows in the future, but that hasn't happened at the moment. Official information on the file system on the Microsoft website:
Why may a smartphone not launch programs from a memory card? How is ext4 fundamentally different from ext3? Why will a flash drive last longer if you format it in NTFS rather than FAT? What is the main problem with F2FS? The answers lie in the structural features of file systems. We'll talk about them.
Introduction
File systems define how data is stored. They determine what limitations the user will encounter, how fast read and write operations will be, and how long the drive will operate without failures. This is especially true for budget SSDs and their younger brothers - flash drives. Knowing these features, you can get the most out of any system and optimize its use for specific tasks.
You have to choose the type and parameters of the file system every time you need to do something non-trivial. For example, you want to speed up the most common file operations. At the file system level, this can be achieved in different ways: indexing will provide quick search, and pre-reserving free blocks will make it easier to rewrite frequently changing files. Preliminary data optimization in random access memory will reduce the number of required I/O operations.
Such properties of modern file systems as lazy writing, deduplication and other advanced algorithms help to increase the period of trouble-free operation. They are especially relevant for cheap SSDs with chips TLC memory, flash drives and memory cards.
Separate optimizations exist for disk arrays different levels: For example, the file system can support lightweight volume mirroring, instant snapshotting, or dynamic scaling without taking the volume offline.
Black box
Users generally work with the file system that is offered by default by the operating system. They rarely create new disk partitions and even less often think about their settings - they simply use the recommended parameters or even buy pre-formatted media.
For Windows fans, everything is simple: NTFS on all disk partitions and FAT32 (or the same NTFS) on flash drives. If there is a NAS and it uses some other file system, then for most it remains beyond perception. They simply connect to it over the network and download files, as if from a black box.
On mobile gadgets with Android ext4 is most often found in internal memory and FAT32 on microSD cards. Yabloko does not care at all what kind of file system they have: HFS+, HFSX, APFS, WTFS... for them there are only beautiful folder and file icons drawn by the best designers. Linux users have the richest choice, but you can add support for non-native file systems in both Windows and macOS - more on that later.
Common roots
Over a hundred different file systems have been created, but a little more than a dozen can be considered current. Although they were all developed for their own specific applications, many ended up being related on a conceptual level. They are similar because they use the same type of (meta)data representation structure - B-trees (“bi-trees”).
Like any hierarchical system, a B-tree begins with a root record and then branches down to leaf elements - individual records of files and their attributes, or “leaves.” The main point of creating such a logical structure was to speed up the search for file system objects on large dynamic arrays - like hard drives with a capacity of several terabytes or even more impressive RAID arrays.
B-trees require far fewer disk accesses than other types of balanced trees to perform the same operations. This is achieved due to the fact that the final objects in B-trees are hierarchically located at the same height, and the speed of all operations is precisely proportional to the height of the tree.
Like other balanced trees, B-trees have equal path lengths from the root to any leaf. Instead of growing upward, they branch more and grow wider: all branch points in a B-tree store many references to child objects, making them easy to find in fewer calls. Big number pointers reduces the number of the most time-consuming disk operations - head positioning when reading arbitrary blocks.
The concept of B-trees was formulated back in the seventies and has since undergone various improvements. In one form or another it is implemented in NTFS, BFS, XFS, JFS, ReiserFS and many DBMSs. All of them are relatives in terms of the basic principles of data organization. The differences concern details, often quite important. Related file systems also have a common disadvantage: they were all created to work specifically with disks even before the advent of SSDs.
Flash memory as the engine of progress
Solid-state drives are gradually replacing disk drives, but for now they are forced to use file systems that are alien to them, passed down by inheritance. They are built on flash memory arrays, the operating principles of which differ from those of disk devices. In particular, flash memory must be erased before being written, an operation that NAND chips cannot perform at the individual cell level. It is only possible for large blocks entirely.
This limitation is due to the fact that in NAND memory all cells are combined into blocks, each of which has only one common connection to the control bus. We will not go into details of the page organization and describe the complete hierarchy. The very principle of group operations with cells and the fact that the sizes of flash memory blocks are usually larger than the blocks addressed in any file system are important. Therefore, all addresses and commands for drives with NAND flash must be translated through the FTL (Flash Translation Layer) abstraction layer.
Compatibility with the logic of disk devices and support for commands of their native interfaces is provided by flash memory controllers. Typically, FTL is implemented in their firmware, but can (partially) be implemented on the host - for example, Plextor writes drivers for its SSDs that accelerate writing.
It is impossible to do without FTL, since even writing one bit to a specific cell triggers a whole series of operations: the controller finds the block containing the desired cell; the block is read completely, written to the cache or to free place, then it is erased entirely, after which it is rewritten back with the necessary changes.
This approach is reminiscent of everyday life in the army: in order to give an order to one soldier, the sergeant makes a general formation, calls the poor fellow out of formation and commands the rest to disperse. In the now rare NOR memory, the organization was special forces: each cell was controlled independently (each transistor had an individual contact).
The tasks for controllers are increasing, since with each generation of flash memory the technical process of its production decreases in order to increase density and reduce the cost of data storage. Along with technological standards, the estimated service life of chips is also decreasing.
Modules with single-level SLC cells had a declared resource of 100 thousand rewrite cycles and even more. Many of them still work in old flash drives and CF cards. For enterprise-class MLC (eMLC), the resource was declared in the range of 10 to 20 thousand, while for regular consumer-grade MLC it is estimated at 3-5 thousand. Memory of this type is actively being squeezed by even cheaper TLC, whose resource barely reaches a thousand cycles. Keeping the lifespan of flash memory at an acceptable level requires software tricks, and new file systems are becoming one of them.
Initially, the manufacturers assumed that the file system was unimportant. The controller itself must service a short-lived array of memory cells of any type, distributing the load between them in an optimal way. For the file system driver, it simulates a regular disk, and itself performs low-level optimizations on any access. However, in practice, optimization different devices varies from magical to fictitious.
In enterprise SSDs, the built-in controller is a small computer. It has a huge memory buffer (half a gigabyte or more) and supports many data efficiency techniques to avoid unnecessary rewrite cycles. The chip organizes all blocks in the cache, performs lazy writes, performs on-the-fly deduplication, reserves some blocks and clears others in the background. All this magic happens completely unnoticed by the OS, programs and the user. With an SSD like this, it really doesn't matter which file system is used. Internal optimizations have a much greater impact on productivity and resources than external ones.
Budget SSDs (and even more so flash drives) are equipped with much less smart controllers. The cache in them is limited or absent, and the advanced ones server technologies are not applied at all. The controllers in memory cards are so primitive that it is often claimed that they do not exist at all. Therefore, for cheap devices with flash memory, external methods of load balancing remain relevant - primarily using specialized file systems.
From JFFS to F2FS
One of the first attempts to write a file system that would take into account the principles of organizing flash memory was JFFS - Journaling Flash File System. Initially, this development by the Swedish company Axis Communications was aimed at increasing memory efficiency network devices, which Axis produced in the nineties. The first version of JFFS supported only NOR memory, but already in the second version it became friends with NAND.
Currently JFFS2 has limited use. It is still mainly used in Linux distributions for embedded systems. It can be found in routers, IP cameras, NAS and other regulars of the Internet of Things. In general, wherever a small amount of reliable memory is required.
A further attempt to develop JFFS2 was LogFS, which stored inodes in separate file. The authors of this idea are Jorn Engel, an employee of the German division of IBM, and Robert Mertens, a teacher at the University of Osnabrück. Source LogFS is available on GitHub. Judging by the fact that the last change to it was made four years ago, LogFS has not gained popularity.
But these attempts spurred the emergence of another specialized file system - F2FS. It was developed by Samsung Corporation, which accounts for a considerable part of the flash memory produced in the world. Samsung makes NAND Flash chips for its own devices and for other companies, and also develops SSDs with fundamentally new interfaces instead of legacy disk ones. Creating a specialized file system optimized for flash memory was a long overdue necessity from Samsung's point of view.
Four years ago, in 2012, Samsung created F2FS (Flash Friendly File System). Her idea was good, but the implementation turned out to be crude. The key task when creating F2FS was simple: to reduce the number of cell rewrite operations and distribute the load on them as evenly as possible. This requires performing operations on multiple cells within the same block at the same time, rather than forcing them one at a time. This means that what is needed is not instant rewriting of existing blocks at the first request of the OS, but caching of commands and data, adding new blocks to free space and delayed erasing of cells.
Today, F2FS support is already officially implemented in Linux (and therefore in Android), but in practice it does not yet provide any special advantages. The main feature of this file system (lazy rewrite) led to premature conclusions about its effectiveness. The old caching trick even fooled early versions of benchmarks, where F2FS demonstrated an imaginary advantage not by a few percent (as expected) or even by several times, but by orders of magnitude. The F2FS driver simply reported the completion of an operation that the controller was just planning to do. However, if the real performance gain for F2FS is small, then the wear on the cells will definitely be less than when using the same ext4. Those optimizations that a cheap controller cannot do will be performed at the level of the file system itself.
Extents and bitmaps
For now, F2FS is perceived as exotic for geeks. Even in your own Samsung smartphones ext4 still applies. Many consider it a further development of ext3, but this is not entirely true. This is more about a revolution than about breaking the 2 TB per file barrier and simply increasing other quantitative indicators.
When computers were large and files were small, addressing was not a problem. Each file was allocated a certain number of blocks, the addresses of which were entered into the correspondence table. This is how the ext3 file system worked, which remains in service to this day. But in ext4 a fundamentally different addressing method appeared - extents.
Extents can be thought of as extensions of inodes as discrete sets of blocks that are addressed entirely as contiguous sequences. One extent can contain an entire medium-sized file, but for large files it is enough to allocate a dozen or two extents. This is much more efficient than addressing hundreds of thousands of small blocks of four kilobytes.
The recording mechanism itself has also changed in ext4. Now blocks are distributed immediately in one request. And not in advance, but immediately before writing data to disk. Lazy multi-block allocation allows you to get rid of unnecessary operations that ext3 was guilty of: in it, blocks for a new file were allocated immediately, even if it entirely fit in the cache and was planned to be deleted as temporary.
FAT Restricted Diet
In addition to balanced trees and their modifications, there are other popular logical structures. There are file systems with a fundamentally different type of organization - for example, linear. You probably use at least one of them often.
Mystery
Guess the riddle: at twelve she began to gain weight, by sixteen she was a stupid fatty, and by thirty-two she became fat, and remained a simpleton. Who is she?
That's right, this is a story about the FAT file system. Compatibility requirements provided her with bad heredity. On floppy disks it was 12-bit, on hard drives- at first it was 16-bit, but it has reached our days as 32-bit. In each subsequent version, the number of addressable blocks increased, but nothing changed in its essence.
The still popular FAT32 file system appeared twenty years ago. Today it is still primitive and does not support access control lists, disk quotas, background compression, or others modern technologies optimization of data processing.
Why is FAT32 needed these days? Everything is still solely to ensure compatibility. Manufacturers rightly believe that a FAT32 partition can be read by any OS. That's why they create it on external hard disks, USB Flash and memory cards.
How to free up your smartphone's flash memory
microSD(HC) cards used in smartphones are formatted in FAT32 by default. This is the main obstacle to installing applications on them and transferring data from internal memory. To overcome it, you need to create a partition on the card with ext3 or ext4. All file attributes (including owner and access rights) can be transferred to it, so any application can work as if it were launched from internal memory.
Windows does not know how to create more than one partition on flash drives, but for this you can run Linux (at least in a virtual machine) or an advanced utility for working with logical partitioning - for example, MiniTool Partition Wizard Free. Having discovered an additional primary partition with ext3/ext4 on the card, the Link2SD application and similar ones will offer many more options than in the case of a single FAT32 partition.
Another argument in favor of choosing FAT32 is often cited as its lack of journaling, which means more fast operations writes and less wear on NAND Flash memory cells. In practice, using FAT32 leads to the opposite and gives rise to many other problems.
Flash drives and memory cards die quickly due to the fact that any change in FAT32 causes overwriting of the same sectors where two chains of file tables are located. I saved the entire web page, and it was overwritten a hundred times - with each addition of another small GIF to the flash drive. Have you launched portable software? It creates temporary files and constantly changes them while running. Therefore, it is much better to use NTFS on flash drives with its failure-resistant $MFT table. Small files can be stored directly in the main file table, and its extensions and copies are written to different areas flash memory. In addition, NTFS indexing makes searching faster.
INFO
For FAT32 and NTFS, theoretical restrictions on the level of nesting are not specified, but in practice they are the same: only 7707 subdirectories can be created in a first-level directory. Those who like to play matryoshka dolls will appreciate it.Another problem that most users face is that it is impossible to write a file larger than 4 GB to a FAT32 partition. The reason is that in FAT32 the file size is described by 32 bits in the file allocation table, and 2^32 (minus one, to be precise) is exactly four gigs. It turns out that neither a movie in normal quality nor a DVD image can be written to a freshly purchased flash drive.
Copying large files is not so bad: when you try to do this, the error is at least immediately visible. In other situations, FAT32 acts as a time bomb. For example, you copied portable software onto a flash drive and at first you use it without problems. After a long time, one of the programs (for example, accounting or email), the database becomes bloated, and... it simply stops updating. The file cannot be overwritten because it has reached the 4 GB limit.
A less obvious problem is that in FAT32 the creation date of a file or directory can be specified to within two seconds. This is not sufficient for many cryptographic applications that use timestamps. The low precision of the date attribute is another reason why FAT32 is not considered a valid file system from a security perspective. However, its weaknesses can also be used for your own purposes. For example, if you copy any files from an NTFS partition to a FAT32 volume, they will be cleared of all metadata, as well as inherited and specially set permissions. FAT simply doesn't support them.
exFAT
Unlike FAT12/16/32, exFAT was developed specifically for USB Flash and large (≥ 32 GB) memory cards. Extended FAT eliminates the above-mentioned disadvantage of FAT32 - overwriting the same sectors with any change. As a 64-bit system, it has no practically significant limits on the size of a single file. Theoretically, it can be 2^64 bytes (16 EB) in length, and cards of this size will not appear soon.
Another important thing exFAT difference- support for access control lists (ACLs). This is no longer the same simpleton from the nineties, but the closed nature of the format hinders the implementation of exFAT. ExFAT support is fully and legally implemented only in Windows (starting from XP SP2) and OS X (starting from 10.6.5). On Linux and *BSD it is supported either with restrictions or not quite legally. Microsoft requires licensing for the use of exFAT, and there is a lot of legal controversy in this area.
Btrfs
Another prominent representative of file systems based on B-trees is called Btrfs. This FS appeared in 2007 and was initially created in Oracle with an eye to working with SSDs and RAIDs. For example, it can be dynamically scaled: creating new inodes directly on the running system or dividing a volume into subvolumes without allocating free space to them.
The copy-on-write mechanism implemented in Btrfs and full integration with the Device mapper kernel module allow you to take almost instantaneous snapshots through virtual block devices. Pre-compression (zlib or lzo) and deduplication speed up basic operations while also extending the lifetime of flash memory. This is especially noticeable when working with databases (2-4 times compression is achieved) and small files (they are written in orderly large blocks and can be stored directly in “leaves”).
Btrfs also supports full logging mode (data and metadata), volume checking without unmounting, and many other modern features. The Btrfs code is published under the GPL license. This file system has been supported as stable in Linux since kernel version 4.3.1.
Logbooks
Almost all more or less modern file systems (ext3/ext4, NTFS, HFSX, Btrfs and others) belong to the general group of journaled ones, since they keep records of changes made in a separate log (journal) and are checked against it in the event of a failure during disk operations . However, the logging granularity and fault tolerance of these file systems differ.
Ext3 supports three logging modes: with feedback, organized and complete logging. The first mode involves recording only general changes (metadata), performed asynchronously with respect to changes in the data itself. In the second mode, the same metadata recording is performed, but strictly before making any changes. The third mode is equivalent to full logging (changes both in metadata and in the files themselves).
Only the last option ensures data integrity. The remaining two only speed up the detection of errors during the scan and guarantee restoration of the integrity of the file system itself, but not the contents of the files.
Journaling in NTFS is similar to the second logging mode in ext3. Only changes in metadata are recorded in the log, and the data itself may be lost in the event of a failure. This logging method in NTFS was not intended as a way to achieve maximum reliability, but only as a compromise between performance and fault tolerance. This is why people who are used to working with fully journaled systems consider NTFS pseudo-journaling.
The approach implemented in NTFS is in some ways even better than the default in ext3. NTFS additionally periodically creates checkpoints to ensure that all previously deferred disk operations are completed. Checkpoints have nothing to do with recovery points in \System Volume Information\ . These are just service log entries.
Practice shows that such partial NTFS journaling is in most cases sufficient for trouble-free operation. After all, even with a sudden power outage, disk devices do not lose power instantly. The power supply and numerous capacitors in the drives themselves provide just the minimum amount of energy that is enough to complete the current write operation. With modern SSDs, with their speed and efficiency, the same amount of energy is usually enough to perform pending operations. An attempt to switch to full logging would reduce the speed of most operations significantly.
Connecting third-party files in Windows
The use of file systems is limited by their support at the OS level. For example, Windows does not understand ext2/3/4 and HFS+, but sometimes it is necessary to use them. This can be done by adding the appropriate driver.
WARNING
Most drivers and plugins for supporting third-party file systems have their limitations and do not always work stably. They may conflict with other drivers, antiviruses, and virtualization programs.An open driver for reading and writing ext2/3 partitions with partial support for ext4. IN latest version extents and partitions up to 16 TB are supported. LVM, access control lists, and extended attributes are not supported.
There is a free plugin for Total Commander. Supports reading ext2/3/4 partitions.
coLinux is an open and free port of the Linux kernel. Together with a 32-bit driver, it allows you to run Linux on Windows environment from 2000 to 7 without using virtualization technologies. Supports 32-bit versions only. Development of a 64-bit modification was canceled. coLinux allows, among other things, to organize from Windows access to ext2/3/4 partitions. Support for the project was suspended in 2014.
Windows 10 may already have built-in support for Linux-specific file systems, it's just hidden. These thoughts are suggested by the kernel-level driver Lxcore.sys and the LxssManager service, which is loaded as a library by the Svchost.exe process. For more information about this, see Alex Ionescu’s report “The Linux Kernel Hidden Inside Windows 10,” which he gave at Black Hat 2016.
ExtFS for Windows is a paid driver produced by Paragon. It runs on Windows 7 to 10 and supports read/write access to ext2/3/4 volumes. Provides almost complete support for ext4 on Windows.
HFS+ for Windows 10 is another proprietary driver produced by Paragon Software. Despite the name, it works in all versions of Windows starting from XP. Provides full access to HFS+/HFSX file systems on disks with any layout (MBR/GPT).
WinBtrfs is an early development of the Btrfs driver for Windows. Already in version 0.6 it supports both read and write access to Btrfs volumes. It can handle hard and symbolic links, supports alternative data streams, ACLs, two types of compression and asynchronous read/write mode. While WinBtrfs does not know how to use mkfs.btrfs, btrfs-balance and other utilities to maintain this file system.
Capabilities and limitations of file systems: summary table
| File system | Maximum volume size | Limit size of one file | Length of proper file name | Length of the full file name (including the path from the root) | Limit number of files and/or directories | Accuracy of file/directory date indication | Rights dos-tu-pa | Hard links | Symbolic links | Snap-shots | Data compression in the background | Data encryption in the background | Grandfather-ple-ka-tion of data |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FAT16 | 2 GB in 512 byte sectors or 4 GB in 64 KB clusters | 2 GB | 255 bytes with LFN | - | - | - | - | - | - | - | - | - | - |
| FAT32 | 8 TB sectors of 2 KB each | 4 GB (2^32 - 1 byte) | 255 bytes with LFN | up to 32 subdirectories with CDS | 65460 | 10 ms (create) / 2 s (modify) | No | No | No | No | No | No | No |
| exFAT | ≈ 128 PB (2^32-1 clusters of 2^25-1 bytes) theoretical / 512 TB due to third-party restrictions | 16 EB (2^64 - 1 byte) | 2796202 in the catalog | 10 ms | ACL | No | No | No | No | No | No | ||
| NTFS | 256 TB in 64 KB clusters or 16 TB in 4 KB clusters | 16 TB (Win 7) / 256 TB (Win 8) | 255 Unicode characters (UTF-16) | 32,760 Unicode characters, up to a maximum of 255 characters per element | 2^32-1 | 100 ns | ACL | Yes | Yes | Yes | Yes | Yes | Yes |
| HFS+ | 8 EB (2^63 bytes) | 8 EB | 255 Unicode characters (UTF-16) | not limited separately | 2^32-1 | 1 s | Unix, ACL | Yes | Yes | No | Yes | Yes | No |
| APFS | 8 EB (2^63 bytes) | 8 EB | 255 Unicode characters (UTF-16) | not limited separately | 2^63 | 1 ns | Unix, ACL | Yes | Yes | Yes | Yes | Yes | Yes |
| Ext3 | 32 TB (theoretically) / 16 TB in 4 KB clusters (due to limitations of e2fs programs) | 2 TB (theoretically) / 16 GB for older programs | 255 Unicode characters (UTF-16) | not limited separately | - | 1 s | Unix, ACL | Yes | Yes | No | No | No | No |
| Ext4 | 1 EB (theoretically) / 16 TB in 4 KB clusters (due to limitations of e2fs programs) | 16 TB | 255 Unicode characters (UTF-16) | not limited separately | 4 billion | 1 ns | POSIX | Yes | Yes | No | No | Yes | No |
| F2FS | 16 TB | 3.94 TB | 255 bytes | not limited separately | - | 1 ns | POSIX, ACL | Yes | Yes | No | No | Yes | No |
| BTRFS | 16 EB (2^64 - 1 byte) | 16 EB | 255 ASCII characters | 2^17 bytes | - | 1 ns | POSIX, ACL | Yes | Yes | Yes | Yes | Yes | Yes |
The method of storing anything usually always implies some kind of orderliness, but if in human life it is not a prerequisite, then in the world of computers, storing data without it is almost impossible. This orderliness is reflected in the file system, a concept familiar to most users of different electronic devices and operating systems.
A file system can be compared to a kind of markup that determines how, where, and in what way each byte should be written to the media. The first file systems that appeared at the dawn of the electronic era were very imperfect, such as Minix, a file system that has a lot of limitations and is used in the same operating system Minix, which later became the prototype of the Linux kernel.
But time passed, new file systems appeared, more advanced and stable. Today the most popular of them, at least among Windows users, is NTFS, which replaced FAT32, which is now used only in small flash drives and has many disadvantages, the most significant of which is the inability to write files larger than 4 GB. However, NTFS is not without them. So, according to many experts, it lacks efficiency, performance and stability, therefore, the time has come to think about creating an even more advanced file system that can meet the growing demands from first server, and then client systems.
And so, in 2012, Microsoft developers introduced the Resilient File System, or ReFS for short, a recoverable file system positioned as an alternative to NTFS, and in the future, possibly its replacement. In fact, ReFS is a continuation of the development of NTFS, from which it was decided to remove all unnecessary things that never became popular, and add new features instead.
New in Resilient File System:
- Architecture using function (Storage Spaces)
- High fault tolerance. File system errors that led to data loss in NTFS will be minimized in ReFS
- Isolation of damaged areas. If areas of the file system are damaged, the recorded data can be accessed from under running Windows
- Proactive error correction. Automatically scans volumes for damage and applies preventative data recovery measures
- Automatic recovery subfolders and associated files when metadata is corrupted
- Using redundant writes to improve fault tolerance
- The maximum volume size in ReFS can reach 402 EB versus 18.4 EB in NTFS
- A file of 18.3 EB can be written to a file formatted in ReFS
- The number of files in one folder is 18 trillion. versus 4.3 billion in NTFS
- The length of the file name and path to it is 32767 versus 255 in NTFS
What will be removed:
- Data compression support
- Data encryption using EFS technology
- Extended file attributes
- Hard links
- Disk quotas
- Support for short names and object IDs
- Possibility of changing the cluster size (remains in question)
What will be inherited from NTFS:
- Access Control Lists (ACLs)
- Creating Volume Snapshots
- Mount points
- Reprocessing points
- BitLocker encryption
- Creating and using symbolic links
- Recording all changes occurring in the file system (USN log)
Currently, ReFS is in the early testing stage, however, computer geeks can appreciate the benefits of ReFS now, and on the client Windows system 8.1 or 10. To do this, you will need to perform the following registry tweak:
However, using ReFS on permanent basis Not recommended. Firstly, the system is still unfinished, and secondly, there is any possibility of conversion to ReFS and vice versa third party programs missing, thirdly, if you accidentally lose or delete files from a partition formatted in ReFS, there will be nothing to restore them with, since there are no data recovery programs that work with this file system yet.
Should we expect the implementation of ReFS in the near future? We can say with greater certainty that no. If it gets practical use, it will first be on server systems, which will also not happen soon, but users of client Windows will have to wait after that for at least another five years. Suffice it to recall the implementation of NTFS on client systems, and then it took Microsoft seven years. Well, the most important thing is that there is simply no special need for ReFS. When zettabyte disks appear on desktop computers, then perhaps the finest hour will come for ReFS, but for now we just have to be patient and wait.
Have a great day!