I asked about this in 1995...
|
| Author | Content |
|---|---|
| Bob_Robertson Dec 10, 2007 4:41 PM EDT |
The reply was, basically, this: Linux is a multi-user, multi-tasking OS. Unlike Windows, it does NOT try to use the first open space on a disk when you write a file, it looks for an open _block_ of space big enough for the file first. So each time a file is copied, if it was fragmented, it may not be after a while. Second, with a multi-tasking OS, having the r/w heads sweeping across the disk is a _good_ thing. Anyway, I haven't made any defragmentations in years of desktop use and continual updates/upgrades (Debian, don't you know) and have seen no degradation in service or disk speed or "excess disk access". |
| dinotrac Dec 10, 2007 5:56 PM EDT |
Wow, Bob, somebody gave you an utterly BS response. Sorry to hear you took it to heart. Maybe they were being serious, and doing the best they could in light of the limited options available under Linux. In mainframe days, we paid a lot more attention to disk I/O, because nothing could help a business application more than well-tuned I/O. Mind you, that reflects typical mainframe workloads -- not to mention mainframe hardware. In Unixland, cache and intelligent I/O ordering mitigate a lot of sins, but, even in Unixland, flying drive heads are a bad thing. |
| Igor Dec 10, 2007 7:57 PM EDT |
Just for your info. Found it here:
http://www.reactos.org/forum/viewtopic.php?t=4759 ---------------------------------------------------------------- Haos Lot of Linux file systems are highly fragmentation resistant. So highly fragmentation resistant that one defrag program for linux does not have any direct interface with the file system. All it does is scans for a fragmented file from inode information(Linux internal block display so file system neutral) and tells the filesystem to move it and its magically defraged. No special operation required. General usage can trigger defrags. Even just writing a extra block to a file can create a defrag event on some Linux filesystems. Linux will defrag threw general use more than windows. So far that Vista has a background defrag program to keep its fragmentation under control. Linux fragmention is a lot slower due to these minor changes. Note Linux installed on FAT32(yes you can) Suxs on fragmentation about as much as Windows. There are little design problems in the way windows handles files. Nothing not fixable in Ros while keeping support for Windows. It can take years at times to get Linux filesystem above 5 percent fragmented. Now you have seen the first reason. Fragmentation resistance in filesystems. Lot of people under linux reach the defrag point when the hardware needs replacing. So the idea of no need for defragmenting comes from the time to need to defrag comes around. Windows does have a common cause of fragmentation called restore snapshots and altering swapfile sizes. Reactos should support swap partitions outbox. There is a important difference on fragmentation by using them. Spliting of tmp to its own partition can also have positive effects on fragmentation. Theses are little tricks Linux and Unix have been using for a long time due to great positive effects. MS did have support for swap partitions back in Windows 2000. But it default option still was swapfile. Note HDD filled no room for play nothing can help you much then. Please note it is MS fault on that run as Admin in the docs of the Creator of NT design Admin was never ment to be created alone.(Hello Windows NT 3.5 default user Admin level no requirement to create the extra user) User and Admin account should have always be created. Note MS decide that was too user unfriendly and removed it. Now they have 1000s of programs that need it. Whos fault is it. Microsoft's. Buck stops with the person who decide to disobey the Creator advice. Support for fragmentation resistant file systems have to be up there on the wish list. |
| hkwint Dec 10, 2007 10:07 PM EDT |
Thanks Igor, that was interesting. I also read ReiserFS is 'self defragmenting', but only when you don't use the -noatime and -notail options. Probably fragmentation is mainly a problem of FAT partitions, and I consider them antique. Nonetheless, they're still used on media cards, USB sticks etc. and until recently it was the best way to share HD-partitions between UNIX/Windows I think. |
| gus3 Dec 10, 2007 11:11 PM EDT |
Quoting:they're still used on media cards, USB sticks etc.Where fragmentation presents near-zero performance impact. There is no head seek time. Only read-ahead optimizations result in a throughput hit, when non-contiguous reads can no longer be grouped together in a single request. |
| Sander_Marechal Dec 11, 2007 12:21 AM EDT |
Quoting:they're still used on media cards, USB sticks etc. You *want* fragmentation on those devices. As much as possible. In fact, the flash driver does just that underneath the filesystem driver. Fragment everything and scatter it all over the flash card. The reason is that Flash has only limited write capability. A sector wears out after 10.000-100.000 writes or so. So you need to spread it out and wear down the flash card evenly to prolong it's lifetime. |
| hkwint Dec 11, 2007 12:38 AM EDT |
gus3: You're right. I should have said fragmentation used to be a problem on FAT filesystems on old HD's. |
| gus3 Dec 11, 2007 1:01 AM EDT |
Quoting:Fragment everything and scatter it all over the flash card.Does this happen on the driver level, or does circuitry in the card take care of it? My (very shallow) understanding is the latter. |
| Sander_Marechal Dec 11, 2007 1:39 AM EDT |
I'm not sure. The only thing I'm sure of is that it happens below the filesystem level. Edit: Correction, it's both. It should be implemented at the filesystem level, but most removable flash devices (like USB sticks) have a controller chip on board that does wear-levelling so that a regular FAT32 filesystem can be used. http://en.wikipedia.org/wiki/Flash_memory#Flash_file_systems |
| Bob_Robertson Dec 11, 2007 3:32 AM EDT |
> Wow, Bob, somebody gave you an utterly BS response.
Well, from the rest of things, it seems that the only bs was the "heads traveling over the full disk is a good thing". I also ran a _lot_ of mainframe defrags, that and swapping tapes being the great labor on graveyard shift. Thus my concern from when I first put together a Linux system. I'm sure the conversation is still in the Debian-User mailing list somewhere, I may look it up. My recollection may have slipped a bit over time. {edit: not easily found through Google, bummer.} All that said, I would say that the kit gloves with which defraging had to be handled in mainframe world (big sign: Never Kill A Defrag!) and the obvious benefits of a fragment-resistant file system, I don't think I'm going to be worrying about it any time soon on my machine. |
| dinotrac Dec 11, 2007 4:13 AM EDT |
>Well, from the rest of things, it seems that the only bs was the "heads traveling over the full disk is a good thing". Sorry, that's what I was referring to. And yes, the Linux filesystems are far more fragmentation-resistant that FAT32, although they can still fragment over time, and, especially, as disks approach capacity. For example, my /usr directory structure is about 15% fragmented with 1.5 fragments per file. May not be too bad, depending on which files are fragmented how badly. Still, I hear the disk much more now than I did three years ago. Methinks some kind of action is in order. |
| Bob_Robertson Dec 11, 2007 4:33 AM EDT |
I ran the script too, 7.4% with 1.5 frags/file. I am now wondering if ext3 journaling is going to cause problems with a defrag. It might need to be run in minimalist single-user mode, like fsck. If you do, I'm interested in the tools/result. Please post. |
| Bob_Robertson Dec 11, 2007 4:41 AM EDT |
Darn, it says the same thing (mostly) that was said back in 95 and ext2. http://en.wikipedia.org/wiki/Ext3#Defragmentation I guess I should have noticed the article wasn't talking about ext3 and kept out of it... :^) |
| phsolide Dec 11, 2007 5:10 AM EDT |
It appears to me that "fragmentation" in the context of NTFS (and other extent-allocation filesystems, like ODS-11, and SGI's old EFS) has a different meaning than in block-allocation filesystems like BSD FFS, and modern derivatives, like EFS2, UFS, etc. A third meaning might come up in the context of log-structured filesytems (NetBSD's LFS). NTFS keeps track of the actual disk blocks that belong to a file by "extents": a beginning block and a run length. The old VMS filesystem ODS-2 did the same, which isn't too surprising given that NTFS has a lot in common at the disk layout level with ODS-2. Wink wink, nudge nudge. The BSD Fast File System (FFS) keeps track of individual block addresses that belong to a file. Small pieces of files could be kept in special "fragments", which were disk blocks that had pieces of up to 4 or 8 files in them. SGI's EFS, which to me looked like FFS with extents instead of individual block numbers in it, had the same "fragmentation" as NTFS. Except SGI gave everyone a defragementer out of the box, and had a crontab entry to run it once a week or so. So, a "fragmented" NTFS filesystem does not allow the allocation of large extents. A file header (inode) for a file will have a lot of 1 or 2 block "extents" associated with a file. A "fragmented" BSD FFS filesystem might have a lot of small files kept in the specially-designated "fragment" disk blocks. Of course, it could also have the same kind of fragmentation that NTFS exhibits, where the disk blocks allocated to a file might not be in any kind of sequential order. But the on-disk layout of an FFS filesystem almost guaranteed this, with it's "cylinder groups" and multiple pre-allocated inodes. There's huge semantic confusion around this issue: ensure that you speak of a common "fragmentation", and avoid BSing about non-equivalents. |
| jdixon Dec 11, 2007 5:25 AM EDT |
A quick Google search reveals that there is a defrag utility for ext2 and some other file systems. The latest version listed appears to be defrag-0.73. I have no idea how well ti would work with ext3. It appears to be included in Debian. One thing I would add is that while Linux may start fragmenting as the disk becomes full, it will still boot. I once filled up my hard drive and ran out of user disk space. The machine still booted and ran normally. If you get close to filling up a Windows system disk, you'll usually find that it will take 20 minutes or more for Windows to boot up, if it will at all. |
| Bob_Robertson Dec 11, 2007 6:21 AM EDT |
> One thing I would add is that while Linux may start fragmenting as the disk becomes full, it will still boot. Part of this is a standard 5%of the disk that is "reserved". So while it might reach the point of being full, it's never completely out of available space for the kernel and system processes. A Very Good Idea(tm, reg us pat off). Although, just a thought, with the bootable CD it becomes less of an issue. Rescue is possible through multiple vectors now. What's the best rescue disk for MS Windows? A Linux liveCD. |
| phsolide Dec 11, 2007 6:26 AM EDT |
Another data point: I ran the "frag.pl" script on a ReiserFS (V3) that has my home directory on it. I formatted that partition when I moved from Slackware v9 to 10.1. I think I had 10.0 on my test machine at the time, so when 10.1 came out I moved to it quite quickly. This particular machine is on 24x7, in my basement. So, two years of home directory usage, about. bash-3.1# ./frag.pl /home 6.80270065714389% non contiguous files, 1.61829190643809 average fragments. it's unclear to me what this "frag.pl" script measures, and a ReiserFS filesystem may produce gibberish results. |
| jdixon Dec 11, 2007 6:34 AM EDT |
> What's the best rescue disk for MS Windows? UBCD4WIN. Available at http://www.ubcd4win.com/ You need a valid license for XP, and in theory you can't be running that license at the same time you use the disk. |
| phsolide Dec 11, 2007 6:35 AM EDT |
Sorry, another thought occurs to me: IDE disks now support "LBA", logical block addressing. Do you really know if you have "fragmented" or "non-contiguous" files on a disk with LBA turned on? This same dilemma presented itself in the early 90s for BSD FFS systems like SunOS 4.x, when SCSI disks, which have always had something like LBA, became prevalent. The FFS "cylinder groups" where always a bit of a gamble, since SCSI disks could, and did, lie to you about heads/platters/sector size. Note that NTFS, which keeps *extents* in its moral-equivalent-of-inodes ends up suffering when it can't find numerically-adjacent logically-addressed blocks. The number of extents per file header/inode is limited, and if a file occupies enough small extents, NTFS has to allocate a new header/inode, right? Lots of traffic through the $MFT equals a bottleneck for NTFS performance. |
Posting in this forum is limited to members of the group: [ForumMods, SITEADMINS, MEMBERS.]
Becoming a member of LXer is easy and free. Join Us!