Phoronix benchmarks? More like Moronix Benchmarks.
|
| Author | Content |
|---|---|
| phsolide Nov 22, 2010 10:36 PM EDT |
There's so many things wrong with these Moronix benchmarks. First, they seem deliiberately spread out over as many pages as possible. Second, what in the sam scratch are they "benchmarking" anyway? There's never a significant difference between the test OSes. Bradley Chen et al didn't have any problems coming up with benchmarks that actually gave a difference for such wildly different OSes as NetBSD 1.x, Windows for Workgroups and WIndows NT 3.x (http://www.scs.stanford.edu/~dm/home/papers/chen:p5.pdf). Third, these "benchmarks" have no diagnostic value. Things like "lmbench" (http://www.bitmover.com/lmbench/) or even John Osterhaut's old benchmarks (http://www.stanford.edu/~ouster/cgi-bin/papers/osfaster.pdf) actually measured the basic building blocks of programs. Apparently, Moronix benchmarks just blur everything together. Bah, Humbug! |
| chalbersma Nov 23, 2010 2:20 AM EDT |
Crazy isn't it. I don't mind Phoronix making a few extra bucks by page trolling but the fact that their "benchmarks" are so pathetic is well, pathetic. |
| hkwint Nov 23, 2010 1:26 PM EDT |
But which of today's benchmarks _are_ interesting, if these ones aren't? Providing decades old examples is nice, but how about suggesting some current benchmarks instead? |
| dinotrac Nov 23, 2010 1:37 PM EDT |
@hans -- Not to mention some brownie points for trying to address the issue, and... remembering that meaningful is **always** in the eye of the beholder -- and , WRT benchmarks, somebody claiming otherwise clearly doesn't know much about them -- the Phoronix folks have at least provided some historical data by which one can measure successive kernels. |
| phsolide Nov 27, 2010 8:36 PM EDT |
Here's an example of a worthwhile benchmark: http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-ma... Unfortunately, this benchmark does exactly what the old "lmbench" stuff does as well. Phoronix probably has *not* provided historical data on successive kernels due to the benchmarks they concocted. They're meaningless because no benchmark ever shows a distinct difference. There's no data saying things like "yeah, Ubuntu Occidental Ocelot is a little faster than Bashful Beaver, but the statistics say that' isn't significant." Even dumb ol' election polls will say things like "5% margin of error" or similar. Phoronix doesn't say that. |
| gus3 Nov 27, 2010 10:34 PM EDT |
"A distinct difference" is exactly why I used dbench instead of Bonnie++. http://mindplusplus.wordpress.com/2008/07/14/finding-the-fas... (under the "Finding Benchmarks" heading) |
| Steven_Rosenber Nov 28, 2010 2:10 AM EDT |
Running the same jobs on different OSes and reporting the results - it's got to mean something. |
| dinotrac Nov 28, 2010 9:48 AM EDT |
Steven - Yup. |
| Sander_Marechal Nov 28, 2010 12:01 PM EDT |
Quoting:They're meaningless because no benchmark ever shows a distinct difference. Uh, yes they do. Phoronix isn't just running benchmarks. They have built a large testing framework and computerfarm capable of running many different benchmarks over many distributions and configurations automatically. It's even integrated with the Linux kernel's git repository. When there's a large performance drop (or increase) in a specific benchmark, the framework can automatically git bisect and rerun the benchmarks until it fins the exact commit that caused it. That's a huge contribution to development. It's been used quite succesfully in finding and killing performance regressions. Want better benchmarks? Write one, and give it to the Phoronix developers to integrate in their framework. |
| dinotrac Nov 29, 2010 12:01 PM EDT |
@Sander - What you said. |
| azerthoth Nov 29, 2010 12:54 PM EDT |
ah, he's looking for historical data in a format that doesnt make someone else have to go look up the data for kernel X, Kernel Y, and Kernel Z. phsolide the information you want is there, you complaining because it's not presented to you in a specific format. |
| chalbersma Nov 29, 2010 5:28 PM EDT |
Quoting: Uh, yes they do. Phoronix isn't just running benchmarks. They have built a large testing framework and computerfarm capable of running many different benchmarks over many distributions and configurations automatically. It's even integrated with the Linux kernel's git repository. When there's a large performance drop (or increase) in a specific benchmark, the framework can automatically git bisect and rerun the benchmarks until it fins the exact commit that caused it. That's a huge contribution to development. It's been used quite succesfully in finding and killing performance regressions. They do this for Linux and that's great for Linux but what about the other OS's they compare stuff too? For example, they never use the "correct" ways to configure various bits and pieces on FreeBSD. Half the time they use PC-BSD, which isn't bad if you want a desktop version of FreeBSD. Hell I use it for my desktop but if I'm running a web server will I really want all the services PC-BSD turns on by default running? Hell nah. Would I really want every sound driver and chipset driver loaded on every boot? No those options slow down the system considerably. For linux to linux comparisons phoronix is all right but every time it crosses over into *BSD *solaris or other non-Linux bits it never considers how it would be ran in real life. That's why to me, the phoronix benchmarks are just not up to par. |
| Sander_Marechal Nov 30, 2010 1:49 AM EDT |
I don't think I agree. All the Linuxes the benchmark aren't optimised either. They usually run however it was configured out-of-the-box (i.e. ext4 with logging turned on, which makes it slow). |
| chalbersma Dec 01, 2010 11:21 PM EDT |
They tested a beta of FreeBSD 8 with debugging symbols on for the kernel and compared it to Linux proper. |
| dinotrac Dec 01, 2010 11:40 PM EDT |
@chalbersma: >They tested a beta of FreeBSD 8 with debugging symbols on for the kernel and compared it to Linux proper. Did they now? And how do you know this? Did you break into their offices and check their secret files? Did you torture a contributor until he spilled the beans? Or did they just say so? If they disclose what they are testing and the basis for their comparisons, reasonable people should be able to provide appropriate grains of salt and view the tests in context of that information. |
| chalbersma Dec 02, 2010 1:00 AM EDT |
They admitted to it. Someone asked on their forums because in the test FreeBSD 8 came in just god-awful slow. Someone asked them for their kernel config and they said they used the Beta CD image provided with testing. Which comes with debugging symbols turned on by default. So unless they loaded up FreeBSD 8.1 Beta, used it to compile a brand new kernel without debugging symbols then loaded the new kernel they would have debugging symbols. Of course if they didn't have debugging symbols they'd have a kernel config file. Just a bit of that Razor. |
| dinotrac Dec 02, 2010 8:09 AM EDT |
@chalbersma - That's bad on them, then. Nobody should have had to to ask. |
| herzeleid Dec 02, 2010 5:36 PM EDT |
Quoting:they used the Beta CD image provided with testing. Which comes with debugging symbols turned on by default.They test everything "as is", "out of the box" with no fiddling, no special tweaks, no kernel rebuilds etc, and so if that's what's shipped, that's what gets tested. In other words, they don't treat linux any different from the other OSes they test. |
| dinotrac Dec 02, 2010 5:41 PM EDT |
herzeleid - >In other words, they don't treat linux any different from the other OSes they test. That may be so, but, if the FreeBSD really was a beta, that's information that wouldn't be obvious to many readers and should have been noted. |
| herzeleid Dec 02, 2010 8:02 PM EDT |
Quoting:That may be so, but, if the FreeBSD really was a beta, that's information that wouldn't be obvious to many readers and should have been noted.That may be true, but it's more of a nitpick than a heinous crime. |
| chalbersma Dec 02, 2010 8:43 PM EDT |
FreeBSD is not a no-tweak operating system. It's like installing gentoo without compiling. It's apples and oranges. It would be like buying a new sports car and testing it against an old muscle you just bought. The sports car was made to perform "off the line." When you buy that muscle car it's assumed that you have the know how to tune it yourself or the money to hire someone else to do it. So if you compared the new sports car to the old muscle car you would always come up with misleading values. See where I'm coming from? |
| gus3 Dec 02, 2010 10:55 PM EDT |
[rant] I see a certain "storage expert" doing comparisons of filesystems, and never once have I seen him try the different I/O elevators to see which one any given filesystem prefers. CFQ works well with the "native" Linux ext{2,3,4} FS's, but it's tailored for them and not for the filesystems that come from highly-reliable systems (JFS and XFS). In fact, on my desktop, XFS+CFQ at 2.5GHz performs worse than XFS+noop at 1.0GHz. Any system can be tuned by the distributors, to do most things reasonably well. If you want it to do some things very well, you'll need to make some adjustments. [/rant] |
| Steven_Rosenber Dec 03, 2010 11:25 PM EDT |
What I admire about Michael is that he had a vision for what he wanted to do, and he's doing it. He's not in it to slag BSD systems by purposefully running them in one state or another. Sure he's made some less-than-ideal choices when it comes to which releases to pit against each other, but he's doing this stuff week in and week out, and I really admire what he's doing. Of course anybody else is free to come up with their own benchmarks, or even take Michael's and modify them, and do this testing — or any testing of their choosing — and roll it the way they see fit. That's the beauty of open source. If you think you can do better, by all means do. The assumption that FreeBSD is some super-all-powerful "real" Unix system that totally outclasses Linux is just the kind of assumption that I want to see put to the test. Not that FreeBSD doesn't have attributes aside from processing or i/o speed, because it does, but for people out there deciding just which technologies to deploy at their workplace, business or other, this kind of benchmarking does help. |
| dinotrac Dec 04, 2010 12:23 AM EDT |
Steve - Amen. As somebody who has lived happily in both the FreeBSD and Linux world, I enjoy seeing both get some attention. |
| hkwint Dec 04, 2010 9:01 AM EDT |
Quoting:The assumption that FreeBSD is some super-all-powerful "real" Unix system that totally outclasses Linux is just the kind of assumption that I want to see put to the test. So do I! They should learn from .NET. Add to their license: One is only allowed to benchmark FreeBSD if _they_ can decide what optimizations one should enforce before benchmarking, and if they can see the results before publishing, and they can make one redo ones work with different config if they don't like the outcome. Oh wait, adding such restrictions isn't the idea of the BSD license... |
| Steven_Rosenber Dec 06, 2010 1:32 AM EDT |
I think the argument many have with Phoronix is that Michael is often testing beta builds of OSes that aren't the final versions and are slowed down by debugging flags and such. |
Posting in this forum is limited to members of the group: [ForumMods, SITEADMINS, MEMBERS.]
Becoming a member of LXer is easy and free. Join Us!