oh good
|
| Author | Content |
|---|---|
| tuxchick Oct 03, 2007 10:19 PM EDT |
So if the supposed experts and gurus can't even figure this stuff out, what the heck are we supposed to do? |
| gus3 Oct 03, 2007 11:28 PM EDT |
In most other fields, one can become an expert simply by studying what resources are available. However, in info. sec., it's quite the opposite: one needs professional instruction, with foundations in mathematics and psychology (viz. PEBKAC, aka "ID10T syndrome"). There is no such thing as "a little wrong" in info. sec. It's either done right, or it's an open door to an intruder. The SANS semi-weekly newsletter is proof of that. |
| tracyanne Oct 04, 2007 12:49 AM EDT |
The way he talks about pluggable schedulers I find confusing. I understood the idea of pluggable schedulers was that you plugged in a server optimised scheduler or a desktop optimised scheduler at install time, and that was that. But he's talking about pluggable schedulers as if you plug them in and out depending on the load at the time - doing that would indeed defeat the purpose - or is he just creating another straw man argument so he can tear it apart? |
| Sander_Marechal Oct 04, 2007 2:59 AM EDT |
Quoting:But he's talking about pluggable schedulers as if you plug them in and out depending on the load at the time Huh? Where did he do that? I think you misunderstood part of his post. |
| tracyanne Oct 04, 2007 3:58 AM EDT |
Quoting:I understand the "one size" argument, I just disagree vehemently about it having anything to do with a pluggable scheduler. The scheduler does have tuning, most of it 100% automatic (that's what the "fairness" thing is all about!), and none of it needs - or would even remotely be helped by - pluggability. |
| Sander_Marechal Oct 04, 2007 5:26 AM EDT |
I think you're confused by that part you quoted. The argument goes like this: A scheduler's job is to look at all your running applications and decide which application to run next. You cannot plug in one type of scheduler at install time and be done with it. Nobody uses *only* low-latency programs or *only* high throughput programs. So, if you use a pluggable scheduler that does only one thing, it will *always* be the wrong type of scheduler for some of the applications you have running. Not to mention that your computing habits can change over time (e.g. doing lots of low latency stuff now, but lots of high-throughput things a year from now). So pluggable-at-install schedulers is a bad idea. It's always going to be the wrong scheduler. What you need instead is one perfect scheduler that can look at an application and say "this one needs high throughput" and "this one needs low latency". Then balance all the different applications accordingly. And that is exactly what the current scheduler does. It's one scheduler that can do low-latency *and* high-throughput *and* a whole bunch of other things. And it's got a big knob on the front to make it do "a bit more low-latency" or "a bit more throughput". And the best part is that usually this knob turns all by itself, depending on if you do a lot of low-latency stuff or lots of high-throughput stuff. That is basically Linus' arguement and I fully agree with it. He doesn't talk about changing schedulers at run time. He talks about one scheduler that can do everything and then some. He does refer to changing scheduling at run-time at bit later in his post: Quoting:Sure you could make a multi-level scheduler with different pluggable ones for different levels, but that really doesn't work very well, since even in a multi-level one, you do want to have some generic notion of "this one cares about latency" and "this process is about throughput", so then the pluggable stuff wouldn't add any advantage _anyway_ - the top-level decision would have all the complexities of the one "perfect" scheduler you want in the first place! Basically his point here is, that the piece of code that would be responsible for changing schedulers is pretty much identical to a single, perfect scheduler. And that single perfect scheduler would be less complex. |
| dinotrac Oct 04, 2007 6:26 AM EDT |
>That is basically Linus' arguement and I fully agree with it. I'm sure that makes him happy, not that it makes either one of you right. An unfortunate side effect of playing God is that you convince yourself that you know better than the people who actually run your software. Linus's argument boils down to this: I can do a better job of deciding what you need on your computer even though I don't know the mix of work you run and don't know which things are most important to you. You will get what I care about and like it or else. |
| tuxchick Oct 04, 2007 6:40 AM EDT |
The part about schedulers isn't the bit that has me confused, but security. So Linus has decreed "LSM is in the kernel whether anyone else wants it or not, and in spite of the security pros who say it's not a good idea" is how it sounds to me. He rants about the security people not understanding filenames and inodes, ok, that's something Linus knows inside out. But the rest of the security discussion- it's all a confusing mess. |
| Sander_Marechal Oct 04, 2007 9:24 AM EDT |
Quoting:I can do a better job of deciding what you need on your computer even though I don't know the mix of work you run and don't know which things are most important to you. If you understood the argument, you'd see that pluggable schedulers are not the answer to the problem. They are fundamentally flawed. By definition. He doesn't say what the right solution is (aside from the mythical "perfect" scheduler). Just that pluggable schedulers aren't it. @tuxchick: My understanding of Linus' post is that schedulers are a technical problem. Security is a policy problem. You can solve technical problems but not policy problems. Therefor, pluggable security is a necessary evil. |
| dinotrac Oct 04, 2007 9:39 AM EDT |
>If you understood the argument, you'd see that pluggable schedulers are not the answer to the problem. I understood the argument and I don't see that at all, although that depends on the definition of pluggable you use. A true plug-in, something you swap in and out at will seems like a very bad idea. A scheduler is at the very heart of your system. You don't want instability. You don't want overhead. Both seem likely by-products of that kind of pluggability. A compile time choice, however, can address the desire without adding needless overhead or instability. From my standpoint, compile time choices would address most of the reasons behind the original selection. Obviously, it wouldn't do much for a machine that keeps busy 24 hours a day, with very different workloads day and night. |
| Bob_Robertson Oct 04, 2007 9:46 AM EDT |
> You will get what I care about and like it or else. And when has Linus' argument been anything else? What I hope comes out of this in the long run is a truly _tunable_ scheduler. Then it wouldn't be a matter of changing the code, but altering the parameters. I forget who was it on LXer who suggested that first, they mentioned it came from the mainframe environment. With IBM involved, it should be possible. Linus has said that the desktop is where interesting things are happening. That doesn't sound so much like "I know what's best for you" so much as "Get it right, then talk to me about it." |
| dinotrac Oct 04, 2007 9:49 AM EDT |
> I forget who was it on LXer who suggested that first, they mentioned it came from the mainframe environment. I know that I have mentioned my days as a performance tuner/capacity planner in big mainframe shops and how I longed wistfully for the control I had in those days. |
| Bob_Robertson Oct 04, 2007 12:35 PM EDT |
> ...and how I longed wistfully for the control I had in those days. That would be it, then. I had not perceived from the discussions that such things could be tuned "on the fly", but had to be hard-coded. Linus had said "schedulers are boring", but it sure has turned out to be an interesting series of discussions on the subject! |
| dinotrac Oct 04, 2007 1:10 PM EDT |
>That would be it, then. I had not perceived from the discussions that such things could be tuned "on the fly", but had to be hard-coded. Oh my no. We could change things by editing configuration files and we could sit down at the system console and change them via real-time commands. |
| Bob_Robertson Oct 04, 2007 1:28 PM EDT |
Hahaha! Those systems consoles. My first day on the job as an operator on a multiple-ES9000 system(s), the cadre of suits came in, and one of the systems programmers leaned over the console and typed a command. "Woops..." she said. A high-pitched wail issued from back within the sea of blue-and-white cabinets on the raised floor. A lonely, steady pezo-beep without the beep part. I sat there, dumb (a nod toward tuxchick), at the operator station, no clue how to react and dreading even the possibility that they might ask me to do something about it. Luckily, they all knew who I was even though I didn't know them, and a couple years later we all had a laugh about it when I mentioned how clearly I still remembered that empty feeling. The system programmer mentioned that it really was important to not leave a dangling "-" in a "vary offline 220-" command. :^) |
| dinotrac Oct 04, 2007 1:30 PM EDT |
>The system programmer mentioned that it really was important to not leave a dangling "-" in a "vary offline 220-" command. :^) You might even say it's vary important. |
| Sander_Marechal Oct 04, 2007 2:54 PM EDT |
Quoting:I understood the argument ... A true plug-in, something you swap in and out at will seems like a very bad idea ... A compile time choice, however, can address the desire without adding needless overhead or instability. I don't think you do. Linus' entire argument is against install/compile-time schedulers, because whatever scheduler you pick at compile time, it will always be the wrong one for some jobs. He then adds that, if you try to solve the install-time problem by making it dynamically plugable, that you end up with code that does the same as a single "perfect" scheduler, only vastly more complex. He also states that pluggable schedulers will not satisfy the desire for control over job scheduling. You get control by changing the parameters of the scheduler, not the logic. To make a nice analogy: pluggable schedulers are like having three car engines at home. One for slow-speed/high-pulling-power, one middle-of-the-road and one for high-speed/low-pulling-power. So everytime you change your driving behaviour (city trips, long distance highway, pulling a caravan) you need to take out the engine and put a different one in. What you want instead is a gear box - i.e. a tunable scheduler. Quoting:What I hope comes out of this in the long run is a truly _tunable_ scheduler. Then it wouldn't be a matter of changing the code, but altering the parameters. That is exactly what Linus is aiming for. Biggest problem today is that the tuning happens mostly automagically and that adjusting the scheduler manually is hackish. Bill Davidsen: Quoting:on desktops, response is (and should be king), while on a server, like nntp or mail, I will happily go from 1ms to 10sec for a message to pass through the system if only I can pass 30% more messages per hour Linus: Quoting:Bill, that's a *tuning* issue, not a scheduler logic issue. |
| dinotrac Oct 04, 2007 3:06 PM EDT |
> it will always be the wrong one for some jobs. Ah...control freaks of a feather hang together. The problem isn't that a scheduler is wrong for some jobs. The problem is that Linus wants to control which jobs a scheduler is good for and dictate to the world that they will share his priorities. PS: I spent ten years doing performance and capacity planning in some of the world's largest IBM mainframe centers. I also have my name on US and international patents for a work dispatcher/load balancer for a distributed call center system. Based on that background, I would, in all humility, like to think that I possess some modest comprehension of the tradeoffs involved in dispatching work. |
| Sander_Marechal Oct 04, 2007 3:22 PM EDT |
I'm not saying you don't know your schedulers or job control. I'm saying you're not understanding the point Linus is trying to make.Quoting:I know that I have mentioned my days as a performance tuner/capacity planner in big mainframe shops and how I longed wistfully for the control I had in those days. And how did you have control? By tuning the scheduler or by installing a different kernel with different scheduler logic? |
| dinotrac Oct 04, 2007 4:30 PM EDT |
>By tuning the scheduler Definitely by tuning, but that's a matter of mechanism and implementation. I believe that I have often expressed a preference for more tunability, but I'm pretty easy as to means if the ends are accomplished. After all, the Unix way of doing things and the mainframe way are very different, yet each manages to reach its destination. |
| Sander_Marechal Oct 04, 2007 10:20 PM EDT |
Quoting:I'm pretty easy as to means if the ends are accomplished. And Linus' point is that pluggable schedulers are not the right means to this end. Tunability is. And so far the only people arguing in favour of pluggable schedulers are people who are interested in the same kind of control you are. So far, nobody has given any other reason why a pluggable scheduler would be a good idea. |
| dinotrac Oct 05, 2007 3:42 AM EDT |
>So far, nobody has given any other reason why a pluggable scheduler would be a good idea. Actually, there has been a very good argument made, but it is derived rather than direct: People have actually been willing to code up alternative schedulers and try them out, ie Con's and Hey!! now we've got CFS. But I haven't heard of anybody doing squat WRT a tunable scheduler. I'm not in the loop for kernel-y things, so there might be an absolute firestorm of activity going on, but... You know, I seem to recall that, when Linux came out, the conventional wisdom (cough, HURD) was that any new OS should be built on a micro-kernel, that monolithic OS's were a bad idea. They might have been right, but it doesn't matter: here we have the very-popular monolithic Linux. Why do we have it? Because the people who were clucking weren't the people who were doing. In my book, some sort of pluggable scheduler that gets coded up and refined beats the crap out of tunable scheduler that gets talked about and never implemented. |
| Sander_Marechal Oct 05, 2007 3:55 AM EDT |
The existing scheduler already is tunable. It even autotunes itself. It's just not manually tunable because it lacks an interface. That's all. There is no need to "make it tunable". All someone needs to do is build a decent interface for it (something better than the existing /proc/sys/kernel/ interface). |
| jdixon Oct 05, 2007 4:12 AM EDT |
> You know, I seem to recall that, when Linux came out, the conventional wisdom (cough, HURD) was that any new OS should be built on a micro-kernel, that monolithic OS's were a bad idea. Having used OS-9, which was a sort of pre-micro-kernel micro-kernel, I'd say they were correct, and that a micro-kernel is better. However, that's not what Linus did or wanted to do. I believe Minix implements a micro-kernel, if anyone wants to compare them. A very close analogy can be drawn to the RISC/CISC debates. Everyone agreed that RISC was better, but CISC seems to have won the war by adopting all of the best RISC ideas. It seems to me that the Linux kernel has largely adopted most of the concepts of a micro-kernel (loadable modules, most things being in user space, etc.) and implemented them in its monolithic kernel. They're even trying to implement user space device drivers from what I understand. |
| dinotrac Oct 05, 2007 6:32 AM EDT |
>The existing scheduler already is tunable. To an extent, yes, and some of the remaining work may not be entirely scheduler-specific. Some of it may even be ignorance on my part -- which implies ignorance on the part of many people given that some folks have gone so far as to write alternate schedulers, an unlikely activity if the tuning info were easily available. At any rate, some things I don't know: How well does prioritization of CPU cycles map to prioritization of I/O? Protection of resident memory? How easy is it to assign certain tasks to the "I matter so don't mess with me" classification? How easy is it to assign tasks to the "He doesn't even want me to run, but hasn't managed to find the three-levels deep script that starts me up, so make me struggle for any cycle I can find and don't care if I get 99% degraded and 90% paged out" class? Stuff like that. If it's all there but no mere mortals can find it, it might as well not be there. |
| Bob_Robertson Oct 05, 2007 7:35 AM EDT |
One reason that even RMS uses to explain why HURD isn't ready so many years after it was begun, is because it turns out that message passing is a royal bitch to debug. Minix works because it is an exceedingly simple implementation, a "student's breadboard" as it were. |
| Sander_Marechal Oct 05, 2007 9:17 AM EDT |
Quoting:which implies ignorance on the part of many people given that some folks have gone so far as to write alternate schedulers, an unlikely activity if the tuning info were easily available. I don't think it's ignorance. They're probably just paining the bikeshed: http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/misc.ht... (quote below). Trying to create a new scheduler optimized for one single tasks isn't very difficult (compared to hacking other portions of the kernel) and technically a lot more interesting than doing the drudge work required to make the existing (and more complex) scheduler properly manually tunable. Quoting:“What is it about this bike shed?” Some of you have asked me. |
Posting in this forum is limited to members of the group: [ForumMods, SITEADMINS, MEMBERS.]
Becoming a member of LXer is easy and free. Join Us!