(Original at http://linuxmafia.com/pipermail/conspire/2015-November/008218.html .) Hey, Sam Varghese, would you like a posting on what you got wrong in your article 'Linux security: circling the wagons'? It's a good piece, with only minimal sensationalism, excusable on grounds of a need for adding some zing. But there are things you kind of blew and mischaracterised, including utterly missing the point and tenor of Jon Corbet's reference to the 1999 Mindcraft affair. You're still among my favourites, though, my good man! ----- Forwarded message from Rick Moen <rick@linuxmafia.com> ----- Date: Mon, 9 Nov 2015 18:36:21 -0800 From: Rick Moen <rick@linuxmafia.com> To: conspire@linuxmafia.com Subject: [conspire] Huge Nov. 5th Washington Post article on Linux security Organization: If you lived here, you'd be $HOME already. Four days ago, one Craig Timberg published a very, very long article in the _Washington Post_: 'The Kernel of the Argument: Fast, flexible, and free, Linux is taking over the online world. But there is growing unease about security weaknesses.' http://www.washingtonpost.com/sf/business/2015/11/05/net-of-insecurity-the-k... It's gotten a lot of notice, including quite a bit of deserved sarcasm, starting with the awkward reference to 'kernel' and photo of Torvalds at the top, even though the piece is almost entirely _not_ about the Linux kernel. The blunder is obvious and an easy target, but quite possibly not even mostly of Timberg's making: At most publications, article authors have little or no input into the choice of title and presentation. OTOH, Timberg _is_ guilty throughout the piece of being aware of 'Linux' as a kernel differing from 'Linux' as hundreds of different suites (distros) of the kernel plus thousands of userspace codebases, many of the latter cross-platform, and nonetheless blurring the distinction. This is important because if, say, you speak of a flaw on BIND9 or Apache httpd as a problem with 'Linux', you are being either clueless or dishonest or both unless you call it a problem with 'Linux, Windows, MacOS, and everywhere else these programs run'. Also, blaming the kernel for failing to prevent the flaws in badly written userspace applications is a serious failure of perspective unless carefully explained. To jump ahead for a moment, yes, this is one of the many things Timberg screws up -- especially the latter bit. Herewith, a brief effort to sort out the wheat from the chaff, and comment on both. (I don't promise I won't cut short and bail, or at least switch to doing this in installments, for reasons that will be apparent if you note the length and discursiveness of Timberg's piece.) critics warn that [Linux] has security weaknesses that could be fixed but haven't been So nu? This is true of every codebase ever written. Is Timberg ghostwriting some insider or insiders' peeve? Next paragraph: Worse, as Internet security has surged as a subject of international concern, Torvalds has engaged in an occasionally profane standoff with experts on the subject. Torvalds has had occasionally profane standoffs with just about everyone occasionally, but in this case I'm guessing that the people Timberg is ghostwriting for are mainly the maintainers of the PaX and grsecurity out-of-tree security patchsets. I'll mention in advance that I'm -- mostly -- a big fan of those patchsets and the three people who maintain them. (To my distress, starting Sept. 2015, stable patchsets of grsecurity are available only to paid customers and sponsors: https://grsecurity.net/announce.php ) https://pax.grsecurity.net/ https://en.wikipedia.org/wiki/PaX http://grsecurity.net/ https://en.wikipedia.org/wiki/Grsecurity One cynic's take on this who article might be that it's all about PaX Team / grsecurity people's sudden attempt to monetise their work. Hmmm.... There is a history of Torvalds and other core kernel coders having a difficult relationship with some dedicated security people, including the PaX and grsecurity teams. The latter tend to (IMO, justifiably) inflame this relationship by questioning the kernel's community's handling of security bugs and the accuracy of the kernel's documentation about security problems. Torvalds has often expressed the view that dedicated security people are oddballs, and he doesn't think the same way, or want to. And, seven short paragraphs in, Timberg switches without comment from speaking of Linux as a kernel to Linux as an operating system. Even though he was talking a paragraph before about the problem being 'Linus doesn't take security seriously', now it's: The rift between Torvalds and security experts is a particular source of worry for those who see Linux becoming the dominant operating system at a time when technology is blurring the borders between the online and offline worlds.... Eh? Why are the rift between Torvalds and _certain_ security experts of concern outside the kernel? But I need to ignore Timberg doing this, or I'd be stuck on it for a long time. Suffice it to say, he does this sort of thing throughout. Moving along: Even without a potential nuclear disaster, the stakes are high. Operating system kernels are the most essential code on any computer, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ allowing hardware to work smoothly with multiple pieces of software. Meaningless gibberish. Now, consider this: The Linux kernel runs on the New York Stock Exchange, every Android smartphone and nearly all of the world’s supercomputers. Much of the rapidly expanding universe of connected devices uses Linux, as do many of the world’s biggest companies, including Google, Facebook and Amazon.com. Since the only thing all those uses have in common is specifically the _kernel_ (e.g., Android uses its own libc, and even the kernel is pretty seriously non-standard), the rest of the piece should be about the kernel, right? It's not. Timberg has a long digression where he chides Torvalds's habit of ranting online at any kernel regular whose kernel patch breaks userspace. Timberg's point, the connection to kernel security, was unclear until we reach this bit: The effect of Torvalds's approach to managing the kernel -- defensive, gradualist, sometimes cranky -- chilled debate about the security of Linux even as it became a bigger, richer target for hackers. The result, critics argue, is that while Linux in its early days was widely considered a safer choice than Windows or other commercial operating systems, the edge has dwindled and perhaps disappeared. Excuse me, but Torvalds yelling at his immediate kernel-maintenance lieutenants over patches that break userspace has 'chilled debate about the security of Linux'? Such total bullshit. Someone, e.g., perhaps a PaX team member, has fed Timberg a passive-aggressive line and he fell for it. No, Torvalds yelling at his trusted lieutenants for breaking userspace has _not_ chilled debate about the security of the Linux kernel. E.g., the PaX and grsecurity maintainers cheerfully state their views on the Linux kernel mailing list without reservation, and they don't even get yelled at for doing so. The grsecurity and PaX patchsets have never had a prayer of a chance of being merged into mainline kernels. And why is that? Because they change kernel semantics and break some things. As usual, tightening security has serious costs. Also, the patchsets can be applied cleanly and functionally to only _certain_ kernel releases. This addresses, in part, the enduring rift between the core kernel team (not, as Timberg would suggest, just Torvalds) and dedicated security people: The core kernel team feel that the security people are 'crazy' in the sense of having little appreciation of the many interests that an OS kernel must serve and the style of governance required for a rapidly changing codebase. Torvalds and Co., with total justification, feel that if they ran the kernel the way the security people wished they would, progress would slow to a snail's pace, and a lot of functionality would simply break. You can already have a kernel with painfully careful attention to security. It also has a glacial development process, abysmal scope of supported hardware, and regrettably bad performance. It's called OpenBSD. What Timberg _could_ have reasonably said about 'debate about the security of Linux' is not that Torvalds 'chills' it -- which is utter nonsense -- but rather that it tends to be frustratingly un-fruitful because the core kernel people aren't security professionals and lack that mindset. Hence, the two sides of that discussion have little meeting of the minds, and tend to talk past each other. [Article quotes Matthew Garrett, now principal security engineer for CoreOS, as saying that cutting edge security work hasn't been applied to the mainline kernel, and so isn't as secure as man people would wish it to be, and then:] Versions of Linux have proved vulnerable to serious bugs in recent years. AshleyMadison.com, the Web site that facilitates extramarital affairs and suffered an embarrassing data breach in July, was reportedly running Linux on its servers, as do many companies. Those problems did not involve the kernel itself, but [...] Hey, Mr. Timberg? What did you just do there? The kernel hasn't had as good a security record as it might, says Garrett, and so just look at the compromise of AshleyMadison.com! Except, of course, that (very, very probably) didn't involve the kernel. (To the best of my ability to tell, the compromise vector used for AshleyMadison.com was never disclosed and might not even be known. In my experience of such cases, it's astronomically more likely to be some stupid administrative error, an inside job, or bad userspace code than a kernel security problem.) So, what are you saying, Mr. Timberg? He goes on: [...] but experts say the kernel has become a popular target for hackers building 'botnets,' giant networks of computers that can be organized to initiate cyberattacks. And your point is what, Timberg? By the way, as someone who's done security for a long time, the Linux kernel is, on balance, one of the components I worry _least_ about. And for critical systems, there are hardening kits (including PaX and, until just recently, grsecurity). Kernels are almost never the problem. The problem overwhelmingly is vulnerable, badly designed, and/or ridiculously overfeatured applications. It should also be noted carefully that the hardening strategies typically concentrate _mostly_ (verging on overwhelmingly) on making the Linux kernel better protect vulnerable userspace code from successful attacks from outside -- mitigating application exploits. Almost none of the work is directed at fixing kernel correctness and mitigating kernel exploit modes. Ironically, the only clear example of the latter I can point to came _from a mainline kernel coder_, Andrea Anrcangeli, and is a kernel feature (since 2.6.12, this past March) called 'secure computing mode' or 'seccomp'. It is -- finally -- a totally effective sandboxing security for application code. See: https://en.wikipedia.org/wiki/Seccomp This came from the Torvalds camp. It did _not_ come from the security malcontents who bitched to reporter Timberg. In the former category, protecting vulnerable application code from its own bugs being exploitable, the kernel already does quite a bit, such as ASLR (randomizes the physical and virtual address at which the kernel image is decompressed) and NX (No-eXecute bit marking stretches of memory that shouldn't be executed as not valid to be executed by the CPU). Though neither of these is foolproof. By the way, support for NX is one major advantage that's come painlessly with x86_64, and is yet another reason to lose those ancient IA32 boxen -- though the PaX patchset makes NX work on (some?) IA32 boxes. Kees Cook has been starting to coordinate information about both types of kernel-based protective techniques: http://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project And for four years, there has been an active and influential kernel-hardening mailing list: http://www.openwall.com/lists/kernel-hardening/ Timberg stresses that Torvalds doesn't want to merge a number of such out-of-tree security patchsets. OK, and your point, Mr. Timberg? Do you think everyone has to run a mainline kernel for security-sensitive applications? Because we don't. (Users overwhelmingly don't run kernel.org kernels _anyway_. They run vendor kernels, which typically merge some non-mainline security patches but not others.) Timberg's basic song is that kernel security could be better. Well, gosh, welcome to the real world, Mr. Timberg. Security of any codebase could always be better. The question is cost, in time, money, complexity, difficulty of debugging, adverse consequences on other code, and similar things. Security is always an economic calculation. Add hardening, add role-based access controls, reduce risky and unwanted functionality, and you put a much bigger burden on users and administrators to know what they're doing. This is a reason why the patchsets aren't default. This is why even the biggest adopters of SELinux (RHEL, Fedora) apply it only very tentatively ('targeted policy'). Timberg quotss Kees Cook as describing the horrors of what an attacker could do who 'gained control of the Linux kernel itself' without bothering to say that this is like saying 'If I had wings and really low gravity, I could fly.' [Grsecurity's Brad Spengler] soon demonstrated how nearly a dozen known Linux coding bugs could be exploited by malicious hackers to defeat external defenses and take control of the kernel. Yes? Where? I read LWN.net and that I seem to recall this is a serious exaggeration. Let's see, 2010 interview with Spengler: 'The only comments I make are on LWN (like when Ingo was spreading lies about the perf_counter vulnerability) or in exploit headers, because they seem to respond to that (or rather, the commercial influences care about the media publicity generated by such things, which causes them to improve security). It took 8(!) public exploits for null pointer dereference bugs to be fixed as a privilege escalation vector. Granted, after the first public exploit, mmap_min_addr was introduced some months later, but the Linux developers who aren't really security people aren't very good at introducing security features. mmap_min_addr has had at least half a dozen different bypasses since its inception for such a small, simple feature.' https://slo-tech.com/clanki/10001en/ So, misrepresentation by Timberg (probably out of lack of comprehension) of what Spengler showed. Layman's explanation: http://www.zdnet.com/article/linux-exploit-gets-around-security-barrier/ LWN.net coverage: http://lwn.net/Articles/341773/ In short, privilege-escalation. Privilege escalation is a Very Bad Thing, but is not the same as 'coding bugs [that] could be exploited by malicious hackers to defeat external defenses and take control of the kernel'. That is a gross exaggeration. It should also be mentioned that Spengler's beef was primarily with Red Hat's '2.6.18' vendor kernel (which the exploit sliced through without regard to SELinux), _not_ with kernel.org's tree, which had already been patched for 11 days to fix that very problem, when he published his exploit. See: http://lwn.net/Articles/341912/ Torvalds also resisted suggestions that security deserved a special place in the hierarchy of concerns faced by software makers. All flaws, in his view, were equally serious. Bullshit. This is not his view. This attitude was enshrined in a public posting in July 2008 that said: "I personally consider security bugs to be just ‘normal bugs.’ I don't cover them up, but I also don’t have any reason what-so-ever to think it's a good idea to track them and announce them as something special." This is simply not the same as claiming that all flaws are equally serious -- and Timberg either knew this when he made the claim or should have. But in his recent interview with The Washington Post, Torvalds rejected the notion that bugs could be usefully sorted into categories. "I refuse to waste a second of my life -- or any other developer's life -- trying to classify something that can’t be classified," he said. Torvalds has some significant justification for his view, as Timberg would know if he either studied security or consulted more people about the story than just a couple of security folk with grudges. I'm guessing Torvalds's view is that attempting to highlight a specific class of 'security' bugs would be futile because it's often very non-obvious at the time of discovery what bugs will turn out to have security implications, and the time required to make that determination would be better spent just fixing the bugs and moving on. And there are decades of history that tend (for the most part) to support Torvalds in this view. Despite providing a steady supply of defensive innovations, Spengler did not become a popular figure within the upper reaches of the Linux community, where he was seen as extreme in his views and sometimes brittle in his manner. Well, he is. His comments pretty much everywhere come across as sarcastic and bitter. There is no systemic mechanism for identifying and remedying problems [in the Linux kernel] before hackers discover them, or for incorporating the latest advances in defensive technologies. And there is no chief security officer for the Linux kernel. Nor is there a Santa Claus.[1] Chief security officers, where they exist, tend to be preplanned scapegoats for whenever the next security catastrophe occurs. Because, honestly, what organisation is make the guy actually the project czar? That just doesn't happen, so inevitably it's an advisory position without significant decision authority -- and a scapegoat. In the years since Spengler and others began warning about the security of Linux, it has triumphed in the marketplace. Google released its first version of the Android mobile operating system, which is based on Linux, in 2007, allowing Torvalds's work to reach hundreds of millions of smartphones each year. Google also made the kernel the basis of Chrome OS, which is used in an increasingly popular category of cloud-based computers called Chromebooks. Of course, these are significantly modified _forks_ of Torvalds's work. Now, watch for the hat-trick where Torvalds & Co. get slagged for not only the forked Google kernels but also for Google's notoriously lax userspace third-party ecosystem. Cue it in one... two... ...Three! Hackers are more likely to prey upon Oracle's Java and Adobe's Flash and Acrobat. All of which are extremely deprecated in standard Linux -- because they are security basket cases, and not safe to expose to arbitrary files on the Internet. (I don't know about Acrocrud on Android and ChromeOS, but I doubt it.) More recently, in 2014, Linux devotees were unhappy to discover that an Italian surveillance company called Hacking Team had swiftly turned a Linux exploit known as "towelroot" into a skeleton key capable of gaining access to hundreds of millions of Android phones. Which involved Android's failure to patch a June 2014 Linux kernel bug (CVE-2014-3153) in the futex subsystem (fast userspace mutex locking) that could be exploited by a local process to escalate to root privilege. All distros already had fixed kernels out by the time CVE-2014-3153 was published on June 5, 2014. It was a ghastly coding error, and a serious failure -- but the point is that standard Linux fixed it promptly. Why didn't Android? The answer to that is easy and has everything to do with the Google code ecosystem and nothing to do with Linux: It happened because the Android ecosystem has a horribly broken approach to code maintenance -- unlike standard Linux. This is less Google's fault than it is the institutional laxity built into Google's basic agreement with the Open Hardware Alliance OEMs, in which the OEMs are in the driver's seat and not Google. A hardened variant of CyanogenMod (which itself is a totally open source variant form of Android) now exists, called CopperheadOS: https://copperhead.co/android/ I don't know whether their hardened kernel would have evaded the rather ghastly CVE-2014-3153 bug, even though the project boasts 'a PaX kernel, OpenBSD's hardened allocator and compiler hardening features'. But for damned sure, the project's far more likely to actively respond to serious problems than regular Android in the field is, since the latter's functionally little different from 'Use what got preloaded until the device dies.' It should be mentioned in passing that kernel problems in Android don't even rate: The real problem in Android is the ubiquitous attitude of it being OK to run any old code, proprietary or not, out of anywhere. This guarantees security problems that dwarf anything involving the kernel. [skipping discussion of a August 2015 Linux Security Summit keynote address about lurking security problems by Linux Foundation sysadmin Konstantin Ryabitsev[0], and some reactions to that keynote] "We have some measures in place, although we are really not doing everything we can," wrote James Morris, maintainer of Linux's exterior defenses against attackers. That is _not_ James Morris's remit, by the way. Timberg once again did not bother to do basic fact-checking. Morris maintains the LSM (Linux Security Modules) subsystem, the kernel's interface for access control software such as SELinux.[1] LSM is sometimes inaccurately described as 'Linux's security subsystem' -- as if security can be accomplished by containing it within a subsystem -- so Timberg probably picked up the grossly inaccurate description from one of those. But it's yet another thing he flubbed by not doing fact-checking. Good article by Morris, putting LSM in context: https://www.linux.com/learn/docs/727873-overview-of-linux-kernel-security-fe... Timberg cites an encounter between Kees Cook and Torvalds at the 2015 Linux Summit, where Cook promoted some ideas from Morris's Linux Security Summit keynote, including some specific items from PaX / grsecurity, and: The ability to make the kernel execute code in user-space memory is exploitable. The best solution here can be hardware segmentation; Intel's "supervisor-mode execution prevention" and ARM's "privileged execute never" can both block execution from user-space memory. Instrumenting the compiler to set the high address bit on all kernel function calls can block calls into user-space memory (since the kernel's address space is at the upper end of the virtual address range, while user space is at the bottom). Kees also suggested emulating segmentation by using separate page tables for user mode and kernel mode; Linus jumped in at this point to say that this is the kind of idea that makes security people look crazy; such an approach would never perform well. He suggested avoiding talking about ideas that will clearly never make it into the mainline. http://lwn.net/Articles/662219/ It's speculative _how_ big a performance hit this radical idea would exact, but big, anyway. And Torvalds has a point: huge hits to performance just so the kernel can compensate for egregious coding errors in some applications would be wildly unpopular in _general-usage_ kernels. Timberg portrays this as if it were a questionable reaction: There was a revealing moment, however, when Cook raised the possibility of adding an especially intrusive feature long offered by Grsecurity. Torvalds immediately spoke up, saying this was "the kind of idea that makes security people look crazy," according to LWN.net, a site that follows Linux issues. Some 'security' patches don't have big performance hits or have serious functionality impact. To my knowledge, they get mainlined just as often as any other patches -- but part of the point of Torvalds's 'a bug is a bug' (and therefore a bug labelled 'security' is per-se just a bug) attitude that the security gang dislike is that just about _any_ bug might turn out to have security implications, so why privilege some just because of whom they arrive from, or what tag someone attaches, without other reasons? 'Security' patches certainly do get mainlined. They have an easier time doing so if credibly introduced on LKML, and easier if they are modest in scope rather than a huge stupendous change all at once, and easier if they don't break userspace. If those conditions aren't met and the security practitioner in question gets impatient, maintaining a patchset out-of-tree still works. Just ask the PaX / grsecurity people. Who, by the way, aren't even proposing their patches on LKML because they want to get paid mucho dinero for doing so (see: https://lwn.net/Articles/538600/), so it's not surprising they don't get mainlined. So, what's the complaint, again? Timberg closes with a comparison of the Linux ecosystem to the city of Troy, and rhetorically asks who's protecting the Trojans? Well, Mr. Timberg, those who want the occasional serious inconveniences and reduced performance of including the more-far-reaching security patchsets are doing so, on their security-sensitive systems. Those who care less about hardening, which is _mostly_ protecting badly written userspace applications from the consequences of those bugs, aren't. It's called choice and diversity. Some people aren't used to it. It's nowhere near perfect, and there are bugs in all code, and death and taxes also exist. And no surviving citizens of Troy, either. I'll let Spender have almost the last word, because even though he's a cranky-pants gadfly with a questionable sense of perspective, I like him anyway. Slides: https://grsecurity.net/spender_summit.pdf And I'm going to be curious what security-sensitive people do to compensate for the sudden restricted availability of grsecurity. But in conclusion: Sure, security in the Linux kernel is a huge, fat mess, there are problems that persist for rather too long, and there have been far too few attempts at _systemic_ improvement as opposed to the patch-and-pray treadmill. Same old. Timberg seems to _want_ to imagine a magic world where everyone can be made simultaneously happy with a single reformed/refactored kernel source tree arrived at without major delays and expense. Worse, he blames a bunch of things on Torvalds & Co. that are not within their remit at all. Worst of all, Timberg served mostly as a stalking horse for a little gang of security people with grievances but no real clue how to run a kernel project and why. These are _my_ kind of malcontents, but I'm aware that they're out of their depth when they suggest Torvalds & Co. are running the project wrong. Timberg isn't, and IMO got taken for a ride. We needed a good article about the perennial standoff between the core kernel people and security specialists. This was about 1/3 of one. There are real concerns. This article at least puts them on the table, though it does a poor job of discussing them. [0] See also http://kernsec.org/files/lss2015/giant-bags-of-mostly-water.pdf Highly recommended, and extremely funny & wise stuff. I have lived this. [1] It would be an excellent if antiquated gag to rename a village within the Commonwealth of Virginia to be called 'Santa Claus, Virginia'. Except, you'd never be able to find it with a search engine. [2] Morris is also maintainer of sVirt for virtualisation software and of the kernel crytographic API, has done work on SELinux, and does other security work in the kernel. I don't want to denigrate his work, which is important. He's a Red Hat employee living in Sydney. _______________________________________________ conspire mailing list conspire@linuxmafia.com http://linuxmafia.com/mailman/listinfo/conspire ----- End forwarded message -----