no, bromium will not kill all malware forever

over the weekend a discussion broke out on twitter (as discussions are want to do) about a somewhat overly optimistic article concerning the new anti-malware apple of the security community's eye: bromium.

the primary tactic that bromium uses (or at least the primary one that people focus on) is isolation/sandboxing. bromium's vsentry product uses virtualization on a per-process basis to isolate every process from the system and from each other. that level of granularity for isolation is a lot higher than most sandboxing efforts can give you. while there are certainly benefits to that granularity, there are also drawbacks.

perfect isolation is actually not desirable, we want and even need to be able to use the results of one process inside another one. the more sandboxes you have, he harder this is to manage. the folks at bromium have opted to address this issue using rule-based systems to decide what something in a sandbox can access as well as what to do with any changes that are left when the sandboxed process is finished. rules which, in all likelihood, the administrator can modify to suit their needs.

now, while the article in question is reasonably good at explaining what bromium's vsentry does, the author (jason perlow) takes the arguably naive view that this sandboxing technique can stop all possible malware (as evidenced by the article's headline: "Bromium: A virtualization technology to kill all malware, forever"). the reality, however, is that there are limits to what sandboxing can do, and as clever as the folks at bromium are, they aren't clever enough to deliver on the promise that headline makes.

that's a problem, because people are going to read that headline, see nothing in the article to actually contradict it, and believe that it's actually true. have we seen claims like that before? sure we have - saying it can kill all malware forever is not intrinsically different from claiming 100% protection. it's classic snake oil, only in this case it's not the vendor that's spreading it (as far as we know - we don't know exactly what the folks at bromium may have said to mr. perlow, only that they say the headline is his words, not theirs).

i suppose that should mean there's no problem, right? the vendor's hands are clean, after all. the snake oil is being spread by a third party. the vendor isn't doing anything about it in this case or previous cases that have arisen because, let's face it, they benefit from it. it's good for bromium's business if people think vsentry is better than it actually is, at least in the short term. in the long term, the kinds of mismatched expectations that creates are the same kind that the AV industry struggles with daily.

it is bromium's responsibility to control how their products are perceived, and by failing to take action they are giving tacit approval to the snake oil being spread on their behalf. their hands are not actually clean, they are dirty through negligence. however, i didn't really expect any better of them (though i did give them an opportunity to surprise me) and you probably shouldn't either. tread carefully - caveat emptor.

know your enemy: security vendors

just to be clear, i'm not suggesting that vendors are waging some kind of war against their own customers - they aren't (usually) that kind of enemy. but by the same token, vendors are not your friends either. when it comes to laying out strategies for protecting yourself and your stuff, it's important to know what category to place the various players involved, and vendors are best thought of as adversaries.

to better explain what i mean, imagine you're sitting around a table with your friends playing the classic board game monopoly. although these people really are your friends, in the context of the game, their goal is to win at everyone else's expense. in serving their own interests, they act in ways that don't serve yours and in fact may sometimes be in direct opposition to your interests. in this way it can be said that you and your friends have competing interests.

the customer and the vendor are generally not competing with each other in the conventional sense, but their interests are not aligned and in some cases the interests do compete. you as a customer have an interest in keeping your computers, intellectual property, banking credentials, etc. safe and secure. vendors also have an interest in that to a certain extent, but protecting you and your stuff is not a vendor's highest priority.

vendors are companies. as such their highest priority is the bottom line. without the bottom line, the company ceases to be. companies don't just start up out of thin air, they need money; which means they have investors and those investors expect a good return on their investment, or else it's not a good investment and they might not invest anymore in the future, or maybe even pull out their stake in the company. companies also have operating expenses. they need to pay to keep the lights on and the machines running, and they need to pay their employees who themselves have expenses (families they need to feed and put roofs over their heads). therefore the company has to make profit it's priority. the way vendors make money is by vending - they sell a product and the more product they sell the more money they make.

in theory if the product is good then they'll sell more of it, but it doesn't need to be good enough to stop all the threats to you or your stuff - vendors aren't competing with the bad guys, they're competing with each other, so they only need to be better than other vendors. what's more, since technical 'goodness' is difficult for customers to accurately quantify, the vendor only needs their product to be perceived to be good. technical quality is still required up to a point, of course, because you can't fool all the people all the time. but, since your buying decisions as a customer are based on perception, and that perception can be altered/manipulated more cheaply through marketing than through technological advancement, companies engage in this kind of shortcut to help them maintain or even advance their market position.

how does this compete with your interests as a defender of yourself and stuff? well, in a few different ways, actually:

1. by conventional falsehood, they make their product out to be better than it is and so draw you away from something that may actually suit your needs better (example: look at any vendor that's ever claimed to be able to take care of all/100% of any kind of threat)
2. by omission, they make solving your security problems seem easier than they really are because nobody wants to make the customer swallow a bitter pill about how much work is really involved in staying safe, especially when their competitors aren't doing it (example: how many vendors will tell you about what you need to do when their product doesn't work? how many will even talk about that scenario?)
3. by framing the issue, they make the customer think about the customer's security issues in the vendor's terms, thereby favouring the vendor's proposed 'solution' rather than formulating strategies to meet the customers own unique, individual needs (example: a number of anti-malware vendors used to provide generic detective controls in the form of integrity checkers, but those seem to be mostly gone now and vendors instead talk about technologies based on having varying degrees and types of knowledge about threats, while 'generic detection' (of a different sort) has become a glossed over, value added feature of their scanners)

all of these work against your interests in protecting yourself and your stuff. they work against you finding the best tool for your job, or figuring out everything you need to do, or even knowing there's more to it than just using the vendor's product.

before you get the wrong idea, i don't want you to think this is a condemnation of the people who work for vendors. individually, many of them may well be much closer to being your friend and being on your side than the company they work for as a whole is. their interests are never perfectly aligned with yours, of course. you won't see them sacrificing their own interests (their families, their money, their jobs) for your benefit, and you wouldn't really expect them to, would you? some of them (a scant few when you consider the total number that security vendors employ) will sacrifice some of their time and energy to help people (whether their company's customers or no) learn about the threats that are out there and thus be better armed against those threats. just because someone works for a vendor doesn't mean their character is a reflection of the character of the corporate entity that employs them. yes, companies are run by people but it's their collective behaviour that makes the character of the company. the phrase "none of us are as cruel as all of us" doesn't just apply to anonymous, nor does it just apply to cruelty. 

i also don't want you to think this is a condemnation of vendor companies either. remember, they're not exactly enemies in the conventional sense, but rather adversaries. as much as i tend to refer to them as bad actors, or irresponsible, or any number of other judgmental labels, i can't really see how they could work any other way. the judgments are really just a way of highlighting the divergence of interests between the vendor and the customer. there is some variation in the degree to which they do the the things that they do, of course. smaller companies are more easily influenced by noble ideals, in part because of size and in part because they have less at stake and so can afford to be more 'innovative' in how they operate. it doesn't always work that way, and it doesn't mean their bottom line isn't still the bottom line, but some take a more scenic route to their goals.

that being said, the fact remains that vendors' interests do not align with those of their customers (i.e. you). that means it's important to take what they say with a grain of salt and to evaluate whether the things they say or do or produce are really of actual benefit to you. pick over what they have to offer, take what you can use and throw away the rest. in essence, forage on the enemy.

the abc's of security

over the years i've found myself becoming increasingly dissatisfied with the boiler plate advice i formulated when i was younger, as well as all the other boiler plate advice i've seen/heard given by other people, and even the very concept of boiler plate advice itself. this includes things like best practices (aren't you done practicing yet?) and really any simple, prescriptive answer to questions involving how to keep oneself secure. more and more they seem like incomplete or obsolete anachronisms that aren't suited to the diverse and ever changing circumstances in the real world. never mind the fact that everyone's values (and thus their priorities) are slightly different from each other so boiler plate advice is rarely a really good fit - and of course people's priorities change over time, too.

i've grown and evolved as a security user (a user of security), and no boiler plate seems capable of reflecting my reality anymore. it's just not how i think about or approach the problem of keeping myself secure anymore and i find it difficult to direct others down such fixed, one dimentional paths.

and yet i know people still need advice and direction in order to grow themselves. the subject of first principles and fundamentals occasionally comes up and so i thought to myself what is the most fundamental thing in all of security? if there was just one thing about security that i could impart to another human being, what would it be? the answer is surprisingly simple, surprisingly complex, and surprisingly not limited to just security but in fact really a life lesson that happens to have meaning within security.

the most important thing for anyone to remember when it comes to defending yourself and the things and people you care about is this:

when i say this, i don't mean changing mindlessly like some derivative of the crazy ivan maneuver from the movie hunt for red october (although being unpredictable certainly has tactical advantages) but rather that you change what you do to protect yourself in intelligent, mindful ways. you should always be learning, always growing, always evolving, always adapting, always improving. don't stand still because your adversaries certainly won't be and you don't want to fall behind (or at least any further behind).

there are no easy answers, no matter how many people may be offering them (it seems like everyone does), and no matter how well-intentioned they may be.

debating AV effectiveness with security experts

a rather disheartening conversation took place on twitter over the weekend. as public conversations sometimes do, it grew beyond any capability i have to do it justice through description, so instead i'll provide some screenshots and links to a couple of branches of the discussion.

because i don't follow either dan kaminsky or robert graham, i knew nothing about this discussion until someone retweeted the tweet pictured below (i included as much context as i could):

what first made me take interest in this was that robert graham seemed to be talking about 2 different things as though they were the same. the AV that's only 4% effective (or 0% when he's done with it) is different than the AV that organizations pay 40% of their budget on.

the apparently ineffective AV is actually the scanner component of the AV; as you can see he describes his methodology for bypassing it - a methodology that essentially amounts to malware q/a, which happens to be a countermeasure against heuristic detection, which is a feature of scanners.

the AV that organizations pay 40% of their budget on (assuming that's an accurate figure, i wouldn't know) is the enterprise security suite, which includes other things beyond just the scanner. for example the tweet by dan kaminsky that seems to have started the entire conversation alludes to the failure of symantec's product to stop 44 out of 45 pieces of malware in the recently publicized attack on the new york times. but as symantec rightly pointed out, their product included a reputation system which, for all intents and purposes, behaves much like a whitelist - if something doesn't have a good reputation (and new things have no reputation at all) then it will be flagged. that is about as different from a traditional scanner as one can imagine and bypassing it isn't nearly as straightforward.

talking about 2 different "AV"s as though they were the same is symptomatic of not being able to see beyond the abstraction that is AV. conceptually AV is an abstraction that encompasses a variety of disparate preventative, detective, and recovery techniques. most people, however, just see AV as a magic box that you turn on and it just protects things. the only component that behaves anything like that is the real-time scanner, but it is not the only component in a security suite (especially an enterprise security suite) by any stretch of the imagination.

failing to see beyond the abstraction means, unfortunately, that you will fail to argue intelligently about the subject. it also means you will probably fail to make effective use of AV. if you don't know how a tool works, how can you possibly hope to use it to your fullest advantage? furthermore, how can you value something you don't understand? just as you can't price a car based on the effectiveness of the wheels, you shouldn't value AV based on the supposed effectiveness of the scanner.

one of the things that also became clear as i read some of the subsequent tweets was that robert seems to think there's nothing special about his attacks. but the fact is his attacks are special. as a penetration tester he launches targeted attacks. targeted attacks take more effort, more human capital to execute, and he himself described some of that extra effort. this basically means targeted attacks are more expensive to launch than the more automated variety and that they don't scale quite as well. consequently targetted attacks represent a minority of the overall attacks being performed. note, however, that that doesn't necessarily mean targetted attacks are a minority of the attacks a particular organization sees, as it's entirely possible that an organization may be a juicy enough target to receive a great deal of attention from targeted attackers.

from what i can tell, dan kaminsky also has difficulty seeing beyond the abstraction of AV. much of what he says above reflects the idea that AV is a magic box that you simply turn on and it should protect you. in the preceding example it appears he thinks there's an expectation that organizations solve all their security problems with scanners when in fact the expectation is simply that they have AV suites in their toolbox and that they use the appropriate tool (which may or may not be part of the suite) for the job (as rik ferguson attempted to explain).

dan also quoted the DBIR as showing AV to only be effective 3% of the time. i wondered about that so i looked a little deeper. DBIR stands for Data Breach Investigations Report. let the meaning of that phrase sink in a little bit. a data breach investigations report is a report about data breach investigations. data breach investigations are investigations of data breaches, and data breaches only occur when all the effort you put into preventing them failed.

you can't judge how successful something is by only looking at it's failures.

one of the consequences of this is that dan has actually failed to understand the statistic he's reported. the 3% where AV detected the breach still represents a failure because it was a detection after the breach had happened. this can happen due to things like signature updates (a scanner can detect more today than it could detect yesterday).

another consequence is that trying to use the DBIR to evaluate the effectiveness of AV represents a self-selected sample bias because the failure itself causes the event to be included in the study. a success would have excluded the event from the study. now one might have entertained the possibility that dan simply wasn't familiar with selection bias, but as we will see, that appears to not be the case.

it appears that dan has in fact heard of selection bias before, not to mention the WildList too (bravo). unfortunately it doesn't appear that he can use them properly.

• AV testers don't define the WildList, WildList reporters do (in a sense, but it's probably more accurate to say the the WildList is a result of a particular type of sampling)
• AV testers typically don't do WildList testing, although virus bulletin does offer a WildList-based certification in addition to larger, more inclusive tests
• if a product only detects 90%+ of the WildList, it's generally considered to be crap, because the WildList is the absolute bottom of the barrel of performance metrics. anything less than 100% is an embarrassment.
• AV testers don't define the set of malware they're going to test against, they cull samples from as wide a variety of real-life sources as they can and describe them as being 'in-the-wild' so as to distinguish them from malware that only exists in a 'zoo' or malware that was whipped up in a lab for testing purposes (something that's generally frowned on)
• defining what you're going to measure is not actually selection bias. "Selection bias occurs when some part of the target population is not in the sampled population" ("Sampling: Design and Analysis" by Sharon L. Lohr)  (now THAT's a textbook definition of selection bias - good thing was minoring in statistics in university). if testers defined their target to be the samples they already had then by definition there couldn't possibly be any selection bias because the target population and the sample population would be the same set.

this isn't to say there isn't selection bias in the tests performed by AV testers. it's entirely possible that some classes of malware (perhaps even targeted malware, for example) are harder to find samples of due to circumstances outside the testers' control. that being said, that bias is a lot more subtle than looking exclusively at failures.

now, it just so happens that i continued digging into the DBIR beyond just figuring out what went into it, and came across a rather interesting chart.

i highlighted the part that should really be interesting here. just to be clear, this only covers organizations that have to be compliant with PCI, but unless organizations that are legally obligated to run up-to-date AV are somehow magically more stupid than the rest of the organizations, the rest of the organizations actually have less motivation to run up-to-date AV and so their numbers are probably as low if not lower. 

now what this means is that there is really no way at all to use the DBIR to evaluate the effectiveness of AV because it appears that most of the organizations included in the report can't even follow the most basic of AV best practices. it also suggests that dan hasn't read his own sources thoroughly enough. if i'm not mistaken his 3% figure comes from the year when only 53% of PCI-bound organizations were running up-to-date AV. the subsequent year it was 1% of breaches discovered by signature-based anti-virus, and in the most recent one, AV doesn't appear to have helped after the fact at all.

that ever decreasing percentage of organizations running up-to-date AV is actually kind of disturbing, and it makes you wonder what's hurting organizations more; the amount of money they pay to AV vendors or the amount of attention they pay to security experts pontificating on subjects they are demonstrably ignorant of?

i anticipate that there will be those thinking that all i do is criticize and that i have nothing constructive to offer, so let's think about how we'd really measure AV effectiveness. the independent AV tests are apparently not good enough - in fact dan kaminsky went so far as to say this:

so how would we measure effectiveness really? like effectiveness in the field where AV is actually getting used? well first of all we stop limiting our data collection to just those instances where AV failed, because that's just ridiculous. no, we'll need to collect data on each and every time the AV raises an alert as well, to supplement our failure statistics. oh, and we'll have to follow up each of those alerts to make sure they aren't false positives because those certainly don't contribute to the effectiveness of AV. we'll also have to use all of the AV suite, rather than just parts of it, in order that people can determine if the effectiveness justifies the money they pay for the entire suite. additionally, we'll need to control for variables - different organizations have different security measures and controls that they deploy in conjunction with AV that may stop some malware before the AV gets a chance. that's not a bad thing, of course, and if they all used the same security measures then we could collect data on how effective AV is under those particular circumstances. but because each organization has different measures, they'll affect the AV results to differing degrees and that will skew the measurement. so we'll have to get organizations to either all use the same complementary measures or get them all to stop using any complementary measures. neither of which seem very likely in production environments, so that leaves us with trying to simulate what happens in the field - but then we get back into the AV testing lab territory which apparently 'no competent soul on the planet' trusts.

the reality is that it doesn't matter what kind of test you do, it's never going to match people's anecdotal experience. that's because testing is inherently designed around the idea of arriving at a single set of results representing how effective AV can be - it's never going to be able to reflect what happens when variables such as complementary controls, sub-optimal operation, targeting motivation, etc. aren't controlled for - and in the real world they aren't controlled for. tests necessarily reflect ideal circumstances and your mileage may (probably will) vary.

were i to be overly judgmental i might sign off this post with this little gem by none other than robert graham himself that i found yesterday will going through my RSS backlog:

The problem with our industry is that it's full of self-styled "experts" who are adept at slinging buzzwords and cliches. These people are skilled at tricking the masses, but they have actually zero expertise in cybersecurity.

but i prefer the school of thought from my own post about security experts from 2006 - security is just too big for anyone to be an expert in all parts of it. it seems to me that the expertise of dan and robert lie elsewhere.

it's important for people to recognize their own limitations, and i believe it's also important to recognize the limitations of the authorities you're listening to as well, lest you give credence to well meaning but uninformed experts. anti-malware is a complicated field, more complicated than i think either dan or robert realize, and if they have difficulty seeing beyond the AV abstraction imagine, how many other people do as well. i hope someday dan and robert and other experts like them can gain a deeper appreciation for how complex it is so that they can pass that along to those who depend on them to do the heavy cognitive lifting.

imperva's anti-virus study is garbage

Enough is enough! I have had it with these motherf#$%ing flakes on this motherf#$%ing train of thought - (what i imagine samuel l. jackson might say if he were following this nonsense about imperva)

in case you are unfamiliar, imperva (a security vendor of some sort) commissioned a bunch of students from the technion-israel institute of technology to perform an evaluation of the efficacy of anti-virus (all anti-virus as a whole, apparently, rather than comparing them to each other) by uploading 82 found samples to virustotal. yes, you read that right, it's another virustotal-based test.

these days i have a number of alternative avenues to express myself that i didn't have when this blog was still young, and that can often sate my need to express my feelings on some topic. i can make snide comments on twitter, or even parody tweets from a satirical twitter account. in fact i can even make memes about it. unfortunately none of that has proven sufficient in this case because the hits just keep coming.

you see, imperva keeps shopping this quackery out to more and more media outlets where it gets gobbled up and regurgitated uncritically by writers/editors (who really ought to know better if reporting on this sort of topic is part of their actual job) and thus gets put in front of more and more eyeballs of those who realistically can't know better. along the way it can even collect somewhat supporting voices from venerated members of the security community like robert david graham

or wim remes

let me be clear, however - this is all wrong. as has been repeated over and over again, virustotal is for testing samples, not anti-malware. they say it themselves in their about page

The reason is that using VirusTotal for antivirus testing is a bad idea.

and

BAD IDEA: VirusTotal for antivirus/URL scanner testing

those statements alone should be enough but, because virustotal later talks specifically about comparative tests, imperva (and others) have tried to argue that imperva's test is OK because it doesn't compare products to each other. however...

VirusTotal's antivirus engines are commandline versions, so depending on the product, they will not behave exactly the same as the desktop versions: for instance, desktop solutions may use techniques based on behavioural analysis and count with personal firewalls that may decrease entry points and mitigate propagation, etc.

this makes it pretty clear that the product a customer installs is very much a different thing from the program that virustotal uses - they will in most cases behave very differently and so the results that virustotal spits out cannot be considered representative of what actual users of anti-malware products will experience.

(ironically,  a product that appears to fare best in a virustotal-based test may actually be the worst because a higher focus on the type of static (often signature-based) detection that virustotal best measures could be to cover for a weakness in (or absence of) more generic/dynamic detection capabilities.)

but don't just take my word for it, let's hear from a couple of people who actually work at virustotal

yes, that's right, imperva's study is a joke. this shouldn't be surprising to long time readers of this blog since when i first wrote about this problem four years ago the first reason i gave for why you might want to avoid performing virustotal-based tests was that those of us who know better will laugh at you. i'm sure a number of people are laughing at imperva's gross incompetence (hanlon's razor makes me choose this explanation over the more sinister alternatives) but i'm afraid i can't consider the mess they're making to be a laughing matter.

promulgating ignorance in a security context has the potential to do real harm, and that is where i draw the line. that's why i'm writing this, that's why the title gets straight to the point, and that's why i'm going to start naming some names of people/organizations who have helped make this mess and who really ought to have known better. imperva has behaved like a dung beetle, persistently rolling this turd around, but somehow it keeps getting bigger like some katamari damacy of bullshit, and i think it's important to see the scale and scope of it and hold the people responsible accountable. it's worth noting, however, that somewhere deep down someone at imperva must also have seen the potential for their message to do harm - that's why the caveat that they weren't advising eliminating AV was added (as an apparent afterthought).

a non-exhaustive list of people/orgs who really should have known better, tried harder, and ought to be held to account for this growing mess is as follows:

• the technion-israel institute of technology because they presumably performed the test so that the test could have an air of objectivity it would otherwise lack if imperva performed it themselves
• the students who actually did the work because apparently no one bothered to read virustotal's about page
• the students' supervisor(s) who also failed the reading test and whose job it is to look out both for their students and the reputation of the learning institution that employs them
• imperva for coming up with this cockamamie idea in the first place and for later trying to defend it
• rob rachwald who posted the defense of the test on imperva's blog
• tal be'ery who helped with the defense of imperva's test
• amichai shulman (imperva cto) who is quoted far and wide by uncritical reporters
• clinton karr who seems to have written the press release that many others have simply regurgitated as their own reporting, perhaps as a result of getting that press release onto reuters
• the register which took the bait imperva offered
• richard chirgwin who posted an uncritical account of imperva's research
• techworld which took the bait imperva offered
• john e dunn who posted an uncritical account of imperva's research
• cso online which unthinkingly reprinted the techworld article
• the new york times (yes, that's right) who once again took the bait imperva offered
• nicole perlroth who posted an uncritical account of imperva's research
• the wallstreet journal who apparently uncritically summarize the work of others
• michael hickens who ran a summary of nicole perlroth's article as if it was gold inside an aggregate of summarized stories
• dark reading which posted a regurgitation of imperva's press release - is that what dark reading is supposed to be for?
• ihotdesk which took the bait imperva offered
• steven gaskill who posted an uncritical account of imperva's research
• pc magazine which took the bait imperva offered
• damon poeter who posted an uncritical account of imperva's research
• business computing world which took the bait imperva offered
• christian harris who posted a regurgitation of imperva's press release
• security bistro which took the bait imperva offered
• anthony m freed who posted an uncritical account of imperva's research

(i'm aware there's a lot more than this that you can simply find by googling sentences from the press release, i wish i had the time to make this list exhaustive - that said: reuters, the new york times, and the wallstreet journal... that definitely caught a lot of eyeballs)

now, perhaps you're thinking i'm being too hard on the journalists involved here. after all, they aren't experts. frankly, however, they don't have to be experts to see what's wrong with this test. if you're the type of reporter who reports on this type of technology then you should already know about virustotal and about how it can and can't be used. this isn't rocket science, or even some obscure nuance that only matters every 5th wednesday - not in the context of reporting on this subject. this is something reporters covering security technology ought to know. it's table stakes. you need to be this tall to get on the ride.

perhaps you think i'm being too hard on the students and their supervisor(s)? but this is academia we're talking about. they're expected to do their research, and i don't just mean the experimental research, i mean looking up and reading about the issues involved in designing and performing tests on anti-malware products. and their supeverisor(s) should have made sure they were doing their due diligence in this regard. frankly, in my time i've seen lone rank amateurs perform better tests than this with fewer resources. this is not acceptable academic performance.

and as for imperva themselves, well... if you intend to occupy part of the security industry that hopes to steal some of the AV industry's market, then you better know this stuff like it's the back of your hand. the institutional incompetence going all the way up the chain of command to the chief technology officer is astonishing and i'm surprised they managed to find someone with too many dollars and too little sense to give them funding, but i guess p.t. barnum was right about there being one born every minute.

imperva - do yourselves a favour and put a stop to this mess before it gets any bigger. you can't defend this junk computer science, the truth will eventually come out (it seems to have already started). you can't sweep it under the rug either, you've let things get too out of hand. the kind of smear campaign you're currently running was already attempted by the whitelisting industry years before you, and while that industry itself is still around and may even still be pumping out this same kind of junk, it didn't stop them from drifting back into obscurity. the way i see it the only way you can move forward sustainably is

1. admit your error
2. publicly retract your study
3. reach out to the journalists whose reputations have been tarnished by listening to you and apologize
4. assist the students you dragged into this in learning the error of the experimental methodology they followed (you can probably find a lot of good info either on or linked to from the anti-malware testing blog)
5. start over with a more intelligent methodology and try to make your case again with valid data

and if you can't manage to follow these steps then i'll be glad to watch you fade away or get swallowed up in a few years time, because the kind of incompetence you've been proudly displaying so far is not the path to success.

formula for the future

while reading over some of the prediction-posts that tend to spring up towards the end of the year it struck me that many of these predictions follow from pretty basic axioms. sometimes they're particular to the malware world but other times they're even more general and are simply combined with a malware concept to make a malware prediction. at any rate, i thought perhaps i should point out some of these things and help prediction-makers in the future, as well as to offer a different perspective on predictions of these types.

• the end of something as we know it - the only constant is change, you can never step into the same river twice (because having stepped in it the first time changed it), etc, etc. things are always different in the future than they were in the past. note that the prediction is rarely about the end of something but rather the end of something as we know it. all that means is that that something is going to change in a way that some people will consider significant.
• the dynamic equilibrium between focusing on software exploits and social engineering will continue to be dynamic - when exploits are hard to come by the bad guys will focus on social engineering to get the job done, because it is a job and they want to get paid. when exploits are easy to come by, focus on using them will increase because it's easier to fool an automaton consistently than it is to fool people consistently.
• software that has been popular to exploit in the past will continue to be popular to exploit in the future - bad guys will continue to focus on many of the same pieces of software because that's what their victims use and because they've had so much success there in the past. it's where the money is. good guys will continue to focus on many of the same pieces of software because that's what many people use and thus where the greatest impact can be made. in essence, so long as frequently exploited software remains popular among users it will continue to be a valuable target.
• we will learn more things about more things because of attacks - attacks are generally disruptive in one way or another. even if it's an attack that simply compromises confidentiality rather than availability, it still disrupts the process of using a system or service even if the system or service isn't itself disrupted. disruption has a tendency to highlight things that we would never have known if everything continued to run smoothly and the disruption had never taken place. disruption gives us an indirect view into things that might otherwise remain opaque.
• trends that are increasing will continue to increase, trends that are decreasing will continue to decrease - inflection points are rare. if they weren't it would be much more difficult to recognize trends in the first place. as a corollary, emerging trends will go mainstream. 
• the world is increasingly being made out of computers, and as new marriages of old-tech and computers become mainstream the resulting new-tech will become a target for attack - everything new opens up new possibilities for attack, and everything that is made new by sticking a computer in it opens up new possibilities for attacking that computer. furthermore, the more popular something is, the more profit can be had by attacking it, and so the more tempting a target it becomes.
• people will grow tired of defending themselves the same way they always have and try to find new alternatives - it is a quirk of human nature that we are always looking for new things. it is also true that we are largely unsatisfied with our current defensive capabilities (for whatever reason).
• new defenses will be developed to ward off new attacks and those defenses will be met with new offensive countermeasures - this is just the same offensive vs. defensive cat and mouse game that it's always been and always will be.
• new platforms will offer promise and seem secure, until they stop - figuring out how to successfully attack something without a lot of prior knowledge is difficult and time consuming, but it eventually happens, and the more it happens the faster additional attacks can be formulated until eventually we recognize that the platform wasn't the promised land we had hoped for.
• everything old will become new again - over time, as new people shift into a population and old people shift out, that population will collectively forget the past. at least for a while until someone figures out how to do new things using old concepts and then a renaissance occurs. we've already seen that occur with stealth, as well as boot sector infection/modification.

if you see the shadow of a prediction you've made in the list above, congratulations, you probably made a formulaic prediction. don't feel bad, you get better results using a formula than by trying to pluck the future out of thin air.

sector 2012

i wasn't sure i was going to post anything about my experience at sector this year. i mean, there comes a point at which you all must get it that it's a good security conference and you should all go, right? well, some thoughts were brought to the fore at the end that pretty much cemented the fact that i was going to post something about it so it might as well be within the larger context of my usual "this is what i saw/heard/thought at sector this year" post.

this year the conference was back in the metro toronto convention center's south building. that's where it was held the first time i went in 2008, back when sector was still small. it's grown considerably since then. before the conference i was thinking that i kinda preferred the old days when it was small. turns out part of that preference was a preference for the south building. sure, they may be sticking us in the deepest, darkest hole in toronto (actually, it's quite well lit) but the space is so much better (and the washrooms so much bigger - no more lines that stretch out past the door, with presenters begging to cut in line so that they can get things done and still make it to their own talk in time).

i pretty much avoided the vendor hall as best i could. unfortunately that meant i didn't spend much time checking out cool things like the lockpick village, but i really didn't feel like i had the tolerance for all that marketing this year. i will say, though, i thought symantec's choice of putting their name on a rubik's cube was  interesting. i don't know if it was their marketing department's intention, but associating yourself with something that seems like it should be easy but turns out to be fiendishly complex sends a really interesting message.

speaking of questionable marketing, dave lewis grabbed this shot of a flyer that was placed around all the tables in the lunch/keynote hall. in case you're not aware, in this context "flame" refers to the 'super surveillance software' that was apparently related to stuxnet. they're trying to say that their whitelist would have stopped flame, but since flame is said to have been able to spread through windows update and since people who use whitelists generally whitelist the binaries that come through windows update, i have a hard time buying their claim.

the first keynote of day 1 was an excellent talk about lawful access by law professor and copyfighter michael geist. like others, i found the statistic that ISPs handed over subscriber data voluntarily (without a court order) over 90% of the time to be pretty troubling, and i also think it genuinely calls into question the need for lawful access regulations. is that remaining few percent really worth trampling privacy without judicial oversight? i don't think so.

the first regular talk i attended was jamie gamble's talk about the vulnerabilities that time forgot. i was surprised to learn that this was actually a fairly *nix centric talk, and that while *nix had once earned a reputation of being much more secure than windows the reality now may well be the opposite because of all the advancements microsoft has made.

the next talk i went to attend was steve werby's talk about QR codes. unfortunately the talk didn't actually happen. however, i could already anticipate what some of the problems with QR codes probably were and charlie miller's lunchtime keynote on attacking NFC had a number of parallels to what i anticipated the QR problems to be. maliciously crafted QR codes that could exploit the reader code itself or QR codes that pointed to websites that exploit the browser.

that charlie miller keynote was quite entertaining, of course, but i can imagine some more creative ways he might have tried to surreptitiously read his friend's hotel key card than just holding his phone up to his friends arse as they walked around. maybe they wouldn't have gotten the phone into the 4cm range from the key, but even a low probability is better than the zero percent chance associated with not even making the attempt.

following the lunch keynote i went to gunter ollman's talk on threat attribution via DNS. i did this in part to test out a theory. when he blogs he mentions DGA (or domain generation algorithms) a lot, maybe even too much, and i wondered if that was going to come up in the talk. it turns out not so much. unfortunately he does seem to be somewhat softer spoken than a lot of the other presenters and when you combine that with the open door and people milling about and chatting outside the room it seemed he was unwittingly competing with background noise and not always winning. i may just give the video of the presentation a watch in spite of having attended it live.

after that i attended  michael perklin's anti-forensic techniques talk where i got a lot of ideas about what to do to make investigations too long and expensive to be of value if i ever turn to the dark side. also there were countermeasures, but i consider the chances of me performing forensic investigation even less likely than turning to the dark side. still, always interesting to hear about topics outside my usual comfort zone.

finally, for the last talk of day 1 i attended the introduction to web app testing talk by dave miller and assef levy. in part because i thought that it could be relevant to the day job and also because i wanted to get a taste for this new security fundaments track that sector was offering and this talk was slotted into.

as an aside, i think the introduction of the security fundamentals track points to the influence of the guys from the liquidmatrix security digest podcast, as they have a similarly named/themed segment on their podcast that i think really stands out compared to the other security podcasts i've sampled. unless, of course, the fundamentals track was introduced last year (when i didn't attend), in which case i suppose the influence traveled in the opposite direction.

day 2 of the conference started out with a keynote by jim reaves about global efforts to secure cloud computing. unfortunately, at that early hour and with that topic, i found my mind wandering to other things more often than not. i'll touch more on that soon enough though.

the first talk i attended on day 2 was jon mccoy's hacking .net applications, which was very interesting and i plan on sharing it with my collegues at work when it becomes available for viewing. thankfully jon handed out materials at the end of his talk so that i can share stuff before the video even becomes available (probably later today, or earlier today depending on whether one goes by when this is written vs when it's published).

after that i attended ed bellis' talk about the security mendoza line. it didn't really speak to me. oh well, you can't please everyone all the time.

the lunch keynote was kellman meghu's very humourous attempt to star wars into an allegory for security efforts within an organization. the empire, as you may recall, encountered some problems on their way to ruling the galaxy and there are a number of things they could have done better.

following lunch i attended steve werby's talk about busting hashes. in fact, i attended it twice - before and after the fire alarm was pulled (which seems like it could have been an excellent diversion for some nefarious activity). it was interesting to learn about how one actually approaches a task like that, as well as how cheaply it could be done using amazon.

finally, i attended chester wisniewski's talk about the blackhole exploit kit. this was the second talk from the fundamentals track that i attended and frankly i'm confused about it's classification as a security fundamental. for one thing, a single exploit kit is a very specific and narrowly defined topic for a talk classified as security fundamentals. further, in spite of being a student of the malware field for over 20 years, i still found a few things worth taking notes about. in fact, as i mentioned to chester afterwards, his talk was on par with malware related talks i'd attended in years past, before they even introduced the security fundamentals track.

now, i know some people may not pay much attention to which track a talk may be in, but i actually do and i suspect very strongly i'm not the only one. i pay attention because experience doing otherwise has taught me the value of  those classifications. talks in the management track bore me (i'm no manager), and i've found sponsored talks to be rather disappointing in the past. this new fundamentals track i'd interpret as being for when you know you're not strong in a subject that you want or need to know more about it, and the earlier fundamentals talk i attended about web app testing certainly bares that interpretation out.

so why did this particular talk (and i suspect the previous fundamentals talk on targeted malware, which i didn't get a chance to see but which at first blush also seems poorly classified) get put in the fundamentals pile? well, there are 2 main possibilities i can see. the first is that the sector folks got some really good non-fundamentals talks that they really wanted to squeeze in somewhere and they just happened to have space where the fundamentals talks were supposed to go. this certainly seems plausible, but in that case there really was no need to present them to attendees as though they were actually fundamentals just because they're taking up spaces that had been reserved for fundamentals talks. that just winds up giving people a false impression of what the experience of attending the talk will be like.

the other possible contributor is something i've actually seen a fair bit of in general security circles. there's an attitude or school of thought that says essentially "malware is old hat, we know this stuff already", and while that may be true for some, the fact that there are attendees, presenters (including this year), and even thought leaders who appear incapable of drawing even the most basic distinction between viral and non-viral malware and instead simply call everything viruses demonstrates pretty clearly that not everyone actually knows this malware stuff already. sure they're familiar with the malware phenomenon (who isn't these days) but there's a world of difference between familiarity with a subject and actually knowing it. i'm familiar with the television show "dancing with the stars", even though i've never watched it and can't possibly know very much at all about it.

and make no mistake, i'm not talking about obscure little details like the difference between keyloggers, screen grabbers, and form grabbers. viral vs. non-viral malware is one of the most basic and fundamental delineations you can make in the malware set. viral and non-viral malware are as different from each other as plants and animals - sure they're both alive and you can kill them both with fire, but the one that runs away is a heck of a lot harder to kill that way.

what i would like to see, what had my mind pre-occupied during the cloud security keynote, and what the introduction of fundamentals track made me think might actually work, is a true malware fundamentals talk - malware 101 if you will - because from my perspective it's needed. it's painful watching one presenter after another, one thought leader after another, one authority after another, all reinforce in the people trying to learn about security the mental model about malware that your mom and pop had back in the mid-90s. how effective has that mental model really been for your parents? has it empowered them to better control malware-related outcomes? i have a feeling it probably didn't, so is that really the mental model you want to foster at the "security education conference" of toronto?

unfortunately, confirmation bias and the dunning-kruger effect being what they are, i suspect any such fundamentals talk would fall on a lot of deaf ears (or not even be attended, as seemed to be the trend with fundamentals talks this year - they seemed to have the poorest attendance with the most walk-outs of all the talks i attended).

on exploit detection

data is code and code is data. i know that people like to think of data and code as being inherently different and separate from each other but in the end it's all just symbols in various languages on a real-world analog to the turing machine's infinite tape. fetching the next instruction, decoding it, and performing an operation based on what resulted from that decoding is not intrinsically different from fetching the next chunk of data, parsing it, and performing an operation based on what came out of that parsing. the ability to treat code as data is what allows us to distribute software, and the ability to treat data as code is what allows us to add new functionality to our general purpose computers that they weren't able to do before.

the distinction between programming languages and other input languages is simply that a programming language is intended to be used by programmers to create programs that are used by other people and other input languages are intended to be used by everybody else for (ostensibly) less complex purposes. it's really a matter of complexity, more than anything else, but it turns out that there are many input languages that aren't thought of as programming languages but are turing-complete and so are just as complex as any programming language. further, it's not just that there are two groups of languages (trivial and complex), but rather an entire spectrum.

with that in mind it's little wonder that data in the form of exploits can be just as bad as more traditional malware, but the implications go beyond just that. since data and code are not intrinsically different, since exploits are essentially malicious programs written in the input language for a vulnerable piece of software or hardware, some of the things we know to be true about malware should also hold for exploits.

specifically, the problem of deciding if an arbitrary input exploits a vulnerability seems like it should be reducible to the halting problem. i can certainly imagine trivial input languages where the presence or absence of an exploit is easily decidable, such as one where there are only 2 possible inputs, a legitimate one and one that triggers a vulnerability. however, in general, and certainly in no small part due to the existence of turing-complete languages, i'm fairly confident in saying that this problem is analogous to the virus detection problem. and THAT means that the problem of exploit detection, regardless of how it's approached in practice, is ultimately subject to the same limitations as the problem of virus detection.

to that end, testing exploit detection can run into the same methodological problems that testing virus detection can. for example, creating one's own exploits and testing against those instead of drawing exploit samples from the wild presents the same kind of problem that creating one's own viruses and testing against those would. namely that what is created in the lab does not necessarily correspond to what exists in the wild and so a product's ability to detect what was created in the lab doesn't necessarily correspond to it's ability to detect what's in the wild. certainly it's trivial to see how that would be true for products that detect known exploits using a method similar to known-malware scanning, but even for products that attempt to parse suspected exploits exactly the same as a vulnerable application would this would be the equivalent of emulating suspected viral code which we already know can't work all the time either (otherwise the virus problem would be decidable and we could then use it to solve the halting problem) so there could be exploits in the wild that elude such detection. as such we really should be regarding exploit detection testing that uses in-house exploits on equal footing as virus detection testing that uses lab-made viruses.

what is a vulnerability

a vulnerability is generally considered to be a mistake or oversight that allows the vulnerable program or system to behave in an unintended and undesirable way in response to a particular input.

alternatively, in the sense that the inputs a program accepts represent a language, a vulnerability is a condition where unanticipated functionality is exposed and can be called like a programming API by 'programs' written in the input language in question (otherwise known as exploits).

the occasional exposure of unwanted functionality is unfortunately pretty much an inevitability because the complexity of modern systems makes it next to impossible to anticipate all possible outcomes for all possible inputs. only the most trivial of programs can be made proof against this problem.

although vulnerabilities generally do occur as a result of a mistake or failure to anticipate something, it's also possible for undesirable functionality to be exposed intentionally. these are sometimes considered to be a kind of backdoor. however the vulnerability came about, the exposure of that functionality is unintended and undesirable to someone - be they the software vendor or the software consumer.

back to index