Wednesday, May 19, 2010

Vulnerability Market Numbers

Great idea out of unsecurityresearch to do an anonymous survey of vulnerability researchers to identify their experiences interacting with the groups advertising vulnerability purchasing programs and direct buyers (anyone who buys vulnerabilities but doesn't maintain an advertised program).  I'll summarize some of the interesting results below.

iDefense bought the most bugs in total and was the slowest to pay. ZDI was second in both of those, and the slowest to make an offer. They also ranked the highest in "trustworthiness" and preference to sell to. SecuriTeam was second in preference and ranked the highest in "friendliness" with iDefense finishing last there.

I also did an analysis of the numbers by importing the totals into Excel. I discarded any data with less then two quantified samples (numbers over the quantified limit weren't included). Below I've included charts for client side, server side, aggregate as well as percentage of purchases that were "high value" (ie, exceeded the survey threshold).

Note that there are lots of opportunities for improvement in the numbers. First, some buyers buy more specialized bugs (ie, for products with limited market share). These bugs would go for less money due to the market impact and drive the vendor's numbers downwards as they would appear "on average" to pay less money when in fact they might pay normal or above average for the same bugs that others would. Ideally one would shop the same bug to multiple vendors and compare offers and do this for multiple bug classes to get a much better comparison.

Second, since the researchers reported the data anonymously and the survey was advertised to a limited group opportunities for bias exist there. They could be only reporting more interesting bugs, or disproportionately represent a specialty (Oracle products for example) that would skew the results.

From looking at the data it appears likely that the insufficient number of samples is biasing the "Direct" numbers too low. I took out a single low sample and that moved the average Direct price up to ~$9,400, putting Direct in first place. Given the percentage of Direct sales that exceed the "high value" threshold, I would argue that Direct sales are probably the highest on average but we don't have enough data to show exactly how much higher.

For all the (legitimate) complaints out of the NoMoreFreeBugs community and others it's great to see the market reacting and creating both financial incentives and information regarding the market for sellers and buyers. The survey is available here. I would encourage anyone interested in this area to pass the survey on to any researchers you know as increasing the statistical sampling will significantly improve the quality of the data available. Let me know if you want the numbers or more information about the analysis.

Also for further research on the topic including vendors, a good briefing by Pedram and some papers and other material on the topic see my post from September

1 comment:

  1. You are very correct about the small sample size. Now that the link is more public the sample size should increase greatly, but so will the number of false responses.

    It will be interesting to see what the initial survey looked like vrs the results in a couple weeks.

    ReplyDelete

Share It