Having recently learned here at SoftSeq from a client how much money they spent in 6 months on a well-known crowdsourced bug-hunting platform, compared to what an end-to-end security audit would cost, forced us to share our thoughts. This is our first blog post, anywhere, ever.
What is crowdsourced bug-hunting?
It’s a process where a company publicly invites hackers from around the world to hack their product and pays for security issues hackers report. It’s a great tool when used properly. But the current tendency of businesses to forego systematic Security Engineering processes and audits in favor of crowdsourced bug-hunting makes it very expensive and inefficient for companies that misunderstand it.
Executive summary
Crowdsourced bug-bounty programs only make sense after a security audit, otherwise they’re expensive and ineffective.
Neither is a replacement for proper security engineering process during software development.
Core issues of bug-bounties
Cost. In an unprepared, medium sized web application, bug-hunters on average find 50 issues during the first 6 months of private program running, with mean pay-out of $500 per bug. This is a total of $25000 per medium-sized web application – about twice as much as an end-to-end security audit.
Crowdsourced bug-bounties also have a number of non-financial problems, like:
- Intentionally limited scope to manage costs. It excludes most wide-spread vulnerabilities, and many security controls and settings a secure software should implement, that would be found in your app with high certainty.
- Bug hunters won’t spend time understanding your business, to see what data and processes are business-critical. Their primary focus is what you pay for – technical vulnerabilities, and this limits bug-hunting’s effectiveness.
- Leveraging groups of out-of-scope and lower severity security issues won’t be explored, leaving a large security exposure.
- Production logs will light up, preventing effective Security Incident and Event Management based on log analysis. Bug-bounties are normally run on production domains, meaning when something fishy starts happening you won’t know if it’s an evil hacker or a white-hat bug-hunter.
Software Security Auditing
A deep technical security audit of a medium-sized web application costs around $10,000 to $15,000 – half as much as a comparable bug-hunt. More importantly, it results in a fully-transparent report on all features tested and code reviewed, with both positive and negative findings recorded – not just the bugs. Being performed under an NDA, with access to application source code, and in cooperation with product’s developers, audit findings are far superior to what’s possible to uncover in the same time-frame via bug-hunting.
Here’s a real-life anonymized OWASP ASVS compliance audit report from SoftSeq for a medium-sized web application (bugs are reported separately and linked to the spreadsheet). It lets those technically inclined to assess the depth and breadth of a software security audit.
Proper Software Security Engineering
Neither security audits nor bug-hunts tackle another major cost component related to security bugs – the cost to fix.
Proper Software Security Engineering happens during development, combining manual and automated security processes with clever risk assessment and management. This enables Secure Software Development Life-cycle, describing which is way beyond the scope of this article. I may cover it another time if this topic gets any traction.
Getting the most out of bug-hunting
Bug-bounties can be a very cost-effective addition to the security program – if you come prepared. For a well tested application, with most security issues found and fixed, crowdsourced bug-hunting can bring attention of many professionals from around the globe on the cheap. Finding 95% of security issues before bug-bounty starts can cut its cost from $25000 down to about $1000 in the first 6 months – which is a bargain for the attention you get.
So, why bounty payouts reach dozens or even hundreds thousand U.S. dollars?
The answer is in psychology and cognitive biases, specifically – hyperbolic discounting.
A security audit has a calculable cost at the start, an objective figure here and now. On the other hand, a bug-bounty has a starting cost of $0 and a seemingly small pay-as-you-go fee per bug found. Thus, bug-bounties feel cheaper at the start than security audits due to hyperbolic discounting, leading companies to behave irrationally in respect to their security expenses.
It feels even better to do nothing for security and hope for the best. It has a $0 cost, with no complexity and time requirements. Even in 2018, this seems to be the thinking preferred by most companies.
Don’t let your business fall prey to human biases.