Why you can’t tell your applications are “OWASP certified”
..but still you have the tools to make them much more secure
A couple of days ago I went to visit a prospective customer interested in acquiring our cybersecurity services.
The customer is a small company from my city, whose SaaS services act as a middleware between the plethora of Public Administration Service providers and the citizens, who have then through them a single point of contact toward PA.
Besides being small, the company is highly organized and invested a lot of resources in the IT infrastructure and the development of the software platforms. Security as well wasn't overlooked, such that they already have in place a contract with a provider of cybersecurity services. Within the perimeter of such a contract, there is a periodic Vulnerability Assessment of the SaaS services.
I had a meeting with the founder and CEO, whom I was introduced to by a mutual connection.
While amusingly chatting, the CEO told me that the security of their applications was OWASP-certified. I didn't care much about it, since being a nontechnical guy he could have just misexplained the kind of verification service they are currently receiving.
Then the discussion went on, as we got to know better each other and tried to see if there was any room for cooperation. At some point, my interlocutor mentioned again the OWASP certification saying their services were OWASP Top 10 2017 certified. At that point, I was too eager to know more about the certification to not ask. And thus step in with some questions about it.
The CEO told me that the same company provided them with cybersecurity services and periodic vulnerability assessments, released after the first assessment a certificate stating the compliance of their services with the OWASP Top 10 2017. He also showed me the certificate and the web pages of their services, where they put the OWASP logo and stated their services were "secure" thanks to the "OWASP Top 10 2017 certification".
When I saw this, I first thought "Wow! What the hell the provider has been able to sell this customer: an OWASP Top 10 2017 certification!". I was amazed.
That's indeed amazing, for many reasons. To make the long story short and before entering into details of it, the key facts are that neither OWASP is a certification body nor it has defined any software security certification scheme. Additionally, the "OWASP Top 10" which is the list of the 10 most frequent attacks against web applications and that could eventually be used to mitigate the most frequent vulnerabilities, has been updated in 2021.
The purpose of this post is nevertheless to tell the full story clarifying a couple of points that might help many other software companies failing in the same situation described above.
Thus, I will touch on the following 5 points:
What is the OWASP Foundation?
What is the OWASP Top 10 and how can, if it can help, ensure software security?
What does it mean to certify a product?
Why it is challenging to certify software?
How can I ensure my software is reasonably secure?
Let's start then.
The OWASP Foundation.
OWASP is the acronym for Open Worldwide Application Security Project (formerly Open Web Application Security Project) and it identifies a nonprofit body (the OWASP foundation) that promotes software security. OWASP does so by supporting many initiatives (so-called projects) each one addressing some of the countless challenges people face every day in their attempt to make software more secure. OWASP is primarily a community-driven effort since the largest fraction of the work is done by people who grow the tools and frameworks that the foundation officially supports. Some projects, like the OWASP Top 10, the Web Security Testing Guide, the Core Ruleset for ModSecurity, the Zed Attack Proxy scanner, or the SAMM Maturity Model represent the area they cover a de-facto standard recognized by the individual professionals and on a general basis by the industry.
What the OWASP foundation is not, is an international standardization body that can promote a certification standard.
The OWASP Top 10
Top 10 is one of the de-facto standard projects supported by the OWASP foundation and probably is one of the most widely known also outside the security community. It is, in essence, a list, representing the ten most likely attacks against web applications. This list is built on top of empirical data and expert judgment.
Empirical data is provided by organizations contributing to data they have collected about web application vulnerabilities found in various processes. In 2017 organizations contributed data that covered over 114k applications, whilst for the 2021 data call, OWASP received data for over 500k applications. The Top 10 was first released in 2003, and the 2021 represents the 7th update. In 2024 a further update is expected.
The OWASP Top 10 is so popular that many vendors of security solutions for Web applications like Web Application Firewalls or Dynamic Testing solutions ensure coverage of the attacks in the Top 10.
Nevertheless, we need to keep in mind that the Top 10:
only provides a list of the ten most likely attacks against web applications;
that there are many other attacks beyond such a list;
that the attacks in the top 10 are only those statistically more likely. When I say statistically, it is because we must remember that the vast majority of web apps are derived from standard Content Management Systems. So the largest fraction of the attacks is targeting those CMS. Thus, while looking at the Top 10 we should ask ourselves if it represents an identically valid reference also in the case I need to protect my custom application.
the purpose of the Top 10 is to tell which are the attacks we will have most likely to defend our applications from. But it doesn't provide any element to verify how an application is resilient against them. Whether it is the purpose of other projects like the OWASP Testing Guide or the OWASP Application Security Verification Standard, asking to verify our applications are resilient against the Top 10 doesn't provide any guarantee or practical guidance of how such resiliency should be verified.
What does it mean to certify a product?
Obtaining a product certification means that a product has undergone a thorough evaluation and meets specific standards or regulations set by a certifying body or regulatory authority. The certification is essentially a formal declaration that the product complies with established criteria related to safety, quality, performance, or other relevant factors. The specific requirements for certification vary depending on the type of product, the industry, and the geographic location. Manufacturers seeking certification must carefully follow the guidelines provided by the relevant certifying bodies and regulatory authorities to ensure compliance with all necessary standards and regulations.
Obtaining a product certification typically means compliance with standards, and ensures the product has been tested and evaluated to ensure that it meets the predefined standards and regulations. These standards could cover a wide range of aspects, including safety, environmental impact, performance, and quality. It also means adherence to regulations, so that the product complies with legal requirements and regulations relevant to its industry and market. This may involve meeting specific criteria set by government agencies or industry bodies.
Certification also provides quality assurance, as it often indicates that the manufacturer has implemented and maintains a quality management system (QMS). This system ensures consistent product quality through defined processes and controls. Additionally, for products with safety implications, certification assures consumers, regulators, and other stakeholders that the product has undergone testing to ensure it is safe for use according to established safety standards. For products where consistent performance is critical, such as electronic devices, machinery, and medical equipment, the certification ensures the product has demonstrated reliability and performance according to specified criteria.
From a vendor perspective, achieving a product certification has generally a positive impact on consumer confidence, on market access (it might be the prerequisite for entering specific markets), can offer a competitive advantage, and may help to mitigate risks associated with product defects, safety hazards, and non-compliance with regulations, as well as liabilities or legal issues related to product performance or safety.
Certifying a product involves a series of steps to ensure that it meets specific standards and regulations. The certification process can vary depending on the type of product and the industry. As a general rule, certification first requires the identification of the relevant standards and regulations that apply to a product in the target market. This may include safety, environmental, quality, and other specific criteria. Testing to ensure that the product meets the requirements outlined in the identified standards and regulations must be then conducted. Testing may cover aspects such as safety, performance, reliability, and environmental impact. Evidence that the product complies with the established standards must be provided through the necessary documentation, including test reports, technical specifications, and other required information.
The necessary paperwork and documentation to the relevant certification body or agency. This may involve completing application forms, providing test reports, and paying applicable fees. While reviewing the application the certification body may also conduct on-site audits or inspections to verify that your manufacturing processes align with the standards. Corrective actions might be required to implement to address any non-compliance issues identified during
Once the certification body is satisfied that the product meets the required standards, it will issue a certification. This certification typically includes a mark or label that indicates compliance. After certification, it's important to continue monitoring and maintaining compliance with the standards. This may involve periodic audits, retesting, and updating documentation as needed.
It's important to note that the specific steps and requirements can vary widely depending on the industry, product type, and region in which you are seeking certification.
The case of Common Criteria for IT products.
When it comes to security certifications, the Common Criteria (CC) currently defines the highest standard of verification. CC is an international standard for evaluating and certifying the security features and capabilities of information technology products. Developed by a coalition of nations known as the Common Criteria Recognition Arrangement (CCRA), CC provides a standardized framework for assessing the security attributes of various software and hardware components.
The primary goal of Common Criteria Certification is to establish a consistent and reliable method for evaluating the security properties of IT products, allowing organizations and consumers to make informed decisions when selecting technology solutions. The certification process involves rigorous testing and assessment conducted by accredited laboratories, which evaluate a product's compliance with predefined security requirements.
Common Criteria is structured around Protection Profiles (PPs), which define specific security requirements for a given product or system. Vendors aiming for certification must demonstrate that their product meets the criteria outlined in the relevant PP. Additionally, products are evaluated against a set of Common Evaluation Methodology (CEM) guidelines, ensuring a standardized approach to assessment.
The certification process typically involves several stages, including a security evaluation performed by an accredited laboratory. This evaluation results in the production of a Security Target (ST), which details the security features and functionalities of the product. The ST serves as a basis for subsequent testing, and if the product meets the specified criteria, it is granted Common Criteria Certification.
Common Criteria Certification is widely recognized and accepted in many countries, providing a common language for expressing security requirements and capabilities. This helps organizations globally make informed decisions about the security of the IT products they deploy. The certification process enhances the overall security posture of information technology solutions, fostering trust and confidence in the products that undergo the evaluation.
Why it is challenging to certify software?
Looking at the certification requirements above and having on the other side in mind how modern software is built, what can be a typical software lifecycle also with several (in certain cases many releases in a single day), makes it straightforward to see that certifying the security of a software can be, on average, quite challenging.
First, If we think about the software behind a SaaS platform like in the case mentioned, we see it can be easily in nature highly dynamic, and frequently updated or modified. This dynamic nature makes it challenging to ensure that a certified version of the software remains compliant over time, especially if updates are frequent. Additionally, not only the final output (the software) but in modern software also the development processes, continuous delivery practices, and DevOps methodologies can be not stable and instead frequently updated. This might be also because the technology landscape evolves rapidly, introducing new tools, frameworks, and methodologies. Certification processes may struggle to keep pace with all these advancements and to adapt to the rapid pace of development and deployment.
A second challenge is represented by the fact software often interacts with other software and systems, creating a complex web of dependencies. Certifying one piece of software may not be sufficient if it relies on other components or if changes in the external environment affect its behavior.
Third, that software can run on various platforms (operating systems, hardware configurations, etc.) and environments. Certifying software for each possible combination can be impractical, and ensuring compatibility across all scenarios is a significant challenge.
Furthermore, many modern software applications allow user-generated content or customization. Certifying software with such user-driven elements adds complexity, as the behavior of the software can change based on user input.
It might significantly expand the number of scenarios on which to test software, including edge cases, unexpected inputs, and potential misuse scenarios. Ensuring that a software product is both functional and secure in all these situations requires sophisticated testing procedures.
So what we can do to ensure our software is, at least, reasonably secure?
Whilst certifying the security of software is challenging, we have indeed the chance to make it reasonably secure working to mitigate all the risk factors and the possible causes of insecurity.
First, let's consider where the source code comes from.
It can be either code developed by our development team or provided by third parties.
In the first case, two OWASP projects can be of great help. The Application Security Verification Standard (which I have in-depth described here) provides a guided approach to software verification, making available an exhaustive set of verifications according to which the various functionalities and components of a software can be tested.
Each verification of ASVS aims to verify and highlight a particular kind of weakness (many have also a CWE associated) or to prevent a specific kind of risk (e.g. that malicious code is included in the software) and can be carried out following the approach described in the OWASP Testing Guide, which provides a practical way to test web applications.
So, in practice:
the OWASP ASVS tells us thoroughly WHAT to test;
the OWASP Testing Guide tells us in depth HOW to test.
The OWASP ASVS and the Testing Guide are of course not the only two projects we can use and we might need to test Web Applications. There are other tools either inside or outside the OWASP community we can use to test for specific defects or vulnerabilities. Nevertheless, they do provide a broad enough picture on which additional verifications can be eventually added.
What instead about third-party software components?
One could decide to verify directly also the third-party software components,but before doing this we can first make sure that the 3rd parties providing them are reliable and that the processes for incorporating third-party components into our software are sound and secure.
The OWASP Software Component Verification Standard (SCVS) is an open project from the OWASP foundation that aims to support mitigating supply chain attacks.
I have already addressed SCVS in another post (here), but it is anyway useful to remind here that the standard is organized into control families, each one of which contains multiple controls that apply to different aspects of component verification or processes where component verification occurs. Families are six, and I'm reporting again here their names because they provide an immediate feeling of the aspects that the framework allows to cover:
V1: Inventory Requirements
V2: Software Bill of Materials (SBOM)
V3: Build Environment
V4: Package Management
V5: Component Analysis
V6: Pedigree and Provenance
What else?
Adopting standards like OWASP ASVS and SCVS within a development process would be impactful in terms of security. My personal experience unfortunately tells me that we are still away from their large-scale adoption thus adopting them within our team would bring it immediately above the average level. It is not enough but indeed is a first good step toward the mitigation of large-scale attacks.
Nevertheless, we could do more by looking 360° around all the aspects that are part of the development and release process.
For instance, the two standards do not cover secure design aspects (e.g. security requirements, threat modeling), even evaluating the skills of the software security team (architects, developers, testers, etc.) or again the level of awareness of software security issues among all stakeholders involved in the SDLC.
A comprehensive picture of this is captured by projects like the OWASP 5D framework, which promotes a practical approach to evaluating the maturity of an organization's software development life cycle from a security perspective. The framework covers all aspects of software security and provides a comprehensive approach to evaluate the maturity of an organization's SDLC, focusing on five key dimensions (SW Sec Processes, SW Sec Testing, SW Sec Team, SW Sec Awareness, SW Sec Standards).
One of the nice facts about the 5D Framework is that it has been designed to be natively aligned with industry standards such as ISO 27001, NIST, and OWASP SAMM, which are not certifications we can achieve and exhibit as such but are still standards and approaches we can follow to list and mitigate effectively all the possible sources of risk.
Which is indeed the real goal we must pursue.
Conclusions
This post has been ment to discuss implications of certifying software security and why it is at least challenging to certify software security in many modern contexts.
I shortly listed requirements and steps for certification of a product, also using the Common Criteria (that are the most prominent certification for security of IT prodcts) as a running example.
It emerges that certification is largely incompatible with the dynamic nature of the software and of the DevSecOps processes which are nowadays frequently adopted by many organisations.
Nevertheless, we can adopt a number of frameworks to shape the design, development, testing, and release processes in such a way that potential sources of risk are exhaustively listed and all addressed.
References
OWASP Top 10 - https://owasp.org/www-project-top-ten/
OWASP Software Assurance Maturity Model - https://owasp.org/www-project-samm/
OWASP ModSecurity Core Ruleset - https://owasp.org/www-project-modsecurity-core-rule-set/
OWASP Web Security Testing Guide - https://owasp.org/www-project-web-security-testing-guide/
Common Criteria - https://www.commoncriteriaportal.org/index.cfm