A Framework for Open Source Evaluation

The question is not any longer if a project is open source or not, the question is how much open a project is.

Open source is everywhere, but so is fake open source. Recently there have been more and more cases of open source projects turning into non-open source, at the same time there are examples of non-open source (as per OSI definition) projects building communities as if they are open source. How is that possible, aren’t open source projects supposed to remain so for life? 

Open source is not black and white, it has multiple dimensions of openness, transparency, collaboration, and trust. For some open source is any project on Github, for some it has to pass the OSI definition, and for some, there it has to comply with the unwritten but commonly accepted open source norms. Here I’ll share with you how I think about evaluating open source projects by looking at some of the business and technical aspects first and then exploring the community management customs.

Disclaimer

• These are my personal opinions and not of my employer or the software foundations and projects I'm affiliated with.

• This is not a legal or professional opinion (I’m not a lawyer, nor specializing in OSS evaluation), but a layman’s opinion.
  Update: I got feedback from multiple open source lawyers and updated the article!

• This blog post is sponsored by the subscribe and share buttons, hit these buttons to express support.

Intellectual Property

The very first question to ask about an “open source” project is about the ownership of intellectual property. The good news is that, even without understanding these legal implications, there is a simple Litmus test you can apply. Does the project belong to a reputable open source foundation that you trust? For example, FSF owns the copyright for the projects it host, and more often, foundations (such as ASF, LF) aggregate the rights to license contributions to their projects, through contributor license agreements. In either case, you can trust that they will act as good decentralized stewards and won’t change a project’s future direction overnight. If a project doesn’t belong to a reputable software foundation but is backed by a single company, the question is if you trust the company as a supply chain partner. If the answer to these questions is yes, move to the next section. If the answer is no, then you better investigate and find out who the copyright owner is, and what are their long-term prospects and potential risks for you. A single vendor open source project today, can become closed source tomorrow.

License

The reason why trademark comes before the license is because the rights-holder (usually the author) of software grants the end-user permission to use one or more copies of software through a license. A free-software license is a notice that grants the recipient of the source code or its binary form, rights to modify and redistribute that software. Without the license, these actions would be prohibited by copyright law. The important point here is that the rights-holder can change their mind and change the license. The rights-holder can decide to distribute the software under multiple licenses or change the license to a non-open source license at any point. It is also possible the software is in the public domain, in which case it is not restricted by copyright law. Public domain does not equate to an open source license and this is a less popular approach we can ignore here. 

Again, without being a lawyer, here is a layman's Litmus test for a license: Is the project licensed under an OSI approved licenses list? If the answer is yes, then you can rely on the due diligence of these foundations for reviewing, classifying licenses and pointing out any limitations. If the answer is a no, ask your company lawyer to review and interpret every word on the license and the possible license compatibility implications. 

Governance

With the remaining checks, we are moving from more business and legal aspects, towards technological and community concerning areas of open source projects.

Opens source evaluation framework

Assuming that there are no concerns about the trademark holder party (your future partner), the license (the terms under which you consume the open source software), the next question is governance. Governance is the rules or customs by which projects decide who gets to do what, how they're supposed to do it, and when. It defines the duties, privileges, and authorities associated with different project roles, and how people get assigned to and removed from roles. Examples here are small day-to-day activities such as who has the power to approve a pull request, to vote for a release candidate, agreeing on the project architecture, defining a project roadmap, and electing the project governance board. 

If you are evaluating a project that is strategic for your organization, you want to know who is in charge. Not only that, you might even have the ambition for your developers to have a say in the direction of the project. 

There is again a simple Litmus test: for projects in open source foundations, there are clear rules on who can vote on important decisions, and how you can be part of decision-taking committees. In some foundations such as ASF, it is based on individual community member merit, and in some such, as CNCF it starts by being an employee of a paying member organization. In the blockchain based open source projects it is based on the vote of the token holders. Other foundations have different rules but they all strive for neutrality and decentralization of power among multiple participants. If a project is governed by a single company, or a single individual, you are trusting them to make the best decisions in the interest of the project and the community. Some of these projects may have written down governance rules that they follow, and some might not at all. It is up to you to figure the governance dynamics and their importance for your project involvement. In addition to having governance transparency and taking decisions in the open, the other aspect here is the level of trust and reputation of the governing body. When you look at the governance board of a project, is there a leader or a group of leaders with proven technical and social skills that give you the confidence that they can take the project to the next level? Or do you see a group that is constantly arguing in political fights? These are some indicators of whether an open source project will be successful and grow in the long term, or headache and stagnation are to be expected.

Infrastructure

Having an open source license may qualify a project technically as open source, but that doesn’t say if a project is built the open source way. There are many examples of software released under an OSI-approved license but developed behind closed infrastructure. By infrastructure, I mean chat channels for quick questions by users. Forums and mailing lists where deeper developer discussions happen. Source code management systems where pull requests are reviewed, and build servers where tests are run and nightly binaries created.

For business people and lawyers looking at an open source project, these might not be important, but for the techies that are going to use an open source project, these are some of the assumed benefits. The check to do here is to explore if the software is developed the open source way using an open infrastructure and not behind closed doors. Here are a few example questions:

• Can a user ask a question on the project chat and get an answer from another user without a middleman?

• Can a developer reach out to project committers and get a deep technical question answered?

• Can you run the latest build and confirm a reported bug is fixed?

• Can an architect the weekly community call and figure out the future direction of the project? 

With a closed infrastructure, you have to create a support case and pay to get answers to similar questions. With an open infrastructure and open participation, the answers are available for those who know how to work the open source way.

Community and Adoption

One of the main benefits of open source software is that it allows great ideas to be developed and spread around virally. You may have the greatest technology, the most permissive license, and open development, but if the software doesn’t have a growing community and increasing adoption rate, that is a sign to investigate. Different projects will have different adoption rates. Some may quickly grow into the mainstream or be replaced by others who do that. Some projects may have a small but persistent growth rate and a niche community lasting for decades. Community size and adoption rate are the ultimate longevity indicator for an open source project. Here are some of the example questions you can ask:

• How many active developers (committers) are in the project and what is the average commit rate?

• How many users are subscribed to the user forums and how many questions have been asked in the last month?

• How many times the latest stable version of the software has been downloaded?

• What other projects and services are dependent on and using this project?

• How many commercial organizations are backing this project?

• Are there commercial organizations offering products, support, and services around it?

• How many StackOverflow questions are about this project?

• How many books, conference talks, and job descriptions are mentioning this project?

Running these questions will give you an indication if the project is growing and becoming a de facto standard in its domain or stagnating and likely to be replaced with the next big thing. 

Typically, open source is associated with fast-paced development and innovation. At the same time, open source is also a mechanism for creating wide adoption and creating unofficial standards. Many open source projects have turned into standards such as Kubernetes for container orchestration, Apache Kafka for streaming, Apache httpd for web servers, etc. One of the most expensive things in software is finding people with the right skills. Using open source projects that have a high adoption rate will give you a better chance of finding skilled people and allow them to reuse their skills for longer.

Conclusions

Depending on the criticality of the open source project, there are different risks and criteria for evaluation. For strategic, hard to replace projects, that will be at the foundation of your IT infrastructure, you want well-established projects that have turned into de facto open source standards in their fields. It is important here to identify who owns the trademarks of the project and who is going to be your long-term partner. Typically these partners would be the member organizations of the software foundation where the project belongs or the single company holding the project IP. For the latter, you may want to consider the long-term risks such as the chances of core developers forking the project, a hyperscaler offering the project as a service, company acquisition, etc. 

For non-strategic, tactical, short-lived projects where the speed of delivery is most important, you can let your developers to drive the selection and pick a project based on openness, community collaboration, and hotness (important for some frontend technologies). Here short-to-medium term risks such as regular security fixes, developer support, and license compatibility checks might be enough.

In either case, there are no single evaluation criteria that fit all. You will have to balance among long term business risks, technology stability on one hand, with the latest hotness, innovation and developer satisfaction, on the other. The framework here will give you an overview of the areas to explore and some of the risks to consider. Good luck!