Monday, November 9, 2009

Cultivating Open Source Software

About a month ago, I wrote some guest posts for the Collective Imagination blog at ScienceBlogs.* Now that they've run their course over there, I thought I'd re-post them here in case you missed them the first time. Here is part 3 of 4:

In the first part of this article, I discussed the differences between Free Software and Open Source Software. In a Venn Diagram, these are concentric rings. The definition of free software lives within the definition of open source software, which itself sits within the space of all kinds of software, so free software is really a subset of open source software. For the rest of this series, I'll use the term "open source software" generically.

Later, I reviewed Open Source Software in the real world, describing the features that an open source software project must have in order to get itself off the ground.

But once an open source software project has gotten started, how does it continue to thrive? How do you build and maintain a community of user-developers who will be interested in adding new features?

Once again, I'd like to use FreeDOS as a case study, since I consider it a very real example of a successful open source software community.

It starts with a web site

When I created the FreeDOS Project in 1994, the Web didn't really exist. It was not until 1995-1996 that the Mosaic browser became readily available, and several years would pass before most users had access to the Internet from their home. So in the beginning, we used an ftp site (hosted by our friends at SunSITE) to distribute all our software and source code. We conversed with one another, and announced new software releases, via USENET news groups.

Compared to today, that's a very limited way to organize a group of developers. In today's world, any open source software project that wants to thrive must provide—at least—a web site for its users. And more importantly, that web site should be well organized.

The project's web site should provide a continuous stream of news items: where is the project going, new developments, major bugs found and fixed, and mention of the project by outside groups. A great way to encourage developers to contribute is to recognize individuals. To an extent, we all like to see our name in print, even in a transient medium like the Web. Spotlight those who make major contributions—but strike a careful balance. One sure way to kill new interest is to present a culture of "us vs. them".

One common mistake is forgetting that an open source software project is for everyone! Too often, new open source software projects dedicate a page to "members" or "developers". As a result, anyone just joining the project gets the impression that the group is closed, too insular. Remain inclusive; it's not a members-only club!

Your web site must make it easy for new users to find a way to contribute. Not everyone who decides to help will be a developer who can write code. Rather, some users will only be able to provide debugging information on odd hardware configurations. Others may have expertise in user interface design, while others may be skilled technical writers. Find a niche for everyone, and advertise on your web site how new contributors can get involved.

Source code

Being open source software means that you need to provide your source code for inspection. This is also the only way for new user-developers to fix bugs and add new features, if they are so interested. So after setting up a web site, your next task should be to open your source code.

How you share the source code may depend on the size of your project. If your program is relatively small, you may only need provide a zip or tar file containing the source code. For example, when I wrote FreeDOS Cats/Kitten (a library that helps a program support multiple languages) I just shared a zip file with all my source code. Cats/Kitten is very small, about 6KB.

If your project is much larger, you may want to look into a source code repository, such as CVS or Subversion, so that users can download just the components that interest them. Today, this is how most open source software projects provide their source code. Look at any project's web site, and you should be able to quickly find a link to their source code repository.

But for your open source software to be successful, you need to provide the full source code every time you make a release. This expectation is built into the GNU General Public License, but it is also a key to building a successful community. Making the source code available to your users allows for the cooperative development and rapid code improvement that fosters "mind share". Not provide the source code—such as making a "testing-only release" or a "preview version"—means your users will not be able to see how your code works. More importantly, your users will not be able to help you fix bugs. Without a way to contribute, developers tend to lose interest in a project, and find something else to do.


Most users often experience a new open source software project through its documentation. Before going through the trouble of downloading new code and testing it out, people want to get acquainted with how the software is supposed to work, and decide if it is right for them. So it is not surprising that many successful open source software projects often put a large effort into making the documentation a first-rate experience.

Writing good documentation is hard, though. Developers who are very technically minded often are not able to write instruction manuals that are easy to understand. So how do projects produce useful and meaningful documentation?

This is such an important step for many projects that some projects split off a separate, related project just to update the documentation. For example, Linux enthusiast Matt Welsh co-founded an effort to provide documentation for fellow Linux users, by writing recipe guides called HOWTOs. Today, Matt's project lives on as the Linux Documentation Project (LDP) and their HOWTOs receive the same level of praise as any O'Reilly book. In fact, O'Reilly has published several LDP HOWTOs as books.

Recognizing the success of the LDP had in helping users understand Linux, I created a similar documentation effort for FreeDOS. In founding the Linux Documentation Project (FD-DOC) we followed the spirit of Matt's intent with the LDP. It would be well for any similar documentation effort to follow the same goals:

The documentation project is primarily a vehicle for enthusiasts and developers to share their knowledge about the system with other users and co-developers. People are motivated to contribute to the documentation effort because they know it helps many people.

The documentation project, if you have one, should be the de facto place for people to visit to find out about your open source software project.

As such, it is vital that it is as easy as possible for new authors to contribute to the documentation effort. If you institute complex document standards, create voting organizations, or require participation in mailing lists, you will discourage many potential new contributors from helping you.

Likewise, the tools to create documentation should be easy to use, widely-available, free, and well-supported. When we started the FD-DOC, we adopted roff (ms) to typeset our documents. We found it was widely-accepted, and many people already knew how to write with it. Those who were unfamiliar with roff typically had a short learning curve. However, times have changed. I recommend modern open source software projects create a wiki where users can become editors.

It is important to remember that the documentation project should be open for contribution by anyone, and is not a closed, private club. Otherwise, the documentation project will not be interesting to any but the few people who created it in the first place.

Bug tracking

Often a major sticking point for newcomers to open source software is the issue of tracking bugs. To empower your users to be co-developers or user-developers, you must provide them the tools they need to be efficient contributors. A successful open source software project must equip its developers with a bug tracking system. After all, a software project cannot improve itself if it cannot identify what it wrong in the software.

When I think about bug tracking, I am reminded of this quote:
The highest sounds are the hardest to hear. Going forward is a way to retreat. Greater talent shows itself late in life. Even a perfect program still has bugs.
~Geoffrey James,
The Tao of Programming
There are a number of different bug tracking systems that you can take advantage of. Bugzilla seems to be the most popular these days, but I have also used GNATS and Jitterbug with some success. These bug tracking tools all share a common set of features that you should look for in your open source software project:
  1. The bug tracking system must be easy to use: entering new bugs, and following existing bugs. If your system is too difficult to use, no one will want to enter new bugs, and problems in your software will remain unfixed.
  2. The bug tracking system must have a strong query capability. If your users and developers cannot look up existing bugs, you'll find your project will get flooded with lots of duplicate bugs.
The system must be open to all eyes. This is perhaps the most important—and yet most difficult—feature for traditional software managers to understand. In the cathedral model of software development, bug lists are often hidden from customers. After all, there is no beta released before its time, so why would you advertise features that are broken until you have had time to fix them? However, in the bazaar model of open source software, everyone who uses the software is potentially a new developer for it. In order to foster new fixes and patches to the system, developers must be able to see what work needs to be done, and what features need to be addressed.

Respond to submissions

Treating your users as co-developers is your least-hassle route to rapid code improvement and effective debugging. Because the source code is available, anyone can contribute patches and new features. Given a bit of encouragement, your users will diagnose problems for you, suggest fixes, and even send you source code that helps to improve your project far more quickly than you could on your own.

However, the only way to keep your users working with you as co-developers is to respond to the patches and suggestions they send you. Your best bet to keeping your community engaged and stimulated is the sight of constant improvement in the system they are using.

In the FreeDOS Project, I wrote a lot of utilities and other programs. My most active development efforts were aimed at our first Install program, and later the Cats/Kitten library. Looking at the Install program, early in the development cycle, a very talented developer named Jeremy Davis found an interest in how the Install program worked, found some bugs, and submitted a few fixes to me. Eager to have help in my coding, I responded immediately by reviewing his patches and checking them into the source code tree, and releasing a new version. Releases of the Install program became more frequent as bugs were quickly identified and patches.

Similarly, when I first wrote the Cats library, it was a small effort to recreate the Unix catgets functionality. But since many people around the world contribute to FreeDOS, my Cats library became popular very quickly. I received patches from different developers, adding features, improving Unix compatibility, and reducing the memory footprint. Releases happened very quickly, sometimes several versions in a single day if we had enough of us collaborating. Eventually, we had introduced so many new features yet in a smaller size, I renamed the library to Cats/Kitten.

None of this would have been possible had I stopped responding to the users who were acting as my co-developers. The key was constant feedback.

The worst thing that can happen to an open source software project is to become unresponsive to its users. A sure way for a project maintainer to kill interest is to reply to developers with "Thanks for the patch, but I was planning to add that anyway in the next release."

It's not pixie dust

Finally, let me remind anyone interested in starting an open source software project that open source software does not imply automatic success. Just because you release your project as "open source software" does not mean you will develop a community to help you, even if you do everything "right". No project, either open source or proprietary, can guarantee that it will survive the test of time.

Jamie Zawinski, a well-known open source software developer, wrote of his time with Netscape as it became "open sourced” as Mozilla:
Open source does work, but it is most definitely not a panacea. If there's a cautionary tale here, it is that you can't take a dying project, sprinkle it with the magic pixie dust of "open source," and have everything magically work out. Software is hard. The issues aren't that simple.
In other words, open source is not a magic bullet. A project must be interesting to other developers in order for them to contribute to it. Realize that your software just may not be that cool to many other people.

An example is a project someone once wrote for FreeDOS. The FreeDOS Internet Services Host (FISH) was written in 1999 by Gregory Ball, as a web server that ran on a PC running FreeDOS. Really, the program was a demonstrate that network applications could be written for FreeDOS using WATTCP, the standard library for TCP/IP in DOS. Gregory deployed FISH on an 80386 SX/25 computer with 4MB of memory, but his web site advertised it would run on a lesser system.

This was a really cool project, and I admired it for its unadulterated hack value. However, no one really believed that people would run serious web servers on a DOS-based system. Apache was a much more common system that supported more features. So FISH sort of fell by the wayside, and failed to generate enough interest to maintain a community. Eventually, the last FISH server was turned off, and no one noticed. Being cool just wasn't enough if people weren't interested in using it.
In part 4, I'll talk about the final stages in an open source software project, and how to maintain an effective transition of the maintainer role.

No comments:

Post a Comment