It’s All About the Applications
An operating system is nothing without applications to run on it. This is one area in which NetBSD excels. The packaging system, pkgsrc, has grown beyond its roots and is now available for several other platforms, including the other BSDs; Linux; and several flavors of commercial UNIX, such as Solaris and Mac OS X. DragonFly BSD has adopted it as its native packaging system, and Sun has contributed hardware to the project to encourage better Solaris support. Geert Hendrickx describes some of the attractions of pkgsrc:
pkgsrc provides a solid, consistent, portable framework for installing and managing an ever-growing number of third-party packages, both in source and in binary form. pkgsrc doesn’t let packages install files where they want it; they must all fit into a coherent (and configurable)directory structure. pkgsrc features package selection by license (users can specify which software licenses they accept and which not), automatic checking of packages against a list of known vulnerabilities, etc.
Originally, pkgsrc started as a fork of FreeBSD’s ports system. Since then, it has grown enormously. Like the rest of NetBSD, pkgsrc has a focus on portability, and most of the packages in it have been tested to compile on all supported architectures and operating systems. This, of course, comes with its own problems.
Many of the architectures supported by NetBSD are very slow. Although a 5MHz MicroVAX or a 25MHz 68000 might be fast enough to run some applications usefully, building 6,000 packages on one would take a very, very long time. Joerg Sonnenberger suggested a few approaches that are being used to address this problem. The first is emulation:
It’s certainly faster to build packages for Vax in an emulator running on a beefy amd64 box than to build them natively. Partial CPU emulation of the userland could work, too—similar to our Linux emulation, and I think some people are working on it, but I don’t know the status of it.
The partial CPU emulation referred to here was the subject of a paper published a couple of years ago. This approach is particularly interesting; building an emulator that can run configure scripts and GCC quickly is a very different problem from building a general purpose emulator. For one thing, system calls can be executed on the host machine in much the same way in which system call translation works: When the emulator encounters a system call, it performs any required byte-order manipulations and calls the real system call. This is quite easy to do on NetBSD because the system calls are all in the same place on all systems.
Some programs that are frequently invoked, such as the shell, can be implemented natively, which can speed things up a lot. Although this is a fascinating direction from a research perspective, there is a slightly simpler option, namely using distcc:
All configuration and preprocessing stages would run on the native host, but the actual cross-compiling can be done on faster machines on the network. pkgsrc supports using distcc as a compiler.
The distcc program is a wrapper around another compiler, which allows the distributing of a build job across a network of machines. This means that the heavy lifting work of compiling can be done on other machines; as long as the native machine is fast enough to run the configure script, most of the rest of the work can be delegated.
The ideal solution would be for all the packages to support cross-building as well as the base system does; you can build NetBSD almost anywhere, including on Windows, which makes it particularly nice for developing embedded systems that don’t have the necessary power to perform a native build. Unfortunately, a large number of open source programs use build environments designed without cross-building in mind, and it would require a huge amount of effort to move all of them to a new system.