Why a Thread Pool?
When programming the crawler in the previous section, a problem that would soon present itself is the number of threads to use. A crawler may have to visit tens of thousands of pages, and you certainly do not want to create tens of thousands of threads because each thread imposes a certain amount of overhead. If the number of threads grows too large, the computer will be spending all of its time switching between threads, rather than just executing them.
To solve this problem, you must create a thread pool. The thread pool is given some fixed number of threads to use. The thread pool will assign its tasks to each of these threads. As the threads finish with old tasks, new ones are assigned. This causes the program to use a fixed number of threads, not to be continually creating new threads.
Unfortunately, there is no thread-pooling feature built into Java; thread pooling must be implemented by the programmer. I will now show you my thread pool.