- Traces and JITs
- Speculative Inlining
- The Problem of Numbers
Traces and JITs
This is a big advantage for web scripts, where a "program" is typically compiled every time it's run, and you don't want to have to wait for the whole thing to compile before you can run any of it. Another option is background-compiling, in which the code is compiled in one thread and interpreted in another thread. As the compiler finishes each function, it updates the interpreter to call the compiled version instead of interpreting it.
You can even combine these two approaches by keeping the functions to be compiled in an ordered list and moving them closer to the head every time they're run in the interpreter. This technique will result in the frequently called functions (the ones that will have the biggest impact on overall performance) being compiled first. If a function is never called, no one cares if it isn't compiled. Some virtual machines will have a threshold of a small number of calls (for example, 16) before a function is compiled. This design lets you avoid wasting time compiling code that runs at the start and then is never run again; instead, it will run in the interpreter the first time, and you don't gain anything from compiling it after that.
So far, I've talked about compiling functions or methods. These are useful blocks of code for the programmer to ponder, but they don't often correspond directly to the flow of execution through a real program. It's very unusual to have a program that runs one function, another, another, and so on. Most programs do a bit of work in one function, call another, and do a bit of work there. The second function may call a third, do a bit of work there, and then have it return. The program then spends a bit of time in the second function before returning to the first. If you never had to maintain this code, you could create a much more efficient version of it by rearranging the boundaries between functions. It would be completely unreadable, but it would be faster.
This technique can be very effective. Often, functions contain several code paths, but only one is commonly used. A tracing VM can join together the commonly used paths through a short series of functions, creating a single non-branching block of executable code. While you're using this "hot path," the code will run much faster than it would have with the original structure.