Hidden Class Transforms
One of the downsides of the prototype-based approach is that accessing object properties is very expensive. Accessing an instance variable in a Smalltalk object is just a matter of (the compiler) adding a constant offset to the object pointer to get the address. The same operation in Self requires a dictionary lookup.
However, often there are a lot of objects with a very similar set of properties. In this case, a clever compiler can implement a class-based model. It transforms a group of objects into instances of a (hidden) class, which describes the fixed layout. Accessing the fixed set of fields is very cheap, while other properties are accessed via the same slot mechanism.
This structure also allows some other optimizations. For example, you can cache method lookups more easily if the lookup result is shared among a few objects. The Self team had a very fast implementation in the 1990sfaster than any Smalltalk implementation of the time, and around 50% of the speed of an equivalent C++ implementation. (Although C++ compilers in those days were pretty bad, so that wasn't as much of a boast as it would be now.)
In principle, you can implement any prototype-based program in a class-based language by having a single class for each object. Indeed, that's one of the simplest ways of implementing the hidden class transform. You begin by implicitly creating a class for every object in the system, inheriting from the object's prototype's class. If the new class doesn't declare any new methods or instance variables, you remove it. Eventually, you're left with a relatively large number of classes that have a lot of instances, and a few that have only one or two.