First off, what is special about framework developers and why do they need to refactor so much? Frameworks often expose a large set of APIs that other developers use to build their application. I say APIs, but of course that is a heavily overloaded term. An API in this context is really just a set of interfaces or a set of method signatures. You promise behavior, but are free to implement it in any way you want. In other words, you are free to refactor.
Is this really a unique situation for framework developers? No, any application essentially does the same thing. It promises a certain set of behaviors, but makes little or no promises on the implementation. There is still a lot of room and freedom to refactor behind the scenes. Actually you have a lot more freedom in many situations.
A public interface in a framework cannot be easily refactored. Many of Fowler's refactorings involve changes to the signature of a method or moving a method or class. A public interface has to be backwards compatible, so you can't just change the signature, or move it to a different class, etc. Fowler recognizes this, and describes steps you can take such as keeping the old method/class, but deprecating it and having it delegate to the new, refactored code.
So everybody needs to refactor, Application developers and framework developers alike. Actually there are one set of developers that, in my experience, often think that refactoring is not part of their charter: consultants. This is not globally true, of course. Fowler's opening story revolves around him working as a consultant and suggesting refactorings. It is more true of the "guns for hire" type of consultants. I should know, I used to be one.
For those folks, you are brought to build something that is especially challenging. It is either very complex or is on a super aggressive schedule, or both. The key differentiator is that such consultants do not have a durable interest in the project. They jump in, crank something out, and move on. Sure you want the customer to be happy and ask you back for more work and/or recommend you to their buddies. But usually if it works and is done on time, then the guys writing the checks are happy. Maybe you have to train some folks to take over the code, and that might motivate you to refactor, but it is not the same as if you or a co-worker had to maintain it.
However, refactoring is still important for the guns for hire. Why? Because often those fast and furious projects still shift gears rapidly in mid-project. After all, you are viewed as being so good that you can handle feature creep or product management flip-flopping. Continuous refactoring of your code makes it much easier to handle these situations. If you are on a fixed price project, this is especially important. Even on a time and resource project, you don't want to come off as "oh yeah this change is going to cost you big time." Chances are you will lose the confidence of the employer or they will just skip the change and you just lost money. All because your code needed refactoring and you thought you were above that law of software engineering.