A lot of reasons are given for not supporting the development of Free Software drivers, either by releasing the code or the documentation required for other people to write them. Some of these arguments are valid, but most are misguided or simply wrong, particularly those that relate to not releasing the documentation required for other people to implement the drivers.
Whether you should use a binary driver with a Free Software operating system depends on why you chose a Free OS in the first place. If it was for transparency, security, stability, accountability, or the concept of Software Freedom, then a binary driver is likely to remove some of this attraction for you. If you had other reasons, perhaps using binary drivers is a more "pragmatic" solution.