The Final Decision on Culture
Passing on a job because you're concerned about the culture isn't always the easiest option. If you've got bills piling up, you may have more pressing concerns than whether something's hinky with the company you're joining. Taking a job, despite warning signs, may be the only immediate option. But, if you find yourself in that situation, don't consider yourself locked in. Keep looking for the right job and move on when you find it.
Jumping ship from a company with a toxic work culture shouldn't be shameful. You have to look out for your own career, whether that means passing on a job initially, or leaving after a week on the job.
Furthermore, you have no obligation to fix any culture that exists before you join a company—unless you're being hired for that expressed purpose, your only responsibility is to do the work you're trained for. Some companies respond to problems with their culture by trying to introduce diversity or add less toxic staff to the mix, but those solutions rarely work out.
Joining any company is a decision you have to make on the merits of the offer: whether you're getting what you need out of working for that company. Ultimately, you have to look out for your own interests, just as your employer will look out for its interests.