Depending on how far north you are going in Manhattan, it can be defined as "some way" but in general it takes 30-45 minutes to get to most of the tourist hotspots and neighborhoods people like to visit, which is the average journey time for getting around anywhere New York unless you are staying in Manhattan itself. And yes, the Subway map will be essential no matter where you stay in the city.
Regarding the 'safety' and 'social mix' of the neighborhood - I am proud to live in a neighborhood with such a diverse melting pot of cultures, races and people and there is a mixture of private and social housing, as you will find in any neighborhood in New York City, even in Manhattan. I'm sorry if that mix made you feel uncomfortable.
BedStuy in particular has so many long-term residents, retirees who own their houses/apartments and young professionals and young families moving into the area every day, a testament to its safety, vibrancy and welcoming atmosphere.