I dont think you should quit it.
I used to lsiten to music everytime when I am not at home (in the train, bus, on the road e.t.c). Personally I found out that when I go for long streaks I dont get the feeling to listen to music when I am outside (not at home). I find everything around me more interesting than music but the opposite happens when I relapse. It all has something to do with your dopamine levels.