“I’m not so much concerned about the notes, but the space between the notes.”
Silence is as important as sound in music. It gives time to think and for echoes to die away. It can be dramatic.
There is plenty of music where there are silences. While the silence itself is not music, it can be used to punctuate/emphasize the sound that comes before or after it.
A great master of “silence” in music is Sibelius. There are parts in his music where playing completely stops and then we hear nothing. Then music starts again. These pauses are dramatic and cause the surrounding music to stand out more markedly.