Context:Project for Fundamentals of Digital Signal Theory, a class in NYU Steinhardt.

Role:
Developer

Concept:Interactive 3D website, multi-user audioshare visualizer

Tools:
WebGL, three.js, JavaScript, spatial audio

  The Listening Room is a Three.js powered website in which users can communicate with one another via sharing audio files which are then played back on other users’ sites. The website uses JavaScript in conjunction with Python, using Three.js as a 3D frontend.

Code
In the listening room itself,the turntable acts as a “file upload” button, allowing users to post the .wav file of their choice into the server. 


Once an audio file is uploaded to the server, a folder watchdog takes note, and sends a message to python to take the uploaded file and turn it into a spectrogram. That code looks like this:



And it produces an image of a spectrogram that looks like this:



Once this image is generated, another file watchdog communicates to our server that we’ve created a fresh spectrogram graph. It’s then processed as a texture on a cube in the Listening Room, along with a black & white version that’s used as a displacement map. That creates a 3D object that looks like this: 


Additionally, the user-uploaded audio is played back to everyone else on the server, and is spatialized to the speakers.

Design
The “room” part of the listening room is an original design made in Blender, with textures mapped from AmbientCG.


The project also makes use of a variety of prefabricated .gltf assets from Sketchfab.



these are all available for free download, except the disco ball.

Putting them all together creates the room shown in the first image.

Challenges
This is one of those projects that doesn’t require much coding, but does require a variety of relatively obscure libraries and very specific commands. Finding a library that would make python scripts run in a JavaScript dev environment was tricky! I ended up using PythonShell, and it seems to work well enough - easier than a Flask server. Running this on a server also requires the installation and initialization of a Python virtual environment to run within the JavaScript environment.

Next Steps
Because this code is so modular, it’s possible to do any type of signal processing on the Python side and pass it to other users on a server using the framework in this code. I’d love to continue experimenting with other Python A/V features, and make the listening room even more exciting!