Platform: HTC Vive, 12-speaker surround (optional)
Duration: 15 Weeks
Language: C#, Unity ShaderLab, HLSL, SC Lang
Engines: Unity, SuperCollider
Team Size: 6
Roles: Graphics, Interaction, Systems Programmer
Music in Motion was a student-pitched project aiming to explore the relationship of physical music to sound creation in virtual reality. Over fifteen weeks, we explored around twenty prototypes in interactions, graphics, and sound to try to create an experience that would get the average person with no musical experience to feel like they have intuitive musical control in our environment.
As the lead graphics programmer, I was responsible for developing most of our visual effects and systems that would be strongly coupled with the interactions and sounds of our experience. Over the course of the project, I worked on GPU particle simulation, shader-based water simulation, mesh tessellation, and many externally controllable material and skybox shaders for our designers to adjust for the final product. The largest amount of work was put into the GPU particle simulation (based off of Keijiro Takahashi's Spray), where we prototyped several different visuals and interactions that utilized particles, such as an "aurora borealis" that can be influenced by the player and sound inputs, a stylistic hand render to make a lossy hand tracking method more appealing to the user, and an ethereal flowing line of particles that could be traced arbitrarily through space (which can be seen in our demo materials!).
Alongside my graphics development, I also contributed to the gameplay interactions and various game systems, including many positional and movement controls for our sound engine SuperCollider, efficient and uniform spline generation for the tracing paths, the build process to package our sound engine along with our game builds, and the DMX control system that we used to pair our DMX light setup with the lighting inside our experience.
As the lead graphics programmer, I was responsible for developing most of our visual effects and systems that would be strongly coupled with the interactions and sounds of our experience. Over the course of the project, I worked on GPU particle simulation, shader-based water simulation, mesh tessellation, and many externally controllable material and skybox shaders for our designers to adjust for the final product. The largest amount of work was put into the GPU particle simulation (based off of Keijiro Takahashi's Spray), where we prototyped several different visuals and interactions that utilized particles, such as an "aurora borealis" that can be influenced by the player and sound inputs, a stylistic hand render to make a lossy hand tracking method more appealing to the user, and an ethereal flowing line of particles that could be traced arbitrarily through space (which can be seen in our demo materials!).
Alongside my graphics development, I also contributed to the gameplay interactions and various game systems, including many positional and movement controls for our sound engine SuperCollider, efficient and uniform spline generation for the tracing paths, the build process to package our sound engine along with our game builds, and the DMX control system that we used to pair our DMX light setup with the lighting inside our experience.
Notable Experiences
- Developed a GPU particle simulation system based on Keijiro's Spray and used it in many static and dynamic contexts.
- Created a render texture-based water simulation system that can handle many objects of different sizes and velocities entering and exiting the surface simultaneously.
- Implemented and upgraded several tools aimed at increasing prototype speed, including the sound-interaction "linkage" system originally developed by Yujin Ariza that allowed quick pairing and tweaking of user interactions with our procedural sound generation in SuperCollider.
Download
NOTE: The game was originally designed for our custom 12-speaker setup using SuperCollider, which makes it somewhat fragile to port. If you have trouble running it but would still like to give it a try, shoot me an email!