Spectrascade

Sunday May 20, 2012

If you have a Webgl enabled browser (check), play with Spectrascade here. It runs best in Google Chrome.

Spectrascade is project Anna and I started a couple months ago and have been working on here and there since. We finally decided to put it online so others could enjoy. Mostly, we wanted to learn Webgl and were inspired by Lights and the particle based music game Auditorium. The result is a 3d particle-based music visualization. It's pretty fun and a bit mesmerizing after awhile. Enjoy!

Features

  • Particles are attracted or repelled by the block which you can interact with by dragging.
  • 3d camera control with right mouse button, reset by double clicking.
  • Various tunable parameters. You can also link directly to parameter settings or randomize them.
  • A few different modes: automatic camera rotate, camera movement, spawning particles at mouse location.

Technology

  • WebGL and Javascript.

    The particle system is running purely in the browser (rendering with shaders on the video card). Because we have physics logic for every live particle on every iteration, as well as updates based on the frequency analysis, the computational cost directly scales with the number of particles. We can, handle somewhere between 1,000 - 20,0000 smoothly depending on your computer's hardware though it may be possible to do the logic all on the video card by writing particle state into texture buffers (allowing for an order of magnitude more particles), but we weren't ambitious enough to try this yet. See this page, for an example of this method, also, see my previous post for more info on particles in webgl.

    We use spidergl, a webgl library (with some modifications). Much thanks Marco Di Benedetto for making this excellent software available.
  • HTML5 Audio

    This was pretty straightforward. One irritating thing is that Firefox won't play mp3s, and Safari won't play ogg-vorbis, so we have to switch between these based on the client browser.
  • Frequency Analysis

    We get features from the song with a Fourier transform of the wav file using Python's numpy library server-side. Javascript gets these features via Ajax to update the particle visuals.
1471 days ago (5/2012)
You're not logged in -- to comment please supply some info:
Display Name Are you human? PIN (lets you delete later)
Thibault Molleman Posted 459 days ago (2/2015)

Can i use my own MP3’s?

Dr. Sounds Posted 880 days ago (1/2014)

Hi

Im intrested to fork your code to use in my music video project for my music release “Sounds from Future”.

http://www.substream.se?view=release&code=mareld58

nemo Posted 1412 days ago (7/2012)

resent. and yes.

Jeshua Bratman Posted 1414 days ago (7/2012)

Hmm don’t see it. You emailed jeshuabratman@gmail.com right?

nemo Posted 1414 days ago (7/2012)

BTW, I did reply by e-mail a week ago, but never heard anything back.
Maybe I ended up in your spam folder.

nemo Posted 1421 days ago (7/2012)

Right. Browser sniffing isn’t necessary. Just provide both sources and browsers will play what they play if/when they support it. Degrades gracefully. Heck, you can provide other sources too. wav, flac, whatever entertains you.
But. Aight, e-mail :)

And, of course, I can’t resist since you reiterated the “Chrome plays everything” bit – you know why Firefox is avoiding mp3 right? :)

Admittedly they may give up on that one like they gave up on webm after, well, Chrome kinda screwed them on that front :-p

But, in that case, if they give up on it, w/ both sources, it’ll use mp3 then, assuming you listed that source first, and assuming you actually prefer mp3 audio quality…

Jeshua Bratman Posted 1422 days ago (7/2012)

Hi Nemo. Yes, I am doing browser sniffing to switch between oggs and mp3s (firefox won’t play mp3s, safari won’t play oggs, chrome plays everything). Anyway, I’d like to try to debug the problem your having. I recently updated the graphcis driver on my laptop and the particle textures no longer show up (it’s in integrated intel card) which sounds like the problem you documented. But, let’s chat via email — it’s a little easier that way [jeshuabratman at gmail].

nemo Posted 1424 days ago (7/2012)

Hm. Weird. Does seem to work just fine on my nvidia machine.
Maybe something is up w/ the sucky driver for my ATI card.

Well. That’s in Firefox. Chromium doesn’t play the music at all on either machine. Perhaps you are doing some browser sniffing instead of the source thingy below? :)

nemo Posted 1439 days ago (6/2012)

Oh. WRT switching… You don’t really need to do that. Just construct an audio element and add both mp3 and ogg sources. If you give them the right mime type, the unsupported source won’t even get fetched.

song.appendChild ( document.createElement ( “source” ) );
song.lastChild.src = “song.ogg”;
song.lastChild.type = “audio/ogg”;
song.appendChild ( document.createElement ( “source” ) );
song.lastChild.src = “song.mp3”;
song.lastChild.type = “audio/mpeg”;

Extra spaces ’cause blog software was trying to parse the brackets.

nemo Posted 1441 days ago (6/2012)

Still nothing but the gray cube, controls and music (nice tunes btw) in Firefox. Could be due to statcounter counter_xhtml.js being blocked (out of my control) – although I don’t know why that would impact the demo.

In Google Chrome, seems even more broken. The music does not play at all. There is just a blank area next to the play arrow, and nothing happens when you click that or the skip button (although it shows the song info)

Jeshua Bratman Posted 1453 days ago (6/2012)

Nemo, yes you’re right there were warnings in the browser error console for me as well. Other than the ALPHA->SRC_ALPHA issue, the problem seemed to be with gl.enable — I was following the discussion on http://learningwebgl.com/blog/?p=2042 After removing it the texture still seem to display for me on several browsers. Do they show up for you now? I’ll look into it further when I’m back from my vacation :) Thanks again for pointing this out. --Jesh

nemo Posted 1453 days ago (6/2012)

Aaaaand finally tried Chromium 18, ATI + Linux. Console spewed an endless stream of:
[30052:30052:86882404109:ERROR:gles2_cmd_decoder.cc] glBlendFunc: sfactor GL_INVALID_ENUM

Presumably Firefox would have had as many in the error console if error suppression hadn’t kicked in.

nemo Posted 1453 days ago (6/2012)

FWIW, Also failed in Firefox 13, Linux, ATI card and Firefox 12, Windows, nVidia card using Angle.

nemo Posted 1453 days ago (6/2012)

Maybe you didn’t notice the errors in the error console? (ctrl-shift-j)
BTW. Still erroring.

nemo Posted 1453 days ago (6/2012)

I was using Firefox 15 under Linux.
bjacob said that no recent version of Firefox should have accepted this.
Personally know nothing of the internals. But perhaps some quirk of Angle under Windows or something?

Jeshua Bratman Posted 1453 days ago (6/2012)

Benoit and nemo — thanks for letting me know! Neither chrome 19 nor firefox 12 threw those warnings. What browser are you running? (and yes that should be SRC_ALPHA)

Benoit Jacob Posted 1454 days ago (6/2012)

The warnings reported in nemo’s comment show that these are bugs in this WebGL page. If any browser accepts these WebGL calls without generating a WebGL error, please report a bug to them.

Indeed: 0x1906 is gl.ALPHA, is not a valid parameter to blendFunc. Did you mean SRC_ALPHA? And 0x0 is gl.NONE, not a valid parameter to enable.

nemo Posted 1455 days ago (6/2012)

Timestamp: 06/05/2012 08:41:54 AM
Warning: Error: WebGL: enable: invalid enum value 0x0
Source File: http://www.jeshua.me/spectrascade/Stage.js
Line: 248

Timestamp: 06/05/2012 08:41:54 AM
Warning: Error: WebGL: blendFunc: sfactor: invalid enum value 0x1906
Source File: http://www.jeshua.me/spectrascade/Stage.js
Line: 251

And no particles.

Jeshua Bratman Posted 1469 days ago (5/2012)

Glad you like it! Server load is fine — though it hasn’t been stress-tested at all. The music is up on amazon s3 now and I cache the audio analysis so there’s really not much for the server to do.

Albert Posted 1470 days ago (5/2012)

super neat! how’s the server load?