Kinect MIDI Controller (KMIDIC) – On a Mac!

Posted: April 26, 2011 in Ableton Live, Electronic, Hardware, Kinect, KMIDIC, Launchpad, MIDI Controller, Music, PMIDIC, Processing, Software

Instead of making a stop motion video of my giant Lego man (you can see him in the video below) like I was planning to do, I ended up playing with Processing again. Basically I didn’t really have the right space to setup my stop motion, and I really needed to have a green screen to do what I wanted to do. But its all good, no time wasted as I was very productive anyway!

Firstly, heres the video of my creation today:

Now an explanation…

In summary, I am waving my hand around in the air like an idiot and it is controlling the music coming from my computer. The hardware and software components that are at play here are Microsoft Kinect -> Processing -> Ableton Live and Novation Launchpad. I had a chat to my friend DJ Gustavo Bravetti and he had some good tips for me on how to setup Ableton clips to make the transitions smoother and sound more musical. When I have time, I’ll set up a whole song and give a better performance!

A bit more detail…

Basically, the Kinect is sending the location of my hand to Processing which is in turn sending MIDI note on messages to both Ableton Live and the Novation Launchpad. In this version, I have separated the grid into 4 quadrants, each on playing a different MIDI note that is going into 2 channels in live. The first channel has an arpeggiator triggering an Impulse drum kit, and the second channel has an arpeggiated synth. The lights on the Launchpad are also set to light up each of the quadrants as they are triggered.

For those wanting to delve into the code, its not highly commented, but you should be able to get the idea of what I’m doing. Any issues, just leave a message here or send me an email on benxtan [at] gmail [dot] com.

Processing source code and Live set are available here (UPDATED):

http://benxtan.com/kmidic/kmidic_processing_v0.2.zip

http://benxtan.com/kmidic/kmidic_processing_v0.1.zip

You will need to install Processing, and the rwmidi and libfreenect libraries in the libraries folder of your Processing sketches.

Here are some links if you are after more information.

Software Links:
http://www.ableton.com/
http://processing.org/
http://ruinwesen.com/support-files/rwmidi-0.1c.zip
http://ruinwesen.com/support-files/rwmidi/documentation/RWMidi.html
https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing

Hardware Links:
http://www.xbox.com/kinect
http://www.novationmusic.com/products/midi_controller/launchpad

Once again, if you are a musician or music business in Australia, its free to sign up to http://rockstarhookups.com.au so go do it! I’m giving you free stuff, so help me out here ok? 🙂

Advertisements
Comments
  1. Rolf says:

    very nice, comprehensive work….I have been working with the Kinect/Processing for only a few months, creating a musical instrument for disabled people. It’s working quite nicely but I have not had any success in sending output velocity (controller7). No matter what I send, the loudness stays the same. Grateful for any suggestions…

    • benxtan says:

      Hi Rolf,
      Thanks for your interest. I haven’t tried outputting velocity yet but I’ll try it and let you know what I find. Are you using the same midi and kinect library that I am?

  2. Hi Guys. I’m not sure if it’s even possible but if so I would greatly appreciate info. I should say that I don’t understand programming much but I was wondering how difficult would it be to set up kinect to control Tempo with your right hand and velociy/Dynamics(CC1-modwheel) with your left hand. We are doing a project in Exeter where I will be conducting whole Requiem using Osculator (Wii remotes) but I always wondered if we could use kinect instead. PS please feel free to email me. Thank you

  3. Zachary says:

    Great video, and thanks for the info! I love this concept… however I’m very new to it & am not a developer… I’ve downloaded the files, but am really unsure how it is supposed to work. Is it as simple as connecting the Kinect & runnng Processing? or do I need to install something else on the Mac first or do anything to the Kinect? … Also I am unsure about the Processing library folder. Is it the one that I would find in viewing the contents of the app? I’m not sure about the “libfreenect” file either as there seems to be a couple different downloads on that link… Sorry if I’m asking too many stupid questions, but I am a total beginner with this. Any help would be appreciated.

    • benxtan says:

      Hey Zachary,
      It might a bit tricky for a non-developer to set it up without any instruction.
      I’ve had a bunch of people asking me for instructions now, so I’m going to have to write up some instructions and put it up online.
      Heres a first draft:
      (Screenshots to come later)

      How to get KMIDIC running on a MAC and get it to send MIDI to a virtual MIDI device

      1) Add a virtual MIDI device.
      You only need this if you want KMIDIC to send MIDI messages to software (e.g. Ableton Live) as opposed to hardware.
      Set up a MIDI bus like so – http://cmpercussion.blogspot.com/2008/08/internal-midi-bus-in-mac-os-x.html
      Basically:
      a) Open “Audio Midi Setup” using Spotlight (Command + Space)
      b) If you can’t see a “MIDI Studio” window, click on View > Show MIDI Window
      c) Double click on “IAC Driver”
      d) Click on the “+” button to add a port. The default name is “IAC Bus 1” but you can rename if you wish.

      2) Download and install Processing (http://processing.org/download/)

      3) Download and install RWMidi (http://ruinwesen.com/support-files/rwmidi-0.1c.zip)
      Note: Libraries go in /(Your Processing sketchbook location)/libraries/
      You can get and set (Your Processing sketchbook location) in Processings preferences.
      You should now have /(Your Processing sketchbook location)/libraries/rwmidi

      4) Download and install libfreenect (https://github.com/downloads/shiffman/libfreenect/openkinect.zip)
      You should now have /(Your Processing sketchbook location)/libraries/openkinect

      5) Download and unzip the latest version of KMIDIC (http://pmidic.com/)
      Latest version is (http://benxtan.com/kmidic/kmidic_processing_v0.2.zip)
      This goes into /(Your Processing sketchbook location)/

      6) Open Processing and open the the KMIDIC project.

      7) Get the ID of the MIDI device you want KMIDIC to output to.
      In Processing, run KMIDIC by clicking on the Play button on the top left corner or press Command + R.
      Look at the output in the console.
      Do you want KMIDIC to talk to software or hardware?
      If your answer was software, then remember the MIDI bus you created in Step 1?
      Look for it (IAC Bus 1 Apple Inc.) in the Output Devices list and get the index number (e.g. [0]) to the left of it.

      8 ) Set the ID of the MIDI device you want KMIDIC to output to.
      In Processing, click on the first tab (Kinect _MIDI_Controller_vx.x) and look for the line that says: int midiOutputDeviceId = 0;
      Set the value to the index of the output MIDI device you want to use.
      For example, if the index was [1], you would change the code to say: int midiOutputDeviceId = 1;
      Now KMIDIC is set up to send MIDI message to that device.
      If you had a hardware device plugged in, like a synthesiser, you could easily make KMIDIC send MIDI to it by changing this ID.

      9) Hit RUN!

  4. Eben says:

    5) Download and unzip the latest version of KMIDIC (http://pmidic.com/)
    Latest version is (http://benxtan.com/kmidic/kmidic_processing_v0.2.zip)
    This goes into /(Your Processing sketchbook location)/
    ……………………….
    is this as in the same as the previous processing/libraries??

    6) Open Processing and open the the KMIDIC project.
    ………………………………………………………………………..
    i open processing and open KMIDIC but which file do i open in processing????

    -kinect_midi_controller_v0.2pde
    -kinecttracker.pde
    -grid.pde
    -midideviceutils.pde
    -devices.pde
    -midi.pde

    do you place all .pde files ect all in libraries without the main folder keeping them togeather??

    cos when i try run any of them it says

    unsatisfied link error/users/eben/documents/processing/libraries/openkinect/library/libkinect.jnilib:

    [0] Real Time Sequencer Sun Microsystems

    Output devices:
    [0] Real Time Sequencer Sun Microsystems
    [1] Java Sound Synthesizer Sun Microsystems

    Output Device: Real Time Sequencer Sun Microsystems
    Exception in thread “Animation Thread” java.lang.UnsatisfiedLinkError: /Users/ebengreaves/Documents/Processing/libraries/openkinect/library/libKinect.jnilib:
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1822)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1739)
    at java.lang.Runtime.loadLibrary0(Runtime.java:822)
    at java.lang.System.loadLibrary(System.java:993)
    at org.openkinect.Context.(Context.java:43)
    at org.openkinect.processing.Kinect.start(Kinect.java:40)
    at Kinect_MIDI_Controller_v0_2$KinectTracker.(Kinect_MIDI_Controller_v0_2.java:432)
    at Kinect_MIDI_Controller_v0_2.setup(Kinect_MIDI_Controller_v0_2.java:104)
    at processing.core.PApplet.handleDraw(PApplet.java:1608)
    at processing.core.PApplet.run(PApplet.java:1530)
    at java.lang.Thread.run(Thread.java:613)

    Any help would be awsome bro.

    Cheers

    Eben

    • Eben says:

      dragged everythn into processing and it cant find kinect still.
      any chance you could export it as an application???
      cheers

    • benxtan says:

      Hey Eben,
      1) To get your /(Your Processing sketchbook location)/ directory, look in preferences.

      Inside this directory will be all your projects.
      Each project goes into its own folder, e.g. Kinect_MIDI_Controller_v0_2.
      Leave the folder name as is, don’t rename it.
      Inside that folder will be all the .pde files.

      Also create a folder inside /(Your Processing sketchbook location)/ called libraries and put all the libraries inside it.

      2) When you open, use this file kinect_midi_controller_v0.2pde

      • toot! says:

        I’ve followed your directions and I get this when I try to run:

        Exception in thread “main” java.lang.RuntimeException: java.lang.ClassCastException: com.sun.media.sound.MidiInDeviceProvider$MidiInDeviceInfo cannot be cast to de.humatic.mmj.spi.CoreMidiProvider$JSMDevInfo
        at processing.core.PApplet.runSketch(PApplet.java:7391)
        at processing.core.PApplet.main(PApplet.java:7580)
        Caused by: java.lang.ClassCastException: com.sun.media.sound.MidiInDeviceProvider$MidiInDeviceInfo cannot be cast to de.humatic.mmj.spi.CoreMidiProvider$JSMDevInfo
        at de.humatic.mmj.spi.CoreMidiProvider.getDevice(CoreMidiProvider.java:122)
        at javax.sound.midi.MidiSystem.getMidiDevice(MidiSystem.java:200)
        at rwmidi.RWMidi.getOutputDevices(Unknown Source)
        at Kinect_MIDI_Controller_v0_2$MIDIDeviceUtils.getOutputDeviceName(Kinect_MIDI_Controller_v0_2.java:620)
        at Kinect_MIDI_Controller_v0_2.(Kinect_MIDI_Controller_v0_2.java:65)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at processing.core.PApplet.runSketch(PApplet.java:7389)
        … 1 more

        Any ideas? I’d be really interested to try this out.
        Cheers

      • benxtan says:

        Hmm…not sure what that means. Is there any other output in the console?

  5. Tom says:

    Hey,

    firstly I want to say thank you for sharing this hard with us!! I have just started with this whole world of programming and I am somewhat overwhelmed. Alongside working my way through Pure Data and Processing tutorials I am also trying to get my Kinect to work as a midi controller for Ableton Live.

    I have followed all your instructions and everythingseems to be done correctly, except for this error message:

    “Exception in thread “Animation Thread” java.lang.NullPointerException
    at org.openkinect.processing.Kinect.enableDepth(Kinect.java:82)
    at Kinect_MIDI_Controller_v0_2$KinectTracker.(Kinect_MIDI_Controller_v0_2.java:430)
    at Kinect_MIDI_Controller_v0_2.setup(Kinect_MIDI_Controller_v0_2.java:101)
    at processing.core.PApplet.handleDraw(PApplet.java:2241)
    at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:243)
    at processing.core.PApplet.run(PApplet.java:2140)
    at java.lang.Thread.run(Thread.java:680)”

    Do you have any idea what this could be?

    I have ableton set to receive midi from my new Midi bus but I don’t see any incoming information.

    Any help would be much appreciated!

    If there is any way I can help you please let me know. I am not living in australia but I am morethan happy to help promote your work in any way you can.

    Thanks again!!

    T

  6. […] it all at website Benxtan further details and […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s