Here are some experiments in VR that I made after watching the Doctor Strange movie.
Visuals created using Unity, sounds created using Ableton Live.
Archive for the ‘Hardware’ Category
Doctor Strange Experiments
Posted: November 13, 2016 in Ableton Live, Oculus Rift, Unity 3D, Virtual Reality1
Insight
What if you could make things appear in front of you with the power of your voice?
Idea
Skyfall
Execution
I had an idea of making a virtual reality (VR) demo where you can say things and make them fall from the sky.
For the initial demo, the only objects that you can magically summon are:
- a cube
- a white car – Ferrari
- a red car – Dodge Viper
- a house
- Deadpool
- a Star Destroyer
The “car” command randomly picks between one of 2 cars.
You can also say the word “clear” to remove all the objects from the scene.
Additionally, you can also move around the 3D space using an XBox controller or keyboard and mouse.
All this works within the Oculus Rift DK2.
To take this project further I can:
- add more objects and commands
- add Leap Motion support so you can interact with the objects using your hands
Tech Details
Problem
One of the roadblocks I hit early on was that Unity 5.3.2 does not support the .NET speech recognition DLL’s since it is using Mono.
Solution
In order to get around this, I created 2 applications and used a TCP port for them to communicate with each other.
Components
1) Server: C# .NET application
The voice recognition is being done using System.Speech.Recognition.
A GrammarBuilder object is created with the following words.
private string[] VALID_COMMANDS = {“clear”, “cube”, “car”, “house”, “deadpool”, “star destroyer” };
Once a word is recognised, a message is sent to a specified port via TCP.
2) Client: Unity application
On the client side, there is a TCP listener running on a thread that listens for TCP messages.
If a word is received (e.g. cube) the model named “Cube” is then cloned and added to the scene in front of and above where the user is looking.
First person controls and VR support was also added to make the experience more immersive.
MCA ARTBAR Futures Quartzite Hologram & VR
Posted: June 28, 2015 in 3D, Hardware, Music, Virtual RealityTags: 3D, ableton, art, Music, science, vr
On Friday the 26th of June 2015, a collaborative VR artwork between Vaughan O’Connor and myself was exhibited at the MCA. The 2 artworks, a holographic print and a Virtual Reality (VR) experience were both based on 3D scans of quartzite rocks from Oberon, NSW.
The artworks were on display at the Museum of Contemporary Art (MCA) ARTBAR – Futures, curated by Dara Gill. Facebook event link here.
I created the VR component of the artwork based on the scans that Vaughan supplied.
Inspiration
Inspiration came from lyrics to a song by Covenant called Bullet.
“As the water grinds the stone,
we rise and fall.
As our ashes turn to dust,
we shine like stars.”
Idea
The idea behind the artwork is that it is a place outside of time. Here, concepts such as gravity, space and time do not behave as you would expect. Tempus Incognito. The events that occur here, in the event horizon, cannot affect an outside observer. The 4 classical elements, fire, air, water and earth are present in this place, reflecting the essential building blocks for all things. Even though we are outside of time, matter still exists albeit in a rather astonishing manner.
Technology
The Quartzite VR artwork was created using Unity (a game engine) and experienced via the Oculus Rift DK2 (a virtual reality headset). The ambient soundscape was composed in Ableton Live and is binaural.
Photos
Some photos by @mightandwonder
- https://instagram.com/p/4ZC9KfRUHL/
- https://instagram.com/p/4ZDwMmRUIp/
- https://instagram.com/p/4ZCSOMxUGU/
Observations
A few interesting things that happened on the night:
- one of the computers decided to misbehave (no audio, no projector)
- someone decided to walk about 2 metres away from the desk. The Oculus is plugged into a computer!
- 2 people decided to look under the desk
- a lot of people stood up to experience the artwork rather than sit down in the seats provided
- People were amazed at the Oculus Rift
- People found the experience soothing, calming and meditative.
- Some people wanted to stay in side the artwork forever!
- People asked a lot of questions about the technical aspects of the artwork
More
Futures also featured performances, lectures and music by: Vaughan O’Connor & Ben X Tan, Michaela Gleave, Josh Harle, 110%, Mark Brown, Eddie Sharp & Mark Pesce, Andrew Frost, Claire Finneran & Alex Kiers, Kate MacDonald, Sitting & Smiling, Baden Pailthorpe, Hubert Clarke Jr, Polish Club and Ryan Saez.
Kinect MIDI Controller (KMIDIC) – On a Mac!
Posted: April 26, 2011 in Ableton Live, Electronic, Hardware, Kinect, KMIDIC, Launchpad, MIDI Controller, Music, PMIDIC, Processing, SoftwareInstead of making a stop motion video of my giant Lego man (you can see him in the video below) like I was planning to do, I ended up playing with Processing again. Basically I didn’t really have the right space to setup my stop motion, and I really needed to have a green screen to do what I wanted to do. But its all good, no time wasted as I was very productive anyway!
Firstly, heres the video of my creation today:
Now an explanation…
In summary, I am waving my hand around in the air like an idiot and it is controlling the music coming from my computer. The hardware and software components that are at play here are Microsoft Kinect -> Processing -> Ableton Live and Novation Launchpad. I had a chat to my friend DJ Gustavo Bravetti and he had some good tips for me on how to setup Ableton clips to make the transitions smoother and sound more musical. When I have time, I’ll set up a whole song and give a better performance!
A bit more detail…
Basically, the Kinect is sending the location of my hand to Processing which is in turn sending MIDI note on messages to both Ableton Live and the Novation Launchpad. In this version, I have separated the grid into 4 quadrants, each on playing a different MIDI note that is going into 2 channels in live. The first channel has an arpeggiator triggering an Impulse drum kit, and the second channel has an arpeggiated synth. The lights on the Launchpad are also set to light up each of the quadrants as they are triggered.
For those wanting to delve into the code, its not highly commented, but you should be able to get the idea of what I’m doing. Any issues, just leave a message here or send me an email on benxtan [at] gmail [dot] com.
Processing source code and Live set are available here (UPDATED):
http://benxtan.com/kmidic/kmidic_processing_v0.2.zip
http://benxtan.com/kmidic/kmidic_processing_v0.1.zip
You will need to install Processing, and the rwmidi and libfreenect libraries in the libraries folder of your Processing sketches.
Here are some links if you are after more information.
Software Links:
http://www.ableton.com/
http://processing.org/
http://ruinwesen.com/support-files/rwmidi-0.1c.zip
http://ruinwesen.com/support-files/rwmidi/documentation/RWMidi.html
https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing
Hardware Links:
http://www.xbox.com/kinect
http://www.novationmusic.com/products/midi_controller/launchpad
Once again, if you are a musician or music business in Australia, its free to sign up to http://rockstarhookups.com.au so go do it! I’m giving you free stuff, so help me out here ok? 🙂
My awesome day
Posted: November 26, 2010 in C#, Guitar, Kinect, KMIDIC, Life, MIDI Controller, Music, PMIDIC, Software, Throw Catch1. Woke up and checked stats. My most recent software creation, a Kinect MIDI Controller (KMIDIC) is on engadget in the US, Japan and Germany!
2. Managed to get a ticket to one of my favourite bands Kyuss for next year. So happy! Tickets sold out super quick!
3. Got to work, did a bunch of work, then my band’s (Throw Catch) freshly mastered debut EP arrived in the mail, and it sounds amazing! Big thanks to Dylan Adams (recording and mixing ) and Michael Lynch (mastering at shoehorse sound).
4. Did more work, finished work, and off I went to my first guitar lesson in 12 years…with Peter Northcote! I learnt sooo much in that one hour lesson, and have now made the commitment to pratice guitar at least 10 minutes a day.
5. Started practicing as soon as I got on the train and had a random conversation with a stranger, who is a guitarist and IT professional like myself, and I now know of Animals As Leaders. Fantastic band!
All in all, pretty much the BEST day ever!!!
Also…if you are a musician or a music business in Australia, check out Rockstar Hookups for your music classifieds needs!
benxtan 