Musify

This is a project that me and a couple other people worked on at WildHacks 2016, Northwestern University. We decided to go for 'The best use of Clarifai's API' and went through a few ideas before finalizing one.


The basic idea of our project was to provide background music for videos. Quite simply the videos would be passed through our own custom trained Clarifai agent and decide on a mood.

Pixabay. The website we scraped to get the images.

Clarifai is an Artificial Intelligence with a Vision. Basically it's an image and video recognition API. We decided to work with a custom model and trained it on the images we scraped using Pixabay. We landed upon 4 different categories for the video emotions - Action, Sad, Happy, Cinematic.


Clarifai Preview Screen. The preview portal for our custom model.

One basic look at the code for the main file of our code makes it crystal clear that this was a hackathon project. We used a bunch of python libraries as we required. For example, we used pydub to manipulate sound according to video specifications.


Finally one of the main aspect of our project was to actually find the music suited to our project. We obviously didn't have enough time nor the skills to come up with a machine learning to actually make music on the fly so we relied on JukeDeck's music library. JukeDeck had an abudance of pre-made music on their site sorted by similar titles as we had originally decided on. We built our own database and used that to add it to our videos. JukeDeck doesn't have a public API at the moment so we emailed the people there but since it was a weekend, we didn't hear back from them.


We used jukedeck to get the music according to the video specifications.

Before

After

We also tried to design a proper front-end for the client side of the app but we didn't have enough time so we just submitted it as a command line interface.


The proposed UI for the Musify application.

I'm looking to make this into a fully functional webapp with more tags for videos and proper jukeDeck integration. Meanwhile here are the relevant links to the project.