Social Media Monitoring

WARNING: This article is critical of the Loughborough University Enterprise Office (an initiative to help commercials ideas & work from academia into viable business oportunities).

Well over a year ago (back in 2009, actually the idea came in 2008) I entertained myself with the idea of starting up a social media monitoring business. I have worked on the topic within my PhD and I have published on the topic and also did some decent work with a student of mine back through 2008/2009 academic year. Our results were great, winning me an IEEE best PhD student paper award at a conference among some encouraging feedback from my paper reviewers.

The idea was to use my student's existing server infrastructure (he runs a web hosting service in Poland) to accumulate web 2.0 datasets and apply data-mining and statistical techniques summarisation, and finally to build a nice and fancy User Interface based on edgy web-design techniques (which I lectured about to my large Introduction to Web Programming undergraduate class last year) and further wrap it all up into ontologies to be usable by semantic web capable agents!

The above sounded like a decent Business Plan to me, with relatively low risk, as I could have done this part time (aligned with the PhD) and we could have just used, as mentioned, my students server infrastructure to a degree. See this article for a very timely overview of Social Media Monitoring tools, and how significant they have become in business (There is much more academic work, looking at many case studies - drop me a line if you want some references to those).

We therefore decided to seek some initial funding or at least support from Loughborough's student enterprise office, in one form or another and depending on their response we wanted to begin development of the initial prototypes! After a few email exchanges and several phone conversations, I was very dissapointed...

The ent. office consultant was obviously overworked, and complained about her high volume of meetings, business trips and other responsibilities. Once I mentioned that we have published and received an award, her reaction shocked me. Apparently since the work was published we could not patent it, and hence without any possibilities of patenting she lost any interest. I was shocked by this reaction, assuming that patents still rule the world of software is something I consider quite ridiculous (just like the once click amazon buy button patent), in my opinion a barrier to innovation that's what my whole experience dealing with the entr. office at loughborough felt like.

It is ironic that reading the BBC article today, I realise that with a little bit support we could have had a finished beta product by today, covering local demand in social media monitoring in the Midlands area. Great way to throw logs under the feet of young university talent, Loughborough Student Enterprise Office, well done ;-)!

Any comments are welcome, at same time, I do know the Office was behind a few interesting projects, however they have a lot of improvement ahead to become worthy for a university of Loughboroughs Profile!

Prediction API & Big Data API

Google has branched out in many directions since their initial search-engine & adwords success. The company has had such healthy profits (mainly from their internet based advertising) that they were able to dab in pretty much every current interesting application in Computer Science - see here!

For a while they are providing cloud computing services, such as Google Storage for Developers, check out the pricing of that service here. Google maintains all the data within their own infrastructure. I think this article tries to explains how the distributed storage is implemented (of course just a very generic overview). You will notice that the service is naturally scalable and pretty smart in a number of ways.

The most recent activity of google resulted in the announcement of two new APIs (Prediction API and BigData API). The diagram below shows how these fit together.



BigData is used to query a large cloud store (using an SQL dialect over a webservice) and the Prediction API can be used on the data to train google implemented AI models for prediction. This simply seems to be a machine learning library that can be accessed over a webservice. Obviously this runs on google cloud infrastructure and that has it's advantages.

A number of Machine Learning libraries exist, such as WEKA, RapidMiner and many other. I used to write some of my own code for these algorithms, however over the last few years I noticed an amazing increase in the count of ML libraries. In most of my work these days I use open source libraries.

I am not quite sure how the pricing of these APIs works (maybe somebody can enlighten us on this issue), my impression is it is connected to the Google cloud store service, for which these APIs will present another reason to use this store.

You can check out some code samples for the API here.