SuperGlue – Mass Media Metadata Generator

Summary

SuperGlue is a “digestion system” and metadata generator for mass media. An evolving set of analysis modules annotate 14 DirecTV live news broadcast channels as well as web pages and tweets. The video is archived and synchronized with the analysis. The system provides named-entity extraction, audio expression markers, face detectors, scene/edit point locators, excitement trackers, and thumbnail summarization and more. This is used to organize material for presentation, analysis, and summarization. SuperGlue supports other news-related experiments.

Contributions

From summer 2018 to winter 2018, Mike architected and executed the modularization of SuperGlue, and moved most infrastructure to the cloud (Google Cloud Platform, Amazon Web Services, Compose.io). Furthermore, he wrote a news summarization module that uses NLP and machine learning to generate news summaries from news video metadata. The videos are then served to the mobile app Enlightened through an API. The resulting metadata are visualized on this Dashboard. The production pipeline is visualized here.

In spring 2019, he co-led 3 MIT undergraduates to develop a set of modules extracting facial expressions, sentiment of news scrolls, story segmentation and foreground-background separation (Unspoken News). He also developed machine learning models to study the relationship between extracted low-level features and high-level perception of bias (DeepBias).

In summer 2019, he scaled this project by leading 11 MIT & Wellesley College undergraduate developers and designers on seven sub-projects: API development, video processing infrastructure development, speech diarisation, body language detection, foreground-background separation and background scene analysis, machine learning and data visualization.

Skills

Python, Flask, Google uis-rnn, Posenet, Tensorflow, Keras, PyTorch, Scikit-learn, Numpy, Seaborn, Google Vision API, IBM Watson NLP, Affectiva SDK, MongoDB, Redis, Compose.io, Papertrail, AWS S3, Bubble, Postman, Docker, Docker Cloud, Google Kubernetes Engine, Heroku, Systems Architecture (Modularization, APIs), Leadership.