Wrestling Big Data computing down to size (MIT ILP)

The MIT Industrial Liaison Program (ILP) offers a great window into cutting-edge research at the Institute, and an opportunity for companies to partner with leading researchers. One example is Alan Edelman’s work leading the global, open-source collaboration developing Julia, a powerful but flexible programming language for high performance computing (HPC). The ILP reports:

HPC Programming with Ease
By Eric Brown

As high performance computing (HPC) bends to the needs of big data applications, speed remains essential. Yet, it’s not only a question of how quickly one can compute problems, but how quickly one can program the complex applications that do so.
“In recent years, people have started to do many more sophisticated things with big data, like large scale data analysis and large scale optimization of portfolios,” says Alan Edelman, Professor of Applied Mathematics at MIT’s CSAIL. “There’s demand for everything from recognizing handwriting to automatically grading exams.”

The challenge is that there are only so many programmers capable of such wizardry, and the programs are getting more and more complex and time-consuming to develop. “At HPC conferences people tend to stand up and boast that they’ve written a program so it runs 10 or 20 times faster,” says Edelman. “But it’s the human time that in the end matters the most.” Read more…

Prof. Edelman’s course 18.337J / 6.338J Parallel Computing has 10 tutorial videos about Julia, along with perspectives on big data and cloud computing, and it’s in OCW.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s