Scala's mix of functional and object oriented programming paradigms is ideal for building data pipelines. Many smart people have noticed this, and as a result, we now have three distinct APIs for creating MapReduce pipelines in Scala (Scrunch, Scoobi, and Scalding), as well as a standalone data processing framework developed purely in Scala (Spark). Although all of these tools showcase the power and elegance of Scala, there is a larger opportunity for the community to give pipeline developers a single API to use for both in-memory and batch-style data processing. We'll discuss some of the challenges in creating an API that balances the features of both in-memory and batch processing and demonstrate the benefits that this creates in ETL development, exploratory data analysis, and machine learning.

No outline is available