Apache Spark Addresses Data Lake Challenges

By Paula Bernier October 12, 2018

Many organizations have taken the plunge with data lakes. But many of these efforts are still just treading water. But there’s good news. Apache Spark can help businesses realize the promise of big data lakes.

McKinsey & Co. notes that data lakes are appealing because “data are loaded in ‘raw’ formats rather than reconfigured as they enter company systems.” That means they can be used for more than just basic capture. However, the firm adds, integrating data lakes with other elements of the technology architecture, setting use rules, and finding appropriate talent can be challenges.

Meanwhile, author and consultant Dan Woods in a January article for Forbes has this to say about the subject: “With the data lake, while companies can store massive amounts and varieties of data, they have been unable to effectively manage that data and allow a large number of people with moderate expertise levels to explore the data, come up with useful queries, extract the signal through some regular production process that becomes part of the way a business runs.

Some companies, like Netflix, have managed to operationalize a data lake …. But in most other businesses, the data lake got stuck at the proof of concept stage. That is why in general, the data lake is now in need of salvation. The point of saving the data lake is to understand how we go from having a repository of data with signals to operationalizing that information to provide value to the business.”

One of the challenges is that many companies with data lakes are using expensive, proprietary solutions for data ingestion, integration, and transformation related to them. But now many of these same operations are putting a toe in the waters with Apache Spark. And that’s a good thing, because Apache Spark is a strong distributed computing framework that handles end-to-end analytics, data processing, and machine learning requirements.

In a webinar next week, Anand Venugopal, AVP and business head at StreamAnalytix, will offer more details on why Apache Spark is the answer.

Venugopal will present information about cloud-based IoT use cases with event-time, late-arrival, and watermarks. He’ll talk about Python-based predictive analytics running on Spark in cloud environments. And he’ll offer information on visual interactive development of Apache Spark Structured Streaming pipelines.

He’ll also talk about using Apache Spark related to on-premises data lakes. That conversation will explore on-premises advanced monitoring of Spark pipelines. He’ll also discuss data quality and ETL with Apache Spark using pre-built operators in on-premises environments.

Last year around this time Ian Pointer for InfoWorld wrote “From its humble beginnings in the AMPLab at U.C. Berkeley in 2009, Apache Spark has become one of the key big data distributed processing frameworks in the world. Spark can be deployed in a variety of ways, provides native bindings for the Java, Scala, Python, and R programming languages, and supports SQL, streaming data, machine learning, and graph processing. You’ll find it used by banks, telecommunications companies, games companies, governments, and all of the major tech giants such as Apple, Facebook, IBM, and Microsoft.”

Get stories like this delivered straight to your inbox. [Free eNews Subscription]

Executive Editor, TMC

Related Articles

$100M Towards Innovative Capital Options: AppDirect Secures New Funding from CDPQ

By: Alex Passett    1/9/2024

New funding from CDPQ provides AppDirect with the power to help technology providers and advisors in transforming and scaling their businesses while m…

Read More

NextPlane's Affordable Microsoft Teams PSTN Calling Solution for UCaaS and Service Providers

By: TMCnet Staff    5/23/2023

NextPlane's Affordable Microsoft Teams PSTN Calling Solution for UCaaS and Service Providers

Read More

Snapchat Announces its New ChatGPT-Powered 'My AI' Chatbot

By: Alex Passett    2/27/2023

The popular app Snapchat is now offering a new friend, of sorts: My AI. Powered by ChatGPT, the bot is already integrated for select users to experime…

Read More

Black-owned Businesses See Vision Become Reality with Dialpad, and Sacramento King's Davion Mitchell

By: Greg Tavarez    2/24/2023

Dialpad is further expanding its Tech for Black Founders program and partnership with Sacramento Kings' point guard Davion Mitchell to promote Black-o…

Read More

IDEA Showcase 2023 at ITEXPO Gives Startups Opportunity for Exposure

By: Greg Tavarez    2/16/2023

IDEA Showcase 2023 at ITEXPO in Ft. Lauderdale, Florida, gave entrepreneurs a chance to present pitches to a panel of judges.

Read More