Wednesday, December 6, 2023
HomeSoftware EngineeringLakehouse Information Stack with Raj Bains

Lakehouse Information Stack with Raj Bains

As firms transfer to Spark and a Lakehouse structure, they’re realizing that the info instruments are lagging method behind.  You could be a programmer to successfully use Spark and Airflow.

There are some low-code ETL instruments, however is that sufficient?  Firms need to deal with their information pipelines like mission-critical apps.  They need DevOps for information, with GIT, check protection, and CI/CD.  As well as, firms want end-to-end visibility of their information pipelines, together with monitoring, metadata, and column-level lineage.

The place are the info instruments for these capabilities and the way do non-programmers use these?  

In in the present day’s podcast, you’ll see how Prophecy’s low-code information engineering platform, constructed on Spark and Airflow, offers an entire answer for growing, deploying, and monitoring information pipelines.  We’ll converse with Raj Bains, who’s the founder & CEO of Prophecy.  

Raj has spent 20 years constructing highly effective instruments – Microsoft Visible Studio, founding engineer for CUDA at NVIDIA, product supervisor for Hive at Hortonworks. He’s obsessed with making all organizations productive with information and making the lives of knowledge engineers higher by decreasing toil and growing the enjoyment.

Additionally, should you like what you hear, Prophecy has a particular provide for our listeners.  Go to Prophecy.IO/request-a-demo and put in “SED” for the referral code.  For those who resolve to buy inside 3 months of the demo, you get 10% off.  If you are going to buy after 3 months, you continue to get 5% off.

Sponsorship inquiries:




Transcript supplied by We Edit Podcasts. Software program Engineering Every day listeners can go to to get 15% off the primary three months of audio modifying and transcription providers with code: SED. Due to We Edit Podcasts for partnering with SE Every day. Please click on right here to view this present’s transcript.


WorkOS is a developer platform to make your app enterprise-ready. With a couple of easy APIs, you may instantly add widespread enterprise options like Single Signal-On, SAML, SCIM consumer provisioning, and extra. Builders will discover stunning docs and SDKs that make integration a breeze. WorkOS is sort of like “Stripe for enterprise options.” WorkOS powers apps like Webflow, Hopin, Vercel, and greater than 100 others. The platform is rock stable, absolutely SOC-2 compliant, and prepared for even the most important enterprise environments. So what are you ready for? Combine WorkOS in the present day and make your app enterprise-ready. To study extra and get began, go to

Cox Automotive transforms the best way that the world buys, sells, owns, and makes use of automobiles – and so they have the info to grasp how the world will get from level A to level B. With manufacturers like Kelley Blue Guide, Autotrader, and others, Cox Automotive is hiring software program engineers, information scientists, scrum masters, and different technologists to assist create significant change within the business. If you wish to innovate in a collaborative office, one which values your time and work-life steadiness, then go to  COXAUTOTECH.COM  to search out profession alternatives at Cox.

Puppet, the business chief for infrastructure automation, invitations you to a hearth chat to discover ways to improve collaboration and construct a tradition of joint accountability for compliance and safety throughout your group.

Be a part of Alex Hin, Senior Product Supervisor at Puppet, for a dialogue of how IT, safety, compliance, and threat administration groups can work collectively to guard your group from threat.

You possibly can be a part of Puppet for the dialog on Thursday, April seventh at 11 AM jap and begin constructing your individual tradition of accountability that enhances compliance by registering at


strongDM allows you to handle and audit entry to servers, databases, and Kubernetes clusters, irrespective of the place your workers are. With strongDM, you may simply lengthen your id supplier to handle infrastructure entry. You possibly can automate onboarding, offboarding, and transferring individuals inside roles. strongDM. Handle and audit distant entry to infrastructure. Begin your free 14 day trial in the present day at:

Information engineers combating unreliable information depend on Monte Carlo, the world’s first end-to-end, absolutely automated Information Observability Platform! Monte Carlo allows information groups with visibility into the standard and reliability of their analytical information to maximise enterprise affect. Begin trusting your information with Monte Carlo in the present day! Go to to study extra.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments