Spark performance tuning scenario-based interview questions for data engineers and architects. Complete course available here.

How to grow from data engineer to a solution architect?. Do you know the answer or know the career path?. I know one thing. To be a solution architect, you need to start being a solution architect. No one is offering you a role that you are not already…

Big data, data lake, data ingestion and lakehouse architects!. Here is a design scenario and a bunch of interview questions for data ingestion. You have data arriving in a Kafka cluster and you are asked to design a data ingestion pipeline to bring data…

What are the 5-S of Spark performance tuning?.These are the critical areas of performance bottlenecks in a Spark application and are subject to tuning and need special care and follow best practices. Here is the list. 1. Skew.2. Spill.3. Shuffle.4. Stora…

Common misconception. Spark Dataframe supports structured and semi-structured data. But for unstructured data, you must use RDD APIs. Here is another way of saying the same statement. Spark Dataframe supports CSV and JSON. But for reading a text and imag…

Performance tuning is one of the favorite topics for interviewers. Here are some scenario-based interview questions on performance tuning for Bigdata Data engineers. Q1: You have a Spark application that is running slow. How would you approach investiga…

Data Engineers!.What is the most popular data engineering skill for big data engineers?.SQL is the most critical skill. What is the next one?. PySpark and Databricks. Do you want to begin learning PySpark?.Do you know PySpark but want to expand your knowl…

Looks like many of us are running out of job 🤣😂😭

What are the most common questions aspiring Data Engineers or Bigdata engineers ask?. Some are listed below, and let me know if you could relate to any of these. 1. I have work experience in relational databases and data warehouses. How do I transition i…

Here are some easy Data Engineering questions asked in recent interviews and shared by students. Well! It may not be so easy and obvious. Do you know the answers?. How can you find the size of a data frame?.Be careful with this question. The interviewer…

ScholarNest Team at Bangalore. We have a small and passionate team at Bangalore. Some more members working remotely couldn't make it to our weekend gathering. We are growing and a lot more to come.

Data Engineers!.What is the most popular data engineering skill for big data engineers?.SQL is the most critical skill. What is the next one?. PySpark and Databricks. Do you want to begin learning PySpark?.Do you know PySpark but want to expand your knowl…

Do you have a question or query that you want to ask?.Please don't hesitate to ask away anything related to my expertise or your career. I will try my best to help you out 🙂.It is free of cost on Topmate :-)

Now you can also catch me or follow me on topmate. I am offering four services on the platform. This will allow people who want to connect with me for any career guidance or have a question that I might be able to answer.

A Spark interview question that I keep on hearing now and then. Do you know the answer?.If not, you can get your answer in 60 seconds by watching the video. ----------------------------.PS: Unlock your career change and growth in B…

One Spark interview question that I keep on hearing every now and then. Q: If Spark can spill the data to disk, why would it fail with the OOM - out-of-memory exception?. I heard many answers. However, most of them are beating around the bush. Here are…

Have you heard about Microsoft Fabric yet?. A new entry in the market for data engineering and Lake house implementation. It is still in preview mode and not a marketing gimmick. It appears really powerful with no code or low code interface. What exactl…

Just checked.I don't have any of those 2000 Notes. 😆😆😆😆

Last day to avail discount. Fnd your best learning resources for data engineers?.Follow these links. Many of them are also available in Udemy for business. Start your Spark programming journey from the below link. Learn some adv…

Do you need to find the best learning resources for data engineers?.I have a personal learning list for subjects related to big data and data engineering. Start your Spark programming journey from the below link. Learn some adva…