Naveen Nelamali·Mar 25, 2024What is Adaptive Query Execution and How it Improves PySpark PerformanceAdaptive Query Execution (AQE) is one of Spark 3.0's greatest features. It reoptimizes and adjusts query plans based on runtime statistics…
Naveen Nelamali·Mar 25, 2024PySpark — SparkAI and Its Mose Used Methods with ExamplesThis is the second article in a series on leveraging the pyspark-ai package in PySpark to use the natural English language. In my first…
InTowards AIbyNaveen Nelamali·Mar 24, 2024Using English SDK in PySparkHave you ever wondered what PySpark AI English SDK is and how it simplifies using PySpark with out learning complex SQL and DataFrame API
InSparkByExamplesbyNaveen Nelamali·Jan 14, 2024Spark Executor Memory Overhead: Understanding & Best PracticesSpark executor memory overhead refers to additional memory allocated beyond the user-defined executor memory in Apache Spark
InSparkByExamplesbyNaveen Nelamali·Jan 12, 2024When to use Hive Partitioning and Bucketing?Hive Partition is a way to split a large table into smaller tables based on the values of a column(one partition for each distinct value)
InSparkByExamplesbyNaveen Nelamali·Jan 12, 2024Do you know you can use Variables in Hive?Hive variables are key-value pairs that can be set using the set command and they can be used in scripts and Hive SQL.
InSparkByExamplesbyNaveen Nelamali·Jan 12, 2024How to Union Pandas DataFrames?In pandas, you can use the concat() function to concatenate or union the Pandas DataFrames along with a particular axis.
InSparkByExamplesbyNaveen Nelamali·Jan 12, 2024Why Avoid UDFs in Spark & PySpark?User-Defined Functions (UDFs) in Spark can incur performance issues due to serialization overhead, necessitating the conversion of data…
InSparkByExamplesbyNaveen Nelamali·Jan 12, 2024What are the Specific Roles of Spark Driver and ExecutorHave you ever wondered what the different roles of Apache Spark Driver and Executor play when running your application in a distributed…A response icon1A response icon1
InSparkByExamplesbyNaveen Nelamali·Jan 11, 2024Introduction to Spark-Submit: A Comprehensive Guide to Submitting Spark ApplicationsEvery thing you need to know about spark-submit
