Big Data Systems and Analytics

Name: Big Data Systems and Analytics
Author: Chintan Thakkar

Learn about Big Data processing systems.

Preview the first lesson free — get full access to all 4 lessons.

Course: On-Demand

(electronic music) Welcome to Big Data Systems and Analytics. In these lessons, you'll learn about storage systems designed for big data, like Hadoop and its architecture, which includes Hadoop Distributed File System, MapReduce, and different types of available nodes. To process big data, you need a centralized application that allows you to submit, then store data into a relational database, but it's not possible to store large datasets in a relational database because the database will develop a bottleneck. These applications typically only support small systems that can't handle big data. Google solved this problem with the MapReduce programming model. The MapReduce algorithm performs the following functions. Dividing a big job into smaller parts. Executing smaller jobs independently. Integrating results into a single dataset. If you have a large problem to solve, then you can break the algorithm into several smaller tasks, and each task executes independently on distributed infrastructure. Hadoop is designed to run the MapReduce algorithm. Hadoop systems run the MapReduce algorithm, which breaks the job down into several small tasks. It processes the data in parallel, meaning these smaller tasks are carried out simultaneously so it can perform statistical analysis on huge amounts of data at a time. Other benefits of using the Hadoop architecture are that it can help us with handling a situation where if one node fails, then another node will take care of processing. Hadoop can process the data up to the size of one petabyte in parallel. It also provides a scalability solution, meaning that the node can dynamically be added for processing more data. The Hadoop Distributed File System provides reliability, scalability, and availability. Data is replicated across the system for processing in a distributed banner. The files are distributed in large blocks and are replicated to enable optimized parallel reads. With MapReduce, Hadoop executes the job using the data stored on the distributed file system using the MapReduce algorithm. It helps us make computations efficiently with the stored data. Apache Pig is an open-source project that is built on top of MapReduce algorithms. It helps us to create the data analysis program. You can analyze the large data sets using Apache Pig. It also helps us to display the data as data flows. You need to write the data analysis program using the Pig language, and this program will be converted into map and reduce tasks internally. There are several benefits of using Pig. It's similar to structured query language. If you know the query language, then it's easy to pick up this language. It supports operators such as joins and sorting, similar to the relational database. It can handle both structured and unstructured data. Apache Pig is a high-level language built on the MapReduce abstraction layer. MapReduce can be difficult to learn compared to Apache Pig, but being an expert in Java programming can help with learning it. Apache Pig also allows you to write sample script that is already compiled and will generate the MapReduce job internally. The MapReduce job requires a longer compilation time and allows developers to store the data anywhere in the pipeline. It provides the libraries that enable us to create the extract, transform, and loading jobs. Here are some of the most common uses for Apache Pig. Convert data into parquet format. Search log files from the petabytes of data. Process large size data and generate aggregates. Perform image conversion. Apache Hive is also a platform for generating MapReduce operations. Hive provides support for ad hoc queries for generating data aggregates and also provides various techniques that help with the analysis of large datasets. It supports language similar to the SQL server query language. Hive generally works well with structured data compared to Apache Pig and works on different data types, such as structured, unstructured, and semi-structured. Hive allows us to write the script using HiveQL, a query processing and declarative language.

Beginner Provider : Chintan Thakkar 4 Lessons · 18m in Arabic, German, English, Spanish, French, Portuguese, Urdu, Chinese

Course Description

To process Big Data, you need a centralized application that allows you to submit then store data into a relational database. In this Big Data Systems and Analytics course you will learn about storage systems designed for Big Data like Hadoop and its architecture, which includes Hadoop Distributed File System, MapReduce, and different types of available nodes.

The Hadoop Distributed File System provides reliability, scalability, and availability. With MapReduce, Hadoop executes the job using the data stored on the Distributed File System and helps users make computations. Apache Pig is built on top of MapReduce algorithms. It helps users create the data analysis program and analyze large datasets. This course will explain the process of how to run Big Data anyltics on Hadoop.

What You'll Learn

Identify the functions of MapReduce algorithms
Explain what the Hadoop Distributed File System (HDFS) is and its relation to Big Data
List the most common uses of Apache Pig
Recall the nodes in the HDFS architecture
Explain the steps for creating files in HDFS
Describe how to run Big Data analytics on Hadoop

Key Takeaways

Processing Big Data requires a centralized application that lets you submit and store data into a relational database.
Hadoop is a storage system designed for Big Data, with an architecture that includes the Hadoop Distributed File System, MapReduce, and different types of available nodes.
The Hadoop Distributed File System provides reliability, scalability, and availability.
With MapReduce, Hadoop executes jobs using data stored on the Distributed File System and helps users make computations.
Apache Pig is built on top of MapReduce algorithms and helps users create data analysis programs and analyze large datasets.

Frequently Asked Questions

What does this course cover?

The course covers storage systems designed for Big Data like Hadoop and its architecture, including the Hadoop Distributed File System, MapReduce, and different types of available nodes, as well as Apache Pig and the process of running Big Data analytics on Hadoop.

What is the Hadoop Distributed File System and why does it matter for Big Data?

The Hadoop Distributed File System provides reliability, scalability, and availability, and the course explains what it is and its relation to Big Data.

What is Apache Pig used for according to this course?

Apache Pig is built on top of MapReduce algorithms and helps users create data analysis programs and analyze large datasets; the course also lists its most common uses.

What skills will I gain from this course?

You will gain skills in Big Data, Big Data Analytics, Data Processing Systems, Database Systems, the Hadoop Distributed File System (HDFS), and Oracle Big Data.

What topics are taught in the lessons?

The lessons include Introduction to Hadoop, Hadoop Architecture, How to Run Big Data Analytics on Hadoop and HDFS, and the MapReduce Algorithm.

Professional Certifications and Continuing Education Units (CEUs)

Society for Human Resource Management (SHRM®)

Professional Development Credits (PDCs): 0.5

Certification Program Categories:

Leadership & NavigationBusiness AcumenConsultationGlobal MindsetEthical PracticeRelationship ManagementAnalytical AptitudeCommunicationDiversity, Equity & Inclusion

KnowledgeCity is approved by SHRM as a Recertification General Provider to offer SHRM-CP or SHRM-SCP professional development credits (PDCs). By taking the courses approved by SHRM, KnowledgeCity can award SHRM Professional Development Credits (PDCs) for HR knowledge and competency programs related to the SHRM Body of Applied Skills and Knowledge™ (the SHRM BASK™).