What is big data and big data Hadoop complete overview

Got what they needed for that what they need to do. They need to make sure that they analyze this data. So they realize that big data analytics can solve a lot of problems and they can get better business inside through that. So let us move forward and see what type of analysis they did on that data. So before analyzing the data they came to know that energy.

Do utilization and billing were only increasing now after analyzing Big Data the game to know that during Peak load the users require more energy and during off-peak times that users require less energy. So what advantage they must have got from this analysis. One thing that I can think of right now is they contain the industries to use their Machinery only during the off-peak times so that the load will be pretty much balanced and you can even say that time-of-use pricing encourages costs aviary. They like industrial heavy machines to be used off big time. So yeah, they could save money as well because off-peak times pricing will be less than the peak time prices, right? So this is just one analysis. Now, let us move forward and see the IBM sweet that they developed so over here what happens you first dump all your data that you get in this data warehouse after that. It is very important to make sure that your user data is secure then what happens you need to clean that data as I've told you We as well there might be many pieces that you don't require. So you need to make sure that you have only useful material or use data in your data set and then you perform certain analyses. And in order to use this sweet that IBM offered you efficiently, you have to take care of a few things. The first thing is that you have to be able to manage the smart meter data. Now there is a lot of data coming from all these million Smart Meters, so you have to be able to manage that large volume of data and also be able to retain it because maybe a



You're on you might need it for some kind of regulatory requirements or something. And this thing you should keep in mind is to monitor the distribution grid so that you can improve and optimize the overall grid reliability so that it can identify the abnormal conditions which are causing any kind of problem. And then you also have to take care of optimizing the unit commitment. So by optimizing the unit commitment that companies can satisfy their customers even more they can reduce the power outages. That is big and reduced. Power outages so that their customers don't get angry more identify problems and then reduce it obviously and then you have also to optimize the energy trading. So it means that you can advise your customers when they should use their appliances to maintain that balance in the power load and then you also have to forecast and schedule loads. So companies must be able to predict when they can profitably sell the Excess power and when they need to hedge the supply.





Let's talk about how Encore has made use of the I-beam solution. So Encore is an electric delivery company and it is the largest electrical distribution and transmission company in Texas and it is one of the six largest in the United States. They have more than 3 million customers and their service area covers almost 

100  17,000 square miles and they begin the advanced meter program in 2008.

And they have deployed almost three point two five million meters serving customers of North and Central Texas. So when they were implementing it they get three things in mind. The first thing was that it should be instrumented. So this solution utilizes the smart electricity meters so that they can accurately measure the electricity usage of a household every 15 minutes because like we discussed that those Smart Meters were sending out data every 15 minutes.

And it provided data inputs that are essential for consumer insights. Next thing is that it should be interconnected. So now the customers have access to the detailed information about the electricity they are consuming and it creates a very enterprise-wide view of all the meter acid and it helped them to improve the service delivery. The next thing is to make your customers intelligent now since it is getting monitored already about how each of the household or each customer is. Zooming power. So now there can advise the customers about maybe Tell them to wash their clothes at night because they're using a lot of appliances during the daytime. So maybe they could divide it up so that they can use some appliances at off-peak hours so that they can even save more money and this is beneficial for both of them for both the customers and the company as well and they have gained a lot of benefits by using the IBM solution. So what are the benefits they got is that is enabled on the court to identify and fix outages before the Murmurs get an inconvenience. That means they were able to identify the problem before it even occurred and it also improved the emergency response on events of severe weather events and views of outages and it also provides the customers the data needed to become an active participant in the power consumption management and it enables every individual household to reduce their electrical consumption by almost five to 10% And this is how Encore use the IBM solution and made huge benefits out of it.

Just by using big data analytics, that idea is performed, but let me just interrupt right now. So since Ray Smart told us in the beginning as well that there are no free lunches in life. Right? So this is an opportunity but there are many problems to encase this opportunity, right? So let us focus on those problems one by one. So the

The first problem is storing colossal amounts of data. So let us discuss a few stars that are there in front of your screen. The data is already in the past two years is more than the previous history in total. So guys, what are we doing table generating so much amount of data is said that by 2020 total Digital Data will grow to 44 zettabytes approximately and there is one more start that amazes me is about 1.7 m be of new information will be created every

And for every person by 2020 so storing this huge data in a traditional system is not possible. The reason is obvious. The storage will be limited to one system. For example, you have a server with a storage limit of 10 terabytes that your company is growing fast and data is exponentially increasing now what you'll do now at one point, you'll exhaust all the storage so investing in huge servers is not a cost-effective solution. So raised for what do you think? What? And be the solution to this problem according to me a distributed file system will be a better way to store this, uh data because with this will be saving a lot of money. Let me tell you how they can.

Due to this distributed system You can store your data in commodity Hardware instead of spending money on high-end servers. But let me tell you guys it is just one part of the problem. Let's see if you more. So since we saw that the data is not only huge but it is present in various formats as well, like unstructured semi-structured and unstructured so you not only need to Store this huge data, but you also need to make sure that a system is present to store this variety of data generated from various sources, and now let's focus on the next problem. Now, let's focus on the diagram. So what are you can notice that the hard disk capacity?

This is increasing by the distance of performance or speed is not increasing at that rate. Let me explain to you this with an example. If you have only a 100 Mbps input-output Channel and you are processing say one terabyte of data. Now, how much time will it take maybe calculate I will be somewhere around three-point nine winners, right? So we'll be somewhere around two points nine minutes and I've taken an example of one terabyte. What if your

Tossing some zettabytes of data so you can imagine how much time left. Now what if you have four input-output channels for the same amount of data, then it will take approximately point seven two hours or converted to minutes. So it will be around 43 minutes approximately right? And now imagine instead of 1tb. You have Z device of data for me more than storage accessing and processing speed for huge data is a bigger problem.

So if there was no solution for it, if it would take so much time to access the data the recommendation system won't work at all and they make a lot of money just for the recommendation system because a lot of people go there and click over there by that product. Right? So let's consider that that is taking like ours or maybe years to process my that big amount of data.