Background
Our Client needs to build an Advanced Analytics Platform for Oil and Gas Industries. They have various types of data sources from where they were getting Data for Oil and Gas Industry.
They have to buy a subscription for FTP Server in which they got EBCDIC files every month for Oil and Gas for Texas State and also have to get Historical Data from the year 2000 till now regarding for all states in the US, and they have already stored it in SQL Server.
How We build this Platform?
We start working on building this platform in collaboration with Client and following are steps for developing this platform:
Data Collection from Batch Data Sources:
We were having two batch data sources. One was SQL Server and Other was FTP Server having EBCDIC files. So, We use Apache Beam on Cloud Data Flow for collecting data from both data sources and then we store the data as it is to Data Lake.
Data Collection from Streaming Data Sources (Google Pub-Sub) :
As we explained that we were also getting data from Sensors attached to Oil & Gas refinery Machines. So Python Code was deployed inside the Sensor which was writing various states of Machines to Google Pub-Sub. So We use Google Pub-Sub as our Real-time Data Stream.
Predictive and Other Advanced Analytics
So We build and train our models using the Data in BigQuery and then run those ML and DL Algorithms on incoming data to satisfy the defined goals for Oil and Gas Refinery.
Steps Included (Model building using Deep learning algorithms) :
Explore Data
- Launch Cloud Datalab
- Invoke BigQuery
- Draw graphs in Cloud Datalab
Creating a Sampled datasets
- Clone repository
- Run notebook
Creating a Tensorflow Model (Wide and Deep)
- Develop a Tensorflow model in Datalab on a small sampled dataset.
Preprocessing using cloud Dataflow.
- Preprocess data at scale for machine learning.
Training on cloud ML Engine.
- Train large-scale model, hyperparameter tune it, and create a model ready to deploy.
Deploy trained model
- Deploy a trained model as a microservice which can do both online and batch prediction.
Use Model
- Deploy a Model web application that consumes the machine learning service.
- We will use the model on the data and will be transferred and which can be used on cloud bigtable.
Other Support
We also provides the following support to the client: Network Operation Center: Xenon Stack provides 24 * 7 support for maintaining the REST API’s, Databases, Hadoop and Spark Clusters in order to have zero failure or downtime.