The book's release year is 2020 by John Wiley & Sons, Inc., Indianapolis, Indiana
Dan Sullivan
Principal Engineer and Software architect. He specialised in:
1. Data Science,
2. Machine Learning,
3. Cloud Computing.
- Dan is the author of the
1. Official Google Cloud Certified Professional Architect Study Guide.
2. Official Google Cloud Certified Associate Cloud Engineer Study Guide.
3. NoSQL for Mere Mortals.
- Dan has certifications from:
1. Google.
2. AWS
along with a Ph.D. in genetics and computational biology from Virginia Tech.
Business requirements = stakeholder requirements specifications (StRS), describe the characteristics of a proposed system from the viewpoint of the system's end user like a CONOPS.
- Professional Data Engineer
- Professional Data Engineers enable "data-driven decision making" by:
1. collecting,
2. transforming,
3. Publishing data.
- A Data Engineershould be able to:
1. Design,
2. Build,
3. Operationalize,
4. Secure,
5. Monitor data processing systems
with a particular emphasis on security and compliance.
- A Data Engineer should also be able to:
1. Leverage,
2. Deploy,
3. Continuously train pre-existing machine learning models.
- The Professional Data Engineer exam assesses your ability to:
1. Design data processing systems,
2. Ensure solution quality,
3. Operationalize machine learning models.
4. Build and Operationalize data processing systems.
- These GCP storage services are fully managed to scalable and backed by industry leading SLAs.
1. Cloud SQL: Cloud SQL: This fast and compatible storage service allows managing relational, NoSQL, object storage, data warehouse to In memory. These GCP storage services are fully managed to scalable and backed industry leading SLAs.
Cloud Spanner: Another fully managed, relational Google Cloud database service, Cloud Spanner differs from Cloud SQL by focusing on combining the benefits of relational structure and non-relational scalability. It provides consistency across rows and high-performance operations and includes features like built-in security, automatic replication, and multi-language support.
3. Firestore: Firestore is a flexible, scalable NoSQL cloud database to store and sync data for a client- and a server-side development
4. Cloud Bigtable: Cloud Bigtable a fully-managed non-relational database that is suitable for both real-time access and analytics workloads. It is an excellent solution for large-scale, low-latency applications as well as intensive data analytics such as IoT, personalisation, recommendations, monitoring and geospatial datasets.
4. Cloud Bigtable: Cloud Bigtable is a fully-managed non-relational database that is suitable for both real-time access and analytics workloads. It is an excellent solution for large scale, low-latency apps as well as intensive data analytics such as IoT, personalisation, recommendations, monitoring and geospatial datasets.
5. Cloud Storage: Cloud Storage is one of the many storages available on GCP. This highly scalable service can manage an unlimited amount of objects up to 5TB each, such as images and content files.
6. BigQuery: With BigQuery you can perform data analysis via SQL and query stream-data. Since BigQuery is a serverless data warehouse that's fully managed, it built-in Data Transfer Service helps you migrate data from on-premises resources, including Teradata.
7. Memory Store: Designed to be secure, highly available, and scalable, Cloud Memorystore is a fully managed, in-memory Google Cloud data store that enables you to create application caches with sub-millisecond latency for data access.
-Business Requirements to Storage Systems
- Data Engineers will use different types of storage systems for different purposes.
- The specific storage system you should choose is determined, in large part, by the stage of the data lifecycle for which the storage system is used.
- The data lifecycle consists of four stages:
1. Ingest.
2. Store.
3. Process and Analyze.
4. Explore and Visualize.
- What is data lifecycle management?
- Data lifecycle management (DLM) is an approach to managing data throughput its lifecycle, from data entry to data destruction.
- Data is separated into phases based on different criteria, and it mves through these stages as it completes different tasks or meets certain requirements.
- A Good DLM process provides structure and organization to a business's data, which in turn enables key goals within the process, such as data security and data availability.
- Phases of data lifecycle management:
1. Data creation.
2. Data storage.
3. Data sharing and usage
4. Data archival.
5. Data Deletion.
-Benefits of dta lifecycle management
- Data lifecycle management has several important benefits which include:
1. Process improvement.
2. Controlling costs.
3. Data usability.
4. Compliance and governance.
Ingest
-
Comments
Post a Comment