Data Ingestion and Transformation — AWS Certified Data Engineer - Associate

Official references for this objective

• Amazon Web Services — Using Amazon S3 as a target for AWS Database Migration Service - AWS Database Migration Service

  For more compact storage and faster query options, you also have the option to have the data written to Apache Parquet (.parquet) format.

• Amazon Web Services — Working with streaming data on AWS - Build Modern Data Streaming Architectures on AWS

  If open-source technology is critical for your data processing strategy, you're familiar with Apache Kafka and you're looking for real-time latency in less than 70

• Amazon Web Services — Using Lambda to process records from Amazon Kinesis Data Streams - AWS Lambda

  Lambda event source mappings process each event at least once, and duplicate processing of records can occur.

• Amazon Web Services — Building visual ETL jobs - AWS Glue

  AWS Glue triggers can start jobs based on a schedule or event, or on demand.

• Amazon Web Services — Develop producers using the Amazon Kinesis Producer Library (KPL) - Amazon Kinesis Data Streams

  Writes to one or more Kinesis data streams with an automatic and configurable retry mechanism Collects records and uses PutRecords to write multiple records to

• Amazon Web Services — Getting started with streaming ingestion from Amazon Kinesis Data Streams - Amazon Redshift

  Amazon Redshift streaming ingestion uses a materialized view, which is updated directly from the stream when REFRESH is run.

• Amazon Web Services — Working with AWS DMS tasks - AWS Database Migration Service

  showing issues with a supported source and target database that can cause problems during a migration. This can include issues such as unsupported data types,

• Amazon Web Services — What is AWS Database Migration Service? - AWS Database Migration Service

  Data at rest is encrypted with AWS Key Management Service (AWS KMS) encryption. During migration, you can use Secure Socket Layers (SSL)

• Amazon Web Services — AWS Glue ETL - AWS Prescriptive Guidance

  Python shell jobs are designed for running basic ETL scripts written in Python. These jobs run on a single machine, and are better suited for

• Amazon Web Services — Transform data with AWS Glue managed transforms - AWS Glue

  Union : Combine rows from more than one data source that have the same schema.

• Amazon Web Services — Transform source data in Amazon Data Firehose - Amazon Data Firehose

  Amazon Data Firehose supports a Lambda invocation time of up to 5 minutes.