Batch Processing — Data Engineer (Associate)

The official ALIBABA documentation our Batch Processing practice questions are cited to. Review the primary sources, then practise.

Official references for this objective

Alibabacloud — OpenAI Batch API - Alibaba Cloud Model Studio - Alibaba Cloud Documentation Center

Maximum size: 1 MB . Maximum lines: 100 . Concurrency limit: Up to 2 parallel tasks.
Alibabacloud — Batch inference - Alibaba Cloud Model Studio - Alibaba Cloud Documentation Center

Up to 50,000 requests per file, and a maximum file size of 500 MB. If your dataset exceeds these limits, split it into multiple files
Alibabacloud — Batch System: Batch Compute and Cloud-native Serverless Argo Workflows - Alibaba Cloud Community

Portability Weak, only applicable to specific cloud vendor ecosystems Strong, compatible with open-source ecosystems, and support cross-cloud running
Alibabacloud — Cloud-native Offline Workflow Orchestration: Kubernetes Clusters for Distributed Argo Workflows - Alibaba Cloud Community

Argo Workflows allow you to create a DAG to specify the dependencies of each task
Alibabacloud — Flexibly Utilize Cloud Message Service in Your Business - Alibaba Cloud Community

Trigger data load job as soon as file arrives from external system. File watcher application can send file arrival message to Message Service queue
Alibabacloud — What is Data Integration, features of Data Integration - DataWorks - Alibaba Cloud Documentation Center

Batch scenario : One-time full synchronization followed by periodic incremental synchronization. Suitable for data sources that do not require high data timeliness