Data Engineering with AI Assistants

Objective: Set up the AI assistant you will use all week, preview the major topics of the course with prompts that test (rather than replace) your understanding, and practice the working loop used throughout the course: specify, generate, verify, defend. Estimated time: 1-2 hours, before or during the Day 1 morning session.

Data engineers now work alongside AI assistants that write pandas code, draft SQL, and explain protocols in seconds. That does not make this course easier; it moves the value. When code is cheap, what is scarce is the engineer who can specify the problem precisely, verify the result against reality, and take responsibility for the data that feeds downstream decisions. Every module this week includes AI prompts that quiz you, plant bugs for you to find, and audit your choices - this assignment sets that pattern.

πŸ›  Step 0: Set Up Your Tools

  1. Choose the Generative AI assistant you will use this week (e.g., Claude, ChatGPT, Gemini, or Copilot). A free tier may be rate-limited during a full course day.
  2. Install the course TA skill in your assistant (instructions for Claude, ChatGPT/Codex, and Gemini are in the archive). It makes the AI course-aware: the 5-day schedule, every module page, the TCLab examples, pandas and protocol references, and the course policy of coaching instead of solving.
  3. Confirm you can run Python: either install Python with numpy, pandas, and matplotlib, or open any course notebook in Google Colab from the module pages.

πŸ—Ί Step 1: Preview the Course with AI Prompts

Work through these four prompts, one at a time, answering the AI's questions yourself before asking for explanations. Each previews one day of the course.

Prompt 1 β€” Python Foundations (Day 1)

"I am a practicing engineer in a Data-Driven Engineering short course. Ask me 5 questions, one at a time, to find out what I already know about Python data structures (tuple, list, set, dictionary) and NumPy/pandas. Grade my answers, then give me a prioritized 30-minute review plan for the gaps."

Prompt 2 β€” Data Access (Day 2)

"Quiz me with 4 questions, one at a time, on getting engineering data into Python: reading a CSV of vehicle or meter data with pandas, when a SQLite database beats flat files, what a microcontroller (ESP32) adds as a data source, and what is legally and technically different about web scraping vs using an API. Grade my answers and list my misconceptions."

Prompt 3 β€” Industrial Data Streams (Day 4)

"Act as a senior automation engineer interviewing me. Ask me 4 questions, one at a time, about moving industrial data: what Modbus, MQTT, and OPC UA each are and roughly when each is used, and what a plant historian does. I am allowed to say 'I don't know' - when I do, give me the two-sentence version I should remember, then move on. End with an overall assessment."

Prompt 4 β€” Time Series and Models (Day 4)

"Explain, without heavy math, why forecasting a process variable from its own past values (an autoregressive model) works, and what 'data leakage' means when preparing time-series data for such a model. Then ask me 3 questions to check I understood, one at a time, and correct my answers."

πŸ” Step 2: The Working Loop - Specify, Generate, Verify, Defend

Run one complete loop on a realistic mess:

  1. Specify: Give the AI this prompt: ->"Generate a 40-row CSV of 15-minute smart-meter readings (timestamp, meter_id, kWh) that is deliberately malformed: mixed date formats, a thousands separator in some values, a duplicated row, an 'N/A' and a blank cell, one negative reading, and a column renamed halfway down. Keep a private list of the defects and do not reveal it yet."
  2. Generate: Write (or direct the AI to write) pandas code that loads this CSV and produces a clean, numeric, time-indexed DataFrame. You decide what "clean" means: what gets fixed, flagged, or dropped.
  3. Verify: Run the code. Check dtypes, row count, describe(), and a plot. Then ask the AI to reveal the defect list and score yourself: caught, missed, or silently mangled.
  4. Defend: For the negative kWh reading, write two sentences defending your decision (drop, flag, or correct) as if a colleague will bill customers from this data.

✍ Step 3: Write Your Two Reusable Prompts

Review the prompt-structure pages from Machine Learning for Engineers: Agentic Workplan, Agentic Coding, and Agentic Reports. Write two reusable prompts you will use this week: one for learning a new topic (quiz-me style) and one for checking generated code or data (audit style). Each should state context, task, constraints, and how the answer will be verified.

πŸ“‹ What to Turn In

Start a course workbook (a notebook, document, or wiki page you will add to every day this week). Make this the first entry (about one page). You may use Generative AI to help write it, but you must supply the correct outputs, justifications, and assumptions. Answer these questions:

  1. From Step 1: for each of the four preview prompts, one thing you learned or one question you missed (with the corrected answer).
  2. From Step 2: your defect scorecard (caught / missed / mangled), the pandas call that fixed the hardest defect, and your two-sentence defense of the negative-reading decision.
  3. Include your two reusable prompts from Step 3, and name the AI assistant you set up (with the TA skill installed).

Course on GitHub

Python

Access

Communicate

Electrical

Series

Data Engineering

Applications

Object Detection πŸ‘οΈ
Text OCR πŸ‘οΈ
Pose Estimation πŸ‘οΈ
Electromagnetic Relay βš™οΈ
Servo Control SG90 βš™οΈ
Solid State Relay βš™οΈ
Fired Heater Control ⏱️
RAG Similarity Search πŸ—£οΈ
Generative AI πŸ—£οΈ
RAG LLM Integration πŸ—£οΈ
Biomechanics πŸ“ˆ
Air Pressure πŸ“
CO2 Sensors πŸ“
Humidity πŸ“
Light Intensity πŸ“
Location GPS πŸ“
Temperature πŸ“
πŸ‘οΈ=Computer Vision
βš™οΈ=Actuators
⏱️=Time Series
πŸ—£οΈ=Language
πŸ”Š=Audio
πŸ“ˆ=Regression
πŸ“Š=Classification
πŸ“=Sensors

Related Courses

Admin