Data Engineering with AI Assistants
Objective: Set up the AI assistant you will use all week, preview the major topics of the course with prompts that test (rather than replace) your understanding, and practice the working loop used throughout the course: specify, generate, verify, defend. Estimated time: 1-2 hours, before or during the Day 1 morning session.
Data engineers now work alongside AI assistants that write pandas code, draft SQL, and explain protocols in seconds. That does not make this course easier; it moves the value. When code is cheap, what is scarce is the engineer who can specify the problem precisely, verify the result against reality, and take responsibility for the data that feeds downstream decisions. Every module this week includes AI prompts that quiz you, plant bugs for you to find, and audit your choices - this assignment sets that pattern.
π Step 0: Set Up Your Tools
- Choose the Generative AI assistant you will use this week (e.g., Claude, ChatGPT, Gemini, or Copilot). A free tier may be rate-limited during a full course day.
- Install the course TA skill in your assistant (instructions for Claude, ChatGPT/Codex, and Gemini are in the archive). It makes the AI course-aware: the 5-day schedule, every module page, the TCLab examples, pandas and protocol references, and the course policy of coaching instead of solving.
- Confirm you can run Python: either install Python with numpy, pandas, and matplotlib, or open any course notebook in Google Colab from the module pages.
πΊ Step 1: Preview the Course with AI Prompts
Work through these four prompts, one at a time, answering the AI's questions yourself before asking for explanations. Each previews one day of the course.
Prompt 1 β Python Foundations (Day 1)
Prompt 2 β Data Access (Day 2)
Prompt 3 β Industrial Data Streams (Day 4)
Prompt 4 β Time Series and Models (Day 4)
π Step 2: The Working Loop - Specify, Generate, Verify, Defend
Run one complete loop on a realistic mess:
- Specify: Give the AI this prompt: ->"Generate a 40-row CSV of 15-minute smart-meter readings (timestamp, meter_id, kWh) that is deliberately malformed: mixed date formats, a thousands separator in some values, a duplicated row, an 'N/A' and a blank cell, one negative reading, and a column renamed halfway down. Keep a private list of the defects and do not reveal it yet."
- Generate: Write (or direct the AI to write) pandas code that loads this CSV and produces a clean, numeric, time-indexed DataFrame. You decide what "clean" means: what gets fixed, flagged, or dropped.
- Verify: Run the code. Check dtypes, row count, describe(), and a plot. Then ask the AI to reveal the defect list and score yourself: caught, missed, or silently mangled.
- Defend: For the negative kWh reading, write two sentences defending your decision (drop, flag, or correct) as if a colleague will bill customers from this data.
β Step 3: Write Your Two Reusable Prompts
Review the prompt-structure pages from Machine Learning for Engineers: Agentic Workplan, Agentic Coding, and Agentic Reports. Write two reusable prompts you will use this week: one for learning a new topic (quiz-me style) and one for checking generated code or data (audit style). Each should state context, task, constraints, and how the answer will be verified.
π What to Turn In
Start a course workbook (a notebook, document, or wiki page you will add to every day this week). Make this the first entry (about one page). You may use Generative AI to help write it, but you must supply the correct outputs, justifications, and assumptions. Answer these questions:
- From Step 1: for each of the four preview prompts, one thing you learned or one question you missed (with the corrected answer).
- From Step 2: your defect scorecard (caught / missed / mangled), the pandas call that fixed the hardest defect, and your two-sentence defense of the negative-reading decision.
- Include your two reusable prompts from Step 3, and name the AI assistant you set up (with the TA skill installed).