Intro To Data Engineering, Week 14

Meeting Purpose

Review and solve the 'Longest Common Subsequence' algorithm problem using dynamic programming.

Key Takeaways

• Defined a 2D DP table where each cell represents the LCS length of two string prefixes
• If characters match, extend the previous diagonal value; otherwise take the max of the neighboring cells
• Visualizing the DP table's fill process was key to understanding the algorithm

Meeting Purpose

Cover working with Pandas, modularizing pipeline code, and scheduling scripts with Cron and Task Scheduler.

Key Takeaways

• Structured pipeline code into separate extract, transform, and load modules for readability and testability
• Covered ArgParse for command-line arguments and try/except blocks for graceful error handling
• Demonstrated scheduling the pipeline with Cron (Mac/Linux) and Windows Task Scheduler

Meeting Purpose

Review and discuss a coding solution for converting Roman numerals to integers.

Key Takeaways

• Used a map-based approach, subtracting when a numeral is less than the one after it and adding otherwise
• The solution passed all local tests and ran at roughly 90% efficiency on LeetCode
• Walked through the object mapping, running total, and comparison logic in detail

Meeting Purpose

Introduce the importance of logging in Python scripts, especially for remote or cron job scenarios where print statements aren't visible.

Key Takeaways

• Set up Python's logging module and configured the minimum log level
• Covered the different log levels and when to use each
• Emphasized logging as essential for diagnosing scheduled or unattended script runs

Meeting Purpose

Discuss a coding challenge to validate the validity of a Sudoku board.

Key Takeaways

• Input is a 2D array representing a Sudoku board; the task is to check for repeated digits
• Checked each row, column, and 3x3 sub-box for duplicate values
• Used hash-based sets to efficiently track digits already seen in each region

Meeting Purpose

Introduce automated testing, explaining the testing pyramid, and set up PyTest for the first time.

Key Takeaways

• Covered the testing pyramid: unit tests at the base, integration tests in the middle, end-to-end at the top
• Set up PyTest and the pytest-testdocs package for more readable output
• Wrote the first unit test using the Arrange-Act-Assert pattern, deliberately starting from a failing test

Meeting Purpose

Discuss striving for 100% test coverage and write tests for the reorder_columns and analyze modules.

Key Takeaways

• Encouraged writing at least one test per function to approach full coverage
• Compared two approaches for testing reorder_columns's column ordering and preservation
• Introduced PyTest fixtures to set up and tear down a temporary SQLite database for each test

Meeting Purpose

Discuss and solve the 'Integer to Roman' algorithm problem as part of the ACE Academics curriculum.

Key Takeaways

• Used parallel arrays of Roman symbols and values, repeatedly subtracting the largest fitting value
• Multiple solutions ranged from about 65% to over 96% efficiency on LeetCode
• Tracking the largest index already used avoided rechecking larger values on each pass

Meeting Purpose

Introduce integration testing for the extract stage of the ETL process using mocked API responses.

Key Takeaways

• Used the requests-mock library to simulate API calls rather than hitting the real API in tests
• Verified both the returned data and that a 200 status code and correct headers were received
• Planned a follow-up test covering the extract function's handling of a non-200 response

Meeting Purpose

Practice algorithmic thinking using logic puzzles as a break from coding.

Key Takeaways

• Worked through a 'Cake Maker' grid logic puzzle using given clues
• Introduced the 'Four Numbers' math puzzle and LSAT-style logic questions as additional practice
• Recommended an interactive resource for building intuition around Big O notation

Meeting Purpose

Discuss and implement the 'Happy Number' algorithm using two different approaches.

Key Takeaways

• A happy number eventually reaches 1 when repeatedly replaced by the sum of the squares of its digits
• Implemented a set-based approach to detect cycles, plus Floyd's Tortoise and Hare for a more space-efficient version
• Compared the tradeoffs between the two approaches for cycle detection

Meeting Purpose

Discuss approaches to the 'Make Change' problem: finding the fewest U.S. coins needed to make a given amount.

Key Takeaways

• Discussed starting with the largest coin denominations and visualizing the problem before coding
• Agreed that writing pseudocode first helps break down the problem before implementation
• Discussed the value of a simple first solution before optimizing for efficiency

Meeting Purpose

Review integration tests for the load module and add a test verifying database column order.

Key Takeaways

• Verified the load function creates a database table, writes rows, and includes the expected columns
• Debugged an end-to-end pipeline test where extraction was returning None due to a missing URL/token
• Added a test using PRAGMA TABLE INFO to confirm the database's column order matched expectations