Intro To Data Engineering, Week 13

Meeting Purpose

Introduce and begin work on LeapCode algorithm problems, starting with 'Container With Most Water'.

Key Takeaways

• Added a new 'LeapCode' folder to the repo containing classic algorithm problems
• Students work through each problem independently in breakout rooms before group discussion
• First problem included an explanation file, a stubbed solution file, and tests

Meeting Purpose

Discuss and solve the 'Container With Most Water' coding problem.

Key Takeaways

• Solved the problem using a two-pointer approach, moving the shorter pointer inward each step
• A visual walkthrough of the technique significantly improved understanding for the group
• The JavaScript solution passed all LeetCode tests on first submission

Meeting Purpose

Introduce the new data engineering module and Python basics to the class.

Key Takeaways

• Python was chosen for its readability, simplicity, versatility, and job-market demand
• Covered syntax basics and primitive data types: strings, integers, floats, booleans, and None
• Recommended VS Code extensions: the Python suite, a Black formatter, and iSort

Meeting Purpose

Discuss Python virtual environments, lists, and list comprehensions.

Key Takeaways

• Virtual environments isolate project dependencies; created with python3 -m venv and requirements.txt
• Covered common list methods: append, insert, remove, pop, sort, and reverse
• List comprehensions provide concise syntax for transforming or filtering a list in one line

Meeting Purpose

Daily ACE Academics meeting to discuss the Leap Code problem: grouping anagrams using JavaScript's built-in Map object.

Key Takeaways

• Task: group anagrams from an array of strings using a Map keyed by sorted characters
• Reviewed the Map methods needed: instantiation, has(), and set()
• Session had low attendance due to it following National Coffee Day

Meeting Purpose

Review a HashMap-based solution to the LeetCode 'Group Anagrams' problem.

Key Takeaways

• Solution sorts each string's characters to build a unique key for the HashMap
• Each original string is stored in the array associated with its key
• Running the test suite confirmed the solution before submitting to LeetCode

Meeting Purpose

Provide an overview of Python, covering dictionaries, functions, and a guessing-game exercise.

Key Takeaways

• Compared Python to JavaScript and C#, covering dictionaries and the .get() method
• Built a phone book app exercise using dictionaries to add, look up, and list contacts
• Introduced functions with default parameters, named arguments, and type hints, then built a number-guessing game

Meeting Purpose

Review and discuss solutions for a LeetCode problem on finding the length of the longest substring without repeating characters.

Key Takeaways

• Solved with a sliding-window technique and a hashmap tracking each character's last seen index
• Achieved O(n) time complexity, outperforming a naive O(n^2) approach
• The solution passed 94% of LeetCode's test cases

Meeting Purpose

Review Python data engineering concepts and build a data-cleaning project together.

Key Takeaways

• Built a modular ETL pipeline for cleaning library checkout data with Pandas
• Structured the project with a main pipeline script and a utilities package
• Debugged issues with dropping rows that had missing fee values

Meeting Purpose

Review and refine the data-engineering ETL process for a library data project, and introduce an upcoming API mini-project.

Key Takeaways

• Fixed issues dropping rows with spaces, nulls, and 'N/A' entries, and used fillna() for missing fees
• Broke the ETL process into reusable utility functions like drop_rows, fill_values, and coerce_types
• Introduced the next mini-project: extracting from a live REST API and loading into SQLite

Meeting Purpose

Discuss and solve a LeetCode problem on searching in a rotated sorted array.

Key Takeaways

• Applied a modified binary search, checking which half of the array is sorted to guide the search
• Implemented the solution in both JavaScript and Python
• The Python solution outperformed 91% of submissions in runtime

Meeting Purpose

Demonstrate an API-to-SQLite pipeline and prepare students for their own projects.

Key Takeaways

• Extracted data from a public API, flattened nested JSON with pandas.json_normalize, and loaded it into SQLite
• Handled naming conflicts and converted camelCase fields to snake_case
• Introduced python-dotenv for managing API keys ahead of students building their own pipelines

Meeting Purpose

Daily algorithm practice: solve 'Climbing Stairs' using memoization to optimize a recursive solution.

Key Takeaways

• Compared recursive-without-memoization (O(2^n)), memoized recursion (O(n)), and iterative (O(n) time, O(1) space) approaches
• Memoization caches prior results in a map to avoid redundant recursive calls
• The naive recursive approach without memoization times out for larger inputs

Meeting Purpose

Review student-built ETL pipelines pulling from a wide range of public APIs into SQLite.

Key Takeaways

• Students presented pipelines built on APIs including iTunes, NASA APOD, Harry Potter, Spotify, Steam, and MLB The Show
• Feedback focused on error handling, code organization, and consistent naming conventions
• Several projects added extras like data visualizations and country-flag rendering in the terminal