Lecture 3: Collections

BAA1118 - Foundations of Programming for Analytics & AI

Damien Dupré

For Today

Requirements

  • ✅ Assigning values to names
  • ✅ int, float, str, bool
  • ✅ Reading an error message without panicking

Programme

  1. Lists
  2. Indexing and slicing
  3. Dictionaries, tuples and sets
  4. Nested structures
  5. How data is stored on disk

The Problem With One Value Per Name

sales_jan = 1200
sales_feb = 1450
sales_mar = 1310

This works for 3 months. It does not work for 3,000 customers.

  • You cannot loop over separate names
  • You cannot ask “how many are there?”
  • You cannot sort them

Collections solve this: one name, many values.

1. Lists

Creating a List

Square brackets, values separated by commas.

sales = [1200, 1450, 1310]
print(sales)
print(len(sales))       # how many
[1200, 1450, 1310]
3
print(sum(sales))       # total
print(max(sales))       # largest
print(sorted(sales))    # a new, sorted list
print(1450 in sales)    # membership
3960
1450
[1200, 1310, 1450]
True

A list can hold anything, even a mix of types, but a useful list holds one kind of thing.

Adding and Removing

countries = ["Ireland", "France", "Spain"]

countries.append("Italy")        # add at the end
print(countries)

countries.insert(1, "Germany")   # add at position 1
print(countries)
['Ireland', 'France', 'Spain', 'Italy']
['Ireland', 'Germany', 'France', 'Spain', 'Italy']
countries.remove("France")       # remove by value
print(countries)

last = countries.pop()           # remove and return the last
print(last, countries)
['Ireland', 'Germany', 'Spain', 'Italy']
Italy ['Ireland', 'Germany', 'Spain']

Methods Change the List Itself

This is the big difference with strings.

scores = [5, 3, 9]
scores.sort()          # changes scores, returns nothing
print(scores)
[3, 5, 9]
scores = [5, 3, 9]
result = scores.sort()
print(result)          # None!
None

Important

list.sort() sorts in place and returns None. sorted(list) returns a new sorted list. Writing scores = scores.sort() destroys your data.

Your Turn: A Shopping List

  1. Create a list called basket with these prices: 4.50, 12.99, 3.25, 8.00
  2. Add 19.99 at the end
  3. Remove 3.25
  4. Print how many items are in the basket, the total, and the most expensive item
  5. Print the basket sorted from cheapest to most expensive, without changing basket

Solution

basket = [4.50, 12.99, 3.25, 8.00]
basket.append(19.99)
basket.remove(3.25)

print(len(basket))
print(sum(basket))
print(max(basket))
print(sorted(basket))
print(basket)      # unchanged
4
45.48
19.99
[4.5, 8.0, 12.99, 19.99]
[4.5, 12.99, 8.0, 19.99]

2. Indexing and Slicing

Indexing

countries = ["Ireland", "France", "Spain", "Italy", "Portugal"]
Value Ireland France Spain Italy Portugal
Index 0 1 2 3 4
From the end -5 -4 -3 -2 -1 
print(countries[0])
print(countries[-1])    # the last, without knowing the length
Ireland
Portugal
print(countries[5])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[48], line 1
----> 1 print(countries[5])

IndexError: list index out of range

Slicing

list[start:stop:step] returns a new list, from start up to but not including stop.

print(countries[1:4])
print(countries[:2])     # from the start
print(countries[-2:])    # the last two, to the end
['France', 'Spain', 'Italy']
['Ireland', 'France']
['Italy', 'Portugal']
print(countries[::2])    # every second element
print(countries[::-1])   # reversed
['Ireland', 'Spain', 'Portugal']
['Portugal', 'Italy', 'Spain', 'France', 'Ireland']

Leaving a side empty means “all the way”. [::-1] works on strings too.

Mutability and the Copy Trap

Lists are mutable: an element can be replaced in place.

a = [1, 2, 3]
b = a          # b is not a copy!
b[0] = 99      # replace the first element of b
print(a)       # a has changed too
[99, 2, 3]

b = a gives the same list a second name. To copy:

a = [1, 2, 3]
b = a.copy()   # or list(a), or a[:]
b[0] = 99
print(a, b)
[1, 2, 3] [99, 2, 3]

Warning

This trap reappears in week 7 with DataFrames, where pandas warns you about it explicitly.

Your Turn: Slicing Quarters

monthly = [120, 135, 148, 160, 155, 149, 171, 168, 180, 190, 205, 240]
  1. Print the first quarter (first 3 months) and its total
  2. Print the last quarter and its total
  3. Print every third month
  4. Print the list in reverse
  5. Replace the value for June (index 5) with 152 and print the new annual total

Solution

monthly = [120, 135, 148, 160, 155, 149, 171, 168, 180, 190, 205, 240]

q1 = monthly[:3]
q4 = monthly[-3:]
print(q1, sum(q1))
print(q4, sum(q4))
print(monthly[::3])
print(monthly[::-1])

monthly[5] = 152
print(sum(monthly))
[120, 135, 148] 403
[190, 205, 240] 635
[120, 160, 171, 190]
[240, 205, 190, 180, 168, 171, 149, 155, 160, 148, 135, 120]
2024

3. Dictionaries, Tuples and Sets

The Problem With Positions

customer = ["Damien", 42, "Dublin", 1250.50]

What is customer[2]? You have to remember. In six months, you will not.

A dictionary stores values by name instead of by position.

customer = {
    "name": "Damien",
    "age": 42,
    "city": "Dublin",
    "spend": 1250.50
}
print(customer["city"])
Dublin

Curly braces, unique keys, and values of any type, even lists. Think of it as a single row of a spreadsheet with its column headers.

Reading and Writing

customer = {"name": "Damien", "city": "Dublin", "spend": 1250.50}

print(customer["name"])       # read
customer["spend"] = 1400.00   # update
customer["country"] = "IE"    # add a new key
del customer["country"]       # remove a key
Damien
print(customer["email"])
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[57], line 1
----> 1 print(customer["email"])

KeyError: 'email'

get() reads safely, returning None or a default instead of an error:

print(customer.get("email", "not provided"))
not provided

Exploring a Dictionary

print(customer.keys())
print(customer.values())
dict_keys(['name', 'city', 'spend'])
dict_values(['Damien', 'Dublin', 1400.0])
print("city" in customer)     # tests the keys
print(len(customer))
True
3
for key, value in customer.items():
    print(key, "->", value)
name -> Damien
city -> Dublin
spend -> 1400.0

We will unpack that for line properly next week.

Tuples: Lists That Cannot Change

Round brackets instead of square ones.

coordinates = (53.3853, -6.2450)
print(coordinates[0])
print(len(coordinates))
53.3853
2
coordinates[0] = 0
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[63], line 1
----> 1 coordinates[0] = 0

TypeError: 'tuple' object does not support item assignment

Use a tuple when the collection is not supposed to change: a coordinate, a date, a pair of values returned by a function.

Unpacking

latitude, longitude = coordinates
print(latitude)
print(longitude)
53.3853
-6.245

This works for lists too, and is used constantly:

first, second, third = ["a", "b", "c"]
print(second)
b

You will see it in week 9 as fig, ax = plt.subplots().

Sets: Unique Values Only

regions = ["North", "South", "North", "East", "South", "North"]
print(set(regions))
print(len(set(regions)))
{'South', 'East', 'North'}
3

Sets answer “which distinct values are in this column?”, and compare groups with no library at all:

customers_2023 = {"A", "B", "C", "D"}
customers_2024 = {"C", "D", "E"}

print(customers_2023 & customers_2024)   # retained
print(customers_2023 - customers_2024)   # churned
print(customers_2024 - customers_2023)   # new
{'D', 'C'}
{'B', 'A'}
{'E'}

The Four Collections

Syntax Ordered Changeable Duplicates
list [1, 2, 3] yes yes yes
tuple (1, 2, 3) yes no yes
dict {"a": 1} yes yes keys: no
set {1, 2, 3} no yes no

In practice: list for a column of values, dict for a labelled record, set for uniqueness, tuple for fixed pairs.

Your Turn: A Customer Record

  1. Create a dictionary customer with keys id, name, city and orders, where orders is a list of order amounts: 45.0, 120.5, 33.25
  2. Add a new order of 78.0 to the list inside the dictionary
  3. Print the number of orders and the total spent
  4. Print the average order value, rounded to 2 decimals, using an f-string
  5. Safely print the value of a key email that does not exist

Solution

customer = {
    "id": "C001",
    "name": "Damien",
    "city": "Dublin",
    "orders": [45.0, 120.5, 33.25]
}

customer["orders"].append(78.0)

n = len(customer["orders"])
total = sum(customer["orders"])

print(n, total)
print(f"Average order: {total / n:.2f}")
print(customer.get("email", "no email on file"))
4 276.75
Average order: 69.19
no email on file

4. Nested Structures

A List of Dictionaries

This is how almost all real data arrives, from an API, a JSON file, or a database.

customers = [
    {"name": "Damien", "city": "Dublin", "spend": 1250.50},
    {"name": "Aoife",  "city": "Cork",   "spend": 890.00},
    {"name": "Liam",   "city": "Dublin", "spend": 2100.75}
]

Read the brackets left to right:

# customers        -> the list
# customers[1]     -> the second dictionary
# ["city"]         -> the value under "city" in that dictionary
print(customers[1]["city"])
print(customers[-1]["name"])
Cork
Liam

Why This Matters

customers
[{'name': 'Damien', 'city': 'Dublin', 'spend': 1250.5},
 {'name': 'Aoife', 'city': 'Cork', 'spend': 890.0},
 {'name': 'Liam', 'city': 'Dublin', 'spend': 2100.75}]

Turn your head sideways: names across the top, one record per row. This is a table.

import pandas as pd
pd.DataFrame(customers)    # week 7: one line converts it
name city spend
0 Damien Dublin 1250.50
1 Aoife Cork 890.00
2 Liam Dublin 2100.75

A Table Has Two Directions

Each row is one observation: one product, one customer, one country-year.

Each column is one variable: one thing measured about every observation.

Source: UBC Master of Data Science, Programming in Python for Data Science

Your Turn: Nested Data

sales = [
    {"month": "Jan", "region": "North", "units": 120},
    {"month": "Jan", "region": "South", "units": 95},
    {"month": "Feb", "region": "North", "units": 138},
    {"month": "Feb", "region": "South", "units": 101}
]
  1. Print the number of records
  2. Print the region of the third record and the units of the last
  3. Add a fifth record for March, North, 150 units
  4. Print the total units of the first two records only

Solution

sales = [
    {"month": "Jan", "region": "North", "units": 120},
    {"month": "Jan", "region": "South", "units": 95},
    {"month": "Feb", "region": "North", "units": 138},
    {"month": "Feb", "region": "South", "units": 101}
]

print(len(sales))
print(sales[2]["region"])
print(sales[-1]["units"])

sales.append({"month": "Mar", "region": "North", "units": 150})

print(sales[0]["units"] + sales[1]["units"])
4
North
101
215

Adding up the whole column still needs a loop. That is next week.

5. Data on Disk

Where Data Actually Lives

Everything so far has been typed into the notebook. Real data arrives as a file, and you will meet four formats over and over.

Format Looks like Holds
.csv text, one row per line one table
.json text, nested brackets anything, even nested records
.xlsx an Excel document sheets, formatting, formulas
.parquet a binary data file one table, typed and compressed

Today you only need to recognise them and know what shape of Python each one maps onto. Reading them comes in week 5.

CSV: A Table as Text

Comma Separated Values. Open it in a text editor and you can read it:

roasted_stores.csv
store,city,county,seats,floor_m2
Grafton Street,Dublin,Dublin,42,96
Camden Street,Dublin,Dublin,28,64
Blanchardstown,Dublin,Dublin,60,140
  • First line: the column names
  • One line per row, values separated by commas
  • No types. 42 is the text “42” until something decides otherwise
  • Nothing else: no formulas, no colours, no second sheet

That is a list of dictionaries written down. It is the format you should ask for.

The CSV Trap

a_real_export.csv
name,city,note
"Murphy, John",Cork,"He said ""grand"" and left"
Ó Súilleabháin,Baile Átha Cliath,
  • A comma inside a value, so the value is quoted
  • A quote inside a quoted value, so it is doubled
  • An empty value at the end of the line
  • Accented characters, which need the file to be saved as UTF-8

This is why you never split a CSV by hand with .split(","). A proper reader handles all of it, and you will use one from week 5.

JSON: Python, Written Down

JavaScript Object Notation. Look closely at this:

customers.json
[
  {"id": "C0148", "name": "Aoife",   "store": "Grafton Street", "spend": 24.90},
  {"id": "C0201", "name": "Cian",    "store": "Camden Street",  "spend": 61.40},
  {"id": "C0377", "name": "Saoirse", "store": "Grafton Street", "spend": 9.80}
]

You wrote that ten minutes ago. JSON is a list of dictionaries, saved as text. Square brackets for a list, curly braces for a dictionary, colons for keys.

JSON: Python, Written Down

The json module turns one into the other. Imports come in week 5, so read this rather than worrying about the first line:

import json

text = '[{"name": "Aoife", "spend": 24.90}, {"name": "Cian", "spend": 61.40}]'
customers = json.loads(text)

print(type(customers))
print(type(customers[0]))
print(customers[1]["spend"])
<class 'list'>
<class 'dict'>
61.4

Almost every website and app sends its data as JSON, precisely because it can hold nested structures that a table cannot: a customer with a list of orders inside them.

Excel: A Document, Not a Data File

An .xlsx file is a zipped folder of XML. It can hold:

  • Several sheets
  • Formulas that recalculate
  • Merged cells, colours, comments, charts, pivot tables
  • Three different date formats in the same column

Warning

Excel is excellent for people and awkward for code. When you are given one, expect a title row above the headers, a blank column somebody used as a spacer, and totals in the middle of the data. Half of week 7 is dealing with exactly that.

Parquet: The One Built for Data

PAR1 ... (binary, unreadable in a text editor) ... PAR1
  • Typed: a date column comes back as a date, not as text
  • Columnar: reading one column of a hundred does not read the other ninety-nine
  • Compressed: usually a third the size of the same CSV
  • Unreadable by humans, and that is the whole trade

It is the standard for data that is passed between systems rather than emailed to a person. You will meet it in any serious data job, and almost never in an inbox.

Thanks for your attention and don’t hesitate to ask if you have any questions!

@damien_dupre

@damien-dupre

https://damien-dupre.github.io

damien.dupre@dcu.ie