Evidence

Every number produced in the analysis of the full file, as plain tables. Each carries its unit and the query that produced it. The SQL is in sql/10_evidence.sql; the machine-readable output, including the exact statement beside every result, is out/evidence.json.

Dataset and its limits

Dataset: 311 Service Requests – Customer Initiated, City of Toronto
Resource: 311 Service Requests 2025 · f3db05ab-2588-4159-89f7-56c74d1d8201
Licence: Open Government Licence – Toronto
Rows: 500,269
Period: 2025-01-01 to 2025-12-31
File encoding: cp1252
Fields present: Creation Date · Status · First 3 Chars of Postal Code · Intersection Street 1 · Intersection Street 2 · Ward · Service Request Type · Division · Section
Fields absent: Free-text description (stripped before publication) · latitude / longitude (not published)
Coverage limitation: The City documents that this dataset covers 6 of 45 city divisions and roughly 30–35% of all 311 contacts, only requests geospatially validated at intake. Confirmed against the file: it contains 6 operating divisions plus 311 itself. It is not a census of city problems.

Coverage and canonicalisation

Measure	Value	Unit
Total requests	500,269	requests
With both intersection streets	77,833	requests
Share with both streets	15.56	percent
With one street only	624	requests
Distinct canonical intersections	13,155	intersections
Distinct service request types	539	types
Distinct divisions	7	divisions
Distinct wards	25	wards
Rows where street normalisation changed the raw string	4,357	rows
Clusters spanning more than one ward	8 of 64,438	clusters (0.012%)

Source: queries coverage and cross_ward_clusters. The normalisation figure is reported so the reader can see how much work the canonical-key function is really doing: the source data is city-generated from a street centreline file and already consistently abbreviated, so the answer is “not much”.

Duplicate rate

Cluster = (service request type, canonical intersection, 14-day head-anchored window). The first report opens a cluster and anchors it; later reports within 14 days of the anchor join it; the first report past 14 days opens a new cluster. Head-anchored rather than chained: chaining would let one busy intersection form a single cluster running all year and would wildly overstate deduplication.

Measure	Value	Unit
Requests examined (intersection-located)	77,833	requests
Clusters	64,438	clusters
Clusters of 2 or more	6,855	clusters
Requests in a cluster of 2 or more	20,250	requests
Duplicate rate	26.02	percent of examined
Redundant reports (closable as duplicates)	13,395	requests
Redundancy	17.21	percent of examined
Redundancy as a share of the whole file	2.68	percent of 500,269
Largest cluster	278	reports

Source: query dupe_rate_intersection. The denominator is the intersection-located subset, because the intersection key is undefined outside it; dividing by all 500,269 rows would divide by rows the method never examined.

Sensitivity to the two artifacts

Scope	Requests examined	Clusters	Duplicate rate	Redundant	Redundancy
as published, no exclusions	77,833	64,438	26.02%	13,395	17.21%
field conditions only (311-feedback types removed)	76,582	64,365	24.81%	12,217	15.95%
field conditions, automated Bathurst St submitter also removed	74,489	64,340	22.70%	10,149	13.62%

Units: requests, clusters, percent. Source: query sensitivity_field_only. Both exclusions make the headline smaller. Evidence for each is on the Concentration page.

Top service request types by duplicate rate

Volume floor of 500 intersection-located requests. Without it the list is won by types with four reports in the whole year. The floor is stated, not hidden.
Service request type	Total requests	Located	Clusters	In cluster of 2+	Duplicate rate	Largest cluster
Staff Service - Complaint - 311 Toronto	752	752	26	752	100.00%	58
Moving Motor Vehicle Noise	2,802	2,302	206	2,139	92.92%	278
Public Spaces Complaint	1,059	529	346	283	53.50%	15
Traffic Signal Equipment - Graffiti Complaint	818	585	409	277	47.35%	7
Traffic Signal Repair	5,585	5,455	3,800	2,514	46.09%	19
Sidewalk Snow Clearing Required	10,241	4,159	3,092	1,756	42.22%	14
Pick up Dead Wildlife	15,513	3,675	2,654	1,547	42.10%	41
Watermain-Possible Break	2,121	715	549	268	37.48%	10
Road Pothole / Road Damage	11,304	5,998	4,754	2,095	34.93%	10
Bus Stops Snow Clearing Required	849	613	492	204	33.28%	6

Units: requests, clusters, percent. Source: query top_types_by_dupe_rate. The first two rows are the two artifacts documented on the Concentration page and should be read as data-quality findings, not as street conditions.

Largest clusters

Service request type	Intersection	Ward	Reports	First	Last	Span, days
Moving Motor Vehicle Noise	BATHURST ST & CANYON AVE	York Centre	278	2025-01-01	2025-01-15	14
Moving Motor Vehicle Noise	BATHURST ST & CANYON AVE	York Centre	273	2025-06-08	2025-06-22	14
Moving Motor Vehicle Noise	BATHURST ST & CANYON AVE	York Centre	243	2025-04-26	2025-05-10	14
Moving Motor Vehicle Noise	BATHURST ST & CANYON AVE	York Centre	143	2025-01-15	2025-01-28	13
Moving Motor Vehicle Noise	BATHURST ST & CANYON AVE	York Centre	129	2025-10-13	2025-10-26	13

Units: reports, days. Source: query largest_cluster.

Duplicate detection accuracy

150 hand-constructed labelled pairs: 75 true duplicates and 75 hard negatives. Thresholds chosen by maximising F1 over this set; every value tried is recorded in out/eval_results.json under tuning_log, losers included.

1.000

precision

unit: fraction · TP / (TP+FP)

1.000

recall

unit: fraction · TP / (TP+FN)

1.000

unit: fraction · harmonic mean

Confusion matrix at the chosen thresholds (cosine distance < 0.65, link at combined ≥ 0.35).
	Predicted duplicate	Predicted distinct
Labelled duplicate	75	0
Labelled distinct	0	75

A perfect score here is a statement about the eval set, not about the system. Every one of the 75 negatives is rejected by a hard filter, not by the score.

By pair category. Categories (a) and (b) are rejected by the hard filters almost by construction and inflate the aggregate. Category (c) is the only genuinely hard one and is the number to read.
Category	Pairs	Accuracy	TP	TN
same intersection, different service type in a different division, days apart	25	1.000	0	25
same service type, same week, 2-12 km apart	25	1.000	0	25
same corner, days apart, deliberately overlapping everyday wording, different problems needing different crews	25	1.000	0	25
two members of one real 2025 cluster: same service type, same intersection, inside 14 days	75	1.000	75	0

Units: pairs, fractions. Source: scripts/eval.py → out/eval_results.json. Score formula: 0.60*(1-cos_dist) + 0.25*max(0,1-m/150) + 0.15*max(0,1-d/14). Thresholds: F1 saturates across 10 of 234 settings tried, so the argmax is arbitrary; among those achieving the maximum the strictest is taken. Every setting tried, losers included, is in tuning_log.

Ablation, what the embedding does on its own

If the classifier had put both members of a confusable pair in the same service type, which for deliberately blurred wording it easily might, the service-type filter stops firing and meaning is the only thing left. This removes that filter and asks what the embedding can do alone.

Measure	Value	Unit
Best achievable F1 without the service-type filter	0.8571	fraction
at precision	0.7500	fraction
at recall	1.0000	fraction
True duplicates, cosine distance, min / median / max	0.1771 / 0.4223 / 0.6173	cosine distance
Confusable negatives, cosine distance, min / median / max	0.356 / 0.4518 / 0.581	cosine distance
Confusable negatives falling inside the true-duplicate range	25 of 25	pairs

The distributions overlap completely. The embedding does not separate the hard negatives, the classifier does.

The vector index supplies recall: it is what matches “big hole in the road” to Road Pothole / Road Damage when the resident never uses the city's word. The taxonomy classifier supplies precision. That is the opposite of the usual case for vector deduplication, and it is what this data says.

Source: ablation_no_type_filter in out/eval_results.json, over the 75 true duplicates and the 25 confusable negatives.

So the classifier was measured on its own

It is the component carrying precision, so it is the component that has to be measured. Ground truth is the service type of the real 311 record each text was generated from.

Measure	Confusable pairs	Ordinary reports	Unit
Exact service type, out of 539 classes	0.760	0.600	fraction
Correct division, the crew actually dispatched	0.960	0.840	fraction
Truth present in the top-8 retrieval shortlist	0.840	0.750	fraction
Confusable pairs kept in different service types	25 of 25		pairs

That last row is what the perfect precision above rests on. When the classifier collapses both members of a pair into one type, the filter stops firing and nothing downstream can tell them apart.

Source: scripts/eval_classifier.py → out/eval_classifier.json. n = 50 confusable-pair texts, n = 100 ordinary reports.

Adversarial test, cases written outside the build

11 hard negatives and 5 recall pairs, supplied from outside and not tuned against. Each pair ran at its own real intersection with no seeded request within 2 km, so B could only ever match A.

10/11

hard negatives held

unit: pairs · source: adversarial_results.json

false positive, two problems merged

unit: pairs · the failure mode that matters

5/5

true duplicates merged

unit: pairs

10/10

of those held by the service-type filter, not by meaning

unit: pairs · consistent with the ablation above

Every pair. “Held by” records the mechanism that actually rejected the merge.
Case	Expected	A classified as	B classified as	Decision	Similarity	Metres	Days	Combined	Held by
1	no_merge	Sidewalk-Water Ponding	Icy Sidewalk Needs Salting	NEW	,	,	,	,	service-type filter
2	no_merge	Traffic Signal Repair	Park Lighting Maintenance	NEW	,	,	,	,	service-type filter
3	no_merge	Tree Emergency Clean-Up	Tree Emergency Clean-Up	DUPLICATE, FALSE POSITIVE	0.5879	0.0	2.13	0.7300	nothing, merged
4	no_merge	Clean up Illegal Dumping on City Road Allowance	Res / Organic&Garbage / Not Picked Up	NEW	,	,	,	,	service-type filter
5	no_merge	Road Pothole / Road Damage	Damaged Concrete Sidewalk	NEW	,	,	,	,	service-type filter
6	no_merge	Stationary Motor Vehicle Noise	Moving Motor Vehicle Noise	NEW	,	,	,	,	service-type filter
7	no_merge	Pick up Dead Wildlife	Res / Garbage / Not Picked Up	NEW	,	,	,	,	service-type filter
8	no_merge	Tree Emergency Clean-Up	Clean up Debris on Road	NEW	,	,	,	,	service-type filter
9	no_merge	Sidewalk Snow Clearing Required	Bus Stops Snow Clearing Required	NEW	,	,	,	,	service-type filter
10	no_merge	Traffic Signal Equipment - Graffiti Complaint	Traffic Signal Repair	NEW	,	,	,	,	service-type filter
11	no_merge	Construction Noise	Amplified or Musical Instrument Noise	NEW	,	,	,	,	service-type filter
12	ambiguous	Catch Basin - Blocked / Flooding	Road Water Ponding	NEW	,	,	,	,	service-type filter
13	merge	Road Pothole / Road Damage	Road Pothole / Road Damage	DUPLICATE	0.4672	0.0	1.13	0.6683	merged as intended
14	merge	Sidewalk Snow Clearing Required	Sidewalk Snow Clearing Required	DUPLICATE	0.4921	0.0	2.13	0.6725	merged as intended
15	merge	Pick up Dead Wildlife	Pick up Dead Wildlife	DUPLICATE	0.4568	0.0	1.13	0.6620	merged as intended
16	merge	Watermain-Possible Break	Watermain-Possible Break	DUPLICATE	0.5347	0.0	3.13	0.6874	merged as intended
17	merge	Traffic Signal Repair	Traffic Signal Repair	DUPLICATE	0.4297	0.0	2.13	0.6351	merged as intended

Units: cosine similarity, metres, days, combined score. Source: scripts/adversarial_test.py → out/adversarial_results.json. Thresholds unchanged from out/thresholds.json; the test never writes them.