Mining file paths from free-form text

Failure notes and agent memories often mention files in passing (src/foo.go, app/handlers/…). Correlating memories by shared file refs starts with extracting those strings from unstructured text.

Credit: extractFileRefs in Agent-Observer (internal/patterns/correlator.go). Package-level fileRefPatterns is a []*regexp.Regexp of path-like matchers (not shown here) — this post focuses on the extraction loop only.

The function

func extractFileRefs(text string) []string {
	seen := map[string]bool{}
	var refs []string
	for _, pattern := range fileRefPatterns {
		for _, match := range pattern.FindAllString(text, -1) {
			if len(match) > 5 && !seen[match] && len(match) < 200 {
				seen[match] = true
				refs = append(refs, match)
			}
		}
	}
	if len(refs) > 10 {
		refs = refs[:10]
	}
	return refs
}

What it does

1. Runs each compiled path-ish pattern over text.
2. Keeps matches that are longer than 5 and shorter than 200 characters.
3. Dedupes with a seen map (first occurrence wins).
4. Truncates the result to at most 10 refs.

Those refs later drive search for related memories that mention the same paths.

Why this shape

• Regex over free text beats brittle NLP for “looks like a path.”
• Length floors drop junk like a.go-style noise; a 200-char ceiling avoids swallowing paragraphs.
• A hard cap of 10 bounds downstream search fan-out — correlation stays cheap.

Gotcha

Order is pattern order, then left-to-right match order, not importance. After the tenth unique match, later (possibly better) paths are dropped. Patterns also false-positive on version strings, URLs fragments, or word.ext that are not filesystem paths — treat refs as soft hints for search, not as validated file identities.

Takeaway

Mining structure from prose is mostly gates: dedupe, length, and cardinality. The clever part is knowing what not to feed the correlator.