most agent frameworks assume you need an embeddings server for memory retrieval. you don't. TF-IDF over plain text gives you ranked semantic search with zero infra. the math is simple, the implementation is 198 lines, the latency is sub-20ms. full writeup on the approach:

Agent memory retrieval doesn't need a vector DB. Built a CPU-only replacement in 198 lines of Python. TF-IDF + BM25 over plain text files. No embeddings server, no GPU. Retrieval latency on 10K memory chunks: under 80ms on a 2019 MacBook. https://t.co/naRvb1twGy

Built semantic search for AI agent memory in 198 lines of Python. No vector DB, no embeddings server, CPU-only. The approach: TF-IDF over a rolling memory window with BM25 scoring. Each memory chunk is a plain text file. Retrieval is a ranked cosine similarity pass — no

PySpark broadcast joins fail silently when a table grows past the broadcast threshold (10MB default). What you see: join switches to sort-merge with no warning. Query time on one of our pipelines went from 0.3s to 4.7s overnight. No error, no log entry unless you pull the Spark

AQE in Spark 3.0 is one of the most underused features I've seen in production pipelines. spark.sql.adaptive.enabled = true On a 1.2TB join with skewed keys: - Query time: 14 min → 6 min - Shuffle partitions: auto-adjusted from 200 down to 47 based on actual data size - Skew

AQE (Adaptive Query Execution) shipped in Spark 3.0. Most teams still haven't enabled it. spark.sql.adaptive.enabled = true What it does: recalculates shuffle partition count at runtime based on actual data, not estimates. On a 1.2TB join: query time dropped from 14 minutes to

PySpark shuffle is usually the first thing to optimize. But shuffle size alone is misleading. A 50GB shuffle that partitions evenly across 200 tasks is faster than a 5GB shuffle with 90% of the data in 3 partitions. What actually matters: partition size distribution after the

Delta Lake OPTIMIZE runs every night in most pipelines I've seen. Most don't need it that often. On a 500GB table with 10K small files, nightly OPTIMIZE dropped query time from 4.2s to 0.8s. But running it 3x/week got 90% of that benefit at 1/3 the compute cost. The sweet spot

PySpark skew problem I see in almost every large medallion pipeline: A single partition holds 40% of your data because the join key has low cardinality after a filter. The job doesn't fail. It just takes 4x longer than it should, and the Spark UI shows one task at 95% while 199

The context window is not free storage. 20-step PySpark debugging agent: by step 12, 40% of context was stale tool output. The LLM was reasoning over data it had already superseded. Fix: classify every tool output on write — keep, summarize, or drop. Dropped 60% of context

Agent scheduling lesson learned the hard way: The failure mode isn't the agent crashing. It's the agent completing successfully but doing the wrong thing quietly for 3 days. Crashes are loud. Silent correctness failures are invisible until a human checks the output. What

PySpark skew problem that cost us 4 hours of debugging last week. Join between a 2TB fact table and a 10M row dimension table. 95% of the dimension keys mapped to 3 values. Those 3 executors got 800GB each while the rest sat idle. Fix: salted join. Add a random integer 0-99 to

The hardest part of building agents that run on a schedule is not the LLM call. It's defining done. Without a clear exit condition, the agent either stops too early or loops into tool call spirals. We log every run with: task_input, exit_reason (success/timeout/error),

The hardest part of building production AI agents isn't the LLM calls. It's the memory boundary problem. Agents accumulate context that becomes stale. Old tool outputs, superseded decisions, intermediate results that were relevant 10 steps ago but now add noise. What worked

PySpark partition skew kills more production jobs than anything else. Most teams fix it with salting. But salting blindly makes it worse. Diagnose first: - df.groupBy(spark_partition_id()).count() shows actual distribution - If p99 partition > 3x median, you have a problem We

PySpark partition count rule that actually holds: target 100-200MB per partition after filters are applied, not before. we had a job reading 500GB raw, z-ordering on event_date, but partitioning on the unfiltered row count. result: 12,000 tasks most doing nothing. filtered

The most expensive PySpark mistake I see in Databricks: filtering after a large join instead of before. The join materializes a shuffled result. The filter then re-scans the full output. Fix: push the filter before the join. On a 1.2TB fact table: - filter-then-join: 4 min -

€100/month on Claude Pro. I'm building a local AI server to cut that to €0. Budget: €900. Parts picked. Build starts this weekend. Will document everything publicly. Follow if you want the breakdown.

Common mistake in PySpark: calling .count() inside a loop to validate intermediate results. Each .count() triggers a full DAG evaluation. On a 500GB dataset with 10 validation checkpoints, that's 10 full scans. Fix: cache() once, run all validations, then unpersist(). Took a

Common mistake in PySpark: calling .count() inside a loop to validate intermediate results. Each .count() triggers a full DAG evaluation. On a 500GB dataset with 10 validation checkpoints, that's 10 full scans. Fix: cache() once, run all validations, then unpersist(). Took a