SWE-rebench-V2 contains 32,079 software-engineering tasks from 3,617 repositories across 20 programming languages. Each task comes from real development work, with a GitHub issue and the code changes made to resolve it.
I've been interested in a smaller problem within this dataset: localization.
issue + repository
|
v
predicted file paths
Given an issue and the source code, I want to predict which files need to change. I can then compare my predictions against the actual fix.
I started with 96 issues from 24 JavaScript and TypeScript repositories. I could evaluate 85 of them, spanning 22 repositories.
I like localization because it's a fun problem to experiment with. I don't need to run a large coding agent through an entire repair to test an idea. The models can be relatively small, the output is easy to score, and I can try quite a few approaches without spending much money.
There's another part I enjoy: optimizing queries over codebases often teaches me something about the codebase itself.
That's what led me to Attune.
I've also built an interactive research overview containing the measurements and some of the code behind them.
An obvious starting point is an embedding model.
Embeddings turn text into numerical vectors. I can embed the source files, embed the issue, compare the vectors, and rank files by similarity.
This is a fairly standard approach to retrieval, including code localization. It gives me a useful baseline before introducing anything more complicated.
I've been using Qwen3-Embedding-8B.
Eight billion parameters isn't exactly tiny, but it's relatively modest compared with the large reasoning models used for full coding-agent workflows.
I also don't need the model to generate a lengthy response for every file. It produces vectors, and the ranking is ordinary numerical computation.
On my initial 85-task panel, Qwen placed at least one required fixing file in its first ten results on 85.9% of the issues.
That's pretty good.
Finding every file is harder. On that measure, Qwen reached 44.7%.
A bug might require changing five files, but finding just one is enough to count toward the first number. For someone actually fixing the issue, the remaining files can matter quite a bit.
For comparison, here's how Qwen's ranking performed against the GPT-6.1-Sol repository-inspection condition on the same 85 eligible issues:
| Localization measure | Qwen | Sol |
|---|---|---|
| Required file ranked first | 41.2% (35/85) | 70.6% (60/85) |
| At least one required file in the first ten | 85.9% (73/85) | 76.5% (65/85) |
| Every required file in the first ten | 44.7% (38/85) | 48.2% (41/85) |
Qwen found at least one required file within ten more often, by 9.4 percentage points. Sol was better at putting a required file first, by 29.4 points, and recovering every required file within ten, by 3.5 points.
These are descriptive comparisons. Sol searched and read repositories under a different answer protocol; it didn't rank the same fixed candidate list as Qwen. Its failures remained misses. The figures don't establish that one model is generally better at localization. The counts and evaluation boundaries are retained in the localization report.
Still, this established something useful. I could get strong partial recovery from a much smaller model, while the larger model retained an advantage on first-file selection and complete recovery.
And it was very inexpensive.
Across 88 issues, I needed embeddings for 27,733 issue-and-file entries. Many of those entries contained the same text.
After identifying repeated inputs, there were 10,555 distinct texts. One empty input didn't require a model call, leaving 10,554 texts sent to Qwen.
The provider reported 10,644,719 input tokens, averaging about 1,009 tokens per embedded text.
The whole run cost $0.141.
For comparison, I also evaluated a condition where GPT-6.1-Sol searched and read repositories using its tools. Pricing its recorded usage at published API rates gave an estimate of approximately $4.65.
That's about 33× more model expenditure than the Qwen embedding run.
These were different approaches to the same localization problem. The Sol condition performed a complete repository-inspection turn, whereas Qwen supplied the initial file ranking. They also used different answer protocols, and neither cost includes the surrounding infrastructure.
But the difference is substantial.
The latency measurements were interesting too:
| Operation | Median | Count |
|---|---|---|
| Qwen embedding request | 2.43 seconds | 351 |
| Jev shortlist judgment | 0.37 seconds | 88 |
| Sol repository-inspection turn | 28.94 seconds | 88 |
The Qwen requests could contain several texts. The Jev timer starts after preparing a shortlist. The Sol timer includes tool interaction.
The Qwen request has a roughly 12× shorter median wait than the Sol inspection turn, although that isn't an end-to-end latency measurement for the full Qwen pipeline.
The measurements are retained in the interactive research overview.
The costs got me interested in another property of this problem.
A GitHub issue changes from task to task. The repository's source code doesn't necessarily change.
Several issues may refer to the same version of a repository, or to versions that share most of their files.
Once I've embedded a source file, I can use that vector for another issue, provided the file content, model, and embedding recipe still match.
repository
|
v
source embeddings
|
+----------+----------+
| | |
issue A issue B issue C
| | |
v v v
embedding embedding embedding
| | |
+----------+----------+
|
v
file rankings
Each new issue still needs its own embedding. But I don't necessarily need to embed the repository again.
This is why the 10.64 million tokens shouldn't be interpreted as the length of the issues themselves. That total includes the source-file embeddings shared across the 88 tasks.
In the retained experiment, the cold run used 351 embedding requests. An unchanged replay required zero new provider requests.
Loading and comparing the saved vectors still requires work, and the complete warm localization wall time wasn't measured. But the repeated model calls were avoided.
Attune makes these acquisition boundaries explicit using Flix effects.
The relevant idea is small:
// A calculation asks for an embedding.
//
// source or issue
// |
// v
// Embedding.vectorFor(request)
// |
// +----> saved vector
// |
// +----> provider request
// |
// v
// ranking
//
// The handler supplies the answer.
// The calculation doesn't need to know where it came from.
pub eff Embedding {
def vectorFor(request: Request): Result[String, List[Float64]]
}
An effect doesn't automatically provide caching. The handler still has to check the exact request, including the model and preparation recipe.
But separating the calculation from the work required to supply its inputs lets me experiment with different implementations without rewriting the ranking logic.
I wanted to apply a similar idea to repository structure.
The embedding model gives me a useful ranking. What else can I learn about the files it retrieves?
Consider three TypeScript files:
// receipt.ts
import { calculateTotal } from "./pricing";
// checkout.ts
import { calculateTotal } from "./pricing";
The relationship between them looks like this:
receipt.ts ----imports----> pricing.ts
^
|
imports
|
checkout.ts
Suppose the embedding model ranks receipt.ts highly.
A developer reading it might notice that it imports pricing.ts. They might follow that import, then look for other files that depend on the same module.
That's two operations:
receipt.ts
|
| imports
v
pricing.ts
|
| imported_by
v
receipt.ts, checkout.ts
This is ordinary graph traversal.
In Flix, I can express that sequence directly:
// receipt.ts --imports--> pricing.ts
// |
// imported_by
// |
// v
// receipt.ts, checkout.ts
Theory.follow(
facts,
Attune.files(world, List#{"receipt.ts"}),
List#{Atom.Imports, Atom.ImportedBy}
)
The result is a set of files containing receipt.ts and checkout.ts, assuming the relationships in the example have been extracted successfully.
The implementation of follow is small:
pub def follow(
facts: List[Fact],
start: Frontier,
route: List[Atom]
): Result[String, Frontier] =
match route {
// No operations left. Return the current answer,
// provided the facts belong to the same repository world.
case Nil =>
if (List.forAll(
fact -> fact#world == Attune.worldOf(start),
facts
))
Ok(start)
else
Err("a fact from another world cannot apply here.")
// Follow the next relationship, then continue.
//
// files --imports--> files --defines--> symbols
//
// Each step receives the result of the previous one.
case step :: rest =>
forM (
next <- Attune.apply(facts, start, step);
answer <- follow(facts, next, rest)
) yield answer
}
Attune currently has six directed relationships: defines, defined_in, imports, imported_by, calls, and callers.
Those operations have different input and output types.
file ----defines----> symbol
file ----imports----> file
symbol ----calls----> symbol
A file can't be used as the input to calls. Since routes are assembled dynamically, the evaluator checks their types as it executes them.
If an import can't be resolved, Attune doesn't invent a relationship to fill the gap.
This is useful when I start generating programs instead of writing every sequence by hand.
Sometimes I want to ask a question that isn't just a sequence of operations.
For example, which two files import the same dependency?
receipt.ts ----imports----> pricing.ts <----imports---- checkout.ts
In Flix, I can write:
// Find two DIFFERENT files that import the same dependency.
//
// first ----imports----> dependency
// second ----imports----> dependency
//
// Both import facts must hold.
pub def sharedRules():
#{ Imports(FileId, FileId),
Shared(FileId, FileId, FileId) } = #{
Shared(first, second, dependency) :-
Imports(first, dependency),
Imports(second, dependency),
if (first != second).
}
That's ordinary Datalog.
I can inject the repository's known imports into the relation, run the rule, and collect matching files.
The query doesn't tell me whether receipt.ts and checkout.ts both need changing. But it identifies a relationship I can test as part of localization.
Suppose Qwen gives me a file that seems relevant to an issue.
I'd like to find other files that might be involved. Following imports is one possible heuristic, but there are plenty of others.
Direct import:
anchor ------imports------> candidate
Two imports:
anchor --imports--> helper --imports--> candidate
Shared dependency:
anchor ----imports----> helper <----imports---- candidate
The anchor is a file Qwen already ranked highly. The candidate is another file I might want to move up the ranking.
Each diagram is a different hypothesis about what makes two files related.
A direct import might be useful. Perhaps following two imports is better. Maybe files that share a dependency tend to change together.
I don't know which relationships will help. Fortunately, localization gives me a cheap way to test them.
And rather than writing each query myself, I can write a program that generates the queries.
Attune represents a query as a list of relationships.
The actual representation in Language.flix is:
// One requirement:
//
// (relationship, source variable, target variable)
//
// Example:
//
// (imports, A, B)
// A ----imports----> B
//
// Every requirement in a program must hold.
pub type alias Atom = (Int32, Int32, Int32)
pub type alias Program = List[Atom]
The current implementation uses integers for the relationships and variables. With names in place of those numbers, its vocabulary looks like this:
// File and symbol relationships:
//
// file ----imports----> file
// file ----defines----> symbol
// symbol ----calls----> symbol
let imports = 0;
let defines = 1;
let calls = 2;
// Require two variables to be different:
//
// A != B
let different = 3;
// The roles we're trying to connect:
//
// anchor --------?--------> candidate
// |
// helper
//
// The anchor comes from Qwen's ranking.
// The candidate is a file we might promote.
// The helper is an optional intermediate file.
let anchor = 0;
let candidate = 1;
let helper = 2;
// Slots 3–6 represent symbols.
With that vocabulary, the three diagrams become:
// anchor ----imports----> candidate
let direct = List#{
(imports, anchor, candidate)
};
// anchor --imports--> helper --imports--> candidate
let chain = List#{
(imports, anchor, helper),
(imports, helper, candidate)
};
// anchor ----imports----> helper <----imports---- candidate
let shared = List#{
(imports, anchor, helper),
(imports, candidate, helper)
};
These are three different programs in the same small language.
The implementation checks that relationships connect the right kinds of things. An import can't accidentally connect a file to a symbol.
I also restrict the search to at most eight relationship atoms and six variables. That gives me a bounded space of programs to experiment with.
This is where my earlier fuzzing work starts becoming relevant.
In coverage-guided fuzzing, I take an input, mutate it, execute it, and see whether it exposes behavior I haven't observed before.
I can do something similar with these queries.
Start with a direct import:
anchor --imports--> candidate
Then try introducing an intermediate file:
anchor --imports--> helper --imports--> candidate
That changes the question from "Does the anchor import this file?" to "Can I reach this file by following two imports?"
Here's the mutation in Flix, with the numeric tags replaced by the names above:
// One mutation: replace an import with a two-step path.
//
// BEFORE:
//
// source --------imports--------> target
//
// AFTER:
//
// source --imports--> helper --imports--> target
//
// `p` is the current program.
// `vars` contains the variable slots it already uses.
let intermediate = List.filterMap(
match (kind, source, target) ->
// Only split imports.
// The helper slot must be free.
if (kind == imports and not Set.memberOf(helper, vars))
Some((
"intermediate",
// `remove` deletes the old direct edge.
// `:::` preserves the rest of the program.
List#{
(imports, source, helper),
(imports, helper, target)
} ::: remove((kind, source, target), p)
))
else None,
p
);
I haven't executed a query yet. I've constructed another program to try.
There are several other mutations:
// ADD:
// A ----imports----> B
// A ----imports----> B ----imports----> C
//
// REMOVE:
// A ----imports----> B ----imports----> C
// A ----imports----> B
//
// REVERSE:
// A ----imports----> B
// A <---imports----- B
//
// IDENTIFY:
// A ---> X A ---> Y
// X = Y
//
// SEPARATE:
// A ---> X B ---> X
// A ---> X B ---> Y
//
// INTERMEDIATE:
// A ----imports----> B
// A --imports--> C --imports--> B
//
// SECOND TARGET:
// symbol ----calls----> first
// \--calls----> second
let all =
additions :::
removals :::
reversals :::
ids :::
separation :::
intermediate :::
second;
Of course, not every mutation produces a useful or even valid program.
Some connect files to symbols incorrectly. Others contain unrelated relationships. Two programs might also differ only because I gave an internal variable another name.
So the generator checks its results:
// These two programs have the same structure:
//
// anchor --defines--> symbolA --calls--> symbolB
// anchor --defines--> symbolC --calls--> symbolD
//
// Renaming internal variables doesn't create a new hypothesis.
//
// Canonicalization removes these naming duplicates.
// It doesn't prove that different programs return different answers.
let values = List.filterMap(
match (operation, proposal) -> {
let normalized = canonical(proposal);
let id = key(normalized);
// VALID:
// file --defines--> symbol --calls--> symbol
//
// INVALID:
// file ----calls----> file
//
// Also reject disconnected or oversized programs.
if (valid(normalized) and id != key(p))
Some((
operation + ":" + id,
render(normalized),
normalized
))
else None
},
all
) |> List.toSet |> Set.toList;
valid checks the relationship types, number of variables, size of the query, and whether its relationships form a connected program.
Now I have a set of small, valid programs to test.
Datalog isn't an unusual choice for code analysis.
Graphenium uses it to check architectural changes proposed by AI coding agents. GitHub's CodeQL uses a Datalog-derived language for source analysis, and Doop uses Datalog for program analysis.
What I particularly like about Flix is that its Datalog constraints are first-class values.
In Kernel.flix, I turn the generated query into executable Datalog by combining compiled fragments.
Here's the central function, with names expanded and the evaluation illustrated:
pub def construct(
identity: String,
program: Language.Program
): Fragments.Rules = {
// Example input:
//
// anchor --imports--> helper --imports--> candidate
//
// Requirements:
//
// Imports(anchor, helper)
// Imports(helper, candidate)
//
// The planner chooses an evaluation order and tracks
// which variables each step has already bound.
let ordered = Language.execution(program);
List.foldLeft(
(rules, entry) -> {
let (step, (atom, boundMask)) = entry;
let (kind, source, target) = atom;
// Turn one relationship into a Datalog fragment.
//
// (imports, anchor, helper)
// |
// v
// require: anchor --imports--> helper
//
// Inequality uses a different fragment:
//
// (different, anchor, candidate)
// |
// v
// require: anchor != candidate
let next =
if (kind == different)
Fragments.different(
source, target, identity, step
)
else
Fragments.atom(
kind, source, target,
boundMask, identity, step
);
// Compose this requirement with the previous ones.
//
// finish
// <+> Imports(anchor, helper)
// <+> Imports(helper, candidate)
//
// Both import facts must hold.
rules <+> next
},
Fragments.finish(identity, List.length(ordered)),
List.mapWithIndex((i, v) -> (i, v), ordered)
)
}
The fragments are compiled Flix code. I'm selecting and combining them rather than generating arbitrary source code and compiling it at runtime.
After that, I can execute the resulting Datalog program against relationships extracted from a repository.
The experiment looks something like this:
direct import
|
v
mutate the query
|
+----------------+----------------+
| | |
v v v
reverse two imports shared dependency
| | |
+----------------+----------------+
|
v
check and run queries
|
v
compare what they found
|
v
choose what to try next
The difficult part is that last step.
In fuzzing, new execution coverage is often a useful signal. An input that reaches previously unexplored behavior might deserve more mutations.
I'd like to find an equivalent signal for structural queries.
Maybe a program deserves further exploration when it finds relationships that earlier programs missed.
But finding something new isn't the same as finding something useful.
A query could return hundreds of files sharing a common utility without getting any closer to the files needed for an issue.
I've already seen rules that improve individual localization results while making other results worse. The more recent searches haven't established a rule-selection method that consistently beats my strongest baseline.
So I'm interested in measuring two things separately: whether a query discovers different structural relationships, and whether those relationships improve localization.
I haven't established a reliable coverage-guided search heuristic yet. That's something I want to investigate.
What I like about the approach is that every experiment is inspectable. I can read the query, see the relationships it requires, execute it against another repository, and check what it found.
And the act of searching often teaches me something about how those repositories are organized.
That's what makes Flix interesting here. I can write the queries, the language describing the queries, and the program that searches through them in the same implementation.
OpenRouter advertises Qwen3-Embedding-8B at a starting price of $0.01 per million input tokens.
I've spent plenty of time getting models to run locally on my NixOS/Asahi laptop. I enjoy working through the hardware and software problems involved.
But prices this low made me reconsider the economics.
I live in Ohio, where residential electricity averaged 19.45 cents per kilowatt-hour in July 2026, according to the US Energy Information Administration.
Suppose a local machine consumes 500 watts while processing 1,000 embedding input tokens per second.
Processing the same 10.64 million tokens from my experiment would take just under three hours and cost approximately 28.8 cents in electricity alone.
That's about twice the 14.1 cents the provider actually charged.
Under those assumptions, the machine would need to sustain roughly 2,040 tokens per second just to match my recorded bill on electricity.
To match OpenRouter's advertised starting price of one cent per million tokens, it would need to sustain around 2,700 tokens per second.
Neither comparison includes the hardware.
These are hypothetical throughput and power figures, not measurements of my laptop. Better local hardware, cheaper electricity, or different provider prices could change the result. Privacy and independence matter too.
What interests me is why I'm even making this comparison.
I started with a problem that could have been approached using a large reasoning model. By narrowing it to localization and finding that a much smaller embedding model did useful work, I reduced the model spending enough that electricity became an interesting unit of comparison.
And with tens of thousands of potential issues, the scale makes those small costs worth thinking about.
This leads to another systems problem I've been following.
One company I find interesting is Modular, which develops Mojo and MAX.
The part of its work I find most compelling isn't a particular benchmark result. It's the problem of heterogeneous compute.
Different workloads make different demands on hardware.
Some operations are limited by arithmetic throughput. Others are limited by memory bandwidth. GPUs, CPUs, and other accelerators have different strengths, and the best hardware for one stage of a workload may not be the best hardware for another.
Even the prefill and decode stages of a language model can have quite different computational characteristics.
Modular's approach is to make high-performance execution portable across different hardware, with MAX providing the runtime and Mojo providing a way to express and specialize kernels.
Its MLSys 2026 discussion explores these questions in more detail.
I find that framing useful because the economics of AI aren't determined by the model alone.
An embedding request, a graph traversal, and a short model judgment are very different operations.
My localization experiments use Qwen to establish a ranking, Flix programs to investigate repository relationships, and sometimes Jev to judge a prepared set of candidates.
Those stages have different costs, different performance characteristics, and different opportunities for reuse.
Choosing how to perform each stage can matter as much as making any one operation faster.
In the original localization study, combining Qwen retrieval with structural context and a small Jev judgment stage increased top-ten hit rate from 85.9% to 91.8%.
Complete fixing-file recovery rose from 44.7% to 55.3%.
The recorded model expenditure for Qwen, structural context, and Jev together was approximately $0.165.
That's roughly 28× less than the $4.65 API-price estimate for the Sol inspection condition.
The procedures weren't identical, so this isn't a claim of 28× cheaper complete software engineering. It does demonstrate a much smaller model bill for this localization experiment.
And the structural results weren't uniformly positive.
One source-supported import rule moved a required Ember file from rank 36 to rank 3.
The same kind of intervention could make other repositories worse.
Later searches through thousands of small Datalog programs haven't established a generally useful rule-selection method that consistently beats the stronger Studio baseline.
That makes the next part of the problem particularly interesting to me.
I can now generate and execute many small programs. What I don't yet have is a reliable way to choose which ones are worth exploring.
The coverage-guided fuzzing analogy suggests one possibility, but a new structural relationship doesn't necessarily imply better localization.
I'd like to investigate that further, ideally with larger experiments and retained execution evidence showing where different programs help or fail.
A repository contains years of decisions made by the people who built it.
Some of those decisions are visible in imports, definitions, calls, and the way components depend on one another.
Recovering that information requires work, but another issue against the same repository may be able to use it again.
A source file might remain unchanged across many issues. An embedding can be retained. Structural facts can serve many different queries. Two queries might share an intermediate answer.
But those optimizations don't always pay off.
A repository might change too frequently. A query might be too cheap to bother caching. Or a particular structural relationship might not improve localization at all.
Those are things I want to measure rather than assume.
I've made Attune open source because I'd like the implementation and evidence to be inspectable. The code is still changing, and I don't consider the research finished.
If you're curious, explore the measurements. Try reading one of the Flix programs, or give it to ChatGPT or Claude and ask what it computes. See whether a simpler query produces the same result. Look at an experiment that didn't work.
I'd be interested in what other people find.
I started with localization because I wanted an inexpensive way to experiment with how agents navigate code.
The surprising part has been how much of the problem can be separated into relatively small operations, each of which can be measured independently.
Once the model is small enough, the cost of computation becomes a question about more than model size. It becomes a question about scheduling, hardware, electricity, and whether previously acquired information remains useful.
That's the economic question behind Attune.