We've advertised our 300,000-token context window. But can Supermaven actually use all those tokens to improve its suggestions?
In our new post, we evaluate Supermaven's long-context performance.
We've added a free tier to Supermaven!
It doesn't have our unique 300,000 token context window, but it still has a great model with unmatched latency. And it can see your current diff, so it understands what you're working on.
The Supermaven Jetbrains plugin has been submitted to Jetbrains for review and should be ready in a couple days.
We'd be grateful to anyone who wants to beta test the plugin - DM us if interested.
Supermaven 0.1.23 (VSC) makes a significant change to suggestions in the middle of the line. They will now display off to the right so your text doesn't move around as you're typing:
There is no longer a delay for indexing when using Supermaven with a new repository. Supermaven will now return completions immediately and run indexing in the background.
The Supermaven Jetbrains plugin is ready (in beta). We tried to DM everyone who messaged us about it but we got rate limited by Twitter, so you can access it here:
Supermaven for VS Code 0.1.31 supports enabling/disabling Supermaven for specific languages. For example, you can disable Supermaven in Markdown files.
We're excited to announce support for Jetbrains IDEs!
Thanks to all our beta testers who have helped work out issues with the plugin over the past weeks.
In addition to .gitignore, Supermaven now supports .supermavenignore to give you additional control over which files are indexed by Supermaven.
Syntax is the same as .gitignore.
Starting from version 0.1.33, Supermaven is disabled by default in files ignored by .gitignore, and these files will not be sent to our servers.
You can re-enable Supermaven in these files by clicking the status icon:
We added a Code Policy to our website to give customers additional clarity on the protection of their proprietary data. We do not train models on customer code.
Some users on Windows reported that the Supermaven extension would keep a blank terminal window open. We reproduced the issue and it is fixed in extension version 0.1.13.
A long context window doesn't just help Supermaven look up function definitions - it also lets us learn your personal style and write code the way you would write it.
@SupermavenAI
is the only AI tool I've paid for. I never got much value out of GH Copilot. It's just too slow. I am able to type out what's in my head faster than what Copilot could suggest. This is not so with Supermaven. Not only is Supermaven faster, but it writes like me.
We made changes to Supermaven's handling of large diffs that should significantly improve quality and latency when making many changes or adding large files.
We changed Supermaven to be less conservative in suppressing suggestions in the middle of a line. To get these changes, update to extension version 0.1.17.
There was an issue with incorrect handling of carriage returns that caused Supermaven to fail to return completions on Windows in some cases.
This issue is fixed. It is a server-side change, so no update is required.
There was an issue where Supermaven would consume a large amount of system memory when the repository contained large files. This is fixed in extension version 0.1.14.
Thanks to
@PhilipKung5
for reporting the issue.
A common evaluation for a long-context model is the "needle in the haystack", where a piece of text (the "needle") is hidden within a large context (the "haystack"). The model is tested on whether it can retrieve the needle after consuming the entire context.
In this task, the model is given a sequence of key-value pairs. Each pair occurs twice. The model is scored on whether it can use the first occurrence of the pair to correctly guess the value when it sees the pair a second time.
But the needle in the haystack task is made easier because the needle stands out from the rest of the text: to get it correct, the model only needs to remember the needle.
To give a harder test, we made a "dense retrieval" benchmark.
@ITLSR
@cursor_ai
We are planning to add a chat feature in the future.
Note Cursor and Supermaven are fully compatible since Cursor is a VSCode fork. You'll just want to disable Copilot++ so they don't conflict
Lastly, we plot prediction error as a function of token index on our internal code, averaging over several hundred orderings of the files to get a smooth result.
Here, we plot the dense retrieval performance as a function of the number of tokens separating the first and second occurrence of a pair.
Many groups have found models are best at remembering the start and end of the context, and we reproduce this finding.
@jullerino
We haven’t changed anything that would affect this in the past few days. Can you give some more context about this example? What was the suggestion that was accepted here and how did it mess up the brackets?
@DeiAlexTVT
Sometimes Supermaven will not suggest. Are you saying that even when you go to a new place in the file and start typing, you get no suggestions?