feat(batches): show name of additional fields that cause errors

Long-term local retention of user telemetry as 'event logs' can now be disabled entirely via the telemetry: { disableLocalEventLogs } site configuration.

• Adds cost estimates to the ModelConfiguration returned by .api/llm/supported-models.json
• Adds llmTokenUsageCostEstimate telemetry value with estimates of each requests LLM cost in pennies

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Add the sourcegraph API client user-agent to telemetry payload types under source.server.server_client.user_agent.

• Propagates new X-Sourcegraph-API-Client-Name and X-Sourcegraph-API-Client-Version headers to request clients

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• New OpenAPI API docs within the instance at the URL $SRC_ENDPOINT/api/openapi/public. Alternatively, reach the page at "User > Settings > OpenAPI Reference".

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• The /.api/completions/stream API now includes used input/output tokens and the upstream model name in the response body when using the query parameter api-version=5 or higher. Does not apply to: Vertex Anthropic, Azure OpenAI (mo model name, and no token usage in streaming response), Gemini (no model name in streaming response)
• LLM chat completions now support stream: false when using Azure OpenAI and OpenAI-compatible providers.

• The raw HTTP API now accepts the query paramter format=cody or header Accept: application/cody to render files and directories as context items for Cody. Example request path: /github.com/sourcegraph/cody/-/raw/agent/src/index.ts?format=cody.

• add library that writes to TEST_INFRASTRUCTURE_FAILURE_FILE if it is defined for test failures

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Cody Gateway: add new claude-3-5-haiku-latest & claude-3-5-opus-latest models to Cody Gateway allow list Cody Gateway: add new claude-3-5-haiku-latest dotcom models list
Backport 11e7481ba3c810ae5b47d32b8cf32066e2f0b2bb from #1470

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

The sub repository permissions implementation now emits logs that describe the rules that were evaluated for a given request whenever tracing is enabled. (We limit these logs for only when tracing is enabled since they can be quite verbose).

• Added embeddable page for file snippets

Prevents default browser shortcuts from modifying the file content locally.

This fixes a bug where we would apply a phrase boost if a content: filter was specified with keyword search enabled. This led to inconsistent behavior (regexp vs keyword search) and it also rendered the content: filter ineffective, because we ran a general text search instead of a just a content search.

Fixes a bug where hovers were not triggered for less mainstream languages like F#, Standard ML, Visual Basic, Pkl, Hack, MATLAB etc.

Introduces new limits for auto-indexing inference to reduce the risk of continuously growing auto-indexing queues.

• The number of jobs spawned per round of auto-indexing inference per repo is capped (default: 100) to reduce risk of clogging of auto-indexing queues. Excess jobs will be discarded.
• The number of paths inspected for a single round of auto-indexing inference per repo is capped (default: 500) to reduce risk of timeouts. Excess paths will be discarded.

Fixes handling of SCIP uploads when the hash passed to the -commit flag of src-cli corresponds to the hash of an annotated tag instead of the hash of a commit. Previously, these uploads were not accessible for code navigation.

Fixes a bug where the retention tab for precise indexes would show a nil pointer exception instead of retention information.

fix(batches): enable request splitting by default

fix(batches): handle spec being nil for suffix salt

fix(batches): enable GQL request splitting via BATCH_CHANGES_REQUEST_SPLITTING

fix(batches): display "Deleted Namespace" instead of throwing an error

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

fix(code-monitor): fail monitors with missing/unverified primary email address

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Added token usage information to .api/llm/chat/completions

Fixes a bug where logs and traces were sometimes missing key-value pairs recorded alongside errors.

NA

The documentation for gitserver's ReadDir method has been clarified to reflect that the return iterator's Name() method returns the full path of the file, as opposed to just the basename.

The documentation for the "exclude" section in the github code host connection schema has been clarified to explain that each block is OR'd together, and the expressions within each block are AND'd together.

• When there are multiple auth providers configured that point to the same GitHub URL, as can be the case when using private GitHub Apps, user external account tokens will now refresh correctly.

• GitHub code host connections using repositoryQuery will no longer delete repositories if the sync fails for reasons like a GitHub outage or a token expiring.

• Reduce peak memory required for Zoekt indexing
• Improve search ranking by using repo freshness as a scoring signal

With this change we rank repository search results by similarity. Before, repository search results were ordered by star count instead. This affects suggestions and repository search results. To disable this feature and return to the previous behavior, set { "experimentalFeatures": { "disableOrderBySimilarity": true}} in global settings.

The GraphQL endpoint search.results.repositories now returns the list of repositories in the order of the matches we found. Before, repositories were sorted by id.

• TypeScript: Fixes references to object properties in various places
• Ruby: Fixes references to Opus:: in Ruby tests

feat(batches): verbose error logging for batch syncer with BATCH_CHANGES_SYNCER_VERBOSE_ERROR_LOGGING

feat(batches): add experimental feature to add a fork name suffix to avoid name collisions

• /.api/completions/stream endpoint now accepts "file" and "repo" parts alongside "text" and "image_url"

• /.api/completions/stream now supports vision using the OpenAI-compatible base64 encoding of images. Example "content": [{"type": "image_url", "image_url": { "url": "data:image/png;base64,{{IMAGE_BASE64}}" } }]. Requires the query parameter api-version=3 or higher.

• enable aspect workflows ci telemetry

• generate constraints.go using bazel for onconflict linter

• add cody-gateway environments for subcommand live

• implement using the release-registry for the banner

• Remove upgradetest from the release promotion ci

• Allow PRs to block a release from going out 😈

• BitBucket Cloud, BitBucket Server, GitLab, Gerrit, Gitolite, AWS CodeCommit Azure DevOps code host connections now support configuring SSH key authentication from the UI instead of mounting from disk.

• GitHub and generic Git code host connections now support configuring SSH key authentication from the UI instead of mounting from disk.

• add pgvector to server and postgresql-12 images

• build and add pgvector extension

• Introduce a new experimental embeddings index and context retriever.

We have updated the client-side query validation to allow combining rev: filters with query-based search contexts.

For very large repositories, search jobs could time out while fetching the repository. Now, search jobs can fetch and search over repositories, even if they take a long time to fetch.

We added a new database index to speed up a janitor job which is run as part of Search Jobs. Before, the janitor job might have significantly delayed migrations during an upgrade.

The new usagesForSymbol API correctly searches repos whose names contain spaces or other special characters.

Go cross-repo navigation for tagged versions should work better when using scip-go for auto-indexing. Navigation based on SCIP indexes generated using scip-go in CI or cron jobs should be unaffected.

• Fixed incorrect executors version reporting which indicated 0.0.0+dev even though it was the correct executor AMI version running under the hood.

• override internal/version.version with x_defs at build time in executor

fix(batches): disabled "select all" checkbox if all nodes are already disabled

• Fix an issue where duplicate telemetry can be exported when the worker service is scaled horizontally

modelsAPIEnabled is always false for dotcom

• The /.api/llm/chat/completions endpoint will now return an empty string "" for finish_reason instead of the string "unknown_please_report_bug()".

• dump more information out about the branch if it is out of sync

• remove embeddings in promotion script

• fix PGDATASOURCE format when using unix sockets

• Fixes a bug that can cause errors when rendering many mermaid diagrams on a page.

This fixes a bug where we added "readme" too often to the context.
Backport 28ff196a663f537c6cb6340f976a91431509a90e from #582

fix(batches): workaround for a bug in GitHub
Backport aad3a04f8c93561a61c404e69132e70a22d0acba from #576

n/a

• Add a container registry deny list to complement the allow list.
• Configure in site config via "batchChanges.containerRegistryDenylist".
• Mutually exclusive with "batchChanges.containerRegistryAllowlist".

• New POST /.api/cody/context REST endpoint to retrieve a list of relevant source locations given a natural language query.

When searching an unindexed commit we would consult indexed commits for speeding up results. If our index contained a commit that no longer existed in git we would error out due to a regression in v5.4.5099. This is now fixed.

fix(batches): avoid "Name already exists on this account" from creating fork by fetching the repo when the error happens

Fixes a regression in Cody context where questions containing the word 'or' could return noisy or no results.

• LLM API endpoints (/.api/llm) now return JSON-encoded HTTP bodies for non-200 status codes.

• LLM API /.api/chat/completions now returns OpenAI-compatible finish_reason.

• For compatibility with OpenAI clients, it's possible to use Bearer TOKEN header with all API endpoints that start with the prefix /.api/llm.

• prevent web-sveltekit commands from running if there are untracked files under src/routes

repo-updater now emits logs that log the result of every code host sync.

• the cancel funcs used by sg commands are now wrapped in sync.OnceFunc to prevent duplicate execution

• sg: fix panic when using wolfi update-hashes
• sg: add deprecation notice for wolfi update-hashes

• sg - fix panic in Cloud Ephemeral listing when listing instances that are not Ephemeral

• ensure deployment / instance names are clamped in all places for cloud ephemeral