One of my biggest UX gripes with GitHub Copilot (in VS) is that you have to wait for the whole response to be generated before you can read it due to the auto-scrolling as the response streams in. Would be a big plus to be able to stop the auto-scroll!

Anyone else having issues with Azure OpenAI (timing out trying to connect via API, failing to load model deployments in Azure OpenAI Studio...)?

Anyone else having issues getting into the Azure OpenAI Studio today?

Good article (Azure OpenAI) on considerations for right-sizing your Provisioned Throughput deployment, spilling peak workloads over to PAYG for cost optimisation: Right-size your PTU deployment and save big ( https://t.co/p6bU4lZlDr)

Important to make a conscious assessment, else you run the risk of hitting production with a workload that the LLM deployment model/provider cannot fulfil reliably. Depending on use case/customers, you may then be forced into a hasty switch to Provisioned Throughput

PAYG models are great and have low outlay, but usually don't have guarantees and usually higher latency. Provisioned Throughput models give consistency and lower latency but a huge jump up in committed spend, where you end up paying for a lot of quiet time in a non 24/7 scenario

Transitioning to Production with LLM capabilities, when you care about consistency of guaranteed throughput and latency is a key challenge that needs to be considered carefully up front.

Full Fibre 900 - £41.99/month Full Fibre 900 for a customer of many years (me) - £55.54/month Don't know what's going on with @Plusnet these days, but that's shocking. Making me wonder if longer term, it'll be wound down (as part of consolidating Plusnet/BT/EE, per mobile)

Any of my followers (or beyond) have experience with Amazon Bedrock for hosting LLMs (on a token usage basis and/or Provisioned Throughput basis)? Keen to hear thoughts, experiences etc. Please RT!

Good to see a new Learning Path for Semantic Kernel on MS Learn: https://t.co/ifLPI3l9sT

Early days, but early signs looking promising - around 50% reduction in duration, increased reliability/stability due to stable plan reuse. Even looking like there's potential to use a different (cheaper and faster) LLM when using cached plans.

To save generating plans afresh every time for similar inputs, I'm building a prototype Semantic Kernel "Plan Cache". Still needs LLM (+ Vector Search), but with a much smaller input context, and a more focused task.

Switched over recently to the new gen Planners (Handlebars) as part of Semantic Kernel v1.x upgrade. Next step is to optimise the "plan generation" phase of a workflow. Generating plans is expensive (time and cost) and relatively prone to error.

But the trend is pretty consistent on Azure OpenAI with 1106-Preview. Running tests on isolated model deployments, using low token usage test case (<500 prompt tokens, < 300 completion tokens), various points throughout the day, different machines/code implementations etc etc

Having significant latency issues with GPT 4-Turbo (1106-Preview) on Azure OpenAI, making it consistently, noticeably slower than GPT-4 (0613). Anyone else encountering this? e.g. 48.8s (GPT-4-Turbo) vs. 13.9s (GPT-4) Snapshot times, so fluctuates, but wildly so for 4-Turbo.

Currently watching:

Microsoft Graph activity logs ( https://t.co/KjggluKFcr) is long overdue. Finally able to get a tenant-wide view on usage - dusting off my KQL skills!

Finishing up prep for an internal talk I'll be giving tomorrow on the Semantic Kernel. There are demos, and then there are Generative-AI demos - let's hope it all behaves!

Enjoyed the keynote, as you'd expect!

Checked in, ready for MS Build AI day to start