NVIDIA just released a free new chatbot that runs locally on Windows PCs. You need a GeForce RTX 30 Series GPU or higher and at least 8GB of VRAM to use the bot.

• NVIDIA released a demo version of a chatbot that runs locally on your PC, giving it access to your files and documents.

• The chatbot, called Chat with RTX, can answer queries and create summaries based on personal data fed into it.

• It supports various file formats and can integrate YouTube videos for contextual queries, making it useful for data research and analysis.

  • GenderNeutralBro@lemmy.sdf.orgEnglish
    1·
    9 months ago

    Pretty much every LLM you can download already has CUDA support via PyTorch.

    However, some of the easier to use frontends don’t use GPU acceleration because it’s a bit of a pain to configure across a wide range of hardware models and driver versions. IIRC GPT4All does not use GPU acceleration yet (might need outdated; I haven’t checked in a while).

    If this makes local LLMs more accessible to people who are not familiar with setting up a CUDA development environment or Python venvs, that’s great news.