It’s a bit of a weird shower thought but basically I was wondering hypothetical if it would be possible to take data from a social media site like Reddit and map the most commonly used words starting at 1 and use a separate application to translate it back and forth.

So if the word “because” was number 100 it would store the value with three characters instead of seven.

There could also be additions for suffixes so “gardening” could be 5000+1 or a word like “hoped” could be 2000-2 because the “e” is already present.

Would this result in any kind of space savings if you were using larger amounts of text like a book series?

  • SpaceNoodle@lemmy.world
    link
    fedilink
    arrow-up
    90
    arrow-down
    1
    ·
    1 year ago

    That’s kind of how file compression works - by replacing patterns with shorter symbolic representations!

    • RvTV95XBeo@sh.itjust.works
      link
      fedilink
      arrow-up
      17
      ·
      1 year ago

      If I remember correctly, you can even pre-build dictionaries for some compression algorithms to speed up the process if you’re working with compressing lots of similar data.