• running_ragged@lemmy.world
    link
    fedilink
    English
    arrow-up
    44
    arrow-down
    6
    ·
    2 months ago

    Google (used to) scrapes the specific details authorized by robots.txt and uses it to make your content visible.

    OpenAI scrapes everything it can technically see, ignoring robots.txt and feeds i to a black box and regurgitates it claiming it’s something new, that it deserves to be paid for.

    Quite different actually.

    • masterspace@lemmy.ca
      link
      fedilink
      English
      arrow-up
      8
      arrow-down
      24
      ·
      2 months ago

      So if OpenAI complies with Robots.txt files then there’s no issue right?

      Because then they’re identical. Open AI spent a bunch of money building a powerful system they feed those results to, as did Google.

        • Zarxrax@lemmy.world
          link
          fedilink
          English
          arrow-up
          9
          arrow-down
          1
          ·
          2 months ago

          Actually Google tries their hardest NOT to point you to content. They scrape the data from sites and display it directly in the search results so that you don’t need to visit any site except Google. Their new AI answers that they are pushing on users are just another step in that direction.

          • Z3k3@lemmy.world
            link
            fedilink
            English
            arrow-up
            10
            arrow-down
            1
            ·
            2 months ago

            Which is why Google is no longer my default browser. I’d be quite happy if it reverted Back to don’t be evil or just ceased ro exist

        • masterspace@lemmy.ca
          link
          fedilink
          English
          arrow-up
          5
          arrow-down
          19
          ·
          2 months ago

          Literally every page Google shows you, where it also shows you those ads it makes money from, is Google’s content and it is derived from the data it gets scraping the web.

          • grue@lemmy.world
            link
            fedilink
            English
            arrow-up
            16
            arrow-down
            1
            ·
            2 months ago

            What the fuck are you even talking about? Making a list of website identifiers (names and URLs) so that people can go to them isn’t even slightly the same as making a derived work of the websites’ contents.

                • masterspace@lemmy.ca
                  link
                  fedilink
                  English
                  arrow-up
                  3
                  arrow-down
                  6
                  ·
                  2 months ago

                  Google has to scrape and process the entire webpage and analyze the content to figure out which links to provide you. The end result of them presenting you with those links is from them copying and processing all your site’s data, including when you aren’t listed in the search results.