00:00:06 <fizzie> (And some crawler has rediscovered the HackEso hgweb repo -- https://zem.fi/tmp/hack.png -- so I may turn that one back off at some point. Though right now it's chugging along okay. Probably because it's fetching files instead of diffs.)
00:00:47 <aadenboy> sounds like something an ant would do
00:02:07 <HackEso> thank? No such file or directory
00:04:53 <fizzie> The thing is, you can browse the entire file tree at any revision, and there's around 12557 revisions and 24491 current files, so assuming the number of files has grown linearly (probably not true) that amounts to some 330 million URLs. Most of which are effectively the same file duplicated thousands of times, but that's not obvious. So they won't be done any time soon.
00:10:16 <esolangs> [[AWA5.0]] https://esolangs.org/w/index.php?diff=182089&oldid=118355 * Squidmanescape * (+416)
00:31:24 <int-e> fizzie: I see "only" 5405 current files (in the repo).
00:31:50 -!- ais523 has quit (Quit: quit).
00:41:58 <fizzie> Oh, I accidentally also counted .hg, even though I definitely thought of excluding it.
00:42:51 <int-e> fizzie: sampling every 100-th changeset:https://int-e.eu/~bf3/tmp/repo-size.png
00:43:41 <int-e> > 3500*12500 -- still a lot
00:44:07 <int-e> (I'm lowballing the average)
00:52:11 -!- Acid190 has quit (Quit: Leaving).
01:05:41 <int-e> fizzie: hmm, reload for full data (the peak above 10k was somebody trying to build perl, I guess. (rev. a16537d2fe07)
01:17:08 -!- salpynx has quit (Quit: Leaving).
01:35:09 <int-e> (hmm, the numbers are *slightly* off (4 files in the end) because I'm adding file added/removed numbers and for branch merges I'm not sure what those mean)
01:39:03 -!- amby has quit (Quit: so long suckers! i rev up my motorcylce and create a huge cloud of smoke. when the cloud dissipates im lying completely dead on the pavement).
02:31:03 <esolangs> [[User:CodePentuplets48]] https://esolangs.org/w/index.php?diff=182090&oldid=181975 * CodePentuplets48 * (+14)
02:34:29 <esolangs> [[Language list]] https://esolangs.org/w/index.php?diff=182091&oldid=182055 * CodePentuplets48 * (+14) /* X */
02:37:17 <esolangs> [[User:Tommyaweosme]] https://esolangs.org/w/index.php?diff=182092&oldid=182006 * Tommyaweosme * (+0)
03:18:35 -!- aadenboy has quit (Quit: goodbye for now! back another day).
03:31:01 -!- chloetax1 has joined.
03:31:53 -!- chloetax has quit (Read error: Connection reset by peer).
03:31:53 -!- chloetax1 has changed nick to chloetax.
06:30:37 <esolangs> [[Esolang:Introduce yourself]] https://esolangs.org/w/index.php?diff=182093&oldid=182033 * The-coder256 * (+228)
07:08:11 -!- Sgeo has quit (Read error: Connection reset by peer).
07:40:48 -!- b_jonas has quit (Quit: leaving).
07:49:57 -!- BadGuyAlex has joined.
08:03:48 <esolangs> [[]] N https://esolangs.org/w/index.php?oldid=182094 * The-coder256 * (+918) Created page with "'''''' is an [[esoteric programming language]] made by [[User:The-coder256]]. It uses emojis for instructions. ==Instructions== uses an accumulator A and a stack for memory. Here are all of the instructions: {| class="wikitable" !Emoji !Meaning |- | | Increments
08:06:38 <BadGuyAlex> "Unfortunately we have had to restrict certain expensive pages" blah blah, yeah...
08:09:27 <esolangs> [[Special:Log/newusers]] create * GladICanAtLeastMakeAnAccount * New user account
08:10:16 <esolangs> [[Joke language list]] https://esolangs.org/w/index.php?diff=182095&oldid=181393 * The-coder256 * (+19)
08:11:00 <FireFly> BadGuyAlex: alas, the scraping spiders come for us all (which is to say, anyone serving files to the public internet)
08:17:25 <esolangs> [[]] M https://esolangs.org/w/index.php?diff=182096&oldid=182094 * The-coder256 * (+1)
09:04:36 <fizzie> Hmm. I could try setting up some sort of alternative path that goes through something CAPTCHA-like (I feel like the problematic crawlers generally even try to bypass those, they just go for quantity over quality) to set up a cookie that would then be accepted like the wiki login one, for people who aren't interested in making an account.
09:08:05 <BadGuyAlex> though what kind of CAPTCHA. Current AI models are pretty smart (though often not when you actually need them).
09:18:33 -!- salpynx has joined.
09:19:25 <salpynx> is part of the problem that esolangs bot sends diff URLs to IRC, and #esolangs IRC logs are online and archived?
09:19:49 <salpynx> any bot that looks to archive or process urls from IRC will have a lot to digest
09:24:21 <fizzie> I haven't looked closely, but I don't think the URLs have been collected from the logs. You do get to all the diffs (well, at least all diffs of consecutive revisions) by just following links on the site.
09:25:31 <fizzie> And anecdotally Anubis seems to be helping other sites stay online, and that's not even a CAPTCHA.
09:29:06 <salpynx> right, Special:RecentChanges contains all the diffs too, as do the logs at https://logs.esolangs.org/libera-esolangs , if they can find one they probably found the other
09:30:06 <salpynx> IRC logs are not the smoking gun I thought they were :)
09:34:17 <fizzie> Fun fact, logs.esolangs.org itself gets hardly any traffic, and the little there is is mostly from "honest" crawlers (as in, ones that set the user agent), like ClaudeBot, AmazonBot. Dunno what that says about how the dishonest crawlers operate.
09:42:38 <BadGuyAlex> π€ User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:150.0) Gecko/20100101 Firefox/150.0 πΏπΏπΏπΏ
10:35:38 <esolangs> [[]] https://esolangs.org/w/index.php?diff=182097&oldid=182096 * The-coder256 * (+109)
10:37:52 <BadGuyAlex> I'm thinking of: a program (and data) represented as arbitrary size integer N (BigInt). An interpreter does `operation = N % opcount`, executes an operation on N and updates N. Operations have to be chosen in a way to satisfy Turing-completeness.
10:38:13 -!- somefan has joined.
10:42:53 <esolangs> [[Special:Log/newusers]] create * Windows95 * New user account
10:51:52 <salpynx> that's like a single register Minsky machine, with prime encoded 'virtual' registers, but the code is stored as an int in the lowest (for ease of access) reg ...
10:53:37 <salpynx> I think I can see how that might work. I was working on a lang that stored the instruction pointer in a prime encoded register, this would just store the program data in one too
10:57:49 <salpynx> it's a single register Von Neumann Minsky machine
11:08:06 <salpynx> I like this idea, I've got a few Minsky machine experiments in progress that might be extendable to storing code in a encoded register. I thought https://esolangs.org/wiki/Ring-around-the-Rosie was novel for shifting instruction pointer storage into the one register. Moving the program there is a next step.
12:05:48 -!- amby has joined.
12:10:01 <esolangs> [[User:Tommyaweosme]] https://esolangs.org/w/index.php?diff=182098&oldid=182092 * Tommyaweosme * (+206)
12:33:05 <esolangs> [[User:Tommyaweosme/sandbox/history of every rule in Esolang:Policy]] N https://esolangs.org/w/index.php?oldid=182099 * Tommyaweosme * (+3278) Created page with "this article is related to esolangs in a way that it is related to the esolang wiki itself, and it is WIP. == All content must be public domain or equivalent. == this rule's original form was created on dece
13:05:20 -!- salpynx has quit (Quit: Leaving).
13:43:30 <esolangs> [[Talk:Important lesbian virtual machine]] https://esolangs.org/w/index.php?diff=182100&oldid=182041 * Miui * (+85) /* import as from lang */
13:43:44 <esolangs> [[Talk:Important lesbian virtual machine]] M https://esolangs.org/w/index.php?diff=182101&oldid=182100 * Miui * (+70) /* import as from lang */
13:50:38 <esolangs> [[Usernames]] https://esolangs.org/w/index.php?diff=182102&oldid=181947 * PrySigneToFry * (+173)
13:55:35 -!- impomatic has joined.
14:08:00 -!- b_jonas has joined.
14:14:28 <esolangs> [[Talk:Important lesbian virtual machine]] https://esolangs.org/w/index.php?diff=182103&oldid=182101 * Miui * (+69)
15:01:54 <esolangs> [[User:Tommyaweosme/sandbox/history of every rule in Esolang:Policy]] https://esolangs.org/w/index.php?diff=182104&oldid=182099 * Aadenboy * (+216) blockquotes
15:10:00 <esolangs> [[Arbitrary memory emulation]] https://esolangs.org/w/index.php?diff=182105&oldid=182086 * Aadenboy * (-78) optimize slightly
15:15:59 -!- impomatic has quit (Quit: Client closed).
15:24:10 -!- impomatic has joined.
15:30:51 -!- BadGuyAlex has quit (Quit: BadGuyAlex).
16:11:57 <esolangs> [[User:ChuckEsoteric08]] https://esolangs.org/w/index.php?diff=182106&oldid=169657 * ChuckEsoteric08 * (-110)
16:39:37 -!- Lord_of_Life has quit (Excess Flood).
16:40:04 -!- Lord_of_Life has joined.
16:44:14 -!- Lord_of_Life_ has joined.
16:44:24 -!- Lord_of_Life has quit (Ping timeout: 246 seconds).
16:46:04 <esolangs> [[Arbitrary memory emulation]] https://esolangs.org/w/index.php?diff=182107&oldid=182105 * Aadenboy * (+6) fix
16:47:04 -!- Lord_of_Life_ has changed nick to Lord_of_Life.
16:50:07 <esolangs> [[Arbitrary memory emulation]] M https://esolangs.org/w/index.php?diff=182108&oldid=182107 * Aadenboy * (-8) link change
16:50:23 <esolangs> [[Arbitrary memory emulation]] M https://esolangs.org/w/index.php?diff=182109&oldid=182108 * Aadenboy * (+0) sigh
17:11:19 <esolangs> [[User:Hammy/Box full of sand/nothing (DRAFT)]] N https://esolangs.org/w/index.php?oldid=182110 * Hammy * (+1437) Created page with "<table style="border:1px solid darkred;background-color:red;max-width:256px;padding:5px;color:white"> <tr> <td> <big><big><big>'''DRAFT'''</big></big></big> </td> </tr> <tr> <td> This esolang is currently a draft. Please do not edit it
17:40:14 <korvo> fizzie: Anubis does not actually help much. It used to help when it was new, but it only took a few weeks for serious scrapers to rig up a way to precompute challenges.
17:49:41 <fizzie> Mhm. I was basing that on the fact that I still see it around on places like, say, the Debian bug tracker, and (unlike pre-Anubis) those sites still seem to be usable. But I've certainly not researched this.
17:49:48 <fizzie> I guess it might be a victim of its popularity: if a critical mass of sites adopted it, that also means it became worthwhile to defeat it.
17:50:04 <fizzie> I don't think anyone's going to spend any human effort for scraping our wiki specifically, and while I'm sure a generic LLM can work out the sort of CAPTCHA I might add -- like the one we have on new user registrations -- I'm not entirely convinced they've really built up a feedback/orchestration loop to make it do that at scale.
17:52:40 <fizzie> I mean, if they were inclined to spend actual human effort they could just manually set the login cookie to any non-empty value, it's not like I'm validating it or anything. (Though I guess it's possible a set-but-invalid cookie will make MediaWiki serve an error or a cookie-reset redirect or something instead of the page contents, in which case I'd accidentally closed that loophole.)
17:57:07 <korvo> https://github.com/sleeyax/anubis-solver from over a year ago. A pile of Go that solves Anubis challenges using Go on the CPU. It's somewhat faster than the browser's cryptography API.
17:57:23 <korvo> I struggle to imagine that no lab is solving Anubis challenges with GPU cryptography.
18:42:28 <int-e> korvo: it could be that if you put that much thought into crawling you'll also consider rate limits
18:43:31 -!- kata has joined.
18:47:35 <FireFly> I think Anubis challenges have changed in the past year as well
18:47:48 <FireFly> (but of course it'll always be a cat-and-mouse game, heh)
18:47:55 <int-e> The super aggressive botnet scraping (that Anubis apparently still works against) doesn't make much sense to me unless there's some market where you can sell, I don't know, 1TB of scraped data for $10.
18:50:24 <korvo> There are paid versions of Common Crawl, yes. The demise of Books3 in USA courts came with the understanding that if somebody wanted to build a *private* version of Books3, with the rule that it was backed 100% by real books on real shelves in a real storage facility, then they could sell access to it for training.
18:50:41 <korvo> That's *Bartz v. Anthropic* and *Kadrey v. Meta* for folks who want to go do reading.
18:51:08 <korvo> Oh, also, such private libraries wouldn't have to disclose their existence, terms of use, or fee schedule to authors.
19:38:00 -!- ais523 has joined.
19:39:04 <ais523> re: blocking bots, I think a lot of progress on this can be made by exploiting the way they do connections
19:39:33 <ais523> I can imagine captchas not to tell humans and computers apart, but to tell common scraperbot frameworks apart from other users
19:40:37 <ais523> for example, an interstitial page that's just a button that sends a POST request (and possibly JavaScript to automatically click on it), that sets a short-lived session cookie (thus not violating GDPR rules) that contains the IP address at the time the page was visited
19:41:15 <ais523> most of the bots don't send POST requests and rotate through IPs like crazy, so would find this extremely hard to get past, even though it's kind-of trivial for any human
19:43:35 <ais523> (except me, because I browse with cookies off by default)
20:05:32 <zzo38> There is those issues. I am not quite sure what to do, although checking the headers for claiming to be a browser that does not support cookies, might be another way to make it possible to bypass. (In some cases someone also might want to download a single file using curl.)
20:05:50 <zzo38> There are also possibilities such as, use mutual TLS for write access (but optional TLS for read access).
20:06:18 <zzo38> These would solve different problems, but if all of them occur, then there are things to do to do them, but then you have to consider what other problem they cause
20:08:34 <zzo38> I had suggested port knocking in the past.
20:12:02 -!- salpynx has joined.
20:18:27 <esolangs> [[ShiftAlpha]] N https://esolangs.org/w/index.php?oldid=182111 * Broxol * (+3266) Created page with "ShiftAlpha (shift for short) is a programming language developed by [[User:Broxol]] in which programming the language is a block-sliding puzzle. This is not a turing-complete language. I intend to make a turing complete version, which I call ShiftAleph. ==INTRODUCT
20:19:13 <ais523> port-knocking on a public website would certainly be an unconvential solution
20:19:26 <ais523> although you could do it in a standard browser, so it would be possible to give instructions
20:25:01 <esolangs> [[If a==1:]] https://esolangs.org/w/index.php?diff=182112&oldid=164951 * CodePentuplets48 * (+662)
21:26:04 -!- impomatic has quit (Quit: Client closed).
21:50:27 -!- Sgeo has joined.
21:53:43 -!- salpynx37 has joined.
21:56:15 -!- salpynx has quit (Ping timeout: 246 seconds).
22:25:18 -!- salpynx37 has quit (Quit: Leaving).
22:38:01 <esolangs> [[]] M https://esolangs.org/w/index.php?diff=182113&oldid=182097 * The-coder256 * (-2)
23:00:07 <esolangs> [[Special:Log/newusers]] create * Geoff * New user account
23:02:07 <esolangs> [[Esolang:Introduce yourself]] https://esolangs.org/w/index.php?diff=182114&oldid=182093 * Geoff * (+108) /* Introductions */
23:04:52 -!- somefan has quit (Remote host closed the connection).
23:33:47 <esolangs> [[User:Miui]] https://esolangs.org/w/index.php?diff=182115&oldid=182084 * Miui * (+128) /* personal esopages */
23:48:55 -!- amby has quit (Quit: so long suckers! i rev up my motorcylce and create a huge cloud of smoke. when the cloud dissipates im lying completely dead on the pavement).