This is an archive of past discussions with User:JL-Bot. Do not edit the contents of this page. If you wish to start a new discussion or revive an old one, please do so on the current talk page.
The maximum DOI is automatically calculated from the last Crossref pull. Currently, the Crossref pull is done after the dump file is processed. Unfortunately, that means if a new reference in the latest dump includes a DOI above the last limit, it will get flagged as bad. Instead of basing the Crossref pull on the existance of a new dump, I probably should just have it always execute on the 1st and 20th so it will be done before the dump is available. -- JLaTondre (talk) 23:40, 10 August 2023 (UTC)
Yeah probably. Could update all the JL-Bot/DOI stuff on those dates too, which would let me create the newest DOI redirects ahead of the full dump processing. Headbomb {t · c · p · b}00:02, 11 August 2023 (UTC)
I notice the bot hasn't processed the dump yet? Normally it's done within the first 3-4 days of the month, but we're on day 6 now... 142.169.80.39 (talk) 17:53, 6 May 2024 (UTC)
There was an error while running. I have kicked off processing again, but it will take awhile to complete. -- JLaTondre (talk) 22:17, 6 May 2024 (UTC)
@Headbomb : The output for the May 20 dump is producing significant less citations than normal. For example, the A's end on page 100 this time when they typically go to 111. I am investigating to see what is going on. -- JLaTondre (talk) 19:52, 22 May 2024 (UTC)
It looks like the enwiki-20240520-pages-articles.xml.bz2 dump file is missing content. It is only 18G where the last one (20240501) was 20G. It usually increases each month so that is an unexpected (and pretty big) decrease. There are no processing errors and no changes in the expected citation templates. -- JLaTondre (talk) 20:13, 22 May 2024 (UTC)
Hmm, I did find this announcement. It doesn't explain why the dumps have not been completed, but sounds like there might be a format change which could impact parsing once it arrives. I use a library for the parsing so not sure if it will impacted or not. -- JLaTondre (talk) 14:16, 8 June 2024 (UTC)
Well at least it's in progress. I checked earlier this month around the 3rd and it hadn't started.
Long answer: {{Cite techreport}} was moved to {{Cite tech report}} back in June. Everything managed to still work okay until Citation botupdated the template usage in the articles. The parsing is based on the "real" template name. It is smart enough to also look for redirects to the template name, but it expects the primary name to be a non-redirect.
If it's been fixed, I can rerun the bot manually. Otherwise, it will get resolved in next weekend's run (assuming it has been fixed by then). -- JLaTondre (talk) 22:07, 27 July 2024 (UTC)
So "total DOIs" being the total number of times {{cite xxx|doi=}} or {{doi}} appears? The total number of {{doi}} templates is already on the page. Since it looks like both templates support |doi-access=free, I assume so, but wanted to check. And would the |doi-access=free count be all or distinct DOIs (since the other sub measurement would be distinct)? -- JLaTondre (talk) 21:33, 13 August 2024 (UTC)
Got it. The first should be easy. The |doi-access=free will require updating the dump parsing. I will try to get that in before the 20th dump. -- JLaTondre (talk) 23:01, 13 August 2024 (UTC)
Awesome. And before the next dump too. We'll get to see just how many more we got from the recent CS1 update (Aug 17) that flagged more free DOIs. We've got about 27% of all such citations that have been idenfitied as free to read. Not bad. Headbomb {t · c · p · b}17:43, 18 August 2024 (UTC)
You state "redlinks (with at least one dot)". However, if there was a [[One. Two]] blue link, you would not want a [[One Two]] redlink? -- JLaTondre (talk) 12:58, 18 August 2024 (UTC)
Upon further reflection, I don't know why I wanted at least one dot. I thought it would cut down on a certain class of common cases, but I can't think of them at the moment. So yeah, forgot that part for now. All redlinks that differ only by dots from a bluelink. Headbomb {t · c · p · b}17:41, 18 August 2024 (UTC)
Wikipedia:JCW/DOTS has been created. I ended up going with a single page as the maintenance processing was implemented with single page output. If the DOTS output grows and needs to be broken into multiple pages, I can do that. But I wasn't sure if DOTS was going to be used to clean citations up - I didn't want to do extra work if the actual result was the DOTS output would get smaller over time. DOTS will need to be added to the {{JCW-Main}} template and a description to the Maintenance page. Please let me know if you see any issues with the output. -- JLaTondre (talk) 23:47, 19 August 2024 (UTC)
Looks great. DOTS should shrink pretty fast. I don't know if I'll have time to make a dent in it before the next time, but by September it'll probably be under 100 entries. Headbomb {t · c · p · b}00:11, 20 August 2024 (UTC)
I think the only thing I'd change is make it case insensitive. And treat , and . as equivalent to each other (i.e. if redlinks differ only by commas or dots). Headbomb {t · c · p · b}01:38, 20 August 2024 (UTC)
The dump has been out for a few days. Normally the bot edits on the 3rd or 4th, but we're now the 5th... Has it crashed? Or something else? Headbomb {t · c · p · b}14:55, 5 September 2024 (UTC)
There was an issue with the server that has been fixed. Job is running now. It will take awhile to complete and upload results. -- JLaTondre (talk) 20:11, 5 September 2024 (UTC)
I made the change. It is now picking up items like entry #21 (where "Crit Rev Eukaryot Gene Expr . " matches). The delta between page revisions is a little hard to process as I realized it was not sorting the "Entries (Citations, Articles)" box and the result order would vary from run to run. It is now sorted so future changes will be easier to see.
However, it is not picking up the "Microbiol Immunol ." case as the original request was to capture redlinks that differ from bluelinks by dots. In the "Microbiol Immunol" case, there is no bluelink to match against (see M33). The redlink only cases are already picked up the "Spaced dot" case of the patterns matching (see #67 at Patterns) so doesn't seem like it needs to be added to the DOTS processing. -- JLaTondre (talk) 18:38, 18 October 2024 (UTC)
Fix has been made. The bot is currently running on any project page that has "vital" in its title. It will update the remaining ones during its normal run this weekend. -- JLaTondre (talk) 00:37, 7 November 2024 (UTC)
It looks like for some reason the page text was not returned for these DOI redirect pages so the bot was unable to parse out the target. It worked today. I'll change the code so that it stops processing if this happens. Hopefully just a one time API issue, but the change will avoid bad results if it does happen again. -- JLaTondre (talk) 22:33, 13 November 2024 (UTC)
There does not appear to be "Front Health Serv" or "Front. Health Serv." citations to match against (see F28 and F29]). The dots processing is only matching against other citations as per the other maintenance reports. -- JLaTondre (talk) 00:04, 11 November 2024 (UTC)
These (and all other maintenance reports, if that's how they work) should be matched vs existing links/targets. Headbomb {t · c · p · b}10:45, 12 November 2024 (UTC)
They are normally happening on the 1st and 20th? 12:56, 23 December 2024 (UTC) — Preceding unsigned comment added by Headbomb (talk • contribs)
Crossref had a slight change in the format of their results which caused the bot a problem. I have adjusted for it and am running the job again. -- JLaTondre (talk) 19:57, 23 December 2024 (UTC)
Thanks, hope you are having a good holiday season also. I had some free time so I implemented email from the server. Now, I should get an email when it fails vs. someone having to notice it didn't complete a run. -- JLaTondre (talk) 12:57, 26 December 2024 (UTC)
I see the issue. It is not properly getting the revid associated with the Publishers.cfg/A–M or Publishers.cfg/N-Z pages so it did not detect that this edit should have caused a reprocessing. It will currently only trigger if the other configuration pages are changed. When the original publisher page was subdivided, I didn't add handling for the /- which need to be URL encoded. I made a change for that and it should run tonight. I will validate in morning. -- JLaTondre (talk) 01:05, 18 January 2025 (UTC)
The dumpfile is dated 20250123 instead of 20250120. I updated the code to look for that date & it should pull it tonight. Hopefully they get back to a regular schedule. If not, I will have to update the bot to watch the RSS feed and use that to detect a new version. -- JLaTondre (talk) 00:36, 28 January 2025 (UTC)
Did the bot crash? Normally the bot edits at around 2AM my time when there's been changes to the .cfg pages, and this didn't happen today. Did the bot crash? Headbomb {t · c · p · b}13:46, 26 February 2025 (UTC)
Yes, it received an error when retrieving a page. I kicked it off again and everything seems fine. I did get an email notifying me of the failure (so that is working), but I didn't have a chance to look into until recently. -- JLaTondre (talk) 23:45, 26 February 2025 (UTC)
Thanks! I didn't realize that there was a single category that contained software, people, ideas, companies, and so on. I'll try that and see how it goes. Rjjiii (ii) (talk) 17:23, 1 March 2025 (UTC)
Initial version created. For the dash cases, I made it require a space before the dash as otherwise there were a lot of false positives (examples JBR-BTR, KRXI-TV) -- JLaTondre (talk) 15:29, 8 March 2025 (UTC)
Well it's the wrong category for sure (should be {{R from abbreviation}}), but the goal of the worklist is to catch as many as possible, so expansion is the solution for the bot.
Change made. There were 14k #invoke usages. Of those, roughly 180 were new journals/magazines cited and the remainder were increased counts of journals/magazines already reported. The individual pages are uploading now. The other reports (targets, popular, citewatch, etc.) will update whenever something triggers their processing. I can manually kick them off if needed, but usually there are changes to config, etc. that cause regular processing of those. -- JLaTondre (talk) 23:24, 8 August 2025 (UTC)
If, for example, you have |journal=Аcta Вaltico‑Slavica, where А and В comes from the Cyrillic alphabet and the others from the Latin alphabet, it would be useful in the complilation to highlight this sort of thing, i.e. when an entry has characters from two different alphabets. If it's from a single alphabet, no highlighting is needed.
Yes, that is doable. Perl, which is what that part of the citation processing is written in, makes it easy to check language scripts. Perl can recognize all the ones listed at perlunicode#Scripts (all the ones you are requesting are on that list). For Chinese, it would really be detecting for Han script - which in my understanding is used for several Eastern languages. It will probably be a couple of weeks before I can complete it. -- JLaTondre (talk) 23:14, 27 August 2025 (UTC)
Yes, if it's the Han alphabet, then that's the character set that should be highlighted. The point is to detect names that have multiple character sets in them, which should be rare, and usually limited to case like |journal=The Journal of Things = Το ημερολόγιο των πραγμάτων.
It's probably simpler to collect them and have them all reported on their own WP:JCW/Multiscript subpage, with that highlighting only in effect on that page.
Should it report cases where there is a language template? For example, what should it do with Sidirotrohia ({{langx|el|Σιδηροτροχιά}}) which will produce Sidirotrohia (Σιδηροτροχιά) (after the change discussed below)? There are also cases where people enter titles in multiple languages without the use of a template? Should it only report a mismatch when it happens within a single word? -- JLaTondre (talk) 23:53, 28 August 2025 (UTC)
If there's a language template, that can be ignored IMO. I suppose to start, mismatches could happen accross multiple words, this way it could catch things like Acta WhateverА. Headbomb {t · c · p · b}00:12, 29 August 2025 (UTC)
Updated Citation Field Extraction & Template Processing
@Headbomb I have been updating how the |journal= | |magazine= field is extracted from the citation templates. Currently, the bot is using a regex to extract the field, but this occasionally gives a bad result due to the complexity of pattern matching against templates within templates, comments, nowiki markup, and everything else people can embed in a citation template. It doesn't happen often, but there are cases that end up in Invalid titles that are due to parsing errors. The new method will use a tokenizer to split the citation templates into parts and pull out the |journal= | |magazine= field. I have found it to be more reliable.
As part of this change, I needed to tweak how the template expansion works. That led down a rabbit of hole of checking all the template expansions that have been implemented and validating them.
I came up with some items that could use your input.
There are a handful of cases where |journal=.. The old method would miss these whereas the new method will return a period. Do you want these to be reported or dropped? On the article output, this causes an extra period when the template is expanded. For example, at Arleen McCarty Hynes#References, you can see a ". ." on 20, 34, 35. I wasn't sure if this is something you wanted to cleanup.
The current processing removes ({{langx}}) from the end of a citation, but not other language templates. So |journal=Studime Historike ({{langx|en|Historical Studies}}) becomes Studime Historike, but |journal=Hubei Wenshi Ziliao ({{lang|zh-hans|湖北文史资料}}) becomes Hubei Wenshi Ziliao (湖北文史资料). My assumption is that any language templates in parenthesis (or brackets) at the end of a citation should be removed and only the non-parenthesis (non-brackets) section returned. Is this correct?
It will be a bit before this new version is ready to go live. I will do it separately from the language script detection request above so that it's easier to see any unintended effects of either change. JLaTondre (talk) 00:27, 28 August 2025 (UTC)
|journal=. should be reported yeah. No special processing needed, though I suppose they could also be added to WP:JCW/Dots. It definitely needs cleaning up.
Ideally, I think the best thing is to report what is rendered, so |journal=Studime Historike ({{langx|en|Historical Studies}}) can be treated as |journal=Studime Historike (Historical Studies) and |journal=Hubei Wenshi Ziliao ({{lang|zh-hans|湖北文史资料}}) can be treated as |journal=Hubei Wenshi Ziliao (湖北文史资料)
I will remove the special handling of langx in parenthesis. I did look at using the API to expand templates so that I wouldn't have to hard code processing them. This would have simplified things as I would not have to add handling new ones when they popup in dumps or make updates if there is a change in how they operate. However, it would cause additional items to appear in the output that I don't believe you want. For some examples:
{{ill|Die Sprache|de}} produces Die Sprache [de] where I believe only Die Sprache should be output
{{nihongo|BOMB|[[:ja:BOMB|BOMB]]}} produces BOMB (BOMB) where I believe only BOMB should be output
.jpg){kind=link}
