Wikipedia:Village pump (WMF)

 Policy Technical Proposals Idea lab WMF Miscellaneous 

• Table of contents
• First discussion
• End of page
• New post

The WMF section of the village pump is a community-managed page. Editors or Wikimedia Foundation staff may post and discuss information, proposals, feedback requests, or other matters of significance to both the community and the foundation. It is intended to aid communication, understanding, and coordination between the community and the foundation, though Wikimedia Foundation currently does not consider this page to be a communication venue.

• Discussions of proposals which do not require significant foundation attention or involvement belong at Village pump (proposals)
• Discussions of bugs and routine technical issues belong at Village pump (technical).
• Consider developing new ideas at the Village pump (idea lab).
• This page is not a place to appeal decisions about article content, which the WMF does not control (except in very rare cases); see Dispute resolution for that.
• Issues that do not require project-wide attention should often be handled through Wikipedia:Contact us instead of here.
• This board is not the place to report emergencies; go to Wikipedia:Emergency for that.

Threads may be automatically archived after 14 days of inactivity.

« Archives, 1, 2, 3, 4, 5, 6, 7

Update: Scheduled shutdown of Wikidata descriptions on EnWiki

Background: Some time ago some Foundation staff thought Wikidata item-descriptions could be used as convenient short-descriptions for Wikipedia articles. They began displaying them in several locations on our articles or as descriptor/links to the articles. Several concerns arose when the community noticed and examined the practice, resulting in consensus against using Wikidata on/for our articles in this fashion. (Link to one of several discussions.) Several editors and I discussed collaborative resolution with staff. The Foundation created a new feature allowing us to create and manage these descriptions locally (see WikiProject Short descriptions). The Foundation wanted to avoid suddenly blanking all descriptions, so Wikidata descriptions continued to appear when no local description is present. The Foundation agreed to shut complete the shutoff of Wikidata descriptions once we had created approximately 2 million local descriptions.[1]

I am happy to announce there are now 2,250,000 mainspace pages with short descriptions - approximately 1,939,000 articles, approximately 312,000 disambiguation pages, plus thousands of other assorted pages in and out of mainspace (portals, help pages, drafts, redirects, project pages etc).

Almost two months ago I created Phabricator task T248457 for the WMF to implement the agreed shut-off. The task appears to have slipped through the cracks unnoticed/forgotten. Ping DannyH (WMF) who gave the original shutoff commitment (or any Liaison or other staff) to get the ball rolling on this. I am eager to celebrate successful collaboration and resolution on this issue. Thanx. Alsee (talk) 13:39, 15 May 2020 (UTC)

In case it is unclear from the above, the WMF committed to disabling short description Wikidata fallbacks: the WMF plan is to switch from a Wikidata-fallback to full enwiki control when there are 2 million non-blank short descriptions on enwiki (quoting DannyH (WMF) from the discussion linked above). They have not done so yet, even though en.WP has exceeded that criterion by over 10%. – Jonesey95 (talk) 16:53, 15 May 2020 (UTC)

Alsee, Thank you for the update. —¿philoserf? (talk) 17:12, 15 May 2020 (UTC)

It's worth reminding DannyH (WMF) of his statement at the time: "the WMF plan is to switch from a Wikidata-fallback to full enwiki control when there are 2 million non-blank short descriptions on enwiki". Nevertheless, we are going to have to wait for somebody to triage phab:T248457 and then get it assigned and finally get it implemented. There's no timescale for that, nor is there any likelihood that anyone other than DannyH will feel under any obligation to do the job. I can only suggest that everyone who thinks that it's important for staff to respect their promises to the community, should post on that phabricator thread to urge some progress on implementation. --RexxS (talk) 00:05, 16 May 2020 (UTC)

• It would be good to see enwp control. We have a large number of editors working and we can do this. Wikidata is a super fine project, there is no doubt on it. However several things continue to disturb me: example, one Wikipedia article has an IMDb or Twitter external link or some other article value being called from Wikidata. Now you go to Wikidata, and find they are linking back (imported from/Reference) to English Wikipedia. This is confusing and make things circular (note: this particular topic is not in scope in this discussion, and we may discuss somewhere else if you want). Coming back to short descriptions, if we really need short descriptions, we should do it following our own guidelines and thoughts, and not from another project, specially where generic item descriptions are mass-created using QuickStatements and other tools. Regards. --Titodutta (talk) 00:49, 16 May 2020 (UTC)

  @Titodutta: one of the first features I implemented in Module:WikidataIB was to filter out values that were unsourced or referenced only to Wikipedia, That facility has been available for four years now, so there's no excuse for having the feedback loops you describe. If you let me know whenever you find those sort of problems, I'll fix them. --RexxS (talk) 01:20, 16 May 2020 (UTC)

  • I did not prepare a list. I'll create one on the go now. Anyway, a couple of (not the best) examples I can find now: Jan_Schmid external links "FIS Nordic combined skier ID" is on Wikidata, Wikidata says it is imported from English Wikipedia. Similarly "EL" section Official website at WePay. It might be a good idea to directly link, than going to Wikidata and learn it was imported from Wikipedia. --Titodutta (talk) 02:04, 16 May 2020 (UTC)

    Okay - that shows up because Zyxw used #property which doesn't filter. I can fix it by using a Wikidata call, but are you sure you need references for an identifier? An identifier will link to the entry in the relevant database, so following it will verify the id almost always. I don't usually worry about sources for images and their captions for the same reason. Anyway, I'll have a look at any list you want me to. --RexxS (talk) 02:35, 16 May 2020 (UTC)

    First, this is unrelated to the WMF so perhaps discussion should move elsewhere. The last Wikidata RFC was a long time ago, a lot of issues are unresolved, and I would welcome some clarity. If anyone gets the ball rolling on that, please ping me with the location. RexxS: On a personal note I've noticed some of your comments elsewhere to the Foundation, particularly in relation to Wikidata descriptions, and I wanted to express my appreciation. Back to the immediate topic, it sounds like you're considering a "fix" of filtering out values that are sourced from Wikipedia. In this case, I don't think that was the concern. If you recheck Titodutta's comment they didn't want the value gone. They were suggesting that the info should be "directly link"ed from Wikipedia, rather than coming through Wikidata. See my diff. I want to know whether or not the community wants edits like that applied broadly. I think it's an improvement, I believe(?) that is what Titodutta was suggesting, but we're in consensus-limbo on it. Alsee (talk) 11:11, 17 May 2020 (UTC)

• It's truly a pity to have wasted so much efforts adding clutter to articles. Nemo 14:21, 17 May 2020 (UTC)
  • If you mean the short descriptions are clutter then I very strongly disagree - they're extremely useful and the big shame is that they're going to disappear from the majority of articles if wikidata fallback is turned off (if it must be turned off, waiting until local coverage is close to 100% would achieve the goal without the any unnecessary accessibility issues). If you mean that the local (vs soured from WikiData) descriptions are clutter, then I'm not sure I agree - they're certainly unnecessary duplication of effort but a single line in the page source isn't really what I'd call "clutter". Thryduulf (talk) 14:27, 17 May 2020 (UTC)

    I wouldn't mind relying on local short descriptions. A few days ago, someone changed wikidata descriptions to label iCarly seasons as "child pornography" and "isis propoganda", but I only noticed because they bragged on Twitter. Changes to local short descriptions will show up in people's watchlists. Schazjmd (talk) 22:52, 20 May 2020 (UTC)

    Can't we just make wikidata changes show up on watchlists on an opt-in basis? Enterprisey (talk!) 04:07, 11 June 2020 (UTC)

    Can't wikidata get some form of ClueBot running to revert vandalism like that? It's such a big problem. {{u|Sdkb}} ^talk 04:41, 11 June 2020 (UTC)

• I wasn't involved in the prior discussions about Short Descriptions, so I don't have a well-formed opinion about them, but since this is the WMF page, I think the scope of the discussion here ought to be limited to encouraging the WMF to abide by the agreed-upon community consensus, not challenging that consensus, which inappropriately muddles things. As a new page, we need to set down some strong norms about what is okay to post here. That said, there does seem to be a fair amount of sentiment questioning where we are at with short descriptions, so I think it might be appropriate to open up a conversation at a more appropriate venue to take stock of where we are at with short descriptions and to potentially reassess whether we ought to go ahead with using 2 million as the point to cut off Wikidata imports. One piece of data that I can't resist sharing (despite what I just said above) since it might help frame the discussion is this: of the 43,000 Level 5 Vital Articles (example of the kinds of pages in that group here), currently 19,000 (44%) have no short description. {{u|Sdkb}} ^talk 19:43, 20 May 2020 (UTC)

  Sdkb we're all figuring on the scope of this page together. For what it's worth: My concept when I created this page, and according to the notes at the top, this is the correct page for discussing and forming consensus on the relevant topics. Therefore this would be the correct place to consider a new/different consensus. That said, I was also a central participant in the previous discussions. The consensus was actually to eliminate the Wikidata descriptions immediately, effectively blanking everything while we rebuilt local descriptions. Waiting for 2 million descriptions before shutoff was a compromise with the WMF, insofar as it wasn't particularly unreasonable and no one pressed the issue any further. At that time not shutting off wikidata, or waiting for "close to 100%" local descriptions, were pretty well WP:SNOWy territory. I expect that remains true today. Alsee (talk) 22:27, 20 May 2020 (UTC)

  Alsee, that makes sense. I'm concerned a little, though, that having too many conversations directly on this page (given how sprawly they can often get) might make it harder for WMF folks (who are understandably very busy) to follow along with it all. For some issues, it may be better to discuss them elsewhere as a sort of staging area, and then to come here once they're decided enough that we can present a more straightforward "this is what the community thinks" message. (We'd of course link to the full discussion; the idea isn't to hide anything, but just to keep this page more readable.) {{u|Sdkb}} ^talk 08:03, 24 May 2020 (UTC)

Ping Whatamidoing (WMF). The Foundation agreed that this would be done once we hit two million descriptions. There has been no effective response on phabricator T248457 since March 25, and DannyH (WMF) hasn't responded to the ping three weeks ago. Can you help get a response on this? Thanx. Alsee (talk) 23:13, 4 June 2020 (UTC)

We don't appear to be having much luck with this do we. Any ideas for next steps?©Geni (talk) 16:38, 10 June 2020 (UTC)

Given that Danny is active, and his line manager is probably Katherine Maher, probably suck it up. And stop adding short descriptions. Unless someone can get her attention on Twitter. I do not think she is paying any attention to what is happening on Wikiprojects. I am not going to resign an admin bit over it.--Ymblanter (talk) 18:12, 10 June 2020 (UTC)

And do not forget to take it with the candidates on the Board elections which are going to happen next year. Some of you (not me) may even try to check with the candidates from affiliates.--Ymblanter (talk) 18:15, 10 June 2020 (UTC)

One option is to effectively shut off Wikidata descriptions from our end. Right now there is a filter in place to prevent us from setting a description that is blank or only punctuation. We could modify {{short description}} to check if the value is blank, and then fill in some non-blank value of our choosing. (Perhaps "No description yet.") Then we can do a bot run to make sure {{short description}} is present on every article.

Another option (and we can proceed with both options simultaneously) is to draft a message to Executive Director Katherine Maher and submit it as a formal consensus message from the EnWiki community. My concept would be to start by expressing serious community concern at the poor relationship, poor alignment, and poor level of partnership between the Foundation and the community. I'd suggest phrasing it like 'perhaps you are unaware of these systemic problems', as I suspect staff aren't reporting these issues up to the Director. Then we briefly lay out the Short Description case, that Wikidata descriptions were deployed without consulting the community, that there was a commitment to shut off Wikidata descriptions once we created 2 million descriptions, and then lay out how long the Phab task has been ignored, how long the manager who made the agreement has been non-responsive, and how long the liaison has been non-responsive. (Note: The liaison was only pinged 6 days ago, which is not good but also not badly excessive yet. However those figure will presumably be higher if/when we deliver the message.) Then I would close by requesting her help improving Foundation-engagement, and that we have other specific concerns we hope to be able to bring to her attention. I tend to procrastinate, but I'd be willing to draft it and open the proposal if there's support for this route. (Ping me for quickest action.) I have other specific issues that I want raise for consensus-discussion, but I think we should start with this one simple item. Asking the Director to follow through on an existing Foundation-commitment should be comparatively easy, and it will hopefully establish the path for trying to resolve other issues. Alsee (talk) 20:45, 10 June 2020 (UTC)

I pinged DannyH on 16 May in this very thread as he's the one responsible for keeping the promise he made, so he has no excuse for not having enough time. I can't help but wonder whether we could take the unusual step of discussing the issue on Jimbo's talk page first and asking for his help and advice. That would at least make sure that we try all of the avenues available to us before starting a formal complaint to the CEO. It would also perhaps raise the profile of the problems we've encountered because of the number of talk page watchers there. I almost made that very post there today, but I thought it would be reasonable to leave it a few days yet, just in case we do get a reply from WAID as she was only pinged a few days ago. --RexxS (talk) 23:45, 10 June 2020 (UTC)

The end of the fiscal year is always a busy time of year for him. He and I are usually in a meeting together on Mondays; I'll try to ask him about this the next time I see him. That may be more effective right now than asynchronous communication. Whatamidoing (WMF) (talk) 17:39, 11 June 2020 (UTC)

thanks WAID.--Ymblanter (talk) 17:57, 11 June 2020 (UTC)

@Whatamidoing (WMF): It's now been more than a week. * Pppery * _{it has begun...} 15:23, 21 June 2020 (UTC)

Hi everyone, I'd like to let you know that we are currently working on plans to make the switch to entirely using local short descriptions on English Wikipedia, without the Wikidata fallback. We really appreciate that people have put in so much time and work populating the local descriptions.

I don't have an estimate yet for when we expect the work to happen, because there are a few use cases that we need to figure out. Right now, the Wikipedia apps allow people to write and edit short descriptions on their phone, and that's a popular feature that helps readers who may not typically be Wikipedia editors to contribute new descriptions, and to keep existing descriptions accurate. We want that feature to only use the local descriptions now, so we're going to work on that. We also want to make sure that the descriptions work properly when they're dynamically generated from an infobox, as Template:Infobox settlement currently does.

Engineers are talking about this now, and you'll see activity on the Phabricator ticket. I can let you know when we have a stronger idea of how and when the changes are going to be made.

One thing that I'd like to ask about is the point that Sdkb brought up above: how do we get from ~2 million descriptions to ~6 million (minus disambiguations and lists)? When we make this switch, descriptions are going to "disappear" on a lot of articles, and as Sdkb points out, that will include some important articles. I know that some people have been using automated tools to import descriptions from Wikidata, and I'd like to know if folks plan to continue doing that, and if we can help. I'm interested to know your thoughts. — DannyH (WMF) (talk) 21:46, 22 June 2020 (UTC)

@DannyH (WMF): thanks for your announcement, it is very welcome. I'm sure we will all be happy to hear your estimate for when the work will happen as soon as you have that information.

I'd like to suggest that the solutions to the issues you raise could lie in a change of mindset. When you consider the short description text that accompanies searches on mobile and acts as a subtitle on the app, it may be helpful to forget about Wikidata and to think about short description text as being sourced authoritatively from the enwiki short descriptions; then to refocus your efforts into finding ways of making the enwiki short descriptions as comprehensive as possible. These are some examples that come to mind:

• Support a proposal to import missing enwiki short descriptions from Wikidata on a one-off bot run. This won't be easy to generate support for, but it would represent a way of closing the old chapter and starting a new one.
• Redirect mobile apps that allow editing of short descriptions to creating enwiki short descriptions. Worry about getting those descriptions into Wikidata as a secondary issue.
• Support moves to allow bots to update the Wikidata description field periodically to import enwiki short descriptions as the authoritative text. There will be resistance from Wikidatans but it should be possible to offer a good case for doing it.

I know you'll think "that's what I'm already proposing", but I wanted you to consider principally the focus of where the short descriptions come from. While you think about the issue as an uneasy hybrid between enwiki and Wikidata, you can't hit the targets you want. If you embrace the concept that the authoritative text resides on enwiki, you'll mobilise the large workforce on enwiki (who have already done so much amazing work) to continue to improve and expand the coverage of enwiki short descriptions. Having that sense of ownership of our content is fundamental to keeping work going on it. --RexxS (talk) 22:35, 22 June 2020 (UTC)

RexxS: Well, that is still what I'm already proposing. :) We are now thinking about enwiki as the authoritative text for English short descriptions, and with that as the focus, we want to help to get descriptions on those x-million more enwiki articles. I think we're on the same page. — DannyH (WMF) (talk) 05:58, 23 June 2020 (UTC)

that's a popular feature that helps readers who may not typically be Wikipedia editors to contribute new descriptions, and to keep existing descriptions accurate Well, it is also a feature that turns existing descriptions into inaccurate ones and allows people to create new inaccurate ones where no description existed. This seems to be common for Indian caste articles, where glorification and denigration is the name of the game, and it is difficult to track. - Sitush (talk) 09:27, 29 June 2020 (UTC)

-

@DannyH (WMF) How do we get from 2M descriptions to 6M? Make them visible on the article page in desktop web and mobile web. People won’t edit what they can’t see. Once they get used to seeing the shortdesc on good articles, they’ll want to add them to others.

Perhaps, as Alsee suggested, display "No description yet". But now that we have way more than a critical mass of articles with local descriptions, that mightn’t be necessary.

I know the en-wp community tends to be resistant to changes that are opt-out rather than opt-in. But if we can get consensus on this, will the Foundation support us?

Aside: another team at W?F is looking at simple structured tasks to engage new users, short descriptions could be a good fit for that.

Pelagic ( messages ) Z – (12:19 Sat 04, AEST) 02:19, 4 July 2020 (UTC)

Instead of backsliding, and playing for time, if WMF actually did what they said they would and shut down the displays of Wikidata descriptions, now that we have basically paid up on the extortion, it might encourage Wikipedians to add more short descriptions. · · · Peter Southwood ^(talk): 13:04, 8 July 2020 (UTC)

@Pelagic: The worst part is that some non-W?F editors literally implemented all of the features that W?F keeps saying they'll "get around to adding on desktop" in the short description helper gadget. It displays short descriptions at the top of the page under the title; tells whether they're defined locally, being pulled from Wikidata, or were generated automatically by an infobox or similar template (or if they don't exist in any of those places, which is pretty rare at this point); and it gives options to easily and quickly edit them and import/export them from Wikidata. Literally all the W?F needs to do is flip on that gadget by default and pretty much everyone's problems would be solved. Actually, the community could just flip on that gadget by default: setting a gadget as default doesn't require Foundation action and could just be done by an interface admin after getting consensus. Nathan2055^{talk - contribs} 22:38, 8 July 2020 (UTC)

Nathan2055, I don't think the gadget actually does all of those things, though it is a great gadget. · · · Peter Southwood ^(talk): 12:55, 9 July 2020 (UTC)

@DannyH (WMF):: I actually don't think these problems are quite as difficult to solve as they sound. Our own short description helper gadget already handles displaying and editing locally defined, pulled from Wikidata, and infobox-generated descriptions fine, and allows for one-click importing and exporting between Wikidata and enwiki. That gadget could be flipped on by default for logged-in editors, with a MediaWiki extension-based solution eventually adding a version of it for logged out users as well. As for handling descriptions "disappearing" from articles that only have one defined on Wikidata, I think the easiest solution to that is adding an optional imported=yes parameter to {{short description}}, having that parameter stick the pages in a Category:Pages with descriptions automatically imported from Wikidata category or something similar, and then having a single-run bot go through and import in the Wikidata descriptions for every article which doesn't have one defined locally. Then we can just work through that category and verify that all the descriptions look acceptable. I certainly think that's a better choice than just leaving automatic Wikidata population on, because we've already seen just how ridiculously prone to abuse that system is. It was a good idea in theory, but I think we have enough evidence at this point to say that no project should be displaying data stored on another project that hasn't been checked manually by a human first. It's just too dangerous. Nathan2055^{talk - contribs} 22:54, 8 July 2020 (UTC)

@Nathan2055:, If I remember correctly this is one of the proposals that was specifically turned down previously. Not to say you cannot propose it again, but I fear it would still face opposition on the basis that the descriptions in Wikidata should be checked by a Wikipedia editor before inporting them, and that the person who imports a description from Wikidata is personally responsible for ensuring that it is, if not necessarily very good, at least not very bad. A semi-automated tool to run through a list of pages lacking a short description, using the helper would be OK, as an editor would be checking before importing, at least in theory, and can be held accountable for any harm done due to lack of diligence. I might use such a tool myself. · · · Peter Southwood ^(talk): 11:23, 16 July 2020 (UTC)

• Perhaps the Shortdesc helper should become a default feature for all users (or at least all registered users); I've noticed that my adding short descriptions to articles has skyrocketed since then, and I believe that the acceleration of adding short descriptions to articles will far outweigh the possible increase in vandalism. – John M Wolfson (talk • contribs) 03:00, 16 July 2020 (UTC)

  I agree that higher visibility could greatly accelerate the addition of short descriptions. Making the helper a default feature would make short descriptions more visible, so it could help accelerate addition, if it does not start another flare-up of the squabble over whether they should exist in the first place. Unfortunately, the history of how they were basically forced onto Wikipedia will be dredged up and revisited, both by the people who opposed it in the first place, and probably also by a whole batch of people who have managed to remain unaware of the details of that dispute, and possibly even unaware of the existence of short descriptions. Could be worth a try. This would call for an centralised RfC as it would affect everybody, and as I was heavily involved in the original dispute, I would prefer not to draft it myself. My own feeling is that it is time for WMF to display a little good faith and do what they promised. If and when that happens I am willing to support such a proposal. If it does not happen, that failure do deliver on a promise will be used as a weapon to resist making the gadget a default, which could delay or prevent long term predominance of articles having a short description, which to my mind would be a lose-lose result.· · · Peter Southwood ^(talk): 10:57, 16 July 2020 (UTC)

  Given that installing the gadget increased my own short description adding probably around fivefold, I think it should be the default for registered users, which would accelerate the process of making short descriptions near universal. I can draft an RfC to that effect if you'd like, seeing as it might be inevitable. I wonder what the hold up with the Wikidata-description deprecation is given that more than 2 million mainspace articles (albeit including disambiguation pages) that have them, which was the benchmark as I recall. – John M Wolfson (talk • contribs) 08:42, 17 July 2020 (UTC)

As far as WMF Product is concerned, we'd be happy for the short descriptions to have more visibility on desktop or mobile, either opt-in or not. I agree that being able to see them would help to motivate people to add and update them. People were so adamant in previous discussions about not displaying them, that we didn't know if having them 100% hosted on enwiki would make a difference. It's good to know that's on the table, and I'd be happy to talk more about how to get to consensus, and how we can help.

To respond to the questions above about whether we are doing what we've promised, work on the change has begun — if you're into Phabricator, you can see the progress that's being made on this ticket. — DannyH (WMF) (talk) 18:50, 21 July 2020 (UTC)

While we wait for the squabbling at phabricator about avoiding doing something they are ethically obliged to do to finish, perhaps we should do a bot run to add a short description to all articles where a local short description does not already exist, putting them into a maintenance category and with content stating that they have no short description yet, but anyone can help by adding a suitable short description. This would use existing code to discontinue the use of Wikidata content and make the missing descriptions more visible and findable. Then it would no longer matter how long the issue is avoided by those who want to push Wikidata content into English Wikipedia against community consensus and against the word of WMF that it would be stopped. Cheers, · · · Peter Southwood ^(talk): 11:04, 30 July 2020 (UTC)

That sounds like a great idea in theory, but what short description would be displayed in practice? I believe that WMF has technical measures in place to prevent us from overriding an inappropriate or vandalised Wikidata description with a blank one. Certes (talk) 11:43, 30 July 2020 (UTC)

Something like:"This article needs a short description - Click here if you think you can help" The link should take the editor to an explanation of what is needed in a short description, and after reading it they could click a button indicating they understand and will comply with the requirements, which would skip this stage in future and open the short description gadget on the page they clicked from, which should display the usual options, including copying the SD from Wikidata if it is good enough. This cannot reasonably be described as an inappropriate or vandalised short description, nor a blank one. It would serve the purposes of Wikipedia and incidentally also WMF in a way that maximises usefulness to all, and does no harm to Wikidata, so if WMF were to override it it would be strong evidence of unethical and counterproductive behaviour on their part. Or have I missed something? Logged out users would unfortunately have to go through the instructions every time, or maybe a cookie could help here. · · · Peter Southwood ^(talk): 12:43, 30 July 2020 (UTC)

Those editors who have encouraged the adding of missing short descriptions by bot and user-assisted scripts might like to have a look at the miniproject which is currently being planned at Wikipedia talk:WikiProject Short descriptions#Bot proposal to create short descriptions from scratch, for articles requiring one. Given the sensitivities of some editors to the importation of Wikidata descriptions, we will create new SDs for articles that need them from local information only, without pulling anything from Wikidata. Contributors are welcome. MichaelMaggs (talk) 16:56, 14 August 2020 (UTC)

Phab:T248457 has now been triaged and classified as "Later". Will Wikidata descriptions still be removed as promised, or has the WMF's half of the bargain been postponed indefinitely? Certes (talk) 15:17, 2 September 2020 (UTC)

Clarification from the WMF folks would be appreciated - are you going to do this on your end, or do we need to set up a bot to do this ourselves? Tazerdadog (talk) 23:18, 2 September 2020 (UTC)

Summary

So, in summary:

• The WMF implemented, years ago, the addition of Wikidata descriptions to enwiki articles without informing enwiki of this, without providing any way to monitor or overrule this, ...
• After an RfC which overwhelmingly opposed this, they promised to turn this off, but did this only for some uses and not for all
• When this was found out, the opposition was reaffirmed, but the WMF then, instead of finally keeping their original promise, haggled for a compromise based on rather fanciful data, and got us to agree to add 2 million descriptions, and only then they would turn it off.
• In March, this goal was achieved, and the WMF asked to keep their side of the bargain. It then turned out that while many people on enwiki had worked hard to keep our side of the bargain, not a single thing had been done at the WMF side to keep theirs.
• Some six months later (and three months after it was promised that we would see "work being done" at the phabricator ticket), there is zero progress, and the task is in fact scheduled for "later" and not assigned to anyone at the WMF.

Is this somewhat correct? And if so, why is this in any way acceptable? Fram (talk) 08:38, 4 September 2020 (UTC)

Pinging DannyH (WMF). What steps can the volunteer community here at en.WP take to ensure that WMF staff are able to keep the commitment that you made? – Jonesey95 (talk) 16:08, 4 September 2020 (UTC)

Hi, thanks for the ping — this is a misunderstanding based on ticket numbers. :) The ticket that Certes posted about a couple days ago is Phab:T248457 — that ticket was submitted by Alsee, and that's not the one that the developers are using. I mentioned above that the correct ticket is Phab:T256817, which is currently being worked on — you can click on some of the subtasks to see that discussion and work are happening on them today. I posted a link to the correct ticket number above on July 21st, but I didn't stress enough that it was a different ticket; I just linked the phrase "this ticket". Sorry that I didn't make that clear enough. I agree that we have promised to make this change happen, and we are working on making it happen. Let me know if you have more questions. Sorry about the mixup. — DannyH (WMF) (talk) 20:19, 4 September 2020 (UTC)

Thank you for the explanation, Danny. It's comforting to know that this is still a priority for the WMF. Certes (talk) 20:27, 4 September 2020 (UTC)

Off topic

DannyH (WMF) sorry to change the subject, but while you're here I wanted to give you an important update on the Talk Page project. A quick recap for you and everyone else: Back when the Flow prototype was being built, editors told the project manager that correct wikitext support was fundamental and necessary, and that the project could never be deployed without it. The project manager said he had no intention of fixing it, and cut off communication saying further discussion was a waste of time, and proceeded to build a system with needless wikitext corruption issues. Eventually I had to organize an effective Global Community Consensus across three wikis before the Foundation finally acknowledged there was a problem. You stepped in and did an excellent job running the Talk Page Consultation, genuinely listening to what we wanted. You cancelled the last phase of the consultation, and handed the project over to a new manager. That manager has essentially cloned Flow's back-end design problem. Instead of simply inserting the new comment into the page, he decided to needlessly translate everything through Parsoid before inserting the new comment, then retranslate the combined content back through Parsoid for saving. Again, like with Flow, this round-trip double-translation causes wikitext corruption to the page. Again, the manager has declared that he has no intention of fixing it. Again, the Foundation has terminated communication. I honestly can't understand why the Foundation becomes actively-hostile whenever we say "don't screw up wikitext". It's such a simple and basic requirement. I thought you might want to look into this. I'm hoping you can avert an RFC against the new Talk Page Project. Alsee (talk) 23:28, 4 September 2020 (UTC) @Alsee: I think you are exaggerating a bit here. The developers of DiscussionTools seem to care about edits using their tool not corrupting unrelated parts of the page. * Pppery * it has begun... 21:56, 5 September 2020 (UTC) Hi Alsee: I'm working closely with Peter on this project, and I'm aware of your concerns about wikitext corruption. From what I recall from a few months ago, Peter asked you for examples of wikitext that you said were corrupted by the tool, and looked into the problems. Other active contributors appear to be satisfied with the team's progress on the tool; some folks on Meta are discussing it on Meta:Babel, with a lot of support. — DannyH (WMF) (talk) 20:40, 9 September 2020 (UTC)

So, instead of a ticket that is unassigned and "later", we have one that is unassigned and "blocked/waiting"... Still doesn't explain why nothing had been done on the WMF side in the years since their promise and while we were active keeping our side of the (WMF-forced) bargain. Looking at the underlying tickets, it seems that some initial activity in early August was the last that actually happened. Fram (talk) 07:05, 7 September 2020 (UTC)

Is there anything we can do to help unblock it or end its wait? Is the Android app board the best place for the ticket (or is it also in other work flows – I'm unfamiliar with Phab "boards")? Certes (talk) 09:14, 7 September 2020 (UTC)

One thing we as a community could do is to never again modify our behavior based upon any promise from the W?F, citing this example of them not keeping their promises. A would be interested in hearing about any other examples of unkept promises by the W?F. If any come to mind please drop me a note on my talk page. --Guy Macon (talk) 17:51, 7 September 2020 (UTC)

Guy Macon, I'm sure it is challenging to be a Wikimedia developer, but there are multiple examples of this phenomenon. Take ISBN magic links (please!). We sure have worked quite a bit, and continue to work, to transform ISBN magic links into ISBN templates after T145604 (follow links there for discussion) and a related en.WP RFC about magic links. Four years later, we're still waiting on WMF developers for the ability to turn off magic links here at en.WP. – Jonesey95 (talk) 04:07, 8 September 2020 (UTC)

I'm starting to see an underlying theme here. I'm sure we're all grateful to the WMF for providing a stable computing and comms platform and for being the legal body which a project with BLPs and other controversial content needs. The problems seem to start when the WMF imposes technical changes: ISBN must be a magic word; short descriptions must be imported; you will change to this text editor or that talk page format. If they are appropriate for and welcomed at some projects then by all means make them an option, possibly even the default with due notice. However, it should be a requirement of any breaking change that it either have community consensus or be easily turned off by local sysops where it is not wanted. We may have differences on other issues, such as the WMF's proposal to share Wikipedia's title and the UCoC, but making unwanted technical change optional would go a long way towards restoring the WMF's former status as a welcome partner. Certes (talk) 10:30, 8 September 2020 (UTC)

The problem is that that would mean that every technical change would result in two version going forwards which becomes a maintenance headache.©Geni (talk) 00:08, 10 September 2020 (UTC)

Certes, as problematic as the foundation can be. I don't think you have a clue how hard it is to develop this platform. Things you are saying are fun if there were 4000 developers working on all this. —TheDJ (talk • contribs) 07:39, 10 September 2020 (UTC)

According to phab:T257486 it's blocked because there isn't an API to allow editing local descriptions from the mobile app (they don't want users being able to edit descriptions but not able to see those descriptions) yet, but building that API seems to be currently being worked on. – Majavah ^{talk · edits} 05:59, 9 September 2020 (UTC)

What happens when the mobile app edits the description of a page which has a local short description? Do they edit the {{short description}} in the WP article, or the Wikidata description? If the latter then the problem they are trying to avoid already exists for two million articles, so it may be wise to disable app description editing anyway, which would unblock the fix. Certes (talk) 10:18, 9 September 2020 (UTC)

Hey all, I want to let you know that work is still continuing on this. I know that seeing something called "Blocked/waiting" seems disheartening, but in this case, that just means another team needs to do something before the first team can pick it back up. I've been assured that the platform team is working on their part this week, and then the Android team will be unblocked. This is taking a while, because we want to make sure that it's actually done correctly — that every place where someone can edit a short description, we're getting the old description from the correct place (English WP page) and the new description is added/edited in the right place (again, English WP page). I know that it's exciting to accuse the WMF of lying and betraying everyone, and I would be the last person to try to deny folks that particular form of entertainment. In this case, we made a promise that we are currently in the process of keeping. — DannyH (WMF) (talk) 17:39, 10 September 2020 (UTC)

Any reason why the work on this didn't start at the time of the promise? Apparently this isn't easy (considering that we are now 6 months since our part of the deal was done), so shouldn't have some planning, analysis, work ahead have been done? It almost looks as if everyone at the WMF was surprised that this needed to be done, and had to start working on this years too late. Surely that can't be true? And I guess Certes would like a reply to his post of 10:18 9 september directly above, which seems rather on point. I notice that Task 257488 only got someone assigned two days ago (well, it was assigned in July, then almost immediately declared "blocked/waiting", and just now assigned to someone else). This sudden revival is of course unrelated to this thread.

It's just that some of us are rather sick of still getting vandalized Wikidata descriptions in various enwiki displays. E.g. a few dating from at least 4 days ago and still active at the time of writing, some simply rather unprofessional (2020 New South Wales Rugby League gets the description "202p new South Wales"), some weird (Verghese Kurien, 600 pageviews per day, is "Indian founder of dairy-cooperative Amul. Atali buttely delicious amul", 2020 Saskatchewan general election gets "Photo of Current Premier of Saskatchewan -Scott Moe".

Or if you want really high-profile pages, take e.g. Karl Marx, with a vandalized description now since more than a day[2] (visible e.g. at the Commons infobox, not just an enwiki problem). Nalanda, world heritage site, vandalized description since almost 24 hours. BLP violations, like for Jacek Kurski, where the description of this politician/journalist was changed 22 hours ago to Polish politician and government propagandist.

To be fair, anti-vandalism at Wikidata sometimes does work. Sodapoppin, a BLP, had his alias "little genitalia man" removed a few days ago[3], after it had been added in December 2018... On the other hand, Baby Phat is still named AMIT KUMAR on Wikidata, since May 2020[4]: this has been copied to other languages through smart bots... It's a good thing that despite the exaggerated claims of having so many editors and views, very few people actually ever look at Wikidata. The only way most people come into contact with it is when it is being pushed into Commons or enwiki.

So perhaps the pressure we put on this is not just a desire to accuse the WMF of bringing on the end of the world, but also a genuine concern that this (descriptions, and more generally vandalism on Wikidata perpetuated to other, more visible sites) is still not being taken serious by the WMF, and that only pressure from groups like enwiki eventually gets some result, perhaps, someday. Fram (talk) 07:32, 11 September 2020 (UTC)

Just an FYI that work is currently being done to unblock the switch to only use local descriptions: [5][6]. Ryan Kaldari (WMF) (talk) 00:34, 18 September 2020 (UTC)

Current work

Hi all, there was some confusion in the conversation above about which Phabricator tickets you should look at if you want to follow along with the work being done. We've cleaned up the tickets so it's easier to follow now, and the easiest place to look is "T256817: Replace Wikidata descriptions on enwiki with local descriptions". If you click through the subtasks in that tree, you'll see that there are patches currently being worked on and reviewed right now. I hope that helps folks who are monitoring the progress. Please ping me if you've got questions, or something that you want to talk about. — DannyH (WMF) (talk) 19:44, 22 September 2020 (UTC)

It’s hard to see what's progressing with all the sub tasks in Phab, but I observe the following now. Joseph Henry Maiden (Q2583058) has "Anglo-Australian botanist (1859-1925)", "botaniste anglais", "australischer Botaniker". At w:en:Joseph Maiden I see "Missing article description (Add)" on desktop shortdesc helper, no desc on Timeless or mobile/Minerva search drop-down, nor in Related Articles from Margaret Flockton. Desc shows in iOS app at the head of the article, but not in search drop-down. For comparison, w:fr:Joseph Henry Maiden has "botaniste anglais" both below the title and in the search drop-down on mobile. Seems to be mostly turned off now for en. Would that reflect your understanding, DannyH? —Pelagic ( messages ) – (22:20 Thu 15, AEST) 12:20, 15 October 2020 (UTC)

Too late now, but ...

Wikidata descriptions generally aren't bad, maybe because few vandals discovered how much "fun" could be had there. The problem seems to be that lack of visibility meant problem descriptions didn’t get fixed as quickly as desired. With the luxury of hindsight, all this could have been avoided if:

1. Changes to Wikidata descriptions were logged as revisions to the linked articles in the same language (perhaps a null edit with summary like "Description on Wikidata (Q93462772) changed from "..." to "..."). Then they would show in watchlists, recent changes, etc.
2. Descriptions were displayed on articles for web platforms, not just search results and related-articles lists. Sure, search results are where they have great value as disambiguators, but being rendered at the top of every page would help readers notice problems.

The horse has bolted for w:en, but please, Danny (and for #1 maybe Lydia), for the sake of the larger non-English Wikipedias, could you consider those as potential improvements? —Pelagic ( messages ) – (21:04 Thu 15, AEST) 11:04, 15 October 2020 (UTC)

Hi Pelagic, Thanks for the ping! I am the right person for the first one. It is in fact already on my list of next things to ask my team to tackle. The ticket for it is at phab:T191831. --Lydia Pintscher (WMDE) (talk) 12:01, 15 October 2020 (UTC)

Thanks, Lydia Pintscher (WMDE), that's great to know. I found Phab:T257588 "Improve the integration of description and label fields with other Wikimedia projects" but that seems more general. Pelagic ( messages ) – (19:47 Fri 16, AEST) 09:47, 16 October 2020 (UTC)

Pelagic extensive previous discussions have made it clear that the points you list would not be sufficient to address the problems or wikidata-item-descriptions acceptable. It never made sense in the first place to try to store language-local content remotely on wikidata. Alsee (talk) 10:08, 16 October 2020 (UTC)

Current status

Hi all: I believe that now we're only showing enwiki-hosted descriptions on English WP, and not pulling from Wikidata at all. This works for me on the app article pages and search results, on the mobile search results, and in the Visual Editor link modal. One thing that's pending is that adding new descriptions on the iOS app saves to Wikidata and not to English WP right now, so you don't see the description that you just tried to add — this should be fixed in the next app update. Other than that, I believe that the change is now working everywhere. Does anyone see something that I've missed? (ping Pelagic, who was looking at it a couple weeks ago.) — DannyH (WMF) (talk) 23:38, 2 November 2020 (UTC)

Yes, Danny, I see that the iOS app now shows the w:en description. (E.g. for koala c.f. koala (Q36101), the app says "An arboreal herbivorous ...".) – without updating the app, still on 6.6.2 (1745).

"+ Add article description" only shows if there is neither a shortdesc magic word / template nor a wikidata description. Perhaps it should create both? (Wikidata might have their own ideas about the great unwashed body of app users creating descriptions having initial capitals, but *shrug* ...)

Good to know this is almost finished. Now that English Wikipedia has sovereignty over the descriptions, I'd like to see them displayed on Desktop Web (without requiring users opt in to the gadget or some custom script/CSS). Is there any plan for a per-wiki config that can do this, if a community agreed they want to turn it on?

— Pelagic ( messages ) – (20:00 Thu 05, AEST) 10:00, 5 November 2020 (UTC)

IP Address Masking Confirmed As Mandatory

Per a recent update on the IP Masking project, Legal has apparently decided that IP masking is no longer desirable but mandatory, with consultation now limited to implementation form.

Thus far Legal have not provided reasoning on that, but they are set to give a statement (detail level unknown), likely in the next week.

As this will have a significant effect on anti-vandalism efforts, please provide your ideas, concerns, and comments on the discussion page on how to mitigate any negative consequences and utilise any potential positives. Nosebagbear (talk) 10:19, 19 October 2020 (UTC)

This link will be useful here--Ymblanter (talk) 11:01, 19 October 2020 (UTC)

Looks like we need, and will have, an RFC on this. Alsee (talk) 09:38, 23 October 2020 (UTC)

I believe that our traditional workflow is as follows:

1. The W?F proposes something on an obscure meta page. Nobody notices.
2. The W?F posts it somewhere else, and literally everyone who replies hates it.
3. The W?F reaffirms their commitment to listening to user feedback.
4. The W?F announces that they are going to go ahead and do it anyway and you can all go and pound sand.
5. An RfC is posted. Hundreds of people contribute. The result is overwhelmingly negative.
6. The W?F goes ahead and does what they were always planning on doing.
7. The shit hits the fan, admins resign, The Signpost does a feature. Wikipediocracy does a feature. The Register does a feature. The Guardian does a feature. The New York Times does a feature.
8. The board of directors tells the W?F to knock it off. Nobody gets fired or demoted.
9. Return to start.

I will make popcorn. --Guy Macon (talk) 17:19, 23 October 2020 (UTC)

Sometimes they skip step 4 and go straight to step 6, which we then follow with step 5. Extra butter, please. – Jonesey95 (talk) 17:58, 23 October 2020 (UTC)

Depiction of Wikipedia Foundation Wikimedia Foundation destroying Wikipedia with the Fram ban, IP masking, and the 2020 rebrand instead of making obvious but boring improvements to what we have. —pythoncoder (talk | contribs) 18:52, 24 October 2020 (UTC)

The W?F has thrown a lot of crap at us before, but basically saying "we want more vandals" is a new low. Popcorn tastes good. —pythoncoder (talk | contribs) 18:52, 24 October 2020 (UTC)

This might be a good next step. -- RoySmith (talk) 21:30, 1 November 2020 (UTC)

I've said this elsewhere, but didn't gain much traction. Showing IP info to logged-in users isn't a problem. Exposing it to every anon, scraper and mirror is a problem. But W?F want to hide it from all of us. Pelagic ( messages ) – (20:33 Thu 05, AEST) 10:33, 5 November 2020 (UTC)

RFC on Discussion Tools content corruption

As a result of the Talk pages consultation 2019, the Foundation ended their strategy to eliminate&replace Talk pages with Flow. It also resulted in a decision to keep existing Talk pages, to keep existing Talk page editing, and to build new tools on top of the existing Talk system. The current project is called Discussion Tools. Discussion Tools is, in general, a promising product. It aims to add a Reply link after comments, and help automatically insert the comment into the page wikitext at the right spot with appropriate indentation. I hope to see it ultimately improved and deployed. However the RFC below describes a design problem with the product, and the project manager doesn't want to fix it. The proposal below seeks consensus that the product not be deployed until the design flaw is fixed. Alsee (talk) 18:43, 28 October 2020 (UTC)

One of the reasons Flow failed was broken wikitext support, including a wide range of content corruption issues. Discussion Tools does not look like Flow, but the backend code basically clones Flow's design flaw. When Discussion Tools is used, it can and does cause corruption to existing Talk page content. I will try to keep the technical details to a minimum. Here is how Discussion Tools should work:

1. You click Reply and enter your comment.
2. Discussion Tools should insert your comment-text (with indents added) into the page-text, and save the page.

Simple, right? Any programmer knows that should never result in content corruption. Instead Discussion Tools is designed like this:

1. You click Reply and enter your comment.
2. Discussion Tools uses VisualEditor's Parsoid engine to translate the wikitext page content into something that isn't wikitext. (I believe it is called HTMLRDFa). It does this even if you explicitly avoid VisualEditor.
3. It inserts your comment into the non-wikitext translation of the page.
4. It (tries to) translate that back into wikitext, and the result of the round-trip-translation is saved.

In theory double-translation is supposed to return back to the original page content, and much of the time the theory works well enough that you never notice it. However the roundtrip translation is insanely complex with many hidden bugs. Below I document multiple different cases where the Discussion Tools round-trip double-translation causes content corruption to the existing page. The problem wouldn't exist if they didn't use round-trip-translation. Note an important point: The examples below are proof that that there is a systemic design flaw. There are likely hundreds, if not thousands of different examples that trigger content corruption. Arguments over the specific examples, or advocating fixing the specific examples, fail to address the systemic problem. The Flow team was never able to fix these problems, and even if the new team could magically fix each individual case there will still be new cases of corruption popping up with no foreseeable end.

Collapsed examples demonstrating Discussion Tools content corruption

Deleting a template out of the middle of a previous comment, here on EnWiki. (The template should have had a tl| but DiscussionTools shouldn't be creating confusion or damage with mystery deletions.) Mangling an existing 5 line comment onto a single line on FrWiki. BLANKING AN ENTIRE COMMENT in a different section, on KoWiki. - Appears unrelated to DiscussionTools. Moves hidden comments such as the <!--Autosigned by SineBot--> applied by Sinebot, off of an existing comment and incorrectly puts it on the ends of the new comment. Here is a diff reported from FrWiki. The </div> markup was deleted, and the 'Message envoyé' line was moved. Note that Discussion Tools deleted and moved content in an entirely different section from the section where the reply was. This is one staff member making a test reply to another staff member. Quiddity's edit should be be a one line diff, with Quiddity replying "test". Note that it made a various changes to six different lines of Whatamidoing's comment, and then proceeded to wrap both Quiddity's comment comment inside Whatamidoing's table markup. This is a related example, it should be a simple one line diff "Reply to reply 1". Instead it is a multi-page diff, grossly mangling multiple sections in a variety of ways. This test was another one-line-reply made with Discussion tools - "Reply 2." Note that Reply 2 deleted the </span> markup from the previous comment. That causes causing all current and future text to turn red, down to the end of the section. Note that the examples above document at least three completely unrelated cases that trigger corruption. They are surely just the tip of the iceberg of things that will trigger corruption.

Proposed: Discussion Tools not be deployed until it is fixed to avoid round-trip-translation content corruption, including both known and as-yet-unknown cases in this class. (This almost certainly means altering the design to avoid round-trip-translation.) Alsee (talk) 18:43, 28 October 2020 (UTC)

Update: I'm finding and adding new examples (in the collapse box), which have now escalated to DiscussionTools blanking entire comments from an entirely different section. And to reiterate, I don't want DiscussionTools eliminated (as some responses seem to believe) - I'm desperately trying to get it fixed. Alsee (talk) 07:52, 30 October 2020 (UTC)

Discussion Tools RFC responses

• Support as nom. They should simply insert the new comment without a needless Rube Goldberg round-trip-translation. It is inexcusable that the Foundation has such gross disregard for fixable wikitext corruption. Alsee (talk) 18:43, 28 October 2020 (UTC)
• Support the need to fix this fundamental flaw before deploying this tool. Barring that, local WP instances should have the option to disable it until it is fixed. Developers might complain that these are edge cases, examples of GIGO, but garbage exists, and the Wikimedia software does not prevent it from existing. The reply tool should simply post an indented (LISTGAP-compliant, presumably) reply, regardless of the text around it. If it can't do a simple reply for some reason, it should emit an error message asking the editor to reply using regular editing. I would love to trust the developers to deploy a mostly working tool and then diligently fix bugs, but as we have seen with the still-in-beta Visual Editor, well-described bugs that damage wikitext and cause headaches for gnomes are not being addressed (e.g. three-year-old T162291; there are plenty more of those). Developers have moved on to the next shiny thing. – Jonesey95 (talk) 21:24, 28 October 2020 (UTC)
• Oppose, as the benefits of Discussion Tools clearly outweigh the issues. The upside is large: editors not familiar with markup will be better able to participate in talk page discussions, a critical part of editing. The bugs are, as the discussion below documents, rare. I have been using the 2017 wikitext editor, which also uses Parsoid, for a couple years now, in mainspace and on discussion pages, and have never run into a Parsoid parsing bug. "Just go back and re-architecture the entire project when it's already nearly complete to prevent some occasional bugs" isn't a reasonable demand to make. Vahurzpu (talk) 00:08, 29 October 2020 (UTC)

  (edit conflict) The 2017 wikitext editor does not use Parsoid in the way that DiscussionTools does. Making an edit to an un-Parsoid-parsable page using the 2017 wikitext editor does not make any changes to the wikitext other than those you typed in, whereas replying to a comment on such a page does. * Pppery * _{it has begun...} 02:00, 29 October 2020 (UTC)

• Support Alsee's proposal: alter the design to avoid round-trip-translation. "Something must be done. This is something. Therefore this must be done." --Guy Macon (talk) 01:41, 29 October 2020 (UTC)
• Question: Is there some way I can cut and paste the Wikitext from some random page, send it through the dual conversion, and see the result? I would ask that the result show up in some sort of sandbox with no ability to save, not in the edit window where a single click might publish the undead thing. --Guy Macon (talk) 01:41, 29 October 2020 (UTC)

  @Guy Macon, mw:Parsoid/Round-trip testing has information on how to do that. In this particular context, you could also just use the Reply tool for a while (e.g., in a Sandbox page) and see what happens, or look at https://dtcheck.toolforge.org/dtstats.html and see what's actually happened during the last ~16,500 real-world uses. Whatamidoing (WMF) (talk) 06:47, 29 October 2020 (UTC)

  Guy Macon, User:Enterprisey/parsoid-round-trip, hot off the press. Enterprisey (talk!) 07:16, 29 October 2020 (UTC)

• Oppose. I'm not thrilled with Parsoid, and I'm particularly not thrilled with HTML/RDFa. As a practical matter, however, both are here to stay. They are part of the core wikimedia architecture that exists today. Likewise, the model of "convert wikitext to HTML/RDFa, manipulate it in that domain, then convert back to wikitext" is not going away; it's part and parcel of the whole Parsoid architecture. To demand that tools be designed to avoid the round-trip translation is tilting at windmills.

Wikitext is a disaster. I'm not sure if it has a formal grammar, but if it does, it's absurd, allowing constructs which make no sense to anybody who has ever tried to write a parser. So we've got multiple parsers, each of which is broken in a different way. And, with wikitext being hand-generated, it's bound to be written badly. There's tons of unparsable wikitext in the wild right now. If it gets mutated during a round-trip, well, garbarge-in, garbage-out. The more we get away from hand-written wikitext, the better off we are. If there's some bumps in the road, that's the price we have to pay. Parsoid is a line in the sand. It may have bugs, but there's one place they need to get fixed. And HTML/RDFa, despite its problems (I'll be happy to rant about that in another forum), is at least a fixed target with a real grammar against which to write tools. — Preceding unsigned comment added by RoySmith (talk • contribs) 01:55, 29 October 2020 (UTC)

• @RoySmith: Actually, Parsoid has a PEG tokenizer based on several thousand parsing IO tests (I want to say on the order of 10-20k). See mw:Parsoid/Internals#Architecture. --Izno (talk) 03:24, 29 October 2020 (UTC)

  Izno, That may be true. But surely any grammar that allows:

  This is italic and also bold and now just bold.

  is just farshtunken. -- RoySmith (talk) 21:46, 29 October 2020 (UTC)

  I didn't claim that the grammar was sane. :) --Izno (talk) 23:34, 29 October 2020 (UTC)

• (edit conflict) I would prefer using the edit filter to block edits made using DiscussionTools which corrupt the wikitext, but the edit filter isn't currently given enough information to tell whether an edit is made using DiscussionTools, and this proposal does present a serious problem, so I have to support the current suggestion as an interim measure. * Pppery * _{it has begun...} 02:00, 29 October 2020 (UTC)

  @Pppery, I hope you'll try out the Reply tool yourself. In the meantime, here are all of the dirty diffs at enwiki from the last seven days:  [7] and [8]. They remove a single stray space from the end of a line (spaces that do not show when you are reading the page, because the old PHP parser drops them). There have been some serious bugs in the past (e.g., this bug, which was fixed months ago), but I would classify removing a stray space as a minor problem. Whatamidoing (WMF) (talk) 06:32, 29 October 2020 (UTC)

• Oppose The benefit of Discussion Tools is that more casual editors will be able to engage in talk page discussions by breaking down technical barriers for new participants. The harm is that bad wikitext continues to be bad wikitext. If you want a different parser, write a different parser, but I'm not going to support keeping our talk pages inaccessible just because Special:LintErrors exists. — Wug·a·po·des​ 02:26, 29 October 2020 (UTC)

   Developer note FYI there are far more barriers that just writing a different parser - buy in from the WMF for one, but more importantly any existing quirk in wikitext would either need to be kept, which hinders how "different" the other parser would be, or removed, which would require consensus (devs, community, or both?) and effort to both decide, clean up existing uses that would be broken, etc. A different parser would only compound the issues of the differences between the current parser used to render everything and the parser Parsoid uses. DannyS712 (talk) 02:38, 29 October 2020 (UTC)

  I was suspicious that a new parser would be a miracle solution, so thanks for clarifying the design considerations. As an aside, this is my first time using the reply button (thanks for the link @Whatamidoing (WMF)), and seeing it with my own eyes just convinces me more that this would be a boon for editor recruitment and engagement. — Wug·a·po·des​ 02:53, 29 October 2020 (UTC)

  @Wugapodes, thanks for the ping. The three Wikipedias that tested it and then asked for it to be turned on by default (and the Editing team actually agreed to so far; there are a couple of requests pending but not approved yet) are all working with the Growth team on newcomer retention. However, the initial analytics results aren't back yet, and this probably requires an A/B test (planned for November or December), so I can't tell you whether it's actually having the effect you (and I) expect. Whatamidoing (WMF) (talk) 06:04, 29 October 2020 (UTC)

• Oppose this feels like nitpicking over edge cases. I'm using the tool right now (per instructions of Whatamidoing below in js file), I think it's quite neat. I haven't ran into any issues so far, and it feels like a good tool. This whole "enwiki doesn't subscribe to anything new developed" is a bit meh. And if there are bugs, it'll be like any other software with bugs, and I'm sure they'll fix it. Also + Roy above and Izno below. ProcrastinatingReader (talk) 10:22, 29 October 2020 (UTC)
  • Having used this for almost a week now, I'm even more strongly opposed to this. I think this is a well built tool that will facilitate better talk page discussions. This should go live as planned. Reply-link is great, but it wasn't enabled by default and every now and then there's errors on trying to reply; this tool I haven't had such an error with yet. I have not seen any problematic replies in my usage. Very rare, exceptional GIGO cases are totally not a good reason to hold this. This seems stable for production usage, and it seems like it's been thoroughly tested for edge cases, which is reassuring to me as well. Large benefits here which outweigh any small 'errors', and we've got few common usage (ie not-(stress)test) examples of errors here. ProcrastinatingReader (talk) 19:09, 4 November 2020 (UTC)
• Oppose per the data below provided by PPelberg (WMF). A 1% error rate that is still being examined is, I have to say even looking at this page, a lower error rate than is generated by editors in normal editing. I'm guessing as I continue to use the tool I will find more issues but knowing that the "bad code" problems highlighted here are being tracked, and are at such a low rate, is enough to reassure me. Best, Barkeep49 (talk) 01:09, 30 October 2020 (UTC)
• Support per Jonesey95 above and below. Post-release it’s too easy to focus on the next project and not fix the issues here. Cavalryman (talk) 19:58, 1 November 2020 (UTC).
• Oppose: Properly participating in Wikipedia discussions currently requires learning a strange syntax; the new tool improves accessibility for those deterred by this first hurdle. This especially includes users who already constructively participate using VisualEditor, which is unavailable on talk pages. Making it easier for users to discuss conflicts will probably reduce the amount of unnecessary edit wars and the amount of "discussions" occurring in edit summaries of reverts instead of talk pages. No software will ever be free of bugs, and rejecting the tool for the rare occurrence of fixable bugs would be a mistake. ~ ToBeFree (talk) 21:31, 1 November 2020 (UTC)
• Oppose - While the new discussion tools aren't perfect, they seem to be a substantial improvement in the user experience, even for seasoned editors. The few parsing inconsistencies that exist would be rare edge cases for talk page discussions. Kaldari (talk) 17:09, 4 November 2020 (UTC)

RfC Discussion

• @Alsee:, do you have a diff where developers have indicated they're not going to fix this issue? Like you I'm in favor of the goals of this project. I would love for the enwiki community to be able to work collaboratively with the foundation rather than in an us vs them manner. That might not be possible but I worry that the outcome of RfC being successful will be that the foundation writes of enwiki in terms of how it develops this and so we miss out on the value altogether. Frwiki is hardly a small wiki; has there been community discussion about the relative benefits to current drawbacks of the implementation there? Best, Barkeep49 (talk) 21:56, 28 October 2020 (UTC)

  @Barkeep49, I think the answer to your question would depend upon what you call "this issue".  I'm not so sure about these claims.  

  • In Alsee's first diff, Parsoid automagically resolved a stripped tag error on a talk page about six weeks ago.  Resolving Special:LintErrors is generally considered a good thing to do. Moving the blank line around the hidden HTML comment is less desirable, and I believe there's a bug report for this (it might be in the list that will get resolved this WP:THURSDAY).
  • In the second diff, my teammate and I were testing a bug report from Alsee at the mw:Beta Cluster testing site back in February.  This was early-stage testing, months before the tool was offered as a Beta Feature to any community.  This page was corrupted when a wikitext table had all of its formatting syntax with ::: inserted in front of it.  (Wikitext tables require the table formatting codes, such as |-, to be the first characters on the line.) Broken wikitext is expected to produce broken pages, especially when we're talking about super-early stages of software.  To make it harder to create this problem accidentally, the Reply tool hasn't let editors insert tables or templates for months now, unless you put the wikitext in yourself. A longer-term solution requires a significant technical change to wikitext, which may be discussed in the coming months.
  • In the third diff, Alsee ran a test with this content:  {#if:x|<span style="color:red;|<span style="}} font-weight: bold">This should be red 2.</span>.  I have seen Alsee post this string as a deliberate 'stress test' several times on at least three wikis now, but I've never seen an editor type anything like that in a real comment.   There's a bug report for this in Phab, and I understand that it's priority is set to reflect its rarity.

  These diffs are not representative of the everyday experience of using the Reply tool.  If you'd like to see real-world diffs, you can check RecentChanges here at enwiki.  

  You can also look at this table of diffs by wiki, if you want a quick way to find diffs that are more likely to be broken in some way.  The problem rate is low and falling.  Most days, about one out of 150 Reply comments have a whitespace change, and another 1 in 150 have a non-whitespace change.  (Fun fact:  Parsoid isn't the only software that causes 'dirty diffs' on talk pages.  WP:WikEd does, too.)

  If you're interested in why both the Reply tool and User:Enterprisey/reply-link.js are using Parsoid to place comments, then I can tell you more about that, but I think you might be better off trying out the Reply tool for yourself. Unless someone deliberately corrupts this page, then you can try it out right here. Click on https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(WMF)&dtenable=1 and look for the little [reply] links at the end of each signature. I recommend clicking over to the "Visual" section and using the toolbar to ping me or to add a link to your reply. Imagine a world in which you never have to remember how to spell or correctly capitalize editors' usernames. :-D Whatamidoing (WMF) (talk) 22:17, 28 October 2020 (UTC)

  Although I'll give you a hint now: @Izno replied while I was typing, and I didn't have to deal with the edit conflict that would have triggered in any of the regular wikitext editors. Whatamidoing (WMF) (talk) 22:19, 28 October 2020 (UTC)

  Whatamidoing (WMF), I'm not sure how much of your reply is really meant for me and how much is a "I'm going to reply to this comment so I can say everything I want to." Going off the idea that it's at least actually partially directed at me, I have tried the tool. I found it impressive at the early stage I tried it and I found the foundation responsive to the concerns I did raise - more responsive than I'd have actually expected. This is partly why I asked Alsee the questions that I did; I wanted to make sure I wasn't missing something/ Best, Barkeep49 (talk) 22:40, 28 October 2020 (UTC)

  @Barkeep49, if you copy the first happy lines from m:User:Whatamidoing_(WMF)/global.js (this diff), then you can use it all the time. I haven't tried out Enterprisey's reply-links myself, but it's obviously popular, especially with the improvements he made a few months ago. Maybe someday you can tell me how the two compare. Whatamidoing (WMF) (talk) 22:51, 28 October 2020 (UTC)

  Whatamidoing (WMF), thanks for the links to these diagnostic tools. Do similar tools exist for VE while it is still (I hope) being debugged? Here is an example of a broken edit from this tool. I don't know the rules on fr.WP, so I don't know if this was technically a GIGO edit, but it looks like an editor unwittingly altered the formatting of another editor's comment in a significant way, which is usually unacceptable. Given the small scale on which this tool is currently deployed, and based on my experience with stale VE bugs, I worry that multiplying the frequency of this sort of edit by 100 or 1,000 will lead to a lot of problems that developers will dump on gnomes to fix after they have moved on to something more fun than fixing tricky bugs. – Jonesey95 (talk) 02:53, 29 October 2020 (UTC)

  @Jonesey95, what do you mean by "significant way"? It stripped out a line containing only colons between two comments by the same editor. That editor wrote a comment with four colons, then added a blank line with three colons, and then wrote another comment with four colons immediately afterwards. Under our rules about WP:LISTGAPs, any editor would have been justified in removing those stray colons manually.

  The Editing team is working with the Parsing team to reduce the "dirty diffs". The current rate is low and falling. I encourage you to try it out and see what you think about it yourself. Whatamidoing (WMF) (talk) 06:16, 29 October 2020 (UTC)

  Barkeep49, you asked if there's a diff where developers have indicated they're not going to fix this issue. I think this link will do: Phab post by the project manager. Context: The manager is discussing just one of the symptom cases (the tables example). He states they aren't even going to try to fix that case. Their "fix" to avoid corrupting the page (for that particular problem case) is to detect that case and then fully break the REPLY button. The REPLY button won't corrupt the page if the button refuses to let the user post any reply at all. I reported this issue directly to the manager seven months ago. That post from him is exactly one month ago. If they're not fixing that specific case, they clearly don't intend to fix the underlying cause. Alsee (talk) 23:10, 29 October 2020 (UTC)

• As I've said at least once, this proposal is DOA. The technology stack powering DiscussionTools is coming whether you want it or not. I would invite other editors to read through that thread and the other forked thread that Alsee disruptively started twice to try to kill the implementation, as well as the responses from just about everyone else (to wit, the developers have worked issues identified where it is the tool that is at fault and asked for further specific details; Alsee failed then to provide them, so I guess I'm happy there are 3 observed questionable problems above).

  Further, in almost all cases I've seen where round-tripping has caused an actual corruption rather than a suboptimal edit, it has been the fault of frankly bad wikitext. --Izno (talk) 22:06, 28 October 2020 (UTC)

• This seems out of our scope. Software back-end is the domain of the MediaWiki.org community, and WMF-wide configuration policies are set on Meta. The most we can do is decided whether to enable or not enable Discussion Tools on EnWiki. — Wug·a·po·des​ 02:14, 29 October 2020 (UTC)

  If you all decide that you'd like to test this as an opt-in-only Beta Feature, then I can make the request for you. I do not know what the response would be. Whatamidoing (WMF) (talk) 06:42, 29 October 2020 (UTC)

  I think it would be useful. For an opt-in beta feature, I do not think we need a formal RfC or anything similar.--Ymblanter (talk) 07:32, 29 October 2020 (UTC)

  I for one would prefer going ahead with the current rollout scheme across all wikis unless there is consensus to do something different. It seems enwiki is excluded in phase 3 in T266303; what phab task is tracking the deploy to enwiki? Is it roughly known when will it be deployed, and will it be a beta feature at that point or is it enabled-for-all, under the current plans? Unrelated, it seems sometimes the "Reply" button sends me to the top of the page rather than to the comment I've just posted (I have to scroll back down manually). ProcrastinatingReader (talk) 12:12, 29 October 2020 (UTC)

• I've only tried in one niche discussion, but is this tool able to distinguish between * and :? eg in Special:Diff/986048398 it added a : when most of the replies were bullets (it also added a space). This was when I was responding to the main comment, btw. Side feature request: to add a reply to this section I have to edit source. Not sure if it's easily possible, but in the long run maybe some kind of "Add reply" functionality when replying to a section/main discussion rather than an individual user? It may be more difficult to achieve, if at all, I know. ProcrastinatingReader (talk) 14:02, 29 October 2020 (UTC)

  It does know about list types, but it uses that only for when you're replying to something that's already in a list. I.e. it'll sensibly reply to an * comment with * and a : comment with :. When you're just replying to a top-level section (which isn't in any sort of list), it'll default to :. I took a look at the replies where you were, and they're already super-mixed -- in the HTML output there's 5 dl and 4 ul lists, so it's a bit of a mess, and one could argue that even if it was trying to guess based on overall usage it'd have landed on : for that reply.

  The reply links trigger off of finding a block of text with a signature, which is why replying to this section-as-a-whole isn't present. I do agree it'd be helpful if it better supported non-described sections, even if they're less common... DLynch (WMF) (talk) 17:19, 29 October 2020 (UTC)

  Oops, sorry, I gave outdated information. We matched the list type up until about a month ago, when we started replying to everything with :. We decided that we’re still evaluating how best to choose which indentation type should be used for replies... so having a half-done selective approach in place was probably worse than a consistent one while we’re thinking about it.DLynch (WMF) (talk) 19:20, 29 October 2020 (UTC)

  @DLynch (WMF) thanks for this. That's fine, was just curious. BTW, the preview on "Source" actually shows bullets. Every new line is an additional bullet. I actually kinda liked that, but that's not how it saves, so it feels a bit misleading to show it in preview. But otherwise, great work on this reply stuff to all devs/others involved! It's quite cool! Hopefully it encourages talk page participation even more. ProcrastinatingReader (talk) 19:03, 4 November 2020 (UTC)

  Oh hey, thanks for pointing that out! We hadn't noticed the preview issue -- it's a side-effect of forgetting to remove the list-type inference there when we did everything else. A fix for that has been committed and will work its way out in the next deploy cycle. DLynch (WMF) (talk) 16:34, 5 November 2020 (UTC)

• As you all consider this, I thought it would be helpful to do two things:

  1. Share how the Editing Team is thinking about and addressing the risk that edits made with the DiscussionTools make changes to talk pages other than adding content.

  2. Address the statements and diffs in the original post.

What follows isn't exactly "bite-sized." Tho, I thought thoroughness would be worthwhile in this context. If anything here prompts new thoughts or questions, please let us know...

• 1. The potential for DiscussionTools to impact other parts of the page.

  • The core of this proposal is the need for edits made with DiscussionTools to not affect talk pages beyond adding the content people are posting with it. We relate to this proposal in so far as we, too, see it as a requirement that edits made with DiscussionTools should not impact other parts of talk pages, especially in ways that make them more difficult to edit and understand.
  • As Whatamidoing (WMF)  mentioned above, to help us all ensure the tool is working in ways we expect, we've created a way to detect when edits made with DiscussionTools do anything beyond adding new content to talk pages. You can play around with it here: https://dtcheck.toolforge.org/dtstats.html.

  • To date, the tool has analyzed over 16,500 comments.
  • In the past 5 days, an average of ~1.30% of comments do something other than add new content to talk pages. In the past 30 days, an average of ~1.20% of edits made with the Reply Tool have changed other parts of the talk page in some way.
  • Note: The most common change we see is space characters being removed at the beginning or end of comments. We're working to eliminate these as well.

  • Since we started tracking this kind of edits on 3-September, we've seen the 5-day average "dirty diff rate" fall 83%, from ~6% → ~1%.

  2. Statements made in proposal

  Discussion Tools round-trip double-translation causes content corruption to the existing page

  In principle, this statement is true: edits made with the Reply Tool have changed other parts of the page. Although, evidence suggests (see: https://dtcheck.toolforge.org/), the incident rate of these "changes" is falling, and we now have a mechanism to proactively detect when these changes occur.

  The problem wouldn't exist if they didn't use round-trip-translation

  Some context as to why we have been using Parsoid to save comments made with Discussion Tools: "[Parsoid] makes it possible for the software to work out the structure of the page and thus more reliably insert replies in the correct position on the page. Secondarily, by sending talk page content through Parsoid, it enables the reply tool to work on transcluded pages." More context here: https://w.wiki/jGS.

  ...examples demonstrating Discussion Tools content corruption

  Example 1 (fr.wiki)

  • This is an issue we're tracking and talking about a fix for. You can track the progress on this work on Phabricator here: T262448.

  • Note: Matma Rex shared some helpful context about the impact and cause of this category of issues in T262448#6499606, "Many of the diffs we're seeing look really bad because they remove closing tags that actually seem to be matched to an opening tag at a glance, and are only unmatched because of incorrect nesting resulting in the relevant node being implicitly closed earlier."

  Example 2.1 & Example 2.2

  • The above seem to be cases where an edit is made with the Reply Tool on a page that contains broken/incomplete table syntax. As of April 2020, the tool is disabled on most affected pages (T246481). We think this protection might not have worked as expected in this case because of the speed with which these edits were made. See: T263709#6502328.
  • As shown in the second example, it’s still possible to intentionally avoid that workaround, e.g. by opening the reply tool in one browser tab, and then adding the broken syntax to the page in another; or posting multiple comments very quickly. Fixing this is possible, but would cause posting replies using the tool to be slower in the common case where the page isn’t affected by this bug. We investigated this issue back in September and decided it would be safe to not fix this considering, what we see as, the low likelihood of this occurring in "real" usage (T263709#6503764).
  • Long-term, the solution to this problem requires extending wikitext, so that corrupted syntax can be isolated from the rest of the page.  We are planning a technical RFC to do this. You can track the progress on this work in Phabricator here:  T230683.

  Example 3

  • This seems to be similar to the issue we talked about earlier this year.Topic:Vcwvt3bq03o5gv8h
  • As a result of that interaction, the Editing Team decided not to address this issue considering:

  • There are Parsoid-compatible ways of achieving the same intended effect.
  • We are not currently aware of any instances where the syntax you are using here has been used on live talk pages.

  • Technically, this is the same issue as T262448 (unmatched closing tag), and may get resolved along with that.

  There are likely hundreds, if not thousands of different examples that trigger content corruption.

  • As noted above, to date, the tool has analyzed over 16,500 comments. Over the past 30 days, ~1.20% of edits made with the Reply Tool have changed other parts of the talk page in some way.
  • We'd value you having a look at the kinds of changes contained in this "~1.20%" metric by inspecting the diffs we're tracking. If you see diffs that you think involve serious problems, please let us know here, in Phabricator or by commenting on the tool's talk page at mw.org: Talk:Talk pages project/replying. Here is a link to the tool you can use to review all edits made with the Reply Tool:   https://dtcheck.toolforge.org/dtstats.html. PPelberg (WMF) (talk) 00:53, 30 October 2020 (UTC)

  Thanks @PPelberg (WMF), this is helpful context. I'll just note that one kind of "reply" I can't do yet (as I've installed the tool per @Whatamidoing (WMF)'s invitation above) is I cannot participate in the response part of the RfC. Is that in scope for this project? Best, Barkeep49 (talk) 01:07, 30 October 2020 (UTC)

  I'll just note that one kind of "reply" I can't do yet...

  To make sure, we're on the same page, in stating the above are you referring to being able to use the Reply Tool, or something like it, to submit a vote (e.g. Support or Oppose) in the ===Discussion Tools RFC responses=== section?

  If yes, then this is a use case we've started to think about (see: phab:T259865); I've also added the use case I think you are highlighting here to the task's description (see: phab:T259865#6590520).

  {Is that in scope for this project?

  I think the safest answer would be to say "no" in so far we do not yet have specific date in mind for when we will work on supporting this use case. This is not to say we do not think it's important or we won't end up getting to it.

  Rather, we think getting the "communication primitives" (e.g. responding on discussion-style conversations and adding new topics) to a place where people who are newer to the wikis can communicate with others instinctively and ways that comply with community conventions should take precedent. You can see a bit about how we're thinking about the project in this update: Talk_pages_project/Updates#4_Sep_2020.

  ...this is helpful context

  Oh good. If any other questions come up, please do let me or anyone else know. We're striving to make the work around this project as legible as possible. PPelberg (WMF) (talk) 02:00, 30 October 2020 (UTC)

  PPelberg (WMF) I remain mystified at your apparently intense desire to not fix it. If you inserted the new comment into the original page content, all of these problems would be solved in a single stroke. Instead you're expending work chasing after a variety of symptoms, in some cases you're "fixing" it by fully breaking the reply button so the user can't reply at all,[9] you are going to have to chase after all of the new cases that crop up, and then you're leaving us screwed after your immediate project wraps up. Even if you manage to fix today's list of examples, tomorrow's corruption cases are never going to be fixed. We're supposed to file phab tasks on them, and these kind of tasks just vanish into the infinite-backlog on phab after the immediate project closes. Alsee (talk) 06:13, 30 October 2020 (UTC)

  PPelberg (WMF) I just found that DiscussionTools blanked an entire comment in a completely different section![10] How is that not a screaming showstopper?!?!? Alsee (talk) 07:46, 30 October 2020 (UTC)

  I just found that DiscussionTools blanked an entire comment in a completely different section.

  This diff was unsettling to us as well – thank you for linking to it, @Alsee. We investigated this diff when we noticed it ~2 months ago.

  The result of that investigation: the bug that caused this concerning edit was not caused by DiscussionTools, but rather by MediaWiki failing to handle edit conflicts when the replica databases were out of sync. This blanking would've happened regardless of what editing interface/tool someone used.

  So folks are aware: we fixed the underlying issue that caused this problematic diff in T261715.

  Note: you'll see the problematic diff happened on 6-September-2020 and the patch to resolve it was committed on 1-September-2020. The reason that patch did not prevent this particular issue is because the patch landed on ko.wiki (a Group 2 wiki) on Thursday, 10-September, four days after the edit happened. PPelberg (WMF) (talk) 15:33, 30 October 2020 (UTC)

  @PPelberg (WMF) that is the context I was talking about. Thanks and best, Barkeep49 (talk) 14:47, 30 October 2020 (UTC)

  Roger that, okay. Thank you for confirming, @Barkeep49. PPelberg (WMF) (talk) 15:35, 30 October 2020 (UTC)

  PPelberg (WMF) I have stuck that one from the example list. With that one out of the way, could I get a response to my comment (06:13 , 30 October)? There's now something like a dozen known independent cases where DiscussionTools will randomly corrupt the existing page (not all included in my example list). Those cases will randomly be serious or "minor". New ones are found week after week, month after month, and will continue year after year. Alsee (talk) 19:30, 30 October 2020 (UTC)

  ...could I get a response to my comment (06:13 , 30 October)?

  For sure. Although, can you please point me to the specific point within the comment you posted that you'd value a response to? I'd been thinking I'd responded to that post, tho, you may be seeing something I'm not. PPelberg (WMF) (talk) 01:03, 31 October 2020 (UTC)

• @PPelberg (WMF):, I realise that you don't just set a "if we reach x% of edits having a bug, we ship", but you've impressively dropped the bug rate thus far (80% or so, I believe you said). Notwithstanding any substantial changes to the timeline, how much more time is intended in pre-(broader)-release for squashing the blighters? I'm sort of ambivalent at the moment, mainly because of the increased possibility of the tool to have errors outside of the reply itself, which are both harder to detect, more problematic, and if not detected immediately, more fiddly to fix manually. But the progress made seems positive (I am not qualified to weigh in on the pros and cons of not directly using source as the proposal advises), so I'm more on the general tool. Nosebagbear (talk) 10:26, 3 November 2020 (UTC)

  hi @Nosebagbear – thank you for this question. A couple of comments in response below...

  ...how much more time is intended in pre-(broader)-release for squashing the blighters?

  Assuming you are referring to how much more time do I anticipate us needing before we'd consider offering the Reply Tool as an opt-in Beta Feature at en.wiki, I can share some of what we have planned before doing this:

  • Listening for what consensus emerges in this RfC.
  • Making the Reply Tool available as a Beta Feature at an additional ~250 wikis to ensure it continues to work in ways people expect at a broad range of wikis. See phab:T266303.
  • Monitoring how the latest round of backend improvements (see: Phab:T262408) impact the rate at which edits made with DiscussionTools do anything other than add new content to pages.
  • Talking with people who have used the Reply Tool at en.wiki using the script @Whatamidoing (WMF) shared in this comment to understand the extent to which the tool is working as expected.

  ...I appreciate you likely asked the above seeking an estimated date or number of weeks/months before the tool is offered here. Tho, considering the unknowns involved with them, it's hard to offer that at this time. Of course, when we have a better sense of time, we will make sure it is widely known.

  ...the increased possibility of the tool to have errors outside of the reply itself, which are both harder to detect, more problematic...

  Had you and I been having this conversation in August, I would've agreed with you about it being harder for us to detect, at scale, when edits made with the Reply Tool do anything other than adding new content to pages. Although, we now have a tool that enables us (you, everyone in this conversation, the Editing Team, etc.) to detect this category of errors. If you are interested in exploring this for yourself, you might check the tool's "dirty diff detection" results for yesterday, 3-Nov-2020: https://dtcheck.toolforge.org/dtcheck-2020-11-03.html.

  Please let me know if anything here prompts new thoughts or questions. PPelberg (WMF) (talk) 17:08, 4 November 2020 (UTC)

• I think this is a bug. It's adding a new line between what it's replying to. I believe this fails enwiki's MOS:LISTGAP. ProcrastinatingReader (talk) 02:05, 5 November 2020 (UTC)

  Nope, a new line there is fine since it is not a list being replied to. --Izno (talk) 04:02, 5 November 2020 (UTC)

• WMF devs, another thing: I was replying to a discussion and it was closed between the time I started typing and clicked submit. After doing this, the page refreshed and my entire reply was lost (it was not added as an edit). I think this behaviour is undesirable; an error should be shown or the opportunity to save ones reply. ProcrastinatingReader (talk) 20:40, 16 November 2020 (UTC)

Some musing about software engineering

Every piece of software has bugs. Deciding when a piece of software is ready to release is a matter of weighing the value of getting it shipped vs the severity of the known bugs (plus the risk exposure of the bugs you haven't found yet). At some point, you decide that it's good enough, and ship it. Then you have a big release party, celebrate your accomplishments, and hold your breath until the first wave of bug reports come in, hoping that none are too serious.

There's no such thing as perfect. If your release criteria is that you've eliminated every bug, "both known and as-yet-unknown cases", you'll never ship. And then the value of the software is zero.

Where that balance falls depends on what you're writing. If your program's job is to detect that a 737 is about to stall, the consequence of getting it wrong is that hundreds of people die. If your program's job is to insert a comment in a talk page, the consequence is that somebody has to revert the edit. So, clearly, the release criteria are different. To insist that it not be shipped because there are known bugs is absurd. -- RoySmith (talk) 21:40, 30 October 2020 (UTC)

That's a great philosophy, but it implies that developers will be willing to fix bugs that are reported post-release. The WMF developers have a spotty track record on doing so (e.g. the three-year-old T162291 Visual Editor bug; there are plenty more of those, and pleny that are much older). Once bitten, twice shy. – Jonesey95 (talk) 22:07, 30 October 2020 (UTC)

Jonesey95, Just like it's absurd to think software will only get released when there's zero bugs, it's absurd to think every reported bug will get fixed. There's a finite amount of developer time. Every day somebody spends working on one thing means there's something else that didn't get worked on.

I'm not privy to how WMF does things internally, but typically each bug gets scored based on how bad the problem is, how much value there will be in fixing it, and an estimate of how much effort will be required to fix it. Not unlike our importance/quality scoring system for articles. Or our plethora of "article needs improvement" templates. Surely adding a {{Citation needed}} template to an article is tantamount to filing a bug against it. We've currently got 412,324 of those. We should stop writing new articles until we get all of those fixed, right?

I see that T162291 was triaged as low priority. I gather from your comments on the phab ticket that you disagree. Such is life, people don't always agree on how bugs get prioritized, just like editors don't always agree on what edits should be made to an article. -- RoySmith (talk) 01:47, 31 October 2020 (UTC)

I'll just add to what Roy wrote that, assuming the data offered by PPelberg is correct (and I have zero reason to distrust it), the current error rate is for this tool is in the ballpark, if not less, than our manual error rate. Like I know I did a whole bunch of reply indenting wrong before (using :::: when I should have been using *::: for instance) and if this tool can get that correct then our fellow editors (in this case primarily those who need screen readers) benefits. Best, Barkeep49 (talk) 01:57, 31 October 2020 (UTC)

@Barkeep49: One reason I have some concern is that the current set-up is generating bugs that cause issues outside of the posted reply. This has a couple of effects on your's and Roy's comments above. Issues (other than ones that completely bork the whole page) are much more likely to be missed, so the error rate can be both more substantive than the manual rate but also be harder to fix. If the tool damages a post elsewhere (say the one it's replying to, which is most likely), and the poster doesn't notice, it's quite possible several posts will come in before someone notices. As with pending changes, fixing it in that circumstance becomes way more fiddly. So it's not quite an "apples to apples" consideration Nosebagbear (talk) 10:20, 3 November 2020 (UTC)

What kind of error impacts how I feel about it happening. Erasing other's comments is a big concern if it's happening 1% of the time. Some of the other errors shown don't really bother me if it's at a 1% rate. But correctly writing syntax that would otherwise have had errors if be made by a human edit is the dog that doesn't bark and it's why I mention it. And some of those human made errors, as in my example of using :::: when it should have been *::: are likely never fixed. So I am just as concerned about the errors this tool fixes as the errors it introduces. Best, Barkeep49 (talk) 16:48, 3 November 2020 (UTC)

Indeed, by the end of the work, I and others will not need to fix WP:LISTGAP issues on talk pages. --Izno (talk) 17:10, 3 November 2020 (UTC)

RoySmith I don't think your comment resembles the current issue. I am not asking them to fix multiple bugs. In fact I want them to stop wasting time and resources fixing these bugs. Insert the new comment into the page source and the number of translation-corruption bugs becomes zero. Fix the design and the time spent hunting and fixing this class of bugs becomes zero. Alsee (talk) 07:42, 31 October 2020 (UTC)

@Alsee and RoySmith: This right here is the correct solution, and would solve almost every issue I currently have with this system. IIRC, this is what reply-link (which I'm writing this comment with) already does, and it works fine in 99% of cases. In that remaining 1%, it instead errors out and fails to post instead of corrupting the rest of the talk page. Someone will probably say "but making new editors have to go into wikitext mode if there's a problem is Bad™" but that's simply the only option available. Community consensus after Flow imploded was that any further attempts to build new discussion features must be done exclusively as an extension to wikitext talk pages and integrate seamlessly. Unless the WMF somehow gets consensus to throw out wikitext entirely (yeah, that's not happening), new discussion features must not break things for editors using plain wikitext exclusively. Internally to the WMF, this should be looked at the same way Linus Torvalds looks at the "kernel updates must never break userspace software" policy, i.e. an immutable rule that can never be broken under any circumstances (with the sole possible exception of extremely critical security fixes).

Luckily, the WMF's developers have an extremely simple solution to this issue: don't commit back any of the wikitext that was pulled from the page. Visual Editor doesn't really have a choice here, but Discussion Tools does. Pull the wikitext and run it through the parser for previews if you want to, but only actually convert back and commit the added wikitext from the new reply. Congratulations, you've just fixed every single potential issue with this system in one fell swoop. In any event, per WP:TPO this should really be the only acceptable course of action. Automated tools should never be allowed to touch existing comments (beyond archiving, of course, but even then that's moving full sections to a different page without modifying the wikitext contained within) without strong consensus that it's a good and necessary decision that won't break anything. Nathan2055^{talk - contribs} 04:13, 5 November 2020 (UTC)

Wikimedia Login Wiki

What is the interwiki link for Wikimedia Login Wiki? --Ituafmq (talk) 20:05, 31 October 2020 (UTC)

Looking at Special:Interwiki, I don't believe there is one. BethNaught (talk) 21:03, 31 October 2020 (UTC)

@BethNaught: But aren't there supposed to be interwiki links for all WMF projects? Or is my belief incorrect? --Ituafmq (talk) 01:00, 1 November 2020 (UTC)

There are several private/hidden/fishbowl/etc. wikis that the WMF maintains. They're not for general or external use, and my understanding is that most/all of them don't have interwiki links as no links to those wikis should ever be posted. ~ Amory (u • t • c) 12:38, 2 November 2020 (UTC)

@Amorymeltzer: Ok, thank you very much for your reply! --Ituafmq (talk) 16:01, 2 November 2020 (UTC)

Amorymeltzer, What is a "fishbowl wiki"? -- RoySmith (talk) 16:05, 2 November 2020 (UTC)

Well, meatball:FishBowl, but less confusingly: open to read, (largely) not open to write. It also gets tossed around for anything that "looks" like a little fishbowl, like some office/cubicle designs or some silly icebreaker games. In this case, anyone can "look in" from the outside, but only the "fish" are inside and participating. Truthfully, fishbowl wikis can make use of interwiki links, I just lumped them in above with the other "weird/nonstandard" options. ~ Amory (u • t • c) 16:20, 2 November 2020 (UTC)

Community Wishlist

The Community Wishlist goes live in a few days. There have been some changes again this year with the Community Tech team being more explicit about their ability to pick which projects they work on and with there being a seperate "leaderboard" for large and small projects.

I tried to be neutral in the above paragraph with the description. However, I am not a fan. When I worked with this team for changes around WP:NPP, I found them great collaborators and on the whole a pleasure to work with. However, the community has now gone from having one way to actually get some developer time for things it thinks important to zero ways. In order for a community desired change to happen it needs to pass an initial screen to be allowed onto the community wishlist. It needs to attract some level of support (but don't worry if it's not the most since it's neither a vote nor a discussion). It then needs to be selected, by whatever method the team feels like using, for research. Then the research needs to be positive. Then the development needs to be completed successfully That is, by my count, 5 different places where, no matter how much the global community desires a change, that the it could simply not happen. The fact that the team is small is not an indictment of them but of the leadership who allocates funds, including the board. But the rest of this are changes I'm dispirited to see. Best, Barkeep49 (talk) 21:55, 9 November 2020 (UTC)

@Barkeep49: For the global watchlist that was not done by the community tech team, I got a grant to create an extension to do it - once that is done if there are any other wishes that I think I could do, but the community tech team has declined (at any of those 5 stages) I can look into it DannyS712 (talk) 22:05, 9 November 2020 (UTC)

That's a wonderful offer DannyS712 and I am grateful for the work you and other volunteer devs do. However, it doesn't let the foundation off the hook. Volunteers by their very nature are less reliable than a permanent group of devs at the foundation. To use a different example, I am grateful that Enterprisey developed reply-link and my life was made better by it. However, I still the foundation's work on the discussion tool was necessary and useful. Best, Barkeep49 (talk) 22:10, 9 November 2020 (UTC)

@Barkeep49: Hello! We have discussed the feedback provided by you and other volunteers (thanks for sharing it). We have shared a response, which provides some clarification and adjustments, on the talk page. Thanks! --IFried (WMF) (talk) 01:34, 10 November 2020 (UTC)

To summarise two key points of that update, which is substantive enough to warrant a read: all non-declined backlog items will be added to the task list and when they're researching which wishlist items to do, they'll do so in order of most votes received. Nosebagbear (talk) 09:41, 10 November 2020 (UTC)

Strategy event organized by Wikimedia UK and WMF

There will be an online event [11] next week Tuesday organized specifically by Wikimedia UK and the Wikimedia Foundation for the English Wikipedia in relation to the strategy. More details are provided at the link. I have no relation to its organization, though I am planning to attend, but I would like to remark that this could be a chance to have some conversation and to break the usual circle WMF proposes something - Community does not care - WMF comes with a specific proposal - a shitstorm - WMF bacls down - nothing is done, problem not solved. We are now at the stage of "Community does not care", but the process is certainly going to affect us - and I hope this is a chance to think how it will affect us and what we can do about it.--Ymblanter (talk) 20:01, 10 November 2020 (UTC)