In T175672#3635140, @Joe wrote:

Looking at http://php.net/manual/en/mysqli.ssl-set.php, I would think you'd only need to set capath=/etc/ssl/certs, while setting all other parameters to NULL (except maybe cipher, as I have no idea what is the actual default cipherlist for mysqli on HHVM).

In T173696#3620700, @Lucas_Werkmeister_WMDE wrote:

Interesting idea! It feels a bit weird to implement logic like this on top of the cache (I thought that’s the cache’s job?), but you’re the expert :) it sounds like it makes a lot of sense, at least, since the set of regexes is mostly static and the set of values is highly dynamic, with some very commonly used values.

I think I’ll remove the “don’t bother” microtime check, though, since it seems that even for an extremely simple query like SELECT (1 AS ?x) {}, the query service rarely responds in less than 0.04 seconds, and never in less than 0.02 seconds (tested from a Cloud VPS system within the Eqiad cluster).

If want to avoid flooding cache with rarely used long-tail combinations, maybe something like this could be done:

In T176101#3614778, @Anomie wrote:

In T176101#3614184, @aaron wrote:

Problem seems to be:

if ( $this->stage <= MIGRATION_WRITE_BOTH ) {
	$fields[$this->key] = $this->lang->truncate( $comment->text, 255 );
}

...LocalFile already used addQuotes(), and this can remove the ending quote character.

That's not the real problem. The problem is that the different behavior of IDatabase->insertSelect()'s $varMap versus ->insert()'s $a wasn't noticed, so the code was incorrectly quoting the value passed to CommentStore->insert() (from the original pre-CommentStore code) rather than quoting the returned literal fields for passing into IDatabase->insertSelect().

0001-SECURITY-Fix-handling-of-CommentStore-and-insertSele.patch1 KB

Download

Problem seems to be:

if ( $this->stage <= MIGRATION_WRITE_BOTH ) {
	$fields[$this->key] = $this->lang->truncate( $comment->text, 255 );
}

Probably the onMoveTranslationUnits handler should be a closure sent to DeferredUpdates. Anything that needs to COMMIT/BEGIN within it's scope needs full transaction control, which is not usually guaranteed when some hook triggers.

• PROBLEM: in LinksUpdate, runForTitle() starting off with acquirePageLock(), then calling doUpdate() for the secondary update list, and returning without committing. This meant that any caller using this method inside a loop had to call commitMasterChanges() itself somehow, otherwise, the acquirePageLock() call would fail. The multi-title case of RefreshLinksJob had a for-loop that did not do this. Note that acquirePageLock() uses getScopedLockAndFlush() which is intended for "critical sections" (https://en.wikipedia.org/wiki/Critical_section) involving read/writing to the database. Since it makes to sense to acquire a lock and then read a stale snapshot (from REPEATABLE-READ) from *before* lock acquisition, Database demands that any transaction be cleared. It will do so automatically if there are no writes, but otherwise it fails since committing prematurely may break atomicity.
• INTRODUCTION: This was broken since 63a3911a67507731695bad3188f486219a563b7d but nothing used multi-title refreshlinks jobs. 0df49eeaf49dcd84cee5afc678de43ebd6c984c5 introduced a use case for this and made the bug manifest itself.
• AVOIDANCE: since this would seem to happen for any multi-tutle job run, I'm not sure how this got past testing unless there were (a) no links updated and the test jobs were triggered by null or non-link changed edits or (b) the edited test entities only had one backlink. Future backlink change propagation testing should cover these cases.

I'd leave it open. The above change avoids the jobqueue and thus fast tail jobs piling due to slow wikidata jobs in the head of the queue. It should help, I'd assume.

In T174993#3606993, @aaron wrote:

I think LinksUpdate for the page directly edited can probably be moved (back) to doing the actual work post-send.

The 'enqueue' parameter can be removed from MediaWiki::restInPeace() since, unlike in the PRESEND run, the user is not waiting on it to run. If a caller really wants to enqueue a job post-send, it can always use lazyPush() instead of adding an EnqueueableDataUpdate to the POSTSEND deferred update list.

I think LinksUpdate for the page directly edited can probably be moved (back) to doing the actual work post-send.

In T174993#3601614, @GWicke wrote:

As far as I can tell, the page image(s) are handled as part of deferred linksUpdate processing. This means that the updates would be executed after the main web request, but on the same PHP thread that handled the original edit request.

For catching slow queries, we can use logging to logstash when the runtime passes a certain threshold (to avoid spamming the service). A leaderboard could be added to Kibana for the top occurrences of normalized messages.

In T171071#3464513, @Marostegui wrote:

Hey,

I would be nice to do a test with MariaDB 10.0 and 10.1 if possible, to see if there are any regressions.
For that matters, on codfw you can pretty much use any slave for 10.0 (they all have pretty much the same HW), so for the sake of picking one from s1:
db2048

If you want to do the same test with MariaDB 10.1, you could use db2062. db2062 is being used lately to reclone some hosts, so you might want to give us a heads up before using it, to make sure it doesn't have mysql down for one of those maintenances.

Those refreshLInks jobs (from wikibase) are the only ones that use multiple titles per job, so they will be a lot slower (seems to be 50 pages/job) than the regular ones from MediaWiki core. That is a bit on the slow side for a run time of a non-rare job type (e.g. TMH or GWT).

In T173710#3571046, @EBernhardson wrote:

In T173710#3571009, @Legoktm wrote:

Could we always bump page_touched, but only send the purges to varnish if the timestamp is within the past four days? Would that let us run the older jobs faster since if I understand correctly the throttling is to avoid overloading varnish with purges?

Unfortunately the throttling still happens regardless of page touched. Throttling isn't based on actual purges performed but on the number of work items in a job. Work items are a simple count of pages in the job, rather than how many pages will actually be purged. Changing this behavior would basically increase the number of purges we send to varnish.

In T173710#3570037, @Joe wrote:

Correcting myself after a discussion with @ema: since we have up to 4 cache layers (at most), we should process any job with a root timestamp newer than 4 times the cache TTL cap. So anything older than 4 days should be safely discardable.

This would account for about 1% of jobs according to Gwicke's sampling, but I suspect that under large pressure the distribution could get significantly worse.

As far as retries go, the attempts hash for wikidatawiki:htmlCacheUpdate has few entries with run counts no greater than 3. The onl incrementing code is doPop() in MediaWiki, the same code that made them go up to 3 to begin with. If the same job ran many times, I'd expect there to be very high values there.

In T174422#3566350, @Krinkle wrote:

@Ladsgroup @aaron Would $wgUpdateRowsPerQuery be appropiate here, too? Or is it important for this particular query to use a different batch size?

Though this bit is problematic:

Ignored purges still count as work items, yes.

Note that for de-duplication, as long as the job has rootJobTimestamp set, it will ignore rows already touched (page_touched) to a higher/equal value, and likewise not send purges to the corresponding pages. So the CDN aspects *should* already have lots of de-duplication, the job spam notwithstanding.

In T173710#3551156, @aaron wrote:

Secondary purges where for dealing with replication lag scenarios, not lost purges. That was one extra purge (2X).

One easy change I can see to not use CdnCacheUpdate from HtmlCacheUpdateJob (but still for the pages directly being edited). There is already processing delay anyway (and if there is none, there less likely to be replag, though not guaranteed), so there is less "de facto" use in a secondary purge for backlinks.

Secondary purges where for dealing with replication lag scenarios, not lost purges. That was one extra purge (2X).

In T173710#3548223, @daniel wrote:

In T173710#3547580, @aaron wrote:

In other words, base jobs for entities that will divide up and purge all backlinks to the given entity. Note that each job has two entries.

Wait - each job has two entries? You mean, there are duplicates inserted, and not pruned?...

From

mwscript maintenance/runJobs.php wikidatawiki --type htmlCacheUpdate --nothrottle --maxjobs 100 | grep "IsSelf=1"

CommonSettings.php will still need updating.

Looks like using WikiMap is the best choice IMO.

Mostly htmlCacheUpdate jobs on wikidatawiki:

In config, with $wgTrxProfilerLimits you can do:

Another option is to have the MediaWiki.php code use InterWikiLookup::getAllPrefixes() and the 'iw_url' field for each item.

Closing. The two logging-related improvement action item left have their own tasks.

Error rate went from 500-1000/s to 50-80/s.

In T171371#3521365, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2017-08-12T20:00:52Z] <krinkle@tin> Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s)

Yeah that list should be updated. I happen to investigate things there often (for now), though it's not really in my team domain.

Max and I also noticed backlinks to this wiki from commons in global usage tables.

Did you check redis? Are their jobs back in the queues? It's possible some remote wiki is inserting jobs via JobSpecification or something.

We should see if there is any unrelated downside to adding login/wikidata.org. I'd imagine it would be useful for account creation on connections with high latency.

$wgReadOnly is enforced at the LB/DB layer in addition to random callers checked wfReadOnly().

Have you tied something like this?

$wgReadOnly = ( PHP_SAPI === 'cli' ) ? false : 'This wiki is currently being upgraded to a newer software version.'

I commented on the patch, it's a METHOD mismatch problem, so the commit/wait steps don't happen (just the one big one like before).

Roll-out should probably be something like:
a) DB_MASTER connections for testwiki/mediawikiwiki (group 0)
b) DB_MASTER connections for S3 (25%, 50%, then 100% of connections)
c) DB_MASTER connections for external stores (25%, 50%, then 100% of connections)
d) All DB_MASTER connections remaining shards (25%, 50%, then 100% of connections)

Reading things like https://www.percona.com/blog/2013/10/10/mysql-ssl-performance-overhead/ I think this may only make sense for DB_MASTER (also index 0) connections.

I don't see much noise from the logs about refreshLinks at https://logstash.wikimedia.org/goto/a029053d21a195163e68acc0a23e760e.

Read-only activates for bots sooner, which might lower the edit rate and change the distribution of edits (slower/faster, the ratio with navtiming going up, and such). I'm not sure that would make spikes itself. Actually replication lag adds on to save time, so that can certainly make spikes in navtiming due to ChronologyProtector.

In T125735#3421595, @elukey wrote:

@aaron any chance in your opinion that we could use nutcracker on the jobrunners?

In T164173#3420723, @daniel wrote:

@aaron another question: does RefreshLinksJob also purge the CDN cache automatically? should it? It does update the parser cache...

I also wonder why some of those log warnings come from close() and others have the proper commitMasterChanges() bit in the stack trace. Normally, there should be nothing to commit by close() and it is just commits for sanity.