/otwarchive

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

AO3-5207 Add StatCounterIndexer and fix order of reindexing. #3111

Merged
merged 5 commits into from Oct 22, 2017

Conversation

Reviewers

@houndci-bot houndci-bot

@elzj elzj

Assignees
No one assigned
Labels
Awaiting review Priority: Broken on Test (High)
Projects
None yet
Milestone
No milestone
3 participants
@tickinginstant @elzj @houndci-bot
@tickinginstant
Contributor

tickinginstant commented Oct 22, 2017

Issue

https://otwarchive.atlassian.net/browse/AO3-5207

Purpose

The purpose of this pull request is twofold:

  1. Fix an issue that occurs when :start_new_indexing is active, and :stop_old_indexing is not. Because the old indexing code deletes all of the indices in the subqueue after performing reindexing, the new indexing code was being run on the empty list, which was causing an error. (And also meant that nothing was being reindexed with the new system.)

  2. Add in a StatCounterIndexer class so that the new indexing code can perform a stats-only update like the old indexing code, rather than having to reindex the whole work just to update the hit count.

Testing

Check to make sure that the ScheduledReindexJobs are being run without generating error messages.

tickinginstant added some commits Oct 21, 2017

@tickinginstant
AO3-5207 First go at StatCounter reindexing.
d4621c9
@tickinginstant
AO3-5207 Add RSpec tests to make sure it works.
51969de
@tickinginstant
AO3-5207 Small fixes to pass tests.
a40defb
@tickinginstant
AO3-5207 Make sure to call original when stubbing.
3c01187
@tickinginstant
AO3-5207 Finally fixed.

All checks have passed

4 successful checks
Scrutinizer — 7 updated code elements
Details
codeclimate — All good!
Details
continuous-integration/travis-ci/pr — The Travis CI build passed
Details
hound — 6 violations found.
eaa73ec
app/models/indexing/index_subqueue.rb
@@ -35,6 +35,11 @@ def enqueue
end
def run
+ # Do this before the old indexing to make sure that we know which IDs to
+ # pass to the indexer. (Delete this comment after the upgrade is complete.)
+ if $rollout.active?(:start_new_indexing)
@houndci-bot

houndci-bot Oct 22, 2017

Do not introduce global variables.

app/models/search/stat_counter_indexer.rb
@@ -0,0 +1,56 @@
+# A class for reindexing work stats.
@houndci-bot

houndci-bot Oct 22, 2017

Missing magic comment # frozen_string_literal: true.

app/models/search/stat_counter_indexer.rb
+ end
+
+ def index_documents
+ $new_elasticsearch.bulk(body: batch)
@houndci-bot

houndci-bot Oct 22, 2017

Do not introduce global variables.

spec/models/index_queue_spec.rb
@@ -32,4 +32,51 @@
expect(IndexQueue::REDIS.exists("index:work:main")).to be_falsey
end
+
+ describe "#run" do
@houndci-bot

houndci-bot Oct 22, 2017

Block has too many lines. [40/25]

spec/models/stat_counter_indexer_spec.rb
@@ -0,0 +1,60 @@
+require 'spec_helper'
@houndci-bot

houndci-bot Oct 22, 2017

Missing magic comment # frozen_string_literal: true.

spec/models/stat_counter_indexer_spec.rb
@@ -0,0 +1,60 @@
+require 'spec_helper'
+
+describe StatCounterIndexer do
@houndci-bot

houndci-bot Oct 22, 2017

Block has too many lines. [46/25]

spec/models/stat_counter_indexer_spec.rb
+ let!(:work) { create(:work, title: "unique title", posted: true) }
+ let!(:stat_counter) { work.stat_counter }
+
+ describe "#index_documents" do
@houndci-bot

houndci-bot Oct 22, 2017

Block has too many lines. [42/25]

spec/models/stat_counter_indexer_spec.rb
+ end
+
+ it "should not change the search results for title" do
+ expect do
@houndci-bot

houndci-bot Oct 22, 2017

Parenthesize the param change { result_count(title: "unique title") } to make sure that the block will be associated with the change method call.

spec/spec_helper.rb
@@ -168,6 +168,10 @@ def update_and_refresh_indexes(klass_name)
$new_elasticsearch.indices.refresh(index: "ao3_test_#{klass_name}s")
end
+def refresh_index_without_updating(klass_name)
+ $new_elasticsearch.indices.refresh(index: "ao3_test_#{klass_name}s")
@houndci-bot

houndci-bot Oct 22, 2017

Do not introduce global variables.

@elzj elzj added Awaiting review Priority: Broken on Test (High) labels Oct 22, 2017

@elzj

elzj approved these changes Oct 22, 2017
View changes

@elzj elzj merged commit cc2e7e8 into otwcode:master Oct 22, 2017
4 checks passed

4 checks passed

Scrutinizer 7 updated code elements
Details
codeclimate All good!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
hound 6 violations found.

zz9pzza added a commit that referenced this pull request Nov 27, 2017

@elzj @zz9pzza
Re-merge Elasticsearch branch (#3179) 
* AO3-4815 Update Elasticsearch and filter interface (#3104)

* Upgrades faraday (ran bundle update elasticsearch)

* Adds elasticsearch-model to directly replace tire model

Uses rollout to indicate which module to include in the searchable modules

* Allows application to connect to two ES instances

* Switches version of ES in use by Indexer and Query depending on rollout feature

* Fixes feature failure in importing

The image is rendered with https://, not http://

* Adds awesome_print gem, to make inspecting json easier

* Adds class to recreate indices in upgraded ES instance & reindex from remote

* Removes elasticsearch-model

* Removes rollout code from models

* WIP - Making app compatible with ES 5.5

* WIP - Pulls in elasticsearch overhaul code

Starting to debug elasticsearch overhaul code, both in its incompatibilities
with current ES and its underlying bugs.

* Gets search/works_anonymous feature specs to pass with new ES code

* Gets search/works_tags features to pass with new es code

* Fixes search/works_info spec failures

* Fixes test failures in search/work_stats

* Updates ES version read by Travis

* Updates ES instance to point to the url Travis will recognize

* Fixes some failing feature specs for search

* Fixes admins/admin_works specs

* Fixes bookmark_create and collection_dashboard feature specs

* Fixes importing/archivist feature specs

* Fixes other_a/orphan_work feature specs

Still using Tire, and reindexing with both Tire and elasticsearch will not break
anything and will be useful when running the test suite twice against the two
different ES versions

* Fixes all other other_a feature failures

* Fixes works_anonymous feature failures

* Fixes tag_wrangling_relationships feature failures

* Fixes users feature failures

* Fixes works feature failures

* Adds the new ES indexing code wherever Tire is use in test setup

* Fixes works default rails actions spec failures

* Fixes elasticsearch_upgrade_spec failures and refactors ES refresh code

* Fixes work query spec

* Adds some minor refactoring to tests

* Adds bookmark indexing/query code & fixes associated tests

* Updates BookmarkSearch references to BookmarkSearchForm

* Fixes bookmark_create feature failures

* Fixes admin/admin_works feature failures

* Fixes bookmarks feature spec failures

* Fixes collections feature specs

* Fixes other_a and search failures

* Fixes users feature specs

* Removes refrences to tire from tests and specs for now

* Fixes search steps problem when reindexing models with any? false

* Fixes reindexing models without any records in specs

* Removes outdated use of WorkSearch model in works controller

* Fixes typo

* Fixes failing specs caused by fixes to failing features

* Fixes bugs in bookmark query

* Some feature failure cleanup

* Adds code for Tag search using updated ES

* Adds pseud search code

* Small test fixes

* Fixes some failing tests, and attempts to address Travis issues

* Adds fixes for Travis failures that are repeatable locally

* Fixes problem where bookmarks without works can't be searched for

* Reindexes indexes everywhere they are updated in the features directory

This should remove inconsistent, nearly-unrepeatable errors caused by the fact
that Elasticsearch is asynchronous. Yay!

* Fixes bookmark query spec failure

* Refactors some of the index updating logic in features

* Fixes default rails actions for works spec

* Isolates Tire reindexing to one file in features

* Fixes tags and wrangling test failures

* Distinguishes ES versions in specs

* Fixes some typos

* Updates to catch old ES version no matter the environment

* Attempting a run with old ES version

* Fixes Tire import

* Fixes spec helper method

* Separates ES wrappers based on ES version in controllers

* Fixes use of new ES code in helpers for use_old_search?

* Fixes old ES code usage in bookmarks controller

* Fixes search using old ES for bookmarks

* Removes tests that were catching a bug that is no longer a problem

* Fixes works failures for old es version

* Fixes features failing on Travis

* Fixes spec failures for old ES version

* Sets Travis to dl new version of ES to see how build does

* Attempt with ES 6.0 alpha

* Adds fix for bookmark deletion bug

* Adds missing details to work indexer

* Adds rollout to indexing

* Swaps out 'use_old_search?' with 'use_new_search?'

* Fixes param names in tests

* Adds rollout to application record use_new_search? method

* Fixes small details & removes unused code

* Fixes typo

* Fixes typo

* Fixes typo

* Fixes lack of question mark for rollout.active? method

* Runs Travis build on old ES and sets elasticsearch version with rollout

* Fixes inccorect method invocation

* Fixes incorrect method invocation in people controller

* Fixes works stats feature failures

* Adds fixes for failing specs

* Uses appropriate count method for old es version

* Removes debug line from feature

* Upgrades nokogiri due to security concern

* Makes it so app will respond appropriately to ES version

To run on Travis, all three ES urls need to be the same:
ENV['ELASTICSEARCH_URL'] &
ENV['ELASTICSEARCH_1_URL'] &
ENV['UPGRADED_ELASTICSEARCH_URL']

All need to point to the same thing. Whenever running an instance of the app
with two versions of ES, ENV['ELASTICSEARCH_URL'] and ENV['ELASTICSEARCH_1_URL']
need to point to the old version and ENV['UPGRADED_ELASTICSEARCH_URL'] needs to
point to the new version.

* Second attempt with 6.0.0-alpha1 ES

* For test with ES 6.0.0-alpha1

* Fixes Travis run in 6.0.0-alpha1 ES

* Untar

* Try starting both

* Simple test run

* Differentiate the builds

* Differentiate the builds

* Restore all the tests

* Fixes call to new elasticsearch api perform_request

* Updates to test suite to watch how upgraded es version works on its own

* refactors out index updating methods in feature specs

* Adds rollout specs

* Fixes bug in work search form object

* Matches config to travis

* Removes all but rspec tests from travis for now

* Sets up tests & rollout methods use expected behavior

* Fixes switch from string to text or keyword for index definitions

* Updates index definitions to be compatible with 6.0.0.beta2 ES version

* Changes localhost to home IP for old ES urls

* Does not require old es to work when old es is gone

* Switches ES bool queries to true/false

* Fixes spec failures for ES Group 1

* Adds specific test to ensure old elasticsearch does not index when it's not present

* Moves to new elasticsearch document indexing when old es is gone

* Fixes problem with helper method for specs

* Ensures app never attempts to reindex an old es index in the new es instance

* Makes it so that a user doesn't have to be present if use_new_search is 100% enabled

* Runs ES=1

* Runs ES=2

* Runs ES=3

* Runs ES=4

* Back to rspec

* Removes command to rebuild three times upon failure

* Updates set_rollout script to use global  variable

* Adds rollout definitions to test suite to see if that addresses Travis build problem

* Runs build with ES=1

* Runs all builds

* Adds same assurances to features as specs that ES will work w/ both versions

* Fixes imported image URL in feature spec

* Fixes reference to tire methods when tire is gone in feature support methods

* Runs only ES=4

* Moves es/rollout code for feature specs

* Fixes boolean representation in work search form

* Fixes boolean in ES search in bookmark search form

* Fixes bookmarks feature failures

* Fixes works_restricted feature failures

* Runs all four ES groups

* Runs ES=1 and ES=4

* Comments out test that's breaking due to third-party url bug

* Makes Travis only run 2 trials

1 - With the old Elasticsearch, in which we test that the old Elasticsearch
continues to work while start_new_indexing is true (individual tests also test
that the SearchForm classes continue to work as expected, meaning that the new
Elasticsearch also works in this case when use_new_search is true)

2 - With the new Elasticsearch

* Adds old exlclusion filter code

* Adds tests & code for work filtering with exclusion tags

* Addresses Travis test failures

* Adds bookmark exclusion filters

* Downgrades back to 5.6.1 (latest stable) ES version

* Adds single_mapping => true to ES indices

* Fixes failure to index bookmarks when they do not have a bookmarkable object

* Adds comments to help when removing old code

* Removes rollout manipulation from Travis

This was not actually setting features. All Travis needs to do is install one or
both versions of ES depending on the test group being run.

* Adds test cases for use_new_search? controller method

* Adds tests & fixes bugs for filtering

* Removes default 'true' from use_new_search method

Reverts old styles and scripts to master

* Fixes typos

* Fixes more typos

* Fixes rollout test for works controller

* Fixes works controller mistake

* Ports over ALL js and css changes that effect filters

* Removes useless method

* Removes file that has nothing to do with new search

* Fixes failing filter tests

* Pulls in fix for autocomplete bug from master

* AO3-4815: Fill out new batch indexing code

* Fixes merge clobbering of css & js files

* AO3-4815: Add new ES to codeship script and use local.yml instead of … (#3110)

* AO3-4815: Add new ES to codeship script and use local.yml instead of editing the regular config

* Fix ES download url and load local config when testing

* AO3-4815: Ensure correct results when browsing bookmarks by user (#3109)

* AO3-5213 Index authors on works correctly for sorting (#3114)

AO3-5213 Index authors on works correctly for sorting

* AO3-5207 Add StatCounterIndexer and fix order of reindexing. (#3111)

* AO3-5207 First go at StatCounter reindexing.

* AO3-5207 Add RSpec tests to make sure it works.

* AO3-5207 Small fixes to pass tests.

* AO3-5207 Make sure to call original when stubbing.

* AO3-5207 Finally fixed.

* AO3-5212 Index correct user ids for works. Also throw in a couple of updated work type tag ids (#3112)

* AO3-5209 Don't change the appearance of non-work & bookmark filters (#3117)

* AO3-5209 Use old JavaScript and views for old Elasticsearch filters (styling fixes to come)

* AO3-5209 Restyle old filters

* AO3-5209 Ensure new work and bookmark filter styling does not conflict with exisiting collection filter styles

* AO3-5216 Fix setting authenticity tokens when cached pages load (#3118)

This was the fix for AO3-5153, which got lost in a merge.

* AO3-5214 Only run the tests on Travis once for now (#3116)

* AO3-5208: Fix filtering behavior (#3113)

* AO3-5208: Fix filtering behavior by drilling down and setting facets appropriately.

* AO3-5208: Don't error when there are no filters

* AO3-5208: Ensure excluded tags are visible in filters

* AO3-5208: Make sure bookmark facet query knows about exclusions. Also don't index filters and other bookmarkable fields on the bookmark itself since that's what we're trying to avoid.

* AO3-5208: Dry up some tag-related query methods, reduce number of db requests, only exclude children of Character tags, and make exclusion work for bookmark tags again

* AO3-5208: Update some tests to match new behavior

* AO3-5208: Put excluded bookmark tags in the right filter bucket

* AO3-5208: Remove sub tag exclusion code, since the meta tag id is already indexed on works

* AO3-5208: Comment out tests for types of exclusion filtering we aren't currently doing for bookmarks

* AO3-5208: Enable querying on both parent and child bookmark fields and reflect queries in bookmark filters

* AO3-5208: Update tests for bookmark search syntax change and bookmarker indexing encompassing user name

* AO3-5208: Make 'other tags' filtering work for noncanonical tags as well

* AO3-5223: Fix id on bookmarks exclude field

* AO3-5208: Remove child exclusion from filters and enable inclusion for noncanonical tags

* AO3-5220 and AO3-5224: Fix searching bookmarks by notes and tag names and index bookmarkable date for sorting

* AO3-5220: Update old bookmark search code for notes fix, fix spacing in module

* AO3-5225 Fix sidebar counts and scope by user id if a user is present (#3119)

* AO3-5225: Fix sidebar counts and scope by user id if a user is present

* AO3-5225: Update user dashboard feature to check counts in multiple places

* AO3-5231: Fix formatting of search text and don't chop up tag searches

* AO3-5231: Remove duplication and add specs

Some checks haven’t completed yet

1 pending and 1 successful checks
continuous-integration/codeship — Build in progress
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
e2c284a

sarken added a commit that referenced this pull request Nov 27, 2017

@elzj @sarken
AO3-5254: Upgrade to Elasticsearch 6 (#3183) 
* AO3-4815 Update Elasticsearch and filter interface (#3104)

* Upgrades faraday (ran bundle update elasticsearch)

* Adds elasticsearch-model to directly replace tire model

Uses rollout to indicate which module to include in the searchable modules

* Allows application to connect to two ES instances

* Switches version of ES in use by Indexer and Query depending on rollout feature

* Fixes feature failure in importing

The image is rendered with https://, not http://

* Adds awesome_print gem, to make inspecting json easier

* Adds class to recreate indices in upgraded ES instance & reindex from remote

* Removes elasticsearch-model

* Removes rollout code from models

* WIP - Making app compatible with ES 5.5

* WIP - Pulls in elasticsearch overhaul code

Starting to debug elasticsearch overhaul code, both in its incompatibilities
with current ES and its underlying bugs.

* Gets search/works_anonymous feature specs to pass with new ES code

* Gets search/works_tags features to pass with new es code

* Fixes search/works_info spec failures

* Fixes test failures in search/work_stats

* Updates ES version read by Travis

* Updates ES instance to point to the url Travis will recognize

* Fixes some failing feature specs for search

* Fixes admins/admin_works specs

* Fixes bookmark_create and collection_dashboard feature specs

* Fixes importing/archivist feature specs

* Fixes other_a/orphan_work feature specs

Still using Tire, and reindexing with both Tire and elasticsearch will not break
anything and will be useful when running the test suite twice against the two
different ES versions

* Fixes all other other_a feature failures

* Fixes works_anonymous feature failures

* Fixes tag_wrangling_relationships feature failures

* Fixes users feature failures

* Fixes works feature failures

* Adds the new ES indexing code wherever Tire is use in test setup

* Fixes works default rails actions spec failures

* Fixes elasticsearch_upgrade_spec failures and refactors ES refresh code

* Fixes work query spec

* Adds some minor refactoring to tests

* Adds bookmark indexing/query code & fixes associated tests

* Updates BookmarkSearch references to BookmarkSearchForm

* Fixes bookmark_create feature failures

* Fixes admin/admin_works feature failures

* Fixes bookmarks feature spec failures

* Fixes collections feature specs

* Fixes other_a and search failures

* Fixes users feature specs

* Removes refrences to tire from tests and specs for now

* Fixes search steps problem when reindexing models with any? false

* Fixes reindexing models without any records in specs

* Removes outdated use of WorkSearch model in works controller

* Fixes typo

* Fixes failing specs caused by fixes to failing features

* Fixes bugs in bookmark query

* Some feature failure cleanup

* Adds code for Tag search using updated ES

* Adds pseud search code

* Small test fixes

* Fixes some failing tests, and attempts to address Travis issues

* Adds fixes for Travis failures that are repeatable locally

* Fixes problem where bookmarks without works can't be searched for

* Reindexes indexes everywhere they are updated in the features directory

This should remove inconsistent, nearly-unrepeatable errors caused by the fact
that Elasticsearch is asynchronous. Yay!

* Fixes bookmark query spec failure

* Refactors some of the index updating logic in features

* Fixes default rails actions for works spec

* Isolates Tire reindexing to one file in features

* Fixes tags and wrangling test failures

* Distinguishes ES versions in specs

* Fixes some typos

* Updates to catch old ES version no matter the environment

* Attempting a run with old ES version

* Fixes Tire import

* Fixes spec helper method

* Separates ES wrappers based on ES version in controllers

* Fixes use of new ES code in helpers for use_old_search?

* Fixes old ES code usage in bookmarks controller

* Fixes search using old ES for bookmarks

* Removes tests that were catching a bug that is no longer a problem

* Fixes works failures for old es version

* Fixes features failing on Travis

* Fixes spec failures for old ES version

* Sets Travis to dl new version of ES to see how build does

* Attempt with ES 6.0 alpha

* Adds fix for bookmark deletion bug

* Adds missing details to work indexer

* Adds rollout to indexing

* Swaps out 'use_old_search?' with 'use_new_search?'

* Fixes param names in tests

* Adds rollout to application record use_new_search? method

* Fixes small details & removes unused code

* Fixes typo

* Fixes typo

* Fixes typo

* Fixes lack of question mark for rollout.active? method

* Runs Travis build on old ES and sets elasticsearch version with rollout

* Fixes inccorect method invocation

* Fixes incorrect method invocation in people controller

* Fixes works stats feature failures

* Adds fixes for failing specs

* Uses appropriate count method for old es version

* Removes debug line from feature

* Upgrades nokogiri due to security concern

* Makes it so app will respond appropriately to ES version

To run on Travis, all three ES urls need to be the same:
ENV['ELASTICSEARCH_URL'] &
ENV['ELASTICSEARCH_1_URL'] &
ENV['UPGRADED_ELASTICSEARCH_URL']

All need to point to the same thing. Whenever running an instance of the app
with two versions of ES, ENV['ELASTICSEARCH_URL'] and ENV['ELASTICSEARCH_1_URL']
need to point to the old version and ENV['UPGRADED_ELASTICSEARCH_URL'] needs to
point to the new version.

* Second attempt with 6.0.0-alpha1 ES

* For test with ES 6.0.0-alpha1

* Fixes Travis run in 6.0.0-alpha1 ES

* Untar

* Try starting both

* Simple test run

* Differentiate the builds

* Differentiate the builds

* Restore all the tests

* Fixes call to new elasticsearch api perform_request

* Updates to test suite to watch how upgraded es version works on its own

* refactors out index updating methods in feature specs

* Adds rollout specs

* Fixes bug in work search form object

* Matches config to travis

* Removes all but rspec tests from travis for now

* Sets up tests & rollout methods use expected behavior

* Fixes switch from string to text or keyword for index definitions

* Updates index definitions to be compatible with 6.0.0.beta2 ES version

* Changes localhost to home IP for old ES urls

* Does not require old es to work when old es is gone

* Switches ES bool queries to true/false

* Fixes spec failures for ES Group 1

* Adds specific test to ensure old elasticsearch does not index when it's not present

* Moves to new elasticsearch document indexing when old es is gone

* Fixes problem with helper method for specs

* Ensures app never attempts to reindex an old es index in the new es instance

* Makes it so that a user doesn't have to be present if use_new_search is 100% enabled

* Runs ES=1

* Runs ES=2

* Runs ES=3

* Runs ES=4

* Back to rspec

* Removes command to rebuild three times upon failure

* Updates set_rollout script to use global  variable

* Adds rollout definitions to test suite to see if that addresses Travis build problem

* Runs build with ES=1

* Runs all builds

* Adds same assurances to features as specs that ES will work w/ both versions

* Fixes imported image URL in feature spec

* Fixes reference to tire methods when tire is gone in feature support methods

* Runs only ES=4

* Moves es/rollout code for feature specs

* Fixes boolean representation in work search form

* Fixes boolean in ES search in bookmark search form

* Fixes bookmarks feature failures

* Fixes works_restricted feature failures

* Runs all four ES groups

* Runs ES=1 and ES=4

* Comments out test that's breaking due to third-party url bug

* Makes Travis only run 2 trials

1 - With the old Elasticsearch, in which we test that the old Elasticsearch
continues to work while start_new_indexing is true (individual tests also test
that the SearchForm classes continue to work as expected, meaning that the new
Elasticsearch also works in this case when use_new_search is true)

2 - With the new Elasticsearch

* Adds old exlclusion filter code

* Adds tests & code for work filtering with exclusion tags

* Addresses Travis test failures

* Adds bookmark exclusion filters

* Downgrades back to 5.6.1 (latest stable) ES version

* Adds single_mapping => true to ES indices

* Fixes failure to index bookmarks when they do not have a bookmarkable object

* Adds comments to help when removing old code

* Removes rollout manipulation from Travis

This was not actually setting features. All Travis needs to do is install one or
both versions of ES depending on the test group being run.

* Adds test cases for use_new_search? controller method

* Adds tests & fixes bugs for filtering

* Removes default 'true' from use_new_search method

Reverts old styles and scripts to master

* Fixes typos

* Fixes more typos

* Fixes rollout test for works controller

* Fixes works controller mistake

* Ports over ALL js and css changes that effect filters

* Removes useless method

* Removes file that has nothing to do with new search

* Fixes failing filter tests

* Pulls in fix for autocomplete bug from master

* AO3-4815: Fill out new batch indexing code

* Fixes merge clobbering of css & js files

* AO3-4815: Add new ES to codeship script and use local.yml instead of … (#3110)

* AO3-4815: Add new ES to codeship script and use local.yml instead of editing the regular config

* Fix ES download url and load local config when testing

* AO3-4815: Ensure correct results when browsing bookmarks by user (#3109)

* AO3-5213 Index authors on works correctly for sorting (#3114)

AO3-5213 Index authors on works correctly for sorting

* AO3-5207 Add StatCounterIndexer and fix order of reindexing. (#3111)

* AO3-5207 First go at StatCounter reindexing.

* AO3-5207 Add RSpec tests to make sure it works.

* AO3-5207 Small fixes to pass tests.

* AO3-5207 Make sure to call original when stubbing.

* AO3-5207 Finally fixed.

* AO3-5212 Index correct user ids for works. Also throw in a couple of updated work type tag ids (#3112)

* AO3-5209 Don't change the appearance of non-work & bookmark filters (#3117)

* AO3-5209 Use old JavaScript and views for old Elasticsearch filters (styling fixes to come)

* AO3-5209 Restyle old filters

* AO3-5209 Ensure new work and bookmark filter styling does not conflict with exisiting collection filter styles

* AO3-5216 Fix setting authenticity tokens when cached pages load (#3118)

This was the fix for AO3-5153, which got lost in a merge.

* AO3-5214 Only run the tests on Travis once for now (#3116)

* AO3-5208: Fix filtering behavior (#3113)

* AO3-5208: Fix filtering behavior by drilling down and setting facets appropriately.

* AO3-5208: Don't error when there are no filters

* AO3-5208: Ensure excluded tags are visible in filters

* AO3-5208: Make sure bookmark facet query knows about exclusions. Also don't index filters and other bookmarkable fields on the bookmark itself since that's what we're trying to avoid.

* AO3-5208: Dry up some tag-related query methods, reduce number of db requests, only exclude children of Character tags, and make exclusion work for bookmark tags again

* AO3-5208: Update some tests to match new behavior

* AO3-5208: Put excluded bookmark tags in the right filter bucket

* AO3-5208: Remove sub tag exclusion code, since the meta tag id is already indexed on works

* AO3-5208: Comment out tests for types of exclusion filtering we aren't currently doing for bookmarks

* AO3-5208: Enable querying on both parent and child bookmark fields and reflect queries in bookmark filters

* AO3-5208: Update tests for bookmark search syntax change and bookmarker indexing encompassing user name

* AO3-5208: Make 'other tags' filtering work for noncanonical tags as well

* AO3-5223: Fix id on bookmarks exclude field

* AO3-5208: Remove child exclusion from filters and enable inclusion for noncanonical tags

* AO3-5220 and AO3-5224: Fix searching bookmarks by notes and tag names and index bookmarkable date for sorting

* AO3-5220: Update old bookmark search code for notes fix, fix spacing in module

* AO3-5225 Fix sidebar counts and scope by user id if a user is present (#3119)

* AO3-5225: Fix sidebar counts and scope by user id if a user is present

* AO3-5225: Update user dashboard feature to check counts in multiple places

* AO3-5231: Fix formatting of search text and don't chop up tag searches

* AO3-5231: Remove duplication and add specs

* Upgrade elasticsearch version on travis

* Remove single type mapping which is no longer needed

* Update has_parent query syntax

* Update syntax for parent and child filters, which are now only supported as queries

* Update bookmark query spec to match new syntax

* Upgrade codeship

Some checks were not successful

1 failing and 1 successful checks
continuous-integration/travis-ci/push — The Travis CI build failed
Details
continuous-integration/codeship — Build succeeded
Details
2dae45d

sarken added a commit that referenced this pull request Nov 30, 2017

@elzj @sarken
AO3-5221: Bookmarkable indexing (#3186) 
* AO3-4815 Update Elasticsearch and filter interface (#3104)

* Upgrades faraday (ran bundle update elasticsearch)

* Adds elasticsearch-model to directly replace tire model

Uses rollout to indicate which module to include in the searchable modules

* Allows application to connect to two ES instances

* Switches version of ES in use by Indexer and Query depending on rollout feature

* Fixes feature failure in importing

The image is rendered with https://, not http://

* Adds awesome_print gem, to make inspecting json easier

* Adds class to recreate indices in upgraded ES instance & reindex from remote

* Removes elasticsearch-model

* Removes rollout code from models

* WIP - Making app compatible with ES 5.5

* WIP - Pulls in elasticsearch overhaul code

Starting to debug elasticsearch overhaul code, both in its incompatibilities
with current ES and its underlying bugs.

* Gets search/works_anonymous feature specs to pass with new ES code

* Gets search/works_tags features to pass with new es code

* Fixes search/works_info spec failures

* Fixes test failures in search/work_stats

* Updates ES version read by Travis

* Updates ES instance to point to the url Travis will recognize

* Fixes some failing feature specs for search

* Fixes admins/admin_works specs

* Fixes bookmark_create and collection_dashboard feature specs

* Fixes importing/archivist feature specs

* Fixes other_a/orphan_work feature specs

Still using Tire, and reindexing with both Tire and elasticsearch will not break
anything and will be useful when running the test suite twice against the two
different ES versions

* Fixes all other other_a feature failures

* Fixes works_anonymous feature failures

* Fixes tag_wrangling_relationships feature failures

* Fixes users feature failures

* Fixes works feature failures

* Adds the new ES indexing code wherever Tire is use in test setup

* Fixes works default rails actions spec failures

* Fixes elasticsearch_upgrade_spec failures and refactors ES refresh code

* Fixes work query spec

* Adds some minor refactoring to tests

* Adds bookmark indexing/query code & fixes associated tests

* Updates BookmarkSearch references to BookmarkSearchForm

* Fixes bookmark_create feature failures

* Fixes admin/admin_works feature failures

* Fixes bookmarks feature spec failures

* Fixes collections feature specs

* Fixes other_a and search failures

* Fixes users feature specs

* Removes refrences to tire from tests and specs for now

* Fixes search steps problem when reindexing models with any? false

* Fixes reindexing models without any records in specs

* Removes outdated use of WorkSearch model in works controller

* Fixes typo

* Fixes failing specs caused by fixes to failing features

* Fixes bugs in bookmark query

* Some feature failure cleanup

* Adds code for Tag search using updated ES

* Adds pseud search code

* Small test fixes

* Fixes some failing tests, and attempts to address Travis issues

* Adds fixes for Travis failures that are repeatable locally

* Fixes problem where bookmarks without works can't be searched for

* Reindexes indexes everywhere they are updated in the features directory

This should remove inconsistent, nearly-unrepeatable errors caused by the fact
that Elasticsearch is asynchronous. Yay!

* Fixes bookmark query spec failure

* Refactors some of the index updating logic in features

* Fixes default rails actions for works spec

* Isolates Tire reindexing to one file in features

* Fixes tags and wrangling test failures

* Distinguishes ES versions in specs

* Fixes some typos

* Updates to catch old ES version no matter the environment

* Attempting a run with old ES version

* Fixes Tire import

* Fixes spec helper method

* Separates ES wrappers based on ES version in controllers

* Fixes use of new ES code in helpers for use_old_search?

* Fixes old ES code usage in bookmarks controller

* Fixes search using old ES for bookmarks

* Removes tests that were catching a bug that is no longer a problem

* Fixes works failures for old es version

* Fixes features failing on Travis

* Fixes spec failures for old ES version

* Sets Travis to dl new version of ES to see how build does

* Attempt with ES 6.0 alpha

* Adds fix for bookmark deletion bug

* Adds missing details to work indexer

* Adds rollout to indexing

* Swaps out 'use_old_search?' with 'use_new_search?'

* Fixes param names in tests

* Adds rollout to application record use_new_search? method

* Fixes small details & removes unused code

* Fixes typo

* Fixes typo

* Fixes typo

* Fixes lack of question mark for rollout.active? method

* Runs Travis build on old ES and sets elasticsearch version with rollout

* Fixes inccorect method invocation

* Fixes incorrect method invocation in people controller

* Fixes works stats feature failures

* Adds fixes for failing specs

* Uses appropriate count method for old es version

* Removes debug line from feature

* Upgrades nokogiri due to security concern

* Makes it so app will respond appropriately to ES version

To run on Travis, all three ES urls need to be the same:
ENV['ELASTICSEARCH_URL'] &
ENV['ELASTICSEARCH_1_URL'] &
ENV['UPGRADED_ELASTICSEARCH_URL']

All need to point to the same thing. Whenever running an instance of the app
with two versions of ES, ENV['ELASTICSEARCH_URL'] and ENV['ELASTICSEARCH_1_URL']
need to point to the old version and ENV['UPGRADED_ELASTICSEARCH_URL'] needs to
point to the new version.

* Second attempt with 6.0.0-alpha1 ES

* For test with ES 6.0.0-alpha1

* Fixes Travis run in 6.0.0-alpha1 ES

* Untar

* Try starting both

* Simple test run

* Differentiate the builds

* Differentiate the builds

* Restore all the tests

* Fixes call to new elasticsearch api perform_request

* Updates to test suite to watch how upgraded es version works on its own

* refactors out index updating methods in feature specs

* Adds rollout specs

* Fixes bug in work search form object

* Matches config to travis

* Removes all but rspec tests from travis for now

* Sets up tests & rollout methods use expected behavior

* Fixes switch from string to text or keyword for index definitions

* Updates index definitions to be compatible with 6.0.0.beta2 ES version

* Changes localhost to home IP for old ES urls

* Does not require old es to work when old es is gone

* Switches ES bool queries to true/false

* Fixes spec failures for ES Group 1

* Adds specific test to ensure old elasticsearch does not index when it's not present

* Moves to new elasticsearch document indexing when old es is gone

* Fixes problem with helper method for specs

* Ensures app never attempts to reindex an old es index in the new es instance

* Makes it so that a user doesn't have to be present if use_new_search is 100% enabled

* Runs ES=1

* Runs ES=2

* Runs ES=3

* Runs ES=4

* Back to rspec

* Removes command to rebuild three times upon failure

* Updates set_rollout script to use global  variable

* Adds rollout definitions to test suite to see if that addresses Travis build problem

* Runs build with ES=1

* Runs all builds

* Adds same assurances to features as specs that ES will work w/ both versions

* Fixes imported image URL in feature spec

* Fixes reference to tire methods when tire is gone in feature support methods

* Runs only ES=4

* Moves es/rollout code for feature specs

* Fixes boolean representation in work search form

* Fixes boolean in ES search in bookmark search form

* Fixes bookmarks feature failures

* Fixes works_restricted feature failures

* Runs all four ES groups

* Runs ES=1 and ES=4

* Comments out test that's breaking due to third-party url bug

* Makes Travis only run 2 trials

1 - With the old Elasticsearch, in which we test that the old Elasticsearch
continues to work while start_new_indexing is true (individual tests also test
that the SearchForm classes continue to work as expected, meaning that the new
Elasticsearch also works in this case when use_new_search is true)

2 - With the new Elasticsearch

* Adds old exlclusion filter code

* Adds tests & code for work filtering with exclusion tags

* Addresses Travis test failures

* Adds bookmark exclusion filters

* Downgrades back to 5.6.1 (latest stable) ES version

* Adds single_mapping => true to ES indices

* Fixes failure to index bookmarks when they do not have a bookmarkable object

* Adds comments to help when removing old code

* Removes rollout manipulation from Travis

This was not actually setting features. All Travis needs to do is install one or
both versions of ES depending on the test group being run.

* Adds test cases for use_new_search? controller method

* Adds tests & fixes bugs for filtering

* Removes default 'true' from use_new_search method

Reverts old styles and scripts to master

* Fixes typos

* Fixes more typos

* Fixes rollout test for works controller

* Fixes works controller mistake

* Ports over ALL js and css changes that effect filters

* Removes useless method

* Removes file that has nothing to do with new search

* Fixes failing filter tests

* Pulls in fix for autocomplete bug from master

* AO3-4815: Fill out new batch indexing code

* Fixes merge clobbering of css & js files

* AO3-4815: Add new ES to codeship script and use local.yml instead of … (#3110)

* AO3-4815: Add new ES to codeship script and use local.yml instead of editing the regular config

* Fix ES download url and load local config when testing

* AO3-4815: Ensure correct results when browsing bookmarks by user (#3109)

* AO3-5213 Index authors on works correctly for sorting (#3114)

AO3-5213 Index authors on works correctly for sorting

* AO3-5207 Add StatCounterIndexer and fix order of reindexing. (#3111)

* AO3-5207 First go at StatCounter reindexing.

* AO3-5207 Add RSpec tests to make sure it works.

* AO3-5207 Small fixes to pass tests.

* AO3-5207 Make sure to call original when stubbing.

* AO3-5207 Finally fixed.

* AO3-5212 Index correct user ids for works. Also throw in a couple of updated work type tag ids (#3112)

* AO3-5209 Don't change the appearance of non-work & bookmark filters (#3117)

* AO3-5209 Use old JavaScript and views for old Elasticsearch filters (styling fixes to come)

* AO3-5209 Restyle old filters

* AO3-5209 Ensure new work and bookmark filter styling does not conflict with exisiting collection filter styles

* AO3-5216 Fix setting authenticity tokens when cached pages load (#3118)

This was the fix for AO3-5153, which got lost in a merge.

* AO3-5214 Only run the tests on Travis once for now (#3116)

* AO3-5208: Fix filtering behavior (#3113)

* AO3-5208: Fix filtering behavior by drilling down and setting facets appropriately.

* AO3-5208: Don't error when there are no filters

* AO3-5208: Ensure excluded tags are visible in filters

* AO3-5208: Make sure bookmark facet query knows about exclusions. Also don't index filters and other bookmarkable fields on the bookmark itself since that's what we're trying to avoid.

* AO3-5208: Dry up some tag-related query methods, reduce number of db requests, only exclude children of Character tags, and make exclusion work for bookmark tags again

* AO3-5208: Update some tests to match new behavior

* AO3-5208: Put excluded bookmark tags in the right filter bucket

* AO3-5208: Remove sub tag exclusion code, since the meta tag id is already indexed on works

* AO3-5208: Comment out tests for types of exclusion filtering we aren't currently doing for bookmarks

* AO3-5208: Enable querying on both parent and child bookmark fields and reflect queries in bookmark filters

* AO3-5208: Update tests for bookmark search syntax change and bookmarker indexing encompassing user name

* AO3-5208: Make 'other tags' filtering work for noncanonical tags as well

* AO3-5223: Fix id on bookmarks exclude field

* AO3-5208: Remove child exclusion from filters and enable inclusion for noncanonical tags

* AO3-5220 and AO3-5224: Fix searching bookmarks by notes and tag names and index bookmarkable date for sorting

* AO3-5220: Update old bookmark search code for notes fix, fix spacing in module

* AO3-5225 Fix sidebar counts and scope by user id if a user is present (#3119)

* AO3-5225: Fix sidebar counts and scope by user id if a user is present

* AO3-5225: Update user dashboard feature to check counts in multiple places

* AO3-5221: Set up a pathway from bookmarked classes to our indexers

* AO3-5221: Fix code that got re-added on merge

* AO3-5221: Do a little method delegation

* AO3-5221: Make sure stat counter indexing still works

* AO3-5221: Update comments

All checks have passed

2 successful checks
continuous-integration/codeship — Build succeeded
Details
continuous-integration/travis-ci/push — The Travis CI build passed
Details
08a676d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment