更新时间:2022-10-14 22:34:35
NEST的内部实现 Reindex
使用滚动将文档从一个索引移动到另一个索引。
应该是一个很好的起点。 b
$ b
下面你可以找到有趣的代码,从 github 。
var page = 0;
var searchResult = this.CurrentClient.Search< T>(
s => s
.Index(fromIndex)
.AllTypes()
.From(0)
.Size(size)
.Query(this._reindexDescriptor._QuerySelector?(q => q.MatchAll()))
.SearchType(SearchType.Scan)
。滚动(滚动)
);
if(searchResult.Total< = 0)
throw new ReindexException(searchResult.ConnectionStatus,index+ fromIndex +没有文档!
IBulkResponse indexResult = null;
do
{
var result = searchResult;
searchResult = this.CurrentClient.Scroll< T>(s => s
.Scroll(scroll)
.ScrollId(result.ScrollId)
);
if(searchResult.Documents.HasAny())
indexResult = this.IndexSearchResults(searchResult,observer,toIndex,page);
page ++;
} while(searchResult.IsValid&& indexResult!= null&&& indexResult.IsValid&& searchResult.Documents.HasAny());
您还可以查看集成测试 for 滚动
[测试]
public void SearchTypeScan()
{
var scanResults = this.Client.Search< ElasticsearchProject>(s => s
.From(0)
.Size(1)
.MatchAll()
Fields(f => f.Name)
.SearchType(SearchType.Scan)
.Scroll(2s)
);
Assert.True(scanResults.IsValid);
Assert.False(scanResults.FieldSelections.Any());
Assert.IsNotNullOrEmpty(scanResults.ScrollId);
var results = this.Client.Scroll< ElasticsearchProject>(s => s
.Scroll(4s)
.ScrollId(scanResults.ScrollId)
);
var hitCount = results.Hits.Count();
while(results.FieldSelections.Any())
{
Assert.True(results.IsValid);
Assert.True(results.FieldSelections.Any());
Assert.IsNotNullOrEmpty(results.ScrollId);
var localResults = results;
results = this.Client.Scroll< ElasticsearchProject>(s => s
.Scroll(4s)
.ScrollId(localResults.ScrollId));
hitCount + = results.Hits.Count();
}
Assert.AreEqual(scanResults.Total,hitCount);
}
I am using .From() and .Size() methods to retrieve all documents from Elastic Search results.
Below is sample example -
ISearchResponse<dynamic> bResponse = ObjElasticClient.Search<dynamic>(s => s.From(0).Size(25000).Index("accounts").AllTypes().Query(Query));
Recently i came across scroll feature of Elastic Search. This looks better approach than From() and Size() methods specifically to fetch large data.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
I looking for example on Scroll feature in NEST API.
Can someone please provide NEST example?
Thanks, Sameer
Internal implementation of NEST Reindex
uses scroll to move documents from one index to another.
It should be good starting point.
Below you can find interesting for you code from github.
var page = 0;
var searchResult = this.CurrentClient.Search<T>(
s => s
.Index(fromIndex)
.AllTypes()
.From(0)
.Size(size)
.Query(this._reindexDescriptor._QuerySelector ?? (q=>q.MatchAll()))
.SearchType(SearchType.Scan)
.Scroll(scroll)
);
if (searchResult.Total <= 0)
throw new ReindexException(searchResult.ConnectionStatus, "index " + fromIndex + " has no documents!");
IBulkResponse indexResult = null;
do
{
var result = searchResult;
searchResult = this.CurrentClient.Scroll<T>(s => s
.Scroll(scroll)
.ScrollId(result.ScrollId)
);
if (searchResult.Documents.HasAny())
indexResult = this.IndexSearchResults(searchResult, observer, toIndex, page);
page++;
} while (searchResult.IsValid && indexResult != null && indexResult.IsValid && searchResult.Documents.HasAny());
Also you can take a look at integration test for Scroll
[Test]
public void SearchTypeScan()
{
var scanResults = this.Client.Search<ElasticsearchProject>(s => s
.From(0)
.Size(1)
.MatchAll()
.Fields(f => f.Name)
.SearchType(SearchType.Scan)
.Scroll("2s")
);
Assert.True(scanResults.IsValid);
Assert.False(scanResults.FieldSelections.Any());
Assert.IsNotNullOrEmpty(scanResults.ScrollId);
var results = this.Client.Scroll<ElasticsearchProject>(s=>s
.Scroll("4s")
.ScrollId(scanResults.ScrollId)
);
var hitCount = results.Hits.Count();
while (results.FieldSelections.Any())
{
Assert.True(results.IsValid);
Assert.True(results.FieldSelections.Any());
Assert.IsNotNullOrEmpty(results.ScrollId);
var localResults = results;
results = this.Client.Scroll<ElasticsearchProject>(s=>s
.Scroll("4s")
.ScrollId(localResults.ScrollId));
hitCount += results.Hits.Count();
}
Assert.AreEqual(scanResults.Total, hitCount);
}