且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

在ElasticSearch NEST API中滚动示例

更新时间:2022-10-14 22:34:35

NEST的内部实现 Reindex 使用滚动将文档从一个索引移动到另一个索引。



应该是一个很好的起点。 b
$ b

下面你可以找到有趣的代码,从 github

  var page = 0; 
var searchResult = this.CurrentClient.Search< T>(
s => s
.Index(fromIndex)
.AllTypes()
.From(0)
.Size(size)
.Query(this._reindexDescriptor._QuerySelector?(q => q.MatchAll()))
.SearchType(SearchType.Scan)
。滚动(滚动)
);
if(searchResult.Total< = 0)
throw new ReindexException(searchResult.ConnectionStatus,index+ fromIndex +没有文档!
IBulkResponse indexResult = null;
do
{
var result = searchResult;
searchResult = this.CurrentClient.Scroll< T>(s => s
.Scroll(scroll)
.ScrollId(result.ScrollId)
);
if(searchResult.Documents.HasAny())
indexResult = this.IndexSearchResults(searchResult,observer,toIndex,page);
page ++;
} while(searchResult.IsValid&& indexResult!= null&&& indexResult.IsValid&& searchResult.Documents.HasAny());

您还可以查看集成测试 for 滚动

  [测试] 
public void SearchTypeScan()
{
var scanResults = this.Client.Search< ElasticsearchProject>(s => s
.From(0)
.Size(1)
.MatchAll()
Fields(f => f.Name)
.SearchType(SearchType.Scan)
.Scroll(2s)
);
Assert.True(scanResults.IsValid);
Assert.False(scanResults.FieldSelections.Any());
Assert.IsNotNullOrEmpty(scanResults.ScrollId);

var results = this.Client.Scroll< ElasticsearchProject>(s => s
.Scroll(4s)
.ScrollId(scanResults.ScrollId)
);
var hitCount = results.Hits.Count();
while(results.FieldSelections.Any())
{
Assert.True(results.IsValid);
Assert.True(results.FieldSelections.Any());
Assert.IsNotNullOrEmpty(results.ScrollId);
var localResults = results;
results = this.Client.Scroll< ElasticsearchProject>(s => s
.Scroll(4s)
.ScrollId(localResults.ScrollId));
hitCount + = results.Hits.Count();
}
Assert.AreEqual(scanResults.Total,hitCount);
}


I am using .From() and .Size() methods to retrieve all documents from Elastic Search results.

Below is sample example -

ISearchResponse<dynamic> bResponse = ObjElasticClient.Search<dynamic>(s => s.From(0).Size(25000).Index("accounts").AllTypes().Query(Query));

Recently i came across scroll feature of Elastic Search. This looks better approach than From() and Size() methods specifically to fetch large data.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html

I looking for example on Scroll feature in NEST API.

Can someone please provide NEST example?

Thanks, Sameer

Internal implementation of NEST Reindex uses scroll to move documents from one index to another.

It should be good starting point.

Below you can find interesting for you code from github.

var page = 0;
var searchResult = this.CurrentClient.Search<T>(
    s => s
        .Index(fromIndex)
        .AllTypes()
        .From(0)
        .Size(size)
        .Query(this._reindexDescriptor._QuerySelector ?? (q=>q.MatchAll()))
        .SearchType(SearchType.Scan)
        .Scroll(scroll)
    );
if (searchResult.Total <= 0)
    throw new ReindexException(searchResult.ConnectionStatus, "index " + fromIndex + " has no documents!");
IBulkResponse indexResult = null;
do
{
    var result = searchResult;
    searchResult = this.CurrentClient.Scroll<T>(s => s
        .Scroll(scroll)
        .ScrollId(result.ScrollId)
    );
    if (searchResult.Documents.HasAny())
        indexResult = this.IndexSearchResults(searchResult, observer, toIndex, page);
    page++;
} while (searchResult.IsValid && indexResult != null && indexResult.IsValid && searchResult.Documents.HasAny());

Also you can take a look at integration test for Scroll

[Test]
public void SearchTypeScan()
{
    var scanResults = this.Client.Search<ElasticsearchProject>(s => s
        .From(0)
        .Size(1)
        .MatchAll()
        .Fields(f => f.Name)
        .SearchType(SearchType.Scan)
        .Scroll("2s")
    );
    Assert.True(scanResults.IsValid);
    Assert.False(scanResults.FieldSelections.Any());
    Assert.IsNotNullOrEmpty(scanResults.ScrollId);

    var results = this.Client.Scroll<ElasticsearchProject>(s=>s
        .Scroll("4s") 
        .ScrollId(scanResults.ScrollId)
    );
    var hitCount = results.Hits.Count();
    while (results.FieldSelections.Any())
    {
        Assert.True(results.IsValid);
        Assert.True(results.FieldSelections.Any());
        Assert.IsNotNullOrEmpty(results.ScrollId);
        var localResults = results;
        results = this.Client.Scroll<ElasticsearchProject>(s=>s
            .Scroll("4s")
            .ScrollId(localResults.ScrollId));
        hitCount += results.Hits.Count();
    }
    Assert.AreEqual(scanResults.Total, hitCount);
}