m_shige1979のときどきITブログ

プログラムの勉強をしながら学習したことや経験したことをぼそぼそと書いていきます

Github(変なおっさんの顔でるので気をつけてね)

https://github.com/mshige1979

bluemixのRetrieve and Rankを試す(curlでチュートリアル)

質問入力に対して回答候補とランク付けした回答候補を返す

らしい
簡単に考えてみて質問に対していくつかの回答候補を出すやつと思っていく

環境

ホスト

Mac

ツール

cURL
Python2.7.10

関連ツール

Bluemixのcfツール
※予めインストールしておく

デザインがちょっと変わったので全部コマンドラインでやる

なんかバグっぽい挙動していてうまく動かないんだもん(´・ω・`)
最後に全部削除しています。

サービス作成

ログイン
cf login -a https://api.au-syd.bluemix.net/

※国によってエンドポイントを変える必要があります。(今回はシドニー

サービス一覧を確認
cf marketplace
cf marketplace -s retrieve_and_rank
サービスを作成
cf create-service retrieve_and_rank standard rr_tutorial_01
資格情報を作成
cf create-service-key rr_tutorial_01 rr_tutorial_01-credentials
資格情報を確認
cf service-key rr_tutorial_01 rr_tutorial_01-credentials

{
 "password": "7kpbrZJ4n1Cq",
 "url": "https://gateway.watsonplatform.net/retrieve-and-rank/api",
 "username": "d2affa4a-a293-4bba-98c4-b03501299e66"
}

ディレクトリ準備

ディレクトリを作成して移動
mkdir r_r1
cd r_r1

クラスター作成

クラスター作成
curl -X POST -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
    "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters" -d ""

{
  "solr_cluster_id":"sc8ac937f6_4cc5_44a2_98b3_d249c853391f",
  "cluster_name":"",
  "cluster_size":"",
  "solr_cluster_status":"NOT_AVAILABLE"
}
クラスター作成を確認
curl -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
  "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/sc8ac937f6_4cc5_44a2_98b3_d249c853391f"

{
  "solr_cluster_id":"sc8ac937f6_4cc5_44a2_98b3_d249c853391f",
  "cluster_name":"",
  "cluster_size":"",
  "solr_cluster_status":"READY"
}

※READYがあること

コンフィグファイルのサンプルをダウンロード
wget https://github.com/watson-developer-cloud/doc-tutorial-downloads/raw/master/retrieve-and-rank/cranfield-solr-config.zip

※今回はサンプル

コンフィグファイルをアップロード
curl -X POST -H "Content-Type: application/zip" \
  -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
  "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/sc8ac937f6_4cc5_44a2_98b3_d249c853391f/config/example_config" \
  --data-binary @cranfield-solr-config.zip

{
  "message":"WRRCSR026: Successfully uploaded named config [example_config] for Solr cluster [sc8ac937f6_4cc5_44a2_98b3_d249c853391f].",
  "statusCode":200
}
コンフィグファイルを関連付け
curl -X POST \
  -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
  "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/sc8ac937f6_4cc5_44a2_98b3_d249c853391f/solr/admin/collections" \
  -d "action=CREATE&name=example_collection&collection.configName=example_config"

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <lst name="responseHeader">
    <int name="status">0</int>
    <int name="QTime">12332</int>
  </lst>
  <lst name="success">
    <lst name="10.176.142.253:6173_solr">
      <lst name="responseHeader">
        <int name="status">0</int>
        <int name="QTime">2541</int>
      </lst>
      <str name="core">example_collection_shard1_replica2</str>
    </lst>
    <lst name="10.176.39.48:5590_solr">
      <lst name="responseHeader">
        <int name="status">0</int>
        <int name="QTime">3026</int>
      </lst>
      <str name="core">example_collection_shard1_replica1</str>
    </lst>
  </lst>
</response>
ドキュメントファイル用のサンプルをダウンロード
wget https://github.com/watson-developer-cloud/doc-tutorial-downloads/raw/master/retrieve-and-rank/cranfield-data.json
ドキュメントファイルのアップロード
curl -X POST -H "Content-Type: application/json" \
  -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
  "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/sc8ac937f6_4cc5_44a2_98b3_d249c853391f/solr/example_collection/update" \
  --data-binary @cranfield-data.json

{
  "responseHeader":{
    "status":0,
    "QTime":1673
  }
}

ランカー用設定

グランドトゥルースファイルのサンプルをダウンロード
wget https://github.com/watson-developer-cloud/doc-tutorial-downloads/raw/master/retrieve-and-rank/cranfield-gt.csv
トレーニング用pythonスクリプトをダウンロード
wget https://github.com/watson-developer-cloud/doc-tutorial-downloads/raw/master/retrieve-and-rank/train.py
トレーニングスクリプト実施
python ./train.py -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
  -i cranfield-gt.csv \
  -c sc8ac937f6_4cc5_44a2_98b3_d249c853391f \
  -x example_collection \
  -n "example_ranker"

Input file is cranfield-gt.csv
Solr cluster is sc8ac937f6_4cc5_44a2_98b3_d249c853391f
Solr collection is example_collection
Ranker name is example_ranker
Rows per query 10
Generating training data...
Generating training data complete.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  304k    0   319  100  304k     24  23456  0:00:13  0:00:13 --:--:-- 17948
{
  "ranker_id":"c852bax18-rank-2794",
  "name":"example_ranker",
  "created":"2016-10-24T13:47:07.199Z",
  "url":"https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/rankers/c852bax18-rank-2794",
  "status":"Training",
  "status_description":"The ranker instance is in its training phase, not yet ready to accept rank requests"
}
Solrの検索クエリをブラウザで確認
https://d2affa4a-a293-4bba-98c4-b03501299e66:7kpbrZJ4n1Cq@gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/sc8ac937f6_4cc5_44a2_98b3_d249c853391f/solr/example_collection/select?q=what is the basic mechanism of the transonic aileron buzz&wt=json&fl=id,title

  "responseHeader":{
    "status":0,
    "QTime":16,
    "params":{
      "q":"what is the basic mechanism of the transonic aileron buzz",
      "fl":"id,title",
      "wt":"json"
    }
  },
  "response":{
    "numFound":164,
    "start":0,
    "docs":[
      {
        "id":"496",
        "title":[
          "a theory of transonic aileron buzz, neglecting viscous effects ."
        ]
      },
      {
        "id":"1268",
        "title":[
          "stable combustion of a high-velocity gas in a heated boundary layer ."
        ]
      },{
        "id":"903",
        "title":[
          "two dimensional transonic unsteady flow with shock waves ."
        ]
      },{
        "id":"520",
        "title":[
          "wing-tail interference as a cause of 'magnus' effects on a finned missile ."
        ]
      },{
        "id":"199",
        "title":[
          "measurement of two dimensional derivatives on a wing-aileron-tab system ."
        ]
      },{
        "id":"643",
        "title":[
          "an investigation of wing-aileron flutter using ground launched rocket models ."
        ]
      },{
        "id":"1072",
        "title":[
          "ignition and combustion in a laminar mixing zone ."
        ]
      },{
        "id":"262",
        "title":[
          "the formation of a blast wave by a very intense explosion ."
        ]
      },{
        "id":"1248",
        "title":[
          "an analytic extension of the shock-expansion method ."
        ]
      },{
        "id":"660",
        "title":[
          "the fundamental solution for small steady three dimensional disturbances to a two dimensional parallel shear flow ."
        ]
      }
    ]
  }
}
ランカーの作成を確認
curl -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
  "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/rankers/c852bax18-rank-2794"

{
  "ranker_id":"c852bax18-rank-2794",
  "name":"example_ranker",
  "created":"2016-10-24T13:47:07.199Z",
  "url":"https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/rankers/c852bax18-rank-2794",
  "status":"Available",
  "status_description":"The ranker instance is now available and is ready to take ranker requests."
}
ランク結果をブラウザで確認
https://d2affa4a-a293-4bba-98c4-b03501299e66:7kpbrZJ4n1Cq@gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/sc8ac937f6_4cc5_44a2_98b3_d249c853391f/solr/example_collection/fcselect?ranker_id=c852bax18-rank-2794&q=what is the basic mechanism of the transonic aileron buzz&wt=json&fl=id,title

{
  "responseHeader":{
    "status":0,
    "QTime":679
  },
  "response":{
    "numFound":164,
    "start":0,
    "maxScore":10.0,
    "docs":[
      {
        "id":"313",
        "title":[
          "on alternative forms for the basic equations of transonic flow theory ."
        ]
      },{
        "id":"496",
        "title":[
          "a theory of transonic aileron buzz, neglecting viscous effects ."
        ]
      },{
        "id":"520",
        "title":[
          "wing-tail interference as a cause of 'magnus' effects on a finned missile ."
        ]
      },{
        "id":"903",
        "title":[
          "two dimensional transonic unsteady flow with shock waves ."
        ]
      },{
        "id":"1268",
        "title":[
          "stable combustion of a high-velocity gas in a heated boundary layer ."
        ]
      },{
        "id":"1072",
        "title":[
          "ignition and combustion in a laminar mixing zone ."
        ]
      },{
        "id":"262",
        "title":[
          "the formation of a blast wave by a very intense explosion ."
        ]
      },{
        "id":"643",
        "title":[
          "an investigation of wing-aileron flutter using ground launched rocket models ."
        ]
      },{
        "id":"199",
        "title":[
          "measurement of two dimensional derivatives on a wing-aileron-tab system ."
        ]
      },{
        "id":"1242",
        "title":[
          "some considerations on the laminar stability of time-dependent basic flows ."
        ]
      }
    ]
  }
}

削除

クラスター削除
curl -i -X DELETE \
   -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
   "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/sc8ac937f6_4cc5_44a2_98b3_d249c853391f"

HTTP/1.1 200 OK
X-Backside-Transport: OK OK
Connection: Keep-Alive
Transfer-Encoding: chunked
Date: Mon, 24 Oct 2016 14:43:16 GMT
Content-Type: application/json
Server: -
Set-Cookie: Watson-DPAT=OQsYXi8uoqg6kh5E4EU2qrm9VwsHdb%2BiXd%2Fw4%2FddxcYCDDr54ieiie7IqLO4VCOpn%2BjFquCI6Qx%2BNnPQirRQjwUjQ76mGx1bNC%2F30R75gMzM7DH%2BVtn0C8awkB3Q%2FmbrpAtq78TdEXuq9PYewL11w1q1H5Pun8HIVnDZRUGIvTtAyLMYKtLU78puiczdvcAd2os68%2FuWnBUPsnXbAHe03VEvLK6TzadSseoyjzjmcVdgutrRpt1HcR983aY%2FYnpr8I7WkrTNNpHOVX%2FriCxD8bjorCwTuuEZR87DWYK7UEneB%2Bpe6jKXhnftRZcoezBv66xtpJq07%2FdCswBxKLgeH9PKtAUggYCqMOyn3vfEUdTyo0iD66P901Qg5gI9mWtE8MQfTbhWcsStpcWpN6R%2FnCrd5cKcvCxfT3qK4BzYi8ZbEsKAr3TLud504FM%2FZxfKIT8PY%2B%2BzWE%2B1HmvHFGozUQucznWgn0ZDEBcx%2BWwezYlcQrlfNtJfz6V7D%2BC%2BmUnP97BTfItDwYbJ0IQtDRaNj0or1jW%2FpxRfve7ZrmokSjzm%2F9PEXyHEELxHu5rW3hL%2Bpr8zF2i9vkusu8q40Jg%2BSwlny1tZpfpoiGtZjm8mMjkOW2Rx6iVd7v4KhEpgXaOqBBBqXI1Vnpq8b8yM1daJbpprza9ufrHvJFbBWsvqJs1x1pqoRlmPHGUvraRNCzw5v3Y44UZez8dMRrS18qWNTLuTtnk1dQIQBEpttRj%2FvmG1cQnbP8F5FRLIgs%2B4PoLFmvM21Jik7Gq9O2Kyfq5tbSIBUiV6%2BiLheafrHt%2Fhx9Ql5dMuXSgU5E0beWfh%2FLs3bKI9W8QFklA5gC03UJ7DtYljp4HS%2FTqbzSDF2OJV00Xs4Q1E%2F9RTd62b9HfPTMUSq2eHZBzoaivr41Ls8%2FjgebwILaO3Dvn5cFqKZqtxDbN3HwJsHiMmPWtyHE%2FLDbznhMorRaXMoR8vi0tKV9AdELroSVUwJe5b3DmlZughvrJy07WflTLtug%3D%3D; path=/retrieve-and-rank/api; secure; HttpOnly
X-Client-IP: 119.104.7.50
X-Global-Transaction-ID: 51546924
X-DP-Watson-Tran-ID: gateway-dp01-51546924

{
  "message":"WRRCSR023: Successfully deleted Solr cluster [sc8ac937f6_4cc5_44a2_98b3_d249c853391f].",
  "statusCode":200
}
ランカー削除
curl -X DELETE \
  -u "d2affa4a-a293-4bba-98c4-b03501299e66":"7kpbrZJ4n1Cq" \
  "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/rankers/c852bax18-rank-2794"

{}
サービス削除
cf delete-service-key rr_tutorial_01 rr_tutorial_01-credentials -f
cf delete-service rr_tutorial_01 -f

所感

今回はどのようなデータを用意すればよいのか不明だったのでチュートリアルをそのままやってみた。
質問に対して回答リストを取得している感じ、まだ、ランカーの順位付けや日本語対応などはわかっていないので
少しずつ調べていく。