json,python-2.7,elasticsearch,google-search-api , Parsing Google Custom Search API for Elasticsearch Documents


Parsing Google Custom Search API for Elasticsearch Documents

Question:

Tag: json,python-2.7,elasticsearch,google-search-api

After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've tried automatically loading, but not results.

Below is some example input that doesn't include things like id or index. I'm trying to focus on creating the correct data structure. I've tried modifying graph algorithms like depth-first-search but am running into problems with the different data structures.

Here's some example input:

# mock data structure
google = {"content": "foo", 
          "results": {"result_one": {"persona": "phone",
                                     "personb":  "phone",
                                     "personc":  "phone"
                                    },
                      "result_two": ["thing1",
                                     "thing2",
                                     "thing3"
                                    ],
                      "result_three": "none"
                     },
          "query": ["Taylor Swift", "Bob Dole", "Rocketman"]
}

# correctly formatted documents for _source of elasticsearch entry
correct_documents = [
    {"content":"foo"},
    {"results": ["result_one", "result_two", "result_three"]},
    {"result_one": ["persona", "personb", "personc"]},
    {"persona": "phone"},
    {"personb": "phone"},
    {"personc": "phone"},
    {"result_two":["thing1","thing2","thing3"]},
    {"result_three": "none"},
    {"query": ["Taylor Swift", "Bob Dole", "Rocketman"]}
]

Here is my current approach this is still a work in progress:

def recursive_dfs(graph, start, path=[]):
  '''recursive depth first search from start'''
  path=path+[start]
  for node in graph[start]:
    if not node in path:
      path=recursive_dfs(graph, node, path)
  return path

def branching(google):
    """ Get branches as a starting point for dfs"""
    branch = 0
    while branch < len(google):

        if google[google.keys()[branch]] is dict:

            #recursive_dfs(google, google[google.keys()[branch]])
            pass

        else:
            print("branch {}: result {}\n".format(branch,     google[google.keys()[branch]]))

        branch += 1

branching(google)

You can see that recursive_dfs() still needs to be modified to handle string, and list data structures.

I'll keep going at this but if you have thoughts, suggestions, or solutions then I would very much appreciate it. Thanks for your time.


Answer:

here is a possible answer to your problem.

def myfunk( inHole, outHole):
    for keys in inHole.keys():
        is_list = isinstance(inHole[keys],list);
        is_dict = isinstance(inHole[keys],dict);
        if is_list:
            element = inHole[keys];
            new_element = {keys:element};
            outHole.append(new_element);
        if is_dict:
            element = inHole[keys].keys();
            new_element = {keys:element};
            outHole.append(new_element);
            myfunk(inHole[keys], outHole);
        if not(is_list or is_dict):
            new_element = {keys:inHole[keys]};
            outHole.append(new_element);
    return outHole.sort();

Related:


Count function counting only last line of my list


python,python-2.7
Count function counting only last line of my list N = int(raw_input()) cnt = [] for i in range(N): string = raw_input() for j in range(1,len(string)): if string[j] =='K': cnt.append('R') elif string[j] =='R': cnt.append('R') if string[0] == 'k': cnt.append('k') elif string[0] == 'R': cnt.append('R') print cnt.count('R') if I am giving...

List of tuples from (a, all b) to (b, all a)


python,list,python-2.7,tuples
I am starting with a list of tuples (a,all b). I want to end with a list of tuples (b,all a). For example: FROM (a1,[b1,b2,b3]) (a2,[b2]) (a3,[b1,b2]) TO (b1,[a1,a3]) (b2[a1,a2,a3]) (b3,[a1] How do I do this using Python 2? Thank you for your help....

Sorting in Ruby on rails


ruby-on-rails,json,sorting
I'm very new to Ruby on rails. I try to edit the following api. I want to sorting "can_go" which is true are shown at the top of list. I added this row before sending data, but the result is still order by "user_id". user_infos.sort { |a, b| - (a['can_go']<=>b['can_go'])...

access the json encoded object returned by php in jquery


php,jquery,ajax,json
I want to post some data to php function by ajax, then get the encoded json object that the php function will return, then I want to get the information (keys and values) from this object, but I don't know how, here is my code: $.ajax({ url: "functions.php", dataType: "JSON",...

What is the difference between <> and == in python?


python-2.7
In [142]: (MON,TUE,WED,THR,FRI,SAT,SUN)<>range(7) Out[142]: True In [143]: (MON,TUE,WED,THR,FRI,SAT,SUN)==range(7) Out[143]: False ...

do calculation inside JSONArray in Java


java,arrays,json
I have a simple issue but cannot solve it as I am not very good at algorithm! I have an JSONArray in this form: [{"values":[{"time":1434976493,"value":"50"}, {"time":1434976494,"value":"100"}],"counter":"counter1"}, {"values":[{"time":1434976493,"value":"200"}, {"time":1434976494,"value":"300"}],"counter":"counter2"}, {"values":[{"time":1434976493,"value":"400"}, {"time":1434976494,"value":"600"}],"counter":"total"}] What I want to do is to get the integer value for counter 1 and counter 2 and then divide...

Check for duplicates in JSON


javascript,jquery,json,duplicates
I have a json object: var object1 = [{ "value1": "1", "value2": "2", "value3": "3", }, { "value1": "1", "value2": "5", "value3": "7", }, { "value1": "6", "value2": "9", "value3": "5", }, { "value1": "6", "value2": "9", "value3": "5", }] Now I want to take each record out of that...

Parse text from a .txt file using csv module


python,python-2.7,parsing,csv
I have an email that comes in everyday and the format of the email is always the same except some of the data is different. I wrote a VBA Macro that exports the email to a text file. Now that it is a text file I want to parse the...

convert from json to array


javascript,php,json
I am trying to get data to display in a table. I don't know what I am doing wrong, but when I get the data from my page it is an array of single characters. I could parse this myself but would prefer to know what I am doing wrong....

How do I copy a row from one pandas dataframe to another pandas dataframe?


python,python-2.7,pandas,dataframes
I have a dataframe of data that I am trying to append to another dataframe. I have tried various ways with .append() and there has been no successful way. When I print the data from iterrows. I provide 2 possible ways I tried to solve the issue below, one creates...

How to extract efficientely content from an xml with python?


python,xml,python-2.7,pandas,lxml
I have the following xml: <?xml version="1.0" encoding="UTF-8" standalone="no"?><author id="user23"> <document><![CDATA["@username: That boner came at the wrong time ???? http://t.co/5X34233gDyCaCjR" HELP I'M DYING ]]></document> <document><![CDATA[Ugh ]]></document> <document><![CDATA[YES !!!! WE GO FOR IT. http://t.co/fiI23324E83b0Rt ]]></document> <document><![CDATA[@username Shout out to me???? ]]></document> </author> What is the most efficient...

php array returns undefined but print_r shows otherwise [duplicate]


php,arrays,json,undefined
This question already has an answer here: PHP: “Notice: Undefined variable” and “Notice: Undefined index” 11 answers foreach($jsonmediationinfo as $value1) { echo $value1['mstatus']; print_r($jsonmediationinfo); } Output: 1Array ( [0] => Array ( [mstatus] => 1 [mhearingnum] => first [mminutes] => adakjaflafjlarjkelfkalfkd;la ) [1] => Array ( [mhearingnum] => second...

Codeigniter Select JSON, Insert JSON


json,codeigniter,select,insert,routing
I have very simple users database: user_id, user_name, user_email My model this: class Users extends CI_Model { private $table; private $table_fields; private $table_fields_join; function __construct() { parent::__construct(); $this->table = 'users'; $this->table_fields = array( $this->table.'.user_id', $this->table.'.user_name', $this->table.'.user_email' ); $this->table_fields_join = array(); } function select(){ $this->db->select(implode(', ', array_merge($this->table_fields, $this->table_fields_join)));...

Serializing a java bean into a cookie: Is it bad?


java,json,cookies
In the organization that i work for, there was a serious debate about the following. Scenario: There is a POJO with 6 different properties all are of type Strings. These values need to be persisted as cookies so that it can be picked back when someone does a booking on...

Is there a way to say for every x values, do this?


python,python-2.7
Before I start, I am new to Python, so any low-level description would be incredibly helpful! I have a list of lets say 60 values (representing one hour, from 8:00-9:00) and I want to run average, maximums, minimums, and standard deviation for each set of 15. (I already have the...

Replacing elements in an HTML file with JSON objects


javascript,json,replace
I am writing some Javascript to: Read in and Parse a JSON file Read in an HTML file (created as a template file) Replace elements in the HTML file with elements from the JSON file. It is only replacing the first element obj.verb. Can someone please help me figure out...

Nested JSON structure to build side-by-side d3.js charts


javascript,json,d3.js
I'm currently stuck on how to traverse a JSON structure (I created) in order to create side-by-side donut charts. I think I've created a bad structure, but would appreciate any advice. I'm working from Mike Bostock's example here: http://bl.ocks.org/mbostock/1305337 which uses a .csv file for source data. In that example...

'utf8' codec can't decode byte 0xf3


python,json,character-encoding
I am using python 2.7 to read the json file. my code is import json from json import JSONDecoder import os path=os.path.dirname(os.path.abspath(__file__))+'/json' print path for root, dirs, files in os.walk(os.path.dirname(path+'/json')): for f in files: if f.lower().endswith((".json")): fp=open(root + '/'+f) data = fp.read() print data.decode('utf-8') But I got the following error....

Pass function call as a function argument


python,python-2.7,python-3.x
Code: def function1(a,b): return a-1,b-1 def function2(c,d): return c+1,d+1 print function1(function2(1,2)) Error: Traceback (most recent call last): File "C:\Users\sony\Desktop\Python\scripts\twitter_get_data.py", line 6, in <module> print function1(function2(1,2)) TypeError: function1() takes exactly 2 arguments (1 given) [Finished in 0.1s with exit code 1] Why the above error? ...

How to check for multiple attributes in a list


python,python-2.7
I am making a TBRPG game using Python 2.7, and i'm currently making a quest system. I wanted to make a function that checks all of the quests in a list, in this case (quests), and tells you if any of of the quests in the list have the same...

Parse JSON output from AlamoFire


json,swift,nsurlrequest
I am using the latest Alamofire to manage a GET http request to my server. I am using the following to GET and parse the JSON: Alamofire.request(.GET, "*******") .responseJSON {(request, response, JSON, error) in if let statusesArray = JSON as? NSArray{ if let aStatus = statusesArray[0] as? NSDictionary{ //OUTPUT SHOWN...

Create n:m objects using json and sequelize?


javascript,json,node.js,sequelize.js
I am trying to learn sequelize, but am having trouble getting a n:m object created. So far I have my 2 models that create 3 tables (Store, Product, StoreProducts) and the models below: models/Store.js module.exports = (sequelize, DataTypes) => { return Store = sequelize.define('Store', { name: { type: DataTypes.STRING, },...

How can I resolve my variable's unexpected output?


django,python-2.7
I have a variable in django named optional_message. If I debug the variable then it says Swenskt but when I try to print the variable on my page the following comes out: (u'Swenskt',) and the variable can't be tested for its length etc. What should I do if I only...

Response 200 OK but Jquery shows error?


javascript,jquery,json
I am stuck in kind of a weird problem where I am trying to make a JSONP request to the Open Weather history API to get the weather information for the previous years on the a particular date. fetchHistoryUrl = 'http://78.46.48.103/data/3.0/days?day=0622&mode=list&lat=39.77476949&lon=-100.1953125'; $.ajax({ method: 'get', url: fetchHistoryUrl, success: function(response){ console.log(response); },...

Parsing Google Custom Search API for Elasticsearch Documents


json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

Python split by comma delimiter and strip


python,python-2.7
I am trying to open a file in python in read mode then write striped and split data to an output file. I am unsure how to split and strip the same data. Do I need to create a different line? Do I need to write the data out first?...

why i don't get return value javascript


javascript,jquery,html,json,html5
When i debug my code i can see i have value but i don't get value to createCheckBoxPlatform FN function createCheckBoxPlatform(data) { var platform = ""; $.each(data, function (i, item) { platform += '<label><input type="checkbox" name="' + item.PlatformName + ' value="' + item.PlatformSK + '">' + item.PlatformName + '</label>' +...

Identify that a string could be a datetime object


python,regex,algorithm,python-2.7,datetime
If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

Call function on Server from iOS app - Objective C


ios,objective-c,json,server,backend
If you are familiar with Parse.com's Javascript SDK, this is what I am trying to do for my own server for my iOS app (Objective-c). I want to be able to send some a string to the function that is on my server, have the server run its function and...

Fatal error catched by register_shutdown_function and update json_encode


php,json,fatal-error
I can catch a fatal error with register_shutdown_function('shutdown') function shutdown(){ $error = error_get_last(); $result['message'] = $error['message']; } and I have an echo json_encode($result); Unfortunately this echo json_encode shows nothing because json is not updated with the message of the fatal error. What can I do to fix this?...

Deserializing JSON into c# class


c#,json,couchdb
I have a JSON in the following format (from couchDB view) {"rows":[ {"key":["2015-04-01","524","http://www.sampleurl.com/"],"value":1}, {"key":["2015-04-01","524","http://www.sampleurl2.com/"],"value":2}, {"key":["2015-04-01","524","http://www.sampleurl3.com"],"value":1} ]} I need to create a "service" to get this data from couchDB and insert it on SQL Server (for generating reports..) in a efficient way. My first bet was to bulk insert this json...

Link to another resource in a REST API: by its ID, or by its URL?


json,api,rest,api-design,hateoas
I am creating some APIs using apiary, so the language used is JSON. Let's assume I need to represent this resource: { "id" : 9, "name" : "test", "customer_id" : 12, "user_id" : 1, "store_id" : 3, "notes" : "Lorem ipsum example long text" } Is it correct to refer...

Strange Behavior: Floating Point Error after Appending to List


python,python-2.7,behavior
I am writing a simple function to step through a range with floating step size. To keep the output neat, I wrote a function, correct, that corrects the floating point error that is common after an arithmetic operation. That is to say: correct(0.3999999999) outputs 0.4, correct(0.1000000001) outputs 0.1, etc. Here's...

how to fetch a column in browse_record_list in orm browse method in openERP


python-2.7,orm,openerp-7
I'm beginner in openERP. I'm trying to get a column in a table. While using ORM browse method and iterating that object i got the result in browse_record_list as browse_record(table.name,21). I want to fetch that particular id 21 alone through that browse method but instead im getting same browse_record as...

JQuery mutiple post in the same time


javascript,php,jquery,json
I have three functions ,and every function post to special php page to get data.. and every function need some time because every php script need some time .. function nb1() { $.post("p1.php", { action: 1 }, function(data) { console.log(data); }, "json") .fail(function(data) { console.log("error"); }); } function nb2() {...

Why i get can not resolve method error in class android?


android,json
I use this tutorial: JSON and part of code to this: protected String doInBackground(String... urls) { TextView lbl=(TextView) findViewById(R.id.provCODE); person = new Person(); person.setName(lbl.getText().toString()); //person.setCountry("1853"); //person.setTwitter("1892"); return POST(urls[0],person); } but in the this line: person.setName(lbl.getText().toString()); i get this error: Can not resolve method. How can i solve that?what happen? my...

How to rearrange CSV / JSON keys columns? (Javascript)


javascript,json,csv,papaparse
I am converting a JSON object array to CSV using Papa Parse JavaScript Library. Is there a way to have the CSV columns arranged in a certain way. For e.g; I get the column as: OrderStatus, canOp, OpDesc, ID, OrderNumber, FinishTime, UOM, StartTime but would like to be arranged as:...

Use JSON file to insert data in database


javascript,json,mongodb,meteor,data
I'm using my JSON file like this to insert data in my collection : var content = JSON.parse(Assets.getText('test.json')); console.log('inserting...'); Profiles.insert({ user: id, data:content }; But I would like to have a "data's tree" like that : [ user: "rtegert23423131", firstname:"test", surname:"test2", // ... ] Not like that : [ user:...

Pandas Dataframe Complex Calculation


python,python-2.7,pandas,dataframes
I have the following dataframe,df: Year totalPubs ActualCitations 0 1994 71 191.002034 1 1995 77 2763.911781 2 1996 69 2022.374474 3 1997 78 3393.094951 I want to write code that would do the following: Citations of currentyear / Sum of totalPubs of the two previous years I want something to...

String comparison in AngularJS


javascript,json,angularjs,ionic-framework,string-comparison
I'm trying to compare two strings in AngularJS, and I've seen examples online. As I understand it, you can use angular.equals(str1, str2), you can use ===, you can use == if you're sure that both are strings... I've tried all three, but I don't get the result. Something must be...

Slicing a Python OrderedDict


python-2.7,slice,ordereddictionary
In my code I frequently need to take a subset range of keys+values from a Python OrderedDict (from collections package). Slicing doesn't work (throws TypeError: unhashable type) and the alternative, iterating, is cumbersome: from collections import OrderedDict o = OrderedDict([('a', 1), ('b', 2), ('c', 3), ('d', 4)]) # want to...

Uncaught error: Invalid type for google table column


javascript,json,google-maps,google-visualization
I recently started using google visualization due to its simplicity. But while implementing in my recent project i seem to have run into a bit of trouble. The program shown is used to pull data from a JSON object and utilize it to display data in the google tables. Now...

KendoUI Grid - Complex JSON with inconsistent keys


javascript,json,kendo-ui,kendo-grid
I know this is probably been asked before, but having tried to find an answer - I am guessing either I am not comprehending some of the answers right, or I am looking at the problem all wrong. I am using a complex SLC loopback query - and the api...

Deserializing Json data to c# for use in GridView - Error Data source is an invalid type


c#,json,gridview,serialization,.net-4.5
I am relatively new to working with C# (v5.0) and Json data (Newtonsoft.Json v6.0) and am seeking assistance in resolving an error when attempting to populate a .Net 4.5 GridView control. My sample Json data as returned by a web service is: { PersonDetails: [ { Book: "10 ", FirstName:...

Python Popen - wait vs communicate vs CalledProcessError


python,python-2.7,error-handling,popen
Continuing from my previous question I see that to get the error code of a process I spawned via Popen in python I have to call either wait() or communicate() (which can be used to access the Popen stdout and stderr attributes): app7z = '/path/to/7z.exe' command = [app7z, 'a', dstFile.temp,...

How to remove structure with python from this case?
python,python-2.7
How to remove "table" from HTML using python? I had case like this: paragraph = ''' <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quidem molestiae consequuntur officiis corporis sint.<br /><br /> <table> <tr> <td> text title </td> <td> text title 2 </td> </tr> </table> <p> lorem ipsum</p> ''' how...

I need to make sure that only certain characters are in a list?


python,regex,string,list,python-2.7
I have this to get input and put it in a list: def start(): move_order=[raw_input("Enter your moves: ").split()] And I only want the characters A, D, S, C, H (it's for a game >_>) to be allowed. I've tried using the regular expressions stuff: if re.match('[ADSCH]+', [move_order]) is False: print...

Insert data in collection at Meteor's startup


javascript,json,meteor,data,startup
I would like to insert data at Meteor's startup. (And after from a JSON file) At startup, I create a new account and I would like to insert data and link it to this account once this one created. This is the code that creates the new account at startup:...

Can't save json data to variable (or cache) with angularjs $http.get


json,angularjs,web-services,rest
I have weird angularjs problem. I'm trying to fetch data from Rest Webservice. It works fine, but I can't save json data to object. My code looks like: services.service('customerService', [ '$http', '$cacheFactory', function($http, $cacheFactory) { var cache = $cacheFactory('dataCache'); var result = cache.get('user'); this.getById = function(id){ $http.get(urlList.getCustomer + id).success(function(data, status,...