Get Search Results from Google using Python

February 2nd, 2010 by Kevin | No Comments | Filed in programs

In my previous post, I have explained how we can use the Twitter API with Python to perform user based operations such as update Twitter status, check your home timeline, etc. Based on a similar method of using web services, we can query the Google search engine and get the search results using a Python script.

The Google search engine can be accessed by JavaScript using JQuery. However, non JavaScript users can query the Google search engine using the webservice

http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q="Your query string here"

The returned format is JSON(Javascript Object Notation).
The following Python script uses the http GET request to query the word “Kevin Rodrigues” in the Google search engine.

import urllib
import httplib2
import base64
import json

def google_search(phrase):
	"""Get google search results for the phrase"""

	url = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&" + urllib.parse.urlencode({'q':phrase});
	referer = "http://kevinrodrigues.com"

	h = httplib2.Http(".cache")
	resp, content = h.request(url, "GET", headers={'Referer': referer})

	if resp.status == 200:
		str_content = str(content)[2:-1]
		search_result = json.loads(str_content)
		print(search_result)
	else:
		print('Error')

if __name__ == '__main__':
	google_search('Kevin Rodrigues')

Find more information at http://code.google.com/apis/ajaxsearch/documentation/#fonje

 

Related Posts:

Tags:

Using the Twitter API with Python

January 30th, 2010 by Kevin | 1 Comment | Filed in programs

Twitter provides the Twitter API which can be used by developers to perform all the functions provided by a Twitter client such as update the twitter status, check the home timeline or the public timeline, retweet, update your profile etc.

To access all the twitter functionality, there are already user libraries created in most of the programming languages. You can find these libraries here.

Twitter API consists of three parts: two REST API and a Streaming API. The REST API are used to perform the user functionality like update status, view timeline, etc. The Streaming API are used for search and trend data. This tutorial is an attempt to explain in a simple way how we can use the Twitter REST API. I will be showing three examples for this. All of the other Twitter REST API can be used in a similar way.

I will be using Python 3.1 and the httplib2 library for the HTTP request/response.

1) Check your rate limit in Twitter

Rate limit is the number of times you can access certain twitter API or rather functionality in a particular hour. For example, you can update your twitter status with a rate limit of 150 requests per hour. Any attempt to use more than this will return an error. The rate limit can be based on the user (for authenticated API calls) or on the IP address (for unauthenticated API calls). Twitter allows increase in the rate limit to 20,000 for developers. Find more information about the twitter rate limit here.

The following Python script explains how to use the Twitter Rate Limit API to get the rate limit for the user.

def get_rate_limit(auth):                    #auth is a dictionary {'username': username, 'password': password}
	"""Get the rate limit for the user"""

	url = "http://twitter.com/account/rate_limit_status.json" #Twitter API for getting the rate limit (using JSON)

	h = httplib2.Http(".cache")                            #Get the http object (using httplib2)
	h.add_credentials(auth['username'], auth['password'])  #add the Twitter Basic Authentication (username, password)
	resp, content = h.request(url, "GET")                  #Send the http GET request for the rate limit

	if resp.status == 200:                                 #if the GET is successful
		str_content = str(content)[2:-1]               #Python byte to string conversion with b''. Remove it
		rate_limit_info = json.loads(str_content)      #Load the Rate Limit Information like remaining, hour limit
		return rate_limit_info['remaining_hits']       #Get the remaining hits for the hour
	else:
		return -1

The rate limit is not applied for using the rate limit twitter API. The rate limit Twitter API should be used to check if we have exceeded it or not before sending any further Twitter requests.

2) Update your Twitter Status

You can update your twitter status by using the Twitter Status API. This API uses a http POST method. So it does not deplete your rate limit as Twitter has set it mainly for the GET methods. However there is still a limit on the number of status updates you can perform in a day. Like the rate limit API, the Status Update API requires user authentication. Note that in Twitter there are two authentication methods available: Basic Authentication and OAuth Authentication. Twitter recommends using OAuth Authentication as this will be used in future versions. But for now it supports both. In this tutorial, I use Basic Authentication for it’s simplicity.

The following Python script explains how to send a Twitter status update.

def update_status(auth, status):            #auth is a dictionary {'username': username, 'password': password}
	"""Update my twitter status"""

	url = "http://twitter.com/statuses/update.json"        #Twitter API to update status

	h = httplib2.Http(".cache")
	h.add_credentials(auth['username'], auth['password']) #Twitter Basic Authentication

	resp, content = h.request(url, "POST", urllib.parse.urlencode({'status':status})) #Send the status

	if resp.status == 200:
		print(content)
	else:
		print('Error')

3) Get your Twitter Home Timeline

The Twitter Home Timeline API will return the status updates for the user, the followers and all of the retweets. You can get the updates based on filters like after a certain ID, before a certain ID, based on the number of updates or based on a particular page you want. Here I read the user home timeline based on the number of updates I want.

The following Python script explains how to get the home timeline for a user.

def get_home_timeline(auth):         #auth is a dictionary {'username': username, 'password': password}
	"""Get the home timeline for the user"""

	url = "http://api.twitter.com/1/statuses/home_timeline.json"   #Twitter Home Timeline API

	h = httplib2.Http(".cache")
	h.add_credentials(auth['username'], auth['password'])
	resp, content = h.request(url, "GET", urllib.parse.urlencode({'count':2}))   #Get 2 updates

	if resp.status == 200:
		str_content = str(content)[2:-1]
		str_content = str_content.replace('\\\\"', '')   #returned object not JSON serializable
		str_content = str_content.replace('\\', '')      #So modification to make it JSON serializable
		list_content = json.loads(str_content)
		print(list_content[0])                               #printing one status update
	else:
		print('Error')

There are many more Twitter API for your use and you can find all of them at the Twitter API website.

Have you developed any stand alone or web based application which uses the Twitter API or the API libraries? Which method of Authentication do you recommend, Basic Authentication or OAuth?

 

Related Posts:

Tags:

Getting BSE Stock Price by Market Data API

January 28th, 2010 by Kevin | No Comments | Filed in programs

In my post Using Python to get BSE price, I had mentioned a second way of getting the data from the Stock Market Data API

Unfortunately, I am unable to get the proper output even though the status code returned is 200 OK. Maybe I am using the API in a wrong way. So I have asked the guys at Stock Market Data API for some help.

The SMD Get Quote API follows simple HTTP GET syntax: start with a base URL and then add arguments and values after a question mark (?). For the most recent snapshot feed, you need to provide SMDAPIId, MarketId, SymbolId and Format parameters:

http://smdapi.co.cc/api/get_quote?apiid=APIId&market=MarketId&symbol=SymbolId&format=Format

Here is my Python code:

import httplib2

def get_bse(code):
     """Retrieve the BSE information based on the BSE Company Code entered"""

     APIId = "de8290b371"
     MarketId = "BSE"
     Format = "JSON"
     url_string = "http://smdapi.co.cc/api/get_quote?apiid=" + APIId + "&market=" + MarketId +
                   "&symbol=" + str(code) + "&format=" + Format
     print(url_string)
     referer = "http://kevinrodrigues.com"

     h = httplib2.Http(".cache")
     resp, content = h.request(url_string, "GET", headers={'Referer': referer})

     print(resp.status)
     print(content)

     with open('stock.json', 'w', encoding='utf8') as f:
          str_content = str(content)
          f.write(str_content)

if __name__ == '__main__':
     get_bse(500180)   #Get the stock information for HDFCBANK with code 500180

Could any of you help me get this right?

 

Related Posts:

Tags: